Archivierung im IBM DB2 HOST Bereich Hans-Jürgen Schmidt hjs@de.ibm.com Tel. 07034-15-3282 Mobile 0160-8812236

Agenda Warum Archivierung Definition Archivierung Wie werden Daten heute archiviert ? Terminologie DB2 Data Archive Expert for z/OS IBM SAP Archivierungs Lösung

Warum Archivierung In den IT Bereichen wächst das Datenvolumen überdurchschnittlich Die Anzahl der Datenbanken wächst expotential Gale Research Inc.

Warum Archivierung Inaktive Daten existieren überall Transaktionen Data Warehouses e-mail etc. Inaktive Daten werden noch benötigt, nur die Zugriffswahrscheinlichkeit ist geringer als bei aktiven Daten Warum sollen inaktive Daten gehalten werden ? Gesetzliche Gründe Geschäftliche Gründe Trend Analysen Auswertungen von Vergangenheitsdaten Reaktivierung Inactive data means that the frequency or probability of access to the data is low. Inactive (or “dormant data”, as coined by Bill Inmon) is not unusable – unusable data should be deleted. We are simply referring to data that is probably historical and for which there is either a business or legislative need to keep. An interesting phenomenon described by Bill Inmon is that as a data store grows in size, so does the percentage of inactive data. From personal experience, I see this phenomenon on my laptop computer. A few years ago, I was much more diligent in managing my inactive data when I had a three gigabyte hard drive. Today, with an 80 GB hard drive, I spend much less time managing my inactive data and I know that my percentage of inactive (and unusable!) data is much higher than in the past.

Warum Archivierung Es ist ein Phänomen, dass bei steigendem Datenvolumen auch der Prozentsatz der inaktiven Daten überprozentual zunimmt 10 GB 10 % inaktiv 100 GB 40 % inaktiv 2 TB 70 % inaktiv 50 TB 85 % inaktiv *Inmon, Bill. Webinar on data warehousing and dormant data. http://archive.mshow.com/61039 Inactive data means that the frequency or probability of access to the data is low. Inactive (or “dormant data”, as coined by Bill Inmon) is not unusable – unusable data should be deleted. We are simply referring to data that is probably historical and for which there is either a business or legislative need to keep. An interesting phenomenon described by Bill Inmon is that as a data store grows in size, so does the percentage of inactive data. From personal experience, I see this phenomenon on my laptop computer. A few years ago, I was much more diligent in managing my inactive data when I had a three gigabyte hard drive. Today, with an 80 GB hard drive, I spend much less time managing my inactive data and I know that my percentage of inactive (and unusable!) data is much higher than in the past.

Warum Archivierung Das Problem  inaktive Daten erzeugen Kosten Wachsende Besorgnis der IT Manager : wie gehe ich mit steigendem Datenvolumen um Kapazitätserweiterung ist nicht immer der effizienteste Weg mit Wachstum umzugehen Ungezügeltes Datenwachstum beeinträchtigt das IT Umfeld in folgenden Bereichen : Performance Kosten für Prozessor, Speicher Handhabung / Wartung Inactive data means that the frequency or probability of access to the data is low. Inactive (or “dormant data”, as coined by Bill Inmon) is not unusable – unusable data should be deleted. We are simply referring to data that is probably historical and for which there is either a business or legislative need to keep. An interesting phenomenon described by Bill Inmon is that as a data store grows in size, so does the percentage of inactive data. From personal experience, I see this phenomenon on my laptop computer. A few years ago, I was much more diligent in managing my inactive data when I had a three gigabyte hard drive. Today, with an 80 GB hard drive, I spend much less time managing my inactive data and I know that my percentage of inactive (and unusable!) data is much higher than in the past.

Warum Archivierung Wo liegen die Kosten ? Performance Zusätzliche I/O Operationen Zusätzliche Prozessor Operationen Hardware Kosten Prozessor Platten Datenredundanz (eingeschränkte Normalisierung) Indexorganisation Handhabung / Wartung REORG / UNLOAD / BUILD INDEX etc. Beeinträchtigung Batch Window Unzufriedenheit externer/interner Kunden Antwortzeitverhalten der Anwendung Inactive data means that the frequency or probability of access to the data is low. Inactive (or “dormant data”, as coined by Bill Inmon) is not unusable – unusable data should be deleted. We are simply referring to data that is probably historical and for which there is either a business or legislative need to keep. An interesting phenomenon described by Bill Inmon is that as a data store grows in size, so does the percentage of inactive data. From personal experience, I see this phenomenon on my laptop computer. A few years ago, I was much more diligent in managing my inactive data when I had a three gigabyte hard drive. Today, with an 80 GB hard drive, I spend much less time managing my inactive data and I know that my percentage of inactive (and unusable!) data is much higher than in the past.

Definition der Archivierung Transfer von inaktiven Daten zu einem Archiv, auf das bei Bedarf schnell zugegriffen werde kann Archive/Retrieve Prozesse Häufig ist die Archivierung ein Bestandteil in der Backup/Recovery Strategie eines Unternehmens Unterschied Archive / Backup Archiv enthält selektive Daten Archive sind Anwendungs orientiert Backup ist Datenspeicher orientiert Mit DB2 z/OS V8 wurde BACKUP/RESTORE in die DB2 Engine übernommen Anforderungen Selektionskriterium soll kompletter SQL Umfang sein Schneller Retrieve Prozess Welche Kosten werden akzeptiert ? Probleme Strukturänderungen der Daten Neue Versionen der Anwendung Inactive data means that the frequency or probability of access to the data is low. Inactive (or “dormant data”, as coined by Bill Inmon) is not unusable – unusable data should be deleted. We are simply referring to data that is probably historical and for which there is either a business or legislative need to keep. An interesting phenomenon described by Bill Inmon is that as a data store grows in size, so does the percentage of inactive data. From personal experience, I see this phenomenon on my laptop computer. A few years ago, I was much more diligent in managing my inactive data when I had a three gigabyte hard drive. Today, with an 80 GB hard drive, I spend much less time managing my inactive data and I know that my percentage of inactive (and unusable!) data is much higher than in the past.

Wie werden Daten heute archiviert ? Meistens mit Eigenentwicklungen Zugeschnitten für jede einzelne Anwendung Wartung sehr teuer Einige Firmen archivieren keine Daten Daten werden gelöscht und es wird darauf vertraut dass sie nicht mehr benötigt werden Es werden immer mehr Ressourcen zur Verfügung gestellt Kostenexplosion bei der IT Infrastruktur Die gesetzlichen Regularien zur Datenaufbewahrung erfordern eine stabile Archivierungs Lösung Archivierung läuft unter Kontrolle der Anwendungen System Programmierer / DBA’s stellen die Infrastruktur zur Verfügung

Wie werden Daten heute archiviert ? Ergebnisse aus Umfragen Von 479 Befragten haben 63 % eine Archivinfrastruktur installiert, von den 37 % die mit Nein antworteten befassen sich 80 % mit dem Thema Neben gesetzlichen Vorgaben spielen auch interne Überlegungen (Fachabteilungen) eine Rolle bei der Archivierung Verwaltung/Personal/Vertrieb : 41 % Produktion/Entwicklung : 34 % Finanz : 19 % Gemeinsamer Request : 6 % Medium für Archive Bestehende Archive : Platte = 38 %, Band = 39 % Neue Archive : Platte = 53 %, Band = 25 % Bedeutung Optischer Speicher : Rückgang von 23 % auf 16 % Kosten : Platte = 25-35 US $, Band = 3-5 US $, Optisch = 5 US $ Mehr als 50 % der Befragten schätzt die Grösse ihres Archivs auf über 1 Terabyte

retrieve (most common) Terminologie Active Data archive Archive Data Terminologie Aktive Daten Archivierte Daten Typen von Archiven Archivierungs Tabellen: direkter Zugriff via SQL Archivierungs Dateien Flat Files (kein DB2 Format) WORM (write once read multiple) Speicherung : Platte, Band, Optisches Medium Archiv Metadaten – Daten über das Archiv Wer, Wann, Wohin (Speichermedium), Was (SQL WHERE Clause), Warum (manuell, periodisch, geplant) Pro Archiv retrieve retrieve (most common) Temp Active data – data that is used on a day to day basis. Archive data – data that is accessed infrequently (sometimes never) Table archives – archived data that reside in tables (in a DBMS world) File archives – archived data that reside in files Archives can contain related data from more than one table. These tables will usually be related thru some sort of RI either DBMS maintained RI or application managed RI.

DB2 Data Archive Expert for z/OS M Storage Management System HSM TSM

DB2 Data Archive Expert for z/OS Components An ISPF interface Specifications Execution A callable Application Programming Interface (API) Archive metadata DB2 Grouper

DB2 Data Archive Expert for z/OS To table Retain SQL access Save disk space (by not requiring fast index access to archived data): Archive is a copy and a delete operation The deletes can be deferred Product insures that no data is deleted that is not safely in the archive Index 40GB Saved 20% in disks costs! (if 50% of the data is inactive) After Archive 100GB 20 Index Data 60GB Data 30 Data 30 <table_name> <table_name> <arc_table>

DB2 Data Archive Expert for z/OS To file No SQL access, but cheaper media options available Consider media format when upgrading storage devices Tape is still about 5 times cheaper than disk Can use products like a 4GL to access data directly DFHSM can be used to automatically migrate files to cheaper storage media Like archive tables, rows are timestamped in order to know when they were archived WORM support

DB2 Data Archive Expert for z/OS To both (table and then file)! Multi-tier archive: 91-364 day archive (table) >364 day archive (file) Active Data High Low Very Low

DB2 Data Archive Expert for z/OS Retrieve To temporary table or source Usually on demand -- could be programmatically controlled Selective from Single archive Multiple archives Keeps track of the archives (metadata) When (timestamp) Who (userid) What (What was archived) SQL WHERE clause Definitions (DDL) Where (Location of the archived data) Why (Scheduled, periodic, or manual) How (via DB2 Unload)

DB2 Data Archive Expert for z/OS By allowing data to be archived to tables, apps can access the archive Limitations apply; not for everyone Uses UNION in VIEW function in DB2 for z/OS Version 7 Index Application still reads from <table_name> Data Data <act_table> <arc_table> Application reads need to supply a date_col predicate to avoid access to <arc_table> Updates need to point to the act_table Prevent updates to arc _table CREATE VIEW <table_name> AS SELECT * FROM <act_table> WHERE DATE_COL >= (CURRENT_DATE-365 DAYS) UNION ALL SELECT * FROM <arc_table> WHERE DATE_COL < (CURRENT_DATE-365 DAYS)

DB2 Grouper A common component of several DB2 IBM Tools, first delivered in Data Archive Expert The problem There are many relationships between DB2 objects, such as tables, in a business application -- Some of these relationships can be discovered easily, while others cannot be easily found The objective Enable the location, augmentation, and management of this information as the basis for consistent data management activities The solution Grouper is a component for discovering, recording, and managing groups of related objects (tables) that comprise a business application

DB2 Grouper Maintain sets of groupings Specify global relationships (non-enforced RI) Kick off dynamic (Units of work) relationship discoveries Specify input values to the group discovery process Group discovery scope (starting points) Boundary objects Additional relationships Launch the group discovery process Browse through the groupings

Interfaces Interfaces ISPF: Create/work with specifications Callable programming interface (SQL CALL statement): stored procedures

IBM’s DB2 for z/OS Tools Produkt Portfolio Application Management Database Administration Utilities DB2 Utilities Suite DB2 High Performance Unload DB2 Automation Tool DB2 Bind Manager DB2 Data Archive Expert DB2 Path Checker DB2 Table Editor DB2 Test Database Generator DB2 Web Query Tool DB2 and IMS Encryption DB2 Administration Tool DB2 Object Compare Performance Management Recovery DB2 Performance Manager DB2 Bufferpool Analyzer DB2 Performance Expert DB2 Query Monitor DB2 SQL Performance Analyzer DB2 Archive Log Accelerator DB2 Change Accumulation Tool DB2 Log Analysis Tool DB2 Object Restore Tool Replication DB2 Data Propagator DB2 II Classic Federation

Authors: Paolo Bruni, Walter Huth, Ernie Mancill, Iain Warnock SG24-7080 Authors: Paolo Bruni, Walter Huth, Ernie Mancill, Iain Warnock