Axel Herbst Performance, Data Management & Scalability SAP AG

Präsentation zum Thema: "Axel Herbst Performance, Data Management & Scalability SAP AG"—  Präsentation transkript:

1 Axel Herbst Performance, Data Management & Scalability SAP AG
Data Archiving @ SAP Axel Herbst Performance, Data Management & Scalability SAP AG

2 The Life Cycle of SAP Data
DATABASE FILE SYSTEM S T O R A G E S Y S T E M t Residence time Data access Audits Deletion Business complete Non changeable Creation Importance t

3 Legal Compliance - Record Retention Periods
Pharmac./ Life Sci. (21 CFR Part 11) 2 years after commercial release - Records relating to the manufacturing, processing, packing of food 3 years after distribution - Records relating to the manufacturing, processing, packing of drugs and pharmaceuticals 5 years after end of manufacturing or product - Records relating to the manufacturing of biological products 5 years - All hospitals must retain records in originally or legally produced form Healthcare (HIPAA) 21 years+ (perhaps for life) - Medical records for minors from birth to 21 2 years after patient’s death - Medical records 3 years - Financial statements Financial Services (SEC 17a-4) End-of-life of enterprise - Member registration for broker/dealers End of account +6 years - Trading account records 30 years after completion of audit - Employee and medical records of individuals exposed to toxic substances OSHA Sarbanes Oxley 4 years after completion - Original correspondences from financial audits or publicly-traded corporations 5 10 15 20 4 3 2 1 Minimum Retention Period on Compliant Media (Years) Source: Enterprise Storage Group, May 2003

4 Architecture of an ABAP-Based SAP Component
SAP GUI Web Server Front End Application (e.g. Word) SAP GUI SAP GUI SAP GUI SAP GUI SAP GUI SAP GUI Gateway Dispatcher Dispatcher Gateway Shared Shared Memory Memory DIA BTC SPO DIA ENQ Enqueue and WP WP WP and WP WP Table Buffers Buffers DBMS DBWP DBWP DBWP DBWP Database

5 Size of Database Survey Results from the American SAP User Group ASUG
(Archiving Track) 1999 2002 12 10 8 8 7 7 6 5 5 5 4 2 1 1 < 300 GB TB >1.5 TB GB GB GB

6 Today's requirements to RDBMS in SAP Systems
Requirements concerning business objects Several TB net data in RDBMS Size of a business object between some KB and some 100 MB Typically at least 3 search criteria Business partner Material/product/service/... Date Need to access business objects for several (even 10+) years Requirements concerning OLTP 1000 database transactions per second 50 MB/s write access to database files Database transactions change hundreds of tables (dozens of business objects) Recovery in less than 15 minutes 7*24 hours (2 hours planned down time a month)

7 System Availability - Downtime Costs Per Hour
Downtime Losses by Application Typical Loss Per Hour of Unplanned Downtime (US$) Financial/Trading $2,400,000 Supply Chain $600,000 ERP CRM $480,000 E-Commerce E-Business Business Application $300,000 Database Messaging $60,000 Infrastructure $42,000 Source: Storage Magazine

8 Use of Resources - Storage Savings
Data Records 10 GB Mirrors 20 GB System Copies 80 GB Archiving Compression to 20 % 2 GB 2 GB Backup to Tape Total savings in this scenario: 97.5%

9 Long-Term Issues Costs Long-term reuse
Efficient archive format: compact, indexable Inexpensive storage system/media Easy administration (low TCO) Long-term reuse Appropriate search/query capability to find archived data Readability of archive Replaceable/maintainable/migrateable hardware and software components Interpretability (self-contained and self-describing format, schema) Application connectivity (open interface/protocol) Application integration Enough semantics/context for future use Archival of referenced documents Security Write once guarantee (-> for legal, tax and audit reasons) Authorization, authentication, nonrepudiation, privacy, encryption Operating (archiving in production environment) Mass data transfer from DB to archive Automation, Monitoring, ...

10 Zusammenfassung: Datenarchivierung – Warum?
Kontrolliertes Wachstum der Datenbank Kosteneinsparungen geringere Hardwarekosten geringere Verwaltungskosten Bessere Performance schnelleres Sichern und Wiederherstellen schnelleres Erstellen der CBO-Statistiken schnellere Release-Wechsel Kürzere Antwortzeiten für Endbenutzer bei Geschäftsvorfällen mit in DB verbleibenden Daten Langfristige Speicherung von Daten in “nutzbarer” Form … wegen neuer rechtlicher Anforderungen (GDPdU, SOA) … trotz mehr “Bewegung” in der IT-Landschaft (End-of-life System vs. Data) DB-Platz Zeit

11 Application-oriented DB Archiving
Architectural classification w.r.t. provider of archiving service DBMS-based DBMS-integrated Application with archiving functionality Application DBMS DBMS with archiving functionality DB Archive DB Archive

12 with archive integration archiving functionality („ADK“)
SAP Data Archiving For SAP, only the DBMS-based approach is an option. However, SAP applications use Basis Archiving Functionality. DBMS independence (DB2/..., Informix, MS SQL Server, Oracle, MaxDB) Availability: DBMS-integrated approach only in DB2/390 RAM Even „Standard“ SQL needs to be unified (SAP „Open SQL“) Tertiary storage integration Needs to be vendor-independent as well: dedicated archiving/CM systems connected via certified interface to store archive files Application „awareness“ of archiving DB schema hardly contains application semantics (almost no integrity constraints on DB level, business context for archive access, ...) SAP application with archive integration SAP basis archiving functionality („ADK“) DBMS DB Archive

13 Datenarchivierung aus SAP-Kundensicht
Auswertungen/Zugriff archivierte Daten lesen Datenbank Datenobjekte Archivdateien Anwendungsdaten in Archivdateien schreiben in Datenbank löschen

14 Anwendungsbezug durch Archivierungsobjekte
Struktur: Definition der logisch (aus betriebswirtschaftlicher Sicht) zusammenhängenden, gemeinsam zu archivierenden Daten – einschl. Kontext = “Schema” (allerdings nicht Archivschema) Verhalten: Programme, Anwendungsspezifische Prüfungen Technische Einstellungen für CO-Belege für FI-Belege Archivierungs- objekt Customizing Daten Programme

15 Programme eines Archivierungsobjekts
… sichtbar als Aktionen für den DA-Administrator

16 Kernfunktionen in Archivierungsprogrammen I
Schreibprogramm Berechtigungsprüfungen Einlesen des anwendungsspezifischen Customizings Lesen von der Datenbank: SELECTs aus allen beteiligten Tabellen/Views Archivierbarkeitsprüfung Erzeugen eines neuen Archivierungslaufs Aufbau komplexer (Daten-)Objekte aus gelesenen Sätzen Sichern der Datenobjekte in Archivdatei Ausgabe eines Protokolls Abschluß des Archivierungslaufs ggf. Anstoßen einer Folgeaktion (Löschen oder Ablegen)

17 Beispiel für Archivierbarkeitsprüfungen
Abhängigkeiten im Belegfluß Faktura SD_VBRK Lieferung RV_LIKP Verkaufsbeleg SD_VBAK ... Berücksichtigung der Residenzzeit Erreicht Änderungsdatum Erfassungsdatum Archivierung Zeit Verkaufsbeleg Residenzzeit Beginn 2 Beginn 1

18 Kernfunktionen in Archivierungsprogrammen II
Löschprogramm Berechtigungsprüfungen Einlesen des technischen objektspezifischen Customizings (z.B. Transaktionsgrenzen) An die aktuelle Plattform und das aktuelle Release (DB-Schema!) angepaßte Lesen der Datenobjekte aus der Archivdatei Löschen der korrespondierenden Daten(sätze) in der Datenbank (SAP Business Information Warehouse: DROP PARTITION) Ausgabe eines Protokolls Abschlußarbeiten ggf. Anstoßen einer Folgeaktion (Ablegen oder Nachlauf)

19 Archivzugriff Sichtbarkeit von archivierten Daten eigenständiges Archiv integriertes Archiv … je nach SAP-Anwendung! Bewußte Unterscheidung DB <-> Archiv oder uniformer Zugriff bei Anzeige aus Anwendungstransaktion heraus Generische Tools Archivinformationssystem Document Relationship Browser

20 Das Archive Development Kit – ADK
ADK = Generische Entwicklungs- und Laufzeitumgebung für ABAP- Archivierungsprogramme Von SAP für alle Archivierungsobjekte eingesetzt Schreiben, Löschen, Lesen, Zurückladen, Umsetzen vom Archivinformationssystem benutzt Für Kunden zur Entwicklung eigener Archivierungsobjekte freigegeben ADK-Komponenten Laufzeitsystem Kapselung von anwendungsunabhängigen Basisfunktionen, vor allem in „Richtung Archiv“ Repository Verwaltungs- und Metadaten Administration Benutzerschnittstelle für DA-Administrator

21 ADK als Laufzeitumgebung
SAP-System Archivadministration ADK- Laufzeitsystem Dateisystem Archivierungs- programme Datenobjektdienste Konvertierungen Dateiverwaltung Statusverwaltung Klassenaufrufe Statistikmodul CMS-Anbindung AS-Aufrufe Monitoring Jobsteuerung Archivdateien Archive- Link/ Content Mgmt.- Service HSM- System AS Anwendungsdaten Verwaltungs- und Metadaten Ablagesystem Hinter-grund-verarbei-tung DA- Monitor Datenfluß (beim Schreiben) Datenbank Steuerfluß

22 Temporäre Konvertierungen bei lesenden Zugriffen
Plattformanpassung Dateipfad entsprechend aktuellem Betriebssystem Zeichensatz (Codepage) und Zahlenformat für ADK-Format-Metadaten (Header, Tags) für zeichenartige und numerische Anwendungsdaten Erfordert „Bootstrapping“ beim Header-Lesen Strukturanpassung -> Umgehen mit Schemaevolution Schema in Archivdatei wird mit aktuellem Schema (aus DDIC) verglichen Bei namensgleichen Strukturen (Tabellen) Namensgleiche Felder müssen zuweisungskompatibel sein Initialwert bei noch nicht zum Archivierungszeitpunkt vorhandenen Feldern Ausblenden nicht mehr vorhandener Felder

23 Transaktionsaspekte Konsistenz Datenbank <-> Dateisystem
Physische Datei erst dann logisch sichtbar, wenn „fclose“ okay; d.h. vom Schreibprogramm übergebene Datenobjekte nicht logisch synchron archiviert „orphan files“ unkritisch Konsistenz Datenbank-Löschen <-> Archiv-Schreiben Pragmatische effektive Lösung durch 2-Phasigkeit Ausschluß von Änderungen durch Anwendung zwischen Schreib- und Löschphase Read-only-Zugriff und Archivierbarkeit über Status gewährleisten oder unkritische Änderungen zulassen oder Anwendungssperren verwalten

24 Repräsentation der Phasen in der Archivverwaltung
Write Phase Delete Phase File 1 Delete Phase File 2

25 Auswahl eines geeigneten Archivierungsobjekts

26 Auswertung von DA-Statistiken

27 Additional Session Extends to Minimum Level
Database Page Concepts Overhead Area Free Space Data Rows Additional Session Extends to Minimum Level Data Removed During Archiving Session Reserved for Update Extension

28 Reserved for Update Extension
Fragmentation Overhead Area Free Space Reserved for Update Extension

29 After Delete Processes
Index Fragmentation Index 1 After Delete Processes 2 Effect: Index range scans: More index leaf blocks read More tree levels -> Index Reorganization/Rebuild! ... ... 3 4 ... ... ... ... ... ... Table

30 Customer Example Starting point Approx. 290GB DB size and approx. 15GB DB growth per month Aim Reduction of DB growth rate to: Reduce hardware costs Maintain stable system performance Response times and system administration Faster implementation of support packages and upgrade projects Local currency conversion Archiving 19 archiving objects from FI, CO, MM, SD, and HR Result (after 15 months) > 200GB archived

31 With regular archiving
Customer Example 0.00 100.00 200.00 300.00 Mar Feb May Jun Jul Aug Sep Oct Nov Dec Jan Apr DB growth: ~15GB/month 'Without' Archiving Initial Reduction: ~60GB With regular archiving DB growth: ~7GB/month Allocated DB content 400.00 500.00 600.00 700.00 Expected size without Archiving Allocated DB size

32 ADK-Based Data Archiving
... proved to be a valued concept and implementation for ABAP-based SAP components. And will continue to do so. However, let‘s review the long-term issues ... and discuss the pros & cons of new ideas.

33 Drivers for XML-Based Archiving
Standards for long-term use of archived objects XML http(s) WebDAV Java J2EE Archive access Independent of “home“ system Potentially cross-system For Java applications as well Central archiving service Reduced redundancy in distributed scenarios Central administration

34 XML-Based Archiving – A Service Approach
XML DAS Connector mySAP X Archive Browser XML App. System Database Services XML Data Archiving Service (XML DAS) Services Database Open and inexpensive storage system Business data archive

35 XML Data Archiving Service XML DAS Adminis-tration
ADK- vs. XML Archiving Local Archive Admin. Local Archive Admin. ADK Archiving Programs XML Archiving Programs ADK XML Archive API XML DAS Connector SAP Web AS SAP Web AS HTTP XML Data Archiving Service (XML DAS) XML DAS Adminis-tration SAP J2EE Engine ArchiveLink WebDAV File system Storage system ArchiveLink File system Resource Storage system File Document File

36 WebDAV-Like Archive Hierarchy
Collection Archive path Name System ID x / b4t /b4t/ 000 Resource Archive path Name URI = Archive path + Name Client x /b4t/000/ bc_sbook_x XML archiving object y /b4t/000/bc_sbook_x/ 2003 Archive Store /b4t/000/bs_sbook_x/2003/ order_schema.xsd order_4711.xml order_4712.xml XSD XML Hierarchical WebDAV-compliant name space -> stable URIs Only restructuring of hierarchy would invalidate locally saved URIs Paths and names not case-sensitive Archive space Root collection without name: First system collection System collections do not contain resources. They are only created during Customizing Naming convention for system collections make future mergers easier Property indexes Not automatically bound to resource content Same properties for sets of resources SAP client-aware (optional) XML schema(s) in collection -> to validate all XML resources in this collection 0..1 schema if XML without name space; 0..n schemas if name spaces are used Schema should be stored in collection before XML resources are archived Optional: style sheets in collection No overwrite (no PUT/MKCOL for the same resource/collection) However, DELETE is supported Document types XML, XSD, XSL, BIN Synchronous transaction-safe requests Basis for update–LUWs in application and “early” direct writing into archive XML XML Arch. Store1 Arch. Store2 Arch. Store3 XML

37 Properties of Resources
WebDAV concept for attributing and finding resources A set of properties can be defined independently of resources. Such a property set is called property index. The property index is used to attach property values to a resource either when a resource is archived or later on. Properties are typed: VARCHAR(n) … Properties are used in value-based queries OrderNumber: 4711 XML OrderDate: 2005/01/10 ShipTo: Eppingen-Rohrbach

38 Storage System Support
Application layer XML document/stream MKCOL PUT URI service interface: WebDAV-like HTTP methods resources (XML doc n) Resources (XML doc 1) Global service layer internal storage abstraction WebDAV client File system I/O Storage system protocols WebDAV URL Phys path Storage layer Collection Directory Resource File

39 Open Archive File Format
A wrapper format ensuring… Efficiency (many data objects, byte-addressable for random access) Long-term interpretability (XML schema, XML, easy to construct) XML resource Standard Decompression Prefix Compressed Resource 1 Compressed Resource 2 Compressed Resource n ... Offset 1 Offset 2 Offset n Header

40 XML DAS Functionality Beyond WebDAV
Write once, resources cannot be modified Insertion of resources into collections can be disabled Deletions possible (occurs with logging) Identification of resources Via unique URI Long-term and stable Archive queries Hierarchical search (using the path) Value-based search (with property indexes) Future: Content search using other engines No WebDAV spec ambiguities No case sensitivity anywhere Within one collection, resources and collections cannot have same name More status codes

41 XML DAS Functionality Beyond WebDAV
XML awareness TYPE = { XML | XSD | XSL | BIN | COL | RES | ALL | ... } Check XML for well-formedness, and validity against schema Keep meta data (XSD, XSL) “close“ to business data Automatic naming Unique name for a resource within a collection Integration of different storage systems Independent of the logical XML DAS hierarchy Supports data life cycle management Archiving application integration For example, support of safe delete from DB, even in one phase Security Authorization, HTTPS, check sum Pack resources Optional Asynchronous

42 ABAP XML Archiving Object BC_SBOOK_X
Transaction SARA also used in XML-based archiving

43 JAVA Archiving: A System Deployment Scenario
SAP J2EE Engine XML Data Archiving Service 1 …n instances Archiving programs run here XML DAS Connector for JAVA Java Application

44 Java Archiving: GUI Prototype

45 Positioning ADK and XML
ADK: primarily for reducing size of database; reliable, stable, secure; for ABAP only XML: advantages when end-of-life of data longer than end-of-life of system and in multi system environments; for Java as well Striking Differences for Users Storage in the form of resources in standardized XML format instead of ADK files Archived resources can be read by your tool Easier interpretation in the long term Compression optional through pack function Application-specific searches with help of property indexes Direct archiving, no separate store phase (WebDAV or file system) Schedule as many delete jobs as reasonable

46 About SAP Data Archiving
Focus SAP R/3 Enterprise Further components SAP BW SAP CRM Detailled information about Technology and administration (ADK) Data storage and data access Implementation of archiving projects Authors Archiving experts at SAP (also mal nicht Küspert, Schaarschmidt, Zeller, Langguth ;-)

47 Copyright 2005 SAP AG. All Rights Reserved
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft®, WINDOWS®, NT®, EXCEL®, Word®, PowerPoint® and SQL Server® are registered trademarks of Microsoft Corporation. IBM®, DB2®, DB2 Universal Database, OS/2®, Parallel Sysplex®, MVS/ESA, AIX®, S/390®, AS/400®, OS/390®, OS/400®, iSeries, pSeries, xSeries, zSeries, z/OS, AFP, Intelligent Miner, WebSphere®, Netfinity®, Tivoli®, Informix and Informix® Dynamic ServerTM are trademarks of IBM Corporation in USA and/or other countries. ORACLE® is a registered trademark of ORACLE Corporation. UNIX®, X/Open®, OSF/1®, and Motif® are registered trademarks of the Open Group. Citrix®, the Citrix logo, ICA®, Program Neighborhood®, MetaFrame®, WinFrame®, VideoFrame®, MultiWin® and other Citrix product names referenced herein are trademarks of Citrix Systems, Inc. HTML, DHTML, XML, XHTML are trademarks or registered trademarks of W3C®, World Wide Web Consortium, Massachusetts Institute of Technology. JAVA® is a registered trademark of Sun Microsystems, Inc. JAVASCRIPT® is a registered trademark of Sun Microsystems, Inc., used under license for technology invented and implemented by Netscape. MarketSet and Enterprise Buyer are jointly owned trademarks of SAP AG and Commerce One. SAP, SAP Logo, R/2, R/3, mySAP, mySAP.com and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned are trademarks of their respective companies.