DATA WAREHOUSE Oracle Data Warehouse Mit Big Data neue Horizonte für das Data Warehouse ermöglichen Alfred Schlaucher, Detlef Schroeder DATA WAREHOUSE.

Präsentation zum Thema: "DATA WAREHOUSE Oracle Data Warehouse Mit Big Data neue Horizonte für das Data Warehouse ermöglichen Alfred Schlaucher, Detlef Schroeder DATA WAREHOUSE."—  Präsentation transkript:

1 DATA WAREHOUSE Oracle Data Warehouse Mit Big Data neue Horizonte für das Data Warehouse ermöglichen Alfred Schlaucher, Detlef Schroeder DATA WAREHOUSE

2 Themen Big Data Buzz Word oder eine neue Dimension und Möglichkeiten
Oracles Technologie zu Speichern von unstrukturierten und teilstrukturierten Massendaten Cloudera Framwork „Connectors“ in die neue Welt Oracle Loader for Hadoop und HDFS Big Data Appliance Mit Oracle R Enterprise neue Analyse-Horizonte entdecken Big Data Analysen mit Endeca

3 Was hat uns bisher interessiert?
Sales & Mktg Information Technology Engineering Supply Management Service Finance Sales & Operational Planning Production

4 Warum und wie Big Data jetzt?
Neue Wege der Datenerzeugung Kosten und andere Analysen Beiläufig entstehende Daten Maschinen-generiert Kommunikation Geo-Bezüge Was sind interessante Daten Wie sind sie zu speichern Welche Analysetechnik / Verfahren Welche Kosten entstehen The interest in big data has reached new highs. Everywhere you turn there is no escaping the buzz about big data. In this presentation we are going to look at some of the use cases for big data. But before we start on that, let’s take a look at why there’s all that interest in big data now. There are two trends that are helping to drive the interest in big data today. First, there is simply a lot more data being generated online today. On the one hand, there is a greater volume of human-generated data – from social media, to photographs, to , and so on. But there is also a lot more machine-generated data being generated as well - today sensors are cheap, and small enough to go anywhere. Think of smart meters, cell phones, security cameras, consumer products that phone home and so on. Interestingly, although there’s probably more human-generated data at the moment, that will certainly change some time in the next few years. The second trend that is driving interest in big data is the decreasing cost of hardware and the emergence of open source tools to store and processes all this data. One could argue that much of this “new data” has been available for many years, but it was never cost-effective to acquire and analyze all of it. Today, however, it is more economically feasible to do so. Analyzing this information can give insight into customers, target markets and so on. And it is this potential to gain new insights and in turn uncover new business cases and opportunities to improve the way you run your business, that is really how big data can pay off. NEW WAYS TO GENERATE DATA - We are finding new ways to monitor activities and processes, building new data streams… HIGH VELOCITY DATA FLOWS - These new data streams are increasingly being generated in real-time and they need to be analyzed in real-time to gain maximum benefit… VAST DATA POOLS - new sources are introducing a wide variety of schema-less data streams that need to be mined and analyzed to gain greater insight… ECONOMICS OF ANALYTICS – The total cost of acquiring, organizing and analyzing massive, varied and complex data sets is declining Sensors are cheap and small enough to go anywhere Growing digital ecosystem New Business Opportunities

5 Use Case gibt es viele Use Cases Financial Service Freizeit
Automaten / Logistik Automotive Retail

6 Ein potentieller Fall Offen aber bleiben Frage wie:
Ein Börsen-Unternehmen misst permanent alle relevanten Aktienkurse über einen längeren Zeitraum. In dem Data Warehouse sind alle Entwicklungen, alle Ups und Downs der letzten 10 Jahre genau dokumentiert. Offen aber bleiben Frage wie: Warum sind diese Ups und Downs zu bestimmten Zeiten entstanden? Beeinflussen öffentliche Nachrichten den Aktienhandel? Parallel zu dem Data Warehouse sammelt das Unternehmen alle öffentlich zugänglichen Nachrichten. Gesucht werden bewertende Aussagen zu Zeitpunkten der Aktienbewegungen.

7 In vielen Lebenssituationen erzeugen wir beiläufig und permanent Daten (z. T. ohne es zu wissen oder zu bemerken) Potential für neue Analysen und Geschäftsmodelle

8 Potential für neue Analysen und Geschäftsideen
Maschinendaten Vergleichsdaten / DWH 50 Kontaktpunkte / Skifahrer / Tag 10 KB pro Kontaktpunkt -> 500KB pro Skifahrer / Tag Bei 20 Millionen Skifahrer in den Alplen/Jahr und durchschnittlich 10 Tagen Aufenthalt sind das -> 10 TB / Tag -> 100 TB insgesamt Personendaten Herkunft (Wohnort) Kartenkaufort Alter (Geb. Datum) Nutzungszeitraum Monat / Woche / Tag Tageszeit Nutzungshäufigkeit Wetterdaten Pistendaten Schneeverhältnisse Schwierigkeitsgrade Höhenmeter Liftdaten Auslasung

9 Technologisch erweiterte Möglichkeiten
1980 2010 Automaten / Deutschland Alle Automaten über Sensorik erfasst und zentral abrufbar Füllstände, Sensoren in der Mechanik der Geräten Wartungszyklen, Routenplanung für Service-Techniker Bitte nicht als Zigarettenwerbung verstehen. Wir finden Rauchen nicht gut!

10 Auto: Der fahrende Computer > 200 Sensoren / Auto
10 KB / Km (?????) 10 MB / Tankfüllung 10 TB / Tankfüllung / 1 Mill Autos 500 TB / km / 1 Mill Autos 1 MB / Km (?????) 1 GB / Tankfüllung 100 TB / Tankfüllung / 1 Mill Autos 5 PB / km / 1 Mill Autos Fahrpedalgeber Kühlwasser Bremspedalgeber Regenfühler Beschleunigung Leerlaufverhalten Drehzahl Kraftstoffverbrauch Bremskraft Reifendruck Stromverbrauch Öldruck Temperatur innen / aussen Getriebeeinstellung Aktivierte Stromverbraucher Motortemperatur

11 Big Data in der Auto-Industrie
Vielfältige Einsatzgebiete Welche Daten werden gesammelt Verwendung “Use Case” Komponenten-Sensoren (z. B. Öldrück, Temperatur etc.) Vorhersage von Pannen Individuellle Service Pläne Welche Stromverbraucher werden wie oft genutzt Welche Extras werden genutzt und sind wirklich wichtig Effektiveres Marketing Eingang in F&E Brems- / Beschleunigungsdaten, Fahrleistung, Schaltverhalten Messen des individuellen Fahrverhaltens Rekonstruktion von Unfällen Individuelle Versicherungsangebote GPS-Geo-Positionen Wo befindet sich das Fahrzeug Welche Strassen werden genutzt Proactive engagement Bessere Grundlagen für künftige Anforderungen (Gelände etc.) Verbrauchsdaten Wer braucht wann und wo und wieviel Treibstoff . Messen des Tankverhaltens. Genauere Grundlage für Tankstellenplanung If the last slide showed breadth, this slide shows some of the depth. Of all the different use cases in automotive, we’ve chosen to focus here just on the data that a manufacturer could acquire by instrumenting cars that customers purchase. You can see the kind of things that they can look at: what’s happening with components, are they in spec or moving out of spec; what features in the new, complex control systems are drivers using or struggling with; how do they drive; and so on. There are lots of different ways to use this new data. If you combine information about all the vehicle sensors with driving techniques and fuel consumption you might be able to generate tailored service plans. The people who drive their car harder and faster might come in for a service earlier than usual, for example.

12 Genauere Kundenanalysen
Mehr als nur klassische Sortimentsanalysen Massenmarkt-Retailer Business Problem Neue Art der Analyse Nutzen Zielgenauere Werbebotschaften Minimieren von Werbeaufwand Bessere Zielgruppen-orientierung Kaufgewohnheiten Kundenbindung Detailliertere Kundenprofile Hinzuziehen von zusätzlichen Informationsquellen Co-Varianz-Analysen s Web logs Effizienteres Marketing Umsatzsteigerung Schärfung des Sortiments Target The first one here comes from an excellent story in the New York Times in February of 2012. Improve timing and relevance of offers The best time to reach a customer with an offer is when they are ready to buy. This is what drives things like “back to school” sales for books and school supplies. The problem is it’s fairly easy to tell who has school age children and when school is about to start, so pretty much any retailer can do that. No differentiation. There are other occasions when people are ready to buy. Childbirth in particular is highly regarded in this respect, because when a family welcomes a new child, not only are they ready to buy, but their whole pattern of purchasing is subject to change. So capturing a new customer then is potentially quite valuable. Birth records are public knowledge so everybody has them. That’s why new parents are deluged by offers for everything as soon as they get home. But what if you could detect this earlier? New Data Analysis The closer somebody gets to giving birth the easier it is for a retailer to figure this out. Registering for baby stuff and buying large quantities of diapers are giveaways but that’s also too late. Starting some years ago, Target started looking at the purchasing history of customers that they knew were pregnant. After a lot of work, they found that certain combinations of around 25 products could be used to predict pregnancy with reasonable accuracy as early as the second trimester. Things like the combination of vitamin supplements followed by lotions, cotton swabs in bulk and larger bags could be used to give a “pregnancy score” and even to predict the month of birth quite well. And the accuracy of this model improved over time with more data like s, website behavior and, of course, more analysis. Payoff Armed with this knowledge, Target was in a position to send the expectant mother offers for things that she would need. They also realized that they might need to do this carefully, so pregnancy-related offers with included with a mix of other items. But this enabled them to get some mindshare with customers getting ready to buy a lot of new things and change their purchasing habits. It has been a very powerful revenue generation tool. Their model was go effective that on at least one occasion, Target identified a pregnant high school girl and targeted her with pregnancy-related programs, before her family even knew she was pregnant. Business Benefit (TBC) One indication of the importance of this approach is that while the story was under development, Target realized what was happening and stopped talking to the NY Times. Perhaps they didn’t want others to realize how effectively they were using big data. But they have boasted publicly of their focus on the mother and baby opportunity as a major part of their strategy. And if you look back 10 years since they started this program and compare their stock price with arguably their biggest competitor, they are not doing too badly.

13 Soziale Stadtpläne Wo in bewegen sich zu welchen Zeiten die meisten Menschen

14 Weitere Big Data Use Cases
EA Big Data Roundtable 3/28/ :59 AM Weitere Big Data Use Cases In allen Branchen AUTOMOTIVE Auto sensors reporting location, problems COMMUNICATIONS Location-based advertising CONSUMER PACKAGED GOODS Sentiment analysis of what’s hot, problems FINANCIAL SERVICES Risk & portfolio analysis New products EDUCATION & RESEARCH Experiment sensor analysis HIGH TECHNOLOGY / INDUSTRIAL MFG. Mfg quality Warranty analysis LIFE SCIENCES Clinical trials Genomics MEDIA/ ENTERTAINMENT Viewers / advertising effectiveness ON-LINE SERVICES / SOCIAL MEDIA People & career matching Web-site optimization HEALTH CARE Patient sensors, monitoring, EHRs Quality of care OIL & GAS Drilling exploration sensor analysis RETAIL Consumer sentiment Optimized marketing TRAVEL & TRANSPORTATION Sensor analysis for optimal traffic flows Customer sentiment LAW ENFORCEMENT & DEFENSE Threat analysis - social media monitoring, photo analysis UTILITIES Smart Meter analysis for network capacity, And this sentiment can be felt across a variety of industries. As you can see from this slide, there are many different use cases for leveraging big data. In each of these cases, new types of data – can be machine generated or human generated - or a greater volume of data is being analyzed to solve traditional business problems. We’ll cover many of these later on, so I’m not going to go into great detail, but I will make one point. This slide illustrates the breadth of opportunity with big data. There are many other industries that we could have included. There’s an opportunity for big data just about everywhere. Challenged by: Data Volume, Velocity, Variety Copyright 2012 Oracle Corporation. All rights reserved.

15 Big Data Opportunities?
Drei Kategorien Senken von Kosten Steigern von Umsatz Neue Geschäftsideen €€€ “In a big data world, a competitor that fails to sufficiently develop its capabilities will be left behind.” Now we’re going to look at some use cases in more detail. Because of the variety of opportunities, we can’t show all the possible use cases. Instead, we’re going to show some representative examples. Even if your industry or company is not shown directly, we hope that you will see something relevant that’s applicable to you, or something that will give you ideas that you can use. The single biggest success factor for big data projects is having a good business case. So what we’re going to do here is offer three different sets of examples oriented around different types of business case. Some projects set out to reduce costs, others to increase revenue and others to provide some kind of new innovation with new products or programs not currently available. All these approaches have the potential to bring long term value to any organization and keep you ahead of your competition. McKinsey Global Institute

16 „New Data“ Paradigma Low value density data processing HDFS
Batch bulk load HDFS Reduced data set Analyse Tools Semi strukturierte Daten Kundenprofile Externe Daten Social Media Blogs, Feeds, Forum Social Media, Mails, Briefe, Verträge Texte, Dokumente Smart Visualisierung Leicht durchführbare Abfragen User Defined Algorithms Mobile Complex event processing Kundenvorlieben (Sentiments Analyse) Massendaten- Analyse Predictive Analysen Kundenprofile Angereicherte Wissensablage Filter Classify corrolate Unstrukturierte Daten Transaction Event Correlation Text Clickstream Web content Logs Tablet NOSQL High volumn, low Latency-Daten-Streams Marktdaten, News Realtime und Selbstlernend Statistical Analysis Transaktionen Aggregate Kennzahlen Web Strukturierte Daten Relational High value data processing Profile Transaktionen Vertriebswege Absatzdaten OLTP: Stateless Delivery Unterschiedliche Kanäle Bank, PoS, Credit Card Office Acquire Organise Analyse, Decide &Deliver

17 „New Data“ Paradigma Low value density data processing HDFS
Batch bulk load HDFS Reduced data set Analyse Tools Semi strukturierte Daten Kundenprofile Externe Daten Social Media Blogs, Feeds, Forum Social Media, Mails, Briefe, Verträge Texte, Dokumente Smart Visualisierung Leicht durchführbare Abfragen User Defined Algorithms Mobile Complex event processing Kundenvorlieben (Sentiments Analyse) Massendaten- Analyse Predictive Analysen Kundenprofile Angereicherte Wissensablage Filter Classify corrolate Unstrukturierte Daten Transaction Event Correlation Text Clickstream Web content Logs Tablet NOSQL High volumn, low Latency-Daten-Streams Marktdaten, News Realtime und Selbstlernend Statistical Analysis Transaktionen Aggregate Kennzahlen Web Strukturierte Daten Realtional High value data processing Profile Transaktionen Vertriebswege Absatzdaten OLTP: Stateless Delivery Unterschiedliche Kanäle Bank, PoS, Credit Card Office Acquire Organise Analyse, Decide &Deliver

18 + „New Data“ Paradigma Zählen von Strings und Mustern Massen Daten
Low value density data processing HDFS Batch bulk load HDFS Reduced data set Analyse Tools Semi strukturierte Daten Zählen von Strings und Mustern Massen Daten Kundenprofile Externe Daten Social Media Blogs, Feeds, Forum Social Media, Mails, Briefe, Verträge Texte, Dokumente Smart Visualisierung Leicht durchführbare Abfragen Unstrukturiert User Defined Algorithms Mobile Complex event processing Kundenvorlieben (Sentiments Analyse) Massendaten- Analyse Predictive Analysen + Kundenprofile Angereicherte Wissensablage Filter Classify corrolate Unstrukturierte Daten Transaction Event Correlation Text Clickstream Web content Logs Tablet NOSQL Klassisches DWH High volumn, low Latency-Daten-Streams Marktdaten, News Realtime und Selbstlernend Statistical Analysis Transaktionen Aggregate Kennzahlen Web Strukturierte Daten Realtional Strukturiert High value data processing Profile Transaktionen Vertriebswege Absatzdaten OLTP: Stateless Delivery Unterschiedliche Kanäle Bank, PoS, Credit Card Klassische Auswertung Office Acquire Organise Analyse, Decide &Deliver

19 + „New Data“ Paradigma Zählen von Strings und Mustern Massen Daten
Low value density data processing HDFS Batch bulk load HDFS Reduced data set Analyse Tools Semi strukturierte Daten Zählen von Strings und Mustern Massen Daten Kundenprofile Externe Daten Social Media Blogs, Feeds, Forum Social Media, Mails, Briefe, Verträge Texte, Dokumente Smart Visualisierung Leicht durchführbare Abfragen Unstrukturiert User Defined Algorithms Mobile Complex event processing Kundenvorlieben (Sentiments Analyse) Massendaten- Analyse Predictive Analysen + Kundenprofile Angereicherte Wissensablage Filter Classify corrolate Unstrukturierte Daten Individualisiertere Sichten Transaction Event Correlation Text Clickstream Web content Logs Tablet NOSQL Klassisches DWH High volumn, low Latency-Daten-Streams Marktdaten, News Realtime und Selbstlernend Statistical Analysis Transaktionen Aggregate Kennzahlen Web Strukturierte Daten Realtional Strukturiert High value data processing Profile Transaktionen Vertriebswege Absatzdaten OLTP: Stateless Delivery Unterschiedliche Kanäle Bank, PoS, Credit Card Klassische Auswertung Office Acquire Organise Analyse, Decide &Deliver

20 + „New Data“ Paradigma Zählen von Strings und Mustern Massen Daten
Low value density data processing HDFS Batch bulk load HDFS Reduced data set Analyse Tools Semi strukturierte Daten Zählen von Strings und Mustern Massen Daten Kundenprofile Externe Daten Social Media Blogs, Feeds, Forum Social Media, Mails, Briefe, Verträge Texte, Dokumente Smart Visualisierung Leicht durchführbare Abfragen Unstrukturiert User Defined Algorithms Direkte Einflussnahme auf Prozesse Mobile Complex event processing Kundenvorlieben (Sentiments Analyse) Massendaten- Analyse Predictive Analysen + Kundenprofile Angereicherte Wissensablage Filter Classify corrolate Unstrukturierte Daten Individualisiertere Sichten Transaction Event Correlation Text Clickstream Web content Logs Direkte Kundenansprache Tablet NOSQL Klassisches DWH High volumn, low Latency-Daten-Streams Marktdaten, News Realtime und Selbstlernend Statistical Analysis Transaktionen Aggregate Kennzahlen Konkrete Aktionen Web Strukturierte Daten Realtional Strukturiert High value data processing Profile Transaktionen Vertriebswege Absatzdaten OLTP: Stateless Delivery Unterschiedliche Kanäle Bank, PoS, Credit Card Klassische Auswertung Office Acquire Organise Analyse, Decide &Deliver

21 Big Data: Infrastruktur Anforderungen
Acquire Organize Analyze Unvorhersehbares Auftreten Hohe Datenmengen Flexible Daten-Strukturen Arbeiten mit vielen Servereinheiten Abfragen mit extrem hohen Daten-Durchsatz Bearbeitung am Speicherplatz Hohe Parallelisierung Explorative Analyse Komplexe statistische Analysen Agile Berichtsentwicklung Massive Skalierung Real Time Ergebnisse

22

23 Oracle’s integrierte Software Lösung
Data Variety Acquire Analyze Organize Unstructured Cloudera Hadoop HDFS Oracle NoSQL DB Oracle Analytics Mining R Spatial Graph OBI EE Oracle MapReduce Oracle (DW) Schema-less Oracle Hadoop Loader Oracle (OLTP) Schema Information Density

24 Oracle Engineered Systems
Data Variety Acquire Analyze Organize Unstructured Big Data Appliance Exalytics Schema-less Schema Exadata Database Machine Information Density

25 Oracle Data Warehouse Architektur für unternehmensweites Datenmanagement
Data Integration Real Time & Batch Any Source BI Server Reporting & Publishing Ad-hoc Analysis Office Integration Mobile Scorecards Interactive Dashboards Oracle Database Management System Information Layer Architecture Concept Data Integration Layer Enterprise Information Layer User View Layer InDatabase ROLAP InDatabase MOLAP Controlling Financial Marketing Sales HR BI Apps Data Management Concept Reference Data Models InDatabase R InDatabase Data Mining Operational Data Layer Dynamic Data Marts Data Quality Rules Checks&Monitoring DWH Logistic Utilities Business Catalogue Technical Auditing Metadata Utilities Lifecycle Management Concept DWH System Monitoring Utilities DWH Security Utilities DWH Backup / Recovery Concept Concept Framework Big Data Solution To explain it a little bit more Lets have look to this map What you see here is a collection of all components you can use in a Data Warehouse and Business Intelligence environement. Oracle Database Management System Exadata noSQL Hadoop Server Cluster Server Cluster Operating System Storage Hierarchy Optimized Network Optimiertes Netzwerk Big Data Appliance Exalytics Exadata / Database Machine / Exalytics

26 Oracle Produkt-Komponenten Data Warehouse / BigData
Oracle EE (37) OLAP (18) InMemory DB Cache(18) AD Comp (9) Partitioning (18) Advanced Analytics (18) Label Sec (9) RAC (8) Diagnostic+Tuning (8) Spatial (13) Data Integration NoSQL EE (8) BigData Connectors (1,5) BigData Appliance (400K) Exadata /DBM (ab 350K) Business Intelligence

27 Oracle Produkt-Komponenten Data Warehouse / BigData
Oracle EE OLAP InMemory DB Cache AD Comp Partitioning Advanced Analytics Label Sec RAC (8) Diagnostic+Tuning Spatial Data Integration NoSQL EE BigData Connectors BigData Appliance Exadata /DBM Business Intelligence

28 Oracle Produkt-Komponenten Data Warehouse / BigData
Oracle R Enterprise Advanced Analytics Oracle Data Mining NoSQL EE Oracle Loader for Hadoop Oracle Direct Connector for HDFS Oracle R for Hadoop Connector BigData Connectors BigData Appliance

29