Institute for Software Science – University of ViennaP.Brezany Web Services und Grid Services im Grid Computing Peter Brezany Institut für Softwarewissenschaften.

Präsentation zum Thema: "Institute for Software Science – University of ViennaP.Brezany Web Services und Grid Services im Grid Computing Peter Brezany Institut für Softwarewissenschaften."—  Präsentation transkript:

1 Institute for Software Science – University of ViennaP.Brezany Web Services und Grid Services im Grid Computing Peter Brezany Institut für Softwarewissenschaften Universität Wien

2 Institute for Software Science – University of ViennaP.Brezany 2 Medien, die radikal die Gesellschaft beeinflußten Web 1500s Druckpresse 1840s Penny Post 1850s Telegraph 1920s Telefone 1930s Radio 1990s 1950s TV 20xx Grid

3 Institute for Software Science – University of ViennaP.Brezany 3 Grid Computing Vision "The Internet is about getting computers to talk together; Grid computing is about getting computers to work together." Tom Hawk, IBM's general manager of Grid computing

4 Institute for Software Science – University of ViennaP.Brezany 4 Grid Computing Vision (2) Tim Berners-Lee replies to the question What did you have in mind when you first developed the Web? by saying "The dream behind the Web is of a common information space in which we communicate by sharing information. If applied to the Grid computing this sentence can be rephrased to The dream behind the Grid computing is a common resource space in which we can work together using shared recources.

5 Institute for Software Science – University of ViennaP.Brezany 5 Web im Vergleich zum Grid Classical Web Classical Grid More computation

6 Institute for Software Science – University of ViennaP.Brezany 6 Web im Vergleich zum Grid (2) Classical Web Semantic Web Richer semantics

7 Institute for Software Science – University of ViennaP.Brezany 7 Web im Vergleich zum Grid (3) Classical Web Classical Grid Semantic Web Richer semantics More computation Semantic Grid Source: Norman Paton

8 Institute for Software Science – University of ViennaP.Brezany 8 Lernziele Motivation für Grids Grundbegriffe Bestehende Architekturen Neue Entwicklungen –Von Web Services zu Grid Services –Weiterentwickung und Integration von Web Services und Grid Services Grid Lösungen

9 Institute for Software Science – University of ViennaP.Brezany 9 Beispiele und logische Konsequenzen Beispiel Wasserversorgung – Früher: Hausquelle / Brunnen – Heute: Wassersammelstelle Leitungen Wasserhahn Beispiel Energieversorgung – Früher: Generator – Heute: Großer Generator Stromleitungen Steckdose –Power Grid Computational Grid / Grid Computing (z.B.: NASA: Information Power Grid (www.ipg.nasa.gov))www.ipg.nasa.gov Logische Konsequenz: Grid Computing Rechenleistung (und vieles mehr) aus der Steckdose Viele Rechner zu einem Großen Netz verbunden; Vorteile: – Komplett neue Möglichkeiten der Zusammenarbeit für Unternehmen – Hardwareersparnis (mieten) (vgl. Generator / Quelle) – Teuere Software mieten statt kaufen – Selbst z.B. Rechenleistung anbieten

10 Institute for Software Science – University of ViennaP.Brezany 10 Grid Computing - Definition Definition nach www.globus.org 1 : The Grid ist eine Infrastruktur, die eine integrierte, gemeinschaftliche Verwendung von Ressourcen erlaubt. Als Ressourcen kommen nicht nur Re- chenleistung und Speicherplatz in Frage, sondern auch ganze (beliebige) Geräte können im Grid gemeinschaftlich verwendet werden, also zum Beispiel Hochleistungscomputer, Netzwerke, Datenbanken, Teleskope, Mikroskope bis zu Elektronenbeschleunigern. Ziel des Grid ist es, dass man auf Geräte zugreifen kann, als ob man sie besitzen würde, ohne sie kaufen zu müssen. Charakteristika von Grid-Anwendungen: - Große Datenmengen - Großer Rechenaufwand –Sicheres Resourcen-Sharing zwischen unabhängigen Organisationen –Aufbau von Virtuellen Organisationen (VO) ----------------------------------------------------------- 1 Praktisch alle wichtigsten Grid Projekte bauen auf middleware Globus (1998 - Globus 1, 2001 - Globus 2, 2003 - Globus 3)

11 Institute for Software Science – University of ViennaP.Brezany 11 VO Beispiel Autohersteller beauftragt: – Application service provider (ASP) Finanzielle Vorhersage – Storage service provider (SSP) (Historische) Daten – Cycle providers Rechenleistung für die Analyse Szenarienanalysen für neue Fabrik (bzw. Standort) durchzuführen.

12 Institute for Software Science – University of ViennaP.Brezany 12 VO Beispiel (2) Figure: An actual organization can participate in one or more VOs by sharing some or all of its resources. We show three actual organizations (the ovals), and two VOs: P, which links participants in an aerospace design consortium, and Q, which links colleagues who have agreed to share spare computing cycles, for example to run ray tracing computations. The organization on the left participates in P, the one to the right participates in Q, and the third is a member of both P and Q. The policies governing access to resources (summarized in quotes) vary according to the actual organizations, resources, and VOs involved.

13 Institute for Software Science – University of ViennaP.Brezany 13 Definitionen: Protokoll, Dienst, API, SDK Protokoll: – Menge von Regeln für Endpunkte von Telekommunikationssystemen zum Informationsaustausch – Standardprotokoll gewährleistet Interoperabilität Dienst (Service): – Netzwerkfähige Instanz mit einer bestimmten Fähigkeit Definiert durch Protokoll und Reaktion auf eine Protokoll-Nachricht (service = protocol + behavior) Application Program Interface (API): – Standardinterface für Zugriff auf Funktionalität (ein Protokoll kann mehrere APIs haben) – Ermöglicht Portabilität Software Develpment Kit (SDK): – Implementiert ein API

14 Institute for Software Science – University of ViennaP.Brezany 14 Grid Protokoll Architektur vs. IP Architektur Application Fabric Controlling things locally: Access to, & control of, resources Connectivity Talking to things: communication (Internet protocols) & security Resource Sharing single resources: negotiating access, controlling use Collective Coordinating multiple resources: ubiquitous infrastructure services, app- specific distributed services Internet Transport Application Link Internet Protocol Architecture

15 Institute for Software Science – University of ViennaP.Brezany 15 Grid Architektur (2) Fabric: – (Computer / Dateisysteme / Archive / Netzwerke / Sensoren /...) (open, read, write, close,...) – Kaum Beschränkungen am low-level solang Schnittstellen erfüllt Connectivity: – Kommunikation (IP, DNS, Routing,...) – Sicherheit (Grid Security Infrastructure, GSI) - Einheitliche Authentifikation - Single sign-on - Delegation - Public Key Technologie

16 Institute for Software Science – University of ViennaP.Brezany 16 Grid Architektur (3) Resource Layer: – Grid Resource Allocation Management (GRAM) Zuweisung, Reservierung, Monitoring, Steuerung von Rechenresourcen – GridFTP Protokoll (FTP Erweiterungen) Hochgeschwindigkeitsdatenzugriff und –Transport – Grid Resource Information Service (GRIS) Zugang zu Struktur- und Statusinformationen – Netzwerkreservierung, Beobachtung und Steuerung – Baut auf Connectivity Layer (GSI & IP) auf.

17 Institute for Software Science – University of ViennaP.Brezany 17 Grid Architektur (4) Collective Layer: – Globale Protokolle und Dienste – Baut auf dem neck auf – ist komplett unabhängig von den Resourcen – Verzeichnisdienste – Monitoring- und Diagnosedienste – Datenreplikationsdienste – etc. Applications: – Verwenden Dienste beliebiger Layer

18 Institute for Software Science – University of ViennaP.Brezany 18 Data Grid Ursprüngliche Motivation: Wissenschaftliche Anwendungen sind sehr daten intensiv und enorm große Menge von Forschern aus der ganzen Welt will einen schnellen Zugriff auf diese Daten haben.

19 Institute for Software Science – University of ViennaP.Brezany 19 State of the Art in 2002 Die bisher diskutierten Konzepte implementiert von mehreren SDK, z.B. Globus (U.S.), Unicorn (EU Projekt), European Data Grid (EU Projekt), usw. Nur in wissenschaftlichen Kreisen gut bekannt und Fokus auf big-science Anwendungen. Fast keine Anbindung von Datenbanktechnologien, Anwendung von flat files. Notwendigkeit näher zum every-day life (e- Business, medicine, usw.) zu sein. Ignorierung von Web Entwicklung – Web Service Technologien Große Firmen (IBM, Sun, Microsoft, usw.) beginen jetzt auch mitzumachen.

20 Institute for Software Science – University of ViennaP.Brezany 20 Grid and Web Services: Convergence? Grid Web GT – Globus Toolkit, OGSI – Open Grid Service Infrastructure However, despite enthusiasm for OGSI, adoption within Web community turned out to be problematic Started far apart in apps & tech OGSI GT2 GT1 HTTP WSDL, WS-* WSDL 2, WSDM Have been converging ? 2004 1991

21 Institute for Software Science – University of ViennaP.Brezany 21 Grid Service – OGSA – OGSI – GT3 OGSA – Open Grid Service Architecture

22 Institute for Software Science – University of ViennaP.Brezany 22 Grid Service – OGSA – OGSI – GT3 (2) Grid Services are defined by OGSA. The Open Grid Services Architecture (OGSA) aims to define a new common and standard architecture for grid-based applications. RIght at the center of this new architecture is the concept of a Grid Service. OGSA defines what Grid Services are, what they should be capable of, what types of technologies they should be based on, but doesn't give a technical and detailed specification (which would be needed to implement a Grid Service). Grid Services are specified by OGSI. The Open Grid Services Infrastructure is a formal and technical specification of the concepts described in OGSA, including Grid Services. The Globus Toolkit 3 is an implementation of OGSI. GT3 is a usable implementation of everything that is specified in OGSI (and, therefore, of everything that is defined in OGSA). Grid Services are based on Web Services. Grid Services are an extension of Web Services. We'll see what Web Services are in the next page, and what Grid Services are in the page after that. I still don't get it: What is the difference between OGSA, OGSI, and GT3? Consider the following simple example. Suppose you want to build a new house. The first thing you need to do is to hire an architect to draw up all the plans, so you can get an idea of what your house will look like. Once you're happy with the architect's job, it's time to hire an engineer who will make detailed blueprints that specify construction details (like where to put the master beams, the power cables, the plumbing, etc.). The engineer then passes all those blueprints to qualified professional workers (construction workers, electricians, plumbers, etc) who will actually build the house. We could say that OGSA (the definition) is the architect, OGSI (the specification) is the engineer, and GT3 (the implementation) is the workers.

23 Institute for Software Science – University of ViennaP.Brezany 23 OGSA - GridService

24 Institute for Software Science – University of ViennaP.Brezany 24 GT 3 Architecture I Grid Services, which we have already seen, are the 'GT3 Core' layer. Let's take a look at the rest of the layers from the bottom up: GT3 Security Services: Security is an important factor in grid-based applications. GT3 Security Services can help us restrict access to our Grid Services, so only authorized clients can use them. For example, we said that only our New York, Los Angeles, and Seattle offices could access MathService. We want to make sure only those offices have access to MathService and, of course, we want all the data exchanged between MathService and clients to be encrypted so we can keep malicious users from intercepting our data. Besides the usual security measures (putting the web server behind a firewall, etc.) GT3 gives us one more layer of security with technologies such as SSL and X.509 digital certificates. GT3 Base Services: This layer actually includes a whole lot of interesting services: Managed Job Service: Suppose some particular operation in MathService might take hours or even days to be done. Of course, we don't want to simply stand in front of a computer waiting for the result to arrive (specially if, after 8 hours of waiting, all we get might simply be an error message!) We need to be able to check on the progress of the operation periodically, and have some control over it (pause it, stop it, etc.) This is usually called job management (in this case, the term 'job' is used instead of 'operation'), The Managed Job Service allows us to treat our invocations like jobs, and manage them accordingly.

25 Institute for Software Science – University of ViennaP.Brezany 25 GT 3 Architecture II Index Service: Remember from A short introduction to Web Services that we usually know what type of Web Service we need, but we have no idea of where they are. This also happens with Grid Services: we might know we need a Grid Service which meets certain requirements, but we have no idea of what its location is. While this was solved in Web Services with UDDI, GT3 has its own Index Service. For example, we could have several dozen MathServices all around the country, each with different characteristics (some might be better suited for statistical analysis, while others might me better for performing simulations). Index Service will allow is to query what MathService meets our particular requirements. Reliable File Transfer (RFT) Service: This service allows us to perform large file transfers between the client and the Grid Service. For example, suppose we have an operation in MathService which has to crunch several gigabytes of raw data (for a statistical analysis, for example). Of course, we're not going to send all that information as parameters. We'll be able to send it as a file. Furthermore, RFT guarantees the transfer will be reliable (hence its name). For example, if a file transfer is interrupted (due to a netwok failure, for example), RFT allows us to restart the file transfer from the moment it broke down, instead of starting all over again. GT3 Data Services: This layer includes Replica Management, which is very useful in applications that have to deal with very big sets of data. When working with large amount of data, we're usually not interested in downloading the whole thing, we just want to work with a small part of all that data. Replica Management keeps track of those subsets of data we will be working with. Other Grid Services: Other non-GT3 services can run on top of the GT3 Architecture.

26 Institute for Software Science – University of ViennaP.Brezany 26 Service Data

27 Institute for Software Science – University of ViennaP.Brezany 27 Service Data

28 Institute for Software Science – University of ViennaP.Brezany 28 Service Data

29 Institute for Software Science – University of ViennaP.Brezany 29 Notification Interfaces

30 Institute for Software Science – University of ViennaP.Brezany 30 Pull-Notifications

31 Institute for Software Science – University of ViennaP.Brezany 31 Push-Notifications

32 Institute for Software Science – University of ViennaP.Brezany 32 Notifications in GT3

33 Institute for Software Science – University of ViennaP.Brezany 33 Challenge: Advanced Grid Applications Example: Knowledge Discovery in Grid Databases

34 Institute for Software Science – University of ViennaP.Brezany 34 Motivation Business Medicine Scientific experiments Simulations Earth observations Data and data exploration cloud Data and data exploration cloud

35 Institute for Software Science – University of ViennaP.Brezany 35 Data Warehouse Knowledge Cleaning and Integration Selection and Transformation Data Mining Evaluation and Presentation The Knowledge Discovery Process OLAP Online Analytical Mining OLAP Queries

36 Institute for Software Science – University of ViennaP.Brezany 36 The GridMiner Project in Vienna GridMiner : A knowledge discovery Grid infrastructure (http://www.gridminer.org/) OGSA-based architecture Workflow management Grid-aware data preprocessing and data mining services Data mediation service OLAP service GUI Implementation on top of Globus Toolkit 3.0 Application : Management of patients with traumatic brain injuries

37 Institute for Software Science – University of ViennaP.Brezany 37 GridMiner Architecture GMMS Mediation GMPPS Preprocessing GMDMS Data Mining GMPRS Presentation GM DSCE Dynamic Service Control GMDIS Integration GMOMS OLAM GMIS Information GMRB Resource Broker GridMiner Core GMCMS OLAP / Cubes GridMiner Base GridMiner Workflow Grid Core Services Security File and Database Access Service Replica Management Grid Core Grid ResourcesData Source Fabric

38 Institute for Software Science – University of ViennaP.Brezany 38 Collaboration of GM-Services Example 3:

39 Institute for Software Science – University of ViennaP.Brezany 39 The Control Layer Control Layer –Provision of the whole knowledge discovery process to a client Knowledge discovery process in GridMiner –services to execute not known –order of service execution –sequential and concurrent execution Approaches investigated: –Data Mining Query Language –Standard Workflow Orchestration Approach (BPEL4WS, WSFL, GSFL, …) –Our approach: Dynamic Service Control

40 Institute for Software Science – University of ViennaP.Brezany 40 The Control Layer Standard Service Orchestration Approach (BPEL4WS)

41 Institute for Software Science – University of ViennaP.Brezany 41 Workflow Models Composition by Service PublisherComposition by Service Consumer

42 Institute for Software Science – University of ViennaP.Brezany 42 The Control Layer - Approaches: Dynamic Service Control Dynamic Service Control Language (DSCL) –based on XML –easy to use –supports OGSA Grid Services –specially design to support knowledge discovery processes Dynamic Service Control Engine (DSCE) –processes workflow according to DSCL DSCE Service A Service C Service D Client OGSA Grid Services Notification sink DSCL subscribequery results notify (re)connect Start, stop, resume… Service B

43 Institute for Software Science – University of ViennaP.Brezany 43 Dynamic Service Control Language (DSCL) Features –Control flow »concurrent execution of activities »sequential execution of activities –Activities »creation of new Grid Service Instances »invoking operations on Grid Service Instances »querying information of Grid Service Instances »destroying of Grid Service Instances

44 Institute for Software Science – University of ViennaP.Brezany 44 DSCL - Structure variables composition dscl qreate Service invoke query SDE qreate Service invoke query SDE qreate Service invoke

45 Institute for Software Science – University of ViennaP.Brezany 45 Initializing by simple type value Initializing by arrays <ns1:value xsi:type=xsd:int xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:xsd=http://www.w3.org/2001/XMLSchema xmlns:ns1=http://ogsa.globus.org>4711 DSCL - Variables <ns1:value xsi:type=soapenc:Array soapenc:arrayType=xsd:int[2] xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance xmlns:soapenc=http://schemas.xmlsoap.org/soap/encoding/ xmlns:xsd=http://www.w3.org/2001/XMLSchema xmlns:ns1=http://ogsa.globus.org> 23 -112

46 Institute for Software Science – University of ViennaP.Brezany 46 Initializing by a complex type value Austria 1090 Vienna Liechtensteinstr. 18 DSCL - Variables <xsd:schema targetNamespace="http://www.gridminer.org/test/" xmlns:tns="http://www.gridminer.org/test/"......

47 Institute for Software Science – University of ViennaP.Brezany 47 DSCL Control Flow composition dscl sequence parallel invoke activityID=act2.1 … invoke activityID=act2.2 … createService activityID=act1 … sequence variables act1 act2.1 act2.2 …

48 Institute for Software Science – University of ViennaP.Brezany 48 Grid and Web Services: Convergence: Yes! Grid Web The definition of WSRF means that Grid and Web communities can move forward on a common base First publications on WSRF: January 2004 WSRF Started far apart in apps & tech OGSI GT2 GT1 HTTP WSDL, WS-* WSDL 2, WSDM Have been converging Web Services Resource Framework - WSRF

49 Institute for Software Science – University of ViennaP.Brezany 49 Literatur 1.Grid Computing – Making the Global Infrastructure a Reality. By F. Berman, G. Fox, T. Hey (Eds.), Wiley 2003 2.www.globus.orgwww.globus.org 3.www.gridminer.org (unser Forschungsprojekt)www.gridminer.org 4.Viele Dokumente im Web