1 AGD: ResourceIntegration, Harry Enke (AIP) Components of Grid Development 1. Middleware 2. Resource Integration 3. Connectivity 4. Acceptance of Grid in the Community Sustainability Overview

2 AGD: ResourceIntegration, Harry Enke (AIP) Two pillars of D-Grid concept Communities DGI Problem: Concept of Grid is still not widely percieved as the way of integrating all resources into a collaborative framework for scientific work Example: discussion about german Supercomputer National frame

3 AGD: ResourceIntegration, Harry Enke (AIP) Middleware Complementary development and extention of Middleware stacks from DGI and Communities Starting point is (mostly) Globus 4.x Open Source based and oriented Integrative towards community specific tailored middleware (based on globus) (HEP) und specialized middleware (Unicore) Components of Grid Development

4 AGD: ResourceIntegration, Harry Enke (AIP) Resource integration 1 Level: National integration Unified Certification Policies Building of an efficient certification chain –for users –for services Service Certification is still not sufficient –for hardware (Hosts) it is sufficient –for Services the Certification (Webservices, Compute- Services etc) no unified system –different approaches not integrated (Shibboleth) VO Management still not available on national level Components of Grid Development

5 AGD: ResourceIntegration, Harry Enke (AIP) Resource Integration Level 1: National Integration Obstacles: Resource-providers in scientific community face legal ond procedural obstacles which hamper the grid way provding resources with Vos, especially the SCC Community-Resources are more flexible in that aspect In the long run we need a change of these procedures and substitute them with grid-based usage policies (establishing new comittees for eg. allocating resources to a science project) Components of Grid Development

6 AGD: ResourceIntegration, Harry Enke (AIP) Resource Integration Level 2: Grid-Middleware Accounting and Billing procedures are still not available Reasons: 1. different community procedures and requirements 2. software for these procedures still in concept phase 3. resource integration into a functional grid only now comes into being Job monitoring and quality assessments for QoS agreements for resource usage is one of the crucial steps in the next year, as a base for even raising the question of resource allocation procedures toward grid ways BMBF sponsored considerable hardware resources, which provide a great opportunity for progress, since there are no legal (and local) restrictions imposed Components of Grid Development

7 AGD: ResourceIntegration, Harry Enke (AIP) Connectivity The Grid is hungry for bandwidth Useability of grid integrated resources is very limited without availability of sufficient network connectivity Requirement for scientific communities is: Centers of Grid development are the nodes of DGI amd Communities HEP-Community already has a network concept for the LHC-Experiment at least connect all BMBF hardware nodes with 10GBit/s (Example BaWü: for science institutions a countrywide Gbit network without fees) Components of Grid Development

8 AGD: ResourceIntegration, Harry Enke (AIP) Connectivity For QoS of grid services with respect to network connectivity no middleware stack has any component to offer Without incorporating QoS for network connectivity accounting/billing is only partially possible Components of Grid Development

9 AGD: ResourceIntegration, Harry Enke (AIP) Improving of acceptance of the Grid is one of the most important tastks of the Community grids Getting scientific communites to use the grid infrastructure for their day to day work is one of the most important contibutions towards the sustainability Grid integration of huge projects is a pilot towards this goal Example: LOFAR Building Institute resource grids and VOs, establishing resource sharing on that level is one of the roads Getting more associated partners into Community projects by offering support for middleware installation to get them started Collaboration on international level with other projects within the communities Example : IVOA / GAVO Example : GAIA Example : Ukraine Astrophysics Grid Components of Grid Development

10 AGD: ResourceIntegration, Harry Enke (AIP) Get the resources in a useable state: 1.Deploy our VOMRS on all resources, have all local RA use the tool 2.Ensure a regular update of the gridmaps on all grid resources 3.Run a daily health-checker on each resource thats in MDS to report the running services, the state of the local gridmaps and if possible, the gridusers and their jobs 4.Start an issue tracking module for the resource providers to report on problems with the grid machines, establish a procedure for resolving conflicts with gridjobs and gridusers 5.Draft a set of policies for grid resource usage 1.Basic policy: all AstroGrid-D grid resources are available to all our VO members 2.Exceptions are ok, but only for specialized resources (eg dont try to run a taskfarming job on a cluster, or a program without gravitational interaction on a cluster with GRAPE boards) 3.Local users run with a higher priority than grid users Resource Integration: Next steps

11 AGD: ResourceIntegration, Harry Enke (AIP) Special for clusters: 1.Incorporate more resources from AstroGrid-D, especially the pending clusters 2.Start to define default libraries and environments for the clusters (mpirun etc), document standards for the different local queue-managers and policies 3.Provide above information via commandline facilities as well as hardware info etc, incorporate queue status information Special for workstations: 1.Provide owners (main user) with a means to suspend grid jobs for some time and bar starting new ones if this state is signaled 2.Get this suspended status communicated to the information service 3.Add additional security related monitoring of globus ports wrt security and attacks Resource Integration: Next steps

12 AGD: ResourceIntegration, Harry Enke (AIP) BMBF Hardware for our Nodes: For ZAH and Potsdam already agreed: Installation of same OS (Scientific Linux 4.3 or 4.4) Source installation of Globus 4.03 Start an admin group for these resources across institute borders Ensure the participation of the community already in the planning stages of the integration of all middleware stacks to ensure a forward solution Resource Integration: Next steps

13 AGD: ResourceIntegration, Harry Enke (AIP) Nachhaltigkeitsmodelle: Motivation Ziele und Erfolgskriterien Science: Erzielung von wissenschaftlichen Erkenntnissen durch effiziente Nutzung von Resourcen, Know-How und Personal Erfolgskriterien Wissenschaftlicher Output, Publikationen, Grundlagen für neue Technologien Business: Erzielung von Gewinn durch effiziente Nutzung von Resourcen, Know-How und Personal Erfolgskriterien wirtschaftlicher Erfolg, Gewinn Nachhaltigkeitsmodelle in Science und Business

14 AGD: ResourceIntegration, Harry Enke (AIP) Nachhaltigkeitsmodelle: Möglichkeiten der Zusammenarbeit Motivationen für die Entwicklung von grundlegenden Verfahren und Technologien Science: neue Verfahren werden entwickelt, um in einem bestimmten Gebiet die Erkenntnismöglichkeiten zu erweitern Business: neue Verfahren werden entwickelt, um Konkurrenzvorteile und höhere Gewinnchancen zu erlangen Nachhaltigkeit bedeutet in beiden Kontexten: langfristige Planung der Entwicklungen Science: Grossprojekte haben eine Planungs und Laufzeit von Jahrzehnten, wesentlich bestimmt von Science-Output Business: eine längerfristige Entwicklung ist wesentlich von der Kapitalumschlagszeit / ROI und anderen wirtschaftlichen Gesichtspunkten bestimmt (Zeiträume von <=5 Jahren) Nachhaltigkeitsmodelle in Science und Business

15 AGD: ResourceIntegration, Harry Enke (AIP) Nachhaltigkeitsmodelle: Möglichkeiten der Zusammenarbeit Beispiel für eine mögliche Zusammenarbeit im Grid-Kontext: Wissenstransfer durch Zusammenarbeit mit Hardware-Lieferanten bei der Installation von Default-D-Grid-Middleware auf kleinen und mittleren Clustern für Industrie und Wissenschaft Entwicklung von Methoden zur Bewertung von QoS des Grids Entwicklung von Montioring/Accounting im Grid-Kontext Nachhaltigkeitsmodelle in Science und Business