UNICORE – Towards production quality grid environment Piotr Bała N. Copernicus University – Toruń (PL) ICM Warsaw University (PL) Funding by European.

Präsentation zum Thema: "UNICORE – Towards production quality grid environment Piotr Bała N. Copernicus University – Toruń (PL) ICM Warsaw University (PL) Funding by European."—  Präsentation transkript:

1

2 UNICORE – Towards production quality grid environment Piotr Bała N. Copernicus University – Toruń (PL) ICM Warsaw University (PL) Funding by European Commission Grant No. IST-1999-20247 (EUROGRID) and IST-2001-32257 (GRIP)

3  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała2 EUROGRID Vision Build a European Grid infrastructure that gives users a seamless, secure access to High Performance Computing resources and that advances computational science in Europe

4  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała3 GRIP Project Extend EUROGRID to cooperativity with globus middleware. Provide UNICORE users with access to resources avaliable trough globus.

5  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała4 EUROGRID Infrastructure and Components  Based on UNICORE system  Develop additional GRID components efficient data transfer ASP infrastructure resource broker application coupling interactive access  Integration of new components by Intel (former Pallas) and Fecit

6  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała5 UNICORE architecture UNICORE Client UNICORE Gateway A UNICORE Server (NJS) TSI NQE (Cray T3E) UNICORE Gateway B UNICORE Server (NJS) TSI PBS (Linux cluster) Insecure Internet

7  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała6 UNICORE Security  Based on the PKA  Industrial standard X509  Secure communication Gateway, NJS cetificates Gateway, NJS check user certificate Multiuple CA accepted  User certificates User certificate stored in client Public key stored in UUDB database at each site PKA mapped to user account (XLOGIN) –more than one certificate can map to xlogin –multiple CA allowed Multiple certificates allowed

8  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała7 UNICORE  Stable and reliable  Allows access to production systems  OpenSource  Commercial Support Available (UNICORE Pro)  Interoperability with Globus (GT2, GT3)  (see R. Menday talk tomorrow)

9  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała8 UNICORE  Stable and reliable  Allows access to production systems  OpenSource  Commercial Support Available (UNICORE Pro)  Interoperability with Globus (GT2, GT3) (see R. Menday talk tomorrow)  Pratial support of OGSA standard (see next talk)

10  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała9 EUROGRID resources  European HPC GRID testbed  Agreement on security standards, certification, access policies etc. CRAY SV1 (16PE) SUN E1000 (24PE) SGI O2000 (32PE) Linux Cluster (4 PE) Linux Cluster (16PE) SGI O2000 (64 PE) SGI Onyx 2000 (64PE) IBM SP4 CRAY T3E - 1200 (816 PE) FUJITSU VPP300 (8 PE) SGI O2000 (128 PE) SGI O3000 (256 PE) IBM SP3 (8 PE) NEC SX5 cluster (40 PE) IBM Power4 (256 PE, 1.3 TFLOPS) COMPAQ Linux Cluster (24 PE) Linux Intel Cluster (36 PE) IBM (1PE) CRAY T3E - 600 (512 PE) CRAY T3E - 1200 (512 PE) FZJ

11  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała10 UNICORE Applications  Job files for user applications script tasks command tasks file transfer tasks (client-vsite, vsite-vsite)  Plug-in for job and input preparation  User can run the same script at all VSITEs Introduction of IDB entries for site dependent installation GROMOS96=/pkg/gromos96; export GROMOS96 GAUSS_EXEDIR=/pkg/gaussian/g98; export GAUSS_EXEDIR AMBERHOME=/pkg/amber6; export AMBERHOME CHARMM=/pkg/charmm/c27b2/exec/t3e/charmm; export CHARMM

12  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała11 BioGRID  UNICORE Client (4.1.5 or UNICORE Pro 1.0.7) unix and Windows Easy installation (www.unicore.org, biogrid.icm.edu.pl)  Example jobs: Gaussian98, TBDFT, Amber, Gromos  Plugins Gaussian98 AMBER CPMD (V. Huber FZJ) BLASS (S. Haubold, TU Dresden) PDB Search Sequence Search Visualization Plugin PluginLoader List Jobs

13  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała12 UNICORE Client  Single application  Job preparation  Job monitoring

14  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała13 UNICORE Client

15  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała14 UNICORE Client – Gaussian98 job preparation

16  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała15 UNICORE Client – edit resources for job

17  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała16 Gaussian98 in EUROGRID

18  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała17 Gaussian98 plugin Input preparation reads existing input, recognizes keywords Molecule coordinate editor formats: XYZ Z-matrix text

19  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała18 Gaussian98 plugin CPU time estimate based on known algorithm scaling O(N 4 ) Checks program avaliability

20  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała19 Gaussian98 plugin Prepares and transfers files for visualization

21  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała20 Gaussian98 plugin Output

22  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała21 Gaussian Plugin J. Pytliński, A. Nowiński Visualization panel

23  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała22 Gaussian Plugin J. Pytliński, A. Nowiński

24  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała23 Gaussian Plugin J. Pytliński, A. Nowiński

25  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała24 Gaussian Plugin J. Pytliński, A. Nowiński

26  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała25 Amber 6.0 plugin Ł. Skorwider  Input preparation extended input preparation wizard covers all most popular options for NVT, NVE, NPT simulations NMR simulations are not covered yet  Extended user help short help available for each button Keyword list and keyword locator –user is navigated to the proper panel JavaHelp entries  Plugin checks Amber availability SOFTWARE_RESOURCES entries

27  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała26 Amber plugin Input preparation

28  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała27 Amber plugin Input can be edited by user Errors are marked in red

29  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała28 Amber plugin Panel for each group of options Default options Help window

30  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała29 Amber plugin Find parameter button

31  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała30 Visualizer Plugin K. Benedyczak

32  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała31 Visualizer Plugin K. Benedyczak

33  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała32 CPMD plugin V. Huber (FZJ) CPMD input

34  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała33 CPMD plugin CPMD wizard

35  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała34 PDB search plugin M. Wroński  Prepares database query: Protein Data Bank (PDB) database Sequence database  Databases available through web interface Plugin uses HTTP protocol to generate query Database mirrored at various locations  Output visualization PDB structure can be visualized in the plugin using external software –JMV (Java Molecular Viewer) –K. Shulten group – Chicago –Requires Java3D –JMol –Sourceforge –RasMol/RasWin

36  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała35 PDB search plugin Lite search

37  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała36 PDB search plugin Advanced search

38  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała37 PDB search plugin User can use different mirrors The best one is selected (fastest answer)

39  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała38 PDB search plugin Search results

40  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała39 PDB search plugin Visualization with external packages

41  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała40 PDB search plugin Rasmol JMV Jmol User- defined

42  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała41 Sequence search plugin M. Wroński Search interface

43  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała42 Sequence search plugin Datailed view

44  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała43 Sequence search plugin Simple visualizatio based on BIOJava

45  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała44 Blast Plugin S. Haubold (TU Dresden)  Input preparation

46  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała45 List All Jobs Plugin K. Benedyczak  List all users jobs Scans USITEs and VSITEs Lists Job information Information on running jobs –Alternative mechanism based on the additional buttons which can be defined in the Job window can be used. Direct connection to JMC pannel

47  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała46 List All Jobs Plugin

48  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała47 File Filter Plugin J. Wypychowski  Access to the work directory of the running job  Filter status file Status file Intermediate files (eg. coordinates)  Display results in the Client  Task running as separate job Immediate queue User has to be able to run two jobs

49  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała48 File filter plugin Pick up job from JMC Define file to process Submit job

50  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała49 File filter plugin Tail of the Amber status file

51  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała50 DBAccess Plugin M. Wroński Access to the SQL databases  MySQL  Postgress ... Results can be saved in the file XML output

52  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała51 Generic plugin - basic idea J. Wypychowski  Plugin is not a higher level language program No Java  Applications typically require a set of parameters from a basic set of types (integer, string, float)  Constraints for the input data must be available  UNICORE tasks and interaction must be simple  Interface should generate itself on the fly  Text based configuration (XML)  Independent from UNICORE internal representation

53  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała52 GenericPlugin - implementation  Plugin within plugin – generic plugin generator  XML based configuration with strict grammar definition  Small set of basic parameter types  Auto-generated interface  Constraints and parameters are scriptable with JavaScript  Simplified UNICORE tasks and data  Common API to create application plugins  VSite based application plugin storage

54  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała53 GenericPlugin in UNICORE

55  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała54 Script Plugin in XML 01 02 03 04 sh 05 06 07 08 sh 09 11 12 13

56  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała55  Access to the external devices Thermometer connected through serial port to TSI server Deamon for the device running on the target system DeviceSteer Plugin DeviceSteer Krzysztof Benedyczak

57  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała56 DeviceSteer

58  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała57 DeviceSteer

59  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała58 BioGRID  Operate a GRID for biomolecular simulations  Develop interfaces to existing biological and chemical codes  Web site: biogrid.icm.edu.pl

60  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała59 Web site Information on software  info  home page  local info  VSITEs  USITEs

61  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała60 Web site Step by step user guides

62 Funding by European Commission Grant No. IST-1999-20247 (EUROGRID) and IST-2001-32257 (GRIP) biogrid.icm.edu.pl www.eurogrid.org www.grid-interoperability.org Piotr Bała J. Pytliński, Ł. Skorwider, M. Wroński, K. Benedyczak, J. Wypychowski, A. Nowiński, M. Nazaruk, K. Wawruch ICM Warsaw University (PL) N. Copernicus University – Toruń (PL)

63  Hier klicken, um Master-Textformat zu bearbeiten.  Zweite Ebene CGW03 27/10/2003Piotr Bała62 EUROGRID Workshop  Eurogrid Workshop (29.10.2003) Cracow Grid Workshop 2003 Kraków, 27-29.10.2003 UNICORE Tutorial 28.10.2003 - evening –Room for 30-40 people Lectures29.10.2003 – morning –UNICORE overview- K.-D. Ortel, Intel –Meteo-GRID: Performing Local Weather Forecast Using GRID Computing- K.-J. Lenz, DWD –Globus/Unicore interoperability - R. Menday, FZJ –Plugin development - K. Benedyczak, M. Wroński ICM/UMK –Unicore in OpenMolgrid project- M. Romberg, FZJ