IceCube Detector Readout Karl-Heinz Sulanke EL-Gruppe DESY Zeuthen 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Inhalt Einführung IceCube DAQ allgemein DOM (Digital Optical Module), kurz DOR (Digital Optical module Readout card), ausführlicher DOR-Produktion und -Test Schlussbemerkung I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

Amundsen-Scott South Pole Station Südpol Skiway Dome AMANDA IceCube 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen IceCube Detector 4/10/2017 K.-H. Sulanke, DESY Zeuthen

Bohrhütte und Schlauchtrommel 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Deployment 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen December 2010: 86 Strings im Eis! 5160 DOMs 4/10/2017 K.-H. Sulanke, DESY Zeuthen

IceCube DAQ, Hardware Komponenten DAQ basiert auf standard IT-Technik (PC-Farm, Switches,…) Ethernet für Datentransfer drei “custom made” Hardware Komponenten: DOM (Digital Optical Module), im Eis MCU (Master Clock Unit) + DSB (Domhub Service Board) GPS-signal-fanout by J. Przybilski, LBNL Berkeley DOR (DOm Readout-card) by K.-H. Sulanke, DESY Zeuthen Spezialkabel für DOM <-> DOR Verbindung Schwedische Firma Ericsson I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

IceCube DAQ Block Diagram DORs I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. DOMs 4/10/2017 K.-H. Sulanke, DESY Zeuthen

DOM (Digital Optical Module) 4/10/2017 K.-H. Sulanke, DESY Zeuthen

DOM Mainboard Block Diagram 4/10/2017 K.-H. Sulanke, DESY Zeuthen 11

DOM Montage und Test in Zeuthen Jürgen Pieper aus der Elektronikwerkstatt Etwa 1500 DOMs wurde in Zeuthen montiert Klimakammer fuer Langzeit-Tests (R. Nahnhauer) mit Hilfe von LED + Lichtleitfasern I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

String Kabel (Ericsson) 4/10/2017 K.-H. Sulanke, DESY Zeuthen 13

IceCube Kabel (3 km lang) 16+2 Quads, 0.9 mm Kupferdraht, „solid wire“ Je ein Quad, bestehend aus zwei verdrillten Paaren, für 4 DOMs 145 Ohm Impedanz, DC-Widerstand <140ohm / 2.5 km Wichtig (!) wenig crosstalk zwischen den Drahtpaaren >50 db suppression near end cross talk >30 db suppression far end cross talk Akkurate mechanische Konstruktion nötig I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen DOR (Dom Readout card) 8 DOMs pro Karte, 32 bit PCI Businterface, DMA fähig In System Firmware Update per Software 96V DC und Kommunikation (je 2 DOMs) über ein twisted wire pair DOM Datenrate, >= 20KB / DOM, -> 48KB -Kommunikation, Paket-basierend inkl. 32 bit Checksumme Zeitkalibration, Genauigkeit besser als 5ns, -> 3ns - virtuell : 5160 gleichlaufende Uhren im Eis !!! GPS Interface (UTC, ASCII Time String -> PCI) DOM Power-Switch (96V) inkl. Strom / Spannungsauslese DOM Power-Ramping DOM Power-Überwachung per Firmware I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen DOR_rev0 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen DOR_rev1b 4/10/2017 K.-H. Sulanke, DESY Zeuthen

DOR - DOM Readout card, Revision 1b Power Control Ch0..Ch3 96 V On Cur Vol Comm. Ch0 ADC / DAC 20MHz PLL 10Mhz 1PPS Time-string 10MHz In0 In1 DOM quad cable In_sel Comm. FPGA Osc. 10MHz Comm. Ch1 ADC / DAC SRAM 1 MByte Cfg Comm. Ch2 ADC / DAC Local Bus 33MHz DOM quad cable PLD Cfg Req FLASH 2 MByte Comm. Ch3 ADC / DAC PCI FPGA JTAG JTAG PLD FPGA PLL-In_sel JTAG Mem. Bus PCI Bus 33MHz 4/10/2017 K.-H. Sulanke, DESY Zeuthen

Cable Interface Scheme FPGA ADC 10 Bit PREAMP 10 Cable Con. +48V -48V RS485 DAC 8 Bit 8 alternative use I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke

K.-H. Sulanke, DESY Zeuthen ASK Encoding, 1MBit/s Rx, 3500m Kabel Tx Derzeit implementiert DC-free, über Trafo eingekoppelt, 1=pulse, 0=kein Signal, Ist default encoding für den DOM nach power on DOM-Datenrate von 50 KByte/sec 4/10/2017 K.-H. Sulanke, DESY Zeuthen

Rx / Tx Data Path, one Wire Pair PCI Bus FPGA Comm. DAC Framing, Encoding Tx_FIFO_A,B 32 8 8 Data_out Data_in Cable Con. Empty AlmEmpty ReadEna WriteEna Address Decoder 2 Wire Pair Control (8) State Machines BusCycle 4 2 Message_rcvd Interrupt Control 1 Interrupt Diff. Rec. Comm. ADC DeFrame, Decoding Rx_FIFO_A,B 10 8 Data_in Data_out I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. AlmFull Empty WriteEna ReadEna Internal FIFOs will be replaced by external SRAM 4/10/2017 K.-H. Sulanke

K.-H. Sulanke, DESY Zeuthen Firmware, ein Beispiel Statemachine zur Steuerung der halbduplex Kommunikation mit den DOMs A,B 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Zeitkalibration I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. Kalibrations-Zyklus komplett in Firmware realisiert initiert durch Software, typ. 1 x pro Sekunde 4/10/2017 K.-H. Sulanke, DESY Zeuthen 23 23

Time Calibration In-ice DOMs Time IceTop automatic, every few seconds for 76 DOMs

K.-H. Sulanke, DESY Zeuthen 8B10B Encoding, 2MBit/s Tx Rx, 3500m Kabel Verdoppelung der Datenrate Erste Labortest waren o.k. (1 x DOR + 8 DOMs), auch Mix von ASK / 8B10B Bei vollen Domhub ( 8 x DOR, 60 DOMs) , unter Linux, gab es Probleme Linux-Treiber, zu hohe Gesamtdatenrate ? Firmware –Bug ? 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen DOR Bestückung Prototypen, Bestueckung in der Zeuthener Elektronik-Werkstatt Serienfertigung für etwa 800 boards extern Hersteller I: Mittelstaedt / ESL GmbH, Bln. Tempelhof etwa 30+60 boards kostengünstig, aber … veraltete Produktionsanlagen (Stand Ende 2004) Probleme (keine Erfahrungen) mit dem BGA-Löten kein Löten unter Schutzgas (war in Aufbauphase) I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

DOR Bestückung, fortgesetzt Hersteller II: Intratec / Elbau GmbH, Bln. Weissensee etwa 800 boards geliefert moderne Produktionsanlagen Löten unter Schutzgas keine Probleme mit BGAs geringe Probleme, verursacht durch Handbestückung LED 180° gedreht (1x) einzelne Pins nicht gelötet (~4x) Komplettangebot inkl. Materialbeschaffung möglich kann empfohlen werden I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen DOR Test Erste Firmware-Tests unter DOS, 130 Mhz Pentium, 32MB Speicher -> Borland_C, Templates fuer Linux Treiber (by John Jacobsen) Nach Herstellung, 100%-iger Funktionstest Test unter Linux Test Software von Arthur Jones, LBNL Berkeley, California Selbsttest mit Hilfe rückgeführter Ausgänge / Testadapter Testlog wird im DOR-Flash abgelegt einmalige Nummer per Server vergeben und aufgeklebt R1B0674D05 Rev. 1b abs. Nr. Serien Nr. I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

DOR Fehleranalyse, bezogen auf 225 Boards sofort funktioniert (Fa. Intratec) : ~85 % nach “einfacher” Reparatur, Ausbeute : ~ 98% LP-design, Fehler durch schlechte Footprint-Libraries schlechte Pad Geometrie versursacht Lötfehler z.B. SO-8 Footprint mit zu kurzen Pads echte Bestückungsfehler durch verbogene Pads (schlechter Lieferzustand) durch Handbestückung durch schlechten Lötpastendruck (selten) defekte Bauelemente: 2x 5V/5V - DC / DC Wandler Operationsverstärker OPA237NA I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen DOM hub From GPS Unit DOR card DSB Card CPU Cat5-cable with: 10 MHz 1 Pulse per Second RS-232 Time String 10 MHz, 1 pps. Time String Power Supplies Fans Monitors I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

DOM hub (Industrie-PC) DOM Power Supplies Power Distr. Card Chassis Fans Hard Drive I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. Backplane with 12 PCI slots DOR Cards (2 of 8 shown) CPU DSB for GPS distr. ~300 W running 60 DOMs 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Rack with DOMhubs LEDs for Rx, Tx and DOM-Power PCI Bus access I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

Unterkunft ->Zukünftiges IceCube Lab 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen IceCube Lab im Winter 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Schlussbemerkungen IceCube Auslese funktioniert zuverlaessig, Detektor uptime >99% Firmware / Software Entwicklungsaufwand war betraechtlich Kommunikations-Protokoll zu komplex (?) PINGU als evtl. IceCube Erweiterung, 10 Gev...100 GeV Auslese basierend auf PCIe, uTCA, ATCA (?) Verwendung der existierenden Software moeglich (?) Besseres Encoding (?), z.B. PSK (Phase Shift Keying) I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Verwendete Quellen Animationen, entnommen dem IceCube docushare/public/ „Design and Performance of the IceCube Electronics“, Vortrag von R.G. Stokstad (LBNL Ca.), 2005 eigene Dokumente und Bilder I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen 36 36

Danke für die Aufmerksamkeit ! I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Backup Slides I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen IceCube String 60 optical Main Cable sensors DOM Base with 1400 m HV generator Electrical feedthrough for power + data OM Spacing: 17 m Photomultiplier DOM Board Gel String Glass pressure sphere. Rated to 10000 psi. Outer diameter: 13" 2400 m 4/10/2017 K.-H. Sulanke, DESY Zeuthen 39

K.-H. Sulanke, DESY Zeuthen DOR Leiterplatte Entwurf: Reiner Roitsch, Berlin 8 Lagen, 311 x 122 mm, SMD beidseitig Technik: part. chem NiAu auf Cu 3 BGAs, 484 / 256 / 100 pins Herstellung Fa. Mittelstaedt, schlechte Qualität Fa. Gillett, schlechte Daten-Vorverarbeitung Fa. straschu, ausreichend gute Qualität I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. 4/10/2017 K.-H. Sulanke, DESY Zeuthen 40 40

Drill tower Hose reel Hot water generator IceTop tanks The drilling site in January, 2005

Each 2 m dia. IceTop tank contains two Digital Optical Modules Each 2 m dia. IceTop tank contains two Digital Optical Modules. The freezing of the water is done in a controlled manner to produce clear ice.

K.-H. Sulanke, DESY Zeuthen Data Packet Format 4/10/2017 K.-H. Sulanke, DESY Zeuthen 43

K.-H. Sulanke, DESY Zeuthen Zeitkalibration I would like to present you two examples of fast link interfaces, designed in our lab. For both cards block schemes, test results and conclusions will be presented. Kalibrations-Zyklus komplett als Firmware realisiert initiert durch Software, z.B. 1 x pro Sekunde 4/10/2017 K.-H. Sulanke, DESY Zeuthen

Timing verification with flashers 1.74 ns rms { All 60 DOMs

Alte und neue Station (im Bau befindlich) 4/10/2017 K.-H. Sulanke, DESY Zeuthen

K.-H. Sulanke, DESY Zeuthen Polemarker 4/10/2017 K.-H. Sulanke, DESY Zeuthen