PCI Express DMA Engine für Active Buffer Projekt im CBM Experiment Wenxue Gao, Andreas Kugel, Reinhard Männer, Holger Singpiel, Andreas Wurz Uni. Mannheim DPG Tagung, Gießen 14 März 2007

Inhalt Einleitung Blockdiagramm Realisierung Leistung 2 von 15

Einleitung – CBM Experiment CBM TSR, Jan. 2006

Einleitung – PCI Express 2,5 Gbps pro Link Point-to-Point TLP (Transaction Layer Packet) Post: MWr (Memory Write Request), … Non-post: MRd (Memory Read Request), … Completion: CplD, Cpl, … Message: Msg 4 von 15

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx Tx

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx Tx MWr1

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx Tx MWr1

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx MWr1 Tx

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx MWr1 Tx MWr2

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx MWr1 Tx MWr2

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx MWr1 MWr2 Tx

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx MWr1 MWr2 Tx MWr3

PCI Express – Post TLP (MWr, …) Trn. Host End-Point Rx MWr1 MWr2 Tx MWr3

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx Tx

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx Tx MRd1

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx Tx MRd1

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx MRd1 Tx

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD2 CplD1 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD2 CplD1 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD2 CplD1 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 CplD2 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 CplD2 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 CplD2 MRd1 Tx MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 CplD2 MRd1 MRd1 Tx MRd2 MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 CplD2 Tag[7:0] MRd1 MRd1 Tx MRd2 MRd2

PCI Express – Non-post TLP (MRd, …) Trn. Host End-Point Rx CplD1 CplD2 Tag[7:0] MRd1 MRd1 Tx MRd2 MRd2

Einleitung – SG DMA SG(Scatter/Gather) Voll-Duplex „Done“ Zustand Multiple-descriptor chain Voll-Duplex Downstream: Host  Endpoint Upstream: Endpoint  Host „Done“ Zustand Status Register Interrupt Downstream Upstream Host Endpoint

Blockdiagramm PCIe Transact . Layer Interface Tag Channel Buffer RAM Rx Tx Tx Arbitrator Memory BRAM + FIFO + Registers Upstream DMA Channel Downstream PIO Rx Resolution PCIe Transact . Layer Interface Channel Buffer Tag RAM

Channel Buffer TLP Channel FIFO TLP ohne Payload TLP mit Payload Breite = 128 Tiefe = 15 TLP ohne Payload Alles im Word TLP mit Payload Lokale Adresse Zusätzliche Informationen LAdr Hdr2 Hdr1 Hdr0 Rx Tx xxxx 95 127 63 31 9 von 15

Realisierung – DMA teilen 4 KB Grenze verboten Address/Length Combination

Realisierung – „Done“ bestätigen Wann ist DMA beendet? „Done“ Zustand nötig CplD‘s für unterschiedliche MRd‘s kommen nicht folgend Mögliche Lösungen Tag RAM lesen CplD zählen Channel Buffer leer Letzten Tag triggern (x) Bitmap füllen 128-bit Register für 7-bit Tags 11 von 15

Leistungsparameter Zielbaustein FFs LUT4s RAMb16 Slices Virtex4 XC4VFX60-11ff672 FFs 9 834 out of 50 560 ( 19 % ) LUT4s 11 464 out of 50 560 ( 22 % ) RAMb16 58 out of 232 ( 25 % ) Slices 9 426 out of 25 280 ( 37 % ) Frequenz ( trn_clk ) 250 MHz Verzögerung (Transaction layer) PIO: 52 ns (MRd  CplD ) DMA: 80 ns (DMA „Start“  Tx TLP) Theoretische Bandbreite 2Gbps x4 = 8Gbps, bi-directional 12 von 15

4-Lane Tests

Offene Fragen Kleinerer Channel Buffer Bessere Behandlung von Fehlern Meistens reichen 64-bit, statt 128-bit Bessere Behandlung von Fehlern Teilweise unvollständig Überschreiben von CplD zu vermeiden Time-out tag Recycling Höhere Bandbreite für downstream DMA

Zusammenfassung PCI Express Vorteile Virtual channels Xilinx Lösung Parallelität Skalierbarkeit Virtual channels 2 DMA Channels 1 PIO Channel Xilinx Lösung 62,5 MHz für x1 250 MHz für x4 15 von 15

x4-ABB Design Summary -------------- Logic Utilization: Number of Slice Flip Flops: 9,834 out of 50,560 19% Number of 4 input LUTs: 11,464 out of 50,560 22% Logic Distribution: Number of occupied Slices: 9,426 out of 25,280 37% Total Number 4 input LUTs: 12,993 out of 50,560 25% Number used as logic: 11,464 Number used as a route-thru: 643 Number used for Dual Port RAMs: 202 Number used as Shift registers: 684 Number of bonded IPADs: 18 out of 62 29% Number of bonded OPADs: 16 out of 24 66% Number of bonded IOBs: 1 out of 352 1% Number of BUFG/BUFGCTRLs: 5 out of 32 15% Number used as BUFGs: 4 Number used as BUFGCTRLs: 1 Number of FIFO16/RAMB16s: 58 out of 232 25% Number used as FIFO16s: 0 Number used as RAMB16s: 58 Number of DSP48s: 2 out of 128 1% Number of DCM_ADVs: 1 out of 12 8% Number of GT11s: 8 out of 16 50% Number of GT11CLKs: 1 out of 8 12%

X4 Test

DMA Prozess Buffer-descriptor Start/Stop Befehl SA (Source Address) DA (Destination Address) NXA (Next Descriptor Address) Length (Length in bytes) Control (Control register) Start/Stop Befehl Upstream: MWr + MRd (dex) Downstream: MRd Busy/Done Zustände erkennen Status Register Interrupt (Msg)

Blockdiagramm Rx Memory BRAM + Registers + FIFO Tx Tx Arbitrator MWr_usp MRd_dsd MRd_usd MRd_dsp Cpl/D MWr Memory BRAM + Registers + FIFO Tag RAM MRd: CplD Cpl MRd Rd Wr Rx Resolution US: Msg DS: DMA Upstream Engine Registers Downstream

Verifizieren PIO + DMA ($random) Output checking Transaction length Address-pair Chain length (DMA) Descriptor Address (DMA) Flow control: *_rdy_n Output checking tsof/teof Data Deskriptor abteilen Downstream (Write) Upstream (Read) Root Endpoint 1 2

Memory Space BRAM FIFO Registers Eventuelle Erweiterung 16KB 32 x 32 Loop-back Registers Write / Read Control / Status Eventuelle Erweiterung DDR (BRAM ähnlich) GbE (FIFO ähnlich) BRAM Registers Loop-Back Wr Rd OFIFO IFIFO