06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 1 Vertex Processor Board...

06/12/2006Wilco Vink / Martin van Beuzekom

Leo Wiggers / Sander Mos 1

Vertex Processor BoardOptical Tx Board

PRR Pile-Up System



Outline• Pile Up System overview

– Dataflow– Data reordering– Output Board

• Optical Tx board– Functionality– Board specs– Clock tree– Radiation hardness– Board tests– PCB changes needed

• VEPROB– Functionality– Board specs– Board tests– Concluding remarks

• Interconnection tests• Conclusions / Outlook



Pile-Up System overview

Small system, but still:4 different types of active boards (some highly complex)6 different types of passive boards



OpticalTx

Boards

Dataflow• All data of one event (2048 bits) goes to a single processor

– I/O limitation -> multiple (4) processors in round robin– Each event transmitted in 4 clock cycles

• Getting the bits at the right place at the right moment– Simultaneous transmission of ¼ silicon data– Algorithm per ¼ sensor (left / right in parallel)

event 4 3 2 1

OpticalTx

Boards

Veprob 1Veprob 2Veprob 3Veprob 4



Data reordering

Data reordered via:– Passive repeater cards– Cat6 cabling to balcony– Passive transition boards– Optical TX boards– Optical patch panel

Seems simple, but:



Complications:• First/second Si wheel mirrored• Complex layout of sensor

– Mix of stripno - beetle channo and stripno - beetle channo

– Gives a twist when multiplexed in time, hence 2 flavors of Optical TX boards (jumper selectable)

• Layout of hybrid is not compromised– Unavoidable routing irregularities

– Signals on 3 different repeater cards

– 2 types of transition boards

J12a-low

J11d-low J11c-low J11a-low

J13c-high J13a-highJ13d-high


3_4_7

3_8_11

3_12_15

4_4_7

4_8_11

4_12_15

11_4_7

11_8_11

11_12_15

12_4_7

12_8_11

12_12_15

3_0_3

4_0_3

11_0_3

12_0_3

0_0_3

7_0_3

8_0_3

15_0_3

0_4_7

0_8_11

0_12_15

7_4_7

7_8_11

7_12_15

8_4_7

8_8_11

8_12_15

15_4_7

15_8_11

15_12_15

J14c-high J14a-high


J14d-high

J11d-high J11c-high J11a-highJ11d-highJ11d-high

J12c-high J12a-highJ12d-high

J12d-low J12c-low

Project: LHCb

Project part: Pile-Up

Date 23-06-2006

This overview displays all interconnects to Transition Board Inner.

File: trajects_RPT_CD_TransBrd_Inner

It covers all signals from Repeater Board C and half of the signals from Repeater Board D.The displayed interconnect colors do not correspond with the Excel listing colors.

0_0_3

0_4_7

0_13...15

Repeater C Repeater D Repeater B

RPT B

Transition Board Inner

RPT C RPT D

Trajects with at least 2 differential pairs, passing through RPT C.

Traject parts with at most 2 differential pairs, both RPT C and RPT D.

FPGA input groups

3_4; 6; 7

3_8_11

3_12_15

4_13

4_4_7

4_14 ; 15

4_0_3

7_4

7_8_11

7_12_15

8_14 ; 15

8_4_7

8_0_3

7_6 ; 7

8_13

11_6 ; 7

11_8_11

11_12_15

11_4

12_4_7

12_13...15

12_0_3

15_4; 6; 7

15_8_11

15_12_15

Rj45 connectors

8_13

7_4

11_6 ; 7

11_4

4_13

0_12

7_5

8_1215_5

3_5

4_12

11_5

12_12

0_12

15_5

3_5

11_5

12_12

15_0_3

12_8_11

7_0_3

4_8...10

0_8_11

3_0_3

Outputs:Inputs:

Inputs: Outputs:

Trajects with at least 2 differential pairs, passing through RPT D.

Trans.Board Trans.BoardUpper Lower

Time 14:00Nikhef / ET / jpfrans

4_14 ; 15

Outp. Outp.

4_11 ; 12

7_5

11_0_3

4_11

8_8...10

8_11 ; 12

7_6 ; 7

8_11

J1ahJ1ap

J1al J1ad

J1anJ1af

J1am

J1ae

J1ak

J1ai

J1ag

J1aa

J1ab

J2

J3

J1bp

J1bo

J1bn

J1bh

J1bg

J1bf

J1be

J1bm

J1bd

J1bl

J1bk J1bc

J1bj J1bb

J1bi J1ba

J1

J1h

J1e

J1c

J1a

J3a

J3b

J3c

J3d

J3e

J3g

J3h

J2b

J2f

J2d

J1c

J1g

J1e

J2f

J2b

J1g

J1d

J1f

J1b

J2e

J2c

J2a

J3f

J2e

J2d

J3h

J3b

J3f

J1ao

Data reordering, some details

Repeater -> optical board connections (1 of 2)



PRR

• Active boards– Hybrid, reviewed Jan 2006– Optical Tx Board : today– Processor Board : today– Output Board : no proto yet



Output Board• Merges Pile Up decision from 4

Processor Boards• Large number of monitoring histograms• Board items

– 9Ux400mm 2.4 mm 8 layer pcb– Xilinx Virtex 4

• FPGA 10% of logic and 85% of memory used

– 4 serial inputs 1.6 Gb/s copper– Latency Virtex 4 might be too long

• Fallback -> 4 extra serial inputs with TLK2501

– 2 optical serial outputs -> L0DU– CCPC / Glue Card– TTCrq

• FPGA code almost done & partially simulated

• Schematic entry in progress, finished before the end of 2006

• Layout finished in January 2007• Board production finished March 2007

DES

DES

DES

DES

FIFO

FIFO

FIFO

FIFO

MUX DEMUX

SER

SER

HISTOGRAMMING& MEMORY

ERR. COUNT

Input: 4 x 1.6 GBit

Ouput: 2 x 1.6 GBit

Xilinx Virtex 4 XC4VFX60

OpticalTransmitters

To L0DU

TTC RQ CCPC Output board



Optical Tx Board



Optical Tx Board

• Data input– Clock synchronization– Time alignment

• Event time multiplexing• BCID labeling• VELO control board interface

– ECS• I2C

– TFC (TTCrx)• LHC clk• BCID reset• Power on reset

• Clock de-jitter and distribution• Power-Up

– Gol(crt4t) POR via ECS – Flash based FPGA

128 LVDSData 80 Mb/s LVDS receivers

GOL

Data sync Event

multiplexingBCID labeling

LHC clock

Qpll-DeJitterClocktree

ParallelOptical

transmitterJTAG (lab only)

I2C, POR, BCIDrst

Optical Tx Board

GOLGOL

Actel APA300

Power

VE

LO

ctrl brd

12 VEPROB



Optical Tx Board PCB details

• 9U X 400mm double height • 10 Layer 1.6mm

– 4 power layers (GND, 2v5, 3v3, GND)

– 6 routing layers• Power via standard LHC

backplane (TELL1)• Custom backplane P2 P3 (cPCI)

LVDS inputs via transition boards in backside of crate

• Controlled impedance differential traces

• Signal integrity verified with ICX (Mentor Graphics)



Data to clock synchronization

• Time align all inputs• Compensate timing differences in cables & PCB

traces• Input sampling

– Four FF on different clocks edges• 3.25 ns resolution

– Max 12.5 ns compensation• Sampling phase of each input bit selectable via

ECS• Global clock phase, via delay selection (delay25)

on VELO control board (0.5 ns steps)

DFF

DFF

DFF

DFF

D

DFF

DFF

DFF

clka

clkb

clka

clkb

clkselect(0)

clkselect(0)

clkselect(1)

clka

clkb

Beetle data

ff1 ff2 ff3

Fine tune clk delay (control board)



OpticalTx

Board

Event multiplexing

• 8 Optical Tx Boards in optical station multiplex events to four processor boards– Time multiplexing four

quarters, each in 25ns • Each processor board

processes one complete (out of four) event– large number of strip inputs

to vertex finder algorithm

event4 3 2 1

OpticalTx

Boards

Veprob 1Veprob 2Veprob 3Veprob 4



Clock distribution• LHCb clock (40.08 MHz) received from VELO control board

– Optical Tx Board is synchronous system• Clock must be de-jittered for GOLs

– QPLL on board• Board uses both 40 and 80 MHz clock• Clock distribution based on MC100LVEP family

– LVPECL standard– Similar to MUON ODE Board

QPLL

1 to 10 clockDriver

(MC100LVEP111)

LVDS 40MHz

LVPECL 1 to 5 clockDriver

(MC100LVEP14)

LVPECL

GOlGOl

GOlGOl

3X

Diff-> single ended(DS90LV048)

FPGA(Actel ProAsic)

1 to 10 clockDriver

(MC100LVEP111)

LVDS 80MHz



PassiveDelay

Line (3 ns)

LVPECL

LVPECL

LVPECL

LVCMOS

LVCMOS

LVCMOS

3X

LVDS 40MHz

Optical Tx Board



Radiation Hardness

• Optical station– Located on balcony, 10m from beam– Radiation dose 640 Rad in 10 years (including safety factor 2)– Additional safety factors: 2x2 => dose 2.5 kRad in 10 years– All static registers triple redundant

• Single event upset counters – All active devices qualified for radiation hardness

• Rest of trigger logic– Behind wall



Radiation Hardness(2)Component Tested by Technology TID reference

GOL CERN MIC Rad-Tol http://proj-gol.web.cern.ch/proj-gol/publications/paperLEB2001.pdf

CRT4T CERN MIC Rad-Tol http://mic-asics.web.cern.ch/mic-asics/documents/CRT4T.pdf

Qpll CERN MIC Rad-Tol http://proj-qpll.web.cern.ch/proj-qpll/images/qpll2IrradiationResults.pdf

LHC4913 CERN MIC Rad-Tol http://rd49.web.cern.ch/RD49/

Agilent transmitter

HFBR772

LHCb Muon

Marseille

COTS 15krad LHCb note 2004-013

Actel ProAsicPlus LHCb Muon

INFN – LNF

COTS 68 krad ODE PRR presentation

Clock driver

MC100LVEP111

LHCb Muon

INFN – LNF

COTS 68krad ODE PRR presentation

Clock driver

MC100LVEP14

LHCb Muon

INFN – LNF

COTS 68krad ODE PRR presentation

LVDS receiver

DS90LV048ATMTC

Atlas Muon

RPC

COTS 70krad http://sunset.roma1.infn.it:16080/muonl1/reviews/12march2002/RPC_RHA_status.pdf

LVDS driver

DS90LV047ATMTC

Atlas Muon

RPC

COTS 70krad http://sunset.roma1.infn.it:16080/muonl1/reviews/12march2002/RPC_RHA_status.pdf

Open drain Buffer

MC74LCX07DTG

Atlas Muon

RPC

COTS 780krad LHCb note 2006-01



Optical Tx Board tests• ECS control

– I2C via VELO control board OK – Control via PVSS panels read/write OK

• Minor changes in SDA/SCL termination

• TFC– Fast controls via TTCrq on VELO control board– LHC LVDS clock 40 MHz via delay line (delay25) OK– (BCID reset from TTCrx)

• Power up GOL– 2v5 power via CRT4T for each GOL– On/Off via power on reset (POR VELO control board)

OK • Clock jitter measurement

– De-jitter by Qpll – Low jitter clocktree– GOL always locked OK

• Link test results later in presentaion

• Input timing– Verified OK

• Board temperature– Temperature cycling, to be done– Crate doesn’t fit in climate chamber– All devices T < 45 C

Edge to edge σ=18 ps

10u period σ=26 ps

100u period σ=72 ps



PCB changes neededFew bugs in layout• Center pad shorts pads(CRT4T QPLL)

– Solution on proto, center pads isolated with kapton tape

– Not wanted for series production– Geometry changed:

• CRT4T done• QQPLL done

• AC coupling in gigabit traces between GOL’s and HFBR772 missing

– Capacitors added on proto– To be changed in layout

• Mirrored geometry JTAG connector (Actel programmer)

• I2C SCL termination changed

All modifications “local” -> very low risk

Center pad



VErtex PROcessor BoardVEPROB



Optical Ribbon

Orc Card

VEPROB

• 24 input fibers• Xilinx Virtex2-pro100

FPGA 1704 pins– 24 TLK input busses– CCPC Glue-Card

interface to ECS– 1.6 Gb/s outputs (Xilinx

Rocket I/O, MGT)• TTCrq timing fast control

signals– L0 trigger – LHC clock– BCID reset

• ECS via CCPC and Glue card

Optical Ribbon

Orc CardChannel

Sync

Ethernet

CCPCGlue Card

Opticaltransmitters

JTAG

Xv2pro100

VertexFinder

Opt in.TTCRq

Monitor&

control

VEPROB

MGT

L0buffer MGT

MGT

TFC

ECS

Output Board

TELL1



VEPROB vertex finder

• Vertex finder algorithm in single FPGA

• Each event 8 groups of 256 input strips

• Correlation matrix number of combinations 4082 (left) + 3594 (right)

• 1st , 2nd peak position & height– 128 bins (1,2,3,5 mm width)

• Multiplicity (# active strips)• counters & histogramming:

70% FPGA resources• Latency: 875ns (35LHCclks) L0 buffer

1st peak finder

Peakmasking

Vertex finder

Beam-150 0 150

Z-position

Top view detector

Al Bl

BrAr

right

left

Sensor:

Decisionlogic

CorrelationMatrix

+histogram

2nd peak finder

CorrelationMatrix

+histogram

pipeline

Strip data

Multiplicity



VEPROB PCB details• 9U X 400mm single height • 12 Layer 1.6 mm PCB

– 4 power layers (1v8, 2v5, 3v3, GND)– 8 routing layers– Stiffener across board

• Main functionality in Xilinx Virtex2Pro100 FPGA 89% slices used

– LUT4 82%, RAM 20%, registers 28%• Gigabit LVDS traces & data path ORC to

FPGA 50 Ohm controlled impedance• Front side I/O

– 2 optical ribbon, 2 ORC Mezzanines• Back side I/O

– Power via standard LHC backplane (TELL1)

– TTCrq Mezzanine fiber– CCPC & Glue-Card, ethernet – Output to TELL1 buffer– Output to Output Board



VEPROB tests

• ECS control– Slow controls and monitoring via CCPC and Glue card

• TTCrq via I2C OK • FPGA access registers and memories through local bus OK

• TTCrq– 40 & 80 MHz QPLL LVDS clocks to FPGA OK

• FPGA / algorithm– Data path OK– Synchronization input links OK – Debug functionalities OK

• Board temperature cycling test (to be done)– All devices T < 55 C OK

• Link tests later in presentation



Some remarks

• Difficulties in assembly of large BGA– Due to board thickness– 1704 Ball grid array 1 mm pitch, 45 x 45 mm– Extensive test during solder process with mechanical sample– Solder process under control OK

• Design flaw in PCB– Missing pull up at FPGA configuration signal (mounted at solder side

connector) OK

• No re-design needed



Link tests



Pile-Up test setup dataflow

• All boards (proto-types) present

for testing @Nikhef onlytests done

Hybridskaptoncables

repeaterboards

transitionboards

opticalboards

VELOctrl brdspecsmaster

ECS pvss, test software C

UTBpatterngen.

VEPROBboards

outputboard

L0DU@IP8

default dataflow

TTCvx

VMEcpu

AnalogTell1

digitaltell1

UTBdatabuffer

VMEcpu

TTCvi

Odin

1

2

3



Interconnection tests• Gigabit Optical connections

– Optical Tx Board to VEPROB• 2 optical ribbons (GOL to ORC

TLK2501)– 2 fibers out to TELL1 (ORC receivers)

• Xilinx rocket output -> HFBR5720 -> TLK2501

• Gigabit LVDS coaxial output– VEPROB to Output Board

• Xilinx Rocket output v2pro -> Xilinx Rocket input Xilinx Virtex4

• For test: Rocket-out to Rocket-in v2pro

MPO-SC cassete

SC-MPO cableMPO-MPO

0,6,9dB MPO-MPO

MPO-LC cable

SMA coax0,6,9dB MPO-MPO

1

2

3



Gigabit interconnection tests• BERR test on all links

– Link 1 : 12 output links • Optical Tx Board -> Orc card(Veprob)• 0 dB 48 h -> 0 bit errors • 6 dB att. 60h -> 0 bit errors• 9 dB att. 60h -> 0 bit errors• 15 dB att. many errors

– Link 2: 2 output links, VEPROB -> ORC on Tell1

• Xilinx Rocket I/O -> HFBR5720 -> Orc• 0 dB 60h -> 0 bit errors• 6 dB att. 48h -> 0 bit errors• 9 dB att. 60h -> 0 bit errors• 15 dB att. many errors

– Link 3: 1 Output link VEPROB -> Output Board (Cu link)

• Tested: Xilinx rocket Output -> Coax VEPROB Xilinx Rocket Input

• 0 dB 48h -> 0 bit errors• 3 dB att. 48h -> 0 bit errors• 6 dB att. 60h -> 0 bit errors

• All links conform specs

Link 3 σ=28 ps (electrical)

Link 1: σ=57 ps(electrical)



Conclusions

• Optical Tx Board– All functionality as expected, all tests (excl. thermal) OK– Optical Tx Board PCB needs minor changes, but low risk

• Ongoing– PCB production time schedule

• 2 weeks to new layout• PCB production 4 weeks• Assembly 4 weeks, after all components in house

• VEPROB– All functionality as expected, all tests (excl. thermal) OK– PCB layout OK , no re-design needed

• Pull up, FPGA configuration missing, mount on connector– PCB production time schedule

• 5 boards in house • All components in house except $$ FPGA • Assembly 4 weeks• VEPROB series assembly can be started when FPGA’s are delivered



Outlook

• Board production ready: 3 months• Test programme

– Individual board tests 1 month– System tests 1 month (maybe 2)



Spare Slides



Passive boards

• Kapton cables– In house

• Repeater boards: 3 different types– 1 type produced & tested– 2 other types layout done

• Transition boards: 2 types– inner silicon strips, proto board used

in test setup– outer silicon strips, layout done

6U vme

30x30 cm

20x75 cm



LatencyLHC Clocks

spec / sim measured

Hybrid 50ns 2

Cabling to optical station 18 m (5ns/m) 108 ns 5

VETO Optical Station

Synchronisation and demultiplexing 4

GOL 64 ns max 3

Optical Ribbon 52m (old:60m) (4.5ns/m) 260 ns 11 (4m ribbon)

Vertex processor board

Orx-card(tlk2501) 4 43

channel sync and vertex finder algoritm(xilinx FPGA) 875ns 35 (was 42)

MGT Rocket i/o transmitter + cable 4

Output board MGT Rocket i/o receiver 4

Mux 1

MGT Rocket i/o transmitter 3

Cable to L0DU 90ns 4

Total: 84(87)

06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 1 Vertex Processor Board...

Documents

Transcript of 06/12/2006 Wilco Vink / Martin van Beuzekom Leo Wiggers / Sander Mos 1 Vertex Processor Board...