Level-1 data driver card design review report, T. Alexopoulos et al.

59 Pages·2015·41.89 MB·English

Draftversion1.0
ATLAS NOTE
February14,2015
Level-1 Data Driver Card Design Review Report
1
T.Alexopoulosa,C.Bakalisa,P.Gkountoumisa,G.Iakovidisb,A.Koulourisa
2
aNationalTechnicalUniversityofAthens,Greece
3
bBrookhavenNationalLaboratory,USA
4
Abstract
5
InthisdocumentwedescribetheoveralldesignandspecificationsofthefinalLevel-1Data
6
Driver Card for the electronics review process of the New Small Wheel. Furthermore, the
7
roadmap to the final board through out the evaluation of several prototype cards will be
8
presented along with the various tests that have to be performed in order to validate the
9
proposeddesign. Finally,anupdatedtimescheduleandthemanpowerplanneededtocover
10
thedesignandproductionoftheLevel-1DataDriverCardswillbegiven.
11
©Copyright2015CERNforthebenefitoftheATLASCollaboration.
ReproductionofthisarticleorpartsofitisallowedasspecifiedintheCC-BY-3.0license.
February14,2015–12:37 DRAFT 1
Contents
12
1 Introduction 3
13
2 Overalldesignandspecifications 5
14
2.1 Architecture,3Dmodels,dimensions,components-connectors,toolingused . . . . . . 5
15
2.2 Boardarchitectureandmaterials(Layerstack,Differentialwidth/gap,Differentialimpedance,
16
FR4,etc.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
17
2.3 Powerdistributionandconsumption . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
18
2.4 LocationoftheL1DDCcardsondetectorandservicesrouting . . . . . . . . . . . . . . 13
19
2.4.1 Micromegascardplacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
20
2.4.2 sTGCcardplacement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
21
2.4.3 Micromegasservicesrouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
22
2.4.4 sTGCservicesrouting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
23
2.5 SpecificdesignforMMandsTGC,sharemostofthedesign . . . . . . . . . . . . . . . 16
24
3 Specificationofthefunctionstobeperformed 18
25
3.1 Connectivityandcompatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
26
3.1.1 InterfacewithFE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
27
3.1.2 InterfacewithADDC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
28
3.1.3 InterfacewithFELIX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
29
3.2 GBTxASICbankconnectivityandconfiguration . . . . . . . . . . . . . . . . . . . . . 21
30
4 Tests 21
31
4.1 Validationinradiationandmagneticfield . . . . . . . . . . . . . . . . . . . . . . . . . 23
32
4.2 Reliability-SEUdetection/correction . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
33
4.3 Debuggingtoolsandprograms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
34
4.4 ProceduresforthefinalL1DDCtests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
35
5 L1DDCPrototype1 23
36
5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
37
5.2 DesignchoicesandDimensions(Layerstack,Differentialwidth/gap,Differentialimpedance,
38
FR4etc.) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
39
5.3 PowerdistributionandPowerconsumption(Componentdescription,Rampupsequence,
40
VoltageLevels) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
41
5.4 Componentdescriptionandspecifications(FPGA,EthernetPHY,SFP) . . . . . . . . . . 33
42
5.5 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
43
5.5.1 Clockdomain(internalandexternalclocks) . . . . . . . . . . . . . . . . . . . . 38
44
5.5.2 ConnectivitywithFEs,ADDCandDAQ . . . . . . . . . . . . . . . . . . . . . 41
45
5.6 Firmware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
46
5.6.1 CommunicationProtocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
47
5.6.2 Systemcore-Userlogic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
48
5.7 Adapterboards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
49
5.8 Testsanddebugging(Toolsandapplicationsused) . . . . . . . . . . . . . . . . . . . . . 48
50
5.9 Issues-errorstobecorrected . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
51
5.10 TestswithVMMequippedfrontendprototypes . . . . . . . . . . . . . . . . . . . . . . 53
52
6 Prototype2&3androadmaptofinalboard 54
53
7 Timescheduleandmanpower 55
54
February14,2015–12:37 DRAFT 2
Appendices 57
55
A AdaptationoffirmwaretoArtix7FPGA 57
56
February14,2015–12:37 DRAFT 3
1 Introduction
57
HighspeedserialdatatransmissionhasbecomethenormindataacquisitionsystemsofHEPexperiments
58
followingthedevelopmentofcommercialhigh-speedserialprotocolssuchasPCIexpress,10GEthernet,
59
infiniband, etc. serializers and deserializers are the key components of such systems. Off the shelf
60
components(COTS)aretypicallynotsufficientlyradiationresistantand, inparticular, quitesusceptible
61
to single event upsets (SEU) caused by energetic (E > 20MeV) hadrons. For this reason the CERN
62
microelectronicsgrouphasbeendevelopingaradiationhardchipset,theGigaBitTransceiver(GBT)[1].
63
TheGBTchipsetconsistsofa4.8Gb/sserializer/deserializerASIC,aversatileopticaltransceiveralong
64
with a laser driver IC, a pin diode receiver IC and a Slow Controls Adaptor (SCA). This separation of
65
functions allows different combinations of transceivers, optical drivers, laser sources, and photodiodes
66
to be used according to the needs of a particular application. Event data, Timing, Trigger and Control
67
(TTC) signals, front-end configuration, and DCS data can be exchanged via a single fibre pair between
68
thefront-endandUSA15usingtheGBT.ItisassumedthattheUSA15endisimplementedinanFPGA
69
ratherthanaGBTASIC.
70
TheGBTASICcanbeviewedasmultiplexinganumberofseriallinks,calledE-links,with80,160,or
71
320Mb/sratesontoonefibrebytransferring2,4,or8bitsperE-linkviaan80-bitword(optionally116
72
bits) every 25ns, for an aggregate GBT bandwidth of 3.2Gb/s. One e-link, consisting of three pairs of
73
differential lines (6 wires) being the clock (Clk+ and Clk-), the Data In (Din+ and Din-) and Data Out
74
(Dout+andDout-). TheelectricallevelsfollowtheLVDSstandard.
75
The Slow Control Adapter (SCA) ASIC provides an interface to several protocols needed for the front-
76
endconfigurationandmonitoring. TheGBT-SCAimplementsdifferentbusesliketheJTAGandtheI2C
77
which allow the configuration of the ASICs while features an ADC for monitoring input signals. The
78
SCAfeaturesaspecificE-linkforthedatacommunicationwiththeGBTx. Bothmicromegas(MM)and
79
small Thin Gap Chambers (sTGC) place an SCA on each front-end board. Although this results in a
80
substantial number of SCAs, it is far more reliable, more compact and less expensive than connectors
81
andcablestoamorecentrallocation.
82
TheLevel-1DataDriverCard(L1DDC)cardwillserveasanintermediatestagebetweentheFEboards
83
and the Felix interface system. The two technologies of NSW sTGC and Micromegas (MM) use the
84
samebuildingblocksforthereadoutL1DDCcards.
85
The micromegas system of the NSW needs to provide muon segment candidates to the Sector Logic
86
(SL) within the Level-1 latency of 41 clock ticks of 40MHz. This corresponds to a latency of 1025ns.
87
OncethesegmentisconfirmedbytheSLthetrackingprimitiveshavetobereadoutandprocessed. The
88
micromegassystemconsistsof∼2.1Mchannels.
89
Thefront-endASICprovidingthetriggerandtrackingprimitivesistheVMM.Forthemicromegas,eight
90
VMMASICswillbeinstalledinafront-endboardcalledMMFE.Thiscreatesafront-endboardcapable
91
ofreading512micromegasreadoutstrips. TheMMFE,apartoftheVMMASIC,willfeatureaGBT-SCA
92
for the configuration, calibration and monitoring of the MMFE. Moreover a custom readout ASIC will
93
be integrated in the MMFE capable to gather, format and serially transmit the information of the eight
94
VMMintoasinglee-linkoutput. Aseparatee-linkwillbeusedfortheconfigurationpath. TheMMFEis
95
alsocapableofdrivingtheARTsignalstoaseparatetransmissionlineforeveryLHCbunchcrossingof
96
25ns. AllthecomponentsoftheMMFEareinstalledinaPCBof215×60mm2. TheARTsignalsofthe
97
eight MMFE boards are multiplexed into a trigger board called ART Data Driver Card (ADDC) which
98
is not described in this document. The readout data of eight MMFE, are multiplexed through e-links
99
intoadatadrivercardcalledLevel-1DataDriverCard(L1DDC).TheL1DDCfeaturesoneGBTx. Itis
100
capableofdrivingtheLevel-1datathroughtheGBTxintoafibertoanetworkinterfacecalledFELIX.It
101
is also capable to configure the ADDC and the MMFE through dedicated to configuration E-links. The
102
L1DDC is built in a PCB of the same dimensions like the ADDC. An overview representation of the
103
February14,2015–12:37 DRAFT 4
NSWelectronicscomplexisshowninFigure1. ThetotalnumberofL1DDCboardsis1024(512boards
104
forthemicromegasdetectorsand512boardsforthesTGCdetectors).
105
BE detector electronics (USA15)
Figure1: OverviewrepresentationoftheNSWmicromegaselectronicscomplexity. TheMMFEfrontend
cardsareconnectedtotheL1DDC(lightgreen)cardsviae-Links.
The front end electronics will be installed on the micromegas wedges radially along both sides. This
106
provides a way of equalizing the load on both sides of the detector and the cable routing. Figure 2
107
showsamultiplexingdiagramofthemicromegasfrontendelectronicsboards. TheL1DDCimplements
108
the configuration e-links uniform along the radial direction with a bit-rate of 80Mbps e-links while the
109
readout e-links, with a bitrate of 160Mbps for the four outer MMFE and 320Mbps for the inner four.
110
Thisisduetothehigherparticleratewhichrequiresahigherbandwithintheinnerradiusofthedetector.
111
2 Overall design and specifications
112
In thissection thefinal L1DDCboard will bedescribed. Thereis a detaileddescription ofthe architec-
113
ture of the board and the components that will be used, the estimation of power consumption and the
114
placement on the detector side. Also, it is mentioned the design choices (like layer stack, differential
115
impedance,differentialwidth/gap)andthematerialsthatwillbeusedinordertofabricatethefinalboard
116
(dielectricconstant,FR4).
117
2.1 Architecture,3Dmodels,dimensions,components-connectors,toolingused
118
Thereisalimitationoftheoverallavailablespacefortheplacementoftheboardtothedetector. Because
119
ofthisheightlimitationthefinalboardwasdecidedtohaveconnectorsonlyintheonesideoftheboard
120
and not in both sides. The other side will accommodate only the components that need to be cooled
121
through the cooling channel. The final board will be 200mm in length, 50mm in width and about
122
18.4mm in height. A representation of the placement of the L1DDC board on the detector is shown in
123
Figure 3. In this figure in the upper margin of the detector is placed the L1DDC board. In the top side
124
of the board the connectors are visible (the power connector, the miniSAS connectors and the VTRx
125
transceiver) and on the bottom side the GBTx ASIC along with the DC-DC converter, the foam that
126
February14,2015–12:37 DRAFT 5
MMFE e
r br
fi
e
br
fi
80 Mbps e-Link
160 Mbps e-Link
320 Mbps e-Link
160 Mbps twin-ax
Figure2: Frontendelectronicsmultiplexingdiagram. Thefrontend(MMFE)electronicscardsarecon-
nected via e-Links to L1DDC and ADDC cards. In addition a communication is established between
L1DDCandADDCcards. TheoutputofthereadoutcardsisconnectedtotheFELIXdistubutionsystem
viafiberopticscables.
keepsintouchtheIntegratedCircuitswiththecoolingchannelandfinallythegroundpinsthatholdthe
127
boardonthedetector. Onthebottomside,ontopofthehotwateroutputisplacedthemicromegasFront
128
Endboard(MMFE8).
129
ThefinalL1DDCboardwillaccommodatethefollowingbasiccomponents: theGBTxASIC,theVTRx
130
optical transceiver, the DC-DC converter and the connectors (for power and data). The GBTX is a
131
radiation tolerant chip that can be used to implement multi-purpose high speed (3.2 − 4.48Gb/s user
132
bandwidth) bidirectional optical links for high-energy physics experiments, see Figure 4 where the top
133
andbottomviewsareshown.
134
ItwillprovidethreedistinctdatapathsforTimingandTriggerControl(TTC),DataAcquisition(DAQ)
135
and Slow Control (SC) information to the front end electronics and the ADDC boards as shown in Fig-
136
ure 5. The GBTX can be electrically interfaced with the on detector electronics using different topolo-
137
gies. The simplest one consists of interconnecting the GBTX and a front-end device through a parallel
138
lane while the most sophisticated allows the GBTX to interface simultaneously with up to 40 front-end
139
devices via duplex local Electrical serial links (E-links). The use of one single parallel lane, the use
140
of multiple parallel lanes or individual serial connections are valid subsets of the E-link programmable
141
features. EachE-linknormallyconsistsofthreesignallines(differentialpairs):
142
• DifferentialClockline(dClk+/dClk-): ClockdrivenbyGBTXtofront-endmodule.
143
• DifferentialDownlinkdataoutput(dOut+/dOut-): DatalinefromGBTXtothefront-endmodule.
144
February14,2015–12:37 DRAFT 6
Figure 3: On detector placement of the L1DDC board; top board on the side of MM wedge is shown.
TheboardbetweenthetworeadoutpanelsofthemicromegaswedgeistheFrontEndMMFE-8. Also,in
thisfigure,thecrosssectionofthevariouscables(High&LowvoltageandFibercables)runningalong
thecoolingcutoutsisshown.
(a)TopsideofGBTx (b)BottomsideofGBTx
Figure4: TheGBTxASIC
Figure5: OverallfunctionalityoftheGBTXASIC
• DifferentialUplinkdatainput(dIn+/dIn-): Datalinefromfront-endmoduletoGBTX [6]
145
Figure 6 represents the general interconnection topology between the GBTX chip and the Front End
146
electronics using E-links. For each group the E-Link clock signals (dClk+/dClk-) can be programmed
147
February14,2015–12:37 DRAFT 7
Figure6: ElinksbetweenthefrontendboardsandtheGBTx.
independently to get any of the following frequencies: 40MHz, 80MHz, 160MHz or 320MHz. If, for
148
example,theE-Linkclocksareprogrammedtobe40MHzforthe2×datarate(80Mb/s)thelinksand
149
itsassociatedclocksbasicallyrunasaDoubleDataRate(DDR)connectionwiththeclockhavingrising
150
andfallingtransitionsoccurringinthemiddleofthebitperiodasshowninFigure7 [6].
Figure7: E-Linkclocksanddataoutputs.
February14,2015–12:37 DRAFT 8
The connectivity of the L1DDC board with the on detector electronics (Front Ends and ADDC boards)
151
willbethroughtheMolex36pminiSASconnectors(PartNo75783-0132)asshowninFigure8aand3M
152
miniSAScables(PartNo8F36-AAA105)asshowninFigure8b [9]. Thesmallsizeoftheconnectors
153
(8.47mmheightand17.8mmwidth)isshowninFigure9aandTable9balongwiththehighlyroutable
154
cables is the reason for choosing them for the update of the NSW. The miniSAS 36 position cable
155
can accommodate up to 4 receiving and 4 transmitting differential pairs, as long as 4 receiving and 4
156
transmitting pins are dedicated to sidebands. The L1DDC board will communicate with the Read Out
157
CompanionASIC(ROC)ofthefront-endboardswhichistheassignedchiptocollectthedatafromthe
158
VMMs and transmit them to the L1DDC. The front-ends are equipped with a second ASIC (the Slow
159
ControlAdapterSCA)whichisresponsibleforcontrollingandmonitoring. BothASICs(ROCandSCA)
160
mustcommunicatewiththeL1DDCthroughoneE-linkeach.
161
(a)Molex36positionMiniSASconnector. (b)3Mmini36positionminiSAScable.
Figure8: MiniSASconnectorandcable
(b)CircuitSize
(a)MiniSASdimensions.
Figure9: MiniSASconnectordimensionsandspecifications
ThegeneralarchitectureoftheGBTXASICanditsmainexternalconnectionsaredisplayedinFigure10.
162
TheGBTXconnectstotheGBLDlaserdriverASICandtotheGBTIAtrans-impedanceamplifierASIC.
163
TheClockandDataRecovery(CDR)circuitreceiveshighspeedserialdatafromtheGBTIA.Itrecovers
164
andgeneratesanappropriatehighspeedclocktocorrectlysampletheincomingdatastream. Theserial
165
data is then de-serialized (that is converted to parallel form) and then DECoded, with appropriate error
166
corrections,andfinallyDeSCRambled(DSCR).
167
In the transmitter part the data to be transmitted is SCRambled (SCR), to obtain DC balance, and then
168
encodedwithaForwardErrorCorrection(FEC)code,beforebeingserializedandsendtotheGBLDlaser
169
driver. TheconfigurationoftheGBLDcanbeperformedviaasimplifiedI2C-Lightconnectionfromthe
170
GBTX. A clock manager circuit takes care of generating and manage the different high speed and low
171
speed clocks, needed in the different parts of the GBTX. A programmable phase shifter is available to
172
generate 8 external user clocks with programmable frequency and phase. An external clock, or an on-
173
February14,2015–12:37 DRAFT 9
Figure 10: GBTx full functionality and connectivity with the Front End boards & the rest of the DAQ
system.
package crystal oscillator, is used during start-up as a locking aid for the CDR circuit and as a clock
174
referencefortheASICwatchdogcircuit. Generalcontrolandmonitoringlogictakescareofcontrolling
175
the different parts of the chip according to the operation mode selected and the ASIC configuration
176
information. Initialconfigurationinformationistakenfromtheonchipe-Fusesthatcanthenbemodified
177
viatheopticallinkitselforviaanI2Cslaveinterface. AJTAGinterfaceisavailableforboundaryscan.
178
Connections to the front-end modules or ASICs are made through sets of local Electrical Links (E-
179
Links). Depending on the data rate and transmission media used, E-Links allow connections that can
180
extenduptoafewmeters. E-LinksuseLow-VoltageDifferentialSignalling,withsignalamplitudesthat
181
are programmable to suit different requirements in terms of transmission distances, bit rate and power
182
consumption. TheE-LinksaredrivenbyaseriesofePortsontheGBTXandareassociatedwithE-link
183
ports in the front-end modules. The number of active E-Links and their data rate are programmable.
184
Parallelfront-endinterfaceswithdifferentbitwidthsarevalidsub-setsoftheflexibleE-Links[6].
185
There are two options for miniSAS cables, one with sidebands as shown in Table 11b and one without
186
sidebands as shown in Table 11c. Also 3M fabricates custom miniSAS cables in any length in case it
187
needed. InourschemefortheconnectiontotheFrontEndthesamelengthofcableswillbeused. This
188
willleadtothesameclockanddatadelaypropagation.
189
ThereceiverportsoftheGBTXASICusestheScalableLow-VoltageSignallingfor400mV(SLVS-400)
190
andtheLVDSstandard. TheLVDSsignalsusesadifferentialvoltageswingof400mVcentredon1.2V.
191
TheSLVSstandardisalsodifferentialbutwithareducedvoltageswingof200mV,centredon0.2Vas
192
showninFigure12.
193
The VTRx optical transceiver consists of two ASICs, the GBTIA and the GBLD. The GTIA (GigaBit
194
TransImpendance Amplifier) has a bit rate of 5Gb/s (min) and a total jitter smaller than 40ps P-P. Its
195
supplyvoltageis2.5Vanditspowerconsumptionis250mW [8]. TheGBLD(GigaBitLaserDiode)is
196
alsoaradiationtolerantASICfabricatedin130nm. Ithasalsoabitrateof5Gb/s(min),supplyvoltage
197