Table Of Content

Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology [J.Res.Natl.Inst.Stand.Technol.106,279–292(2001)] Statistics and Measurements Volume 106 Number 1 January–February 2001 M. Carroll Croarkin Formorethan50years,theStatisticalEn- surveyofseveralthematicareas.Theac- gineeringDivision(SED)hasbeenin- companyingexamplesillustratetheim- National Institute of Standards and strumentalinthesuccessofabroadspec- portanceofSEDinthehistoryofstatistics, Technology, trumofmetrologyprojectsatNBS/NIST. measurementsandstandards:calibration Gaithersburg, MD 20899-8980 Thispaperhighlightsfundamentalcontribu- andmeasurementassurance,interlaboratory tionsofNBS/NISTstatisticianstostatis- tests,developmentofmeasurementmeth- [email protected] ticsandtomeasurementscienceandtech- ods,StandardReferenceMaterials,statisti- nology.Publishedmethodsdevelopedby calcomputing,anddisseminationof SEDstaff,especiallyduringtheearlyyears, measurementtechnology.Abrieflookfor- endureascornerstonesofstatisticsnot wardsketchestheexpandingopportunity onlyinmetrologyandstandardsapplica- anddemandforSEDstatisticianscreated tions,butasdata-analyticresourcesused bycurrenttrendsinresearchanddevel- acrossalldisciplines.Thehistoryofstatis- opmentatNIST. ticsatNBS/NISTbeganwiththeforma- tionofwhatisnowtheSED.Examples fromthefirstfivedecadesoftheSEDil- Keywords: calibrationandmeasure- lustratethecriticalroleofthedivisionin mentassurance;erroranalysisanduncer- thesuccessfulresolutionofafewofthe tainty;designofexperiments;historyof highlyvisible,andsometimescontroversial, NBS;interlaboratorytesting;measurement statisticalstudiesofnationalimportance. methods;standardreferencematerials; Areviewofthehistoryofmajorearlypub- statisticalcomputing;uncertaintyanalysis. licationsofthedivisiononstatistical methods,designofexperiments,anderror analysisanduncertaintyisfollowedbya Availableonline: http://www.nist.gov/jres 1. Introduction Formorethan50years,StatisticalEngineeringDivi- expandedandnewmethodshavebeendevelopedtoad- sion(SED)staffhaveplayedacriticalroleinthesuccess dressrecentchallengesandtakeadvantageofthestatis- of a broad spectrum of metrology projects at NBS/ tical literature and the tremendous surge in statistical NIST.Duringthistime,statisticsatNBS/NISThaspro- computing capability. gressedwiththeconstantgoalofimprovingandcharac- SED research contributions cover: quantification of terizing measurement methods. Methods and uncertaintyinmeasurements,statisticaldesignofexper- publications which were developed early in the life of imentalinvestigations,montecarlomodeling,parameter thedivisionarestillcornerstonesforstatisticalanalyses estimation,stochasticmodeling,exploratorydataanaly- and are applied across all disciplines and metrologies. sisandempiricalmodeling,modelvalidation,computer Overtheyears,existingmethodshavebeenrefinedand intensivestatisticalmethods,reliabilityanalysis,statisti- 279 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology calsignalprocessing,imageanalysis,timeseriesanaly- sis, hypothesis testing, and quality control. Statisticiansparticipateintheplanningofexperimen- tal studies and conduct rigorous uncertainty analysis of results and develop theoretical models to augment ex- perimentalworkdonebyNISTcollaborators.Examples ofsuchworkincludeMonteCarlosimulationofphysi- calprocesses,suchasneutronscattering,andstochastic differentialmodelingofaerosolparticlespectrometers. Typically, SED staff develop long term relationships with collaborators in the other NIST laboratories and develop intimate knowledge of the scientific field in which they work. Here we highlight areas where SED contributes to metrology work at NIST with examples fromrecentcollaborationsalongwithanhistoricalper- spective for viewing statistical contributions to metrology. 2. History 2.1 Early Days Churchill Eisenhart (see Fig. 1) came to NBS from theUniversityofWisconsinin1946whenEdwardCon- don, Director of NBS, resolved to establish a statistical consulting group to “substitute sound mathematical analysis for costly experimentation” [26]. As the first Chief of the Statistical Engineering Laboratory (SEL), Fig.1. ChurchillEisenhart. heshapedthedirectionthatstatisticswouldtakeatNBS for many years and personally laid the foundation for differentiateeffectscausedbyAD-X2fromeffectsdue error analysis related to measurement science. to the charging line. They also insisted on a formal In its early days SEL, in its work with scientists at randomization scheme for selecting the batteries for NBS,wasdrawnintoseveralinterestingactivitiesasthe treatment in order to avoid conflicts in the analysis. Secretary of Commerce encouraged NBS to become Afterthisaccomplishment,thereensuedabriefmoment involved in outside activities. The most important of of panic when they realized that a design was also these was the controversy over battery additive AD-X2 neededforavisualtestwheretheelectricalplateswould [13].TheNBSDirector,A.V.Astin,hadbeenpressured be disassembled and 45 paired comparisons would be byvarioussenatorsandtheadditiveproducertotestthe made of treated and untreated batteries. Fortunately, additive for its ability to improve battery performance. Joseph Cameron found a suitable incomplete block de- The statisticians, under severe time constraints, were sign, thus avoiding the risk of having to construct such responsibleforrecommendingexperimentaldesignsfor a design in the urgency of the moment [26]. testing the additive. The resulting analysis by SEL of this experiment, Therewere32batteriesavailableforthetest,andthe conducted by the Electrochemistry Section, confirmed manufacturerwantedtoputall16batteriesthatwereto that the additive had no significant positive effect on betreatedwithAD-X2ononechargingline.Thestatis- batteries,butinwhatwastoquicklybecomeaninterest- ticians disagreed with the manufacturer and Jack You- ingsidelightofhistory,theAssistantSecretaryofCom- den(seeFig.2)proposedadesignwiththe32batteries merce for Domestic Affairs announced that Astin had groupedinpairsfortestingonthreecharginglines.On not considered the “play of the marketplace” in his lines1and2,bothbatteriesofapairweretobetreated judgmentandrelievedhimasDirectorofNBS.Eventu- with AD-X2 or both were to be untreated. On line 3, ally,theNationalAcademyofScienceswascalledinto therewasonetreatedandoneuntreatedbatteryineach review NBS’s work, which was labeled first rate, and pair. The statisticians were satisfied that this design for Astin was reinstated [24]. testingtheelectricalperformanceofthebatteriescould 280 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology wasestablishedtoaddressthenumberonecauseoffire deaths in this country, namely, cigarette ignition of up- holstered furniture and bedding. Extensive testing by federal, private, and industrial laboratories of lit cigarettesonfurnituremock-upsresultedinheateddis- cussionsasmightbeexpectedwhenanindustryisfac- ing potential regulation. NIST was asked to intervene, and Eberhardt led in the evaluation of screening tests conductedbytheFireScienceDivisionoftheBuilding andFireResearchLaboratory(BFRL).Thisworkledto thedevelopmentoftwotestmethodsandacarefully-designed interlaboratory evaluation of the test methods. Because standard statistical procedures for analyzing interlaboratory studies do not apply to the analysis of proportions,amethodologybasedonasimplemodelfor “extra-binomial variation” [25] was developed specifi- cally for analyzing this data. The cigarette industry re- sponded with a new study that seemed to imply that fabrics selected by NIST for the study were atypical. Careful re-analysis of this data by Eberhardt demon- stratedtoaTechnicalAdvisoryGroupmadeupofrepre- sentatives from government, industry, consumer advo- cates,andtestinglaboratoriestheflawsintheindustry’s analysis.SEDcollaborationwithBFRLoncigaretteig- nitioncontinuesastheindustrytriestorespondtoCon- gressional mandates for less fire-prone cigarettes and interlaboratory tests are mounted to assess progress in this direction. Fig.2. JackYouden. 2.3 Publications on Statistical Methods 2.2 High Visibility Studies One of the first large-scale contributions of the SEL NISTcontinuestorelyontheStatisticalEngineering to measurement science at NBS was the publication of Division for advice on experimental design, analysis, NBSHandbook91[23]whichhasguidedresearchersat andinterpretationwhenevertheinstitutioniscalledupon NIST for four decades in the planning and analysis of asarbiterintechnicalconflictsofnationalimportance. scientific experiments. In 1954, Eisenhart was ap- In1970,whentheSelectiveServiceSystemwasroundly proached by the Army’s Office of Ordnance Research andproperlycriticizedforallowingbirthdatetobiasthe andaskedtoproduceaManualofExperimentalStatis- draft lottery, Joan Rosenblatt, Chief of SED, led the ticsforOrdnanceEngineersasaguideformilitaryand NBS team that revised the procedures to ensure a fair civilian personnel with responsibility for planning and and random lottery [1]. In the 1980s, when Congress analysisoftestsofArmyequipment.Eisenhartassigned waslobbiedbytheaudiorecordingindustryforprotec- primaryauthorshiptoMaryNatrella,whohadcometo tivelegislationtorequireDigitalAudioTapesystemsto SELfromtheU.S.Navy’sBureauofShipswithexten- be fitted with a copy prevention decoder, NIST was sive experience as a sampling inspection expert. The asked to test the industry’s claim that a “notch” built material was first printed for limited distribution as a into the recorded materials at high frequencies would seriesoffiveU.S.ArmyOrdnancePamphletsaspartof notalterthequalityoftherecordings.Aseriesofdou- the AMC Engineering Design Handbook series. ble-blind listening tests, designed by Keith Eberhardt, AlthoughNatrellawasprincipalauthor,thematerial, uncovered significant, although subtle, differences whichwasseveralyearsinpreparation,wastheresultof amongsubjects’abilitiestodetectanotchandresulted the combined experience and knowledge of the entire in the legislation being denied [9]. SED staff at the time. It proved to be of great benefit Inthe1990s,theFire-SafeCigaretteActwasenacted because of the clear elucidation of difficult statistical to determine the practicability of developing a perfor- concepts accompanied by worked examples. mance standard for less fire-prone cigarettes. The act 281 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology In1963,thematerialswerepublishedasNBSHand- 2.4 Design of Experiments book 91 and offered for sale to the general public. The StatisticalresearchintheStatisticalEngineeringLab- 23chapterheadingslistedinTable1(abbreviated)indi- oratoryinthe1950s,ledbyWilliamConnorandMarvin cate the range of statistical methodologies that were Zelen,focusedonthedevelopmentofexperimentalde- employedintheNBSlaboratoriesatthetime.AllenV. signswhichwerepublishedaspartoftheNBSApplied Astin, Director of NBS at the time, says in the preface Mathematics Series (AMS). AMS 48, AMS 54, and that, “although originally developed with the needs of AMS 49 are devoted to factorial designs with factors the Army in mind, it promises to be equally useful to restrictedto2and3levels;AMS62isdevotedtocyclic othergroupsconcernedwithresearchanddevelopment, designs; and AMS 63 is devoted to partially balanced both within and outside the Government.” Its strength block designs. The papers on factorial designs contain came from the clarity of exposition, which was a hall- fractional designs or subsets of complete factorials mark of Natrella’s writing style, and from its detailed whichare“optimized”forestimatingindividualfactors guidance,completewithnumericalexamples,onstatis- andinteractionsamongfactors.Thisclassofdesignsis tical computations that accompanied each test and pro- probably the most important class for assessing the ef- cedure.In1983,itwasreprintedforcommercialsaleby fect of various factors on measurement processes. Wiley Interscience as part of its Selected Government ThedesignswerecreatedbySELstaffwiththeassis- Publications series. In 1985, the American Society for tanceofmanydedicatedsummerstudents;designsthat Metals(ASM)publishedacondensationoffourchapters were published before 1960 were created without elec- onplanningandanalysisofcomparativeexperimentsas troniccomputers.Thepublicationswereofferedforsale part of the Statistics Section of Volume 8 of the 9th by the Government Printing Office for as little as 40 edition of the ASM Handbook. Over the years, it has cents per copy. been NIST’s second-best selling publication. Factorial designs are such an important class of ex- A few years later, a compendium of papers by the perimentaldesignandhavefoundsomanyapplications StatisticalEngineeringLaboratory,waspublishedinthe at NBS/NIST that it is impossible to give a representa- NBS series on Precision Measurement and Calibration tive accounting of their usage. A study in the Electro- [19].Thisbookcontainsmanyhistoricalpapersinclud- magnetic Technology Division of the Electronics and ing some of the papers referenced in this article and ElectricalEngineeringLaboratoryinBoulderillustrates some that could not be referenced because of space the use of factorial designs for optimizing a measure- considerations. The primary focus is error analysis of ment process [6]. This particular study examined eddy calibration and interlaboratory studies with the materi- currentprobesensitivityasafunctionofcoilconstruc- als organized under the following topics: tionparameters.Eddycurrentscanbeusedfordetecting cracks in metal, such as airplane wings, and are mea- (cid:127) The Measurement Process, Precision, Systematic sured by changes in the probe’s electromagnetic field. Error, and Accuracy Theexperimentalarrangementwasafractionalfactorial (cid:127) Design of Experiments in Calibration with each factor at two levels. The primary goal of the (cid:127) Interlaboratory Tests study was to identify probe construction factors and (cid:127) Functional Relationships interactions with the largest effect on detector sensitiv- (cid:127) Statistical Treatment of Measurement Data ityastheprobeismovedfromanunflawedregionofthe (cid:127) Miscellaneous Topics metal to a flawed region of the metal. The analysis of Table1. TableofContentsofHandbook91:ExperimentalStatisticsbyMaryNatrella Ch.1. Somebasicstatisticalconcepts Ch.12. Factorialexperiments Ch.2. Characterizingmeasuredperformance Ch.13. Randomizedblocks,Latinsquares Ch.3. Comparingwithrespecttotheaverage Ch.14. Experimentstodetermineoptimumconditions Ch.4. Comparingwithrespecttovariability Ch.15. Someshortcuttestsforsmallsamples Ch.5. Characterizinglinearrelationships Ch.16. Testswhichareindependentofdistribution Ch.6. Polynomialandmultivariablerelationships Ch.17. Thetreatmentofoutliers Ch.7. Characterizingqualitativeperformance Ch.18. Controlchartsinexperimentalwork Ch.8. Comparingwithrespecttoatwofold Ch.19. Extreme-valuedata classification Ch.20. Theuseoftransformations Ch.9. Comparisonwithrespecttoseveralcategories Ch.21. Confidenceintervalsandtestsofsignificance Ch.10. Sensitivitytesting Ch.22. Notesonstatisticalcomputations Ch.11. Considerationsinplanningexperiments Ch.23. Expressionofuncertaintiesoffinalresults 282 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology sensitivitytookadvantageofanoptimizationschemefor techniqueincorporatesinterlaboratorytestdataonresis- pinpointingexactsettings(overallfactors)formaximiz- tance, as well as a detailed analysis of the nonlinear ing sensitivity and produced an empirical equation for relationship between resistance and resistivity, as esti- predicting sensitivity based on the levels of the various mated from extensive historical data. factors. Research on the subject of uncertainty is still evolv- ing,andrecentworktakesadvantageofmodernstatisti- caltechniquessuchasBayesianmethodswhichprovide 2.5 Error Analysis and Uncertainty aunifiedapproachtocombiningrelevantinformationin Uncertainty analysis is one of the primary responsi- the measurement experiment [20]. bilitiesoftheNISTstatisticianwhoisinvolvedinreport- ing measurement results. Uncertainty quantifies the quality of a measurement result. In the early 1950s, 3. Calibration and Measurement precisionandaccuracywerecommonlyusedforcharac- Assurance terizingthequalityofmeasurementprocessesalthough therewaslittlecommonagreementorunderstandingas Calibrationistheassignmentofavaluetoatestitem to their meaning and consequences. Eisenhart was or an instrument based on measurements made on the drawntothisissueasitrelatedtocalibrations,whichhe testitemandonareferencestandardwithknownvalue. calledrefinedmeasurementmethods.AsChiefofSEL, Calibrations are of two types: (1) single-point calibra- hesetouttoputtheconceptsofaccuracyandprecision tionsuchasassignmentofamasstoanunknownweight on a solid statistical basis. and(2)calibrationoveraregimesuchasacalibrationof Inapaperthatwastobecomethefoundationforerror alinewidthstandardfrom0.1(cid:1)mto10(cid:1)m.Experimen- analysis at NBS [10], Eisenhart synthesized his own talconfigurations,calledcalibrationdesigns,forsingle- workandthewritingsofstatisticaltheoristsandpracti- pointcalibrationsspecifymeasurementstobemadeon tioners, Walter Shewhart, Edwards Deming, Raymond testitemsandreferencestandards.Designsofthistype Birge,andR.B.Murphy,intoconceptsofqualitycon- arethefoundationforartifactcalibrationsatNIST.The trol that could be applied to measurement processes. solutionstothesedesignsarebasedonrestrainedleast- Three basic concepts in the paper have been embraced squares techniques [34] where the known value of the and practiced by metrologists at NBS ever since: (1) a reference standard(s) is the restraint on the system of measurement process requires statistical control; (2) equations. statisticalcontrolinthemetrologycontextimpliescon- The Statistical Engineering Division has created a trol of both reproducibility and repeatability; and (3) a largeportfolioofdesignsforNISTcalibrationlaborato- measurement result requires an associated statement of riesandaddsnewdesignstothiscollectiontorespondto uncertaintywhichincludesanypossiblesourceofbias. specific situations, as needed, or to take advantage of Hispaperwasfollowedbyotherswhichlaidthefounda- advancesininstrumentationandmetrology.Recently,an tion for future developments in uncertainty analysis at automated balance that was introduced into the NIST NBS.ParticularlynoteworthyisashortpaperbyH.H. masslaboratoryrequirednewdesignstotakeadvantage Ku[18]onpropagationoferrorwhichiseasilythemost of the high precision of the balance and also deal with enlightening paper ever written on the subject. the limitations that it imposed on the experimental Thestatisticaldeterminationofuncertaintyinmetrol- setup. ogyisoftencomplex,requiringcarefulconsiderationof The contributions of statisticians at NIST to calibra- the magnitudes of multiple sources of variability. A tion designs date from the late 1950s when Joseph measurementmaydependonthesesourcesinanonlin- Cameron seized on advances in experiment design and earway.Evaluatingtheindividualcomponentsofuncer- electronic computing to introduce new calibration de- taintycanrequirethecombinationofmultiplesourcesof signsintoNBSlaboratories.Theearliestdesignsofthis data, taken on various quantities upon which the pri- typewerecreatedforintercomparingmassstandardsor mary measurement depends, both at NIST and from weights, and were referred to as “weighing designs”. multipleoutsidelaboratories.Oneexamplewhichillus- CameronandCharlesReevecreateddesignsforabroad trates these points is an SED collaboration with the range of dimensional and electrical quantities that in- Semiconductor Division of the Electronics and Electri- clude:thelengthofagageblock,roundnessofasphere, calEngineeringLaboratory.Thisworkinvolvedthein- massofaweight,degreeofanangleblock,voltageofa directmeasurementofthegeometryofthinpure-copper standard cell, resistance of a one ohm resistor, and the films, using an approach which exploits the relation- like, which are the basis for calibrations at NIST and ships among resistance, resistivity, and conductor ge- throughout the U.S. metrology community. The unique ometry [29]. The uncertainty analysis for the proposed aspect of the designs created by the statisticians is that 283 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology they each have provision for a check standard to be the reader is advised to go to http://www.itl.nist.gov/ “calibrated” with the test artifacts. div898/pubs/slist.html for specific publications and Thecheckstandarddatabaseisthebasisforapplying to http://www.nist.gov/stat.handbook/mpc/section3/ statisticalcontroltheorytomeasurementprocesses,and mpc34.htm for a catalog of calibration designs for thestatisticiansworkedtoimplementthesestrategiesin weights, standard cells for voltage, resistors, gage the calibration laboratories of NBS. They also merged blocks,angleblocks,roundnessstandards,andhumidity the check standard concept and quality control proce- standards. dures to form a cohesive practice, known as measurement assurance, as a means of tying measurement results to a specified reference and quantifying uncer- 4. Interlaboratory Tests taintyrelativetothereferencebase.Thefirstdocumen- tation of a measurement assurance program in a NBS Oneofthemostwidelyusedtoolsforcharacterizing calibration laboratory appears to be a tutorial on mass andvalidatingmeasurementmethodsistheinterlabora- calibrations [27]. Measurement assurance programs tory test where test results are gathered on the same or now abound in metrology areas as diverse as dimen- equivalent materials by several qualified laboratories. sional measurements and semiconductor devices, and Statisticians are involved in the design and analysis of statisticalcontrolprocedures,basedoncheckstandards, interlaboratory tests over a huge range of disciplines at arethebasisforcontrollingtheoutputofNISTcalibra- NIST.Statisticiansaremembersofteamsthatdesignthe tion processes. initialinterlaboratoryexperiments,providingbothanal- Of equal importance, measurements on check stan- ysis, interpretations and recommendations for further dards form the basis for uncertainty determinations in intercomparisons. The analysis tools depend not only manyareasofmetrology.Incollaborationwithscientists upontheproblemathandbutalsoonthepurposeofthe inthecalibrationlaboratories,statisticiansdeveloperror interlaboratorytestandarenotlimitedtospecificstatis- models,forexplainingsourcesofvariabilityinthemea- tical techniques, a popular misconception. surement process that are applicable to measurements Interlaboratory tests sponsored by NIST are often onthecheckstandards.Thisiscriticaltotheassessment undertakenfortheexpresspurposeofproducingacon- of uncertainty because measurements on check stan- sensus value. In such a case, the goal drives both the dardsaretheonlyrecurringmeasurementsinacalibra- design of the experiment and the analysis which must tionsetting;thustheyprovidetheonlydataforestimat- frequently deal with the problem of outlying laborato- inglong-termcomponentsofuncertaintywhichcanbe ries.Asinallcollaborations,buttoagreaterdegreefor related to the uncertainties of values assigned to test important international comparisons, the statistician items [7]. must not only recommend and implement statistical Asmentionedpreviously,NISTalsoprovidescalibra- methods that are appropriate to the task but must also tions of quantities, such as force, where an instrument, win the confidence of the participants. The solution is such as a force sensor, is calibrated over a region of neverhisorherchoicealone,andmuchcareistakento interest. The resulting function is a calibration curve ensure that all parties understand and concur with the which defines the relationship between the surrogate method of analysis. A study to determine a calibration measurement and its reference. In general, neither the factor for relating infrared absorption measurements to calibration curve nor its functional form is known and the interstitial oxygen content of silicon is an example. mustbeestimatedfromexperimentaldata.Strategiesfor NISTstatisticianswereresponsibleforplanningtheex- estimatingthecalibrationcurvesundervariousscenarios perimentsandestimatingthefinalconversionfactorand arecontinuallyexploredbySEDstatisticians.Thediffi- associateduncertaintyforawiderangeofoxygencon- cult statistical task of computing the uncertainty of the tents from round robins of both infrared and absolute “calibrated value” from the inverse of the calibration measurements [2]. This was an important study for the curve is also the domain of the SED. Eisenhart solved world-wide semiconductor industry as many measure- theproblemoftheuncertaintyofthecalibratedvaluefor mentsarenowslavedtothecalibrationfactor.Problems asingleapplicationofalinearcalibrationcurve[44]in that had to be dealt with included non-equivalence 1935. However, the general solution for multiple appli- among samples that were circulated to the participants cationsofacalibrationcurvehasbeenanopenproblem and suspect results from some absolute measurements. inthemetrologycommunityforyears;asolutionusing In1990,theInternationalTemperatureScale(ITS-90) atoleranceintervalapproach[22]istheresultofexten- replaced the 1968 International Practical Temperature sive experience with NIST calibration activities. Scale(IPTS-68).Becauseofadiscontinuityinthe1968 Becausethelistofpapersoncalibrationandmeasure- scale that led to anomalies in temperature differences ment assurance is too extensive for this publication, betweenthetwoscalesintherange630(cid:2)Cto1064(cid:2)C, 284 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology the Consultative Committee on Thermometry of the tificates.SEDstaffarecontributingtokeycomparisons International Committee for Weights and Measures, formeasurementsoftemperature,thermalconductivity, throughitsWorkingGroup2,organizedacollaborative differential and absolute pressure, humidity, vibration effort among national metrology institutes (NMIs) to andacceleration,andbasicelectricalquantities,includ- generate new experimental data for type S thermocou- ingcapacitance,sound,ultrasonicpower,andlinescale. plesinthatrange.TheNISTstatisticianswereresponsi- Key comparisons serve as the technical basis for ble for creating new reference functions and inverse judging measurements around the world and must, functions for type S thermocouples [3, 4]. These func- therefore, accurately reflect the true relationships be- tions are now the basis for all temperature measure- tweenmeasurementsystemsmaintainedbyNMIs.SED mentswithintherangeofthesethermocouples.Tomit- statisticiansprovideguidanceoncomparisondesignsto igate the effect of outlying laboratories, the reference ensurethatdatacollectionwillbeaseffectiveaspossi- equation and associated uncertainties were computed bleforquantifyingbothdifferencesanduncertaintyand using iteratively reweighted least squares regression. implement analyses which account for covariances in Improvementsinthedesignandanalysisofinterlabo- the measurements and ensure that uncertainties have a ratory tests in SED began in the 1960s when W. J. specified confidence level. Youden sought to shed light on errors in measurement processesthroughexperimentaldesign.Inhisworkwith chemists and ASTM committees, Youden left a huge 5. Development of Measurement Methods body of literature on the subject. He approached interlaboratory testing as a means of uncovering biases in Development of new measurement methodology is measurement processes, and the so-called Youden plot probablythemostcriticalelementintheNISTmission. [33] has become an accepted design and analysis tech- SEDstatisticianscontributetotheseeffortsviacollabo- niquethroughouttheworldforcomparingprecisionand rative research efforts which typically proceed in sev- bias among laboratories. Work in graphical methods, eral stages. Initially, the problem is studied for proper whichbeganwiththeYoudenplot,continuestoday,no- understanding, and statistical issues are identified and tablyinrecentworkofNISTchemistDavidDuewer[8]. communicated to the research team. An experiment is Likelihood and non-parametric methods were pio- designed and statistical methods are applied to the re- neered by John Mandel, culminating in a book on the sulting data. New statistical methods, or modifications analysisoftwowaytables[21].Mandel,althoughnota ofexistingmethods,areoftenrequired.Finally,statisti- staff member of the SEL, spent his career as a statisticians participate in the preparation of NIST written cian working within the chemical community of NBS records or archival journal publications. Some collabo- andwiththeASTMcommunitytodevelopmethodsfor rations are one-time associations; others proceed itera- quantifying within-laboratory and between-laboratory tivelyoverseveralyears,withtheresultsofoneproject precision. His methodology, originally applied to the providing the foundation for the next investigation. chemicalandpaperindustries,hasbeencodifiedinna- A collaboration on magnetic trapping of ultra cold tional[1]andinternational[17]standards.Newinterpre- neutronsiscurrentlyunderwaywiththeIonizingRadia- tations of some of Mandel’s work by SED statisticians tionDivisionofthePhysicsLaboratory[15].SEDstaff [53], and the solution of outstanding problems, notably are part of an international team of researchers from estimationwherenotalllaboratoriesareoperatingwith NIST, Harvard University, Los Alamos National Labo- thesameprecision,hasgarneredrecognitionwithinthe ratory, and the Hahn-Meitner-Institute in Berlin. The statistical community. team proposed a new technique to trap ultra cold neu- Recently, the statistical modeling of interlaboratory tronsinamagneticfield.Withthistechnology,theteam test data has led to advances in the theory of linear planstomakeahighprecisionmeasurementofthemean mixed-effectsmodelsfromgraphicalandlikelihoodap- lifetime of the neutron. Along with other experimental proaches and to Bayesian solutions to combining mea- data,themeanlifetimeoftheneutronallowsonetotest surements over multiple laboratories or methods [32]. the consistency of the standard model of electroweak Internationalcomparisonsofbasicmetrologicalstan- interactions.Themeanlifetimeoftheneutronisalsoan dards are currently an important component of SED important parameter in astrophysical theories. Eventu- activities at NIST. Studies, known as key comparisons, ally,thismethodshouldyieldalifetimeestimatewithan forcomparingmeasurementsamongNMIshavetakena uncertainty 10 to 100 times smaller than the current critical place in the NIST mission. Their purpose is to uncertainty. establishthedegreeofequivalenceofnationalmeasure- Statistical contributions to this project include plan- mentstandardsmaintainedbyNMIsandprovideforthe ning of a multi-run experiment which is performed at mutualrecognitionofcalibrationandmeasurementcer- the NIST Cold Neutron Research Facility. A magnetic 285 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology trap is filled with neutrons for a prescribed time. After [12]. Optoelectronic devices are critical for high band- thefillingstageofeachrun,theneutronbeamisblocked width measurements of high performance optical fiber and decay events plus background events are observed systems.Aphotodiodeconvertsanopticalsignalintoan duringtheeventobservationstageofeachrun.Basedon electricalsignal.Thiselectricalsignalisdetectedwitha a birth-death stochastic model of the neutron trapping highspeedequivalenttimesamplingoscilloscope.Both process, the statisticians have developed an algorithm thephotodiodeandoscilloscopehaveimpulseresponse whichdeterminestheoptimalamountoftimeforfilling functions which distort the signal of interest. and the optimal amount of time for observing events. SEDstaffaredevelopingstatisticalmethodsandasso- Thisalgorithmhasplayedacriticalroleintheplanning ciated software for calibration of high-speed digital of the second generation of the experiment now under- sampling oscilloscopes and characterizing the impulse way. Some of the data from these experiments and a responseofphotodiodes.Statisticaltasksincludedevel- schematic diagram of the magnetic trap are shown in opmentofestimationmethodsandalgorithmsfortime- Figs. 3 and 4. base distortion estimation and correction; drift estima- Another example is a collaboration which began in tion;signalalignment;andtimingjitterestimation.SED 1988withtheOptoelectronicsDivisionoftheElectron- staff have developed statistical methods and associated ics and Electrical Engineering Laboratory to develop software used in a measurement system for sampling statistical signal processing methods for analysis of oscilloscopes(upto50GHz)tocorrectsignalsforsys- time-domain optoelectronic response measurements tematicerrorsduetotimebasedistortion,drift,andjitter. Fig.3. Theseplotsshowobserveddatawithassociatedstandarduncertaintiesfortwoexperiments.Foreachexperiment,thepredictedcountrate isshownasasolidline. 286 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology Fig.4. Schematicdiagramofthemagnetictrapwhichconfinesultracoldneutrons. In the near future, the experimental work will be LakeSuperiorfishtissue;Portlandcementandurbanair extendedtohigherwavelengths.Statisticalmethodswill particulate matter. Multi-constituent SRMs offer spe- be developed to evaluate the overall uncertainty of the cial challenges because the experimental configuration estimatedpowerandphasespectrumoftheoscilloscope isoftenhighlyunbalancedandcannotalwaysbetreated and photodiode impulse response functions, and time by standard statistical methods. domain measurements will be compared to heterodyne SRMs from other NIST laboratories and measure- measurements. ment areas cover a variety of applications that include sinusoidalroughnessofsteelblocks;magnificationlevel forscanningelectronmicroscopes;resistivityofsilicon 6. Standard Reference Materials wafers; transmittance of optical density filters; charac- terization of radionucleides in ocean sediment, Rock- One of the ongoing metrology activities that SED wellCScaleforhardness;fracturetoughnessofceram- supportsatNISTisthecertificationofStandardRefer- ics; wavelength reference for a hydrogen cyanide enceMaterials(SRMs).SRMsareartifactsorchemical absorption cell; and thermal resistance of fibrous glass compositions that are manufactured according to strict insulation.Alistingofpublicationsrelatedtothecertifi- specifications and certified by NIST for one or more cationofSRMscanbefoundathttp://www.itl.nist.gov/ chemical or physical properties. SRMs are a primary div898/pubs/subject/srm.html. vehicle for disseminating measurement technology to The development of a new SRM typically takes 2 to industry.Inthe1970s,theStatisticalEngineeringDivi- 5yearsandencompassesdesignofaprototype,stability sionenteredaphaseofintensiveinteractionswithdevel- testing,quantificationofsourcesoferror,andcertifica- opersofSRMsatNIST.Thisactivitypersiststothisday, tion and uncertainty analysis. Statisticians collaborate andSEDstaffareheavilyinvolvedinthecertificationof with NIST chemists and scientists and advise on the large numbers of SRMs each year. designandanalysisofexperimentsatallphases;develop ThelargestnumberofSRMsarechemicalcomposi- estimation methods; reconcile interlaboratory differ- tions from the Analytical Chemistry Division of the ences;testandestimatetheeffectofinhomogeneityon Chemical Sciences and Technology Laboratory. These thecertifiedvalue;andcombineallinformationtopro- SRMsareincrediblyvariedandmany,particularlythose duce a certified value and statement of uncertainty. A forenvironmentalapplications,arecertifiedfortheper- methodforcombiningmeasurementsovermultiplelab- centageconcentrationof50constituentsormorewhere oratoriesormethodswhichhavesignificantdifferences theconstituentsarecontainedinanaturalmatrixsuchas is addressed in a 1991 paper [28]. sludge from a river bottom. Typical multi-constituent Non-standardmetrologies,suchasvideoimaging,are SRM materials include marine sediment; uric acid; also disseminated via SRMs and often present special 287 Volume106,Number1,January–February2001 Journal of Research of the National Institute of Standards and Technology challenges. Automation of semiconductor production treated as a probability density function, a sharp SEM requiresscanningelectronmicroscopes(SEMs)thatare image corresponds to a spectrum which has a large capableofmeasuringfeaturesizeswithouthumaninter- shoulderoraflatshape.Thetestproceduremonitorsthe ventionforlongperiodsoftime.AnSEDstaffmember kurtosis(flatness)statistictodetectanyincreaseinkur- has collaborated with the Precision Engineering Divi- tosisthatsignalsdegradationinsharpness.Thistypeof sion of the Manufacturing Engineering Laboratory to collaboration,whichbeginswithanindustrialmeasure- developastatisticalmethodfortestingtheperformance ment problem and results in artifacts and test methods of scanning electron microscopes [35] that is the basis that allow proper use of the artifacts, requires perhaps for a new SRM (see Fig. 5). severalyearsworktobringtofruitionasthemethodol- AsimpleexplanationisthatanSEMimagewithfine ogymustbedeveloped,testedatNIST,andfinallytested detailsis“sharp”.Ifthenormalizedspatialspectrumis in the industrial setting. Fig.5. Parts(a)and(c)ofthefigureshowtwomicrographstakenwithanSEM.Micrograph(a)appearstobefarlesssharpthan micrograph(c),takenwhenthesameinstrumentwasoperatingmoreoptimally.Parts(b)and(d)showthe2-DspatialFourierfrequency magnitudedistributionsforthetwomicrographs.NoticethatthemagnitudedistributionoftheFouriertransformoftheimagesiswider for(c)thanfor(a).Treatingthenormalizedspectrumsasprobabilitydensityfunctions,thesharpnessofanSEMimagecanthenbe determinednumericallybyitsmultivariatekurtosis. 288