Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal Statistical Agencies

265 Pages·2022·3.5 MB·English

by  Panel on Transparency and Reproducibility of Federal Statistics for the National Center for Science and Engineering Statistics

THE NATIONAL ACADEMIES PRESS
   
This PDF is available at http://nap.edu/26360 SHARE
Transparency in Statistical Information for the National
Center for Science and Engineering Statistics and All
Federal Statistical Agencies (2022)
DETAILS
264 pages | 6 x 9 | PAPERBACK
ISBN 978-0-309-27045-8 | DOI 10.17226/26360
CONTRIBUTORS
GET THIS BOOK Panel on Transparency and Reproducibility of Federal Statistics for the National
Center for Science and Engineering Statistics; Committee on National
Statistics; Division of Behavioral and Social Sciences and Education; National
Academies of Sciences, Engineering, and Medicine
FIND RELATED TITLES
SUGGESTED CITATION
National Academies of Sciences, Engineering, and Medicine 2022. Transparency in
Statistical Information for the National Center for Science and Engineering
Statistics and All Federal Statistical Agencies. Washington, DC: The National
Academies Press. https://doi.org/10.17226/26360.

Visit the National Academies Press at NAP.edu and login or register to get:
– Access to free PDF downloads of thousands of scientific reports 
– 10% off the price of print titles

– Email or social media notifications of new titles related to your interests
– Special offers and discounts
Distribution, posting, or copying of this PDF is strictly prohibited without written permission of the National Academies Press.
(Request Permission) Unless otherwise indicated, all materials in this PDF are copyrighted by the National Academy of Sciences.
Copyright © National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All Federal...
Panel on Transparency and Reproducibility of Federal Statistics for
the National Center for Science and Engineering Statistics
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
A Consensus Study Report of
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and All...
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by a contract between the National Academies of
Sciences, Engineering, and Medicine and the National Science Foundation under
grant number 1822391. Any opinions, findings, conclusions, or recommendations
expressed in this publication do not necessarily reflect the views of any organization
or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-27045-8
International Standard Book Number-10: 0-309-27045-6
Digital Object Identifier: https://doi.org/10.17226/26360
Additional copies of this publication are available from the National Academies
Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; [phone] or
[phone]; http://www.nap.edu.
Copyright 2022 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine.
2022. Transparency in Statistical Information for the National Center for Science
and Engineering Statistics and All Federal Statistical Agencies. Washington, DC:
The National Academies Press. https://doi.org/10.17226/26360.
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
The National Academy of Sciences was established in 1863 by an Act of C ongress,
signed by President Lincoln, as a private, nongovernmental institution to advise
the nation on issues related to science and technology. Members are elected
by their peers for outstanding contributions to research. Dr. Marcia McNutt is
president.
The National Academy of Engineering was established in 1964 under the char-
ter of the National Academy of Sciences to bring the practices of engineering
to advising the nation. Members are elected by their peers for extraordinary
contributions to engineering. Dr. John L. Anderson is president.
The National Academy of Medicine (formerly the Institute of Medicine) was
established in 1970 under the charter of the National Academy of Sciences to
advise the nation on medical and health issues. Members are elected by their
peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau
is president.
The three Academies work together as the National Academies of Sciences,
Engi neering, and Medicine to provide independent, objective analysis and
advice to the nation and conduct other activities to solve complex problems and
inform public policy decisions. The National Academies also encourage education
and research, recognize outstanding contributions to knowledge, and increase
public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine
at www.nationalacademies.org.
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
Consensus Study Reports published by the National Academies of Sciences,
Engineering, and Medicine document the evidence-based consensus on the
study’s statement of task by an authoring committee of experts. Reports typi-
cally include findings, conclusions, and recommendations based on information
gathered by the committee and the committee’s deliberations. Each report
has been subjected to a rigorous and independent peer-review process and it
represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and
Medicine chronicle the presentations and discussions at a workshop, symposium,
or other event convened by the National Academies. The statements and opinions
contained in proceedings are those of the participants and are not endorsed by
other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies,
please visit www.nationalacademies.org/about/whatwedo.
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
PANEL ON TRANSPARENCY AND REPRODUCIBILITY
OF FEDERAL STATISTICS FOR THE NATIONAL CENTER
FOR SCIENCE AND ENGINEERING STATISTICS
DANIEL KASPRZYK (Chair), NORC at the University of Chicago
PHILIP ASHLOCK, GSA Technology Transformation Services, General
Services Administration
DAVID BARRACLOUGH, Practices and Solutions Division, Organisation
for Economic Co-operation and Development
CHRISTOPHER CHAPMAN, Sample Surveys Division, National Center
for Education Statistics
DANIEL W. GILLMAN, Office of Survey Methods Research, U.S. Bureau
of Labor Statistics
LINDA A. JACOBSEN, Population Reference Bureau, Inc.
H. V. JAGADISH, Department of Computer Science and Engineering,
University of Michigan
FRAUKE KREUTER, Joint Program in Survey Methodology, University
of Maryland
MARGARET LEVENSTEIN, Inter-university Consortium for Political
and Social Research, University of Michigan
PETER V. MILLER, U.S. Census Bureau (retired)
AUDRIS MOCKUS, Department of Electrical Engineering and Computer
Science, University of Tennessee, Knoxville
SARAH M. NUSSER, Center for Survey Statistics and Methodology,
Iowa State University
ERIC RANCOURT, Modern Statistical Methods and Data Science Branch,
Statistics Canada
WILLIAM L. SCHERLIS,* School of Computer Science, Carnegie Mellon
University
LARS VILHUBER, Department of Economics, Cornell University
*Resigned from panel on October 28, 2019
MICHAEL L. COHEN, Senior Program Officer
MICHAEL SIRI, Associate Program Officer
CONNIE F. CITRO, Senior Scholar
JILLIAN KAUFMAN, Program Coordinator (until January 15, 2020)
ANTHONY MANN, Program Coordinator
JOHN GAWALT, Consultant (until May 18, 2020)
v
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
COMMITTEE ON NATIONAL STATISTICS
ROBERT M. GROVES (Chair), Office of the Provost, Department
of Mathematics and Statistics and Department of Sociology,
Georgetown University
LAWRENCE D. BOBO, Department of Sociology, Harvard University
ANNE C. CASE, Woodrow Wilson School of Public and International
Affairs, Princeton University
MICK P. COUPER, Survey Research Center, Institute for Social Research,
University of Michigan
JANET M. CURRIE, Woodrow Wilson School of Public and
International Affairs, Princeton University
DIANA FARRELL, JPMorgan Chase Institute, Washington, DC
ROBERT GOERGE, Chapin Hall at The University of Chicago
ERICA L. GROSHEN, The ILR School, Cornell University
HILARY HOYNES, Goldman School of Public Policy, University of
California, Berkeley
DANIEL KIFER, Department of Computer Science and Engineering,
The Pennsylvania State University
SHARON LOHR, Consultant and Freelance Writer
JEROME P. REITER, Department of Statistical Science, Duke University
JUDITH A. SELTZER, Department of Sociology, University of
California, Los Angeles
C. MATTHEW SNIPP, Department of Sociology, Stanford University
ELIZABETH A. STUART, Department of Mental Health, Johns Hopkins
Bloomberg School of Public Health
JEANETTE WING, Data Science Institute, Columbia University
BRIAN HARRIS-KOJETIN, Board Director
MELISSA CHIU, Deputy Board Director
CONNIE F. CITRO, Senior Scholar
vi
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
Acknowledgments
A Consensus Study Panel requires many individuals to assist the panel in
studying the issues identified in the panel’s statement of task. The Panel on
Transparency and Reproducibility of Federal Statistics for the National
Center for Science and Engineering Statistics is no different. Many experts
were called upon to discuss issues, provide their expertise, and discuss their
perspectives for the panel’s consideration. The panel thanks all these indi-
viduals for the assistance and knowledge.
The panel benefitted greatly from the presentations provided in its
open sessions. The experts the panel heard from can be clustered into
the following perspectives and areas of expertise (see Appendix C for the
agendas for open meetings): NCSES staff: Emilda Rivers, May Aydin,
Tiffany Julian, and Francisco Moris; experts in metadata standards as used
internationally: Olivier Dupriez (World Bank), Pascal Heus (Metadata
Technology North America), Heidi Koumarianos (Institut National de
la Statistique et des Études Économiques), and Juan Munoz (National
Institute of Statistics and Geography, Mexico); experts from the fed-
eral statistical system: William Bell (Census Bureau), Marcus Berzofsky
(RTI International), Christopher Carrino (Census Bureau), Leighton L
Christiansen (Bureau of Transportation Statistics), Brad Edwards (Westat),
John Eltinge (Census Bureau), Dennis Fixler (Bureau of Economic Analysis),
Nick Hart (Data Coalition), Nancy Potok (formerly Office of Management
and Budget), Mark Prell (Economic Research Service), Marilyn Seastrom
(National Center for Education Statistics), Tori Velkoff (Census Bureau),
and Zack Whitman (Census Bureau); experts in computer science: Jeremy
Iverson and Dan Smith (Colectica), and Natasha Noy (Google); experts in
vii
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
viii ACKNOWLEDGMENTS
administrative records data: John Czajka and Mathew Stange (Mathematica
Policy Research); and an expert in the federal statistical user commu-
nity: Jason Jurjevich (University of Arizona). We also heard from expert
users of NCSES data: Kimberlee Eberle-Sudre (Association of American
Universities) and Anne-Marie Knott (Washington University in St. Louis).
In addition to these public presentations, panel and staff participated in
meetings and conference calls with staff from NCSES and the Interagency
Council on Statistical Policy as well as George Alter (Inter-university
Consortium for Political and Social Research), Jeremy Iverson (Colectica),
and Rolf Schmitt and Leighton L Christiansen (Bureau of Transportation
Statistics). Further, to gain insight into what is currently carried out in
major statistical programs in terms of documentation and archival policy,
the panel sent an informal questionnaire to the leaders of 20 programs
of the federal statistical system, receiving responses from 11. The results of
this questionnaire are provided in Chapter 2.
The panel and staff also studied a number of domestic and international
documents that called for greater openness and transparency concerning
national statistics. This included documents from NCSES, the Committee
on National Statistics, the U.S. Office of Management and Budget (OMB),
the United Nations Economic Commission for Europe (UNECE), Statistics
Canada, the American Association for Public Opinion Research (AAPOR),
and the White House.
The panel is also indebted to John Gawalt, previous director of NCSES,
who not only helped to develop the funding for this study, but also served
as unpaid consultant until May 2020. His knowledge of the federal statisti-
cal system and NCSES was invaluable as the panel interpreted its charge
and organized its open sessions. In addition, John actively participated in
weekly meetings or conference calls with the chair and staff which greatly
helped clarify what issues the panel needed to focus its attention on and
which helped organize the structure of the report.
The panel itself could draw on its own considerable expertise advis-
ing on programs from the federal statistical system, or in areas relevant to
the new directions that had been discussed at a prior workshop on trans-
parency. By subject area, these experts included: from federal statistical
system: Philip Ashlock (General Services Administration, including data.
gov), Christopher Chapman (National Center for Education Statistics),
Dan Gillman (Bureau of Labor Statistics, Census Bureau), Dan Kasprzyk
(Census Bureau, National Center for Education Statistics), Peter Miller
(Census Bureau), and Sarah Nusser (Iowa State University); concern-
ing metadata standards and tools: David Barraclough (Organisation for
Economic Co-operation and Development [OECD]) and Dan Gillman;
from inter national statistical agencies: David Barraclough (OECD), Frauke
Kreuter (Joint Program of Survey Methodology and the University of
Copyright National Academy of Sciences. All rights reserved.
Transparency in Statistical Information for the National Center for Science and Engineering Statistics and ...
ACKNOWLEDGMENTS ix
Mannheim), and Eric Rancourt (Statistics Canada); concerning computer
science tools applicable to federal statistics: H.V. Jagadish (University of
Michigan), Audris Mockus (University of Tennessee), and Lars Vilhuber
(Cornell University); concerning archiving: Margaret Levenstein (Inter-
university Consortium on Political and Social Research) and Lars Vilhuber;
and from the statistical user community: Linda Jacobsen (Population
Reference Bureau).
In creating the chapters of our report, the following individuals played
a key role: the first draft of the Summary was completed by Connie Citro of
CNSTAT; Chapter 1 and the tables in Chapter 7 were primarily drafted by
Peter Miller; Chapter 3 was primarily drafted by Lars Vilhuber, Margaret
Levenstein, and Frauke Kreuter; important parts of Chapter 4 were drafted
by Audris Mockus and Linda Jacobsen; Chapter 5 was drafted by Dan
Gillman and David Barraclough, and sections of this chapter were drawn
from material provided by Michael Lenard and Andrea Thomer, both of
the University of Michigan, consultants to the panel. Under the panel’s
guidance, Lenard and Thomer also completed the first draft of Appendix A,
while Dan Gillman drafted Appendix B.
Finally, the panel thanks staff for the preparation of the entire report.
Michael Cohen and Michael Siri provided tireless energy and enthusiasm
to the panel and its work, organizing open meetings, individual phone
calls, and Zoom meetings, following up on a myriad of issues and com-
ments, and organizing and drafting the report. Following through on the
comments and ideas of panel members was a significant undertaking. The
panel appreciated their interest and effort. Jillian Kaufman and Anthony
Mann provided excellent administrative support during the panel’s data
gathering activities.
This Consensus Study Report was reviewed in draft form by individuals
chosen for their diverse perspectives and technical expertise. The purpose
of this independent review is to provide candid and critical comments that
will assist the National Academies of Sciences, Engineering, and Medicine
in making each published report as sound as possible and to ensure that
it meets the institutional standards for quality, objectivity, evidence, and
responsiveness to the study charge. The review comments and draft manu-
script remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report:
Katharine G. Abraham, Joint Program in Survey Methodology, University
of Maryland, College Park; Christopher Carrino, Office of the Chief
Information Officer, U.S. Census Bureau; Leighton L Christiansen, Bureau
of Transportation Statistics; Mick P. Couper, Institute for Social Research,
University of Michigan; Robert L. Griess, Department of Mathematics,
University of Michigan; Pascal Heus, Metadata Technology North America;
Nicholas Horton, Statistics and Data Science, Amherst College; Juan
Copyright National Academy of Sciences. All rights reserved.