DOCUMENT RESUME CE 081 270 ED 449 354 AUTHOR Kirsch, Irwin; Yamamoto, Kentaro; Norris, Norma; Rock, Donald; Jungeblut, Ann; O'Reilly, Patricia; Berlin, Martha; Mohadjer, Leyla; Waksberg, Joseph; Goksel, Huseyin; Burke, John; Rieger, Susan; Green, James; Klein, Merle; Campbell, Anne; Jenkins, Lynn; Kolstad, Andrew; Mosenthal, Peter; Baldi, Stephane Technical Report and Data File User's Manual for the 1992 TITLE National Adult Literacy Survey. National Center for Education Statistics (ED), Washington, INSTITUTION DC. NCES-2001-457 REPORT NO 2001-01-00 PUB DATE NOTE 648p.; Project Officer: Andrew Kolstad. ED Pubs, P.O. Box 1398, Jessup, MD 20794-1398. Tel: AVAILABLE FROM 877-433-7827 (Toll Free), Fax: 301-470-1244, TTY/TDD: 800 437-0833, E-mail: [email protected], Web site: http://www.ed.gov/pubs/edpubs.html. Research (143) Reports Non-Classroom (055) PUB TYPE Guides Tests /Questionnaires (160) MF03/PC26 Plus Postage. EDRS PRICE Adult Basic Education; *Adult Literacy; *Data Processing; DESCRIPTORS Educational Research; Illiteracy; Literacy Education; *Material Development; *National Surveys; Prisoners; Research Administration; Research Design; Research Methodology; Research Problems; Sampling; Scaling; State Surveys; Statistical Analysis; Validity; Weighted Scores *National Adult Literacy Survey (NCES) IDENTIFIERS ABSTRACT Chapter 1 of this report and user's manual describes design and implementation of the 1992 National Adult Literacy Survey (NALS). Chapter 2 reviews stages of sampling for national and state survey components; presents weighted and unweighted response rates for the household component; and describes non-incentive and prison sample designs. Chapter 3 addresses weighting procedures. Chapter 4 describes NALS's conceptual framework and development of the background questionnaire and literacy tasks. Chapters 5 and 6 document field operations for the household and prison surveys. Chapters 7-9 detail data processing; the missing data procedures; and models and procedures used to scale NALS results, estimate respondents' proficiencies, and conduct statistical analyses. Chapter 10 presents a summary of literature on use of monetary incentives in survey research, experimental features of the field test and national study, and research results. Chapters 11-15 describe statistics, components of variance, and statistical methods used to derive outcome estimates; estimate and test discriminant validity of the three literacy scales from the perspective of correlation or covariance; summarize establishment of NALS literacy levels; explore importance of the response probability convention in reporting prose literacy results; and discuss use of SPSS (Statistical Program for the Social Sciences) and SAS (Statistical Analysis System) for analyzing NALS data. Appendixes include 98 references, additional data and notes, and instruments. (YLB) Reproductions supplied by EDRS are the best that can be made from the original document. ATIONAL CENTER FOR EDUCATION STATISTICS c7, Technical Report January 2001 Q Technical Report and Data File User's Manual for the 1992 National Adult Literacy Survey U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement ED CATIONAL RESOURCES INFORMATION CENTER (ERIC) This document has been reproduced as received from the person or organization originating it. 0 Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. BEST COPY AVAILABLE U.S. Department of Education Office of Educational Research and Improvement NCES 2001-457 2. Customer Survey of OERI Publication Users To help us improve future editions of this publication and give 5. For what purposes did you use this OERI you better customer service, we would appreciate your (Check all that apply.) publication? comments on this survey form. Please check the appropriate Planning box(es) below for each question. Responses will be kept Policy or legislation completely confidential. You may return the survey by mail or FAX. It can be folded and taped closed to allow mailing to the Administrative decisions address listed on the reverse side of this form, or it can be Teaching, class material returned by FAX to 202-219-1321. Many thanks for your customer feedbackit is very important to usl 1:1 Research/analysis General information la. Name of publication Writing news articles, TV or radio material Marketing, sales, or promotion lb. Publication number NCES 2001-457 Other (please describe) lc. Author name How did you receive a copy of this publication? 2. 6. Did the publication help you accomplish whatever you Bought it needed it for? Borrowed it Partially No C]l Yes 1:1 Mailing list membership 7. What is your occupation? Telephone request Administrator Teacher Parent Internet request Statistician Researcher Librarian 1:1 Other (please describe) Student Policy Analyst Journalist/writer Program Planner Other (please specify) Was this publication easy to get? 3. Not at all Somewhat Very How did you find out about this and other OERI 4. 8. How could this OERI publication (or other OERI publications) better meet your needs? (Check all that apply.) publications? (Check all that apply.) Conferences More important topics in education Journal articles More timely release of data Teacher/educator More text introductions to each section Professional associations More research statistics Internet (WWW) Shorter reports (less than 10 pages) Publication announcement Other (please describe) Received in mail OERI staff contact Dissatisfied Satisfied Very Satisfied Overall, how satisfied are you 9. with this product? Comprehensiveness of information a. Clarity of writing (readability, interpretability) b. Clarity of presentation (e.g., tables, charts) c. Timeliness of information d. Accuracy of information e. Clarity of technical notes f. E:11 Usefulness of resources and bibliography g. (=I Organization h. Length i. CIE j. Format [:11 PAPERWORK BURDEN STATEMENT Office of Education Research and improvement (OERI) Publication Customer Survey control number. The valid According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB OMB control number for this information collection is 1880-0529.The time required to complete this information collection is estimated to average 10 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s), suggestions for improving this form, or concerns regarding the status of your individual submission of this form, write directly to: P. Quinn, Room 204, Media and Information Services, OERI, U.S. Department of Education, 555 New Jersey Avenue NW, Washlnaton. nn onon.A570. BEAT COPY AVAILABLE OERI Publication Customer Survey NO POSTAGE Media and Information Services NECESSARY U.S. Department of Education IF MAILED Washington, DC 20202 IN THE UNITED STATES Official Business Penalty for Private Use, $300 BUSINESS REPLY MAIL FIRST-CLASS MAIL WASHINGTON DC PERMIT NO. 012935 POSTAGE WILL BE PAID BY U.S. DEPARTMENT OF EDUCATION U.S. Department of Education Mail Code: 5570 400 Maryland Avenue, S.W. Washington, D.C. 20277-2935 11111111111111111111111111111111111111111111111111111111 Fold on lineTAPE CLOSEDDO NOT STAPLE 10. Do you have any suggestions regarding the content or format of future editions of this publication or other comments? . 4 MIS 1999-6532 NATIONAL CENTER FOR EDUCATION STATISTICS Technical Report January 2001 Technical Report and Data File User's Manual for the 1992 National Adult Literacy Survey Irwin Kirsch, Kentaro Yamamoto, Norma Norris, Donald Rock, Ann Jungeblut, and Patricia O'ReillyEducational Testing Service Martha Berlin, Ley la Mohadjer, Joseph Waksberg, Huseyin Goksel, John Burke, Susan Rieger, James Green, and Merle Klein--- Westat, Inc. Anne CampbellDine College Lynn JenkinsWordsworth Writing and Editing Andrew KolstadNational Center for Education Statistics Peter MosenthalSyracuse University Stephane BaldiAmerican Institutes for Research Andrew Kolstad Project Officer National Center for Education Statistics U.S. Department of Education NCES 2001-457 Office of Educational Research and Improvement 5 U.S. Department of Education Richard W. Riley Secretary Office of Educational Research and Improvement C. Kent McGuire Assistant Secretary National Center for Education Statistics Gary W. Phillips Acting Commissioner The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, and reporting data related to education in the United States and other nations. It fulfills a congressional mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in the United States; conduct and publish reports and specialized analyses of the meaning and significance of such statistics; assist state and local education agencies in improving their statistical systems; and review and report on education activities in foreign countries. NCES activities are designed to address high priority education data needs; provide consistent, reliable, complete, and accurate indicators of education status and trends; and report timely, useful, and high quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, practitioners, data users, and the general public. We strive to make our products available in a variety of formats and in language that is appropriate to a variety of audiences. You, as our customer, are the best judge of our success in communicating information effectively. If you have any comments or suggestions about this or any other NOES product or report, we would like to hear from you. Please direct your comments to: National Center for Education Statistics Office of Educational Research and Improvement U.S. Department of Education 1900 K Street, NW Washington, DC 20006-5651 January 2001 The NOES World Wide Web Home Page is: http://nces.ed.gov The NOES World Wide Web Electronic Catalog is: http://nces.ed.gov/pubsearchfindex.asp Suggested Citation U.S. Department of Education. National Center for Education Statistics. Technical Report and Data File User's Manual for the 1992 National Adult Literacy Survey, NCES 2001-457, by Irwin Kirsch, Kentaro Yamamoto, Norma Norris, Donald Rock, Ann Jungeblut, Patricia O'Reilly, Martha Berlin, Leyla Mohadjer, Joseph Waksberg, Huseyin Goksel, John Burke, Susan Rieger, James Green, Merle Klein, Anne Campbell, Lynn Jenkins, Andrew Kolstad, Peter Mosenthal, and Stephane Baldi. Andrew Kolstad, Project Officer. Washington, DC: 2000. For ordering information on this report, write: U.S. Department of Education ED Pubs P.O. Box 1398 Jessup, MD 20794-1398 or call toll free 1-877-4EDPubs. Content Contact: Andrew Kolstad (202) 502-7374 TABLE OF CONTENTS 1 CHAPTER 1: THE NATIONAL ADULT LITERACY SURVEY: AN OVERVIEW 1.1 Introduction 1 3 1.2 Defining Literacy 3 1.3 The Sample 4 1.4 Weighting 5 1.5 The Survey Instrument: Measuring Literacy 7 1.6 Field Operations 8 1.7 Data Processing and Missing Data 9 1.8 Scaling and Proficiency Estimates 10 1.9 Establishing Literacy Levels 11 CHAPTER 2: SAMPLE DESIGN 11 2.1 Overview 13 2.2 Sampling for the National Component 14 2.2.1 First-Stage Sample 14 2.2.1.1 Westat's master sample of PSUs 14 2.2.1.2 Selecting the sample of PSUs for the national component 21 2.2.2 Second-Stage SampleSelecting Census Blocks (Segments) 22 2.2.2.1 Measures of size and sampling rates 23 2.2.2.2 Minimum segment size 24 2.2.2.3 Segment sample selection 24 2.2.2.4 TIGER maps 25 2.2.2.5 Listing sample segments 26 2.2.3 Third-Stage SampleSelecting Housing Units 26 2.2.3.1 Within-segment sampling rate 28 2.2.3.2 Overall probabilities of selection 28 2.2.3.3 Procedures for selecting missed structures and missed dwelling units 29 2.2.4 Fourth-Stage SampleSelecting Persons Age 16 or Older 30 2.3 The Non-Incentive Sample ..30 2.4 Sampling for the State Literacy Surveys 31 2.4.1 Sample of PSUs 31 2.4.2 Sample of Segments 32 2.4.3 Sample of Housing Units 32 2.4.4 Sample of Persons 32 2.5 Weighted and Unweighted Response Rates 34 2.6 Sampling for the Prison Survey 34 2.6.1 Sample of Correctional Facilities 34 2.6.1.1 Sampling frame and selection of correctional facilities 37. 2.6.2 Selection of Inmates Within Facilities 39 CHAPTER 3: WEIGHTING AND POPULATION ESTIMATES 39 3.1 Goals of Weighting 41 3.2 Calculating Sample Weights for the Household Population 41 3.2.1 Preliminary Steps in Weighting 43 3.2.2 Computing Base Weights 44 3.2.3 Nonresponse Adjustments and Poststratification 45 3.2.4 Compositing Data from the National and State Components 45 3.2.4.1 Composite estimation procedure TABLE OF CONTENTS CONTINUED 3.2.4.2 Deriving the PSU design effect 48 3.2.4.3 Estimating composite factors 49 3.2.5 Computing Final WeightPoststratification Through Raking Ration Adjustments 54 3.3 Replicated Weights for Variance Estimation in the Household Population 55 3.3.1 Household Sample Replication for the National Component 57 3.3.2 Household Sample Replication for the State Component 57 3.3.3 Final Household Sample Replication for the National and State Components 58 3.4 Calculating Sample Weights for the Prison Population 58 3.4.1 Computing Inmate Base Weights 58 3.4.2 Nonresponse Adjustments 62 3.4.2.1 Facility nonresponse adjustment 62 3.4.2.2 Inmate nonresponse adjustment 63 3.4.3 Poststratification Procedures 64 3.4.4 Final Inmate Weights 67 3.5 Replication Weights for Variance Estimation in the Prison Population 68 CHAPTER 4: DEVELOPMENT OF THE SURVEY INSTRUMENTS 70 4.1 Conceptual Framework 70 4.2 The Scope of the Background Questionnaire 72 4.2.1 General and Language Background 73 4.2.2 Educational Background and Experiences 73 4.2.3 Political and Social Participation 74 4.2.4 Labor Force Participation 74 4.2.5 Literacy Activities and Collaboration 75 4.2.6 Demographic Information 75 4.2.7 Prison Survey Background Questionnaire 76 4.2.8 Spanish Versions of the Questionnaires 76 4.3 Development of the Simulation Tasks 76 4.3.1 Organizing Framework for Task Development 77 4.3.2 Materials/Structures 77 4.3.3 Adult Contexts/Content 78 4.3.4 Processes/Strategies 79 4.3.5 Task Difficulty 87 4.3.6 Development of Scoring Guides 88 4.3.7 Assembling the Tasks for Administration 89 CHAPTER 5: THE HOUSEHOLD SURVEY 93 5.1 Overview 93 5.2 Listing 94 5.2.1 Staff Organization for Listing 94 5.2.2 Training Listers 95 5.2.3 Listing Materials 95 5.2.4 The Listing Operation 96 5.2.5 Quality Control Procedures 97 5.2.5.1 Quality control of listing sheets 97 5.2.5.2 Quality control of the listing operation 97 5.3 Data Collection Instruments and Interviewer Materials 99 5.3.1 The Soreener 99 5.3.2 Interview Guides for Exercise Booklets 101 5.3.3 Non-interview Report Forms 101 e ii CONTINUED TABLE OF CONTENTS 104 5.3.4 Interviewer Manuals 104 5.3.5 Field Aids 104 5.3.5.1 Aids used for locating and contacting respondents 105 5.3.5.2 Aids used for obtaining respondent cooperation 106 5.3.5.3 Aids used during the interview 106 5.4 Field Organization and Training 106 5.4.1 Field Organization 106 5.4.1.1 Lines of responsibility 107 5.4.1.2 Interviewer recruitment 109 5.4.2 Training 109 5.4.2.1 Supervisor training 110 5.4.2.2 Interviewer training 114 5.4.2.3 Editor training 114 5.5 Field Operations 115 5.5.1 General Approach to the Field Effort 116 5.5.2 Schedule and Production 117 5.5.3 Reporting Systems 117 5.5.3.1 Automated Survey Control System (ASCS) 118 5.5.3.2 Interviewer reports to the supervisor 118 5.5.3.3 Supervisor reports to the home office 119 5.5.3.4 Home office staff reports to ETS and to NCES 119 5.6 Quality Control of Data Collection 119 5.6.1 Introduction 119 5.6.2 Editing 121 5.6.3 Validation 122 5.6.4 Observation 123 5.6.5 Supervisor Observations 123 5.7 Response Rates 125 5.7.1 Reasons for Non-response 126 5.7.2 Characteristics of Non-respondents 128 5.7.3 Discussion 131 CHAPTER 6: THE PRISON SURVEY 131 6.1 Sample Design 132 6.2 Gaining Cooperation 133 6.3 Interviewer Selection and Training 134 6.4 Data Collection 134 6.5 Quality Control 136 CHAPTER 7: PROCESSING THE DATA 136 7.1 Receipt Control 136 7.1.1 Screener 137 7.1.2 Background Questionnaire 137 7.1.3 Exercise Envelope 138 7.2 Coding and Scoring 138 7.2.1 Coding Background Questionnaires 139 7.2.2 Scoring Simulation Tasks 140 7.3 Data Entry 141 7.4 Editing and Quality Control iii 9 TABLE OF CONTENTS CONTINUED CHAPTER 8: ESTIMATING LITERACY PROFICIENCIES WITH AND WITHOUT COGNITIVE DATA 142 8.1 The Normal Treatment of Missing Cognitive Data 143 8.1.1 Omitted Answers and Questions Not Reached 143 8.1.2 Statistical Imputation Through Scaling 144 8.2 Reasons Cognitive Data Were Missing/Not Reached 146 8.2.1 Non-interview Reports and Low Literacy Skills 151 8.2.2 Internal Evidence for the Validity of Reasons 153 8.3 Using 'Reasons' to Improve Treatment of Missing Cognitive Data 155 8.3.1 Five Logical Imputation Methods Considered 156 8.3.2 Five Methods Applied to 1991 Field Test Data 158 8.3.3 The Method Selected 163 8.4 Final Evaluation 164 CHAPTER 9: SCALING AND PROFICIENCY ESTIMATES 165 9.1 Scaling 165 9.2 Scaling Methodology 168 9.2.1 The Scaling Model 168 9.2.2 Design for Linking the 1992 Scales to the 1985 Scales 169 9.2.3 Item Parameter Estimation 170 9.3 Proficiency Estimation Using Plausible Values 177 9.3.1 Generating Proficiency Scores 177 9.3.2 Linking the 1992 Scale to the 1985 Scale 182 9.3.3 Evaluation of Differential Group Performance 183 9.4 Statistical Tests 187 9.4.1 Analysis of Plausible Values 187 9.4.2 Partitioning the Estimation Error Variance: A Numerical Example 188 9.4.3 Minimum Sample Sizes for Reporting Subgroup Results 190 9.4.4 Estimates of Standard Errors with Large Mean Squared Errors 190 CHAPTER 10: THE ROLE OF INCENTIVES IN LITERACY SURVEY RESEARCH 191 10.1 Literature Review 191 10.2 The 1991 Field Test 193 10.2.1 Field Test Design 193 10.2.2 Summary of Field Test Results 194 10.2.3 Field Test Response Rates 196 10.2.4 Representation of the Target Population in the Field Test 198 10.2.5 Relationship Between Incentive Level, Self-Selection, and Performance in the Field Test 200 10.2.6 Survey Costs for the Field Test 208 10.2.7 Conclusions from the 1991 Field Test 209 10.3 The 1992 Incentive Experiment 210 10.3.1 Sample Design for the 1992 National Adult Literacy Survey 210 10.3.2 1992 Incentive Experiment Design 211 10.3.3 Analysis of Response Rates 212 10.3.3.1 Screener 213 10.3.3.2 Background questionnaire and exercise booklet 214 10.4 Summary and Conclusion 217 10 iv fly