At or Above Level 3 Cut Score

109 Pages·2016·2.31 MB·English

by  Michael Bunch

Smarter Balanced
Assessment Consortium:
Achievement Level Setting Final Report

January 7, 2015
ALS Final Report

Executive Summary ................................................................................................................................ 5
CoOnvetrevinewt so f Tasks and Processes ........................................................................................................... 5
Online Panel Results ................................................................................................................................ 5
In-Person Workshop Results ................................................................................................................... 7
Results of Cross-Grade Review ............................................................................................................ 10
Impact by Subgroup .............................................................................................................................. 12
Action by Chiefs ..................................................................................................................................... 16
Section 1. Introduction ........................................................................................................................ 18
Smarter Balanced Background ............................................................................................................ 18
Purpose and Organization of this Report ............................................................................................ 19
Section 2. Communications Plan ........................................................................................................ 20
Purpose and goals ................................................................................................................................ 20
Targets ................................................................................................................................................... 20
Research and Planning ......................................................................................................................... 21
Environmental scan. ......................................................................................................................... 22
Focus group study. ............................................................................................................................ 23
Recruitment and Outreach ................................................................................................................... 23
Social media campaign/paid promotion. ........................................................................................ 24
Media Strategy/Communications Planning ........................................................................................ 24
Communications planning for key activities and state approval. .................................................. 24
Contingency planning ........................................................................................................................... 25
Intake interviews: Development of contingency plan, messaging and protocols. ........................ 25
Ebola contingency response. ........................................................................................................... 26
Digital and Creative Support ................................................................................................................ 26
Expansion of video production scope. ............................................................................................. 27
Achievement Level Naming Research ................................................................................................. 27
Interviews with assessment experts. ............................................................................................... 28
Focus groups among teachers. ........................................................................................................ 29
Online surveys among parents and students. ................................................................................. 29
Section 3. Logistics ............................................................................................................................. 31
Schedule ................................................................................................................................................ 31
Facilities ................................................................................................................................................. 32
Hardware and Software Support.......................................................................................................... 33
Meetings ................................................................................................................................................ 33
1
ALS Final Report

In-person workshop........................................................................................................................... 33
Cross-grade review. ........................................................................................................................... 36
Technical Advisory Committee Meetings ............................................................................................. 36
Collaboration Conferences ................................................................................................................... 36
Advisory Panel Meetings ....................................................................................................................... 37
National Conference on Student Assessment .................................................................................... 37
Other Meetings ...................................................................................................................................... 37
Security .................................................................................................................................................. 38
Section 4. Audit ................................................................................................................................... 40
Purpose of the Audits ........................................................................................................................... 40
Selection of the Auditors ...................................................................................................................... 40
Major Audit Functions ........................................................................................................................... 41
Review of tools and materials. ......................................................................................................... 41
On-site review of procedures. ........................................................................................................... 41
Review of documentation. ................................................................................................................ 42
Legal review ........................................................................................................................................... 42
Section 5. Panelist Recruitment .......................................................................................................... 43
Online Panel for Achievement Level Setting ....................................................................................... 43
Implementation. ................................................................................................................................ 43
Findings and Refinements ................................................................................................................... 46
In-Person Panel for Achievement Level Setting .................................................................................. 46
Implementation. ................................................................................................................................ 47
Selection Process .................................................................................................................................. 48
Findings and Refinements ................................................................................................................... 50
Cross-Grade Review Panel ................................................................................................................... 54
Implementation. ................................................................................................................................ 54
Findings and Refinements ................................................................................................................... 55
Section 6. Standard Setting Design and Implementation ................................................................... 57
The Bookmark Procedure ..................................................................................................................... 57
Creating ordered item booklets ....................................................................................................... 57
Item mapping. ................................................................................................................................... 59
Typical application of the bookmark procedure. ............................................................................. 59
Design and Implementation of the Online Panel ................................................................................ 60
Software development. ..................................................................................................................... 61
The home page.................................................................................................................................. 61
Online panel activities. ..................................................................................................................... 70
2
ALS Final Report

Conduct and results of the online panel. ........................................................................................ 70
Design and Implementation of the In-Person Workshop.................................................................... 71
Recruitment and selection of panelists ........................................................................................... 72
Preparation of materials ................................................................................................................... 72
Training of facilitators and table leaders ......................................................................................... 73
Orientation and training .................................................................................................................... 73
Round-by-round item review and discussion .................................................................................. 74
Data analysis and reporting ............................................................................................................. 78
Design and Implementation of the Cross-Grade Review Panel ......................................................... 78
Follow-Up Activities ............................................................................................................................... 80
Actions by Chiefs. .............................................................................................................................. 81
Review of ALDs. ................................................................................................................................. 82
Section 7. Field Test ............................................................................................................................ 84
Field Testing the Online Tool ................................................................................................................ 84
Preparations for field testing. ........................................................................................................... 84
Implementation. ................................................................................................................................ 85
Feedback. .......................................................................................................................................... 86
In-Person Workshop .............................................................................................................................. 89
Software support. .............................................................................................................................. 90
Vertical Articulation ............................................................................................................................... 91
Debriefing Session with Panelists ........................................................................................................ 91
Software issues. ................................................................................................................................ 92
Presentation/preparatory session usefulness ................................................................................ 92
Global observations .......................................................................................................................... 92
Debriefing Session with Smarter Balanced Staff ................................................................................ 93
Lessons Learned – Preparations for October ..................................................................................... 94
Section 8. Management Activities ....................................................................................................... 95
Meetings ................................................................................................................................................ 96
Weekly calls. ...................................................................................................................................... 96
Twice-monthly calls. .......................................................................................................................... 96
Collaborative Calls and Meetings ........................................................................................................ 96
Technical Advisory Committee ............................................................................................................. 96
Advisory Panel ....................................................................................................................................... 96
Other Meetings ...................................................................................................................................... 97
Reports .................................................................................................................................................. 97
References ............................................................................................................................................ 98
3
ALS Final Report

Appendix A – Communication Plan and Related Documents ............................................................. 99
Appendix B – Logistics Documentation ............................................................................................. 312
Appendix C – Audit Report and Related Documentation .................................................................. 314
Appendix D – Recruitment Documentation ....................................................................................... 315
Appendix E – Online Panel Documentation ...................................................................................... 316
Appendix F – In-Person Workshop Documentation ........................................................................... 317
Appendix G – Cross-Grade Review Panel Documentation ................................................................ 318
Appendix H – Follow-Up Activities ..................................................................................................... 319
Appendix I – Field Test Documentation ............................................................................................ 320

4
ALS Final Report

Executive Summary

Achievement level setting for Smarter Balanced assessments in Mathematics and English/Language
Arts and Literacy occurred in three phases: online panel, in-person workshop, and cross-grade
review. This report documents each of the three phases and provides results and recommendations
for cut scores for Smarter Balanced Assessments to be given in the spring of 2015.

Overview of Tasks and Processes
The purpose of the three activities was to derive cut scores using a subset of 2014 field test
assessment data and based on a transparent, scientifically rigorous, well documented procedure
that would stand up to external scrutiny. For this purpose, Smarter Balanced contracted with
Measurement Incorporated (MI) to devise and carry out such a plan; MI worked with McGraw-Hill
Education CTB to develop ALS design details. MI presented that plan at the Collaborative Meeting on
April 29, 2014, and it was unanimously approved. Essentials of the plan included the following:

• Recruitment of thousands of individuals in Smarter Balanced states to participate online
• Recruitment of 504 individuals in Smarter Balanced states to participate in an in-person
workshop
• Recruitment of 64 individuals from among the 504 to participate in cross-grade review
• Use of the Bookmark Procedure to guide participants through the achievement level setting
process
• Development of software to deliver assessments and achievement level setting tools to
online and in-person participants

Measurement Incorporated and McGraw-Hill Education CTB staff carried out all tasks under the
supervision of Smarter Balanced, conducting a pilot test of all activities August 14–20, the online
panel October 6–17, the in-person workshop October 13–19, and the cross-grade review on October
20, 2014. Results of the three October activities are provided below.

Online Panel Results
Over 10,000 individuals registered for the online panel and 2,660 participated by entering one cut
score—for Level 3—for one test. Participation by state is summarized in Table ES1, while participation
by role is summarized in Table ES2. Recommended cut scores are summarized in Table ES3.
Additional detail for participation as well as cut scores and impact is provided in the body of the
report.

5
ALS Final Report

Table ES1. Online Panel Participation by State.
Online Online Online
State Participants State Participants State Participants
California 459 Missouri 393 South Dakota 68
Connecticut 121 Montana 43 Virgin Islands 1
Delaware 48 North Carolina 10 Vermont 23
Hawaii 57 North Dakota 57 Washington 402
Iowa 15 New Hampshire 48 Wisconsin 132
Idaho 89 Nevada 40 West Virginia 116
Maine 75 Oregon 314 Wyoming 5
Michigan 69 Pennsylvania 4 Other 71
Total 2,660

Table ES2. Online Panel Participation by Role.
Online Online
Role Participants Role Participants
Higher Ed.
Teachers 1,730 Administrator 18
Non-Teacher Educators 209 Parents 65
Administrators 272 Business 14
Higher Ed. Faculty 187 Other 165
Total 2,660

6
ALS Final Report

Table ES3. Online Panel Cut Score Recommendations for Level 3
Test Cut Score % At or Above
Grade 3 English Language Arts/Literacy 411 46.8
Grade 3 Mathematics 409 53.7
Grade 4 English Language Arts/Literacy 468 42.9
Grade 4 Mathematics 474 42.7
Grade 5 English Language Arts/Literacy 453 62.9
Grade 5 Mathematics 520 36.5
Grade 6 English Language Arts/Literacy 516 46.9
Grade 6 Mathematics 540 37.8
Grade 7 English Language Arts/Literacy 510 54.9
Grade 7 Mathematics 584 27.0
Grade 8 English Language Arts/Literacy 551 47.8
Grade 8 Mathematics 632 17.7
Grade 11 English Language 54.9
544
Arts/Literacy
Grade 11 Mathematics 650 22.0
In-Person Workshop Results
The in-person workshop took place in Dallas October 13–17, 2014, in three waves: Grade 11 on
October 13–15, grades 6-8 on October 15–17, and grades 3–5 on October 17–19. There were 16
panels in all – one for each subject/grade combination in grades 3–8 and two each for grade 11
English language arts/literacy and mathematics. Dr. Michael Bunch (MI) oriented participants to the
7
ALS Final Report

Common Core State Standards and Smarter Balanced Achievement Level Descriptors. Participants
then took the Smarter Balanced online Practice Tests before receiving instruction from Dr. Daniel
Lewis (CTB) in the Bookmark Procedure. Contractor staff led Bookmark orientation exercises, and
then led participants through three rounds of review of ordered item booklets and discussion, during
which participants also reviewed results of the online panel. Table ES4 summarizes participation by
state, while Table ES5 summarizes participation by role. Results of the final round of cut score
recommendations (in scale score values) are summarized in Table ES6.

Table ES4. In-Person Workshop Participation by State.
In-Person In-Person In-Person
State Participants State Participants State Participants
California 66 Missouri 14 South Dakota 10
Connecticut 26 Montana 23 Virgin Islands 18
Delaware 24 North Carolina 8 Vermont 10
Hawaii 22 North Dakota 20 Washington 32
Iowa 17 New Hampshire 20 Wisconsin 28
Idaho 20 Nevada 15 West Virginia 31
Maine 29 Oregon 21 Wyoming 14
Michigan 14 Pennsylvania 0
Total 482

Table ES5. In-Person Workshop Participation by Role.
In-Person In-Person
Role Participants Role Participants
Educator: SWD
Educator: ELL Experience 36 31
Experience
Educator: General 242 General Public 28
Educator: Non-Teaching 89 Higher Education 56
Total 482
8
ALS Final Report

Table ES6. In-Person Workshop Cut Score Recommendations: Final Round
Level 2 % At or Level 3 % At or Level 4 % At or
Test Cut Above Cut Above Cut Above
Grade 3 English Language 362 66.5 427 40.1 485 19.1
Arts/Literacy
Grade 3 Mathematics 383 67.3 436 38.9 506 10.8
Grade 4 English Language 413 64.4 470 42.0 530 18.9
Arts/Literacy
Grade 4 Mathematics 400 77.6 470 44.7 541 15.6
Grade 5 English Language 406 78.6 450 63.9 574 16.9
Arts/Literacy
Grade 5 Mathematics 459 63.5 532 31.4 583 13.8
Grade 6 English Language 467 66.2 527 42.2 614 12.2
Arts/Literacy
Grade 6 Mathematics 491 58.3 563 28.7 604 15.3
Grade 7 English Language 474 68.3 547 40.1 660 6.6
Arts/Literacy
Grade 7 Mathematics 513 53.2 609 19.4 674 5.8
Grade 8 English Language 471 76.4 547 49.4 663 10.2
Arts/Literacy
Grade 8 Mathematics 534 51.3 605 25.6 683 7.4
Grade 11 English 490 72.7 565 47.4 677 11.6
Language Arts/Literacy
Grade 11 Mathematics 533 59.0 644 23.6 740 5.8

9