Table Of ContentAssessing Literacy in a Digital World
Yumei Zhang
Assessing Literacy
in a Digital World
Validating a Scenario-Based
Reading-to-Write Assessment
YumeiZhang
BeijingInternationalStudiesUniversity
Beijing,China
FundedbyBeijingInternationalStudiesUniversity2022AnnualAcademicWorkPublishing
Funding.
ISBN 978-981-19-5714-7 ISBN 978-981-19-5715-4 (eBook)
https://doi.org/10.1007/978-981-19-5715-4
JointlypublishedwithSocialSciencesAcademicPress
TheprinteditionisnotforsaleinChina(Mainland).CustomersfromChina(Mainland)pleaseorderthe
printbookfrom:SocialSciencesAcademicPress.
©SocialSciencesAcademicPress2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublishers,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishersnortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublishersremainneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Acknowledgements
Thisbookwouldnotbepossiblewithouttheassistanceofmanygenerousandincred-
iblysupportivepeople.ThehelpIhavereceivedalongtheprocesshasinspiredand
encouragedmeinmyacademiccareerandbeyond.
Firstofall,IwouldliketoexpressmygratitudetoProfessorShaoqianLuowho
has been a helpful advisor and a kind friend to me. I am thankful for her advice
for my study. I am also grateful for her trust in me and the opportunities she gave
me to participate in her projects from which I have learned more about language
educational assessment. I am also very grateful to Professor Wenxia Zhang, Lian
Zhang,BaochengHan,QiangWang,ZunminWu,XingweiMiao,XiaotangCheng,
andLiLin.Theiradvicehasguidedmeintheresearchdesignandwriting.
Apartfrommyteachers,therearesomanyotherpeoplewhohaveofferedgenerous
andtimelyhelpwhenIfeltconfusedandfrustrated.Theytouchedmetotearsand
inspired me to offer help to others as well. First, I want to thank my participants,
includingtheexperts,raters,thestudents,andtheirteachers,includingMs.Wenxin
Wu,Ms.XueyingYao,Ms.YujieJia,Ms.Wang,andMr.Jiang.Theirparticipation
made this work possible. I am greatly indebted to Mr. Jiangjiang Li. He was the
one who spent days helping me realize my assessment tasks online and revising
the tasks many times.Then I would like toshow gratitude to the researchers from
ETS,includingDr.ZuoweiWang,JohnSabatini,TenahaO’Reilly,andYiSong.My
first attempt at emailing their team brought me kind responses, helpful materials,
andconstructivesuggestions.Amongthem,Dr.Wanghasbeenofferingtremendous
helpduringthetaskdesignandofferingmeadviceondataanalyses.
TherearealsomanyothersIhaveconsultedwith.Dr.HaiyongZhao,LingZeng,
andLvnaYangprovidedmewithfeedbackontheveryfirstdraftofthetasks.Dr.Zhao
andZenghavealsohelpedmereadthefirstdraftandgiventheirfeedback.Besides,
I have also received help with data analyses from students from the psychology
departmentatBeijingNormalUniversity.
I would also like to thank Beijing International Studies University for the fund
thathelpedwiththepublishingofthisbook.Andthanksalsogototheanonymous
reviewers and my editors who played important roles in shaping this project to
v
vi Acknowledgements
completion and bringing this book to publication. They have provided insightful
suggestionsforfurtherimprovementstothisprojectanditspresentation.
Lastbutnotleast,Ialsofeelveryluckytohavemyfamilyandverygoodfriends
aroundme.Myfamilyhasbeenverysupportiveofmydecisions.Myparents’finan-
cial support for my academic trips abroad and for part of the study expenses has
mademystudyeasier.Thespiritualsupportfrommydearestfriendsisalsooneof
themainreasonsthatIcouldkeepthefaithandcarryon.Specialthanksgotomy
roommateLinlinChenwhogavemeverysincereadviceandhelpineveryway.
Contents
1 Introduction ................................................... 1
1.1 Background ............................................... 1
1.2 Purpose ................................................... 4
1.3 Significance ............................................... 4
1.4 LayoutoftheBook ......................................... 5
References ..................................................... 6
2 LiteratureReview .............................................. 9
2.1 Changes of Literacy Contexts Wrought by Information
Technology ................................................ 9
2.2 TheUseofComputerTechnologyinLanguageAssessment ...... 11
2.3 Scenario-BasedAssessment .................................. 12
2.3.1 DefinitionandKeyFeatures ........................... 12
2.3.2 MainConcernsTowardSBA ........................... 15
2.3.3 PreviousResearchonSBA ............................ 16
2.4 ValidityTheoriesandValidationMethods ...................... 21
2.4.1 TheEvolutionofValidityTheoriesandValidation
Frameworks ......................................... 21
2.4.2 Argument-BasedApproachtoValidation ................ 23
2.5 Reading-to-WriteLiteratePractice ............................ 31
2.5.1 TheoreticalFrameworks .............................. 32
2.5.2 PreviousStudiesontheReading-to-WriteAbility ......... 37
2.5.3 ArgumentationasReading-to-WritePractice ............. 40
2.6 SummaryoftheLiteratureReview ............................ 42
References ..................................................... 43
3 ResearchMethodology .......................................... 51
3.1 ResearchQuestions ......................................... 51
3.2 TheoreticalFramework ...................................... 52
3.3 Participants ................................................ 55
3.3.1 Experts ............................................. 55
3.3.2 StudentParticipants .................................. 55
vii
viii Contents
3.3.3 Raters .............................................. 57
3.4 Instruments ................................................ 57
3.4.1 TheSBRtWandRtWTasks ........................... 57
3.4.2 ExpertJudgmentGuideline ............................ 58
3.4.3 StimulatedRecalls ................................... 58
3.4.4 Questionnaire ....................................... 60
3.5 Procedures ................................................ 60
3.5.1 Stage1:StimulatedRecalls ............................ 61
3.5.2 Stage2:AQuasi-Experiment .......................... 62
3.5.3 Stage3:TheFinalAssessment ......................... 63
3.6 DataAnalyses ............................................. 64
3.6.1 AnalysesoftheStimulatedRecallData ................. 64
3.6.2 PreliminaryStatisticalAnalyses ........................ 65
3.6.3 RepeatedMeasuresANOVAandPairedSamplet-Tests .... 65
3.6.4 AnalysesBasedonGeneralizabilityTheory .............. 65
3.6.5 Many-FacetRaschMeasurement ....................... 67
References ..................................................... 69
4 TheDevelopmentoftheSBRtWAssessment ...................... 73
4.1 DevelopmentProcedureandGuidelines ....................... 73
4.2 PurposeandPotentialUseoftheTest .......................... 74
4.3 TheScenariosandtheFeatures ............................... 75
4.4 Topics .................................................... 77
4.5 ReadingTexts .............................................. 78
4.6 ItemTypesandFormats ..................................... 80
4.7 ScoringRubricsfortheWritingTasks ......................... 82
4.7.1 Bottom-Up Extraction of the Holistic Rubric
andAnalyticCriteria ................................. 82
4.7.2 RubricRefinementInformedbyTheoriesandPrevious
Rubrics ............................................. 84
4.7.3 RevisionBasedonExpertJudgmentandTrialScaling ..... 86
References ..................................................... 86
5 Results ........................................................ 89
5.1 AppraisaloftheScoringInference ............................ 89
5.1.1 TheAppropriatenessoftheScoringRule ................ 89
5.1.2 RatingAccuracyandRaterAgreement .................. 94
5.2 AppraisaloftheGeneralizationInference ...................... 98
5.2.1 TheRepresentativenessoftheObservationSample ....... 99
5.2.2 GeneralizabilityandBiasAnalyses ..................... 99
5.3 Appraisal of Extrapolation Inference and Theory-Based
Interpretations ............................................. 116
5.3.1 TheAuthenticityoftheSBRtWAssessmentTasks ........ 117
5.3.2 TheUnderlyingProcessesintheSBRtWAssessment ..... 118
5.3.3 TheUsefulnessoftheSBAFeaturesinFacilitating
Performance ........................................ 124
Contents ix
5.3.4 TheRelationshipBetweenReadingandWriting .......... 130
5.3.5 TheProgressivenessAcrossGradeLevels ............... 134
5.4 Summaryoftheresults ...................................... 134
References ..................................................... 136
6 Discussion ..................................................... 139
6.1 ValidityArgumentfortheSoringInference ..................... 139
6.1.1 The Importance of Scoring Rubric to Construct
Validity ............................................. 140
6.1.2 TheRoleofScoringReliabilityinValidation ............. 141
6.2 ValidityArgumentfortheGeneralizationInference .............. 144
6.2.1 TaskSamplingforGeneralization ...................... 144
6.2.2 TheEffectofTopicalIssues ........................... 145
6.3 Validity Argument for Extrapolation and Theory-Based
Interpretations ............................................. 146
6.3.1 TaskAuthenticityforValidity .......................... 146
6.3.2 UnderlyingProcessesasanIndicationofAttributes
atPlay .............................................. 148
6.3.3 TheEffectofScenario-BasedFeaturesonWriting
Performance ........................................ 149
6.3.4 The Role of Reading Comprehension in Writing
Performance ........................................ 150
6.3.5 VarianceofWritingPerformanceAcrossGradeLevels .... 151
References ..................................................... 152
7 Conclusions .................................................... 155
7.1 SummaryoftheMainFindings ............................... 155
7.2 Implications ............................................... 157
7.2.1 ImplicationsforFutureApplicationofSBA .............. 157
7.2.2 ImplicationsforDigitalLiteracyAssessment ............. 158
7.2.3 Implications for Technology Use in Language
Assessment ......................................... 158
7.3 LimitationsandSuggestionsforFutureResearch ................ 160
References ..................................................... 161
AppendixA: TheGuidelinesforExpertJudgments(Originally
inChinese) ........................................... 163
AppendixB: The Protocol for Stimulated Recall Procedure
(EnglishVersion) ..................................... 165
AppendixC: The Questionnaire (Form 1 as an Example,
OriginallyinChinese) ................................. 167
AppendixD: InformedConsent ..................................... 169
AppendixE: AnExampleoftheSBRtWAssessment ................. 171
x Contents
AppendixF: TheHolisticScalefortheWriting ...................... 177
AppendixG:TheAnalyticRatingScaleForWriting .................. 179
UncitedReferences ................................................ 185
