Validity of Educational Assessments in Chile and Latin America PDF
Preview Validity of Educational Assessments in Chile and Latin America
Jorge Manzi María Rosa García Sandy Taut Editors Validity of Educational Assessments in Chile and Latin America Validity of Educational Assessments in Chile and Latin America · · Jorge Manzi María Rosa García Sandy Taut Editors Validity of Educational Assessments in Chile and Latin America Editors JorgeManzi MaríaRosaGarcía PontificiaUniversidadCatólicadeChile PontificiaUniversidadCatólicadeChile Santiago,Chile Santiago,Chile SandyTaut MinistryofEducation Gunzenhausen,Germany ISBN978-3-030-78389-1 ISBN978-3-030-78390-7 (eBook) https://doi.org/10.1007/978-3-030-78390-7 JointlypublishedwithEdicionesUC TranslationfromtheSpanishlanguageedition:ValidezDeEvaluacionesEducacionalesEnChileYLati- noaméricabyJorgeManzi,MaríaRosaGarcíaandSandyTaut,©EdicionesUniversidadCatólicade Chile,2019.OriginalPublicationISBN978-956-14-2471-5.Allrightsreserved. PreviousEdition:TranslationfromtheSpanishlanguageedition:Validezdeevaluacioneseducacionales enChileyLatinoaméricabyJorgeManzi,etal.,©EdicionesUC2019.PublishedbyEdicionesUC.All RightsReserved. ©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature SwitzerlandAG2021 Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether thewholeorpartofthematerialisconcerned,specificallytherightsofreprinting,reuseofillustrations, recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionor informationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodologynowknownorhereafterdeveloped. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublishers,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishersnortheauthorsor theeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublishersremainneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Contents 1 Introduction ................................................. 1 JorgeManzi,MaríaRosaGarcía,andSandyTaut 2 Is Validation a Luxury or an Indispensable Asset forEducationalAssessmentSystems? ........................... 17 SandyTautandSiugminLay PartI ValidityofStudentLearningAssessments 3 HowtoEnsuretheValidityofNationalLearningAssessments? PriorityCriteriaforLatinAmericaandtheCaribbean ........... 37 MaríaJoséRamírezandGilbertA.Valverde 4 Contemporary Practices in the Curricular Validation of National Learning Assessments in Latin America: AComparativeStudyofCasesfromChile,Mexico,andPeru ...... 69 GilbertA.ValverdeandMaríaJoséRamírez 5 LearningProgressAssessmentSystem,SEPA:EvidenceofIts ReliabilityandValidity ........................................ 93 AndreaAbarzúaandJohanaContreras 6 Validation Processes of the National Assessment of Educational Achievements—Aristas: The Experience oftheINEEd(Uruguay) ....................................... 117 CarmenHaretcheandMarianoPalamidessi 7 TheValidityandSocialLegitimacyoftheChileanNational Assessment of Learning Outcomes (SIMCE): The Role ofEvidenceandDeliberation ................................... 145 LorenaMeckesandMaríaAngélicaMena v vi Contents PartII ValidityofInternationalAssessments 8 TestComparabilityandMeasurementValidityinEducational Assessment ................................................... 173 JorgeGonzálezandRenéGempp 9 MeasurementofFactorInvarianceinLarge-ScaleTests ........... 205 VíctorPedrero 10 InvarianceofSocioeconomicStatusScalesinInternational Studies ....................................................... 235 ErnestoTreviño,AndrésSandoval-Hernández,DanielMiranda, DavidRutkowski,andTylerMatta 11 Measuring Attitudes Towards Gender Equality inInternationalTests:ImplicationsforTheirValidity ............ 259 JuanCarlosCastillo,DanielMiranda,andAngélicaBonilla PartIII ValidityofAdmissionsandCertificationAssessments 12 Validity of Assessment Systems for Admissions andCertification .............................................. 283 MaríaVerónicaSantelices 13 IstheSABER11thTestValidasaCriterionforAdmission toColombianUniversities? ..................................... 307 JuliánP.Mariño,AdrianaMolina,andYadiraGómez 14 ValidityEvidenceoftheUniversityAdmissionTestsinChile: PruebadeSelecciónUniversitaria(PSU) ........................ 331 JorgeManziandDiegoCarrasco 15 Validity of the Single National Examination of Medical Knowledge(Eunacom) ......................................... 353 BeltránMena PartIV ValidityofTeacherEvaluations 16 TeacherEvaluationwithMultipleIndicators:Conceptual andMethodologicalConsiderationsRegardingValidity ........... 373 JoséFelipeMartínezandMaríaPazFernández 17 How Valid Are the Content Knowledge and Pedagogical ContentKnowledgeTestResultsoftheTeacherProfessional DevelopmentSysteminChile? .................................. 395 EdgarValencia,MarthaKluttig,andBeatrizRodríguez Contents vii 18 The Portfolio in the National Teacher Evaluation System in Chile: Collecting Evidence of Validity as Part oftheInstrumentConstructionProcess ......................... 431 DavidTorresIrribarraandÁlvaroZapata Chapter 1 Introduction JorgeManzi,MaríaRosaGarcía,andSandyTaut 1.1 ABookonValidity EducationalassessmentshavehadastrongdevelopmentinLatinAmericainrecent years.Almostallcountriesintheregionregularlyassesstheirstudents’achievement. Severalalsodosoforselectionpurposes—forexample,accesstohighereducation. Itisalsoincreasinglycommonthedevelopmentofassessmentsthataddressteacher performance.Moreover,internationalassessmentshaveshowngreatimpact,which hasledtoasustainedincreaseinthenumberofcountriesintheregionthatpartici- pateinandusePISA,TIMMSandERCEresults.Theresultsoftheseassessments, in addition to the direct use assigned to them according to the declared purposes (training,diagnosis,selection,certificationandpromotion),conformafundamental antecedent for judging the state of progress of education systems, as well as for makingcomparisonsbetweengroups(forexample,menversuswomenandsocioe- conomicandregionalgroups),andforestimatingtrendsineducationalachievement overtime.Asaresult,assessmentshavebecomecentraltoolsforeducationalpolicy, andstronglyinfluenceopinionsthattheeliteandcitizenshaveabouttheeducational system. Inthiscontext,assessmentsintheregionhavereachedacertainlevelofmaturity as have the development of technical capacities to design, implement and use the resultsofassessments.Atthesametime,theyhavebeengivenincreasinglybroader usesthathavegreaterimpactattheindividual,institutionalandnationallevels.Thus, B J.Manzi·M.R.García( ) PontificiaUniversidadCatólicadeChile,Santiago,Chile e-mail:[email protected] J.Manzi e-mail:[email protected] S.Taut MinistryofEducationofBavaria,Munich,Germany e-mail:[email protected] ©TheAuthor(s),underexclusivelicensetoSpringerNatureSwitzerlandAG2021 1 J.Manzietal.(eds.),ValidityofEducationalAssessmentsinChileandLatinAmerica, https://doi.org/10.1007/978-3-030-78390-7_1 2 J.Manzietal. thefundamentalquestionthisbookseekstoansweris:Istheinformationprovided bythesetestsandassessmentprogramsofsufficientqualitytomakeinterpretations andproposedusesvaluableanddefensible? Itisknownthatassessmentsshouldnotbeusedintheeducationalsphereifthere are no studies to support the validity of the interpretation of the scores, the use of suchtests,aswellasthepossibilityofmakingcomparisonsoftheirscoresovertime andbetweengroups.However,thereisaconsensusthat,inspiteoftherecognition giventovalidity,relativelyfeweducationalassessmentprogramshaveaccumulated sufficientevidencetosupporttheirinterpretationsanduses. The problem is that without validity evidence, it is not feasible to know how the scores or trends resulting from those assessments should be interpreted, nor is it possible to support the uses of those scores, even if they are explicitly reported orembodiedinalegalbody.Inshort,theabsenceofvalidityevidencerepresentsa seriousthreattoassessments,compromisingtheirvalueaswellastheirpoliticaland technicalviability. WithintheframeworkoftheStandardsforEducationalandPsychologicalTesting revision,publishedin2014(AmericanEducationalResearchAssociation[AERA] et al., 2014), this book addresses conceptual, methodological and applied aspects related to the efforts made in several countries to validate different types of large- scaleeducationalassessments.Theseeducationalassessmentsrangefromtraditional teststhatassessstudentlearningachievementtorecentassessmentsofnon-cognitive aspects, teacher assessments, as well as certification and selection tests. Addition- ally,thebookcompilestheexperienceofvaliditystudiesregardingthemaininterna- tionalprogramspresentinLatinAmerica(PISA,TIMSS,ERCE,ICCS).Finally,it revealsthechallengesthatmustbeconsideredwhenassessmentsareusedtocompare countries,groupsorachievementtrendsovertime. Thisbookmakesauniquecontributioninourregionduetoseveralreasons.One isthefactthatthisbookreflectsamorecontemporaryviewofvalidity.According to the Standards, validity refers to the amount of evidence that exists to support particularinterpretationsforthespecificproposedusesoftestscores.Thus,itisnot about establishing validity of an assessment in a general way, since what needs to bevalidatedistheinterpretationanduseofthescoresinagivencontextandfora specifiedpopulation. Consequently, as stated by Kane (2006), validation is a process that involves accumulating evidence that allows us to articulate (or refute) a validity argument. Validationbeginsbyexplicitlystatingtheintendedinterpretationsanduses.Inthis context,validityisabroadandflexibleconcept.Thebookillustratesthisaspectby includingchaptersthatmakeaconceptualcontributiontovalidity,asdescribedlater on,andotherchaptersthatreviewnotonlyempiricalevidenceoftestscoresbutalso theprocessofdevelopmentofassessmentandtheirconsequencesshowingthebroad meaningofvalidityasitiscurrentlyunderstood. A second contribution of this book is based on the fact that validity has been recognized as a requirement that should be asumed by those who decide to create and use assessment systems. The Standards are clear on this issue and state that test developers and those who decisions based on test results are responsible for 1 Introduction 3 conductingvalidationstudies.Therefore,itisessentialtotakevalidityintoaccount fromthemomentadecisionismadeinordertodevelopanassessmentprogram.This impliesthatresourceshavetobeallocatedtoprovidesufficientevidencetosupport orrefutetheinterpretationanduseoftheinstrumentsinvolved.Thisrequirementis especiallyrelevantinthecaseofassessmentsthathavehighimpactforindividuals, groups,orinstitutionsbasedontheresultsofeducationalassessments. Thirdly,thisbookillustrateshowdifferentcountrieshaveaddressedthecollection ofvalidityevidenceinvariouseducationalassessmentsystems.Inthisway,thisbook seekstoreaffirmtheneedfordecision-makerstoachievesufficientawarenessonthis matter, so that development and use of assessments, especially those with highest stakes,havesufficientsupportforthedecisionsthataremadebasedonthem. Wewillreviewthebook’smaincontributionsinmoredetailbelow,considering (i)conceptualcontributions;(ii)technicalcontributions;and(iii)chaptersthatillus- tratevalidationprocessesbasedonmultipleorspecificevidence.Thus,thedifferent chapterswillbecoveredaccordingtothecontributiontheymake.Thispresentation doesnotfollowthebook’sTableofContentsorder. 1.2 ConceptualContributions The book contains chapters that constitute important conceptual contributions regardingvalidityinparticularspheres.Thechapterof TautandLay,IsValidation a Luxury or an Indispensable Asset for Educational Assessment Systems? Initi- atesthisvolumewithaconceptualdiscussiononvalidityasitreviewsthehistoryof theconcept—consideringthecontributionsofmainreferents:Messick(1989,1994, 1995),Cronbach(1989)andKane(1992)—andestablishesthecurrentdefinitionof thisconstructaccordingtotheinternationalStandardsforEducationalandPsycho- logicalTesting(AERAetal.,2014).BasedontheStandards,itsummarizesthefive areasinwhichitispossibletocollectvalidityevidenceonassessmentsandarebased on (i) test content; (ii) cognitive processes; (iii) internal structure; (iv) relations to other variables; and (v) test consequences. Regarding this last source of evidence, theauthorsdiscussthecontroversythathasexistedamongthosewhoarguethatthe studyoftheconsequencesofassessmentsshouldnotbepartoftheconceptofvalidity (Borsboom et al., 2004; Cizek, 2012; Maguire et al., 1994; Popham, 1997; Wiley, 1991), and those who, in contrast, consider it relevant to study both intended and unintended consequences as part of validity, especially in high-stakes assessments for examinees (Kane, 2013; Lane & Stone, 2002; Lane et al., 1998; Linn, 1997; Messick,1989,1995;Shepard,1997). This chapter’s contribution also relies on explaining how these different pieces of evidence should be integrated into a line of argumentation on validity, and the differentactorsthatneedtobeinvolvedintheprocess.Finally,theauthorsanalyze the political dimension of validation, pointing out the main obstacles that need to be tackled when research, policy and practice converge in the development of a validationprogram.
The list of books you might like
