Opportunity to Learn, Curriculum Alignment and Test Preparation : A Research Review

140 Pages·2017·1.48 MB·English

by  Jaap Scheerens (eds.)

SPRINGER BRIEFS IN EDUCATION
Jaap Scheerens Editor
Opportunity to
Learn, Curriculum
Alignment and
Test Preparation
A Research Review
123
SpringerBriefs in Education
More information about this series at http://www.springer.com/series/8914
Jaap Scheerens
Editor
Opportunity to Learn,
Curriculum Alignment
and Test Preparation
A Research Review
With Contributions from:
Marloes Lamain
Hans Luyten
Peter Noort
123
Editor
JaapScheerens
Oberon Research andConsultancy
Utrecht
TheNetherlands
ISSN 2211-1921 ISSN 2211-193X (electronic)
SpringerBriefs inEducation
ISBN978-3-319-43109-3 ISBN978-3-319-43110-9 (eBook)
DOI 10.1007/978-3-319-43110-9
LibraryofCongressControlNumber:2016946940
©TheAuthor(s)2017
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
foranyerrorsoromissionsthatmayhavebeenmade.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAGSwitzerland
Acknowledgments
This study was carried out as cooperation between Oberon Research and
ConsultancyandtheDepartmentofResearchMethodology,MeasurementandData
Collection of the University of Twente. The work was supported by a grant from
NRO, the Netherlands Initiative for Educational Research in The Hague.
v
Contents
1 Focus and Design of the Review Study ... .... .... .... ..... .... 1
Jaap Scheerens
2 Conceptualization ... .... ..... .... .... .... .... .... ..... .... 7
Jaap Scheerens
3 Meta-Analyses and Descriptions of Illustrative Studies .. ..... .... 23
Jaap Scheerens
4 Review and “Vote Count” Analysis of OTL-Effect Studies .... .... 55
Marloes Lamain, Jaap Scheerens and Peter Noort
5 Predictive Power of OTL Measures in TIMSS and PISA ..... .... 103
Hans Luyten
6 Recapitalization, Implications for Educational Policy
and Practice and Future Research... .... .... .... .... ..... .... 121
Jaap Scheerens
vii
About the Book
Alignment between educational goals, intended and implemented curricula, and
educational outcomes is considered as a characteristic of effective education. The
expectation is that better alignment leads to better student performance. The con-
cept of opportunity to learn, abbreviated as OTL, is commonly used to compare
contentcovered,aspartoftheimplementedcurriculum, withstudentachievement.
Assuchitistobeseenasafacetofthebroaderconceptof“alignment.”Oneofthe
aims of this study is to further clarify these concepts, identify how they have been
used in research studies, and are employed in practice; the other major aim is to
assess the state of the art on OTL effects. This is done on the base of review of
meta-analyses, review of 51 research studies carried out during the last twenty
years,andasecondaryanalysisofTIMSSandPISAresults.Theresultsofthestudy
indicate a modest, but educationally significant effect size for OTL. Legitimate
forms of test preparation are seen as a promising approach to optimize OTL.
Enhancing the curricular validity of high-stakes tests and closer monitoring of
alignment chains are seen asrelevant interventions for educational policy aimed at
improving OTL.
ix
Chapter 1
Focus and Design of the Review Study
Jaap Scheerens
Abstract Alignment between educational goals, intended and implemented cur-
ricula and educational outcomes is considered as a characteristic of effective edu-
cation.Theexpectationisthatbetteralignmentleadstobetterstudentperformance.
The concept of Opportunity to Learn, abbreviated as OTL, is commonly used to
compare content covered, as part of the implemented curriculum, with student
achievement.Assuchitistobeseenasafacetofthebroaderconceptof“alignment”.
Oneoftheaimsofthisstudywastofurtherclarifytheseconcepts,identifyhowthey
havebeenusedinresearchstudies,andareemployedinpractice.Asecondmajoraim
ofthisstudywastoreviewthestateoftheartonresearchonOTLeffectsbymeansof
a search for meta-analyses and recent primary studies. The following research
questionswereaddressed:(1)Whichfacetsaretobedistinguishedinclarifyingthe
overall concept of OTL, and how are these to be placed as part of more general
modelsofeducationalalignmentandsystemicreform?(2)Whatistheaverageeffect
size of OTL (association of OTL with student achievement outcomes), as evident
from available meta-analyses, review studies, secondary analyses of international
data-setsand(recent)primaryresearchstudies?(3)Whataretheimplicationsofthe
resultson(1)and(2)foreducationalpolicyandpractice?
Study Aims and Research Questions
Alignment between educational goals, intended and implemented curricula and
educational outcomes is considered as a characteristic of effective education. The
expectationisthatbetteralignmentleadstobetterstudentperformance.Theconcept
J.Scheerens(&)
UniversityofTwente,Enschede,TheNetherlands
e-mail:[email protected]
J.Scheerens
Oberon,Utrecht,TheNetherlands
©TheAuthor(s)2017 1
J.Scheerens(ed.),OpportunitytoLearn,CurriculumAlignment
andTestPreparation,SpringerBriefsinEducation,
DOI10.1007/978-3-319-43110-9_1
2 J.Scheerens
ofOpportunitytoLearn,abbreviatedasOTL,iscommonlyusedtocomparecontent
covered,aspartoftheimplementedcurriculum,withstudentachievement.Assuchit
istobeseenasafacetofthebroaderconceptof“alignment”.Oneoftheaimsofthis
studyistofurtherclarifytheseconcepts,identifyhowtheyhavebeenusedinresearch
studies,andareemployedinpractice.Althoughopportunitytolearnwasoriginally
studied within the context of curriculum research, it has also obtained a place in
educational effectiveness research. Within this research orientation OTL is seen as
“aneffectivenessenhancingcondition”andcanbecomparedwithothersuchfactors
foritsinfluenceonstudentachievement.Asamatteroffact,resultsofmeta-analyses
would suggest that OTL has a relatively substantial average-effect size when it is
compared to other effectiveness enhancing conditions, such as learning time and
instructional leadership (Scheerens 2016). Yet, the number of meta-analyses and
reviewstudiesontheeffectsofOTLisratherlimited.Thisstudyseekstomakeastep
towards updating the state of the art, by means of a search for meta-analyses and
recentprimarystudies.
Several recent trends in the ongoing global efforts to improve the quality of
educationprovidefurtherperspectivetoassessingthestateoftheartonOTL,these
are alignment within the context of systemic reform, the accountability movement,
and task related cooperation between teachers.
Alignment Within the Context of Systemic Reform
In influential reports by the OECD and McKinsey the quality of educational sys-
tems is considered in systemic terms, as a whole of impulses and mechanisms at
system, school and classroom level (OECD 2010; McKinsey & Company 2010).
Alignmentbetweenlevelsinvariousfunctionaldomainsisakeyconceptinfinding
out why certain educational systems do better than others. The expectation is that
systems do better when aims, objectives, curricula and assessment programs are
well-aligned. The conceptualanalysisinthisreport,starting outfrom OTL intends
to further clarify the complexity of alignment between “curricular elements” and
opens up discussion on alternative interpretations, for example by comparing
proactive structuring and retroactive planning.
Accountability and Its Influence on Teaching
As indicated in the above, OTL originates from curriculum theory and research.
According to a pro-active logic, aims are operationalized to standards, worked out
asintendedcurricula,whichareexpectedtobeimplementedwithacertainfidelity,
and finally evaluated and assessed, by means of examinations and formative and
summative assessment. This is still a valid logic, although developments in the