Alternative model for the administration and analysis of research-based assessments

0.18 MB·

by  Bethany R. Wilcox

An alternative model for administration and analysis of research-based assessments
Bethany R. Wilcox,1 Benjamin M. Zwickl,2 Robert D. Hobbs,3
John M. Aiken,4 Nathan M. Welch,1 and H. J. Lewandowski1,5
1Department of Physics, University of Colorado, 390 UCB, Boulder, CO 80309
2School of Physics and Astronomy, Rochester Institute of Technology, Rochester, NY 14623
3Department of Physics, Bellevue College, 3000 Landerholm Cr SE, Bellevue, WA, 98007
4School of Physics, Georgia Institute of Technology, 830 State Street, Atlanta, GA 30332
5JILA, National Institute of Standards and Technology and University of Colorado, Boulder, CO 80309
Research-based assessments represent a valuable tool for both instructors and researchers inter-
6 ested in improving undergraduate physics education. However, the historical model for dissemi-
1 natingandpropagating conceptualandattitudinalassessments developedbythephysicseducation
0 research (PER) community has not resulted in widespread adoption of these assessments within
2 the broader community of physics instructors. Within this historical model, assessment developers
n create high quality, validated assessments, make them available for a wide range of instructors to
u use, and provide minimal (if any) support to assist with administration or analysis of the results.
J Here,wepresentanddiscussanalternativemodelforassessment dissemination,whichischaracter-
ized by centralized data collection and analysis. This model provides a greater degree of support
9
for both researchers and instructors in order to more explicitly support adoption of research-based

assessments. Specifically, we describe our experiences developing a centralized, automated system
]
h foranattitudinalassessment wepreviouslycreated toexaminestudents’epistemologies andexpec-
p tationsaboutexperimentalphysics. Thissystem providesaproof-of-concept that weusetodiscuss
- the advantages associated with centralized administration and data collection for research-based
d
assessments in PER. We also discuss the challenges that we encountered while developing, main-
e
. taining,andautomatingthissystem. Ultimately,wearguethatcentralizedadministrationanddata
s collection for standardized assessments is a viable and potentially advantageous alternative to the
c
default model characterized by decentralized administration and analysis. Moreover, with the help
i
s of online administration and automation, this model can support the long-term sustainability of
y
centralized assessment systems.
h
p
PACSnumbers: 01.40.Fk
[

3
I. INTRODUCTION interviewed 72 physics faculty on their use of various as-
v
6 sessmentstrategiesandfoundthatonly33%reportedus-
9 ingRBAsintheircourses. Toexplorethequestionofwhy
Oneofthesignificantcontributionsofthephysicsedu-
8 RBAs are not being adopted by a larger portion of the
cation research(PER) community has been the creation
7 broadercommunity of physics instructors,we draw from
of a number of research-based,standardizedassessments
0
literature on promoting and sustaining adoption of edu-
. (see Ref. [1] for an extensive list). Research-based as-
1 cational innovations. This work argues that, in addition
sessments (RBAs) include (but are not limited to) both
0 to designing innovations that meet the needs and goals
concept inventories and instruments targeting students’
6 ofpotentialusers,successfuladoptionofinnovations like
1 epistemologies. Theseassessmentsaretypicallybasedon
RBAs also depends on two distinct processes: “dissem-
: known student ideas, validated through expert and stu-
v ination” (i.e., spreading the word), and “propagation”
dentreview,pilottestedtodemonstratestatisticalvalid-
i (i.e., promoting sustained adoption) [8]. Dissemination
X ity and reliability, and administered for the purpose of
focuses on whatthe developerdoes whereaspropagation
r low-stakes formative assessment [2, 3]. RBAs provide a
a standardandvalidmeasureofstudentoutcomesthatcan focuses on the users of educational innovations [9].
be compared across semesters, instructors, institutions, Historically, RBA developers have focused almost ex-
and pedagogies. Such measures help to quantify the rel- clusivelyondisseminationthroughstrategiessuchaspre-
ativesuccessofeducationalstrategies,andthuscanhave sentations at professional conferences, publications in
significant impact on the initial adoption and continued peer-reviewed journals, and online access to the instru-
use of new innovations by individual instructors, as well ment for interested instructors [10, 11]. These historical
asattheprogrammaticlevel[4–6]. Additionaldiscussion dissemination strategies focus on informing physics in-
ofthe importanceandvalue ofassessment,including the structors that RBAs are useful and available. However,
use of RBAs, can be found in Refs. [4, 7]. beyond the creation of comprehensive “How to” guides
Despite the utility andpotentialimpactofRBAs,pre- and scoring resources (e.g., electronic scoring spread-
viousworksuggeststhattheseassessmentsarenotbeing sheets) [1, 12], RBA developers have rarely, if ever, ex-
utilized by undergraduate physics instructors as com- plicitly addressed issues of propagation. This failure to
monly as members of the PER assessment community explicitlyattendtopropagationisconsistentwithanim-
might expect or want. For example, Henderson et al. [4] plicit assumption that the only necessary ingredient for
2
a) b)
RBA Instructor Analysis Research RBA Central Administrator Research
Development Course 1 Course 1 Course 1 Development Data Aggregation Analysis Courses 1 to N
give RBA Courses 1 to N Courses 1 to N
student data
Students online access
to RBA
Course 1
course summary student data
information of results
Instructor (PER) Analysis Research
Course 2 Course 2 Course 2 Instructor Students
give RBA Course 1 give Course 1
student data RBA
Students Instructor link Students
Course 2 Course 2 Course 2
.
. .
.
. .
.
. .
Instructor Analysis Research
Course N Course N Course N Instructor Students
Course N Course N
give RBA
student data
Students
Course N
FIG. 1. a) A schematic representation of the default model for the propagation of RBAs that has been used historically in
PER. Here, light grey arrows denote rare or non-existent connections. This model depends almost entirely on the instructor
to drive theadministration and analysis of their students’results. There is also little to no mechanism for data from multiple
courses to be centrally aggregated or compared. b) A schematic representation of an alternative, centralized model for the
administration and analysis of RBAs. This model dependsalmost entirely on a centralized administration system to facilitate
administration and analyze students’ responses. This model also creates a centralized pool of comparison data that can be
used for research and analysis purposes.
the sustained adoption of RBAs is to make physics in- this process (e.g., information on best practices for giv-
structorsawareoftheirexistenceandvalue[5,8,10]. The ingtests,toolstoautomatethecollectionandanalysisof
resultofthisassumptionisadefaultmodelforthepropa- RBA data).
gationofRBAs. Inthismodel,individualinstructorsare Challenges that instructors encounter when admin-
solely responsible for administering the RBA, as well as istering and analyzing RBAs are not meaningfully ac-
analyzing and interpreting their students’ performance. knowledgedinthe default modelRBA propagation(Fig.
This process is typically done with minimal, if any, di- 1a). Instead, it is assumed that individual instructors
rectsupportfromtheassessmentdeveloper. Aschematic will successfully administer, score, and analyze these as-
representationof this default propagationmodelis given sessments independently, despite the fact that many in-
in Fig. 1a. structors report encountering difficulties in this process.
The default modelforRBA administrationandanaly- Moreover,Madsen et al. [7] found that instructors often
sis fails to accountfor important factors that can poten- face additional hurdles as they attempt to productively
tially impact successful propagation. For example, the interpret RBA data from their course (e.g., interpreting
dynamics of the broader instructional system, which in- results in courses with few students, identifying action-
cludes the instructor, the department, and the institu- ableimplicationsforteaching). Inparticular,instructors
tion,canencourageordiscourageinstructorsfromutiliz- areofteninterestedinhowtheirstudents’responsescom-
ingRBAs[8]. Tobetter understandtheseadditionalfac- pare to those of students in similar courses nationally
torsthatcontributetoinstructors’use(ornot)ofRBAs, [7]. However, collection and analysis of RBA data has
Madsenet al. [7]interviewedphysicsfacultyanddepart- historically been localized to individual faculty, thus ap-
ment heads who were familiar with RBAs about their propriate comparisondata is often difficult to find. This
experiences with, and perceptions of, these instruments. lack of any clear mechanism for accessing and/or aggre-
They identified a variety of issues around faculty use of gating comparison data severely limits one of the main
RBAs, not all of which can be overcome through im- affordancesofRBAs– providingastandardizedmeasure
proved propagation strategies (e.g., misalignment of the of student learning across classroom formats, curricular
RBA with faculty learning goals). However, lack of sup- variations, pedagogical strategies, and institutions.
port during the process of utilizing RBAs and interpret- Additionally, the localization of the collection and
ingtheresultswascitedasabarrierbymanyinstructors analysis of RBA data to individual faculty in the de-
[7]. In particular, instructors encounter challenges with fault model for the propagationof RBAs also has conse-
the logistics of administering RBAs and want help with quencesfor PERcommunity. Unless the instructoris,or
3
iscollaboratingwith,aphysicseducationresearcherwith structor summarizing their students’ responses relative
Institutional Review Board approval to conduct human to similar courses in the data set.
subjects research, students’ responses are rarely made This centralized model for the propagation of RBAs
availableforresearchpurposes. Incaseswheredatafrom (Fig.1b)addressesmanyofthefactorsidentifiedbyMad-
RBAs are used for research purposes, the resulting re- senet al. [7]asbarrierstoinstructorsinterestedinusing
searchisoftenhighlycontextualizedtoasinglecourseor RBAs. The centralized administrator not only helps to
institution. Additionally, researchcontexts areoften not reduce the burden on individual instructors by dealing
captured and reported in a systematic or consistent way withmanyofthelogisticalaspectsofadministeringRBA,
across multiple studies. This contextualization, along but also provides a concrete point of contact who can
with the lack of standardized administration, analysis, respond to instructor questions and provide targeted re-
and reporting of results from RBAs, can make it more sources. Additionally,byaggregatingdatafrommultiple
difficultforresearcherstoidentifyandprovidevalidcom- coursesandtaggingthem withcourse-specificmeta-data
parison data from similar courses. Significant contextu- (e.g., class size, level, type of instruction), the central
alization also places significant limitations on the types administrator can provide appropriate comparison data
ofquestionsthatcanbe tackledbyresearchers. Without that can help an instructor determine how their course
centralized data from multiple courses and institutions compares to similar courses. This aggregate data can
along with standardized information about course con- also be used by the administrator for research purposes,
text, it often becomes logistically impractical to answer as well as to feed back into improving the analysis that
questions about, for example, transferability of effective is ultimately reported to instructors.
coursetransformations,orgeneralizabilityofstudentdif-
ficulties.
In recent years, several groups in the PER commu- A. An example from the E-CLASS
nity have made explicit attempts to address the chal-
lenges inherent in the default model for the propagation Tomoreclearlyillustrate this centralizedRBAmodel,
of RBAs through analternative model that is character- this section presents a concrete example of this model
ized by centralized administration, data collection, and in practice. This example centers around a newly auto-
analysis. Examples of this kind of centralized adminis- mated administration system for the Colorado Learning
tration of standardized assessments can also be found in AttitudesaboutScienceSurveyforExperimentalPhysics
several other disciplines [13–15] and contexts [16]. The (E-CLASS) to multiple laboratory courses. Note that
goal of the remainder of this paper will be to provide a while this section includes some backgroundinformation
proof-of-concept for this centralized RBA model within ofthe E-CLASSitself, the focus here is onthe E-CLASS
PER by presenting, in detail, the design and implemen- administration system. Discussion of the E-CLASS as a
tationofanewlylaunched,automatedonlinesystemcre- validassessmentinstrumenthasbeenreportedpreviously
ated to facilitate the administration and analysis of the (see Refs [17, 18]). The goal of this section is to provide
Colorado Learning Attitudes about Science Survey for sufficientdetailonthehistory,development,andmainte-
Experimental Physics [17] (E-CLASS, Sec. IIA). Addi- nance of the E-CLASS system both to demonstrate the
tionally, we also briefly discuss related efforts by several model and to provide a base from which interested de-
otherresearchteams(Sec.IIC),andendwithdiscussion velopers of RBAs could replicate or build off this model
of the characteristics,advantages, and challenges associ- in the future.
ated with a centralized administration and data collec-
tion model for the propagation of RBAs in PER (Sec.
III). 1. The E-CLASS assessment
The E-CLASS was developed by researchers at the
II. AN ALTERNATIVE MODEL University ofColoradoBoulder(CU) [17] to support on-
going, local and national initiatives to improve labora-
To address many of the shortcomings of the default tory instruction[19]. The E-CLASSis a 30 item, Likert-
model for the propagation of RBAs, we propose an al- stylesurveydesignedtomeasurestudents’epistemologies
ternative model that localizes administration, data col- and expectations regarding the nature of experimental
lection, and analysis to a single centralized system (Fig. physics. Items on the E-CLASS feature a paired ques-
1b). In this model, instructors interested in a particular tion structure in which students are presented with a
RBAprovideinformationabouttheir courseto acentral statement (e.g., “Calculating uncertainties helps me un-
administrator. The administrator then provides online derstandmyresultsbetter.”) andareaskedtoratetheir
access to the RBA for the instructor and their students. level of agreement on a 5-point Likert scale both from
Students’responsestotheRBAfromallcoursesarethen their personalperspective and that of a hypothetical ex-
collected and analyzed directly by this central adminis- perimental physicist.
trator. The administrator uses the aggregate data from The E-CLASS is generally administered online and
all courses to produce individualized reports for each in- typically outside of class time. Instructors have his-
4
torically been recruited to use the E-CLASS with their ministered since its initial development in fall of 2012.
students through a variety of means including, personal Prior to the development of the automated system, sur-
communication,emailstoprofessionalemaillists,presen- vey generation, management, and the report generation
tations, and publications [17, 20, 21]. Informationabout was done by-hand by a part-time administrator. While
the instrument is also available online [22]. The instru- managingthesystembyhandwassuccessfulintheshort
ment was initially developed and validated at CU, but term, the personnel requirements of this method likely
has now been validated using data from more than 70 make it unsustainable over long time scales.
courses at roughly 45 institutions [18].
3. Developing and maintaining the E-CLASS system
2. The E-CLASS system
The most demanding aspect of the E-CLASS system,
To administer the E-CLASS via our automated sys- bothfinanciallyandintermsofexpertise,wasdeveloping
tem, an interested instructor first completes our Course and implementing the automation. As indicated previ-
Information Survey (CIS). Questions on the CIS collect ously, automation is not strictly necessary in order to
logistical information (e.g., course start date, number runacentralizedassessmentsystem,atleastintheshort
of student enrolled, department demographics), as well term. TheE-CLASSwascentrallyadministeredformore
as information about course learning goals, equipment, than5semestersbeforethesystemwasautomated. How-
andpedagogy. Logisticalinformationfromthe CISfeeds ever, operating the system without automation required
intotheautomatedsystemdescribedinSec.IIA3where adedicatedadministrativeassistantwhowasresponsible
responses to particular questions are used to generate for generating, managing, and analyzing pre- and post-
unique pre- and post-survey links for each course. The surveys for 40-60 courses each semester. The automated
system then emails these links to the instructor at the system has reduced the administrative requirements to
beginning and end of the course respectively. Each link checking the system periodically to ensure it is function-
isactivatedanddeactivatedautomaticallybythesystem ing and responding to email requests from participating
based on dates selected by the instructor on the CIS. instructors. Thisreductionintheneedfordedicatedper-
In order to utilize student responses to the E-CLASS sonnel greatly increases the sustainability of this system
for research purposes, we have approval from the CU in future semesters when grant funding for the develop-
Institutional Review Board to conduct human subjects ment of the E-CLASS has ended.
research using this online system. Consistent with the The program that governs the automation of the E-
requirements of that approval, the survey link also in- CLASSsystemwaswrittenintheprogramminglanguage
cludes a consent form notifying students that data from Python [23]. Python was selected over a more formal
the assessment may be used for research purposes and relational database [24] for several reasons. Python is
providinginstructionsforhowtowithdrawparticipation. well known in physics, and many physicists have a basic
Afterthepre-andpost-instructionsurveyshaveclosed, knowledge of Python or a similar coding language. This
thesystemautomaticallyemailstheinstructorwithalist fact increases the sustainability of the program as fu-
of the names and ID numbers of all students who com- ture personnelwill needto be familiar withboth physics
pleted each survey, which the instructor can use to pro- content and basic Python programming. Python also
vide a small amount of participation credit. Due to con- includes anumber ofpowerfulstatisticalandrepresenta-
straints around performing human subjects research,in- tional packages that can facilitate data analysis [25–28].
structorsneverreceiveidentifiable,rawstudentresponses One of the authors, an undergraduate computer science
to the survey. Instead, student responsesare fed into re- major familiar with the Python programming language,
portgenerationsoftware,whichcreatesanindividual,on- was hired to help design and then write the code for the
linereportforeachcourse. Theseaggregatereportssum- surveyautomation. TheE-CLASSsurveysarehostedon
marize students’ responses to the E-CLASS both over- a commercial survey platform known as Qualtrics [29].
all and by-item, and also include comparison statistics Qualtrics is CU’s official survey platform, and the uni-
pulled from our growing data set composed of student versity provides access to this service for all faculty and
responses from previous semesters. While generally in- staff. The E-CLASS automation was designed to inter-
tended to be shared between instructors, course reports face with the Qualtrics application programming inter-
are only accessible via a unique and idiosyncratic link face (API) in order to allow the system to automatically
delivereddirectlytotheindividualinstructor,soanindi- generate, activate, deactivate, and pull results from sur-
vidual instructor can choose to keep the report for their veys on Qualtrics. Additionally, hosting the surveys on
class private. Qualtrics maintains the security of the E-CLASS items
As of the fall semester 2015,the E-CLASS system has as Qualtrics surveys are only accessible via their unique
been almost fully automated, requiring only minimal in- survey links and cannot be accessed by Googling “E-
putfromahumanadministratorfromtheinitialcomple- CLASS” or the question prompts.
tion of the CIS through to the distribution of the final In addition to the undergraduate student who pro-
reports. However, the E-CLASS has been centrally ad- duced the code for the automation, one of the authors
5
was brought on as an outside contractor with both PER inresponseto commentsfrominstructorsthatsuggested
and programming experience to create the report gener- theoriginalorderingconventionmadeitdifficulttocom-
ation software, also written in Python. The report gen- pare graphs across distinct courses and/or semesters.
erator takes students’ raw responses from Qualtrics and Severalinstructorsalsorequestedthatthereportinclude
exportsastaticHTMLreportforeachcourse,whichcan documentationoftheaccepted“expert-like”responsefor
beviewedandsharedviatheweborprinted. Reportsin- each question. This information has now been incorpo-
clude aggregatestatistics on the overall E-CLASS score, rated into a separate “Questions” tab in the online re-
average item scores, and student demographic data for ports.
both the course and similar-level courses from previous
semesters, as well as a brief description of how the data
were analyzed. The decision to generate static, rather
than dynamic, reports was made primarily to enhance
B. Initial impacts of the E-CLASS system
their sustainability. A web application capable of gener-
ating dynamic reports with continuously updating com-
parison data or interactive graphs would require real- Inthissection,wediscusswaysinwhichthecentralized
time developer support should the report system ever administration system for the E-CLASS has supported
encounter a problem. Alternatively, static reports, once researchefforts and faculty use of the E-CLASS.
generated, can be made available indefinitely and would
notbe affectedshouldthe reportgeneratorencounteran
unforeseen problem. The automation, report generator,
and raw student responses are all stored and run on a
1. Research with growing national dataset
local,dedicatedandsecureserverpurchasedforthispur-
pose. Local university IT personnel assisted with the
process of purchasing, setting up, and maintaining the Over the previous six semesters during which the E-
E-CLASS server. CLASS has been centrally administered, we have col-
As the major deliverable from the E-CLASS system, lected over 5200 matched pre- and post-instruction stu-
thefinalreportshavegonethroughmultiple iterationsof dent responses from 120 distinct courses spanning 64 in-
review and refinement to ensure that they are clear and stitutions across the United States. These courses in-
useful to instructors. An example E-CLASS report can clude introductory labs for non-majors up through ad-
be accessed from Ref. [22]. To investigate instructors’ vancedlaboratorycoursesforseniorphysicsmajors. The
perceptionsofthe reports,wesolicitedfeedbackfromin- existence of such an extensive data set provides oppor-
structorswhohadreceivedE-CLASSreportsinprevious tunities for addressinga multitude of researchquestions.
semesters. Nine of these instructors also participated in For example, we previously used these data to robustly
phoneinterviewsinwhichtheywereaskedtodiscussany establish the statistical validity and reliability of the in-
positive,negative,and/orconfusingaspectsofthereport. strument for a much broader student population than is
Interviewswereconductedbyoneoftheauthorsandtyp- typically possible for RBAs [18]. Additionally, ongoing
icallylastedbetween15-30minutes. Allnineinterviewees research includes analysis of these data with respect to
madepositivecommentsabouttheE-CLASSand/orthe gender differences in E-CLASS scores [30]. Using demo-
final report. For example, a representative quote from graphicdataonthestudentsalongwithmeta-dataonthe
one of the interviewees was, “The feedback that I get course, we are able to examine gender differences while
about the statistics about my students is very well laid controlling for confounding variables (e.g., student ma-
out. Thereisaverygoodexplanationatthebeginningon jor,courselevel). Moreover,thesizeoftheavailabledata
how to look at the data and interpret the graphs, and it setallowsforexaminationofthe intersectionalimpactof
iswellorganizedandsystematicandeasytofollow.” The different variables while still maintaining significant sta-
presentationofindividualcoursescoresside-by-sidewith tistical power.
significantcomparisondatafromothercoursesnationally
Future work with this growing, national data set will
was also mentioned as one of the major useful features
utilize course meta-data collected by the automated sys-
oftheE-CLASSreport. Additionally,intervieweesnoted
tem to investigate the impact of differences in peda-
thatthereportscontainedalargeamountofinformation
gogy and laboratory structure on students’ epistemolo-
andthatthevisualrepresentationsofstudentsscoresand
gies about experimental physics. Aggregation of data
shifts helped to facilitate interpretation.
frommultiple coursesovermultiple semestersatCUand
Despitetheinstructors’overallpositivefeedbackabout a few other institutions can also be used to examine
the report, they made a number of suggestions for im- changes in students’ epistemologies over time using lon-
provement or clarification. Many of these suggestions gitudinal and pseudo-longitudinal data. Ultimately, this
have since been incorporated into the report generation data set may help to provide concrete suggestions for
software. For example, the ordering convention for data both instructors and researchersinterestedin the nature
points in several of the graphs was modified to support of students’ epistemologies about experimental physics
greaterconsistencybetweenindividualreports. Thiswas and how they change over time.
6
2. Faculty use of the E-CLASS results dents’responsestoavarietyofRBAs. TheDataExplorer
is not designed to be a fully centralized assessment sys-
temasitdoesnotassistwithadministrationordatacol-
Overall, roughly 90 physics instructors have received
lection. Rather, this system represents a halfway point
one or more E-CLASS reports over the past five
betweenthe defaultmodelandafully centralizedmodel.
semesters. To better understand how these faculty per-
In this halfway model, data collection is still primarily
ceive and use these reports, we distributed a short email
in the hands of the instructors, but the system will pro-
survey to a subset of these instructors (N=55) and re-
videamechanismforaggregatingcentralizedcomparison
ceived 17 responses. We also performed follow-up inter-
viewswith9oftheseinstructorstogainadditionaldetail dataandoffertoolsforstandardizedanalysis. Unlikethe
E-CLASSsystem,where the analysisfunctions generally
regarding their responses to the survey. A significant
themefromtheseinterviewswasrelatedtotheperceived require that the data were collected directly by the sys-
tem,theDataExplorerwillalsoprovidethe opportunity
valueofhavingsignificantandappropriatenationaldata
withwhichtocompareindividualcourseresults. Instruc- forinstructorstoanalyzedatafromsupportedRBAsthat
they have already collected from their courses.
tors noted that this comparisondata helped them to de-
termine what questions and trends to pay attention to Recently, the Learning Assistant (LA) Alliance’s has
when interpreting their students’ performance. also created a new centralized, online assessment sys-
The surveys and interviews also showed that the E- tem to support their LASSO study [33]. Like the Phys-
CLASS reports have been used by instructors in multi- Port Data Explorer, the LASSO system offers a range
ple productive ways. For example, 9 of the 17 survey of RBAs; however, similar to the E-CLASS system, the
respondents reported using data from their report to in- LASSO system both hosts the RBAs and manages data
formspecificchangestotheirlabcourses,suchasrevised collection,inadditiontofacilitatingscoringandanalysis.
lab manuals, introduction of reflective assignments, and TousetheLASSOsystem,aninstructordoesnotneedto
changesinthegradingstructuretoexplicitlytargetprac- uselearningassistantsintheircourse,buttheymustjoin
tices such as making predictions and basing conclusions the LA Alliance (which is free). Participating instruc-
on data. In several cases, these changes were made in tors register their course with the LASSO system and
the context of larger scale laboratorycourse transforma- uploada courserosterwith email addressesfor eachstu-
tion efforts involving multiple courses and instructors. dent. The system then emails a unique link to the RBA
Additionally, 13 of 17 survey respondents reported dis- to each student on the roster. After administering the
cussing their E-CLASSreportwith colleaguesordepart- post-test, instructors also receive a summary document
ment chairs, and roughly a third of these respondents reporting their students’ pre- and post-test distributions
(N = 4) included their results in presentations at lo- and averages.
cal or national professionalmeetings. Two faculty mem- One consistent difference between the E-CLASS sys-
bers also included their E-CLASS reportas part of their tem and that of the Data Explorer and LASSO is that
tenure or post-tenure evaluation. These findings sug- both the Data Explorer and LASSO are designed as a
gest that the E-CLASS and its associated report have hub for multiple RBAs, whereas the E-CLASS system
impacted faculty and their laboratory teaching practice offers only the E-CLASS. RBA developers interested in
in multiple ways across multiple institutions. Moreover, utilizing a centralized assessment model for new or ex-
withtheincreasedsustainabilityaffordedbytheautoma- isting RBAs will be able to decide if they wish to create
tion of both the administration and report generation a new system for their assessment or incorporate it into
process, the E-CLASS system can continue to serve as a an existing, multiple-RBA system. Creating a new sys-
resource for new and returning instructors. tem affords a greater degree of control (e.g., access to
raw data) and customizability (e.g., customized instruc-
tor reports), but also requires significant investment of
C. Other centralized assessment systems time and funding to create and maintain the system.
Utilizing an existing multi-RBA system eliminates the
burdenofdevelopingandmaintaininganewsystem,but
In addition to the E-CLASS, there are several other
potentially affords less opportunity to customize the ad-
systems that exemplify some or all aspects of a cen-
ministrationandreportingforthe particularassessment.
tralized model of RBAs (Fig. 1b), including the Phys-
Port Data Explorer [31] and the Learning Assistant Al-
liance’sLearningAssistantSupportedStudentOutcomes
(LASSO) system [32]. While a comprehensive compari- III. DISCUSSION
son of the E-CLASS, PhysPort, and LASSO systems is
beyond the scope of this work, this section briefly de- The E-CLASS system, along with the PhysPort Data
scribes key feature of these two system. Explorer and the LA Alliance’s LASSO system, demon-
The PhysPort Data Explorer is designed to be a tool strates that a centralized model for the dissemination
for scoring, analyzing, and interpreting results of RBAs of RBAs represents a viable and potentially sustainable
[31]. The system is currently in the open beta-testing alternative to the passive dissemination model that has
phase,allowingindividualinstructorstouploadtheirstu- been used historically in PER. Centralized assessment
7
systems have two primary advantages over this default zation. For the E-CLASS system, the majority of this
model. The first advantage is increased instructor sup- funding was used to cover the initial development costs
port for administration, analysis and interpretation of ofthe system,including hardware,software,andperson-
results. Particularly since RBAs are typically given at nel. However, even for automated systems, some min-
the beginning and end of the course, it can be very diffi- imal funding to support a part-time administrator will
cultforbusyinstructorstofindthetimeandresourcesto likely be necessary to respond to user issues and main-
administer an extra assessment. By managing the sur- tain the system indefinitely. (Note that this funding was
vey generation and data collection process, centralized in addition to the initial funding required to supportde-
systems greatlyreduce the burdenplacedoninstructors. velopment and validation of the assessment.) Another
Moreover, particularly for instructors with minimal ex- potential limitation of the centralized assessment sys-
perience with RBAs, the analysis and representation of tems discussed here is that they have been tested only
RBA data to facilitate interpretation of the results can with multiple-choice assessments. Indeed, the adminis-
alsopresentasignificantbarrier. Byprovidingstandard- trative viability of these systems is arguably dependent
ized analysis and reporting, centralized assessment sys- onthesystembeingabletoscorestudentsresponseselec-
tems providea mechanismfor instructors to take advan- tronically. The ability to consistently and reliably score
tage of the collective expertise of the PER community open-endedquestionselectronicallyhasyettobedemon-
with respect to analyzing, representing, and presenting strated within PER. Thus, at the moment, centralized
the results of RBAs, rather than having to develop this assessment systems may be feasible only for RBAs with
expertise for themselves. forced-choice question formats.
The second major advantage of a centralized assess- The E-CLASS and other similar systems have now
ment model is the potential for aggregating large data demonstratedthatamodelfordisseminationofresearch-
sets of responses from a more diverse population of stu- based assessments that centralizes administration, data
dents, institutions, and courses. These large-scale data collection,andanalysisrepresentsa viable alternativeto
sets can then be used both to provide comparison data the default, historical model. We argue that this model
and for research purposes. Comparison data is a key hassignificantadvantagesandrecommendthatmembers
tool for instructorstrying to productively interprettheir of the PER community give legitimate consideration to
students’ performance. Without such comparison data, this alternative model when developing and disseminat-
it can be difficult for instructors to make clear and ac- ing research-based assessments in the future. Research-
tionable conclusions about their students’ performance. based assessments are an important component of fur-
National data sets from RBAs also represent a valuable thering efforts to improve physics education, and cen-
resource for physics education researchers interested in tralized assessment systems have the potential to facili-
broader research questions related to student learning tatetheuseoftheseassessmentsforbothinstructorsand
outcomes, particularly when students’ responses are ac- physics education researchers.
companiedby detailed informationaboutcourse context
andpedagogylikethatcollectedontheE-CLASSCourse
Information Survey.
Despite the clear advantagesoffered by centralizedas-
ACKNOWLEDGMENTS
sessment systems, these systems also have a number of
drawbacks and limitations. For example, the develop-
ment of these systems is time consuming and expensive, This work was funded by the NSF-IUSE Grant DUE-
and they require some degree of maintenance. Each of 1432204 and NSF Grant PHY-1125844. The statements
the systems described in this paper (E-CLASS, LASSO, madeinthisarticledonotnecessarilyrepresenttheviews
and Data Explorer) had external funding from the Na- oftheNSF.ParticularthankstoTakakoHirokawaandto
tionalScience Foundation (NSF) and/oranotherorgani- the members of PER@C for all their help and feedback.
[1] https://www.physport.org/assessments/, (2015). and Tricia Chapman, “Assessment of teaching ef-
[2] WendyKAdamsandCarlEWieman,“Developmentand fectiveness: Lack of alignment between instruc-
validation of instruments to measure learning of expert- tors, institutions, and research recommendations,”
like thinking,” International Journal of Science Educa- Phys. Rev.ST Phys.Educ. Res. 10, 010106 (2014).
tion 33, 1289–1312 (2011). [5] Elaine Seymour, “Tracking the processes of
[3] Bethany R. Wilcox, Marcos D. Caballero, Charles change in us undergraduate education in sci-
Baily, Homeyra Sadaghiani, Stephanie V. Chasteen, ence, mathematics, engineering, and technology,”
Qing X. Ryan, and Steven J. Pollock, “Develop- Science Education 86, 79–105 (2002).
mentandusesofupper-divisionconceptualassessments,” [6] Stephanie V. Chasteen, Bethany Wilcox, Marcos D.
Phys.Rev.ST Phys. Educ.Res. 11, 020115 (2015). Caballero, Katherine K. Perkins, Steven J. Pol-
[4] Charles Henderson, Chandra Turpen, Melissa Dancy, lock, and Carl E. Wieman, “Educational transfor-
8
mation in upper-division physics: The science educa- initial results,” Physical Review Special Topics-Physics
tion initiative model, outcomes, and lessons learned,” Education Research 10, 010120 (2014).
Phys.Rev.ST Phys. Educ.Res. 11, 020110 (2015). [18] Bethany R. Wilcox and H. J. Lewandowski,
[7] Adrian Madsen, Sarah B. McKagan, Mathew Sandy “Students’ epistemologies about experimental
Martinuk, Alexander Bell, and Eleanor C. physics: Validating the colorado learning attitudes
Sayre, “Research-based assessment affordances about science survey for experimental physics,”
and constraints: Perceptions of physics faculty,” Phys. Rev.Phys. Educ.Res. 12, 010123 (2016).
Phys.Rev.Phys. Educ. Res.12, 010115 (2016). [19] Benjamin M Zwickl, Noah Finkelstein, and
[8] Courtney Stanford, Ren´ee Cole, Jeff Froyd, Debra HJ Lewandowski, “The process of transforming an
Friedrichsen, Raina Khatri, and Charles Henderson, advanced lab course: Goals, curriculum, and as-
“Supporting sustained adoption of education innova- sessments,” American Journal of Physics 81, 63–70
tions: The designing for sustained adoption assessment (2013).
instrument,” International Journal of STEM Education [20] Benjamin Zwickl, Noah Finkelstein, and H. J.
3, 1 (2016). Lewandowski, “Development and validation of the col-
[9] RainaKhatri,CharlesHenderson,Ren´eeCole,JeffreyE. orado learningattitudesabout sciencesurveyfor experi-
Froyd,DebraFriedrichsen, andCourtneyStanford,“De- mental physics,” in Physics Education Research Confer-
signing for sustained adoption: A model of develop- ence2012,PERConference,Vol.1513(Philadelphia,PA,
ing educational innovations for successful propagation,” 2012) pp.442–445.
Phys.Rev.Phys. Educ. Res.12, 010112 (2016). [21] Bethany Wilcox and H. J. Lewandowski, “Correlating
[10] Charles Henderson, Andrea Beach, and students’ beliefs about experimental physics with lab
Noah Finkelstein, “Facilitating change in course success,” Physics Education Research Conference
undergraduate stem instructional prac- 2015, PER Conference, 367–370 (2015).
tices: An analytic review of the literature,” [22] tinyurl.com/ECLASS-physics, (2015).
Journal of Research in Science Teaching 48, 952–984 (2011).[23] https://www.python.org/, (2015).
[11] Susan R Singer, Natalie R Nielsen, Heidi A Schweingru- [24] A relational database is a description of tables of data
ber, et al., Discipline-based education research: under- and how they relate to each other.
standing and improving learning in undergraduate sci- [25] Eric Jones, Travis Oliphant, Pearu Peterson, et al.,
ence and engineering (National Academies Press, 2012). “SciPy: Open source scientific tools for Python,”
[12] http://www.colorado.edu/sei/departments/physics.htm, (2001–), [Online; accessed 2015-12-23].
(2016). [26] S. van der Walt, S.C. Colbert, and
[13] Elaine Seymour, D Wiese, A Hunter, and Susan M G. Varoquaux, “The numpy array: A struc-
Daffinrud, “Creating a better mousetrap: On-line stu- ture for efficient numerical computation,”
dent assessment of their learning gains,” in National Computing in Science Engineering 13, 22–30 (2011).
Meeting of the American Chemical Society (2000). [27] Wes McKinney, “Data structures for statistical comput-
[14] Timothy J Weston and Sandra L Laursen, “The under- ing in python,” in Proceedings of the 9th Python in Sci-
graduate research student self-assessment (urssa): Vali- ence Conference,editedbySt´efanvanderWaltandJar-
dationforuseinprogramevaluation,”CBE-LifeSciences rod Millman (2010) pp.51 – 56.
Education 14, ar33 (2015). [28] J.D. Hunter, “Matplotlib: A 2d graphics environment,”
[15] ScottWBonham,RobertJBeichner,AaronTitus, and Computing in Science Engineering 9, 90–95 (2007).
Larry Martin, “Education research using web-based as- [29] http://www.qualtrics.com/, (2015).
sessmentsystems,”JournalofResearchonComputingin [30] Bethany R.Wilcox andH.J. Lewandowski, “Presence or
Education 33, 28–45 (2000). absence of gender differences in students’epistemologies
[16] Systems involving centralized data collection with com- aboutexperimentalphysics,”Phys.Rev.STPhys.Educ.
parisonstonationaldatasetsarealsocommonin,forex- Res. In preparation (2016).
ample, onlinehomework systems. Thesesystems arenot [31] https://test.physport.org/DataExplorer/Preview.cfm,
discussedhereastheyareofadifferentnatureandservea (2015).
differentpurposethantheRBAsusedwithinPER.They [32] https://sites.google.com/a/colorado.edu/la-
are proprietary systems used for high-stakes, individual resources/research/learning-gains-study, (2015).
assessment and are rarely used for research purposes. [33] Ben Van Dusen, Laurie Langdon, and Valerie Otero,
[17] BenjaminMZwickl,TakakoHirokawa,NoahFinkelstein, “Learning assistant supported student outcomes (lasso)
and HJ Lewandowski, “Epistemology and expectations study initial findings,” in Physics Education Research
survey about experimental physics: Development and Conference 2015, PER Conference (College Park, MD,
2015) pp.343–346.