ERIC ED615631: Student Practice Sessions Modeled as ICAP Activity Silos

2021·0.55 MB·English

by  ERIC

Student Practice Sessions Modeled as ICAP Activity Silos
Adam M. Gaweda, Collin F. Lynch
NorthCarolinaStateUniversity
Raleigh,NC,USA
agaweda,cfl[email protected]
ABSTRACT ofcontentorcreationofnovelexternalizedoutputslikesummaries.
There are a number of novel exercise types that students can Finally,Interactivelearning involvesdirectlyengagingwithapeer,
utilizewhilelearningComputerScience,eachwithitsownlevel agent,orinstructortoexploreinformationandreceivefeedback
ofcomplexityandinteractionasoutlinedbytheICAPFrame- whichcanbeexpandedupon.
work [10]. Some are Interactive, like solving coding problems;
Constructive, like explaining code; Active, like retyping source Whiletheseexercisetypeshavemadetheirwayintoclassrooms
code;andPassive,likereviewingslides. Todate,therehasbeen thereislittleevidenceofhowthesetypesofengagementinteract
littleresearchonhowstudentsvarytheirstudyandengagement withone-another. Traditionalinterventionstudieshavefocusedon
habitsbyexercisetypeandwhentheydoso. Inthispaper,we theoverallimpactofoneormoreexercisetypes[20,19,14,8],or
presentourfindingsonstudentactivitysequencesfromanonline ontheautomatedselection/recommendationoffutureexcercises
professionaldevelopmentcourse. Weisolatedstudentactivities baseduponastudentmodel[27],butnotonhowstudentswork
intosessionsandthenproducedactivitytransitionvisualizations withoracrossthemintheabsenceofguidance. Norhasthisrecom-
to compare the behavior of students who complete the course mendationworkbeenextendedtonontraditionallearningcontexts.
tothosewhodonot. Wethenusedmultiplefactoranalysesto Absentanunderstandingofhowstudentsorchestratemultiplein-
examinehowstudentstransitionfromonetypeofactivitytothe teractionmodeswefacechallengesinscaffoldingeffectivelearning
next. Fromthisanalysisweidentifiedplatformsilos instudent’s opportunitiesandinevaluatingtheimpactofnovellearningenvi-
work. Wefurtherexpandthisconcepttothepresenceofactivity ronments. Providingstudentswithineffective,oroverlycomplex
silos grouping by type. We find that this siloing behavior is learningopportunitiesriskstrappingtheminafail/skippractice
consistentinbothcompletersandnon-completersbutisweaker cyclethatwouldinhibitanyfunctionallearninggains[17].
for the latter group. Finally, we discuss our findings and how
instructorsandresearchersmayusethisinformationtoensure Inthispaperwereportourinvestigationofhowstudentsdi-
thatstudentsshowpersistencethroughpractice. recttheirpracticeofCSconceptswhenpresentedwithasetof
options. Ourstudywasconductedinthecontextofanonline
Keywords professionaldevelopmentcourseforPythonprogramming. This
courseispartofaresearchstudyfundedbytheDepartmentof
novel exercises, ICAP framework, study sessions, activity se-
Labor to create novel learning pathways for existing technical
quences,platformsilos,activitysilos,studentmodeling
professionals to move into AI and Data-Science areas. We ex-
tractedstudents’practice/studysessionsandanalyzedtheactivity
1. INTRODUCTION transitionswithineachsession. Wethenanalyzedtheseactivity
CSEducationhaveintroducedanumberofnovelexercisetypes sequencestoanswerthefollowingresearchquestions(RQs):
tobetterscaffoldstudents’experiences. Theseincluderetyping
sourcecode[14],arrangingscrambledcodefragments(Parsons
RQ1 Canwereplicatetheexistenceofplatformsilos introduced
Puzzles)[21],debuggingprovidedcode[8],predictingoutput[26],
in[1]withanewdataset?
fillintheblanks[4],self-explanation[4],andsmallscalecoding
RQ2 Aretherecommonactivitytransitionsbetweenstudents?
exercises[2,12]. Eachoftheseexercisetypescanalsobemapped
RQ3 HowdoactivitysequencesconnectwiththeICAPFrame-
onto the ICAP framework [10]. This framework defines four
work?
categoriesofinstructionalactivitiesbaseduponstudents’levelof
RQ4 How do the practice sessions of completers of the course
engagement: Interactive,Constructive,Active,andPassive.Pas-
differfromnon-completers?
sive learning includesreadingstaticcoursematerialsorwatching
lecturevideos.Activelearning isdescribedasrehearsingorcopy-
ingsolutionsteps.Constructivelearning includesself-explanation To answer these questions, we first produced and analyzed
asetofactivitytransitiondiagramsforstudentsinthecourse
comparingthosewhocompletedthecoursetothosewhodidnot.
Throughthisanalysis,weconfirmedthepresenceofplatformsilos
whichweextendthisnotiontoincludeactivitysilos,wherestu-
Adam Gaweda and Collin Lynch “Student Practice Sessions dentsprimarilyfocusonasinglemodeofengagement(consistent
Modeled as ICAP Activity Silos”. 2021. In: Proceedings of withICAP)duringagivenpracticesession. Whenstudentsdid
The 14th International Conference on Educational Data Mining transitionbetweenmodes,itwasonlytomoveuptheICAPchain
(EDM21). International Educational Data Mining Society, 595-601.
and never to ‘downgrade’ to a lower level of engagement. We
https://educationaldatamining.org/edm2021/
supportourfindingsthroughtwodifferentfactoranalyses,which
EDM’21June29-July022021,Paris,France
helpexplainthe42-62%ofvariancebetweenthesessions. From
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 595
our results, we found ICAP categories isolated into individual need to have an understanding of the snippet as a whole to
sessions,aswellasLMScontentconsumptionandquiztaking. deducewhatneedstobeincludedataparticularblanklocation.
Reviewingincompleteworkedexamplesreducesineffectiveself-
2. BACKGROUND explanationsandenhancesthetransferoflearnedmaterials[4,
5]. Based on the results from Atkinson et. al., we consider
2.1 ICAPFramework
FitB-styleexercisestorequirelowerlevelsofengagementthan
The ICAP Framework seeks to classify the modes of student
self-explanation,andthusclassifythemasanactive ICAPmode.
engagement while engaging in learning activities [10]. These
categories,Interactive,Constructive,Active andPassive respec-
tivel. Passive engagementincludesactivitiessuchasreadinga 2.2.3 ParsonsPuzzles
textorobservingavideo.Active engagementincludesrehearsing ParsonsPuzzles(PP)presentsnippetsofcodethathavebeen
steps or copying solutions. Constructive engagement includes separatedintosegmentsandthenshuffledinorder[21]. Students
self-explanationorcomparingandcontrastingmaterials. Finally, arethentaskedwithplacingthesegmentsbackintothecorrect
Interactive engagementincludesrespondingandinteractingwith order. While Parsons Puzzles are helpful for learning how to
anagent,system,oranotherperson. Ttheframeworkishierar- structure code, performance on Parsons Puzzles has not been
chical,suggestingI>C>A>P,orthatactivitieswithhigher showntocorrelatewithstudents’abilitytoreadortracecode
levelsofengagementpromotethegreatestlevelsoflearning. [11,20]and‘distractor’variantsarenotbeneficialtoyounglearn-
ers[13, 16]. Thesefindingsfurthersupportourresearchgoals,
ChiandWyliepresentaliteraturereviewofseveralempirical asnoteveryexercisetypemaybebeneficialforlearningallthe
studiessupportingtheirICAPhypothesis[10]. Thefirststudy technicalskillsnecessaryforComputerScience.
consistedofallfourmodesofengagementinmaterials science
and showed learning improved significantly at a rate of 8-10% ChiandWylie’sdefinitionofconstructivemodesofengagement
per mode. They then present two studies which used active, include“learnersgenerateorproduceadditionalexternalizedout-
passive,andconstructivemodesinevolutionarybiologyandplate puts... beyondwhatisprovided”. Asmentionedintheprevious
tectonics. Finally, Chi and Wylie present comparisons of two section,ParsonsPuzzlesaresimilartoourtoypeanutbutterand
modesacrossnotetaking,conceptmapping,andself-explanation. jellyexercise. WhileParsonsPuzzlescouldbeconstruedasactive,
In each of these studies, the results again showed that higher studentsmaynotfullycomprehendtheappropriateorderofcode
modesofengagementhadhigherlearninggains. syntaxandmustfigureouttherightsequenceaspartoftheexer-
cise. InourcommunicationswithChiaboutParsonsPuzzles,Chi
ChiandWylie’swork,aswellasourownpersonalcommunica- statesthatdeterminingtheparticularICAPmodeforanactivity
tionswithChi[9],notethatidentifyingthemodeofaparticular canbenon-trivial[9]. ‘Ifastudentalready[knows]therecipe,then
activitycanbeanon-trivialtask. Forexample,atoyexample re-orderingitisjustpickingoutthesequencing,guidedbythese-
taskcouldbepresentingstepstowardmakingapeanutbutterand quenceinformation[they]alreadyknow’thenitisactive. However,
jellysandwichgivenrandomlyshuffledsegmentsofinstructions. ‘ifthestudent[has]notlearnedtheorderfromsomeothersource,
Thistaskcouldbeconstructedasanactive exerciseifthestudent andyouareaskinghertofigureouttherightsequence’,itisacon-
alreadyknowstherecipeandthetaskissimplypickingtheappro- structive exercise. Sincenovicesmaynothaveproficiencyatthe
priatesequencefromthelistofsteps. However,ifthestudenthad timeoftheexercise,weelecttousetheupperboundICAPmode
notlearnedtheappropriateorder,thenthetaskoffiguringout andconsiderParsonsPuzzlesasaconstructive learningactivity.
therightsequencewouldbeconstruedasaconstructive exercise.
ComputerSciencehasasimilartask,knownasParsonsPuzzles, 2.2.4 OutputPrediction
that mirrors this toy example, discussed in more detail in the
OutputPrediction(OP),alsoknownasvariabletracing,exercises
followingsection.
askstudentstoanalyzecodeandthenstatetheexpectedoutputs
ofcodeexecutionortheexpectedvalueofavariableasthecode
2.2 NovelExerciseTypes
progresses. Often,variabletracingisdoneduringconditionaland
Inthissection,wedescribeeightdifferentexercisetypestudiedin loopinstructiontodemonstratehowthevaluesofthevariables
ComputerScienceeducation,providesomeresearchbackground changeaftereachiteration. LikeParsonsPuzzles,OP-styleex-
ontheexercisetype,andjustificationsforwhichICAPmodewe ercisesrequirestudentstoprocesscodesnippetsandexternalize
willclassifythemasforofourstudy. theirexpectedoutputs. Thus,weconsideroutputpredictionas
anotherconstructive learningactivity.
2.2.1 TypingExercises
Typing Exercises (TE) require students to retype source code 2.2.5 Self-Explanation
thathasbeenpresentedtothem[14]. TypingExercisescanbe
Self-explanation(SE)exercisespresentstudentswithsourcecode
used as active learning activities under the ICAP Framework
andaskthestudenttoexplainhowthecodeoperates,describe
astheyrequirestudentstoretypeverbatimthecodepresented
theoverallefficiencyofthecode,orcreateadocumentationstring
to them. Previously, we presented images of source code and
to appear as a comment for the program or function. These
showedthatself-selectedstudentsthatcompletedoptionaltyping
areopen-endedexercisesthataresubjectiveinnatureandare
exercises earned higher course grades and submitted less code
consideredtobeconstructive [10]. However,ChiandWyliedo
withbuildfailures. Leinonenet. al. alsopresentedoptionaltyping
note that students’ may treat the self-explanation activity as
exercisestostudentsbeforeprogrammingtasks[19],butwerenot
active if“thestudent’sself-explanationisverbatimtowhatwas
abletofindthesameresultsasours. Howevertheirstudyonly
read”. However,novicesmaystrugglewithreadingandevaluating
lastedtwoweeksandmanyoftheirselectedparticipantsdidnot
programming code in a linear fashion, focusing more on what
attempttheexercisesatall.
eachlineofcodedid,ratherthanhoweachlineinteractedwith
eachother,oringeneralproducepoorexplanations[25,5]. While
2.2.2 FillintheBlank constructive SEactivitiesmayproducehigherlearninggainsthan
FillintheBlank(FitB)exercisesremoveasmallportionofcode lower-level modes, they may alsonotbe the most appropriate
fromasnippetandasksstudentsto‘fillin’theblank. Students activity for students who are struggling, Thus, similar to our
596 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)
decisionsforParsonsPuzzles,weusetheupperboundtoclassify 3. STUDY
self-explanationasaconstructive learningactivity. 3.1 Design
Westudiedproblemsolvinginthecontextofanonlineprofessional
2.2.6 FindandFixtheBug developmentcourseinPythonprogramming. Thisisapreparatory
Resolvingerrors,ordebugging,isoneofthefirsthurdlesstudents courseforaseriesinAIthatisaimedatnon-traditionalstudents
encounter when learning to program [3]. Once they find that makingacareertransition. ThecourseusedtheMoodleLearning
an error has occurred, it will be necessary for them to resolve ManagementSystemandTYPOS,aCSexerciseplatform[14],and
it before addressing any remaining subgoals for their solution. isorganizedinto10modulesEachmoduleincludesasetofstatic
Forthepurposesofourstudy,weseparateddebuggingintotwo readingmaterial,lectureslides,prerecordedvideos,optionalprac-
separateactivities-FindtheBug(FnB)andFixtheBug(FxB). ticeexercises,andamoduleassessment. Therewere24optional
FindtheBugexercisespresentstudentswithcodethatcontainsa exercisespermodule,3exercisesforeachofthe8typespreviously
commonmisconceptionfornovices. Insteadofresolvingtheerror, described. Studentswerefreetoworkonthepracticeexercisesor
studentsareaskedtohighlighttheareaofcodewherethiserror assessmentsasmuchastheyliked. Theonlyrequirementforpro-
exists. ThoughtheICAPframeworkconsidershighlightingtext gressingtothenextmodulewastoearnapassinggrade(80%or
asanactive learningactivity,ChiandWyliedefineconstructive higher)onthepriormodule’sassessment. Inordertocompletethe
behaviors as requiring some level of“inference”, or adding in course,studentsneededtoearnpassinggradesonallassessments.
additionaldetailorqualification. Sincestudentsmustassessif
alineofcodeis‘correct’ornot,theyareproducingqualifications Wehad69studentsconsenttothestudy. Ofthosestudents,37
andtherefore,weelecttolabelFnBexercisesasconstructive. successfullycompletedthecourse. Studentinteractionsonboth
platforms were logged. We omitted some Moodle interactions
FixtheBugexercisesfollowthenaturalprogressionofdebug- suchaslike“filedownloading”and“viewingthecourse”whichdid
gingtasksbyrequiringstudentstoresolvebrokencode[8]. FxB notpertaintotheexplicitlearningactionswewerefoucusedon.
activitiescouldbeconsideredconstructive or interactive depend-
ing on the context. Similar to FnB exercises, Chi and Wylie Theresultingdatasetcontainedatotalof29,190interactions
include‘repairing’asaconstructive behavior. However,FxB-style from all the students. We then used a similar strategy as [1]
exercisescanalsobeinteractive becausestudentsoftenrelyonthe toextractusersessionsfromtheseinteractionsbaseduponan
codeinterpreter’sfeedbackduringthedebuggingprocess. Fixing exploratoryanalysisofthegapsbetweeninteractions. Thetime
one error may produce new errors with new feedback, or the deltas between course interactions were measured. If a delta
repairmadebythestudentcouldbeincorrect. Thus,weagain betweeninteractionsexceededapredefinedcutoffthreshold,that
chooseusetheupperboundtolabelFxBexercisesasinteractive. sessionwasconsideredover,andanewsessionwascreated. This
was repeated for all interactions a student had for the course.
2.2.7 CodingExercises WhileSeshadriet. al. useda40minutecutofftoestablishtheend
andstartofanewsession,wechosea60minutesasitwasourmost
The final exercise type we used in our study is the de facto
frequentlyobserveddeltabetweeninteractions. Weextracted1,313
standardofintroductoryCScourses-theCodingExercise(CE).
sessionsintotal. Studentsthatcompletedthecourseaccounted
WhileComputerScienceismorethanprogramming,codingex-
for71%,or20,748,ofthecourseinteractions,with1,041sessions
ercisesareoftenusedbyinstructorsasgradedcoursematerialfor
total,atanaverageof28.1(±17.9)sessionspercompleter.
studentstodemonstratetheirunderstandingofthecurrentcourse
topic. Therehasbeenworkontheuseof‘many-smallprograms’
and‘simplesyntaxexercises’,whichsimplyrequirestudentsto 3.2 ActivitySessionTransitionProbabilities
completesmall-scalecodingexercisestobecomefamiliarwiththea The route that students take through online materials can be
particularimplementationbeforeutilizingitaspartoflarger-scale modeledasdiscreteMarkovprocesses,inwhicheachstaterep-
problems[2,12]. Inbothcases,completingthesesmaller-scale resents an activity within the session. For example, a student
programmingexercisesimprovedstudentperformanceandyielded maytransitionfromreviewinglectureslidestoviewinglecture
happierstudents. Studentsoftenrelyonfeedbackfromtheinter- videos on Moodle, or MS→MV. Jeffries et. al. [17] used a
preterastheyconstructtheirsolutionsandmorethanlikelyneed similarprocesstoanalyzesuccessandhelpseekingbehaviorswith
todebugtheirownworkduringthisprocess. Thus,weconsider studentsinanintroductoryCScourse.
CodingExercisesasaninteractive learningactivity.
Figure1visualizesthetransitionprobabilitiesforcompleters:
2.3 StudentModelingandActivityMining transitionswithinTYPOSappearasdashedbluelines,transitions
withinMoodlearesolidredlines,andtransitionsbetweenTYPOS
Seshadriet. al. analyzedhowstudentstudysessionsoperated
andMoodlearesolidblacklines. Forvisibility,onlytransitions
across multiple platforms for three separate courses [1]. Their
involving the start/end of a session or those with a frequency
resultsfoundthatgivenmultipleeducationplatforms,students
above 5% are presented. Module assessment(MA)accounted
willoftenoperatewithinplatformsilos,oronlyutilizeoneeduca-
for39%ofstartingsessionbehavior,TYPOSpracticeaccounted
tionalplatformduringanindividualstudysession. Ofthestudent
for36%,andlectureslidesandvideos(ContentConsumption)
sessions,morethan90%ofthemincludedonlyoneplatform. Ina
accounted for 26%. This figure shows students’ practice was
follow-upstudy,theycomparedtheactivitiesofhigherperforming
largelysiloedbyplatformwitheachsessiontakingplacewithina
studentstothelowerperforminggroupandshowedthatboth
singlemodeofinteraction. WiththeexceptionoftheMS→TE
groupsweremostlikelytostickwithinplatformsilos[15]. Their
transition, students either interacted with TYPOS or Moodle,
workservesas themotivationfor our RQ1. Our hypothesisis
butrarelytogether. SinceMAshowedthehigheststartingses-
thatthepresenceofthese‘platformsilos’willcontinuetohold
sionprobability,oneassumptionisthatstudentsenrolledinthe
acrossothereducationalplatformsnotstudiedintheirresearch.
coursewithpriorcodingexperiencemayhavereviewedthecourse
materialtobecomefamiliarwithPythonsyntaxbeforegoingon
tocompletethemoduleassessment.
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 597
Figure1: Completeractivitytransitionprobabilitiesduringsessions.
One interesting observation from Figure 1 is that TYPOS numberoffactors. Table1showstheeigenvaluesforeachfactor
practicesessionsinvolvedonlyoneorasmallsubsetoftheex- andtheircumulativevariance. Basedontheseresults,weutilized
ercisetypesavailabletothestudents. Forexample,theTE,PP, twoseparatefactoranalyses. Thefirstanalysisuses3-factorsto
andFitB exercisesweretypicallycompletedalone. Amongtwo correspondtothe3eigenvaluesgreaterthan1,assuggestedby
exercisepairs,SE→CE,OP→FnB,andFnB→FxBshowed Kaiser[18]. Thesecondanalysisincreasesto5-factorsbasedon
higherprobabilitiesthanendingthesession. thevarianceextractionrule,whichspecifiesa0.7thresholdfor
eigenvalues[6,24,23].
WecanfurtherclassifytheactivitiesbytheirrespectiveICAP
modes. Forexample,theSE→CE transitioncanalsobeviewed Table1: EigenvaluesforFactorAnalysisofCompleters
as a Constructive→ Interactive transition, MS→TE can be Factor Eigenvalue CumulativeVariance
viewed as Passive→ Active, and so on. From this perspective, 1 3.860695 30.51%
transitionsbetweenactivitiesrarely‘downgraded’toalowermode. 2 1.390921 37.21%
WiththeexceptionofMS→MAandMV→MA,modechanges 3 1.170659 41.79%
primarilyshiftedbyonelevelofengagement. 4 0.916401 52.26%
5 0.810977 62.27%
3.3 ActivitySessionFactorAnalysis 6 0.665925 60.91%
TheprobabilitiesshowninFigure1raisenewquestionsabout 7 0.649612 69.59%
students’practicebehaviors. Notonlywereweabletoconfirm 8 0.486275 70.05%
theexistenceofplatformsilos fromourRQ1,butbasedonthe 9 0.446064 55.23%
observedtransitionprobabilities,wefoundthatstudentsmayalso 10 0.391094 55.42%
operatewithinwhatwetermanactivitysilo,focusingprimarily 11 0.211376 55.42%
ononeICAPmodepersession. Inthissectionwereportontwo
factoranalyseswhichweusetoidentifythelatentvariablesfor Our next task was to identify load factor thresholds for our
eachpracticesessioninordertostrengthenourclaim. latentvariables. Whilethereisnouniversalstandardforloading
thresholds,thegoalistoonlyobservevariablesthatshareastrong
FactorAnalysisisadimensionreductionmethodtodescribe associationwitheachotherandisanon-trivialprocess[22]. For
thevariabilityofobservedvariablesinto,potentially,lowerlatent ourpaper,wewillfocusourattentiontovariablesabovea0.50
variables,orfactors. Toprepareourdatasetforfactoranalysis, (or25%ofthevariable’svariance)threshold,highlightingthem
eachactivitywasconvertedintoabinaryvalue,representingthe inourtablesasgreen. Sincethedifferencebetweena0.50and
presenceorabsenceoftheactivityinthesession. Forexample,if 0.49loadingisminimal,wewillalsohighlightvaluesgreaterthan
asessiononlyinvolvespassivecontentconsumption,theresulting 0.4(or16%variance)inyellowforadditionalreference.
vector for the session would be [1,1,0,0,0,0,0,0,0,0,0], where
the1srepresentthepresenceoflectureslidesandvideosand0 Table2showsthefactorloadvaluesforour3-factoranalysis.
representstheabsenceofallotheractivities. F1 has high loadings for OP, FnB, FxB, SE, and CE. If
we consider the ICAP modes for this factor, this indicates a
In order toevaluate the appropriateness of our data for fac- transitionof Constructive → Interactive TYPOSPractice. F2
toranalysis,weusetheBartlett’sandKaiser-Meyer-Olkintests. has high loadings for FitB and PP, or Active→ Constructive
Bartlett’stestcomparesourcorrelationmatrixagainstaniden- TYPOSPractice. Finally,F3hashighloadforMS,orPassive
tify matrix to test whether our samples are from populations MoodleInteraction. FromTable2,weonceagaincanconfirmthe
with equal variance. Our samples were statistically significant presenceofplatformsilos,howeverweexpandourfactoranalysis
(χ2=3360.05,p=0.0)andthuswecancontinuewithourfactor inordertoseethepresenceofactivitysilos. Moreover,3-factors
analysis. Kaiser-Meyer-Olkincheckstheadequacyforourvari- onlyaccountsfor41.79%ofthecumulativevarianceandsousing
ablestodeterminethesuitabilityoffactoranalysis. OurKMO the0.7eigenvaluethresholdwillallowustoaccountfor62.27%.
scorewas0.83,whichagainshowsourdatasetisadequate.
And finally, Table 3 shows the factor load values for our 5-
Thenextstepforouranalysiswastodeterminetheappropriate factoranalysis. SimilartoTable2,weseeaseparationbetween
598 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)
Table2: Loadingsfor3FactorAnalysisforCompleters. Values differences(χ2=997.8,p>0.0001)andourKMOscorewasalso
greaterthan0.4areinyellowandgreaterthan0.5areingreen.
adequateforanalysis(0.77). SimilartoTable1,wefoundsupport
Activity F1 F2 F3
for 3- and 5-factor analysis, seen in Table 4. We note that a
MS 0.0306 -0.0294 0.5073
6-factoranalysisisalsopossible,buttomirrorthefactoranalysis
MV -0.1006 -0.0999 0.3891
forcompleters,weelectednottopursueit.
MA -0.0269 -0.2862 0.2177
TE 0.0193 0.4336 -0.1345
Table4: EigenvaluesforFactorAnalysisofNon-completers
FitB 0.4703 0.5471 -0.0286
Factor Eigenvalue CumulativeVariance
PP 0.3300 0.6726 0.0392
1 3.480694 26.26%
OP 0.6257 0.3885 0.0216
2 1.503161 34.57%
FnB 0.8215 0.2149 0.0307
3 1.286916 41.39%
FxB 0.8399 0.1546 0.0059
4 0.888718 47.44%
SE 0.6802 0.1020 -0.0933
5 0.821608 54.51%
CE 0.5019 0.0135 -0.1306
6 0.719444 62.12%
7 0.657629 66.31%
8 0.520019 63.79%
TYPOSactivityandMoodleactivity. Further,theactivitiesfor
9 0.499220 57.46%
eachfactorareconfinedtoasingleICAPmode,orwithinone
10 0.375677 57.69%
mode. F1 contains mostly Constructive activities (as well as
11 0.246915 57.69%
FxB),F2containsActive→Constructive activities,F3contains
Passive activities,andF4andF5containInteractive activities. In
addition,F3andF4showaseparationbetweenPassive Moodle Table5showsour3-factoranalysisfornon-completers. The
contentconsumptionandInteractive assessmenttaking. Thus, sameactivitieshavinghighloadingsasF1andF2asthe3-factor
fromtheresultsofour5-factoranalysis,weconfirmthepresence analysisforcompleters(Table2)andalsoshowsimilarplatform
ofactivitysilos withinourstudents. silos. Likewise, the ICAP mode considerations are similar for
eachfactor. F1showsConstructive→Interactive behaviors,F2
Table3: Loadingsfor5FactorAnalysisforCompleters. Values shows Active→ Constructive behaviors,andF3shows Passive
greaterthan0.4areinyellowandgreaterthan0.5areingreen. Moodleinteraction.
Activity F1 F2 F3 F4 F5
MS 0.0224 -0.0065 0.2411 0.1061 -0.0150 Table5: Loadingsfor3FactorAnalysisforNon-completers.
MV -0.0836 -0.1009 0.9838 -0.0982 -0.0264 Activity F1 F2 F3
MA -0.0438 -0.1664 0.1175 0.9759 -0.0124 MS 0.0362 0.0014 0.4167
TE 0.0364 0.3448 -0.0843 -0.2002 -0.0106 MV -0.0334 0.0106 0.6616
FitB 0.4344 0.5629 -0.0451 -0.0445 0.0772 MA -0.0136 -0.2524 0.2999
PP 0.2740 0.7467 0.0004 -0.0193 0.0369 TE 0.0756 0.4637 -0.0719
OP 0.5520 0.4568 -0.0008 0.0282 0.1792 FitB 0.3176 0.6024 -0.0202
FnB 0.8419 0.2321 0.0035 -0.0110 0.0829 PP 0.1082 0.7404 0.0221
FxB 0.8759 0.1568 0.0058 -0.0255 0.0980 OP 0.5310 0.3499 0.1105
SE 0.5963 0.1562 -0.0318 -0.0450 0.2602 FnB 0.5573 0.4837 -0.0910
CE 0.3227 0.0549 -0.0631 -0.0088 0.8891 FxB 0.6636 0.3369 -0.1218
SE 0.8012 0.0568 0.0573
CE 0.5900 0.0201 0.0056
3.4 ComparingCompleterstoNon-Completers
Havingshownthebasicactivitystructuresandidentifiedrelevant
factorswethenchosetoexplorewasthedifferencebetweencom- Table6showsour5-factoranalysisfornon-completers. Non-
pleterandnon-completerstudents. Weusedthesamemethodsfor completersmaintainedthe Constructive→Interactive connection
thenon-completergroupforcomparison. Therewere32students forF1andF2alsomaintainstheActive→Constructiveconnection.
thatfailedtocompleteourcourse. Non-completersmade8,442 Theremainingfactorsdodiffer,F3separatedtheFnB→FxB
courseinteractionsacross341sessions,withanaverage10.7(±7.8) exercises from F1 and F4 focuses primarily on TE. The ab-
sessionspernon-completer. sence of MA was expected since course progression requires
passingmoduleassessments. Fromouranalysis,weconcludethat
non-completersstilloperatedwithinactivitysilos.
Wefirstproducedthesametransitionprobabilitiesdiagram
fornon-completeractivitysessions,seeninFigure2. Similarto
Table6: Loadingsfor5FactorAnalysisforNon-completers.
completers, module assessment accounted for 31% of starting
Activity F1 F2 F3 F4 F5
sessionbehavior,TYPOSpracticeaccountedfor49%,andlecture
MS 0.0271 0.0234 0.0080 -0.0129 0.4040
slidesandvideos(ContentConsumption)accountedfor22%. Non-
MV -0.0191 0.0204 -0.0417 0.0373 0.7123
completersprimarilyoperatedwithinasingleplatform,though
MA 0.0049 -0.1705 -0.0558 -0.1637 0.2922
therewasmoreinteractionsbetweenMoodleandTYPOS.Forex-
TE 0.0541 0.2612 0.0605 0.9570 -0.0716
ample,12%ofMV’stransitionsmigratedtoTYPOSexercisesand
FitB 0.2259 0.6333 0.1698 0.1275 -0.0557
5%ofSEtransitionsmigratedtoMA. Whilecompleterstudents
PP 0.0224 0.6734 0.1463 0.1925 -0.0155
separatedSE→CE andOP→FnB→FxB transitions,these
OP 0.4707 0.4584 0.1711 0.0080 0.0735
twosequenceswerecombinedfornon-completers. However,this
FnB 0.2516 0.4546 0.6255 0.0378 -0.0543
couldpotentiallybeduetothesizedifferences. Bothpopulations
FxB 0.3618 0.2049 0.8444 0.0752 -0.0706
hadsimilarpopulationsizes,butnon-completersdidnotcomplete
SE 0.8072 0.0883 0.2525 0.0740 0.0496
eachmoduleassessmentandwouldnotproduceasmanysessions.
CE 0.6201 0.1006 0.1092 -0.0054 -0.0206
Wethencarriedoutthesamefactoranalysesfornon-completers.
The results of a Bartlett’s test showed statistically significant
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 599
Figure2: Non-completeractivitytransitionprobabilitiesduringsessions.
4. DISCUSSION Second,theexercisetypeswerepresentedinaconsistentman-
Theresultsfromourprobabilitytransitiondiagramsconfirmthe nerforeachmodule. Thustheywereimplicitlysequencedwith
presenceofbothplatformsilos andactivitysilos instudentwork. lower-levelICAPmodesappearingonthetop. Aswementionedin
Theyalsoservetohighlightareaswhereeducatorsandresearchers ourintroduction,discerningtheappropriateorderfor11different
cantailormoreappropriatelearningpathsforstudentsandinpar- activitiesisanon-trivialmatterandmeasuringtheappropriate
ticularthosestudentsthatmaybestrugglingwithcoursematerial. order of exercise types was not a part of our study. Thus, we
presentedexercisesinanorderthatprogressivelyincreasedthe
Ourstudyallowedstudentstoself-selectwhichactivitiesthey level of engagement. This may have influenced next practice
wantedtofocustheirattentionson. Whilethisstyleofcourse selectionsbystudents.
designcouldbeadopted,itdoesstillpresentlimitations. However,
studentsthatprimarilyfocusonlower-levelICAPmodeactivities Finally,weacknowledgethattheICAPmodesassociatedwith
maybereluctanttomoveintohigh-levelmodes. Instructorsorsys- eachexercisetypearesomewhatsubjectiveandopenfordiscussion.
temsthatcanidentifythisstagnatepracticebehaviorcouldencour- MoreovertheexactevaluationofexerciseslikeParsonsPuzzles
agestudentstomoveintohigh-levelICAPmodes. Thereisgrow- orself-explanationmayrequireadditionalresearchandcontext.
inginterestintheconceptofnudgetheory to“alterbehaviorwith- Forthepurposesofthisstudy,whenfacedwithuncertaintywe
outincentivesorbanningalternatives”[7]toencourageprogression. classifiedexercisesaccordingtoahigherlevelmodeofinteraction.
Similarly, we presented students with a number of different 6. CONCLUSIONS
activities,atdifferentcomplexities,fortheirlearningexperience.
Inthiswork,weextractedthepracticeandstudysessionbehaviors
Basedonourresults,studentsweremorethanwillingtocomplete
fromnon-traditionalstudentslearningPython. Amongcompleters
eachtypeofexercise. Somestudentsevenaskedformoreactiv-
andnon-completersofthecourse,theyprimarilyfocusedonasin-
itiesinourpost-coursesurvey. Whileincreasingtheworkloadfor
gleplatform. Theactivitieswithintheseplatformsweremapped
studentsandlearningmaterialcreators,manyoftheactivities
to the ICAP framework. Further, we used factor analyses to
weusedarenotoverlycomplexandrequiredaminimalamount
identify the presence of activity silos within practice sessions.
of time to create, or from the students’ perspectives complete.
Completersandnon-completerssharedsimilarbehaviorsduring
ActivitiesliketypingexercisesorParsonspuzzlescanbecreated
thesepracticesessions,primarilyfocusingononeortwomodes
fromexistingcoursematerialsandofferlittleincentiveforstudents
ofengagementandrarely‘downgraded’tolowerlevelmodes.
tocheat. Theysimplyallowstudentsanopportunitytopractice
theconceptstheylearnedratherpassivelygiventothem,refining
Wecanutilizetheseactivitysequencestohelpshapeouroverall
theirunderstanding,beforeneedingtoapplyittoproblemsolving
coursedesignsforensuringstudentlearning. Lower-levelactivities
activitieslikecodingexercises.
canprovidestudentswiththefoundationalknowledgenecessary
asapartofthetechnicalskillsforthecontent,whilehigher-level
activitiescanrefineandencourageadditionallearninggains. As
theresearchinthisareaexpands,wehopetheinformationpre-
5. LIMITATIONS
sentedinthisstudyencourageseducatorsandresearchersalike
Weacknowledgesomelimitationswithourstudy. First,ourcourse
to provide practice in both levels and can serve as a guide for
ran during the COVID-19 pandemic, which has altered many
recommendationsonhowtobestbuildlong-termproficiencies.
individuals’habits. Ourpopulationalsocontainednon-traditional
studentswhowerebalancingtheirstudieswithworkingfromhome
Acknowledgements
andsupportingotherfamilymembers. Thus,non-completionmay
havebeendrivenbyexternalconstraintsthatarenotreflected ThisworkwassupportedinpartbytheNationalScienceFounda-
inourdataset,andtheobservedhabitsmaychangesomewhat tionunderGrantDRL1721160. KristyBoyer,EricWiebe,and
duringnon-COVIDtimes. CollinF.Lynch(co-PIs).
600 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)
7. REFERENCES [15] NikiGitinabard,TiffanyBarnes,Sarah
[1] SheshadriAdithya,NikiGitinabard,CollinFLynch,Tiffany Heckman,andCollinF.Lynch.Whatwillyoudonext?
Barnes,andSarahHeckman.Predictingstudentperformance Asequenceanalysisonthestudenttransitionsbetween
basedononlinestudyhabits: Astudyofblendedcourses. onlineplatformsinblendedcourses.InProceedingsofthe
InternationalEducationalDataMiningSociety,2018. 12thInternationalConferenceonEducationalDataMining,
[2] JoeMichaelAllen,FrankVahid,AlexEdgcomb,Kelly EDM2019,Montr´eal,Canada,July2-5,2019,2019.
Downey,andKrisMiller.Ananalysisofusingmanysmallpro- [16] KyleJamesHarms,JasonChen,and
gramsincs1.InProceedingsofthe50thACMTechnicalSym- CaitlinLKelleher.Distractorsinparsonsproblemsdecrease
posiumonComputerScienceEducation,pages585–591,2019. learningefficiencyforyoungnoviceprogrammers.In
[3] A.AltadmriandN.Brown.37millioncompilations: Proceedingsofthe2016ACMConferenceonInternational
Investigatingnoviceprogrammingmistakesinlarge-scale ComputingEducationResearch,pages241–250,2016.
studentdata.InProceedingsofthe46thACMTechnical [17] BrynJeffries,TimothyBaldwin,MarionZalk,
SymposiumonComputerScienceEducation,SIGCSE andBenTaylor.Onlinetutoringtosupportprogramming
’15,pages522–527,NewYork,NY,USA,2015.ACM. exercises.InProceedingsoftheTwenty-SecondAustralasian
[4] RobertKAtkinson,SharonJDerry,Alexander ComputingEducationConference,pages56–65,2020.
Renkl,andDonaldWortham.Learningfromexamples: [18] HenryFKaiser.Theapplication
Instructionalprinciplesfromtheworkedexamplesresearch. ofelectroniccomputerstofactoranalysis.Educational
Reviewofeducationalresearch,70(2):181–214,2000. andpsychologicalmeasurement,20(1):141–151,1960.
[5] RobertKAtkinsonandAlexanderRenkl.Interactive [19] AnttiLeinonen,HenrikNygren,NeaPirttinen,
example-basedlearningenvironments: Usinginteractive ArtoHellas,andJuhoLeinonen.Exploringtheapplicability
elementstoencourageeffectiveprocessingofworkedexamples. ofsimplesyntaxwritingpracticeforlearningprogramming.
EducationalPsychologyReview,19(3):375–386,2007. InProceedingsofthe50thACMTechnicalSymposium
[6] DeborahLBandalosandSaraJ onComputerScienceEducation,pages84–90.ACM,2019.
Finney.Exploratoryandconfirmatory.Thereviewer’s [20] MikeLopez,JacquelineWhalley,PhilRobbins,
guidetoquantitativemethodsinthesocialsciences,93,2010. andRaymondLister.Relationshipsbetweenreading,
[7] ChrisBrown.Digitalnudgesforencouraging tracingandwritingskillsinintroductoryprogramming.
developeractions.In2019IEEE/ACM41stInternational InProceedingsofthefourthinternationalworkshopon
ConferenceonSoftwareEngineering: Companion computingeducationresearch,pages101–112.ACM,2008.
Proceedings(ICSE-Companion),pages202–205.IEEE,2019. [21] DaleParsonsandPatriciaHaden.
[8] NickChengandBrianHarrington.TheCodeMangler: Parson’sprogrammingpuzzles: Afunandeffectivelearning
Evaluatingcodingabilitywithoutwritinganycode.In toolforfirstprogrammingcourses.InProceedingsof
Proceedingsofthe2017ACMSIGCSETechnicalSymposium the8thAustralasianConferenceonComputingEducation-
onComputerScienceEducation,pages123–128,2017. Volume52,ACE’06,pages157–163,Darlinghurst,Australia,
Australia,2006.AustralianComputerSociety,Inc.
[9] MicheleneTHChi.PrivateCommunication,August2020.
[22] RobertAPeterson.Ameta-analysis
[10] MicheleneTHChiandRuthWylie.TheICAP
ofvarianceaccountedforandfactorloadingsinexploratory
framework: Linkingcognitiveengagementtoactivelearning
factoranalysis.Marketingletters,11(3):261–275,2000.
outcomes.Educationalpsychologist,49(4):219–243,2014.
[23] KeenanAPituchandJamesPStevens.
[11] PaulDenny,AndrewLuxton-Reilly,andBeth
Appliedmultivariatestatisticsforthesocialsciences:
Simon.Evaluatinganewexamquestion: Parsonsproblems.
AnalyseswithSASandIBM’sSPSS.Routledge,2015.
InProceedingsofthefourthinternationalworkshopon
computingeducationresearch,pages113–124.ACM,2008. [24] JohnRuscioandBrendanRoche.Determining
thenumberoffactorstoretaininanexploratory
[12] JohnEdwards,JosephDitton,DraganTrninic,HillarySwan-
factoranalysisusingcomparisondataofknownfactorial
son,ShelseySullivan,andChadMano.Syntaxexercisesin
structure.Psychologicalassessment,24(2):282,2012.
CS1.InProceedingsofthe2020ACMConferenceonInterna-
tionalComputingEducationResearch,pages216–226,2020. [25] SusanWiedenbeck,VikkiFix,andJeanScholtz.
Characteristicsofthementalrepresentationsofnovice
[13] BarbaraJEricson,LaurenEMargulieux,andJochenRick.
andexpertprogrammers: anempiricalstudy.International
Solvingparsonsproblemsversusfixingandwritingcode.In
JournalofMan-MachineStudies,39(5):793–812,1993.
Proceedingsofthe17thKoliCallingInternationalConference
onComputingEducationResearch,pages20–29,2017. [26] GregWilson.Teaching
TechTogether: HowtoMakeYourLessonsWorkandBuild
[14] AdamMGaweda,CollinFLynch,NathanSeamon,
aTeachingCommunityaroundThem.CRCPress,2019.
GabrielSilvadeOliveira,andAlayDeliwa.Typingexercises
asinteractiveworkedexamplesfordeliberatepracticeinCS [27] GuojingZhou,JianxunWang,CollinF
courses.InProceedingsoftheTwenty-SecondAustralasian Lynch,andMinChi.Towardsclosingtheloop: Bridging
ComputingEducationConference,pages105–113,2020. machine-inducedpedagogicalpoliciestolearningtheories.
InternationalEducationalDataMiningSociety,2017.
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 601