ERIC ED615541: Effects of Algorithmic Transparency in Bayesian Knowledge Tracing on Trust and Perceived Accuracy

2021·0.29 MB·English

by  ERIC

Effects of Algorithmic Transparency in Bayesian
Knowledge Tracing on Trust and Perceived Accuracy
Kimberly Williamson René F. Kizilcec
CornellUniversity CornellUniversity
Ithaca,NY,USA Ithaca,NY,USA
[email protected] [email protected]
ABSTRACT BKT,includinghundredsofarticlesdevotedtoincremental
Knowledge tracing algorithms such as Bayesian Knowledge enhancementsoftheoriginalmodel[20],therearenotmany
Tracing(BKT)canprovidestudentsandteacherswithhelp- real-world applications that use BKT in practice. Some of
ful information about their progress towards learning ob- the most widely used K-12 learning platforms like ASSIST-
jectives. Despite the popularity of BKT in the research ments and Khan Academy decided against using BKT in
community, the algorithm is not widely adopted in educa- favor of simpler models such as N-Consecutive Correct Re-
tional practice. This may be due to skepticism from users sponses (N-CCR) [13]. This raises questions about barri-
anduncertaintyoverhowtoexplainBKTtothemtofoster erstoadoptingknowledgetracingalgorithmsineducational
trust. Weconductedapre-registered2x2surveyexperiment practice. Inparticular,howmuchistherelativecomplexity
(n=170)toinvestigateattitudestowardsBKTandhowthey andopacityofBKTresponsibleforitsslowadoption? Plat-
are affected by verbal and visual explanations of the algo- form providers may be concernedthat educators and learn-
rithm. We find that ostensible learners prefer BKT over a erswillnottrustamodelthatcannoteasilybeexplainedto
simpler algorithm, rating BKT as more trustworthy, accu- them [13, 12, 24, 25, 1].
rate, and sophisticated. Providing verbal and visual expla-
nations of BKT improved confidence in the learning appli- The Technology Acceptance Model (TAM) posits that a
cation, trust in BKT and its perceived accuracy. Findings user’s acceptance and adoption of new technology is based
suggestthatpeople’sacceptanceofBKTmaybehigherthan on its perceived usefulness (PU) and perceived ease of use
anticipated, especially when explanations are provided. (PEOU) [9]. PU and PEOU are beliefs that can be influ-
enced by external factors, such as providing additional in-
formation about a technology. According to TAM, learn-
Keywords
ers’ and educators’ PU and PEOU are essential factors in
Bayesian Knowledge Tracing, Data Visualization, Explain-
the adoption of BKT in practice. Improving their percep-
able AI
tionscouldthereforeincreasetheacceptanceandadoptionof
BKT in real-world applications. Moreover, a better under-
1. INTRODUCTION standing of the mechanisms behind the acceptance of BKT
Knowledge tracing can offer students and teachers a real- is expected to inform the presentation of other knowledge
time understanding of what students have already learned tracing algorithms as well.
and what they are still struggling with [7]. It provides ac-
tionable insights that can lead to better educational out- A large number of knowledge tracing algorithms have been
comes [16]. Among many types of knowledge tracing algo- developed over the years that could benefit from empiri-
rithms, Bayesian Knowledge Tracing (BKT) has been es- cal evidence on how to explain them to users. Recent ad-
tablished and researched most extensively, as evidenced by vances in artificial intelligence have inspired research into
the 114,000 Google Scholar results for ”Bayesian Knowl- more complex algorithms such as deep knowledge tracing
edge Tracing,”17,500 of which published since 2020. BKT (DKT), which uses neural networks [17, 11]. With more
has been tested to help students self-monitor their learning complexalgorithmsthatprovidelessinsightintotheirinner
progress [4, 23], to help teachers understand what students workings, it becomes more important to understand how
have not learned yet [22], and to enable adaptive learning people’s trust in the algorithm and its perceived accuracy
technologiesthatletstudentsskipoverthecontenttheyhave might influence perceptions of usefulness and usability of
mastered [18]. In contrast to the abundance of research on a learning application [1]. Besides BKT and DKT, which
are suitable for modeling understanding and sense-making,
therearealsologisticlearningmodels,suchasAdditiveFac-
tor Models and Performance Factor Analysis [5, 19, 20],
which model memory and fluency [20]. These two types
Kimberly Williamson and Rene Kizilcec “Effects of Algorithmic Trans- of models can also be integrated into one [15]. While there
parencyinBayesianKnowledgeTracingonTrustandPerceivedAccuracy”. are many types of models that can be examined, we choose
2021.In:ProceedingsofThe14thInternationalConferenceonEducational BKT as an example knowledge tracing algorithms that is
Data Mining (EDM21). International Educational Data Mining Society,
relatively simple and popular among researchers.
338-344.https://educationaldatamining.org/edm2021/
EDM’21June29-July022021,Paris,France
338 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)
This research contributes causal evidence to address three plainables.”Theythendesignedanexperimenttodetermine
important research questions. First, do people prefer to the effectiveness between a static and interactive visualiza-
learn with BKT or N-CCR (N-Consecutive Correct Re- tion and found that the static explainable led to a better
sponses) in an ostensible high-stakes test scenario? Second, understanding of the BKT algorithm. More generally, re-
how is their preference related to specific attitudes, includ- search on Open Learning Models (OLMs) has advanced an
ing their confidence in the learning system to do well on a understandingofhowtovisualizeandexplainlearningmod-
test, their trust in the algorithm, and the perceived accu- els[3,2]. OLMsprovideuserswithinteractivevisualizations
racy of the algorithm? And third, how do verbal and/or thatgranttheminsightsintolearningalgorithms,alongwith
visualexplanationsaffectpeople’sattitudesandpreferences the ability to adjust the algorithm. This study will add to
over knowledge tracing algorithms? We answer these re- OLM research by expanding knowledge on how to explain
search questions with data collected from a pre-registered and visualize information to foster positive attitudes.
2x2 factorial survey experiment.
Thecurrentstudyprovidesafoundationalunderstandingof
2. BACKGROUND how individuals perceive BKT compared to N-CCR along
several attitudinal dimensions, and how much verbal and
OneofthesimplestknowledgetracingalgorithmsisN-CCR.
visual explanations of BKT can improve those perceptions.
Itassessesstudentmasterybyevaluatingthenumberofcon-
Our review of prior work informed the following two hy-
secutivecorrectresponsesforaparticularskill. Forexample,
potheses:
the model determines that a student has learned fractions
after correctly answering three fraction questions in a row.
H1. VerbalandvisualexplanationsofBKTleadparticipants
Although N-CCR is easy to understand, its simplicity can
to prefer it over N-CCR.
sometimes make it less accurate than BKT. Still, N-CCR
hasbeenusedinpopularplatforms,includingASSISTments
H2. Verbal and visual explanations of BKT will positively
and Khan Academy [13], and there is mixed evidence as
increase participants attitudes about the BKT algorithm.
to whether BKT outperforms N-CCR at modeling student
learning [8, 10, 13, 21]. Nevertheless, the scientific com-
3. METHODS
munity shows a clear preference for BKT (and other more
The study design, materials, measures and analysis ap-
complexknowledgetracingalgorithms)basedonthealloca-
proacharepre-registeredwiththeOpenScienceFoundation:
tion of research attention.
https://osf.io/7c5zt/. Torefinethestudydesign,measures,
andanalysisplan,weranapilotstudywith26participants
BKT is a two-state Hidden Markov Model where the unob-
andusedbothdescriptiveandinferentialstatisticalanalyses
served hidden state being modeled is student learning, and
tobuildouranalysisplan. Wefirstuseddescriptiveanalysis
for a given knowledge component, a student has a state of
to estimate survey completion time, ensure we had enough
either learned or not learned [6, 17, 11]. Although BKT is
variance in responses, and check that the information pro-
already more sophisticated than N-CCR, critics have sug-
vided to participants was enough information for them to
gestedthatBKTistoosimpleofanalgorithmformodeling
evaluatethealgorithms. Weusedrespondents’answersand
humanlearning. Theypointtodeep(neuralnetwork)learn-
an open-ended question at the end of the survey in which
ingmodelstobetterrepresentallfactorsthatgointostudent
we asked participants for any feedback to improve the sur-
learning [17, 11]. Mao and colleagues [17] found that deep
vey. We took the results from this pilot study to alter the
learningmodelsoutperformedBKTonsomelearningtasks.
visualizationsandinformationprovidedtoparticipantsand
However, they also acknowledge that these gains in perfor-
rephrase some questions to improve clarity. We removed
mancemightnotbeworththelossinmodelinterpretability.
theopen-endedfeedbackquestionfromthesurveyafterthe
While researchers tend to consider BKT as one of the sim-
pilot.
plerandmoreexplainablealgorithmsforknowledgetracing,
practitioners and learners who are the end-users may not
3.1 Participants
share this view.
Participants were recruited from Amazon Mechanical Turk
The explainability of an algorithm, which is partly deter- and received $1.70 for completing a 10-minute survey. The
mined by how transparent, understandable, interpretable it study was advertised as seeking input on test preparation
is, can play an essential role in its adoption into applica- applications. To determine our target sample size of 170,
tions. Barredo Arrieta and colleagues [1] identified these weusedG*Powertoconductapoweranalysis. Ouranalysis
and other reasons for making algorithms more explainable: goals were to obtain 95% power to detect a medium effect
mostrelevanttotheworkonBKTaretrustworthiness,con- size of 0.25 at the standard 0.05 alpha error rate with six
fidence, causality, and accessibility. Prior research on algo- repeated measures and four groups. While we had 170 par-
rithms in education has echoed this finding. Kizilcec [14] ticipants who took the survey, 34 participants either failed
found that increasing transparency by providing users with to answer all of the comprehension questions correctly (29)
additionalinformationaboutanalgorithmmadeuserstrust orhadpriorexperiencewithBKT(4)orboth(1). Analyses
thealgorithmmore(thoughtoomuchinformationcanerode were conducted on the remaining 136 respondents. Table 1
trust). Otherstudieshavemorespecificallyexaminedthein- describes the sample demographics for the sample.
terpretability of BKT in learning applications. Yeung [24]
explored the use of Item Response Theory to make BKT 3.2 Procedure
and deep learning models more explainable, but they have To contextualize the study, participants were provided the
notexaminedhowusersreacttoit. Zhouandcolleagues[25] following narrative with pictures of two sample questions
examined BKT explainability by creating visualization ”ex- taken from the ASSISTments platform:
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 339
(a)Simplevisualization (b)Detailedvisualization
Figure 1: Two versions of a visualization of student performance on questions shown to participants depending on their
condition assignment.
Table 1: Sociodemographics of Participants included in the to ensure that an attentive reader would have no problems
study. answering them correctly.
n % Next, participants saw a short description of the N-CCR
algorithm, which we labeled as 3 Right in a Row (3RR):
Gender Woman 79 58.1
”Atopicwillbeconsideredlearnedonceastudentcorrectly
Man 54 39.7
answers three questions in a row.”A simple table depicting
Transgender Man 1 .7
a sample student’s progression for four topics (table rows)
Gender Variant/Non-Conforming 2 1.5
and questions for each topic (table columns) accompanied
Ethnicity Hispanic 10 7.4
the description. The table looked like Figure 1a. Each
Not Hispanic 126 92.6
cell contained an X or a (cid:88)depending on if the student an-
Race White 100 73.5
swered the question correctly. True to the 3RR algorithm,
Black or African American 10 7.3
each topic was considered learned once three consecutive
American Indian or Alaska Native 2 1.5
questions were answered correctly. At the bottom of the
Asian 16 11.8
page, participants answered several questions about their
Not Listed 5 3.7
attitudes towards the 3RR algorithm (see Measures).
Multiracial 3 2.2
Age 18-24 27 19.9
At this point, participants were randomly assigned to con-
25-34 52 38.2
ditions based on a 2x2 factorial design. There were 33 par-
35-44 37 27.2
ticipants in the No BKT Explanation/BKT Simple Visu-
45-54 8 5.9
alization condition in the final sample, 34 in the No BKT
55-64 11 8.1
Explanation/BKT Detailed Visualization condition, 38 in
Above 65 1 .7
theBKTExplanation/BKTSimpleVisualizationcondition,
and31intheBKTExplanation/BKTDetailedVisualization
condition.
As an admissions requirement for a university program that
you are applying for, you are preparing to take a general The following page mirrored the structure of the previous
knowledge exam. The test is important to you and you need onebutforBKT,providingadescriptionandsamplelearn-
to do as well as possible to get accepted. ingprogressvisualizationbasedontheexperimentalassign-
You have decided to use a test preparation app to help you ment,followedbythesamesetofattitudinalquestionsabout
study for the test. thealgorithm. Next,onthefinalpageofthesurvey,partic-
Akeyfeatureofthetestprepappisthatitpersonalizesthe ipants were asked to compare the two algorithms.
learning experience to help you study efficiently. The app
shows you only questions about topics that you have
not already learned. 3.3 ExperimentalManipulations
Thesystemkeepstrackofyouranswerstoeachquestionand In the no BKT explanationcondition, participants received
automatically moves to the next topic once it deter- this one-sentence description of the BKT algorithm: ”A
mines that you have learned the previous topic. To topicwillbeconsideredlearnedoncethealgorithmestimates
determine if you have learned a topic, the app uses an al- withahighprobabilitythatastudenthaslearnedthetopic
gorithm. Once the algorithm determines you learned a based on their responses up to that point.”In the BKT ex-
topic,itwillstopgivingyoustudyquestionsaboutit. Thus, planation condition, participants additionally received the
it also determines the speed at which you progress in your following information about the BKT algorithm:
test prep.
We would like to get your opinions about the two different After every question you answer, the Bayesian Knowledge
algorithmstounderstandwhichoneyoufindmoreaccurate Tracing algorithm estimates the probability that you have
and trustworthy. nowlearnedatopicusingaprobabilisticmodelthataccounts
for the following data:
On the next page, participants answered three multiple- – an initial probability that you have learned the topic based
choicecomprehensionquestions: (1)Howdoesthetestprep on your first answer: it is higher if you answered correctly
app determine what questions to give you? (2) What de- – a correct guess probability: e.g., 50% for a true/false
termines how quickly you are going to be done with test question
prep? (3) What happens when the system determines that – a slip probability for answering incorrectly even though
you have learned a topic? We pre-tested these questions you already learned the topic
340 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)
–thedifficultyofquestionsyouhaveansweredbasedonhow
BKT Explanation/
many people have answered them incorrectly BKT Detailed Visualization
– performance data such as the number of hints that you BKT Explanation/
BKT Simple Visualization
asked for and the time it took you to answer the question No BKT Explanation/
BKT Detailed Visualization
No BKT Explanation/
Using all of this information, the algorithm estimates
BKT Simple Visualization
the probability that you have learned a topic. If the proba-
4 5 6 7
bility is above 95%, the algorithm moves you on to the next Avg. Algorithm Preference
topic. (1=Strongly prefer 3RR, 7=Strongly prefer BKT)
In the BKT simple visualization condition, participants re- Figure 2: Average algorithm preference by condition.
ceived a simple table depicting a sample student’s learning
progressmirroringtheoneshowninthe3RRalgorithm(Fig-
ure 1a). In the BKT detailed visualization condition, the last planned analysis evaluated H2 by running a linear re-
sametablewasenhancedtoshowtheestimatedprobability gression on each attitudinal construct with the conditions
of having learned the topic using a color scale (Figure 1b). as the predictor variables and the attitudinal construct as
To make the visualizations realistic, we ran a BKT algo- the dependent variable. While the interpretation of linear
rithm over a sample of ASSISTments data and used a 95% regressionoutputisclearandfamiliar,weacknowledgethat
probability to determine mastery. our measures are ordinal and not strictly continuous. We
confirmed that analysis by ordinal logistic regression yields
3.4 Measures equivalent results.
Wemeasuredparticipants’attitudestowardseachalgorithm
For the open-ended question asking participants why they
using six items rated on 5-point unipolar response scales
choose their preferred algorithm, we planned to use simple
(’Notatall’,’Somewhat’,’Moderately’,’Very’,’Extremely’):
thematiccoding. Whileweusedthepilotdatatocreateour
analysis plan, we did remove all pilot data from the final
Confidence: ”How confident are you that the test prep app
dataset.
with this algorithm will prepare you to do very well on the
test?”
4. FINDINGS
Understanding: ”How well do you understand how this al-
gorithm determines if you have learned a topic?” First, we examine which algorithm participants preferred
Sophistication: ”Howcomplexisthisalgorithmfordetermin- overall. Figure 2 shows their average preference in each
ing if you have learned a topic?” condition, which varied between 5 (i.e. Slightly prefer
Accuracy: ”Howaccurateisthisalgorithmatdeterminingif BKT) and 6 (i.e. Moderately prefer BKT). While there
you have learned a topic?” is a suggestive pattern that providing more explanation
Trust: ”Howmuchdoyoutrustthisalgorithmtodetermine for BKT strengthens the preference for BKT, this pattern
what you have learned?” was not statistically significant (linear regression: F3,132 =
Speed: ”Howquicklydoyoulearnthematerialsforthetest 0.7455,p = 0.5268). This means the data do not support
using this algorithm?” H1.
At the end of the survey, participants rated their general
confidence soph trust
preferenceoverthetwoalgorithmsinresponsetothefollow-
ingquestion: ”Nowthatyouhavelearnedaboutthe3Right understand acc fast
in a Row (3RR) and Bayesian Knowledge Tracing (BKT)
algorithms, which one would you prefer to use for your test T(+))
prep?” Response options were on a 7-point bipolar scale: K 2
B
’Strongly prefer 3RR’, ’Moderately prefer 3RR’, ’Slightly o
prefer 3RR’, ’Neither prefer 3RR nor BKT’, ’Slightly pre- −) t 1
R(
ferBKT’,’ModeratelypreferBKT’,’StronglypreferBKT’. R
3
Participants were invited to provide a rationale for their e ( 0
preferenceusinganopen-endedquestion: ”Pleasetelluswhy ns
o
you prefer the algorithm that you choose above.” p
s−1
e
R
3.5 AnalyticalApproach g.
v−2
A
We used the pilot study data to finalize our analysis plan Prefer 3RR/ Slightly/ Strongly Prefer BKT
Neither Moderately Prefer BKT [7] (n=54)
by developing our inferential analysis. For H1, we decided [1,4] (n=31) [5,6] (n=51)
to use linear regression to understand if the conditions had Algorithm Preference Bin
(1=Strongly prefer 3RR, 7=Strongly prefer BKT)
an association effect on the participants’ overall preference.
We used the conditions as the predictor variables and the
Figure 3: Average response on each measure at three levels
preferenceastheoutcomevariable. Wenextdecidedtouse
of preference: Prefer 3RR to Neither, Slightly and Mod-
multiple linear regression to understand the association be-
erately Prefer BKT, and Strongly Prefer BKT. We choose
tween the attitudinal constructs and algorithm preferences.
these groupings because each group represents approxi-
This analysis used the attitudinal constructs as the predic-
mately 1/3 of the sample.
tor variables with preference as the outcome variable. The
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 341
BKT Simple Visualization BKT Detailed Visualization
+))
T(
K
B Confidence Accuracy Trust Sophistication
o
−) t
R(
R2
3
e (
or
c1
S
e
s
n
o
p
s0
e No BKT No BKT No BKT No BKT
R BKT Explanation BKT Explanation BKT Explanation BKT Explanation
g. Explanation Explanation Explanation Explanation
v
A
Figure 4: Average differences (BKT score - 3RR score) in significant attitudinal constructs as a function of the randomly
assigned conditions. Positive scores indicate a higher score for BKT.
Next, we examine how algorithm preference is related to 5. DISCUSSIONS
the six attitudinal measures: confidence in the learning Thisstudyinvestigatedpeople’sattitudestowardsBKTrel-
system, the sophistication of the algorithm, trust, under- ativetoamorestraightforwardknowledgetracingalgorithm
standing, accuracy, and speed. We use the repeated mea- and tested the effect of additional information via explana-
sures design of our study by computing the difference score tions and visualizations on their attitudes. Understanding
for each question: subtracting the participant’s 3RR re- how students might perceive the algorithms used in their
sponse from the BKT response. Figure 3 shows the av- learning applications is a crucial issue for the adoption and
erage response on each measure at three levels of prefer- usabilityofthesetools[9]. Theresultsprovideevidencesup-
ence: Prefer 3RR to Neither, Slightly and Moderately Pre- porting our second hypothesis that additional explanations
ferBKT,andStronglyPreferBKT.Wechoosethesegroup- improve key attitudinal measures of confidence, perceived
ingbecauseeachgrouprepresentsapproximately1/3ofthe accuracy, trust, and sophistication. Qualitative data from
sample. All measures are positively correlated with pref- participants echo this result:
erence as evidenced by their positive slopes (all Pearson’s
r > 0.325,p < 0.0001), but accuracy, confidence, and trust
For something high stake, I’d only trust the
arecorrelatedmorestrongly(r>0.739). Thishighlightsthe
methods that employs a variety of learning
importanceofthesethreeconstructsindeterminingpeople’s
modalities. The analytics for such should match
preferenceoverthealgorithms. Infact,thesixmeasuresex-
the complexities of my learning process as well
plain 66.7% of the variance in preferences (multiple linear
as the nature of the material I’m learning. The
regression: F =43.07,p<0.0001).
6,129 BKT would put me more at ease than the quick
route of the 3RR approach. (Participant as-
Lastly,weexaminehowtheprovisionofverbaland/orvisual
signed to BKT Explanation and BKT Detailed
explanations influenced participant attitudes about BKT.
Visualization who had high confidence, sophisti-
Figure 4 shows the average response in each condition for
cation, accuracy, and trust in BKT relative to
thefourmeasuresthatweresignificantlyaffectedbythein-
3RR)
tervention (i.e., relative understanding and speed did not
change significantly at p < 0.1). We find that confidence
in the learning application with BKT (relative to 3RR) im- Surprisingly,wedidnotfindasignificantincreaseinpeople’s
proved when both a detailed explanation and visualization preference for BKT (H1), even though we found that algo-
were provided (F = 2.88,p = 0.03844). Likewise, the rithmpreferenceisexplainedlargelybypeople’sperceptions
3,132
perceivedaccuracyofBKTimprovedwithbothtypesofex- ofaccuracy,trust,andconfidence. ThispreferenceforBKT
planation provided (F = 3.28,p = 0.02305). Trust in regardless of experimental condition is furthered explained
3,132
BKT improved by providing a detailed explanation, espe- by the qualitative responses from participants:
cially when complemented with the detailed visualization
(F3,132 =2.346,p=0.07575). Finally, and not surprisingly, Idon’tbelievethe3RRalgorithmisatallbenefi-
the more detail was provided, the more sophisticated BKT cialtothestudentattemptingtolearnthetopic.
was perceived to be (F3,132 = 11.17,p < 0.001). This pro- If the student just happens to get 3 exception-
vides evidence in support of H2. ally easy questions in a row, the algorithm will
assume that the student has learned the topic
In all of our analyses, we tested for the presence of demo- whichislikelynotentirelytrue. (Participantas-
graphicheterogeneityinresults. However,nosignificantde- signedtoNoBKTExplanationandBKTSimple
mographic sources of variation were found in our sample. Visualization who Strongly Preferred BKT)
342 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)
Nevertheless, participants generally preferred to use the 2020.
BKT algorithm regardless of the experimental treatment. [3] S. Bull and J. Kay. Open learner models. In Advances
in intelligent tutoring systems, pages 301–322.
Since confidence, trust, and accuracy are important to a Springer, 2010.
user’s preference for BKT, it was notable that those three [4] A. Bunt and C. Conati. Probabilistic student
measures were affected by the experimental manipulations. modelling to improve exploratory behaviour. User
Consistent with prior studies of explainability and trans- Modeling and User-Adapted Interaction,
parency in algorithmic systems, we also found that when 13(3):269–309, 2003.
more information about an algorithm is presented to peo- [5] H. Cen, K. Koedinger, and B. Junker. Learning
ple,theybelievethealgorithmstobemoretrustworthyand factors analysis–a general method for cognitive model
accurate, leading the user to have more confidence in the evaluation and improvement. In International
algorithm [1, 14]. This confidence can, in turn, increase the Conference on Intelligent Tutoring Systems, pages
useofapplicationsshowntoimproveeducationaloutcomes. 164–175. Springer, 2006.
OurresultsfurtherhighlighttheimportanceofOLMtopro-
[6] B. Choffin, F. Popineau, Y. Bourda, and J.-J. Vie.
vide more transparency to gain user trust and confidence.
Das3h: Modeling student learning and forgetting for
In addition, future educational applications using complex
optimally scheduling distributed practice of skills. In
knowledgetracingalgorithmsshouldincludedetailedverbal
International Conference on Educational Data Mining
and visualization explanations of the algorithm to improve
(EDM 2019), 2019.
confidence and trust in the application.
[7] A.T.CorbettandJ.R.Anderson.Knowledgetracing:
Modeling the acquisition of procedural knowledge.
Frankly, we did not expect to find such strong support for
User modeling and user-adapted interaction,
BKT going into this study. Many learning platforms that
4(4):253–278, 1994.
have the capacity to implement BKT or even more com-
[8] Y. B. David, A. Segal, and Y. K. Gal. Sequencing
plex algorithms have opted not to do so. Our informal un-
educational content in classrooms using Bayesian
derstanding was that this was largely due to concerns that
knowledge tracing. In Proceedings of the Sixth
users, such as math teachers who use ASSISTments to as-
International Conference on Learning Analytics &
sign homework, would not understand BKT and not trust
Knowledge - LAK ’16, pages 354–363, New York, New
it and lose confidence in the platform. This understanding
York, USA, 2016. ACM Press.
ledustoexpectthatparticipantswouldreportapreference
[9] F. D. Davis. Perceived usefulness, perceived ease of
for the simpler algorithm and report that they find it more
use, and user acceptance of information technology.
trustworthyandhavemoreconfidenceinasystemthatuses
MIS quarterly, pages 319–340, 1989.
it. Therefore, we are surprised by our positive findings for
BKTandproposefutureresearchdirectionstofollowupon [10] S. Doroudi and E. Brunskill. Fairer but Not Fair
these results. Enough On the Equitability of Knowledge Tracing. In
Proceedings of the 9th International Conference on
Future work in this area should explore different partici- Learning Analytics & Knowledge, pages 335–339, New
pantpopulationsandscenarios. Weareplanningtorunthis York, NY, USA, mar 2019. ACM.
studywithstudentsamplestoseeifwecanreplicatethere- [11] T. Gervet, K. Koedinger, J. Schneider, T. Mitchell,
sults. Whileweaskedourparticipantstoputthemselvesin et al. When is deep learning the best approach to
the shoes of a student needing to prepare for a high-stakes knowledge tracing? Journal of Educational Data
test,runningthisexperimentonastudentpopulationmight Mining, 12(3):31–54, 2020.
make the scenario more realistic to the participants. Addi- [12] W. J. Hawkins, N. T. Heffernan, and R. S. J. D.
tionally, given that many tasks on Mechanical Turk involve Baker. Learning bayesian knowledge tracing
subjects annotating machine learning datasets, this group parameters with a knowledge heuristic and empirical
of participants may have more favorable attitudes to algo- probabilities. In S. Trausan-Matu, K. E. Boyer,
rithms. Another areafor future work includes changing the M. Crosby, and K. Panourgia, editors, Intelligent
scenario. Weranthestudyfromtheviewpointofastudent. Tutoring Systems, pages 150–155, Cham, 2014.
However, we acknowledge that intelligent tutoring systems Springer International Publishing.
deployed in classrooms also need the trust and confidence [13] K. Kelly, Y. Wang, T. Thompson, and N. Heffernan.
of the teachers administering the applications. We plan to Defining Mastery: Knowledge Tracing Versus
rewritethescenariotoallowparticipantstotakeontherole N-ConsecutiveCorrectResponses.In8th International
of a teacher. Conference on Educational Data Mining, 2015.
[14] R.F.Kizilcec.HowMuchInformation? InProceedings
6. REFERENCES of the 2016 CHI Conference on Human Factors in
[1] A. Barredo Arrieta, N. D´ıaz-Rodr´ıguez, J. Del Ser, Computing Systems, pages 2390–2395, New York, NY,
A. Bennetot, S. Tabik, A. Barbado, S. Garcia, USA, may 2016. ACM.
S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, [15] S. Klingler, T. K¨aser, B. Solenthaler, and M. Gross.
and F. Herrera. Explainable Artificial Intelligence On the performance characteristics of latent-factor
(XAI): Concepts, taxonomies, opportunities and and knowledge tracing models. International
challenges toward responsible AI. Information Fusion, Educational Data Mining Society, 2015.
58(October 2019):82–115, jun 2020. [16] K. R. Koedinger and V. Aleven. Exploring the
[2] S. Bull. There are open learner models about! IEEE assistance dilemma in experiments with cognitive
Transactions on Learning Technologies, 13(2):425–448, tutors. Educational Psychology Review, 19(3):239–264,
Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021) 343
2007. jul 2017. ACM.
[17] Y.Mao.Deeplearningvs.bayesianknowledgetracing: [22] S. Ritter, M. Yudelson, S. E. Fancsali, and S. R.
Student models for interventions. Journal of Berman. How mastery learning works at scale. In
educational data mining, 10(2), 2018. Proceedings of the Third (2016) ACM Conference on
[18] E. Milla´n and J. L. P´erez-De-La-Cruz. A bayesian Learning@ Scale, pages 71–79, 2016.
diagnostic algorithm for student modeling and its [23] S. Schiaffino, P. Garcia, and A. Amandi. eteacher:
evaluation. User Modeling and User-Adapted Providing personalized assistance to e-learning
Interaction, 12(2):281–330, 2002. students. Computers & Education, 51(4):1744–1754,
[19] P. I. Pavlik Jr, H. Cen, and K. R. Koedinger. 2008.
Performance factors analysis–a new alternative to [24] C.-K. Yeung. Deep-IRT: Make Deep Learning Based
knowledge tracing. Online Submission, 2009. Knowledge Tracing Explainable Using Item Response
[20] R. Pela´nek. Bayesian knowledge tracing, logistic Theory. EDM 2019 - Proceedings of the 12th
models, and beyond: an overview of learner modeling International Conference on Educational Data Mining,
techniques. User Modeling and User-Adapted pages 683–686, apr 2019.
Interaction, 27(3):313–350, 2017. [25] T. Zhou, H. Sheng, and I. Howley. Assessing post-hoc
[21] R. Pela´nek and J. Rˇiha´k. Experimental Analysis of explainability of the bkt algorithm. In Proceedings of
Mastery Learning Criteria. In Proceedings of the 25th the AAAI/ACM Conference on AI, Ethics, and
Conference on User Modeling, Adaptation and Society, AIES ’20, page 407–413, New York, NY,
Personalization, pages 156–163, New York, NY, USA, USA, 2020. Association for Computing Machinery.
344 Proceedings of The 14th International Conference on Educational Data Mining (EDM 2021)