Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. User-Centered ...

13 Pages·2017·0.42 MB·English

Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. User-Centered Design
and Evaluation of a Human-in-the-Loop Topic Modeling System. Intelligent User Interfaces,2018,12pages.
@inproceedings{Smith:Kumar:Boyd-Graber:Seppi:Findlater-2018,
Author = {Alison Smith and Varun Kumar and Jordan Boyd-Graber and Kevin Seppi and Leah Findlater},
Booktitle = {Intelligent User Interfaces},
Year = {2018},
Title = {User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System},
Url = {http://umiacs.umd.edu/~jbg//docs/2018_iui_itm.pdf},
}
Alison won a best student paper honorable mention (3 out of 300)
Downloaded from http://umiacs.umd.edu/~jbg/docs/2018_iui_itm.pdf
Contact Jordan Boyd-Graber ([email protected]) for questions about this paper.
1
Closing the Loop: User-Centered Design and Evaluation of
a Human-in-the-Loop Topic Modeling System
AlisonSmith VarunKumar JordanBoyd-Graber
UniversityofMaryland UniversityofMaryland UniversityofMaryland
ComputerScience ComputerScience CS,iSchool, UMIACS, LSC
CollegePark,Maryland,USA CollegePark,Maryland,USA CollegePark,Maryland,USA
[email protected] [email protected] [email protected]
KevinSeppi LeahFindlater
BrighamYoungUniversity UniversityofWashington
ComputerScience HumanCenteredDesignand
Provo,Utah,USA Engineering
[email protected] Seattle,Washington,USA
[email protected]
ABSTRACT text[7]bymodelingtheintuitionthatgroupsofwordsthat
Human-in-the-looptopicmodelingallowsuserstoguidethe formacommonthemeappearmoreoftentogetherthannot;
creationoftopicmodelsandtoimprovemodelqualitywithout thediscoveredtopicsaresetsofwords(e.g.,“football,touch-
havingtobeexpertsintopicmodelingalgorithms. Priorwork down,NFL,...”),andeachdocumentcancontainmultiple
inthisareahasfocusedeitheronalgorithmicimplementation topics. However, traditional topic models can include poor
withoutunderstandinghowusersactuallywishtoimprovethe qualitytopics[30]orcanbemisalignedwithadomainexpert’s
modeloronuserneedsbutwithoutthecontextofafullyin- understandingofthecorpus[23]. Pre-processingtechniques
teractivesystem.Toaddressthisdisconnect,weimplemented andparametertuning[36]canimprovemodelquality,buttypi-
asetofmodelrefinementsrequestedbyusersinpriorwork callyrequiremanytime-consumingiterationsandcanonlybe
andconductedastudywithtwelvenon-expertparticipantsto performedbyanalgorithmexpert. Human-in-the-looptopic
examinehowendusersareaffectedbyissuesthatarisewitha modeling(HL-TM)aimstosolvetheseproblemsbyallowing
fullyinteractive,user-centeredsystem. Astheseissuesmirror enduserswhoarenotexpertswithtopicmodelingalgorithms
thoseidentifiedininteractivemachinelearningmorebroadly, todirectlyrefinethemodel(e.g.,changingwhichwordsare
suchasunpredictability,latency,andtrust,wealsoexamined includedinatopic,ormergingorsplittingtopics).
interactivemachinelearningchallengeswithnon-expertend
HL-TM has largely implemented topic model refinements
usersthroughthelensofhuman-in-the-looptopicmodeling.
based on what algorithm developers assume users want, as
Wefoundthatalthoughusersexperienceunpredictability,their
well as algorithm concerns, such as which refinements are
reactionsvaryfrompositivetonegative,and,surprisingly,we
the most straightforward to implement [23, 10, 21, 26]. In
didnotfindanycasesofdistrust,butinsteadnotedinstances
contrast,Leeetal.[27]employedauser-centeredapproachto
whereusersperhapstrustedthesystemtoomuchorhadtoo
identifyasetoftopicrefinementoperationsthatusersexpect
littleconfidenceinthemselves.
tohaveinaHL-TMsystem. However,becausenoexistingim-
plementationsupportedthesetofrefinementoperations(e.g.,
INTRODUCTION
addatopicword,changewordorder,mergetopics),theyused
Topicmodelinghelpsusersunderstandvastdocumentcollec-
Wizard-of-Ozrefinements: theresultingtopicswereupdated
tionswhentherearetoomanydocumentsortoolittletimeto
superficially—not as the output of a data-driven statistical
readindividualdocuments: ajournalistprocessingthereports
model(thegoaloftopicmodels). Thus,thisstudyislimited,
surrounding breaking news or a legal team finding interest-
asitignoresthecommoninteractivemachinelearningissues
inge-mailsduringdiscovery. Topicmodelingautomatically
thatariseinafullyinteractivesystem—suchas,unpredictabil-
discoversthethemes(topics)inlargecorporaofunstructured
ity,latency,andcomplexity—thatcanaffectuserexperience
andimpacthowusersinteractwithaHL-TMsystem.
Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalor
classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed Toaddresstheselimitationsweimplementedabroadsetofre-
forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitation
finementsoperations[27,31]inasinglesystem:addword,re-
onthefirstpage.CopyrightsforcomponentsofthisworkownedbyothersthanACM
mustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orrepublish, moveword,removedocument,changewordorder,splittopic,
topostonserversortoredistributetolists,requirespriorspecificpermissionand/ora mergetopics,andaddtostopwords. UnlikeinLeeetal.[27],
[email protected].
IUI2018,March7–11,2018,Tokyo,Japan. whererefinementswereimmediatelyappliedonlyinthein-
Copyrightisheldbytheowner/author(s).PublicationrightslicensedtoACM. terfaceandwerenotsavedtothebackingmodel,inourstudy
ACMISBN978-1-4503-4945-1/18/03...$15.00.
https://doi.org/10.1145/3172944.3172965
participants chose when to “save” and update the backing weshowhowtoadjusttheDirichlethyperparameters,α and
modelwiththeirrefinements. Weevaluatedthisfullyinterac- β,toencodeuserinformationaboutdocuments(α)andtopics
tive,user-centered HL-TM systemwith12participantswho (β).
werenotexpertsintopicmodeling. Thissystemisalsofast
Giventhismodel, weneedtofindthedistributionthatbest
enoughtosupport(andstudy)interactiveuseoffine-grained
explainsourobserveddocuments. GriffithsandSteyvers[17]
topic modifications, unlike prior systems [21, 10, 23] (e.g.,
propose a collapsed Gibbs sampling based method. In col-
earlyHL-TMsystemstook5 50secondsforeachupdate[23],
− lapsedGibbssampling,theprobabilityofatopicassignment
whereasourstook.09 .63secondstoupdateaftersavingdur-
ingtheuserstudy). In−additiontoobservinghowparticipants z=tindocumentdgivenanobservedtoken,wi,andtheother
topicassignments,z_,is
employedtherefinementstoimproveatopicmodel,wequali-
tativelyassessedoverallexperienceandthepotentialimpacts
ofcommoninteractivemachinelearningchallenges,suchas nw,t+β
P(z=t z_,w)∝(n +α) . (1)
unpredictabilityandlatency. | d,t n +Vβ
t
Participantssubjectivelyandobjectivelyimprovedtopicmodel
qualityusingthegivenrefinements. SimilartoLeeetal.[27], Here, z_ are the topic assignments of all other tokens, nd,t
simplerefinements,suchasremovewordandchangewordor- is the number of times topic t is used in document d, nw,t
derweremostcommon,however,unlikeLeeetal.[27],usage is the number of times token w is used in topic t, and nt
didnotalignperfectlywithperceivedutility,asparticipants is the marginal count of the number of tokens assigned to
foundchangewordordertobeoneoftheleastusefulrefine- topict.Fortraditionaltopicmodels,theGibbssamplerassigns
ments. Ourfullyinteractivesystemalsoexposedpreviously latent topics Z for all tokens in the corpus, going over all
hiddenissues: unexpectedresultsfromusingrefinementsand the documents in the corpus repeatedly until the algorithm
aninabilitytotrackmodelchanges.Participantsalsovariedin converges.
whenandwhytheysavedtheirlocalupdatestotheunderlying
Thestateofthesamplerrepresentsthealgorithm’sbestguess
model. Additionally,participantswhohadmoretrustinthe
ofthetopicassignmentsforeverytoken.Addingahumanin
system(orperhapslessconfidenceinthemselves)werenotas
thelooprequirestheusertobeabletoinjecttheirknowledge
frustratedbyunpredictabilityasothers.
andexpertiseintothesamplingequationtoguidethealgorithm
This work makes the following contributions: (1) efficient tobettertopics.
asymmetric prior-based implementation of a broader set of
user-centeredrefinementoperationsthanhasbeenpreviously
implementedinasinglesystem;(2)extensionofaprevious, Human-in-the-loopTopicModeling
limitedHL-TMuserstudy[27]tothisfullyinteractivesystem; Comparedtotraditionaltoolsthatvisualizestatictopicmod-
(3)understandingofhowinteractivemachinelearningissues els [15, 13, 9], HL-TM provides mechanisms to allow end
such as unpredictability, trust, and lack of control impact userstorefinechangingtopicmodels. Numeroustoolshave
HL-TMusageanddesign;and(4)designprinciplesforuser- beendesignedaroundthisconcept,eachimplementingavari-
centeredHL-TMsystems. etyofrefinements,butwithoutextensiveuserstudiesonreal
world-tasks to show the refinement implementations match
BACKGROUND userexpectations. Andrzejewskietal.[3],buildingonBoyd-
Topicmodelingautomaticallyidentifiesthethemesortopics Graberetal.[6],introducedastatisticalframeworkforusers
thatoccurinacollectionofdocuments. Acommonapproach tospecifypairsofwordsthatshouldorshouldnotbelongto
isLatentDirichletAllocation(LDA)[4],whichisagenerative the same topic through “must-link” and “cannot-link” con-
statisticalmodelthatmodelseachdocumentasadistribution straints. This framework has been extended by numerous
of topics and each topic as a distribution of words. Here, HL-TMtools[23,11,33];however,thisapproachlimitsthe
weprovideabriefbackgroundof LDA andexisting HL-TM possible set of refinements that can be supported and does
systems. We also discuss design challenges for interactive not match users expectations in that end users typically do
machinelearningsystemsmoregenerally. notthinkofspecifyingmodelrefinementsaspairwiseword
correlations[27].
LatentDirichletAllocation
Other HL-TM tools take alternative approaches, such as
LDA[4]isagenerativemodel,whichassumesthateachdocu-
UTOPIAN[10],whichusesnonnegativematrixfactorization
mentdisgeneratedfromafixedsetofktopics. Eachtopicis
andsupportscreating,splitting,andmergingtopics,inaddi-
amultinomialdistribution,φ ,overthevocabulary,V. Each
z tiontochangingwordweights. Lundetal.[28]alsousesa
instanceofaword,ortoken,w,isgeneratedbysamplinga
i fastmatrix-factorizationapproachbutonlysupportsalimited
topicassignmentz fromthedocument’stopicdistributionθ ,
i d set of interactions. ConVisIT [21] uses fragment quotation
followedbysamplingtheselectedtopic’sdistributionφ ,to
zi graphsandsupportssplittingandmergingtopics. Finally,Dis-
generateatokenw.
i tillery [31] uses informative priors to support merging and
Themultinomialdistributionsθ andφ aredrawnfromDirich- removingtopics,aswellasaddingandremovingtopicwords
d z
letdistributionsthatencodesparsity—howmanywordsyou andaddingtostopwords.However,nopriorimplementation
expecttoseeinatopicorhowmanytopicsinadocument— supportsthecompletesetofrefinementoperationsrequested
andcanalsoincorporateexpertknowledgefromusers. Below, byusers[27].
InteractiveMachineLearningDesignChallenges InjectinginformationhappensthroughmodifyingtheDirichlet
Interactivemachinelearning—alsoknownasmixed-initiative parameters for each document, α, and each topic, β.1 Re-
orhuman-in-the-loopsystems—incorporatehumaninputto calltheGibbssamplingconditionalprobabilityP(z=t z_,w)
|
produce an output or a decision. Designers of these must (1), which has two parts: how much a document likes a
accountforchallengesinherentinmachinelearning,which topic—(nd,t+αd,t)—and how much a topic likes a word—
deviates from traditional user interface design. While user (nw,t+βw,t). The priors are added to the topic assignment
interfaces should provide immediate updates [29], be pre- counts; thankstotheconjugacyofmultinomialandDirich-
dictable[18],ensuretheuserfeelsincontrol,andreduceshort letdistributions,thesepriorsaresometimescalled“pseudo-
termmemoryload[34],interactivemachinelearningsystems counts”. OurHL-TMtakesadvantageofthisbycreatingpseu-
arecommonlyslow,unpredictable,sharecontrolbetweenuser docountstoencouragethechangesuserswanttoseeinthe
andsystem,andarecomplex[2,19]. topic.
Priorworkhasexaminedhowpredictability,control,andac- Therefinementoperationsare:
curacyaffecttheuserexperience. Gajosetal.[14]showed
1. Addword: toaddthewordwtotopict weforgetwfrom
thatincreasingpredictabilityandaccuracyleadtoimproved
allothertopicsandencouragetheGibbssamplertoassign
satisfaction, while Kangasraasio et al. [24] showed that al-
topic t for all of the word’s tokens, w. For the former,
lowinguserstoseethepredictedeffectsofanactionbefore i
weforgetthetokens’topicassignments. Forthelatter,we
committingtoitcanimprovetaskperformanceandacceptance.
increasethepriorofwint bythedifferencebetweenthe
Similarly,usersofPeerFinder—atoolthatrecommendssimi-
topic-wordcountsofwandtopic’stopwordw intopict
larstudentsbasedonacademicprofiles—weremoreconfident 0
(i.e.,n n ).
andengagedwhengivenmorecontrolevenwiththenegative w0,t− w,t
2. Removeword: toremovethewordwfromtopict weneed
effectofaddedcomplexity[12]. Inthiswork,weexplorehow
to forget all the word’s tokens w from t and discourage
usersareaffectedbycomplexity,unpredictabilityandlackof i
the Gibbs sampler from reassigningt to the word w. To
controlinHL-TM.
discouragethesamplerfromassigningwtot withahigh
probability,weassignaverysmallprior,2ε,towint.
REFINEMENTIMPLEMENTATION 3. Changewordorder: toreorderwordw2 toappearbefore
Priorwork[27,31]identifiedasetofrefinementsthatusers word w1 in topic t we need to ensure that w2 is ranked
expectedtobeabletouseina HL-TM system. Thereisno higherthanw1inthetopict. Toenforcethis,weincrease
implementationforthebroadsetofuserpreferredrefinements, the prior of w2 in t by the difference between the topic-
so Lee et al. [27] simulated refinements using a Wizard-of- wordcounts(i.e. nw1,t−nw2,t). Intuitively,thisoperation
Oz method. To truly evaluate user experience with a fully resemblesprovidingsupplementalcountstow2 sothatit
functionalHL-TMsystem,weimplementedsevenrefinements rankshigherthanw1inthetopic.
requested by users: add word, remove word, change word 4. Removedocument: in LDA, eachdocumentcanberepre-
order,removedocument,splittopic,mergetopic,andaddto sentedasaprobabilitydistributionovertopics.Toremove
stopwords. thedocumentdfromtopict,weforgetthetopicassignment
for all words in the document d and assign a very small
Theserefinementsincludethesixtoprefinementsidentified, prior,2ε,tothetopict inα .
d
but not implemented, by Lee et al. [27], except for merge 5. Mergetopic: mergingtopicst andt meansthemodelwill
1 2
words. Mergewordswasdiscussedinthatstudyasameans haveacombinedtopicthatrepresentsbotht andt . We
1 2
fororganizingtopicwordsintheinterfaceratherthanadeeper assignt toalltokensthatwerepreviouslyassignedtot .
1 2
specificationthatshouldbeimplementedinthemodel. We Thiseffectivelydeletest fromthemodelanddecrements
2
alsoincludedtworefinementsthatwerenotsuggestedbyusers thenumberoftopics.
inLeeetal.[27],perhapsduetothatstudy’smethod: merge 6. Split topic: to split topic t the user provides a subset of
topicsdidnotarisebecauseusersonlyrefinedindividualtopics thetopic’swords,orseedwords,whichneedtobemoved
andaddtostopwordsmayhavebeenoverlookedbecausethe fromtheoriginaltopic,t,toanewtopic,t . Toimplement
n
studyusedagenericcorpuswithawell-curatedstopwords this,weinvalidatetheoriginaltopicassignmentofallseed
list. words,createanewtopicbyincrementingthenumberof
topics,andassignalargepriorforeachoftheseedwords,
w ,inthenewtopic,t . TheGibbssampler’sjobistosort
RefinementImplementation s n
whichwordslandinwhichofthenewchildtopics.
Whenauserprovidesfeedbacktoatopicmodel,weviewthis
7. Addtostopwords:addingthewordwtoglobalstopwords
ascorrectinganerrorthemodelmade. Wecanthusdivide
removes w from all topics. We exclude that w from the
thisfeedbackintotwobroadclasses: forgettingbadthingsthe
vocabulary,V. This ensures that the Gibbs sampler will
modellearnedandinjectingnewknowledgeintothemodel.
ignorealloccurrencesofwinthecorpus.
Forgetting is accomplished by invalidating the topic-word
assignmentsfortargetedwordtypes. Thisisequivalenttothe
1To implement these refinement operations, we make use of the
modelseeingthatwordfortheveryfirsttime,allowingitto
makebetterdecisions. Intandemwithforgetting, injecting vKecdtiomreinnsteiorpnraeltavteicotnor(rfaotrheeracthhadnosccuamlaern)todftahnedseφpkriisorasV.Tdhiumse,nαsdioinsaal
provides hints that encourage the algorithm to make better vectorforeachtopick.
decisionsgoingforward. 2Weuseε=0.000001forourexperiments.
Interface DatasetandTopicModel
TheHL-TMuserinterface(Figure1)representsatopicmodel Weusedaseparatedatasetandmodelforthetrainingandtest
as a list of topics on the left panel, each displayed as their tasks. Fortrainingwegeneratedamodelwith10topicsfrom
first three words. Selecting any topic in the list shows the adatasetof2,225BBCnewsarticlescorrespondingtostories
full topic view in the right panel, which consists of the top infivetopicalareas(business,entertainment,politics,sports,
20 topic words and snippets of the top 40 topic documents. tech)from2004 2005[16]. ForthetestweusedtheTwitter
Documents are ordered by their probability for the topic t USAirlineSenti−mentdatasetfromKaggle,3whichincludes
giventhedocumentd, or p(t d). Eachword, w, isordered 14,485totaltweetsfromFebruary2015directedtosixpopu-
|
and sized by its probability for the topic t, or p(w t); this larairlines(American,Delta,Southwest,United,USAirways,
|
simplewordlistrepresentationprovidesusersaquicktopic VirginAmerica).Thedatasetincludesmanuallyappliedlabels
understanding[1,35]. Hoveringorclickingontopicwords organizing the tweets into “positive” (2,363 tweets), “neu-
highlightsthewordinthedisplayeddocumentsnippets. tral”(3,099tweets),and“negative”(9,178tweets)sentiment
categories. Wemodeledthe9,178negativesentimenttweets
Usersrefinethetopicmodelusingsimpleinteractivemecha-
with10topicsusingastandardstopwordslist4and300Gibbs
nisms. Werequireuserstoclick“save”toincorporatetheir
samplingiterations. Foreachsubsequentupdateduringthe
specifiedrefinementsinsteadofapplyingthemimmediately
task,30Gibbssamplingiterationswererun. Table1shows
becausethesystemdoesnotsupportrevertingthemodelafter
theinitialsetoftopics(henceforthT1–T10).Weautomatically
an update (we discuss batch vs. immediate refinements in
computedtopicqualityforeachtopicusingatopiccoherence
DiscussionandFutureWork).Instead,theinterfacedisplays
metricbasedonNormalizedPointwiseMutualInformation[5,
intermediatefeedback,suchasboldanditalicizedwords,rep-
NPMI]withWikipediaasthereferencecorpus[25].
resentingusers’specifiedrefinementsbeforesaving,andany
oralloftheoutstandingrefinementscanbeundone. When Procedure
userspress“save”,theirspecificationsareincorporatedinto Sessionsweredesignedtotakeonehour,butinpracticetook
themodel(RefinementImplementation). upto90minutes,andtheywereconductedremotelywithau-
dioandscreen-capturerecording. Weintroducedparticipants
QUALITATIVEEVALUATIONOFAHL-TMSYSTEM totopicmodelingandtothe HL-TM toolusingthetraining
topicmodel.Theinterviewerdescribedeachrefinementopera-
Ourfullyinteractiveuser-centeredHL-TMsystemfocuseson
tionandaskedtheparticipanttopracticesampleoperations.
topicmodelnovices.Participantsexploredandrefinedamodel
builtfromaTwittercorpusofcomplaintsaboutairlines,fol- Participantsthenreviewedtherawtweetsofthetestdataset
lowedbyasemi-structuredinterview.Thestudyfocusedona inacsvfileandweretoldtoimaginetheyhadbeenaskedto
broadsetofoperationsinafullyinteractivesystem(compared organizethesetweetstoidentifydifferentclassesofairline
to [27]),aswellasunderstandinghowinteractivemachine complaints. TheythenopenedtheHL-TMtoolwiththetest
learningchallenges–predictability,complexity,andlatency– topicmodel(Figure1)andwereinstructedthataninitialmodel
complicate topic modeling. For refinements in HL-TM, we of10topicshadbeengeneratedtohelpsummarizethetweets,
dividepredictabilityintocontrolandstability,wherecontrolis but that they may notice flaws and may need to refine the
howmuchtheuser’srefinementisreflectedafterthemodelup- model. Theintervieweraskedafewintroductoryquestions
dates(e.g.,aspecifiedwordisaddedtothetopic),andstability aboutthemodelandthetool,theninstructedparticipantsto
ishowmanyotherchangesnotspecifiedbytheuserappearin thinkaloudwhilerefiningthemodelusingthetooluntilthey
themodel(e.g.,otherunspecifiedwordsareadded).Instability, felt it best categorized the tweets into types of complaints.
inparticular,isaconcernwithHL-TM: smallchangestothe Participants were given a maximum of 20 minutes for the
modelcanpropagateinunexpectedways. task,andafterwardstheyansweredsemi-structuredinterview
questionsaboutthetask,model,andtool.
Method
DataandAnalysis
Thestudyprotocolincludedatrainingtasktofamiliarizeusers
We logged user interaction with the HL-TM tool, including
withtopicmodeling,atesttasktorefineatopicmodel,anda
thestateofthemodelateachiteration,whentheuserpressed
semi-structuredinterviewontheexperience.
“save”,andrefinementusage. Thetaskaudiowasalsotran-
scribedandcodedalongwiththeresponsesforthepost-task
Participants
interview.Codingfollowedathematicanalysisapproach[8]to
We recruited twelve participants (five male, seven female)
uncovertheoverarchingthemesrepresentedbymorespecific
fromcampuse-maillists. Theywereonaverage30.5years
codeswithinthedata. Thecodebookwasorganizedintofive
old(SD=10.3)andfluentEnglishspeakers.Educationalback-
themescontainingatotalof40codes: challenges(10codes),
groundsincludedhuman-computerInteraction(5),informa-
toolrequests(10),refinementrequests(8),savestrategies(6),
tionmanagement(2),education(1),mechanicalengineering
andrefinementstrategies(6). Todetermineagreement,two
(1),computerscience(1),psychology(1),andinternational
researchersindependentlycodedtranscriptsforarandompar-
government(1). Experiencewithtopicmodelingvaried(nine
ticipant. Of21instances,theresearchersagreedonthecodes
withnoexperience,threewithlimited)asdidexperiencewith
for12anddisagreedonnine. Disagreementswereresolved
datascienceormachinelearning(sevenwithnoexperience,
three limited, two significant). Each participant got a $15 3https://www.kaggle.com/crowdflower/twitter-airline-sentiment
Amazongiftcard. WerefertoparticipantsasP1–P12. 4https://raw.githubusercontent.com/mimno/Mallet/master/stoplists/en.txt
Figure1.UserinterfacefortheHL-TMtool.Alistoftopics(left)arerepresentedbytopics’firstthreetopicwords.Selectingatopicrevealsmoredetail
(right):thetop20wordsandtop40documents. Hoveringorclickingonawordhighlightsitwithinthedocuments. Userscanrefinethemodelusing
simplemechanisms:click“x”nexttowordsordocumentstoremovethem,selectanddragwordstore-orderthem,typenewwordsfromthevocabulary
intotheinputboxandpress“enter”toaddthem,selectawordandclickthetrashcantoaddittothestopwordslist,orclick“split”and“merge”(to
therightofthetopicwords)toenterintosplitandmergemodes.
andcodesclarifiedthroughdiscussion, andasecondround Otherrefinementoperationswereusedbyonlythreeorfewer
of coding on transcripts for a different random participant participants(Table2).
achievedbetteragreement(researchersagreedoncodesfor
Whenweaskedparticipantsthestrategiestheyused,wegot
14of15instances). Oneresearcherthencodedtheremaining
similaranswers: removeirrelevantwords(9participants),re-
transcripts.
movetypos(2),skipbadtopics(2),groupcommonwords(2),
changewordordertoname(2),moveirrelevantwordstothe
Findings: SimplicityandImprovement endofthelist(1),andpinpointrefine(1).Toremoveirrelevant
Wediscussfindingsrelatedtorefinementandsavestrategies, words, participants were not consistent, instead employing
abilitytoimprovethetopicmodel, andchallengesfacedin bothremoveword andaddtostopwords. Forexample, P6
usingafullyfunctionalHL-TMsystem. describedthathewould,“firstremoveallsimilarwords(e.g.,
make/makes)ineachtopicandthenputallgenericwordsin
Usersprefersimplerefinements thestopwordslist.” Twoparticipantsdescribedusingchange
LikeLeeetal.[27],simplerefinements,suchasremoveword, wordordernotonlytofixtherelativeimportanceofwords,
changewordorder,andaddwordtostopwordswerethemost but to name a topic, which they did by dragging three de-
commonlyused.Whileperceivedutilityalignedwithusagein scriptivewordstothefrontofthewordlist(eachtopicwas
Leeetal.[27],whichisnotsurprisingasrefinementsdidnot representedbyitstopthreewordsinthetopiclistontheleftof
affectthemodel,thereweretwomisalignedcasesinourstudy: theinterface). Amoreexpectedusageofchangewordorder
change word order and add word (Table 2). Change word camefromP4,whosaid,“Ireorderedtheairlinenamestogo
orderwasthesecondmostcommonrefinement,yetonlytwo totheendasIwasnotinterestedinwhatairlinesattracted
ofthe10participantswhouseditinthetaskthoughtitwasone complaints.” Fordealingwithpoorqualitytopics,twopar-
ofthemostuseful;alternatively,addwordwasonlythefourth ticipantsdescribedtheirstrategytoignorebadtopics,while
mostcommonrefinement,yetallsixparticipantswhousedit one participant described a pinpoint refinement strategy in
thoughtitoneofthemostuseful. Theserefinementsprovide whichshewouldchooseasingletopicwordfromaseemingly
variedcontrol;wediscussthisdiscrepancyinDiscussionand randomtopicandthenuseaddwordandremovewordtomake
FutureWork. thetopicmoreaboutthatsingleword. Finally,wealsonoted
casesofparticipantsusingrefinementstoexplorethemodel.
Detailedrefinementsusageandstrategies Forexample,P10usedtheaddwordrefinementtoseeifwords
Werecordedwhichrefinementsparticipantsused. Themost showedupinthetopic’sdocuments,byfirstaddingawordand
common refinement, remove word, was used by 11 partici- thenhoveringoverittoseeithighlightedinthedocuments.
pants a total of 270 times, followed by change word order P10wouldthenundotheaddedwordifitdidnotappearin
(10participants,136times),addtostopwords(sevenpartic- anyofthetopdocuments.
ipants,90times),andaddword (sixparticipants,41times).
Table1.Initialtesttopicmodelof10topicsgeneratedforthenegativetweetsfromtheairlineTwittercorpus.Topicsarerepresentedbytheirtopwords.
ObservedtopiccoherencecalculatedbyNPMI,whichdeemstopicstobeofhigherqualityiftheycontainwordsthatappearmorefrequentlytogether
thanapartinareferencecorpus.
TopicID NPMI TopicWords
T1 .031 hold,usairways,americanair,call,back,phone,hours,wait,change,minutes
T2 .014 southwestair,virginamerica,ticket,united,amp,fly,website,boarding,time,guys
T3 .024 flight,usairways,delayed,hrs,hours,late,miss,made,delay,connection
T4 .045 united,bag,bags,luggage,lost,baggage,check,find,airport,time
T5 .015 jetblue,http,time,united,email,long,jfk,give,amp,guys
T6 .029 americanair,usairways,people,weather,due,day,airport,hotel,issue,issues
T7 .022 united,plane,gate,waiting,hour,seat,sitting,crew,delay,min
T8 .009 usairways,americanair,make,problems,days,travel,refund,miles,told,booking
T9 .030 service,customer,united,usairways,worst,airline,experience,agents,staff,flying
T10 .025 flight,cancelled,southwestair,flightled,americanair,flights,today,flighted,late,tomorrow
Table2. Listofrefinementsorderedbyin-taskusagewithcountofpar- Table3. Savestrategiesdescribedbyparticipantsandthenumberof
ticipantsthatselectedthespecifiedrefinementasoneofthemostuseful timeseachparticipantsavedduringthetask,orderedfrommosttoleast
orleastusefulrefinements.Simple,word-levelrefinementsareboththe iterations. Therewasnodominantstrategy: saveusageandstrategy
mostcommonlyusedandjudgedtobemostuseful(exceptforchange variedacrossparticipants.
wordorder: onlytwoofthe10participantswhouseditfoundittobe Participant Iterations SaveStrategy
mostuseful).
P8 42 Aftereachrefinement
Refinement Most Least Used Total
P9 28 Whensure
Useful Useful By Usage
P12 19 Afterabatchofrefinements
Removeword 5 1 11 270
P2 19 Aftereachtopicmodified
Changewordorder 2 1 10 136
P7 18 Afterabatchofrefinements
Addtostopwords 3 0 7 90
P10 16 Aftereachrefinement
Addword 6 1 6 41
P11 15 Whensure
Removedocument 0 3 3 20
P4 9 Aftereachtopicmodified
Mergetopic 2 3 2 5
P5 8 Aftereachrefinement
Splittopic 1 5 1 1
P1 3 Forgottosave
P6 1 Forgottosave
P3 0 Afraidtosave
Whenandwhydouserschoosetosavetheirchanges?
Usersrefinethetopicmodelbyapplyingrefinementsandthen
wouldpressthesavebutton.” Incontrast,P6andP1reported
separately clicking “save”. Before saving, users can undo
thattheyforgottosave,andfourotherparticipantsstatedthat
someoralloftheirchanges. Tounderstandwhenparticipants
rememberingtosavewasoneofthemainchallengesofusing
choosetosaveandbecausetheHL-TMsystemdoesnotsupport
thetool.P12wondered,“ifmovingthe[save]buttonoverfrom
undoaftersaving,thesystemdidnotenforceaparticularsave
thesidewouldhavehelpedmeremember[tosave].” Finally,
strategy, such as after every refinement or a set number of
P3 was afraid to save, saying, “I didn’t want to start from
refinements. Instead, participants could specify a series of
scratch.” Shesuggestedthathavingahistoryofrefinements
localrefinements,butthesewouldonlybeappliedtothemodel
thatcouldhavebeenrolledbackmightmitigatetimidity.
once they clicked “save”, which they could do at any time.
Saveusagevariedsubstantially(min=0,max=42,avg=14,
Didusersimprovethemodel?
SD=12);seeTable3.
Todetermineifparticipantsimprovedtheinitialtopicmodel
Userswereaskedaboutstrategiesforwhentoclick“save”: usingtheHL-TMtool,wemeasuredthequalityoftheinitial
aftereachrefinement(4participants),aftereachtopicmodified topicmodelandthefinaltopicmodelsusingqualitativeand
(2), after a batch of refinements (2), when sure (2). These quantitativemethods.
varied strategies suggest that HL-TM should allow users to
Allparticipantsstartedwiththesamemodel. Wecomputed
choose when to save their refinements. Additionally, two
topic quality for the initial model and final models using a
participants forgot to save, and another was afraid to save,
topic coherence metric based on NPMI [25]. The average
whichsuggeststhatsystemsshouldreminduserstosaveand
topiccoherenceforthe10topicsoftheinitialmodelwas.024
supportundo. “Save”countsandstrategyfeedback(Table3).
(min=.01,max=.04,SD=.01)(per-topiccoherenceshown
P8 saved the most frequently (42 times) and described his inTable1). Theaveragetopiccoherenceforthefinalmodel
strategy as saving after each refinement, saying, “I always foreachparticipantrangedfrom.021to.037(M=.027,SD=
pressthesavebuttonwhenImakeanyrefinements.” P9saved .005),whichapairedt-testshowedasignificantimprovement
28times,saying,“onlywhenIamverysureabouttheresult,I fromtherefinementprocess,t(10)=2.89, p=.037.
Participantsgavetheirsatisfactionwiththetopicmodelbefore Stability
andafterthetaskonascalefromonetoseven,withonebeing Auserinterfaceshouldbepredictabletosupportuserconfi-
notatallsatisfiedandsevenbeingverysatisfied. Theaverage denceandunderstanding[18];however,interactivemachine
subjectivemodelsatisfactionincreasedfrom4.7(SD=1.29) learningoftenviolatesthisprinciple[2]asthesesystemscom-
beforethetaskto5.2(SD=0.83)afterthetask. Whilethis binehiddenknowledge—suchaspreviouslylearnedmodels
increasewasnotstatisticallysignificantbyaWilcoxensigned or data—with users’ instructions. We describe system pre-
rank test (Z = 1.04, p=.15), six of the 12 participants dictability as a combination of stability (no changes other
−
commentedunpromptedafterthetaskthattheirfinalmodel thanthosespecifiedoccur)andcontrol(thechangeoccursas
providedagoodorganizationofthecomplaints. Forexample, specified).
P5said,“I’moverallhappywiththe[final]modelandIlike
We asked if participants agreed with the statement, “no
thatIcanusethetooltomakethechangesthatIwant.”
changesotherthantherefinementsImadeoccurredwhenI
Participantsgavethebestandworsttopicsintheinitialmodel clickedupdate”onascalefromone,ornoagreement,toseven,
(Table1). MostparticipantsagreedthebesttopicswereT4(4 orcompleteagreement(Figure2,C).Theaverageresponse
participants),T3(3),T1(3),andT9(3)andtheworsttopics was4.1outofseven(SD=2.0),andthreeof12participants
were T5 (8), T6 (3), and T8 (2), which correlates with the saidinstabilitywasthemostfrustratingchallengewhileno
observedtopiccoherence. ThethreebesttopicsbyNPMIare participantssaiditwastheleast.
T4 (NPMI=.045), T1 (.031), and T9 (.030), while the three
There was a large variance for not only whether users per-
worsttopicsareT8(.009),T2(.014),andT5(.015).
ceivedinstabilitybutalsotheirreactionstoit. Afterthetask,
eightof12participantsmentionedtheyhadperceivedinsta-
Whatchallengesdousersface?
bility. Of those, two participants found this to be positive.
To understand how general interactive machine learning
Forexample,P6observedanunspecifiedchangewhen“new
challenges affect users of HL-TM, we coded four common wordswereaddedontothelisttoreplacetheonesIremoved.
challenges—tracking complex changes, instability, lack of
Itmadethemodelbetter.” P2notedthatafterremovingsome
control,latency—andidentifiedchallengeswithoursystem.
wordsfromatopictherewas“someslightsurpriseatseeing
Participantsalsostatedwhichchallengesweremostandleast
wordsthatIhadnotchosenshowup,butIwasprettysatisfied
frustratingduringthetask. Ofthefourcommonchallenges,
onlookingattheresults.”Threeparticipantsfeltneutralabout
trackingcomplexchangeswasthemostfrustrating,followed
the instability. For example, P7 said, “[instability] did not
byinstability,lackofcontrol,andlatencywastheleastfrus-
impact”hisabilitytoperformthetask,andP4said,“whenI
tratingchallenge.
removedsomekeywords,otherkeywordscameup. Iwasn’t
payingenoughattentiontothisto determineifithelpedor
Trackingcomplexchanges harmed.” Finally,threeparticipantshadnegativereactions,
Whenusersclick“save”,thealgorithmupdatesthemodel,and suchasP9,whowasunsureofwhathadchangedinthemodel
theresultingmodelmayhavesubstantialchanges. Toexplore afteranupdate,butstated,“...butIrememberbeinghappy
whether participants could track these changes, they rated withthetopicandwhenthatchangeditmademeunhappy.”
theiragreementwiththestatement,“Iwasabletoremember This participant also requested the ability to freeze a topic,
what the model looked like before my updates” on a scale meaningitwouldnotbechangedasotherrefinementswere
fromone,meaningnoagreement,toseven,meaningcomplete made.
agreement(Figure2,D)anddiscussedhowthisaffectedthem.
Theaverageresponsewas3.7outof7(SD=1.7),andfourof
Control
the12participantssaidthiswasthemostfrustratingchallenge
Similartothechallengeofinstability,usersshouldalwaysbe
whileonesaiditwastheleast.
incontrolofuserinterfaces[34];however,interactivemachine
Five participants said not being able to remember what the learningisbydefinitionacollaborationbetweenalgorithm
modellookedlikehurttheirperformance. Forexample,P9 and user [19]. Inthis collaboration, wedefine “control” as
said,“amomentago,Iwassatisfiedwiththistopic,butnow whethertheuser’srefinementsareincorporatedintothemodel
it’sgone,andIdon’tthinkIam,butIcan’tremember,”and asexpected. Forexample,ifauserremovesawordfroma
P3andP8feltthelackof“undo”intensifiedthischallenge. topic,thewordshouldnotbeinthetopicaftertheupdate.
P3said,“Ithinkthisisabigissue–I’dliketoknowifI’mcap-
Toexplorewhetherparticipantsfeltincontrolofthesystem,
turingthetruedata–andbeabletostepbacktoearlyversions
theystatedonaseven-pointscalewhethertheyagreedwiththe
ofthemodelbeforesaving,”andP8said,“Idon’tknowwhat
statement,“therefinementsImadewereappliedasexpected
Ihavedonesometimes,andtherearenowaystogoback...”.
whenIclickedupdate”fromnotagreeingatalltocompletely
Fourparticipantsmentionedasimilarchallenge,thatitwas
agreeing(Figure2,B)anddiscussedhowthisaffectedtheir
hardtotellwhatchangedinthemodelafteranupdate,such
task. Theaverageresponsewas5.6outofseven(SD=1.0),
asP10,who“hadtobrushthroughallthewordstoconfirm
meaningoverallusersfoundthesystemtobefairlycontrol-
if [his specified] change occurred,” and P5, who “did not
lable. Oneofthe12participantssaidlackofcontrolwasthe
understanditatfirst,thatthemodelactuallychanges,asthere
mostfrustratingchallengeandonesaiditwastheleast.
was no feedback or indication.” Finally, three participants
requestedalong-termhistoryviewofthemodel,suchasP3, However,duringthetasksevenparticipantsnotedfrustration
whosuggested“havingahistoryofrefinements.” withthelackofcontrolwiththeinterface,andfiveparticipants
A B
C D
Figure2. Countsforresponsesonascalefromonetosevenforparticipants’agreementwithstatementsrelatedtolatency(A),lackofcontrol(B),
instability(C),andtrackingcomplexchanges(D),withsevenmeaningtheydidnotexperienceitandonethattheydid. Mostparticipantsfoundthat
thesystemupdatedquicklyandrefinementswereappliedasexpected,whiletherewassubstantialvarianceforifparticipantscouldrememberwhatthe
modellookedlikebeforeupdatingoriftheyfelttheupdatedmodelincludedotherchangesthanthosespecified.
specificallyobservedthatchangewordorderwasuncontrol- wouldberequiredforthislongerwaittime. P7worriedthis
lable.P4triedtodragimportantwordstothefrontofthetopic waitwouldfurtherhindertheabilitytorememberwhatthe
listandstatedthat,“thereorderingdidn’talwaysgetaccepted,” modellookedlikebeforeupdating,andP3thoughtthiswould
andP8triedtodragunimportantwordstotheendofthelist furtheraffectsavestrategy,suggestingthatitwouldinsteadbe
andsaid,“Itriedtomovethiswordanditjustgoesbackup.” “bettertonot‘save’changes,buttohavehighlightstoshow
whatit‘might’looklikeoncesaved.” Mostparticipantsfelt
thatbothtwominutesand10minuteswouldbeunacceptable
Latency
waittimes.
Priorworkininteractivemachinelearningcallsforrapidin-
teractioncycles[2]tominimizeattentionloss[22]andreduce
Trustandconfidence
short-termmemoryload[34]. However,manyinteractivesys-
Trustisaprimarydesignchallengeforintelligentsystems[32,
temsdonotprovidereal-timeupdates: thislatency—theuser
20]. Anentiresub-fieldofmachinelearningfocusesonin-
hastowaitwhilethealgorithmisperforminganupdate—is
terpretability to build trust and promote adoption of these
typically related to the size of the data and complexity of
systems. Surprisingly, wedid notseeparticipants mistrust-
thecomputation. Forexample,anearlyHL-TMimplementa-
ing the HL-TM system. While users are quick to distrust a
tion[23]took5to50secondstoupdatethemodelbasedon
classificationsystemthatproduceseasilyidentifiable,incor-
refinementoperations.
rectclassifications,topicmodelshavelessobviousincorrect
Ourrefinementimplementationisefficientbydesign,andthe answers.
datasetusedinthisuserstudywasrelativelysmall(bothin
However, participants sometimes put too much trust in the
documentsizeandlength),thereforethealgorithmupdated
systemorlackedself-confidence. Forexample,P10wascon-
almostinstantaneouslyduringthetask(.09 .63seconds).No
− fusedaboutatopicword,saying,“ifthesystemcougheditup,
participantsaidthatlatencywasthemostfrustratingchallenge
theremustbeareasonforit,right?”Someparticipantslacked
while two participants said it was the least frustrating, and
confidenceintheirrefinements:P7saidthatremovedocument
the average response was 6.3 out of seven (SD=0.8) for
istheleastusefulrefinement,because,“Idon’tfeelcomfort-
participantsagreementwiththestatement,“afterclickingthe
ableremovingadocument.” AndwhenP5addedwordstoa
updatebutton,themodelupdatedquickly”(Figure2,A).
topic,shesaid,“it’sputtingmywordsontop...I’veaddedtoo
However, for a more realistic corpus size or alternative re- manywords,whichhavegonetothetopofthelist,soeither
finementimplementation,latencybecomesachallenge. We thealgorithmthinksit’simportantorit’sbecauseI’veadded
askedparticipantstodescribehowtheirabilitytoperformthe them,”followedby,“Idon’tthinkthatitshouldalwaysgive
taskwouldbeaffectedhadthewaittimebeen10seconds,30 more importance [to my added words], because I could be
seconds,twominutes,or10minutes.Mostparticipantsfelt10 wrong!” Thischallengehasadirectconnectiontotheissues
secondswouldbeanacceptabletimetowait:fiveparticipants ofinstabilityandlackofcontrol,whichwediscussinmore
feltthatwaiting10secondswouldhavenoeffectonthetask detailinDiscussionandFutureWork.
andtwoparticipantsfeltthatthislongerwaittimewouldhave
apositiveeffect,forexample,P5statedthatwaitinglonger Whatotherrequestsdousershaveforthesystem?
“wouldbebetterformetorealizethatthetweetshavechanged.” Manyparticipantswantedabetterunderstandingofthemodel
Fora30secondwaittime,twoparticipantsfeltthiswouldbe andthedata. Forexample,twoparticipantsrequestedabetter
anacceptablewaittimewithoutanychangestotheinterface, modeloverview,suchasP7,whowantedto“seetheentirelist
whereas four participants said that changes to the interface ofthetop20wordsforeachtopicononescreentoallowfor
makingbulk,fasterchanges.” Additionally,threeparticipants Supportundo: Whenpossible,HL-TMshouldsupportrevert-
wanted to view words or documents across topics, such as ingtopriorstatesofthemodel: somenotedthatthismade
P12whosuggested,“anoteorcolortoindicatethatacertain themafraidtosaveduringthetask,whileothersspecifically
termappearsonlyinthistopicandnotintheothers.” Two requestedanundofunctionality.
participantsrequestedenhancingthewordincontextfeature,
Allowuserstochoosewhentosave,butremindthemtodo
suchasbyscrollingtotheselectedwordorfilteringtoonly
so:Wehadanticipatedneedingaseparatesaveactiontoallow
documentscontainingtheword. Threeparticipantswantedto
userstoconfirmrefinements(lackingundo)andtocounteract
viewmoredocumentsthanthe40shown,andtwoparticipants
latency,butwealsonoteduserswhocreatedrefinementsasa
wantedtoviewthetotalnumberofdocumentsforatopic.
dataexplorationtoolwithouttheintentofhavingthemupdate
Similar to the merge word operation identified by themodel.Thus,HL-TMsystemsshouldallowuserstochoose
Leeetal.[27],sixparticipantsrequestedarefinementtoadd whentosavetheirrefinementstothemodelinsteadofforcing
phrases(insteadofjustsinglewords),andfourparticipants a save. However, because users forget to save, additional
requestedarefinementtogroupsynonymsandplurals. Asan- informationshouldbeprovidedintheinterfacetoremindusers,
ticipated,participantsusedtheaddtostopwordsrefinement, such as a more prominent count of outstanding refinement
and two participants requested an enhancement to the stop operationsoravisualcuethatdisplaysiftheyhavenotsaved
wordsfunctionality,suchasbeingabletoviewthestopwords recently.
listandremovewordsthathavebeenaddedtoit. However,
Freezetopicstoprotectfrominstability:Userscomplained
sevenparticipantsnotedconfusionbetweentheaddtostop
ofinstabilitywhentopicsthatwereoncehighqualityorabout
wordsandremovewordrefinements,whichshouldbeclarified
aparticularthinghadchanged. Aprocess,suchasfreezing
infutureinterfacedesign. Forexample,P5said,“removinga
atopic,suggestedbyoneparticipantasamechanismtohold
wordfeatureissimilartothedeletefeature,whichgotmea
a particular topic constant during subsequent updates, is a
bitconfused,”andP9said,“Igotconfusedbetweenremoving
promisingsolutiontothisproblemandshouldbeincorporated
keywordsfromaparticular[topic]andtheoverall[topics],
infuturedesign.
soImademistakesinthebeginning.” Tohelpbetterorganize
theview,threeparticipantswantedtonametopics,notingthat Supportmulti-wordrefinements:Participantsrequestedthe
it would be a useful way to remember what the topics are abilitytoaddphrasesandgroupsynonyms. Groupsynonyms
about, and two participants wanted to reorder topics in the couldbeimplementedasthemergewordrefinementdiscussed
list. Finally,twoparticipantswantedtodeleteatopicifitwas inLeeetal.[27],notasanupdatetotheunderlyingmodel,
particularlybad. butasawayoforganizingwordsintheinterface.Ontheother
hand,addphrasesshouldbeimplementedintheinterfaceas
Summary anextensiontoaddword(asrequestedbyparticipants),but
Participants were frustrated by their inability to track how wouldlikelyrequireamorecomplexmodelingapproachthat
themodelchangedthroughouttherefinementprocess.While supportsn-gramsasopposedtosingletokens.
participantsperceivedsysteminstability,theyhadvariedre-
Clarify difference between adding a word to stop words
actions(positiveandnegative). Ontheotherhand,usersdid
andremovingitfromasingletopic: Futuredesignshould
notexperiencesubstantiallatencyorlackofcontrol. Wedid
explicitlydelineatebetweenremovingawordfromalltopics
notfindanycaseswhereusersdistrustedthesystem,butusers
(andthemodelingprocessentirely),addtostopwords,and
perhapstrustedthesystemtoomuchorhadtoolittleconfi-
removingawordfromasingletopic,removeword,asmany
denceinthemselves. Participantsspecificallyrequestedthe
participantsconfusedthetwooperationsduringthetask.
ability to undo changes after saving and to curate the topic
modelview,suchasbyre-orderingthetopiclist,removing Support user-curated model view: Three participants re-
poorqualitytopics,andnamingtopics. Participantsalsore- quested named topics. Two other participants used change
questedmulti-wordrefinements,suchasaddingphrasesand wordorderforadhoctopicnaming. Asthisoperationisnot
groupingsynonyms. always applied as expected, providing a controllable topic
namingfunctionalitywillimproveuserexperience. Partici-
DISCUSSIONANDFUTUREWORK pantsalsorequestedothertechniquesforcuratingtheirmodel
WeoutlineimplicationsforfutureHL-TMsystemdesign,dis- view, which should be incorporated in the design of future
cussopenquestionsrelatedtointeractivemachinelearning, systems, such as the ability to re-order the topic list and to
andprovideareflectiononourHL-TMimplementation. removepoorqualitytopicsentirely.
DesignRecommendations
OpenQuestions
Provide richer history: Participants voiced concerns with
Thisisthefirstsystemtoefficientlyimplementafullsuiteof
theirinabilitytorememberthehistoryofthemodel,andfour
refinementsdesiredbyusersinpriorwork[27,31],enabling
of12participantssaidtheywereunabletotellhowthemodel
thestudyoftruehuman-in-the-loopinteractionsofacompre-
haschangedafteranupdate. HL-TMinterfacesshouldstrive
hensiveHL-TMsystem.Weenumerateopenquestionsabout
to support visualization of short term and long term model
HL-TMdesignthatfollowfromourfindings.
changes;userswanttotrackhowthemodelchangedthrough-
outtherefinementprocess. Thiswasthemostconsistentand Trustvs. instabilityandcontrol: Userswerenotbothered
mostfrustratingissueinthestudy. by instability or lack of control either because they trusted