ebook img

Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. User-Centered ... PDF

13 Pages·2017·0.42 MB·English
by  
Save to my drive
Quick download
Download
Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.

Preview Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. User-Centered ...

Alison Smith, Varun Kumar, Jordan Boyd-Graber, Kevin Seppi, and Leah Findlater. User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System. Intelligent User Interfaces,2018,12pages. @inproceedings{Smith:Kumar:Boyd-Graber:Seppi:Findlater-2018, Author = {Alison Smith and Varun Kumar and Jordan Boyd-Graber and Kevin Seppi and Leah Findlater}, Booktitle = {Intelligent User Interfaces}, Year = {2018}, Title = {User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System}, Url = {http://umiacs.umd.edu/~jbg//docs/2018_iui_itm.pdf}, } Alison won a best student paper honorable mention (3 out of 300) Downloaded from http://umiacs.umd.edu/~jbg/docs/2018_iui_itm.pdf Contact Jordan Boyd-Graber ([email protected]) for questions about this paper. 1 Closing the Loop: User-Centered Design and Evaluation of a Human-in-the-Loop Topic Modeling System AlisonSmith VarunKumar JordanBoyd-Graber UniversityofMaryland UniversityofMaryland UniversityofMaryland ComputerScience ComputerScience CS,iSchool, UMIACS, LSC CollegePark,Maryland,USA CollegePark,Maryland,USA CollegePark,Maryland,USA [email protected] [email protected] [email protected] KevinSeppi LeahFindlater BrighamYoungUniversity UniversityofWashington ComputerScience HumanCenteredDesignand Provo,Utah,USA Engineering [email protected] Seattle,Washington,USA [email protected] ABSTRACT text[7]bymodelingtheintuitionthatgroupsofwordsthat Human-in-the-looptopicmodelingallowsuserstoguidethe formacommonthemeappearmoreoftentogetherthannot; creationoftopicmodelsandtoimprovemodelqualitywithout thediscoveredtopicsaresetsofwords(e.g.,“football,touch- havingtobeexpertsintopicmodelingalgorithms. Priorwork down,NFL,...”),andeachdocumentcancontainmultiple inthisareahasfocusedeitheronalgorithmicimplementation topics. However, traditional topic models can include poor withoutunderstandinghowusersactuallywishtoimprovethe qualitytopics[30]orcanbemisalignedwithadomainexpert’s modeloronuserneedsbutwithoutthecontextofafullyin- understandingofthecorpus[23]. Pre-processingtechniques teractivesystem.Toaddressthisdisconnect,weimplemented andparametertuning[36]canimprovemodelquality,buttypi- asetofmodelrefinementsrequestedbyusersinpriorwork callyrequiremanytime-consumingiterationsandcanonlybe andconductedastudywithtwelvenon-expertparticipantsto performedbyanalgorithmexpert. Human-in-the-looptopic examinehowendusersareaffectedbyissuesthatarisewitha modeling(HL-TM)aimstosolvetheseproblemsbyallowing fullyinteractive,user-centeredsystem. Astheseissuesmirror enduserswhoarenotexpertswithtopicmodelingalgorithms thoseidentifiedininteractivemachinelearningmorebroadly, todirectlyrefinethemodel(e.g.,changingwhichwordsare suchasunpredictability,latency,andtrust,wealsoexamined includedinatopic,ormergingorsplittingtopics). interactivemachinelearningchallengeswithnon-expertend HL-TM has largely implemented topic model refinements usersthroughthelensofhuman-in-the-looptopicmodeling. based on what algorithm developers assume users want, as Wefoundthatalthoughusersexperienceunpredictability,their well as algorithm concerns, such as which refinements are reactionsvaryfrompositivetonegative,and,surprisingly,we the most straightforward to implement [23, 10, 21, 26]. In didnotfindanycasesofdistrust,butinsteadnotedinstances contrast,Leeetal.[27]employedauser-centeredapproachto whereusersperhapstrustedthesystemtoomuchorhadtoo identifyasetoftopicrefinementoperationsthatusersexpect littleconfidenceinthemselves. tohaveinaHL-TMsystem. However,becausenoexistingim- plementationsupportedthesetofrefinementoperations(e.g., INTRODUCTION addatopicword,changewordorder,mergetopics),theyused Topicmodelinghelpsusersunderstandvastdocumentcollec- Wizard-of-Ozrefinements: theresultingtopicswereupdated tionswhentherearetoomanydocumentsortoolittletimeto superficially—not as the output of a data-driven statistical readindividualdocuments: ajournalistprocessingthereports model(thegoaloftopicmodels). Thus,thisstudyislimited, surrounding breaking news or a legal team finding interest- asitignoresthecommoninteractivemachinelearningissues inge-mailsduringdiscovery. Topicmodelingautomatically thatariseinafullyinteractivesystem—suchas,unpredictabil- discoversthethemes(topics)inlargecorporaofunstructured ity,latency,andcomplexity—thatcanaffectuserexperience andimpacthowusersinteractwithaHL-TMsystem. Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalor classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed Toaddresstheselimitationsweimplementedabroadsetofre- forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitation finementsoperations[27,31]inasinglesystem:addword,re- onthefirstpage.CopyrightsforcomponentsofthisworkownedbyothersthanACM mustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orrepublish, moveword,removedocument,changewordorder,splittopic, topostonserversortoredistributetolists,requirespriorspecificpermissionand/ora mergetopics,andaddtostopwords. UnlikeinLeeetal.[27], [email protected]. IUI2018,March7–11,2018,Tokyo,Japan. whererefinementswereimmediatelyappliedonlyinthein- Copyrightisheldbytheowner/author(s).PublicationrightslicensedtoACM. terfaceandwerenotsavedtothebackingmodel,inourstudy ACMISBN978-1-4503-4945-1/18/03...$15.00. https://doi.org/10.1145/3172944.3172965 participants chose when to “save” and update the backing weshowhowtoadjusttheDirichlethyperparameters,α and modelwiththeirrefinements. Weevaluatedthisfullyinterac- β,toencodeuserinformationaboutdocuments(α)andtopics tive,user-centered HL-TM systemwith12participantswho (β). werenotexpertsintopicmodeling. Thissystemisalsofast Giventhismodel, weneedtofindthedistributionthatbest enoughtosupport(andstudy)interactiveuseoffine-grained explainsourobserveddocuments. GriffithsandSteyvers[17] topic modifications, unlike prior systems [21, 10, 23] (e.g., propose a collapsed Gibbs sampling based method. In col- earlyHL-TMsystemstook5 50secondsforeachupdate[23], − lapsedGibbssampling,theprobabilityofatopicassignment whereasourstook.09 .63secondstoupdateaftersavingdur- ingtheuserstudy). In−additiontoobservinghowparticipants z=tindocumentdgivenanobservedtoken,wi,andtheother topicassignments,z_,is employedtherefinementstoimproveatopicmodel,wequali- tativelyassessedoverallexperienceandthepotentialimpacts ofcommoninteractivemachinelearningchallenges,suchas nw,t+β P(z=t z_,w)∝(n +α) . (1) unpredictabilityandlatency. | d,t n +Vβ t Participantssubjectivelyandobjectivelyimprovedtopicmodel qualityusingthegivenrefinements. SimilartoLeeetal.[27], Here, z_ are the topic assignments of all other tokens, nd,t simplerefinements,suchasremovewordandchangewordor- is the number of times topic t is used in document d, nw,t derweremostcommon,however,unlikeLeeetal.[27],usage is the number of times token w is used in topic t, and nt didnotalignperfectlywithperceivedutility,asparticipants is the marginal count of the number of tokens assigned to foundchangewordordertobeoneoftheleastusefulrefine- topict.Fortraditionaltopicmodels,theGibbssamplerassigns ments. Ourfullyinteractivesystemalsoexposedpreviously latent topics Z for all tokens in the corpus, going over all hiddenissues: unexpectedresultsfromusingrefinementsand the documents in the corpus repeatedly until the algorithm aninabilitytotrackmodelchanges.Participantsalsovariedin converges. whenandwhytheysavedtheirlocalupdatestotheunderlying Thestateofthesamplerrepresentsthealgorithm’sbestguess model. Additionally,participantswhohadmoretrustinthe ofthetopicassignmentsforeverytoken.Addingahumanin system(orperhapslessconfidenceinthemselves)werenotas thelooprequirestheusertobeabletoinjecttheirknowledge frustratedbyunpredictabilityasothers. andexpertiseintothesamplingequationtoguidethealgorithm This work makes the following contributions: (1) efficient tobettertopics. asymmetric prior-based implementation of a broader set of user-centeredrefinementoperationsthanhasbeenpreviously implementedinasinglesystem;(2)extensionofaprevious, Human-in-the-loopTopicModeling limitedHL-TMuserstudy[27]tothisfullyinteractivesystem; Comparedtotraditionaltoolsthatvisualizestatictopicmod- (3)understandingofhowinteractivemachinelearningissues els [15, 13, 9], HL-TM provides mechanisms to allow end such as unpredictability, trust, and lack of control impact userstorefinechangingtopicmodels. Numeroustoolshave HL-TMusageanddesign;and(4)designprinciplesforuser- beendesignedaroundthisconcept,eachimplementingavari- centeredHL-TMsystems. etyofrefinements,butwithoutextensiveuserstudiesonreal world-tasks to show the refinement implementations match BACKGROUND userexpectations. Andrzejewskietal.[3],buildingonBoyd- Topicmodelingautomaticallyidentifiesthethemesortopics Graberetal.[6],introducedastatisticalframeworkforusers thatoccurinacollectionofdocuments. Acommonapproach tospecifypairsofwordsthatshouldorshouldnotbelongto isLatentDirichletAllocation(LDA)[4],whichisagenerative the same topic through “must-link” and “cannot-link” con- statisticalmodelthatmodelseachdocumentasadistribution straints. This framework has been extended by numerous of topics and each topic as a distribution of words. Here, HL-TMtools[23,11,33];however,thisapproachlimitsthe weprovideabriefbackgroundof LDA andexisting HL-TM possible set of refinements that can be supported and does systems. We also discuss design challenges for interactive not match users expectations in that end users typically do machinelearningsystemsmoregenerally. notthinkofspecifyingmodelrefinementsaspairwiseword correlations[27]. LatentDirichletAllocation Other HL-TM tools take alternative approaches, such as LDA[4]isagenerativemodel,whichassumesthateachdocu- UTOPIAN[10],whichusesnonnegativematrixfactorization mentdisgeneratedfromafixedsetofktopics. Eachtopicis andsupportscreating,splitting,andmergingtopics,inaddi- amultinomialdistribution,φ ,overthevocabulary,V. Each z tiontochangingwordweights. Lundetal.[28]alsousesa instanceofaword,ortoken,w,isgeneratedbysamplinga i fastmatrix-factorizationapproachbutonlysupportsalimited topicassignmentz fromthedocument’stopicdistributionθ , i d set of interactions. ConVisIT [21] uses fragment quotation followedbysamplingtheselectedtopic’sdistributionφ ,to zi graphsandsupportssplittingandmergingtopics. Finally,Dis- generateatokenw. i tillery [31] uses informative priors to support merging and Themultinomialdistributionsθ andφ aredrawnfromDirich- removingtopics,aswellasaddingandremovingtopicwords d z letdistributionsthatencodesparsity—howmanywordsyou andaddingtostopwords.However,nopriorimplementation expecttoseeinatopicorhowmanytopicsinadocument— supportsthecompletesetofrefinementoperationsrequested andcanalsoincorporateexpertknowledgefromusers. Below, byusers[27]. InteractiveMachineLearningDesignChallenges InjectinginformationhappensthroughmodifyingtheDirichlet Interactivemachinelearning—alsoknownasmixed-initiative parameters for each document, α, and each topic, β.1 Re- orhuman-in-the-loopsystems—incorporatehumaninputto calltheGibbssamplingconditionalprobabilityP(z=t z_,w) | produce an output or a decision. Designers of these must (1), which has two parts: how much a document likes a accountforchallengesinherentinmachinelearning,which topic—(nd,t+αd,t)—and how much a topic likes a word— deviates from traditional user interface design. While user (nw,t+βw,t). The priors are added to the topic assignment interfaces should provide immediate updates [29], be pre- counts; thankstotheconjugacyofmultinomialandDirich- dictable[18],ensuretheuserfeelsincontrol,andreduceshort letdistributions,thesepriorsaresometimescalled“pseudo- termmemoryload[34],interactivemachinelearningsystems counts”. OurHL-TMtakesadvantageofthisbycreatingpseu- arecommonlyslow,unpredictable,sharecontrolbetweenuser docountstoencouragethechangesuserswanttoseeinthe andsystem,andarecomplex[2,19]. topic. Priorworkhasexaminedhowpredictability,control,andac- Therefinementoperationsare: curacyaffecttheuserexperience. Gajosetal.[14]showed 1. Addword: toaddthewordwtotopict weforgetwfrom thatincreasingpredictabilityandaccuracyleadtoimproved allothertopicsandencouragetheGibbssamplertoassign satisfaction, while Kangasraasio et al. [24] showed that al- topic t for all of the word’s tokens, w. For the former, lowinguserstoseethepredictedeffectsofanactionbefore i weforgetthetokens’topicassignments. Forthelatter,we committingtoitcanimprovetaskperformanceandacceptance. increasethepriorofwint bythedifferencebetweenthe Similarly,usersofPeerFinder—atoolthatrecommendssimi- topic-wordcountsofwandtopic’stopwordw intopict larstudentsbasedonacademicprofiles—weremoreconfident 0 (i.e.,n n ). andengagedwhengivenmorecontrolevenwiththenegative w0,t− w,t 2. Removeword: toremovethewordwfromtopict weneed effectofaddedcomplexity[12]. Inthiswork,weexplorehow to forget all the word’s tokens w from t and discourage usersareaffectedbycomplexity,unpredictabilityandlackof i the Gibbs sampler from reassigningt to the word w. To controlinHL-TM. discouragethesamplerfromassigningwtot withahigh probability,weassignaverysmallprior,2ε,towint. REFINEMENTIMPLEMENTATION 3. Changewordorder: toreorderwordw2 toappearbefore Priorwork[27,31]identifiedasetofrefinementsthatusers word w1 in topic t we need to ensure that w2 is ranked expectedtobeabletouseina HL-TM system. Thereisno higherthanw1inthetopict. Toenforcethis,weincrease implementationforthebroadsetofuserpreferredrefinements, the prior of w2 in t by the difference between the topic- so Lee et al. [27] simulated refinements using a Wizard-of- wordcounts(i.e. nw1,t−nw2,t). Intuitively,thisoperation Oz method. To truly evaluate user experience with a fully resemblesprovidingsupplementalcountstow2 sothatit functionalHL-TMsystem,weimplementedsevenrefinements rankshigherthanw1inthetopic. requested by users: add word, remove word, change word 4. Removedocument: in LDA, eachdocumentcanberepre- order,removedocument,splittopic,mergetopic,andaddto sentedasaprobabilitydistributionovertopics.Toremove stopwords. thedocumentdfromtopict,weforgetthetopicassignment for all words in the document d and assign a very small Theserefinementsincludethesixtoprefinementsidentified, prior,2ε,tothetopict inα . d but not implemented, by Lee et al. [27], except for merge 5. Mergetopic: mergingtopicst andt meansthemodelwill 1 2 words. Mergewordswasdiscussedinthatstudyasameans haveacombinedtopicthatrepresentsbotht andt . We 1 2 fororganizingtopicwordsintheinterfaceratherthanadeeper assignt toalltokensthatwerepreviouslyassignedtot . 1 2 specificationthatshouldbeimplementedinthemodel. We Thiseffectivelydeletest fromthemodelanddecrements 2 alsoincludedtworefinementsthatwerenotsuggestedbyusers thenumberoftopics. inLeeetal.[27],perhapsduetothatstudy’smethod: merge 6. Split topic: to split topic t the user provides a subset of topicsdidnotarisebecauseusersonlyrefinedindividualtopics thetopic’swords,orseedwords,whichneedtobemoved andaddtostopwordsmayhavebeenoverlookedbecausethe fromtheoriginaltopic,t,toanewtopic,t . Toimplement n studyusedagenericcorpuswithawell-curatedstopwords this,weinvalidatetheoriginaltopicassignmentofallseed list. words,createanewtopicbyincrementingthenumberof topics,andassignalargepriorforeachoftheseedwords, w ,inthenewtopic,t . TheGibbssampler’sjobistosort RefinementImplementation s n whichwordslandinwhichofthenewchildtopics. Whenauserprovidesfeedbacktoatopicmodel,weviewthis 7. Addtostopwords:addingthewordwtoglobalstopwords ascorrectinganerrorthemodelmade. Wecanthusdivide removes w from all topics. We exclude that w from the thisfeedbackintotwobroadclasses: forgettingbadthingsthe vocabulary,V. This ensures that the Gibbs sampler will modellearnedandinjectingnewknowledgeintothemodel. ignorealloccurrencesofwinthecorpus. Forgetting is accomplished by invalidating the topic-word assignmentsfortargetedwordtypes. Thisisequivalenttothe 1To implement these refinement operations, we make use of the modelseeingthatwordfortheveryfirsttime,allowingitto makebetterdecisions. Intandemwithforgetting, injecting vKecdtiomreinnsteiorpnraeltavteicotnor(rfaotrheeracthhadnosccuamlaern)todftahnedseφpkriisorasV.Tdhiumse,nαsdioinsaal provides hints that encourage the algorithm to make better vectorforeachtopick. decisionsgoingforward. 2Weuseε=0.000001forourexperiments. Interface DatasetandTopicModel TheHL-TMuserinterface(Figure1)representsatopicmodel Weusedaseparatedatasetandmodelforthetrainingandtest as a list of topics on the left panel, each displayed as their tasks. Fortrainingwegeneratedamodelwith10topicsfrom first three words. Selecting any topic in the list shows the adatasetof2,225BBCnewsarticlescorrespondingtostories full topic view in the right panel, which consists of the top infivetopicalareas(business,entertainment,politics,sports, 20 topic words and snippets of the top 40 topic documents. tech)from2004 2005[16]. ForthetestweusedtheTwitter Documents are ordered by their probability for the topic t USAirlineSenti−mentdatasetfromKaggle,3whichincludes giventhedocumentd, or p(t d). Eachword, w, isordered 14,485totaltweetsfromFebruary2015directedtosixpopu- | and sized by its probability for the topic t, or p(w t); this larairlines(American,Delta,Southwest,United,USAirways, | simplewordlistrepresentationprovidesusersaquicktopic VirginAmerica).Thedatasetincludesmanuallyappliedlabels understanding[1,35]. Hoveringorclickingontopicwords organizing the tweets into “positive” (2,363 tweets), “neu- highlightsthewordinthedisplayeddocumentsnippets. tral”(3,099tweets),and“negative”(9,178tweets)sentiment categories. Wemodeledthe9,178negativesentimenttweets Usersrefinethetopicmodelusingsimpleinteractivemecha- with10topicsusingastandardstopwordslist4and300Gibbs nisms. Werequireuserstoclick“save”toincorporatetheir samplingiterations. Foreachsubsequentupdateduringthe specifiedrefinementsinsteadofapplyingthemimmediately task,30Gibbssamplingiterationswererun. Table1shows becausethesystemdoesnotsupportrevertingthemodelafter theinitialsetoftopics(henceforthT1–T10).Weautomatically an update (we discuss batch vs. immediate refinements in computedtopicqualityforeachtopicusingatopiccoherence DiscussionandFutureWork).Instead,theinterfacedisplays metricbasedonNormalizedPointwiseMutualInformation[5, intermediatefeedback,suchasboldanditalicizedwords,rep- NPMI]withWikipediaasthereferencecorpus[25]. resentingusers’specifiedrefinementsbeforesaving,andany oralloftheoutstandingrefinementscanbeundone. When Procedure userspress“save”,theirspecificationsareincorporatedinto Sessionsweredesignedtotakeonehour,butinpracticetook themodel(RefinementImplementation). upto90minutes,andtheywereconductedremotelywithau- dioandscreen-capturerecording. Weintroducedparticipants QUALITATIVEEVALUATIONOFAHL-TMSYSTEM totopicmodelingandtothe HL-TM toolusingthetraining topicmodel.Theinterviewerdescribedeachrefinementopera- Ourfullyinteractiveuser-centeredHL-TMsystemfocuseson tionandaskedtheparticipanttopracticesampleoperations. topicmodelnovices.Participantsexploredandrefinedamodel builtfromaTwittercorpusofcomplaintsaboutairlines,fol- Participantsthenreviewedtherawtweetsofthetestdataset lowedbyasemi-structuredinterview.Thestudyfocusedona inacsvfileandweretoldtoimaginetheyhadbeenaskedto broadsetofoperationsinafullyinteractivesystem(compared organizethesetweetstoidentifydifferentclassesofairline to [27]),aswellasunderstandinghowinteractivemachine complaints. TheythenopenedtheHL-TMtoolwiththetest learningchallenges–predictability,complexity,andlatency– topicmodel(Figure1)andwereinstructedthataninitialmodel complicate topic modeling. For refinements in HL-TM, we of10topicshadbeengeneratedtohelpsummarizethetweets, dividepredictabilityintocontrolandstability,wherecontrolis but that they may notice flaws and may need to refine the howmuchtheuser’srefinementisreflectedafterthemodelup- model. Theintervieweraskedafewintroductoryquestions dates(e.g.,aspecifiedwordisaddedtothetopic),andstability aboutthemodelandthetool,theninstructedparticipantsto ishowmanyotherchangesnotspecifiedbytheuserappearin thinkaloudwhilerefiningthemodelusingthetooluntilthey themodel(e.g.,otherunspecifiedwordsareadded).Instability, felt it best categorized the tweets into types of complaints. inparticular,isaconcernwithHL-TM: smallchangestothe Participants were given a maximum of 20 minutes for the modelcanpropagateinunexpectedways. task,andafterwardstheyansweredsemi-structuredinterview questionsaboutthetask,model,andtool. Method DataandAnalysis Thestudyprotocolincludedatrainingtasktofamiliarizeusers We logged user interaction with the HL-TM tool, including withtopicmodeling,atesttasktorefineatopicmodel,anda thestateofthemodelateachiteration,whentheuserpressed semi-structuredinterviewontheexperience. “save”,andrefinementusage. Thetaskaudiowasalsotran- scribedandcodedalongwiththeresponsesforthepost-task Participants interview.Codingfollowedathematicanalysisapproach[8]to We recruited twelve participants (five male, seven female) uncovertheoverarchingthemesrepresentedbymorespecific fromcampuse-maillists. Theywereonaverage30.5years codeswithinthedata. Thecodebookwasorganizedintofive old(SD=10.3)andfluentEnglishspeakers.Educationalback- themescontainingatotalof40codes: challenges(10codes), groundsincludedhuman-computerInteraction(5),informa- toolrequests(10),refinementrequests(8),savestrategies(6), tionmanagement(2),education(1),mechanicalengineering andrefinementstrategies(6). Todetermineagreement,two (1),computerscience(1),psychology(1),andinternational researchersindependentlycodedtranscriptsforarandompar- government(1). Experiencewithtopicmodelingvaried(nine ticipant. Of21instances,theresearchersagreedonthecodes withnoexperience,threewithlimited)asdidexperiencewith for12anddisagreedonnine. Disagreementswereresolved datascienceormachinelearning(sevenwithnoexperience, three limited, two significant). Each participant got a $15 3https://www.kaggle.com/crowdflower/twitter-airline-sentiment Amazongiftcard. WerefertoparticipantsasP1–P12. 4https://raw.githubusercontent.com/mimno/Mallet/master/stoplists/en.txt Figure1.UserinterfacefortheHL-TMtool.Alistoftopics(left)arerepresentedbytopics’firstthreetopicwords.Selectingatopicrevealsmoredetail (right):thetop20wordsandtop40documents. Hoveringorclickingonawordhighlightsitwithinthedocuments. Userscanrefinethemodelusing simplemechanisms:click“x”nexttowordsordocumentstoremovethem,selectanddragwordstore-orderthem,typenewwordsfromthevocabulary intotheinputboxandpress“enter”toaddthem,selectawordandclickthetrashcantoaddittothestopwordslist,orclick“split”and“merge”(to therightofthetopicwords)toenterintosplitandmergemodes. andcodesclarifiedthroughdiscussion, andasecondround Otherrefinementoperationswereusedbyonlythreeorfewer of coding on transcripts for a different random participant participants(Table2). achievedbetteragreement(researchersagreedoncodesfor Whenweaskedparticipantsthestrategiestheyused,wegot 14of15instances). Oneresearcherthencodedtheremaining similaranswers: removeirrelevantwords(9participants),re- transcripts. movetypos(2),skipbadtopics(2),groupcommonwords(2), changewordordertoname(2),moveirrelevantwordstothe Findings: SimplicityandImprovement endofthelist(1),andpinpointrefine(1).Toremoveirrelevant Wediscussfindingsrelatedtorefinementandsavestrategies, words, participants were not consistent, instead employing abilitytoimprovethetopicmodel, andchallengesfacedin bothremoveword andaddtostopwords. Forexample, P6 usingafullyfunctionalHL-TMsystem. describedthathewould,“firstremoveallsimilarwords(e.g., make/makes)ineachtopicandthenputallgenericwordsin Usersprefersimplerefinements thestopwordslist.” Twoparticipantsdescribedusingchange LikeLeeetal.[27],simplerefinements,suchasremoveword, wordordernotonlytofixtherelativeimportanceofwords, changewordorder,andaddwordtostopwordswerethemost but to name a topic, which they did by dragging three de- commonlyused.Whileperceivedutilityalignedwithusagein scriptivewordstothefrontofthewordlist(eachtopicwas Leeetal.[27],whichisnotsurprisingasrefinementsdidnot representedbyitstopthreewordsinthetopiclistontheleftof affectthemodel,thereweretwomisalignedcasesinourstudy: theinterface). Amoreexpectedusageofchangewordorder change word order and add word (Table 2). Change word camefromP4,whosaid,“Ireorderedtheairlinenamestogo orderwasthesecondmostcommonrefinement,yetonlytwo totheendasIwasnotinterestedinwhatairlinesattracted ofthe10participantswhouseditinthetaskthoughtitwasone complaints.” Fordealingwithpoorqualitytopics,twopar- ofthemostuseful;alternatively,addwordwasonlythefourth ticipantsdescribedtheirstrategytoignorebadtopics,while mostcommonrefinement,yetallsixparticipantswhousedit one participant described a pinpoint refinement strategy in thoughtitoneofthemostuseful. Theserefinementsprovide whichshewouldchooseasingletopicwordfromaseemingly variedcontrol;wediscussthisdiscrepancyinDiscussionand randomtopicandthenuseaddwordandremovewordtomake FutureWork. thetopicmoreaboutthatsingleword. Finally,wealsonoted casesofparticipantsusingrefinementstoexplorethemodel. Detailedrefinementsusageandstrategies Forexample,P10usedtheaddwordrefinementtoseeifwords Werecordedwhichrefinementsparticipantsused. Themost showedupinthetopic’sdocuments,byfirstaddingawordand common refinement, remove word, was used by 11 partici- thenhoveringoverittoseeithighlightedinthedocuments. pants a total of 270 times, followed by change word order P10wouldthenundotheaddedwordifitdidnotappearin (10participants,136times),addtostopwords(sevenpartic- anyofthetopdocuments. ipants,90times),andaddword (sixparticipants,41times). Table1.Initialtesttopicmodelof10topicsgeneratedforthenegativetweetsfromtheairlineTwittercorpus.Topicsarerepresentedbytheirtopwords. ObservedtopiccoherencecalculatedbyNPMI,whichdeemstopicstobeofhigherqualityiftheycontainwordsthatappearmorefrequentlytogether thanapartinareferencecorpus. TopicID NPMI TopicWords T1 .031 hold,usairways,americanair,call,back,phone,hours,wait,change,minutes T2 .014 southwestair,virginamerica,ticket,united,amp,fly,website,boarding,time,guys T3 .024 flight,usairways,delayed,hrs,hours,late,miss,made,delay,connection T4 .045 united,bag,bags,luggage,lost,baggage,check,find,airport,time T5 .015 jetblue,http,time,united,email,long,jfk,give,amp,guys T6 .029 americanair,usairways,people,weather,due,day,airport,hotel,issue,issues T7 .022 united,plane,gate,waiting,hour,seat,sitting,crew,delay,min T8 .009 usairways,americanair,make,problems,days,travel,refund,miles,told,booking T9 .030 service,customer,united,usairways,worst,airline,experience,agents,staff,flying T10 .025 flight,cancelled,southwestair,flightled,americanair,flights,today,flighted,late,tomorrow Table2. Listofrefinementsorderedbyin-taskusagewithcountofpar- Table3. Savestrategiesdescribedbyparticipantsandthenumberof ticipantsthatselectedthespecifiedrefinementasoneofthemostuseful timeseachparticipantsavedduringthetask,orderedfrommosttoleast orleastusefulrefinements.Simple,word-levelrefinementsareboththe iterations. Therewasnodominantstrategy: saveusageandstrategy mostcommonlyusedandjudgedtobemostuseful(exceptforchange variedacrossparticipants. wordorder: onlytwoofthe10participantswhouseditfoundittobe Participant Iterations SaveStrategy mostuseful). P8 42 Aftereachrefinement Refinement Most Least Used Total P9 28 Whensure Useful Useful By Usage P12 19 Afterabatchofrefinements Removeword 5 1 11 270 P2 19 Aftereachtopicmodified Changewordorder 2 1 10 136 P7 18 Afterabatchofrefinements Addtostopwords 3 0 7 90 P10 16 Aftereachrefinement Addword 6 1 6 41 P11 15 Whensure Removedocument 0 3 3 20 P4 9 Aftereachtopicmodified Mergetopic 2 3 2 5 P5 8 Aftereachrefinement Splittopic 1 5 1 1 P1 3 Forgottosave P6 1 Forgottosave P3 0 Afraidtosave Whenandwhydouserschoosetosavetheirchanges? Usersrefinethetopicmodelbyapplyingrefinementsandthen wouldpressthesavebutton.” Incontrast,P6andP1reported separately clicking “save”. Before saving, users can undo thattheyforgottosave,andfourotherparticipantsstatedthat someoralloftheirchanges. Tounderstandwhenparticipants rememberingtosavewasoneofthemainchallengesofusing choosetosaveandbecausetheHL-TMsystemdoesnotsupport thetool.P12wondered,“ifmovingthe[save]buttonoverfrom undoaftersaving,thesystemdidnotenforceaparticularsave thesidewouldhavehelpedmeremember[tosave].” Finally, strategy, such as after every refinement or a set number of P3 was afraid to save, saying, “I didn’t want to start from refinements. Instead, participants could specify a series of scratch.” Shesuggestedthathavingahistoryofrefinements localrefinements,butthesewouldonlybeappliedtothemodel thatcouldhavebeenrolledbackmightmitigatetimidity. once they clicked “save”, which they could do at any time. Saveusagevariedsubstantially(min=0,max=42,avg=14, Didusersimprovethemodel? SD=12);seeTable3. Todetermineifparticipantsimprovedtheinitialtopicmodel Userswereaskedaboutstrategiesforwhentoclick“save”: usingtheHL-TMtool,wemeasuredthequalityoftheinitial aftereachrefinement(4participants),aftereachtopicmodified topicmodelandthefinaltopicmodelsusingqualitativeand (2), after a batch of refinements (2), when sure (2). These quantitativemethods. varied strategies suggest that HL-TM should allow users to Allparticipantsstartedwiththesamemodel. Wecomputed choose when to save their refinements. Additionally, two topic quality for the initial model and final models using a participants forgot to save, and another was afraid to save, topic coherence metric based on NPMI [25]. The average whichsuggeststhatsystemsshouldreminduserstosaveand topiccoherenceforthe10topicsoftheinitialmodelwas.024 supportundo. “Save”countsandstrategyfeedback(Table3). (min=.01,max=.04,SD=.01)(per-topiccoherenceshown P8 saved the most frequently (42 times) and described his inTable1). Theaveragetopiccoherenceforthefinalmodel strategy as saving after each refinement, saying, “I always foreachparticipantrangedfrom.021to.037(M=.027,SD= pressthesavebuttonwhenImakeanyrefinements.” P9saved .005),whichapairedt-testshowedasignificantimprovement 28times,saying,“onlywhenIamverysureabouttheresult,I fromtherefinementprocess,t(10)=2.89, p=.037. Participantsgavetheirsatisfactionwiththetopicmodelbefore Stability andafterthetaskonascalefromonetoseven,withonebeing Auserinterfaceshouldbepredictabletosupportuserconfi- notatallsatisfiedandsevenbeingverysatisfied. Theaverage denceandunderstanding[18];however,interactivemachine subjectivemodelsatisfactionincreasedfrom4.7(SD=1.29) learningoftenviolatesthisprinciple[2]asthesesystemscom- beforethetaskto5.2(SD=0.83)afterthetask. Whilethis binehiddenknowledge—suchaspreviouslylearnedmodels increasewasnotstatisticallysignificantbyaWilcoxensigned or data—with users’ instructions. We describe system pre- rank test (Z = 1.04, p=.15), six of the 12 participants dictability as a combination of stability (no changes other − commentedunpromptedafterthetaskthattheirfinalmodel thanthosespecifiedoccur)andcontrol(thechangeoccursas providedagoodorganizationofthecomplaints. Forexample, specified). P5said,“I’moverallhappywiththe[final]modelandIlike We asked if participants agreed with the statement, “no thatIcanusethetooltomakethechangesthatIwant.” changesotherthantherefinementsImadeoccurredwhenI Participantsgavethebestandworsttopicsintheinitialmodel clickedupdate”onascalefromone,ornoagreement,toseven, (Table1). MostparticipantsagreedthebesttopicswereT4(4 orcompleteagreement(Figure2,C).Theaverageresponse participants),T3(3),T1(3),andT9(3)andtheworsttopics was4.1outofseven(SD=2.0),andthreeof12participants were T5 (8), T6 (3), and T8 (2), which correlates with the saidinstabilitywasthemostfrustratingchallengewhileno observedtopiccoherence. ThethreebesttopicsbyNPMIare participantssaiditwastheleast. T4 (NPMI=.045), T1 (.031), and T9 (.030), while the three There was a large variance for not only whether users per- worsttopicsareT8(.009),T2(.014),andT5(.015). ceivedinstabilitybutalsotheirreactionstoit. Afterthetask, eightof12participantsmentionedtheyhadperceivedinsta- Whatchallengesdousersface? bility. Of those, two participants found this to be positive. To understand how general interactive machine learning Forexample,P6observedanunspecifiedchangewhen“new challenges affect users of HL-TM, we coded four common wordswereaddedontothelisttoreplacetheonesIremoved. challenges—tracking complex changes, instability, lack of Itmadethemodelbetter.” P2notedthatafterremovingsome control,latency—andidentifiedchallengeswithoursystem. wordsfromatopictherewas“someslightsurpriseatseeing Participantsalsostatedwhichchallengesweremostandleast wordsthatIhadnotchosenshowup,butIwasprettysatisfied frustratingduringthetask. Ofthefourcommonchallenges, onlookingattheresults.”Threeparticipantsfeltneutralabout trackingcomplexchangeswasthemostfrustrating,followed the instability. For example, P7 said, “[instability] did not byinstability,lackofcontrol,andlatencywastheleastfrus- impact”hisabilitytoperformthetask,andP4said,“whenI tratingchallenge. removedsomekeywords,otherkeywordscameup. Iwasn’t payingenoughattentiontothisto determineifithelpedor Trackingcomplexchanges harmed.” Finally,threeparticipantshadnegativereactions, Whenusersclick“save”,thealgorithmupdatesthemodel,and suchasP9,whowasunsureofwhathadchangedinthemodel theresultingmodelmayhavesubstantialchanges. Toexplore afteranupdate,butstated,“...butIrememberbeinghappy whether participants could track these changes, they rated withthetopicandwhenthatchangeditmademeunhappy.” theiragreementwiththestatement,“Iwasabletoremember This participant also requested the ability to freeze a topic, what the model looked like before my updates” on a scale meaningitwouldnotbechangedasotherrefinementswere fromone,meaningnoagreement,toseven,meaningcomplete made. agreement(Figure2,D)anddiscussedhowthisaffectedthem. Theaverageresponsewas3.7outof7(SD=1.7),andfourof Control the12participantssaidthiswasthemostfrustratingchallenge Similartothechallengeofinstability,usersshouldalwaysbe whileonesaiditwastheleast. incontrolofuserinterfaces[34];however,interactivemachine Five participants said not being able to remember what the learningisbydefinitionacollaborationbetweenalgorithm modellookedlikehurttheirperformance. Forexample,P9 and user [19]. Inthis collaboration, wedefine “control” as said,“amomentago,Iwassatisfiedwiththistopic,butnow whethertheuser’srefinementsareincorporatedintothemodel it’sgone,andIdon’tthinkIam,butIcan’tremember,”and asexpected. Forexample,ifauserremovesawordfroma P3andP8feltthelackof“undo”intensifiedthischallenge. topic,thewordshouldnotbeinthetopicaftertheupdate. P3said,“Ithinkthisisabigissue–I’dliketoknowifI’mcap- Toexplorewhetherparticipantsfeltincontrolofthesystem, turingthetruedata–andbeabletostepbacktoearlyversions theystatedonaseven-pointscalewhethertheyagreedwiththe ofthemodelbeforesaving,”andP8said,“Idon’tknowwhat statement,“therefinementsImadewereappliedasexpected Ihavedonesometimes,andtherearenowaystogoback...”. whenIclickedupdate”fromnotagreeingatalltocompletely Fourparticipantsmentionedasimilarchallenge,thatitwas agreeing(Figure2,B)anddiscussedhowthisaffectedtheir hardtotellwhatchangedinthemodelafteranupdate,such task. Theaverageresponsewas5.6outofseven(SD=1.0), asP10,who“hadtobrushthroughallthewordstoconfirm meaningoverallusersfoundthesystemtobefairlycontrol- if [his specified] change occurred,” and P5, who “did not lable. Oneofthe12participantssaidlackofcontrolwasthe understanditatfirst,thatthemodelactuallychanges,asthere mostfrustratingchallengeandonesaiditwastheleast. was no feedback or indication.” Finally, three participants requestedalong-termhistoryviewofthemodel,suchasP3, However,duringthetasksevenparticipantsnotedfrustration whosuggested“havingahistoryofrefinements.” withthelackofcontrolwiththeinterface,andfiveparticipants A B C D Figure2. Countsforresponsesonascalefromonetosevenforparticipants’agreementwithstatementsrelatedtolatency(A),lackofcontrol(B), instability(C),andtrackingcomplexchanges(D),withsevenmeaningtheydidnotexperienceitandonethattheydid. Mostparticipantsfoundthat thesystemupdatedquicklyandrefinementswereappliedasexpected,whiletherewassubstantialvarianceforifparticipantscouldrememberwhatthe modellookedlikebeforeupdatingoriftheyfelttheupdatedmodelincludedotherchangesthanthosespecified. specificallyobservedthatchangewordorderwasuncontrol- wouldberequiredforthislongerwaittime. P7worriedthis lable.P4triedtodragimportantwordstothefrontofthetopic waitwouldfurtherhindertheabilitytorememberwhatthe listandstatedthat,“thereorderingdidn’talwaysgetaccepted,” modellookedlikebeforeupdating,andP3thoughtthiswould andP8triedtodragunimportantwordstotheendofthelist furtheraffectsavestrategy,suggestingthatitwouldinsteadbe andsaid,“Itriedtomovethiswordanditjustgoesbackup.” “bettertonot‘save’changes,buttohavehighlightstoshow whatit‘might’looklikeoncesaved.” Mostparticipantsfelt thatbothtwominutesand10minuteswouldbeunacceptable Latency waittimes. Priorworkininteractivemachinelearningcallsforrapidin- teractioncycles[2]tominimizeattentionloss[22]andreduce Trustandconfidence short-termmemoryload[34]. However,manyinteractivesys- Trustisaprimarydesignchallengeforintelligentsystems[32, temsdonotprovidereal-timeupdates: thislatency—theuser 20]. Anentiresub-fieldofmachinelearningfocusesonin- hastowaitwhilethealgorithmisperforminganupdate—is terpretability to build trust and promote adoption of these typically related to the size of the data and complexity of systems. Surprisingly, wedid notseeparticipants mistrust- thecomputation. Forexample,anearlyHL-TMimplementa- ing the HL-TM system. While users are quick to distrust a tion[23]took5to50secondstoupdatethemodelbasedon classificationsystemthatproduceseasilyidentifiable,incor- refinementoperations. rectclassifications,topicmodelshavelessobviousincorrect Ourrefinementimplementationisefficientbydesign,andthe answers. datasetusedinthisuserstudywasrelativelysmall(bothin However, participants sometimes put too much trust in the documentsizeandlength),thereforethealgorithmupdated systemorlackedself-confidence. Forexample,P10wascon- almostinstantaneouslyduringthetask(.09 .63seconds).No − fusedaboutatopicword,saying,“ifthesystemcougheditup, participantsaidthatlatencywasthemostfrustratingchallenge theremustbeareasonforit,right?”Someparticipantslacked while two participants said it was the least frustrating, and confidenceintheirrefinements:P7saidthatremovedocument the average response was 6.3 out of seven (SD=0.8) for istheleastusefulrefinement,because,“Idon’tfeelcomfort- participantsagreementwiththestatement,“afterclickingthe ableremovingadocument.” AndwhenP5addedwordstoa updatebutton,themodelupdatedquickly”(Figure2,A). topic,shesaid,“it’sputtingmywordsontop...I’veaddedtoo However, for a more realistic corpus size or alternative re- manywords,whichhavegonetothetopofthelist,soeither finementimplementation,latencybecomesachallenge. We thealgorithmthinksit’simportantorit’sbecauseI’veadded askedparticipantstodescribehowtheirabilitytoperformthe them,”followedby,“Idon’tthinkthatitshouldalwaysgive taskwouldbeaffectedhadthewaittimebeen10seconds,30 more importance [to my added words], because I could be seconds,twominutes,or10minutes.Mostparticipantsfelt10 wrong!” Thischallengehasadirectconnectiontotheissues secondswouldbeanacceptabletimetowait:fiveparticipants ofinstabilityandlackofcontrol,whichwediscussinmore feltthatwaiting10secondswouldhavenoeffectonthetask detailinDiscussionandFutureWork. andtwoparticipantsfeltthatthislongerwaittimewouldhave apositiveeffect,forexample,P5statedthatwaitinglonger Whatotherrequestsdousershaveforthesystem? “wouldbebetterformetorealizethatthetweetshavechanged.” Manyparticipantswantedabetterunderstandingofthemodel Fora30secondwaittime,twoparticipantsfeltthiswouldbe andthedata. Forexample,twoparticipantsrequestedabetter anacceptablewaittimewithoutanychangestotheinterface, modeloverview,suchasP7,whowantedto“seetheentirelist whereas four participants said that changes to the interface ofthetop20wordsforeachtopicononescreentoallowfor makingbulk,fasterchanges.” Additionally,threeparticipants Supportundo: Whenpossible,HL-TMshouldsupportrevert- wanted to view words or documents across topics, such as ingtopriorstatesofthemodel: somenotedthatthismade P12whosuggested,“anoteorcolortoindicatethatacertain themafraidtosaveduringthetask,whileothersspecifically termappearsonlyinthistopicandnotintheothers.” Two requestedanundofunctionality. participantsrequestedenhancingthewordincontextfeature, Allowuserstochoosewhentosave,butremindthemtodo suchasbyscrollingtotheselectedwordorfilteringtoonly so:Wehadanticipatedneedingaseparatesaveactiontoallow documentscontainingtheword. Threeparticipantswantedto userstoconfirmrefinements(lackingundo)andtocounteract viewmoredocumentsthanthe40shown,andtwoparticipants latency,butwealsonoteduserswhocreatedrefinementsasa wantedtoviewthetotalnumberofdocumentsforatopic. dataexplorationtoolwithouttheintentofhavingthemupdate Similar to the merge word operation identified by themodel.Thus,HL-TMsystemsshouldallowuserstochoose Leeetal.[27],sixparticipantsrequestedarefinementtoadd whentosavetheirrefinementstothemodelinsteadofforcing phrases(insteadofjustsinglewords),andfourparticipants a save. However, because users forget to save, additional requestedarefinementtogroupsynonymsandplurals. Asan- informationshouldbeprovidedintheinterfacetoremindusers, ticipated,participantsusedtheaddtostopwordsrefinement, such as a more prominent count of outstanding refinement and two participants requested an enhancement to the stop operationsoravisualcuethatdisplaysiftheyhavenotsaved wordsfunctionality,suchasbeingabletoviewthestopwords recently. listandremovewordsthathavebeenaddedtoit. However, Freezetopicstoprotectfrominstability:Userscomplained sevenparticipantsnotedconfusionbetweentheaddtostop ofinstabilitywhentopicsthatwereoncehighqualityorabout wordsandremovewordrefinements,whichshouldbeclarified aparticularthinghadchanged. Aprocess,suchasfreezing infutureinterfacedesign. Forexample,P5said,“removinga atopic,suggestedbyoneparticipantasamechanismtohold wordfeatureissimilartothedeletefeature,whichgotmea a particular topic constant during subsequent updates, is a bitconfused,”andP9said,“Igotconfusedbetweenremoving promisingsolutiontothisproblemandshouldbeincorporated keywordsfromaparticular[topic]andtheoverall[topics], infuturedesign. soImademistakesinthebeginning.” Tohelpbetterorganize theview,threeparticipantswantedtonametopics,notingthat Supportmulti-wordrefinements:Participantsrequestedthe it would be a useful way to remember what the topics are abilitytoaddphrasesandgroupsynonyms. Groupsynonyms about, and two participants wanted to reorder topics in the couldbeimplementedasthemergewordrefinementdiscussed list. Finally,twoparticipantswantedtodeleteatopicifitwas inLeeetal.[27],notasanupdatetotheunderlyingmodel, particularlybad. butasawayoforganizingwordsintheinterface.Ontheother hand,addphrasesshouldbeimplementedintheinterfaceas Summary anextensiontoaddword(asrequestedbyparticipants),but Participants were frustrated by their inability to track how wouldlikelyrequireamorecomplexmodelingapproachthat themodelchangedthroughouttherefinementprocess.While supportsn-gramsasopposedtosingletokens. participantsperceivedsysteminstability,theyhadvariedre- Clarify difference between adding a word to stop words actions(positiveandnegative). Ontheotherhand,usersdid andremovingitfromasingletopic: Futuredesignshould notexperiencesubstantiallatencyorlackofcontrol. Wedid explicitlydelineatebetweenremovingawordfromalltopics notfindanycaseswhereusersdistrustedthesystem,butusers (andthemodelingprocessentirely),addtostopwords,and perhapstrustedthesystemtoomuchorhadtoolittleconfi- removingawordfromasingletopic,removeword,asmany denceinthemselves. Participantsspecificallyrequestedthe participantsconfusedthetwooperationsduringthetask. ability to undo changes after saving and to curate the topic modelview,suchasbyre-orderingthetopiclist,removing Support user-curated model view: Three participants re- poorqualitytopics,andnamingtopics. Participantsalsore- quested named topics. Two other participants used change questedmulti-wordrefinements,suchasaddingphrasesand wordorderforadhoctopicnaming. Asthisoperationisnot groupingsynonyms. always applied as expected, providing a controllable topic namingfunctionalitywillimproveuserexperience. Partici- DISCUSSIONANDFUTUREWORK pantsalsorequestedothertechniquesforcuratingtheirmodel WeoutlineimplicationsforfutureHL-TMsystemdesign,dis- view, which should be incorporated in the design of future cussopenquestionsrelatedtointeractivemachinelearning, systems, such as the ability to re-order the topic list and to andprovideareflectiononourHL-TMimplementation. removepoorqualitytopicsentirely. DesignRecommendations OpenQuestions Provide richer history: Participants voiced concerns with Thisisthefirstsystemtoefficientlyimplementafullsuiteof theirinabilitytorememberthehistoryofthemodel,andfour refinementsdesiredbyusersinpriorwork[27,31],enabling of12participantssaidtheywereunabletotellhowthemodel thestudyoftruehuman-in-the-loopinteractionsofacompre- haschangedafteranupdate. HL-TMinterfacesshouldstrive hensiveHL-TMsystem.Weenumerateopenquestionsabout to support visualization of short term and long term model HL-TMdesignthatfollowfromourfindings. changes;userswanttotrackhowthemodelchangedthrough- outtherefinementprocess. Thiswasthemostconsistentand Trustvs. instabilityandcontrol: Userswerenotbothered mostfrustratingissueinthestudy. by instability or lack of control either because they trusted

Description:
Author = {Alison Smith and Varun Kumar and Jordan Boyd-Graber and Kevin Seppi and Leah Findlater}, .. Interface. The HL-TM user interface (Figure 1) represents a topic model as a list of topics on the left panel, each displayed as their first three words. or complete agreement (Figure 2, C).
See more

The list of books you might like

Most books are stored in the elastic cloud where traffic is expensive. For this reason, we have a limit on daily download.