Learning to Effectively Select Topics For Information Retrieval Test Collections 7 1 MucahidKutlu1,TamerElsayed1,andMatthewLease2 0 2 1Dept.ofComputerScienceandEngineering,QatarUniversity,Qatar b 2SchoolofInformation,UniversityofTexasatAustin,USA e F 8 March1,2017 2 ] R Abstract I . Whiletestcollectionsarecommonlyemployedtoevaluatetheeffectivenessofinformationretrieval(IR)sys- s c tems, constructing these collections has become increasingly expensive in recent years as collection sizes have [ grownever-larger. Toaddressthis,weproposeanewlearning-to-ranktopicselectionmethodwhichreducesthe number of search topics needed for reliable evaluation of IR systems. As part of this work, we also revisit the 3 deepvs.shallowjudgingdebate: whetheritisbettertocollectmanyrelevancejudgmentsforafewtopicsora v 0 fewjudgmentsformanytopics.Weconsideranumberoffactorsimpactingthistrade-off:howtopicsareselected, 1 topic familiarity to judges, and how topic generation cost may impact both budget utilization and the resultant 8 qualityofjudgments. ExperimentsonNISTTRECRobust2003andRobust2004testcollectionsshownotonly 7 ourmethod’sabilitytoreliablyevaluateIRsystemsusingfewertopics,butalsothatwhentopicsareintelligently 0 selected,deepjudgingisoftenmorecost-effectivethanshallowjudginginachievingthesamelevelofevaluation 1. reliability.Topicfamiliarityandconstructioncostsarealsoseentonotablyimpacttheevaluationcostvs.reliability 0 tradeoffandprovidefurtherevidencesupportingdeepjudginginpractice. 7 1 : 1 Introduction v i X Test collections provide the cornerstone for system-based evaluation of information retrieval (IR) algorithms in r theCranfieldparadigm[17]. SuchtestcollectionsarevitaltoensuringcontinuingfieldprogressbyenablingA/B a testing of new search algorithms and community benchmarking. However, realistic evaluation requires testing IR systemsatthescaleofinformationthatistobesearchedinpractice,andthishasbecomeincreasinglyproblematic as today’s collection sizes have grown ever-larger. In particular, because larger collections tend to contain more relevant documents (which retrieval systems might return in response to a given search query), human assessors must judge the relevance of ever-more documents for each search topic. Moreover, if insufficient documents are judged, evaluation findings could be compromised [10]. Consequently, a key challenge in IR today is to devise newevaluationmethodstoreducethecostofconstructingtestcollectionswhilecontinuingtoensurethevalidityof findings. 1 Onedirectionofresearchtolowercostshassoughttoreducethenumberoftopicsforwhichhumanrelevance judging is needed at all. NIST TREC test collections have traditionally used 50 search topics, with Buckley and Voorhees [11] reporting that at least 25 topics are needed for stable evaluation, with 50 being better. However, Zobel[60]showedthatonesetof25topicspredictedrelativeperformanceofsystemsfairlywellonadifferentset of25topics. Guiver,Mizzaro,andRobertson[22]conductedasystematicstudyshowingthatevaluatingIRsystems usingthe“right”subsetoftopicsyieldsverysimilarresultsvs.evaluatingsystemsoveralltopics. However, they didnotproposeamethodtofindsuchaneffectivetopicsubsetinpractice. NISTemploysasimpleandeffectivebut costlytopicselectionprocesswhichincludesinitialjudgmentsforeachtopicandmanualselectionfromcandidate topics[54]. Hosseinietal.[27]recentlyproposedaniterativemethodtoautomaticallyfindeffectivetopicsubsets tolowerevaluationcostswhilepreservingevaluationreliability. Foragivenevaluationbudget,anotherrelatedquestioniswhetheritisbettertocollectmanyrelevancejudgments forafewtopics–i.e.,NarrowandDeep(NaD)judging–orafewrelevancejudgmentsformanytopics–i.e.,Wide and Shallow (WaS) judging. Intuitively, since people search for a wide variety of topics with a wide variety of queries, it seems that realistic IR evaluation ought to evaluate systems across a similarly wide variety of search topics and queries. Empirically, large variance in search accuracy is often observed for the same system across different topics [7], suggesting merit in sampling many diverse topics for evaluation in order to achieve stable evaluation among systems being compared. Studies on this issue have led to a fairly consistent finding that WaS judgingtendstoprovidemorestableevaluationforthesamehumaneffortvs.NaDjudging[48,16,9]. However, thisfindingdoesnotholdinallcases.Forexample,Carteretteetal.[14]achievethesamereliabilityusing250topics with20judgmentspertopic(5000judgmentsintotal)as600topicswith10judgmentspertopic(6000judgmentsin total). Moreover,ifweaccountforthetimerequiredtocreateeachtopic,therelativeexpenseofusingmoretopics isfurtherincreased. Inthiswork,weinvestigatehowtoautomaticallyselectinformativesearchtopicsforIRevaluation[22,27]. We beginwithourfirstresearchquestion: • RQ-1: Intelligent Topic Selection. How can we minimize the number of topics (hence the cost) required forreliableevaluationofIRsystems? Towardthisend,weproposeanovellearning-to-rank(L2R)approach in which topics are selected iteratively via a greedy method which optimizes accurate ranking of systems (Section 4.3). Our L2R model utilizes 63 features representing the interaction between topics and ranking of systems (Section 4.3.1). In order to train our model, we describe a method to automatically generate usefultrainingdatafromexistingtestcollections(Section4.3.2). WeevaluateourapproachonNISTTREC Robust 2003 [53] and Robust 2004 [52] test collections, comparing our approach to recent prior work [27] and random topic selection (Section 5). Results show consistent improvement over baselines, with greater relativeimprovementasfewertopicsareused. Asmentionedabove,wealsorevisitthedeepvs.shallowjudgingdebate,investigatinghowintelligentselectionof topicsimpactsoptimizationofevaluationbudget. Inrelationtopaststudies, ourworkisdistinguishedbyinvesti- gatingawiderrangeoffactorsimpactingthistradeoffthanpreviouslyconsidered: • RQ-2: NaDvs.WaS.PriorstudiesofNaDvs.WaSjudginghaveassumedrandomselectionoftopics. How doesintelligentselectionoftopics(byitself)impactevaluationbudgetoptimizationinNaDvs.WaSjudging? WefindthatNaDjudgingoftenachievesgreaterevaluationreliabilityforthesamebudgetthanWaSjudging whentopicsareselectedintelligently. • RQ-3: Judging Speed. Past comparisons between NaD vs. WaS judging have typically assumed constant judging speed [16, 48]. However, data reported by Carterette et al. [15] suggests that assessors may judge 2 documentsfasterastheyjudgemoredocumentsforthesametopic(likelyduetoincreasedtopicfamiliarity). AssumingWaSjudgingleadstoslowerjudgingspeedthanNaDjudging,howdoesthisimpactoptimization ofevaluationbudget? BecausewecancollectmorejudgmentsinthesameamountoftimewithNaDvs.WaS judging,NaDjudgingachievesgreaterrelativeevaluationreliabilitythanshowninpriorstudieswhichdidnot considerthisjudgingspeedbenefitofdeepjudging. • RQ-4:TopicgenerationTime.PriorNaDvs.WaSstudieshavetypicallyignorednon-judgingcostsinvolved intestcollectionconstruction. WhileCarteretteetal.[14]considertopicgenerationtime,the5-minutetime theyassumedisroughlytwoordersofmagnitudelessthanthe4hoursNISThastraditionallytakentocon- struct each topic [1]. How do historically large topic generation costs impact optimization of evaluation budgetintheNaDvs.WaStradeoff? WefindthatWaSjudgingispreferablethanNaDjudgingforshorttopic generationtimes(specifically≤5minutesinourexperiments). However,asthetopicgenerationcostfurther increases,NaDjudgingbecomesincreasinglypreferableoverWaSjudginginoptimizingevaluationbudget. • RQ-5: Judging Error. Several studies have reported calibration effects impacting the decisions and con- sistency of relevance assessors [47, 49]. While NIST has traditionally included an initial “burn-in” judging periodaspartoftopicgenerationandformulation[1],wepositthatdrasticallyreducingtopicgenerationtime (e.g., from4hours[1]to2minutes[15])couldnegativelyimpacttopicquality, leadingtolesswell-defined topics and/or calibrated judges, and thereby less reliable judgments. As suggestive evidence, McDonnel et al. [33] report high judging agreement in reproducing a “standard” NIST track, but high and inexplicable judgingdisagreementonTREC’sMillionQuerytrack[15],whichlackedanyburn-inperiodforjudgesand hadfarshortertopicgenerationtimes. Assumingshorttopicgenerationtimesreduceconsistencyofrelevance judgments, how does resultant judging error impact evaluation budget optimization in balancing deep vs. shallowjudging? Toinvestigatethis,wesimulateincreasedjudgingerrorasafunctionoflowertopicgener- ationtimes. Wefindthatitisbettertoinvestaportionofourevaluationbudgettoincreasequalityoftopics, instead of collecting more judgments for low-quality topics. This also makes NaD judging preferable over WaSjudginginmanycases,duetoincreasedtopicgenerationcost. Theremainderofthisarticleisorganizedasfollow. Section2reviewstherelatedworkontopicselectionand topic set design. Section 3 formally defines the topic selection problem. In Section 4, we explain our proposed L2R-basedapproachindetail. Section5presentsourexperimentalevaluation. Finally, Section6summarizesthe contributionsofourworkandsuggestspotentialfuturedirections. 2 Related Work Constructingtestcollectionsisexpensiveinhumaneffortrequired. Therefore,researchershaveproposedavariety of methods to reduce the cost of creating test collections. Proposed methods include: developing new evaluation measuresandstatisticalmethodsforthecaseofincompletejudgments[5,12,41,59,58],findingthebestsample ofdocumentstobejudgedforeachtopic[18,13,29,36,39],inferringrelevancejudgments[6],topicselection[27, 28,35,22],evaluationwithnohumanjudgments[38,50],crowdsourcing[4,21],andothers. Thereaderisreferred to[37]and[46]foramoredetailedreviewofpriorworkonmethodsforlow-costIRevaluation. 3 2.1 TopicSelection To the best of our knowledge, Mizzaro and Robertson [35]’s study was the first seeking to select the best subset oftopicsforevaluation. Theyfirstbuildasystem-topicgraphrepresentingtherelationshipbetweentopicsandIR systems,thenruntheHITSalgorithmonit.Theyhypothesizethattopicswithhigherhubnessscoreswouldbebetter todistinguishbetweensystems. However,Robertson[40]experimentallyshowsthattheirhypothesisisnottrue. Guiver, Mizzaro, andRobertson[22]experimentallyshowthatifwechoosetherightsubsetoftopics, wecan achieve a ranking of systems that is very similar to the ranking when we employ all topics. However they do not provide a solution to find the right subset of topics. This study has motivated other researchers to investigate this problem: Berto, Mizzaro, and Robertson [8] stress generality and shows that a carefully selected good subset of topics to evaluate a set of systems can be also adequate to evaluate a different set of systems. Hauff et al. [23] reportthatusingtheeasiesttopicsbasedonJensen-ShannonDivergenceapproachdoesnotworkwelltoreducethe numberoftopics. Hosseinietal.[28]focusonselectingthesubsetoftopicstoextendanexistingcollectioninorder toincreaseitsre-usability. KazaiandSung[30]reducethecostforpreferencebasedIRevaluation. Thecloseststudytoourownis[27],whichemploysanadaptivealgorithmfortopicselection. Itselectsthefirst topicrandomly. Onceatopicisselected,therelevancejudgmentsareacquiredandusedtoassistwiththeselection of subsequent topics. Specifically, in the following iterations, the topic that is predicted to maximize the current Pearson correlation is selected. In order to do that, they predict relevance probabilities of qrels for the remaining topics using a Support Vector Machine (SVM) model trained on the judgments from the topics selected thus far. Training data is extended at each iteration by adding the relevance judgments from each topic as it is selected in ordertobetterselectthenexttopic. Further studies investigate topic selection for other purposes, such as creating low-cost datasets for training learning-to-rank algorithms [34], system rank estimation [24], and selecting training data to improve supervised datafusionalgorithms[32]. Thesestudiesdonotconsidertopicselectionforlow-costevaluationofIRsystems. 2.2 HowManyTopicsAreNeeded? Pastworkhasinvestigatedtheidealsizeoftestcollectionsandhowmanytopicsareneededforareliableevaluation. WhiletraditionalTRECtestcollectionsemploy50topics,anumberofresearchersclaim50topicsarenotsufficient for a reliable evaluation [29, 55, 51]. Many researchers report that wide and shallow judging is preferable than narrowanddeepjudging[48,16,9],withoutprovidingthebalancebetweenpooldepthandthenumberoftopics. Carterette et al. [14] experimentally compare deep vs. shallow judging in terms of budget utilization. They find that20judgmentswith250topicsisthemostcost-effectiveintheirexperiments. Urbano,Marrero,andMartn[51] measurethereliabilityofTRECtestcollectionswithregardtogeneralizationandconcludethatthenumberoftopics neededforreliableevaluationvariesacrossdifferenttasks. In order to calculate the number of topics required, Webber, Moffat, and Zobel [57] propose adding topics iterativelyuntildesiredstatisticalpowerisreached,whileSakaiproposesmethodsbasedontwo-wayANOVA[45], confidence interval [43], t test and one-way ANOVA [42]. In another work, Sakai also compares his previous methodsandprovidesguidelinesfortestcollectiondesignwithagivenfixedbudget[44]. Whiletheseworksfocus oncalculatingthenumberoftopicsrequired,ourworkfocusesonhowtoselectthebesttopicsetforagivensizein ordertomaximizethereliabilityofevaluation. Wealsointroducefurtherconsiderationsimpactingthedebateover shallowvs.deepjudging: familiarizationofuserstotopics, andtheeffectoftopicgenerationcostsonthebudget utilizationandthequalityofjudgmentsforeachtopic. 4 2.3 TopicFamiliarityvs.JudgingSpeed Carterette et al. [15] report that as the number of judgments per topic increases (when collecting 8, 16, 32, 64 or 128 judgments per topic), the median time to judge each document decreases respectively: 15, 13, 15, 11 and 9 seconds. Thissuggeststhatassessorsbecomemorefamiliarwiththetopicastheyjudgemoredocumentsforit,and thisgreaterfamiliarityyieldsgreaterjudgingspeed. However,priorworkcomparingdeepvs.shallowjudgingdid not consider this, instead assuming that judging speed is constant regardless of judging depth. Consequently, our experimentsinSection5.4revisitthisquestion,consideringhowfasterjudgingwithgreaterjudgingdepthpertopic mayimpactthetradeoffbetweendeepvs.shallowjudginginmaximizingevaluationreliabilityforagivenbudget inhumanassessortime. 2.4 TopicGenerationCostvs.JudgingConsistency Pastworkhasutilizedavarietyofdifferentprocessestogeneratesearchtopicswhenconstructingtestcollections. These different processes explicitly or implicitly enact potentially important tradeoffs between human effort (i.e. cost) vs. quality of the resultant topics generated by each process. For example, NIST has employed a relatively costlyprocessinordertoensurecreationofveryhighqualitytopics[1]: Forthetraditionaladhoctasks,assessorsgenerallycametoNISTwithsomeroughideasfortopicshav- ingbeentoldthetargetdocumentcollection. Foreachidea,theywouldcreateaqueryandjudgeabout 100documents(unlessatleast20ofthefirst25wererelevant,inwhichcasetheywouldstopat25and discardtheidea). Fromthesetofcandidatetopicsacrossallassessors,NISTwouldselectthefinaltest setof50basedonload-balancingacrossassessors,numberofrelevantfound,eliminatingduplication ofsubjectmatterortopictypes,etc. Thejudgingwasanintrinsicpartofthetopicdevelopmentroutine becauseweneededtoknowthatthetopichadsufficientlymany(butnottoomany)relevantinthetarget documentset. (Thesejudgmentsmadeduringthetopicdevelopmentphasewerethendiscarded. Qrels werecreatedbasedonlyonthejudgmentsmadeduringtheofficialjudgmentphaseonpooledpartici- pantresults.) Weusedaheuristicthatexpectedoneoutofthreeoriginalideaswouldeventuallymake itasatestsettopic. Creatingasetof50topicsforanewswireadhoccollectionwasbudgetedatabout 175-225assessorhours,whichworksouttoabout4hoursperfinaltopic. In contrast, the TREC Million Query (MQ) Track used a rather different procedure to generate topics. In the 2007 MQ Track [3], 10000 queries were sampled from a large search engine query log. The assessment system showed10randomlyselectedqueriestoeachassessor,whothenselectedoneandconverteditintoastandardTREC topicbyback-fittingatopicdescriptionandnarrativetotheselectedquery. Carteretteetal.[14]reportthatmedian timeofgeneratingatopicwasroughly5minutes.Inthe2008MQTrack[2],assessorscouldrefreshlistofcandidate 10queriesiftheydidnotwanttojudgeanyofthecandidateslisted. Carteretteetal.[15]reportthatmediantimefor viewingalistofqueriesis22secondsandback-fittingatopicdescriptionis76seconds. Onaverage,eachassessor viewed 2.4 lists to generate each topic. Therefore, the cost of generating a topic is roughly 2.4∗22+76 ≈ 129 seconds,or2.1minutes. Theexamplesaboveshowavastrangeoftopiccreationtimes: from4hoursto2minutespertopic. Therefore, inSection5.4,weinvestigatedeepvs.shallowjudgingwhencostofgeneratingtopicsisalsoconsidered. Inadditiontoconsideringtopicconstructiontime,wemightalsoconsiderwhetheraggressivereductionintopic creationtimemightalsohaveotherunintended,negativeimpactsontopicquality. Forexample,Scholeretal.[49] 5 report calibration effects change judging decisions as assessors familiarize themselves with a topic. Presumably NIST’s 4 hour topic creation process provided judges ample time to familiarize themselves with a topic, and as noted above, judgments made during the topic development phase were then discarded. In contrast, it seems MQ trackassessorsbeganjudgingalmostimmediatelyafterselectingaqueryforwhichtoback-fitatopic,andwithno initialtopicformationperiodforestablishingthetopicanddiscardinginitialtopicsmadeduringthistime. Further empiricalevidencesuggestingqualityconcernswithMQtrackjudgmentswasalsorecentlyreportedbyMcDonnell etal.[33],whodescribeadetailedjudgingprocesstheyemployedtoreproduceNISTjudgments. Whiletheauthors report high agreement between their own judging and crowd judging vs. NIST on the 2009 Web Track, for NIST judgmentsfromthe2009MQtrack,theauthorsandcrowdjudgeswerebothconsistentwhiledisagreeingoftenwith NIST judges. The authors also reported that even after detailed analysis of the cases of disagreement, they could not find a rationale for the observed MQ track judgments. Taken in sum, these findings suggest that aggressively reducingtopiccreationtimemaynegativelyimpactthequalityofjudgmentscollectedforthattopic. Forexample, whileanassessorisstillformulatingandclarifyatopicforhimself/herself,anyjudgmentsmadeatthisearlystageof topicevolutionmaynotbeself-consistentwithjudgmentsmadeoncethetopicisfurthercrystallized.Consequently, inSection5.4werevisitthequestionofdeepjudgingoffewtopicsvs.shallowjudgingofmanytopics,assuming thatlowtopiccreationtimesmaybealsomeanlessconsistentjudging. 3 Problem Definition Inthissection,wedefineourtopicselectionproblem. WeassumethatwehaveaTREC-likesetup:adatacollection hasalreadybeenacquired,alargepooloftopicsandrankedlistsofIRsystemsforeachtopicarealsoavailable.Our goalistoselectacertainnumberoftopicsfromthetopicpoolsuchthatevaluationwiththoseselectedtopicsyields themostsimilarrankingoftheIRsystemstothe“ground-truth”. Weassumethattheground-truthrankingofthe IRsystemsistheonewhenweusealltopicsinthepoolforevaluation. We can formulate this problem as follows. Let T = {t ,t ,...,t } denote the pool of N topics, S = 1 2 N {s ,s ,...,s } denote the set of K IR systems to be evaluated, and R denote the ranking of systems in 1 2 K S whentheyareevaluatedbasedonevaluationmeasureeoverthetopicsetT (notationusedinequationsandal- gorithmsisshowninTable1). WeaimtoselectasubsetP ⊂ T ofM topicsthatmaximizesthecorrelation(asa measureofsimilaritybetweentworankedlists)betweentherankingofsystemsoverP (i.e., consideringonlyM topicsandtheircorrespondingrelevancejudgments)andtheground-truthrankingofsystems(overT). Mathemati- caldefinitionofourgoalisasfollows: max corr(R ,R ) (1) P⊂T,|P|=M wherecorrisarankingsimilarityfunction(e.g.,Kendall-τ [31]). 4 Proposed Approach The problem we are tackling is challenging since we do not know the actual performance of systems (i.e., their performancewhenalltopicsareemployedforevaluation)andwewouldliketofindasubsetoftopicsthatachieves similarrankingtotheunknownground-truth. 6 Table1: Notationusedinequationsandalgorithms Symbol Name T Topicpool S IRsystemsparticipatedtothepoolofthecorrespondingtestcollection R Ranking of systems in S when they are evaluated based on evaluation measureeoverthetopicsetT N Sizeoftopicpool M Numberoftopicstobeselected D Thedocumentpoolfortopict tc c L Therankedlistresultingfromsystems forthetopict sj(tc) j c Todemonstratethecomplexityoftheproblem,letusassumethatweobtainthejudgmentsforalltopic-document pairs(i.e.,weknowtheground-truthranking). Inthiscase,wehavetocheck(cid:0)N(cid:1)possibilitiesofsubsetsinorder M to find the optimal one (i.e., the one that produces a ranking that has the maximum correlation with the ground- truth ranking). For example, if N = 100 and M = 50, we need to check around 1029 subsets of topics. Since thisiscomputationallyintractable,weneedanapproximationalgorithmtosolvethisproblem. Therefore,wefirst introduceagreedyapproachtoselectthebestsubsetoftopicswhenwehavethejudgmentsforallquery-document pairs(Section4.1). Subsequently,wediscusshowwecanemploythisgreedyapproachinthelackoftherelevance judgments(Section4.2)andthenintroduceourL2R-basedtopicselectionapproach(Section4.3). 4.1 GreedyApproach We first explore an oracle greedy approach that selects topics in an iterative way when relevance judgments are already obtained. Instead of examining all possibilities, at each iteration, we select the ’best’ topic (among the currently non-selected ones) that, when added to the currently-selected subset of topics, will produce the ranking thathasthemaximumcorrelationwiththeground-truthrankingofsystems. Algorithm 1 illustrates this oracle greedy approach. First, we initialize set of selected topics (P) and set of candidatetopicstobeselected(P¯)(Line1). ForeachcandidatetopictinP¯,werankthesystemsovertheselected topicsP inadditiontot(R ),andcalculatetheKendall’sτ achievedwiththisranking(Lines3-4). We then pick the topic achieving the highest Kendall-τ score among other candidates (Line 5) and update P and P¯ accordingly(Lines6-7). WerepeatthisprocessuntilwereachthetargetedsubsetsizeM (Lines2-7). 7 Algorithm1AnOracleGreedyApproachforTopicSelection 1: P ←∅;P¯ ←T (cid:46) selectedsetP isinitiallyempty 2: while|P|, R) 5: topict∗ ←max(τt) (cid:46)chooset∗yieldingthebestcorrelation t∈P¯ 6: P ←P ∪{t∗} (cid:46)addt∗totheselectedsetP 7: P¯ ←P¯−{t∗} While this approach has O(M ×N) complexity (which is clearly much more efficient compared to selecting theoptimalsubset), itisalsoimpracticalduetoleveragingtherealjudgments(whichwetypicallydonothavein advance)inordertocalculatetheground-truthrankingandtherebyKendall-τ scores. 4.2 PerformancePredictionApproach OnepossiblewaytoavoidtheneedfortheactualrelevancejudgmentsistopredicttheperformanceofIRsystems usingautomaticevaluationmethods(e.g.,[50,38])andthenrankthesystemsbasedontheirpredictedperformance. However,relyingonpredictedperformancetorepresenttheground-truthisproblematicsimplybecauseinaccurate predictionsresultinnoisypredictedground-truth,whichcausestheselectionoftopicsthatactuallydonotproduce maximumcorrelationwiththerealground-truthranking. OnenoteworthystudyusingthisapproachisHosseinietal.[27]’sworkwhichpredictsrelevanceprobabilityof document-topicpairsbyemployinganSVMclassifierandselectstopicsinagreedywaysimilartotheapproachin Algorithm1. WeusetheirselectionapproachasabaselineinourexperimentspresentedinSection5. 4.3 ProposedLearning-to-RankApproach In this work, we formulate the topic selection problem as a learning-to-rank (L2R) problem. In a typical L2R problem, we are given a query q and a set of documents D, and a model is learned to rank those documents in terms of relevance with respect to q. The model is trained using a set of queries and their corresponding labeled documents. Inourcontext,wearegivenasetofcurrently-selectedtopicsetP (analogoustothequeryq)andthe setofcandidatetopicsP¯ tobeselectedfrom(analogoustothedocumentsD),andweaimtotrainamodeltorank the topics in P¯ based on the expected effectiveness of adding each to P. The training samples used to train the modelaretuplesoftheform(P,t,corr(R ,R )),wherethemeasuredcorrelationisconsidered thelabelofthetopictwithrespecttoP. Noticethat,inthetrainingdata,thecorrelationiscomputedusingthetrue relevancejudgments. Thisenablesustousethewealthofexistingtestcollectionstoacquiredatafortrainingour modelasexplainedinSection4.3.2. Weapply thisL2Rproblemformulation tothetopic selection problemusingour greedyapproach. Insteadof picking the topic that leads to the best correlation with the ground-truth among all candidate topics, we use the trainedL2Rmodeltorankthecandidatetopicsandthenpicktheoneatthefirstrank. Thealgorithmisillustrated inAlgorithm2. Ateachiteration, afeaturevectorv iscomputedforeachcandidatetopictinP¯ usingafeature t extractionfunctionf (Lines3-4),detailedinSection4.3.1. Thecandidatetopicsarethenrankedusingourlearned 8 model(Line5)andthetopicinthefirstrankispicked(Line6). Finally,thetopicsetsP andP¯ areupdated(Lines 7-8)andanewiterationisstarted,ifnecessary. Algorithm2L2R-basedTopicSelection 1: P ←∅;P¯ ←T (cid:46) selectedsetP isinitiallyempty 2: while|P|

Learning to Effectively Select Topics For Information Retrieval Test Collections

Similar Learning to Effectively Select Topics For Information Retrieval Test Collections