Search Personalization

39 Pages·2014·0.55 MB·English

Search Personalization
Hema Yoganarasimhan ∗
UniversityofWashington
October22,2014
Abstract
Online search or information retrieval is one of the most common activities on
the web. Over 18 billion searches are made each month, and search generates over
$4 billion dollars in quarterly revenues. Nevertheless, a signicant portion of search
sessionsarenotonlylong(consistingofrepeatedqueriesfromtheuser),butarealso
unsuccessful,i.e.,donotprovideanswersthattheuserwaslookingfor. Thispresents
challengestosearchenginemarketers,whosecompetitiveedgedependsonthequality
oftheirsearchengineresults. Inthispaper,wepresentamachinelearningalgorithm
thatimprovessearch resultsthroughautomated personalizationusing gradient-based
ranking. We personalize results based on both short-term or “within-session” behavior,
as well as long-term or “across-session” behavior of the user. We implement our
algorithmondatafromthepremiersearchengine,Yandex,andshowthatitimproves
searchresultssignicantlycomparedtonon-personalizedresults.
Keywords: onlinesearch,personalization,bigdata,machinelearning,rankingalgorithms
∗The author is grateful to Yandex for providing the data used in this paper. She also thanks the participants of
the2014UTDallasFORMSandMarketingScienceconferencesforfeedback. Pleaseaddressallcorrespondenceto:
[email protected].
1 Introduction
Personalization refers to the tailoring of a firm’s products or marketing-mix to each individual
customer. Personalizationremainstheholygrailofmarketing,especiallyindigitalsettings,where
it is feasible for firms to collect data on, and respond to the needs of each of its consumers
in a scalable fashion. Arora et al. (2008) distinguish personalization from the related concept
of customization; the former refers to automatic modification of a firm’s offerings based on data
collectedbyit,whereasthelatterreferstocustomizationbasedonproactiveinputfromtheconsumer.
Amazon’sproductrecommendationsbasedonpastbrowsingandpurchasebehaviorisanexample
of personalization, whereas Google’s ad settings that allow consumers to customize the adsshown
to them based on gender, age, and interests is an example of customization. Customization has
thepotentialadvantageofbeingmoresatisfyingfortheconsumer. However,itisimpracticalfor
manysettings,sinceconsumersfinditcostlytoprovidefeedbackonpreferences,especiallywhen
theoptionsarenumerousandcomplex(HuffmanandKahn,1998;DellaertandStremersch,2005).
Moreover, customization is not possible in the case of information goods, where consumption of
thegoodisapre-requisiteforpreferenceelicitation. Forexample,inthecontextofonlinesearch,a
consumerhastoseeandprocessalltheresultsforaquerybeforeshecanrankthemaccordingto
herpreferences. Thuscustomizationcanonlyhappenafterconsumption,makingitpointless.1
Westudypersonalizationinthecontextofonlinesearch. AstheInternetmatures,theamountof
informationandproductsavailableinindividualwebsiteshasgrownexponentially. Today,Google
indexesover30trillionwebpages(Google,2014),Amazonsells232millionproducts(Grey,2013),
Netflix streams more than 10,000 titles (InstantWatcher, 2014), and over 100 hours of video are
uploadedtoYouTubeeveryminute(YouTube,2014). Whilelargeproductassortmentscanbegreat,
theymake it hardfor consumersto locatethe exactproduct or information they arelooking for. To
addressthisissue,mostbusinessestodayuseaquery-basedsearchmodeltohelpconsumerslocate
theproduct/informationthatbestfitstheirneeds. Consumersenterqueries(orkeywords)inasearch
box, and are presented a set of products/information that is deemed most relevant to their query
basedonmachine-learningsearchalgorithms. Weexaminethereturnstopersonalizationofsuch
searchresultsinthecontextofsearchenginesor‘informationretrieval’.
WebsearchengineslikeBaidu,Bing,Google,Yandex,andYahooareanintegralpartoftoday’s
digital economy. In the U.S alone, over 18 billion search queries were made in December 2013
1Notethatpersonalizationisdistinctfromcollaborativefiltering,wherethefirmmatchesauser’spurchaseandbrowsing
patternswiththatofsimilarusers,andmakesrecommendationsforfutureconsumption. Theunderlyingideaisthatif
userAhasthesamepreferenceasuserBonagivenissue,thenAismorelikelytohaveB’spreferenceonadifferent
issueaswell,comparedtoarandomuser. Incontrast,personalizationreferstostrategiesthatemployinformation
onauser’sownpasthistorytoimproveherrecommendations. PleaseseeBreeseetal.(1998)forabasicdescription
ofcollaborativefiltering. Somewell-knownexamplesoffirmsusingcollaborativefilteringbasedrecommendation
systemsareAmazon,Digg,Netflix,Reddit,andYouTube.
2
andsearch advertisinggenerates over$4billionUS dollarsin quarterlyrevenues(Silverman,2013;
comScore,2014). Consumersusesearchenginesforavarietyofreasons;e.g.,togatherinformation
on a topic from authoritative sources, to remain up-to-date with breaking news, to aid purchase
decisions. Searchresultsandtheirrelativerankingcanthereforehavesignificantimplicationsfor
consumers, businesses, governments, and for the search engines themselves. Indeed, according
some statistics, even a 1% reduction in search time on Google can lead to a savings of 187,000
person-hoursor21yearspermonth(Nielsen,2003).
Early search engines provided generic results – for a given query at a given point in time,
everyonesawthesameresults. Aboutadecadeback,mostmajorsearchenginesstartedpersonalizing
resultsbasedongeographiclocationandlanguage. Thesestrategiescanbeinterpretedassegment-
basedpersonalization,sincetheyassumethatconsumerssharingsimilargeographiesorlanguages
also share similar tastes. More recently, search engines have started personalizing based on
individual-level dataonpast browsing behavior. InDecember2009, Googlelaunchedpersonalized
search,which customizes resultsbasedonauser’sIPaddressandcookie, usingupto180daysof
history, even whena user is notsigned into Google(Google Official Blog, 2009). Since then, other
searchenginessuchasBingandYandexhavealsostartedpersonalizingtheirresults(Sullivan,2011;
Yandex,2013). SeeFigure1foranexampleofhowpersonalizationcaninfluencesearchresults.
Sinceitsadoption,theextentandeffectivenessofsearchpersonalizationhasbeenasourceof
intensedebate,partlyfueledbythesecrecyofsearchenginesonhowtheyimplementpersonaliza-
tion.2 Googleitselfarguesthatinformationfromauser’spreviousqueriesisusedinlessthan0.3%
of its searches (Schwartz, 2012). However, in a field study with 200 users, Hannak et al. (2013)
find that 11.7% of results show differences due to personalization. Interestingly, they attribute
thesedifferencestosearchingwithaloggedinaccountandtheIPuser’saddressratherthansearch
history. Thus, the role of search and click history within and across sessions on the extent and
effectivenessofpersonalizationremainsunclear. Indeed,someresearchershaveevenquestioned
the value of personalizing search results, not only in the context of search engines, but also in
that of hotel industry (Feuz et al., 2011; Ghose et al., 2014). In fact, this debate on the returns to
searchpersonalizationispartofalargeroneonthereturnstopersonalizationofanymarketingmix
variable. (See§2fordetails.)
Broadlyspeaking,therearefourmainquestionsofbothresearchandmanagerialinterestthat
wehavenoclearanswersfor. First,howeffectiveissearchpersonalizationinthefield? Inprinciple,
2InGoogle’scase,theonlysourceofinformationonitspersonalizationstrategyisapostinGoogle’sofficialblog,
whichsuggeststhatwhetherauserpreviouslyclickedapageandthetemporalorderofauser’squeriesinfluenceher
personalizedrankings(Singhal,2011). InthecaseofYandex,allthatweknowcomesfromthefollowingquoteby
oneofitsexecutive,Mr.Bakunov–“Ourtestingandresearchshowsthatourusersappreciateanappropriatelevelof
personalizationandincreasetheiruseofYandexasaresult,soitwouldbefairtosayitcanonlyhaveapositiveimpact
onourusers–thoughweshouldstressthatsearchqualityisamoreimportantmetricforus,”(Atkins-Kru¨ger,2012).
3
if consumers’ tastes show persistence over time, personalization using historic information on
users’browsingbehaviorshouldimprovesearchresults. However,wedonotknowthedegreeof
persistenceintastesandextenttowhichitcanimprovesearchexperience. Thus,theeffectiveness
of search personalization is an open question. Second, which features lead to better results from
personalization? For example, which of these metrics is more effective in improving results – a
user’spropensity to click on the first result orher propensity to click on previouslyclicked results?
Feature engineering is an important aspect of any ranking problem, and is therefore treated as a
sourceofcompetitiveadvantageandkeptunderwrapsbymostsearchengines(Liu,2011). However,
allfirmsthatusesearchwouldbenefitfromunderstandingtherelativeusefulnessofdifferenttypes
of features; e.g., online retail stores (Amazon, Macy’s), entertainment sites (Netflix, Hulu), and
online app stores (Apple, Google). Third, what is the relative value of within and across session
personalization? On the one hand, we might expect within session personalization to be more
informative of consumer’s current quest. On the other hand, across session data can help infer
persistence in consumers’ tastes. For example, some consumers may tend click only on the first
result, while others may be more click-happy. Note that within session personalization has to
happeninrealtime;sounderstandingtherelativevalueofwithinandacrosssessionpersonalization
can help us decide the computational load the system can/should handle. Fourth, which ranking
algorithmisthemostappropriateforthistask,andwhatspeedimprovementsdoesitprovide?
WeaddressthesequestionsusingdatafromYandex. Yandexisthefourthlargestsearchengine
in the world, with over 60% marketshare in Russia (Clayton, 2013; Pavliva, 2013). It generated
over $320 billion US dollars in quarterly revenues in 2013, with 40% growth compared to 2012
(Yandex, 2013). It also has a significant presence is other eastern European countries such as
Ukraine and Kazakhstan. Nevertheless, Yandex faces serious competition from Google, and is
lookingtoimproveitssearchqualityinordertoretain/improveitsmarketshare. Therefore,itnot
onlyinvestsininternalR&D,butalsoengageswiththeresearchcommunitytoimproveitssearch
algorithm by sponsoring research competitions (called Yandex Cups) and making anonymized
searchdataavailabletoresearchers. InOctober2013,Yandexreleasedalarge-scaleanonymized
datasetfora‘YandexPersonalizedWebSearchChallenge’,hostedbyKaggle(Kaggle,2013). The
data consists of information on anonymized user identifiers, queries, query terms, URLs, URL
domains and clicks. We use this data to train our empirical framework and answer the questions
outlinedearlier.
We present a machine-learning framework that re-ranks search results based on a user’s
search/click history so that more relevant results are ranked higher. Our framework employs
athree-prongedapproach–a)Featuregeneration,b)NDCG-basedLambdaMartalgorithm,andc)
FeatureselectionusingtheWrappermethod. Thefirstmoduleconsistsofaprogramthatgenerates
a set of features from the training dataset in a scalabe fashion. The second module consists of
4
the LambdaMart ranking algorithm that maximizes search quality, as measured by Normalized
DiscountedCumulativeGain(NDCG).ThethirdmoduleconsistsofaWrapperalgorithmthatwraps
aroundtheLambdaMartsteptoderivethesetofoptimalfeaturesthatmaximizesearchquality.
Ourframeworkofferstwokeyadvantages. First,itisscalableto‘big-data’. Thisisimportant
since search datasets tend to be very large. The Yandex data, which spans one month, consists
of 5.7 million users, 21 million unique queries, 34 million search sessions, and over 64 million
clicks. Weshowthatourfeaturegenerationmoduleiseasilyscalableandiscriticalforthequalityof
searchresults. Since some partsofthefeaturegenerationalgorithmhavetoberuninrealtime (e.g.,
within-sessionfeatures),itsspeedandscalabilityarekeytotheoverallperformanceofthemethod.
We also show that the returns to scaling the LambdaMart routine and the Wrapper algorithm are
lowafteracertainpoint,whichisreassuringsincewealsofindthatthesearenoteasilyscalable.
Thus,inourframeworkthefeaturegenerationmoduleisexecutedontheentiredataset,whereasthe
LamdaMart and Wrapper algorithms are run on a subset of the data. We find that this offers the
bestresultsintermsofbothspeedandperformance. Second, ourframeworkcanworkwithfully
anonymizeddata. Ingeneral,anonymizationresultsinthelossofthecontextofqueryterms,URLs,
and domains, loss of text data in webpages and the hyperlink-structure between them – factors
which have been shown to play a big role in improving search quality (Qin et al., 2010). Thus,
thechallengeliesincomingupwithare-rankingalgorithmthatdoesnotrelyoncontextual,text,
or hyperlink data. Our framework only requires an existing ranking based on such anonymized
variablesandthepersonalsearchhistoryforauser. Asconcernsaboutuserprivacyincrease,thisis
animportantadvantage.
We apply our framework to the Yandex dataset, and present two key results. First, we show
that personalizationleads to a 4%increase in search quality(as measured by NDCG),a significant
improvementinthiscontext. Thisisincontrasttothestandardlogit-stylemodels. Second,wefind
that both types of personalization are valuable – short-term or “within-session” personalization, as
wellaslong-termor“across-session”personalization.
Ourpapermakesthreemaincontributionstomarketingliteratureonpersonalization. First,it
presentsanempiricalframeworkthatmarketerscanusetorankrecommendationsusingpersonalized
data. Second, it presents empirical evidence in support of the returns to personalization in the
online search context. Third, it demonstrates how large datasets or big-data can be leveraged to
improvemarketingoutcomeswithoutcompromisingthespeedorreal-timeperformanceofdigital
applications.
2 Related Literature
Ourpaperrelatestomanystreamsofliteratureinmarketing,computerscience,andeconomics.
First,itrelatestothetheoreticalliteratureonpricepersonalizationinmarketingandeconomics.
5
Earlyresearchinthisareafoundthatpricediscriminationinstaticsymmetricduopoliescandampen
profits(ShafferandZhang,1995;BesterandPetrakis,1996;FudenbergandTirole,2000),especially
iffirmscannotperfectlypredictconsumers’preferences(Chenetal.,2001). Morerecentlyhowever,
ShafferandZhang(2000,2002);Villas-Boas(2006);ChenandZhang(2009)findthatpersonalized
prices can improve profits if we allow for strategic consumers and inter-temporal substitution.
Anotherdimensioninpersonalizedpricingishaggling. DesaiandPurohit(2004)showthatfirms
may find it profitable to allow consumers to haggle if they face heterogenous haggling costs.
Analyticalresults aremixedin thecontext ofproduct personalization too. Dewan etal. (1999)find
that firms are more likely to personalize if it is cheap to so and if they can second-degree price
discriminate. Ontheotherhand,ChenandIyer(2002)showthatfirms’incentivestopersonalize
is high if consumers’ tastes are heterogenous and cost of personalization is high. While these
analyticalmodelsgivevaluableinsightsintothetrade-offsinvolvedinpersonalization,theirbare-
bones framework cannot inform the empirical effectiveness and costs of personalization, especially
insearchsettings,wheremarketing-mixvariableslikepricedon’tplayanyrole.
Second, our paper relates to the empirical literature on personalization in marketing.3 In an
influential paper, Rossi et al. (1996) quantify the benefits of one-to-one pricing using data on
purchase history and find that personalization improves profits by 7.6%. Similarly, Ansari and
Mela (2003) find that content-targeted emails can potentially increase the click-throughs up to
62%,andAroraandHenderson(2007)findthatcustomizedpromotionsintheformof‘embedded
premiums’ orsocial causesassociated withthe productcan improve profits. Onthe otherhand, a
seriesofrecentpapersquestionthereturnstopersonalizationinavarietyofcontextsrangingfrom
advertisingtorankingofhotelsintravelsites(ZhangandWedel,2009;GoldfarbandTucker,2011;
LambrechtandTucker,2013;Ghoseetal.,2014). Substantively,ourpaperspeakstothisongoing
debateby providingempirical supporttothe ideathatthere areindeedreturns topersonalization,
especiallyinrankingsofonlinesearchengines. However,itdiffersfromtheaforementionedpapers
on two dimensions. First, they focus on the personalization of marketing mix variables such as
price and advertising, while we focus on product personalization (search engine results). Even
papersthatconsiderrankingoutcomes, e.g.,Ghoseetal.(2014),usemarketingmixvariablessuch
asadvertisingandpricingtomakerankingrecommendations. However, intheinformationretrieval
context, there are no such marketing-mix variables. Moreover with our anonymized data, we
don’tevenhaveaccesstothetopic/contextofsearch. Second,fromamethodologicalperspective,
previouspapersusecomputationallyintensiveBayesianmethodsthatemployMarkovChainMonte
Carlo(MCMC)procedures,orrandomized experiments,ordiscretechoicemodels–allof which
arenotonlydifficulttoscale,butalsolackpredictivepower(Ansarietal.,2000;RossiandAllenby,
3Thisisdistinctfromtheliteratureonrecommendationsystems. SeeAnsariandJedidi(2000),Chungetal.(2009)in
marketingandPazzaniandBillsus(2007),Jannachetal.(2010)incomputersciencefordiscussionsonsuchsystems.
6
2003;Friedmanetal.,2000). Incontrast,weusemachinelearningmethodsthatcanhandlebigdata
andofferhighpredictiveperformance.
In a similar vein, our paper also relates to the computer science literature on personalization
of search engine rankings. Qiu and Cho (2006) infer user interests from browsing history using
experimentaldataontensubjects. Teevanetal.(2008)andEickhoffetal.(2013)showthatthere
couldbe returnsto personalization forsome specifictypesof searchesinvolvingambiguous queries
andatypicalsearchsessions. Bennettetal.(2012)findthatlong-termhistoryisusefulearlyinthe
session whereas short-term history later in the session. There are two main differences between
thesepapersandours. First,inallthesepapers,theresearchershadaccesstothedatawith‘context’
– information on the URLs and domains of the webpages, the hyperlink structure between them,
andthequeriesusedinthesearchprocess. Incontrast,weworkwithanonymizeddata–wedon’t
have data on the URL identifiers, texts, or links to/from webpages; nor do we know the queries.
We simply workwith anonymized numericalvalues for queries, whichmakes the tasksignificantly
more challenging. We show that personalization is not only feasible, but also beneficial in such
cases. Second,theperviouspapersworkwithmuchsmallerdatasets,whichhelpsthemavoidthe
scalabilityissuesthatweface.
Morebroadly,ourpapercontributestothegrowingliteratureonmachinelearningmethodologies
forLearning-to-Rank(LETOR)methods. Manyinformationretrievalproblemsinbusinesscontexts
oftenboildowntoranking. Hencerankingmethodshavebeenstudiesinmanydifferentcontexts
such as collaborative filtering (Harrington, 2003), question answering (Surdeanu et al., 2008;
Banerjeeetal.,2009),textmining(MetzlerandKanungo,2008),anddigitaladvertising(Ciaramita
etal.,2008). WereferinterestedreaderstoLiu(2011)fordetails.
3 Data
3.1 DataDescription
We use the publicly available anonymized dataset released by Yandex for its personalization
challenge(Kaggle,2013). Thedatasetisfrom2011(exactdatesarenotdisclosed)andrepresents
27daysofsearchactivity. ThequeriesandusersaresampledfromalargecityineasternEurope.
Thedataisprocessedsuchthattwotypesofsessionsareremoved–thosecontainingquerieswith
commercialintent(asdetectedbyYandex’sproprietaryclassifier)andthosecontainingoneormore
of the top-K most popular queries (where K is not disclosed). This minimizes risks of reverse
engineeringofusers’identitiesandbusinessinterests.
Thedatacontainsinformationonthefollowinganonymizedvariables:
• User: Auser,indexedbyi,isanagentwhousesthesearchengineforinformationretrieval.
Usersaretrackedovertimethroughacombinationofcookies,IPaddresses,andloggedin
accounts. Thereare5,736,333uniqueusersinthedata.
7
• Query: Aquery iscomposedofone ormorewords, inresponsetowhich thesearchengine
returns a page of results. There are a total of 21,073,569 queries in the data.4 Figure 2
showsthedistributionofqueriesinthedata,whichfollowsalong-tailpattern–1%ofqueries
accountfor47%ofthesearches,and5%for60%.
• Term: Eachqueryconsistsofoneormoreterms,andtermsareindexedbyl. Forexample,
thequery‘picturesofrichardarmitage’hasthreeterms–‘pictures’,‘richard’,and‘armitage’.
Thereare197,729,360uniquetermsinthedata,andwepresentthedistributionofthenumber
oftermsinallthequeriesseeninthedatainTable1.5
• Session: Asearchsession,indexedbyj,isacontinuousblockoftimeduringwhichauseris
repeatedlyissuingqueries,browsingandclickingonresultstoobtaininformationonatopic.
Searchenginesuseproprietaryalgorithmstodetectsessionboundaries(Go¨kerandHe,2000);
andtheexactmethodusedbyYandexisnotpubliclyrevealed. AccordingtoYandex,thereare
34,573,630sessionsinthedata,andthedistributionofsessionsperuserisshowninFigure3.
Mostusers,≈ 40%,participateinonlyonesearchsession. Thisdoesnotnecessarilyindicate
high user churn because a user may delete cookies, or return from a different device or IP
address. Nevertheless,asignificantnumberofuserscanbetrackedovertime,andthemedian
userparticipatesintwosessions,withthemeanbeing6.1sessionsperusers.
• URL:Afterauserissuesaquery,thesearchenginereturnsasetoftenresultsinthefirstpage.
TheseresultsareessentiallyURLsofwebpagesrelevanttothequery. Thereare703,484,26
uniqueURLsinthedataURLs,andURLsareindexedbyu. Yandexonlyprovidesdatafor
the first Search Engine Results Page (SERP) for a given query because a majority of users
nevergobeyondthefirstpage,andclickdataforsubsequentpagesisverysparse. Sofroman
implementationperspective,itisdifficulttopersonalizelaterSERPs;andfromamanagerial
perspective,thereturnstopersonalizingthemislowsincemostusersneverseethem.
• URLdomain: IsthedomaintowhichagivenURLbelongs. Forexample,www.imdb.com
isadomain,whilehttp://www.imdb.com/name/nm0035514/andhttp://www.
imdb.com/name/nm0185819/areURLsorwebpageshostedatthisdomain.
• Click: Afterissuingaquery,userscanclickononeoralltheresultsontheSERP.Clicking
a URL is an indication of user interest. We observe 64,693,054 clicks in the data. So, on
average, a user clicks on 3.257 results following a query. Table 2 shows the probability of
4Queriesarenotthesameaskeywords,whichisanothertermusedfrequentlyinconnectionwithsearchengines. Please
seeGabbert(2011)forasimpleexplanationofthedifferencebetweenthetwophrases.
5Searchenginestypicallyignorecommonkeywordsandprepositionsusedinsearchessuchas‘of’,‘a’,‘in’,because
theytendtobeuninformative. Suchwordsarecalled‘StopWords’.
8
clickbypositionorrankoftheresult. Notethesteepdropoffinclickprobabilitywithposition
– the first document is gets clicked 44.51% of times, whereas the the fifth one is clicked a
mere5.56%oftimes.
• Time: Yandex gives information on the timeline of each session. However, to preserve
anonymity,itdoesnotdisclosehowmanymillisecondsareinoneunitoftime. Eachaction
inasessioncomeswithatimestampthattellsushowlongintothesessiontheactiontook
place. Using thesetimestamps, Yandexcalculates ‘dwelltime’for eachclick,which isthe
timebetweenaclickandthenextclickorthenextquery. Dwelltimeisinformativeofhow
muchtimeauserspentwithadocument.
• Relevance: Aresultisrelevanttoauserifshefindsituseful,andirrelevantotherwise. Search
enginesattributerelevancescorestosearchresultsbasedonuserbehavior–clicksanddwell
time. Itiswellknownthatdwelltimeiscorrelatedwiththeprobabilitythattheusertosatisfied
herinformationneedswiththeclickeddocument. Thelabelingisdoneautomatically,hence,
differentforeachqueryissuedbytheuser. Yandexusesthreegradesofrelevance:
• Irrelevant,r = 0.
Aresultgetsairrelevantgradeifitdoesn’tgetaclickorifitreceivesaclickwithdwell
timestrictlylessthan50unitsoftime.
• Relevant,r = 1.
Resultswithdwelltimesbetween50and399timeunitsreceivearelevantgrade.
• Highlyrelevant,r = 2.
Isgiventotwotypesofdocuments–a)thosethatreceivedclickswithadwelltimeof400
unitsormore,andb)ifthedocumentwasclicked,andtheclickwasthelastactionofthe
session(irrespectiveofdwelltime). Thelastclickofasessionisconsideredtobehighly
relevantbecauseauserisunlikelytohaveterminatedhersearchifsheisnotsatisfied.
Incaseswhereadocumentisclickedmultipletimesafteraquery,themaximumdwelltimeis
usedtocalculatethedocument’srelevance.
Insum,wehaveatotalof167,413,039linesofdata.
3.2 Model-FreeAnalysis
Wenowpresentsomemodel-freeevidencethatsupportsthepresenceofpersistenceinusers’tastes,
andpositivereturnstopersonalization. Forpersonalizationtobeeffective,weneedtoobservetwo
types of patterns in the data. First, there has to significant heterogeneity in users’ tastes. Second,
users’tasteshavetoexhibitpersistenceovertime. Below,weexaminetheextenttowhichsearch
andclickpatternssupportthesetwohypotheses.
9
First, we need data on user history to model personalization. So we start by examining user
historymetrics. Figure4presentstheCumulativeDistributionFunction(CDF)ofthenumberof
queriesissuedbyauser. 20%ofusersissueonlyonequery,makinganypersonalizationimpossible
forthem. However,50%ofusersissue4ormorequeries,withthemedianqueriesperuserbeing4.
Further, if we examine the number of queries per session (see Figure 5), we find that 60% of the
sessionshaveonlyonequery. Thisimpliesthatwithinsessionpersonalizationisnotpossiblefor
suchsessions. However,20%ofsessionshavetwoormorequeries,makingthemamenabletowithin
sessionpersonalization. Next, consider thefeasibility ofacross-session personalization. Close to
40% of users participate in only one session (Figure 3). While across-session personalization is
notpossiblefortheseusers,wedoobserveasignificantnumberofrepeat-sessionusers–40%of
usersparticipatein4ormoresessionsand20%in8ormore. Together,thesepatternssuggestthata
reasonabledegreeofbothshortandlong-termpersonalizationisfeasibleinthiscontext.
Figure 6 presents the distribution ofthe average number ofclicks per query forthe users in the
data. Aswecansee,thereisconsiderableheterogeneityinusers’propensitytoclick. Forexample,
more 15% of users don’t click at all, whereas 20% of them click on 1.5 results or more for each
querytheyissue. Toensurethatthesepatternsarenotdrivenbyone-timeorinfrequentusers,we
present the same analysis for users who have issued at least 10 queries in Figure 7, which shows
similarclick patterns. Together, thesetwofigures establishthe presenceof significant user-specific
heterogeneityintasteforclicking.
Next,weexaminewhether thereisvariationorentropyin theURLsthatareclickedfor agiven
query. Figure 8 presents the cumulative distribution of the number of unique URLs clicked for
a given query through the time period of the data. 37% of queries receive no clicks at all, and
closetoanother37%receiveaclicktoonlyoneuniqueURL.Thesearelikelytobenavigational
queries; e.g., almost everyone who searches for ‘cnn’ will click on www.cnn.com. However,
closeto30%ofqueriesreceiveclicksontwoormoreURLs. Thissuggeststhatusersareclicking
on multiple results. This could be either due to heterogeneity in users’ preferences for different
resultsorbecausethereissignificantchurnintheresultsforthequery(apossibilityfornewsrelated
queries,suchas‘earthquake’). ToexamineifthelastexplanationdrivesallthevariationinFigure8,
weplothowmanyuniqueURLsanddomainsweseeforagivenquerythroughthecourseofthe
datainFigure9. Wefindthatmorethan90%ofquerieshavenochurnintheURLsshowninresults
page. Hence, it is unlikely that the entropy in clicked URLs is driven by churn in search engine
results. Rather,itismorelikelytobedriveninheterogeneityinusers’tastes.
Insum,thebasicpatternsinthedatapointtosignificantheterogeneityusers’preferencesand
persistenceofthosepreferences.
10