Regression and Learning to Rank Aggregation for User Engagement Evaluation

0.11 MB·English

by  Hamed Zamani

Regression and Learning to Rank Aggregation
for User Engagement Evaluation
Hamed Zamani, Azadeh Shakery, and Pooya Moradi
SchoolofElectricalandComputerEngineering,CollegeofEngineering,
UniversityofTehran,Tehran,Iran
{h.zamani, shakery, po.moradi}@ut.ac.ir
5
1 ABSTRACT Millions of users are sharing rich information using social
0 User engagement refers to the amount of interaction an in- media sites, such as Twitter, which can be used by social
2 stance(e.g.,tweet,news,andforumpost)achieves. Ranking recommender systems [12]. Item providers often let users
n the items in social media websites based on the amount of express their opinion about an item in social networks. For
instance, users can give a rating to each movie in Internet
a user participation in them, can be used in different appli- MovieDatabase(IMDb)website1 andalsoshareitinTwit-
J cations, such as recommender systems. In this paper, we
9 consider a tweet containing a rating for a movie as an in- ter. This intensifies the importance of considering social
media sites for recommendation and information filtering
2 stanceandfocusonrankingtheinstancesofeachuserbased
systems [31].
on their engagement, i.e., the total numberof retweets and
Product rating prediction is a traditional recommender
] favorites it will gain.
R For this task, we defineseveral features which can be ex- system problem which has been studied extensively in the
literature[10,23,24]. Oneimportantissueinrecommender
I tracted from the meta-dataof each tweet. The features are
. systemsistheengagementwhichcanbegainedbytheusers’
s partitioned into three categories: user-based, movie-based,
c and tweet-based. We show that in order to obtain good re- comments/opinions. When users share their comments on
[ sults, features from all categories should beconsidered. We differentitems, theamount ofuser interactionsachieved by
exploitregression andlearningtorankmethodstorankthe eachcommentcanbeusedtoimprovethequalityofrecom-
1 mender systems. In this paper, we focus on ranking these
tweetsandproposetoaggregatetheresultsofregressionand
v
comments bytheir engagements.
7 learning to rank methods toachieve betterperformance.
WefocusonmovieratingstweetedbyIMDbusersinTwit-
6 We have run our experiments on an extended version of
ter. Hereafter, we use the word“engagement”as the user
4 MovieTweetingdatasetprovidedbyACMRecSysChallenge
interaction which is expressed by adding up the number of
7 2014. Theresultsshow thatlearning torankapproach out-
retweetsandfavoritesatweethasgained. Ourpurposeisto
0 performsmostoftheregressionmodelsandthecombination
rank the tweets of each user, each containing a rating for a
. can improvetheperformance significantly.
1 movie in IMDb,by theirengagements.
0 For this task, we first extract several features from the
Categories andSubject Descriptors
5 tweets. Thefeaturesarecategorizedintothreegroups: user-
1 H.2.8[Database Management]: DataMining;J.4[Com- based, movie-based, and tweet-based. It should be noted
v: puter Applications]: Social and Behavioral Sciences that the content of the tweets are hidden and there is no
i textual feature among our defined features. Then, we pro-
X General Terms pose two different supervised approaches in order to rank
r the tweets. The first approach tires to predict the tweets
a Algorithm, Experimentation engagements globally. In other words, although our pur-
pose is to sort the tweets of each user, we consider tweets
Keywords
of all the users together and then try to predict the tweets
Twitter, User engagement, Rankingaggregation engagements. We can then extract the sorted list of each
user from the global ranked list. Therefore, we fit regres-
1. INTRODUCTION sionmodelstopredicttheengagementofeachtweet. Inthe
second approach, for each user, we rank thetweetsby their
Twitterisanonlinesocialinformationnetworkwhichhas
engagement without predicting the engagements. To this
become tremendously popular in the past few years [19].
aim, we use learning to rank approach which is extensively
exploitedininformationretrieval,naturallanguageprocess-
ing, and recommender systems. Learning to rank methods
Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalor
classroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributed rankthetweetsforeachuser. Incontrarytoregressionmod-
forprofitorcommercialadvantageandthatcopiesbearthisnoticeandthefullcita- els which try to predict the engagements by considering all
tiononthefirstpage. Copyrightsforcomponentsofthisworkownedbyothersthan thetweetstogether,learningtorankmethodsemphasizeon
ACMmustbehonored.Abstractingwithcreditispermitted.Tocopyotherwise,orre- maximizing an objective function for each user. According
publish,topostonserversortoredistributetolists,requirespriorspecificpermission
to the different points of view of regression and learning to
and/[e-mail].
RecSysChallenge’14,October10,2014,FosterCity,CA,USA. 1http://imdb.com
Copyright20XXACMX-XXXXX-XX-X/XX/XX...$15.00.
rank methods, we further propose to aggregate the results 3. METHODOLOGY
obtained by different regression and learning to rank meth- In general, our idea is to extract a number of features
ods to improvetheperformance. foreachtweetandthentrytolearnmachinelearningbased
In the experiments, we use an extended version of Movi- modelsonthetrainingdata. Then,foreachuserintestdata,
eTweetings dataset [9] providedbyACMRecSysChallenge weapplythelearnedmodeltorankhis/hertweetsbasedon
2014 and report the results of a number of state-of-the-art their engagements. In this section, we first introduce the
regressionandlearningtorankmethods,separately. Wefur- features, and then we propose some machine learning ap-
ther discuss the aggregation of the results of these two ap- proachestorankthetweetsbasedontheirengagements. We
proaches. The experimental results show that although the alsotrytoaggregatetheresultsofthesedifferenttechniques
results of regression methods are not so impressive, aggre- to improve the performance. In the following subsections,
gation of regression and learningto rankmethodsimproves we explain our methodology in details.
theresults significantly.
3.1 Features
2. RELATED WORK Eachtweetcontainstheopinionofauseraboutaspecific
Theproblemofengagementpredictionoronlineparticipa- movie. Wepartition thefeatures extractedfrom each tweet
intothreedifferentcategories: user-based,movie-based,and
tion has been studied from different points of view in news
tweet-based features. Overall, we extract several features
websites, social networks, and discussion forums. Several
from eachtweetTtweetedbyuserUaboutmovieM.User-
machine learning algorithms have been used in the litera-
basedfeaturesgiveussomeinformationabouttheuserwho
turefor thistask.
has tweeted his/her opinion about a specific movie. These
Toaddresstheproblemofengagementprediction,several
features have been proposed for training a model. Suh et features are not tweet-specific and they are equal for all
tweetsofeachuser. ThetotalnumberoffollowersofUisan
al. [28] have provided an analysis on the factors impacting
example of user-based features. Movie-based features only
thenumberofretweets. Theyhaveconcludedthathashtags,
includeinformationaboutmovieM,e.g.,thetotalnumberof
number of followers, number of followees, and the account
tweetsaboutmovieM.Tweet-basedfeaturescontainspecific
ageplayimportantrolesinincreasingtheprobabilityofthe
information of tweet T. This information may also contain
tweets to be retweeted. Zaman et al. [34] have trained
theopinionofuserUaboutmovieM.Thetimeandlanguage
a probabilistic collaborative filtering model to predict the
of a tweet are two examples of tweet-based features.
future retweets usingthehistory of theprevious ones.
The name and description of the extracted features are
Linear models have been used in some other studies to
shown in Table 1. These features are extracted for each
predict the popularity of videos on YouTube by observing
tweet T. We specify the category of the features and also
their popularity after regular periods [29]. Petrovic et al.
their type;“N”,“C”, and“B”are used for numerical, cate-
[26]haveproposedapassive-aggressivealgorithmtopredict
gorical, and boolean types, respectively. It should be noted
whether a tweet will be retweeted or not.
thatthefeaturevaluesarenormalizedusingz-scorenormal-
Recognizing popular messages is also one of the similar
ization method.
problems which is used for breaking news detection and
We also perform feature selection to improve the perfor-
personalized tweet/content recommendation. Hong et al.
mance and also toanalyse the effectivenessof theproposed
[13] have formulated this task as a classification problem
byexploitingcontent-basedfeatures, temporal information, features. We exploit backward elimination for feature se-
lection. The bolded features in Table 1 are those that are
meta-data of messages, and theusers social graph.
retained after performing feature selection. We discuss the
Predictingtheextenttowhichanewsisgoingtobebreak-
selected features in Subsection 4.1
ingorhowmanycommentsanewsisgoingtogainisoneof
the engagement prediction problems. Tatar et al. [30] have 3.2 MachineLearningTechniquesforUserEn-
analyzedanewsdatasettoaddressthisproblem. Theyhave
gagement Ranking
focused on sorting the articles based on their future popu-
larity and they have proposed to use linear regression for Inthissubsection,weproposetwodifferentlearningbased
this task. approaches to rank the tweets of each user based on their
It is worth noting that ranking instances is one of the engagements. The first approach is predicting the engage-
problemswhichhasbeenextensivelystudiedininformation ment of tweets, globally. In other words, for predicting the
retrieval,naturallanguageprocessing,andmachinelearning engagementoftweetsofauser,weconsiderthetweetsofall
fields [21]. To solve asimilar problem, Uysal and Croft [31] users for training the model and not only the tweets of the
have proposed“Coordinate Ascent learning to rank”algo- user. To this aim, we use regression models to predict the
rithm to rank tweets for a user in a way that tweets which engagementofeachtweet. Thenextapproachistorankthe
aremorelikelytoberetweetedcomeontop. Theyhavealso tweets for each user without predicting their engagements.
workedonrankingusersforatweetinawaythatthehigher We exploit learning to rank methods to rank the tweets of
the rank, the more likely the given tweet will be retweeted. each user, which focus on ranking the tweets of each user
Several learning to rank algorithms have been proposed in individuallyandtry tomaximize agiven objectivefunction
theliterature. Moreover,therearesomesupervisedandun- for each user. Finally, we propose a supervised method to
supervised ensemble methods to aggregate different rank- aggregate the regression and learning to rank results using
ings, such as Borda Count [2] and Cranking [20]. Previous supervised Kemeny approach [1]. In the following, we ex-
studiesshowthatinmanycases,rankingaggregation meth- plain ourproposed methodsin details.
ods outperform single rankingmethods[8, 21].
Table 1: Extracted features from each tweet T tweeted by user U about movie M
Cat. Feature Name Type Description
Number of follow- N The total numberof users who are following userU in Twitter.
ers
Numberof followees N The total numberof users who are followed by userU in Twitter.
Number of tweets N The total numberof tweets written byuser U.
Number of IMDb N The total number of tweets tweeted by user U using IMBD about different
tweets movies.
Average of ratings N The average of ratings providedby userU about different movies in IMDb.
Number of liked N The total numberof tweets which are liked byuser U.
d tweets
e
s
ba Numberof lists N The total numberof Twitter lists which user U is involved in.
-
r
se Tweeting frequency N The frequency of tweets written byuser U in each day.
U
Attracting followers N The frequency of attracting followers per day. This feature is calculated by
frequency dividing the total number of followers by the membership age of user U in
Twitter in terms of numberof days.
Following frequency N The frequency of following different usersby user U perday.
Like frequency N The frequency of liking tweets byuser U perday.
Followers/Followees N ThetotalnumberoffollowersofuserUdividedbythetotalnumberofhis/her
followees.
Followers- N The differencebetween thetotal numberof followers and followees of user U.
Followees
d Number of tweets N ThetotalnumberoftweetstweetedusingIMDbaboutmovieM.Thisfeature
e
as about M showshowmuchmovieMisratedbydifferentusersaroundtheworldinIMDb.
b
e- Average rating of N The average of ratings reported by differentusers for movieM.
vi
o M
M
Rate N The rating provided by user U for movie M. This rating is a positive integer
up to 10.
Mention count N The total numberof people who are mentioned in tweet T.
Numberof hash-tags N The total numberof hash-tagsused in tweet T.
Tweet age N The age of tweet T in terms of numberof days.
Membership age un- N Thenumberofdaysfrom when userUregistered in Twitter untilwhen tweet
til now T is tweeted.
d
se opinion difference N ThedifferencebetweentheratetweetedbyuserUformovieMandtheaverage
a
b of rates given by differentusers about movieM.
-
t
e
we Hourof tweet C The hour when tweet T is tweeted. This feature is an integer between 0 and
T 23.
Day of tweet C The day of week which tweet T is tweeted.
Time of tweet C The part of the day that tweet T is tweeted. We have partitioned each day
into four parts.
Holidays or not B This feature giveus whethertweet T is tweeted on holidaysor not.
Same language or B This feature illustrates whether tweet T is tweeted in the same language as
not thedefault language of user U or not.
English or not B This feature tells uswhether tweet T is tweeted in English or not.
3.2.1 Regression To aggregate all the mentioned regression and learning
To rank the tweets of each user based on their possible to rank results, we use supervised Kemeny approach [1].
engagements, we can first predict the engagement of each Kemeny optimal aggregation [17] tries to minimize total
tweet and then sort the tweets by their predicted values. number of pairwise disagreements between the final rank-
To predict the engagements, we propose to train regression ing and the outputs of all base rankers. In other words, if
modelsbyusingthefeaturesdefinedinSubsection3.1asthe r1, r2, ..., rn represent the outputs of n different rankers,
featuresandtheengagementsasthelabels. Then,weapply thefinal rankingr∗ is computed as:
the learned model on the same extracted features from the n
∗
test set. r =argmax {Xk(r, ri)}
r
Tocreatetheregressionmodel,weexploitExtremelyRan- i=1
domized Trees (also known as Extra-Trees) [11], Bayesian
wherek(α, β)istheKendalltaudistance[18]measuredas:
RidgeRegression[22],andStochasticGradientDescentRe-
gression (SGDR) [4]. Extra-Trees are tree-based ensemble |(i, j): i<j, αi >αj ∧ βi <βj|
regression methods which are successfully used in several
tasks. In Extra-Trees, when a tree is built, the node split- where αi denotes theith position of rankingα.
WhileinKemenyoptimalaggregationalltherankershave
tingstepisdonerandomlybychoosingthebestsplitamong
thesameimportance,supervisedKemenyapproachassumes
arandomsubsetoffeatures. Theresultsofalltreesarecom-
that there is a weight for each ranker. In more details, in
bined by averaging the individual predictions. SGDR is a
supervised Kemeny instead of counting the number of dis-
generalized linear regression model that tries to fit a linear
agreements, we use the following equation to compute the
model by minimizing a regularized empirical loss function
finalranking:
using gradient descent technique.
n
∗
3.2.2 LearningtoRank r =argmax {Xk(r, ri)∗wi}
r
Insteadofpredictingtheexactengagements,wecanrank i=1
the tweets directly, without predicting the engagements of wherewidenotestheweightofithranker. Tofindtheweight
each tweet. Learning to Rank (LTR) methods are machine ofeachranker,weproposetoperformaRandomizedSearch
learningtechniqueswhichtrytosolverankingproblems[21]. [3]. To this aim, we perform cross validation over training
LTRmethodshavebeenwidelyusedinmanydifferentareas data and findtheoptimal weight for each ranker.
such as information retrieval, natural language processing,
and recommender systems [16, 21]. LTR methods train a 4. EXPERIMENTS
ranking model and use the learned model to rank the in-
In the experiments, we consider an extended version of
stancesusingseveralfeatureswhichareextractedfromeach
MovieTweetingsdataset[9]whichisprovidedbyACMRec-
instance.
Sys Challenge 2014 [27].2 The dataset contains movie rat-
TobuildourLTRmodel,weconsideranumberofranking
ings which are automatically tweeted by theusers of IMDb
algorithms which are among state-of-the-art in many test
iOSapplication. Thereportedresultsthroughoutthiswork
collections: ListNet [7], RankingSVM [15], AdaRank [33],
are those obtained on the test set. The evaluation mea-
RankNet [6], LambdaRank [5], and ListMLE [32]. ListNet
sure is the mean of normalized discounted cumulative gain
is a probabilistic listwise approach to solve ranking prob-
[14] computed for top 10 tweets of each user. We call it
lems,whichexploitsaparameterizedPlackett-Lucemodelto
NDCG@10, hereafter.
computedifferentpermutations. RankingSVMisapairwise
In our experiments, we used Scikit-learn library [25] for
rankingapproachwhichusesSVMclassifierinitscorecom-
all the regression and feature selection algorithms. To se-
putations. The basic idea behind AdaRank is constructing
lect the parameters of the learning methods, we performed
some weak rankers and combining them linearly to achieve
hyper-parameteroptimization usingRandomizedSearch [3]
a better performance. Although, Ranking SVM creates a
with 5-fold cross validation. For the learning to rank algo-
ranking model by minimizing the classification error on in-
rithmsexceptAdaRank,we exploitedan open source pack-
stance pairs, AdaRank tries to minimize the loss function
age, named ToyBox-Ranking3. For AdaRank, we used the
which is directly defined as an evaluation measure (such as
software developed in Microsoft Research [33].4
NDCG@10). RankNet is one of the pairwise methods that
adoptscrossentropyasthelossfunction. RankNetemploys 4.1 Experimental Results and Discussion
a three layered neural network with a single output node
In this subsection, we report and discuss the results of
to compare each pairs. LambdaRank is one of the ranking
different regression and learning to rank methods. We also
algorithms inspired by RankNet which uses Gradient De-
provide the results obtained by aggregating the regression
scent approachtooptimizetheevaluationmeasure. Similar
and learning to rank results using the supervised Kemeny
to ListNet, ListMLE is a probabilistic listwise approach to
approach.
rank instances by maximizing a logarithmic loss function.
To show the impact of feature selection, we report the
3.2.3 AggregatingRegressionandLearningtoRank results of regression andlearning to rankmethods bothbe-
Outputs fore and after feature selection. As mentioned before, the
boldedfeatures inTable1are thoseretained afterperform-
Accordingtotheaforementionedfacts,regressionandlearn-
ingbackwardeliminationmethod. Theselectedfeaturesare
ingtoranktechniquestaketwodifferentpointsofviewinto
consideration and their results might be totally different. 2http://2014.recsyschallenge.com/
Therefore,byaggregatingtheirresults,theperformancecan 3https://github.com/y-tag/cpp-ToyBox-Ranking
potentially beincreased. 4http://goo.gl/xycK0h
Table2: Regressionresultswithandwithoutfeature Table 4: Ranking aggregation results
selection NDCG@10
NDCG@10
LTRs 0.8242044953
REGmethod REGw/FS REGw/oFS
REGs 0.8063031984
XT 0.7441384724 0.7863435909
LTRs+REGs 0.8261454943
BRR 0.7541443109 0.7759180414
SGDR 0.7507494314 0.8168741812
Table 4 represents the results obtained by aggregating
the mentioned regression and learning to rank results us-
ing supervised Kemeny approach. To show the importance
Table 3: Learning to rank results with and without
ofconsideringbothregressionandlearningtorankmethods
feature selection
together,wealsoreporttheresultsachievedbyaggregating
NDCG@10 all the LTR methods and all the regression methods, sepa-
LTRmethod LTRw/FS LTRw/oFS rately. Table4indicatesthatalthoughmostoftheresultsof
regressionmodelsarefarlowerthantheLTRmethods,their
ListNet 0.8243394623 0.8190048552
aggregation improvestheresults. Itshows thataggregating
RankingSVM 0.8225893034 0.8169257071 regression and learning to rank methodsachieves betterre-
sults in comparison with aggregating only LTR methods or
AdaRank 0.8182340058 0.8153622186
regression models. Toshowthatthisimprovementissignif-
RankNet 0.8223464432 0.8169752826 icant, we performed 10-fold cross validation over the train-
LambdaRank 0.8209622031 0.8126243442 ing data and conducted a statistical significant test (t-test)
on the improvements of LTRs+REGs over the other meth-
ListMLE 0.8217342257 0.8174866943
ods. The results show that the improvement achieved by
LTRs+REGsis statistically significant (p−value<0.01).
diffused among all the three feature categories. This shows 5. CONCLUSIONS
the importance of using a combination of different kinds of
In this paper, to rank the tweets of each user based on
features in this problem. The selected user-based features
their engagements, we first defined several features parti-
show how active and popular the user is in Twitter. Inter-
tioned into three different categories: user-based, movie-
estingly,allthebooleanfeaturesareselectedandnoneofthe
based, and tweet-based. We showed that after perform-
categorical features are retained. The reason may be that
ing feature selection, the features are selected from all of
the values of the boolean features are constant and the dif-
thesecategories. Then,weexploitedregressionandlearning
ferencebetweenthemarenotacontinuousvalue. Soitmay
to rank methods to rank the tweets of each user by their
beeasierandmoreefficienttousethesefeatures. Moreover,
engagements. Finally, we aggregated the results of all the
forthecategoricalfeatures,weassignanumbertoeachpos-
regression and learning to rank methods using supervised
sible category and the arithmetic difference between these
Kemeny approach.
numbersis not informative.
WeevaluatedourmethodsonanextendedversionofMovi-
Table 2 shows the results obtained by different regres-
eTweetingdatasetprovidedbyACMRecSysChallenge2014.
sion algorithms, in terms of NDCG@10. In Table 2,“XT”,
Theexperimentalresultsdemonstratethatfeatureselection
“BRR”, and “SGDR” respectively denote Extremely Ran-
significantly affects the performance. The results also show
domized Trees, Bayesian Ridge Regression, and Stochastic
that however the results of most regression models are far
Gradient Descent Regression.
lower than learning to rank methods, their aggregation im-
The results reported in Table 2 demonstrate that fea-
provestheperformance.
ture selection does not help with regression algorithms. In
other words, after performing the feature selection, the re-
sults of regression models are dropped dramatically. This 6. REFERENCES
showsthatbackwardelimination isnotsufficientforregres-
sion models. According to Table 2, there is a considerable [1] A.Agarwal, H. Raghavan,K.Subbian,P. Melville,
difference between the results achieved by different regres- R.D. Lawrence, D. C. Gondek,and J. Fan. Learning
sion models. torank for robust question answering. In Proceedings
Table3showstheresultsofusingseverallearningtorank of the 21st ACM International Conference on
methods. The results also include NDCG@10 before and Information and Knowledge Management, CIKM’12,
afterapplyingfeatureselection. TheresultsreportedinTa- pages 833–842, 2012.
ble3emphasizeontheimportanceofusingfeatureselection [2] J. A.Aslam and M. Montague. Models for
in learning to rank methods; since after performing feature metasearch. In Proceedings of the 24th Annual
selection, the results are improved. Therefore, backward International ACM SIGIR Conference on Research
elimination method works well for LTR methods. Table 3 and Development in Information Retrieval, SIGIR’01,
demonstrates that ListNet performs better than the other pages 276–284, 2001.
LTR methods. Comparing the results of Table 2 and Table [3] J. Bergstra and Y.Bengio. Random search for
3 shows that all the learning to rank methods outperform hyper-parameteroptimization. J. Mach. Learn. Res.,
all theregression models. 13(1):281–305, 2012.
[4] L. Bottou. Stochastic learning. In Advanced Lectures International Conference on Machine Learning, ICML
on Machine Learning, LectureNotes in Artificial ’02, pages 363–370, 2002.
Intelligence, LNAI3176, pages 146–168. 2004. [21] H.Li and G. Hirst. Learning to Rank for Information
[5] C. Burges, R.Ragno, and Q. Le. Learningto rank Retrieval and Natural Language Processing. Morgan &
with non-smooth cost functions.In Advances in Claypool, 2011.
Neural Information Processing Systems 19, 2007. [22] D.J. C. MacKay.Bayesian interpolation. Neural
[6] C. Burges, T. Shaked,E. Renshaw,A. Lazier, computation, 4:415–447, 1992.
M.Deeds,N.Hamilton,andG.Hullender.Learningto [23] T. Nguyen.Web based recommendersystems and
rank using gradient descent. InProceedings of the rating prediction.Master’s thesis, San Jose State
22nd International Conference on Machine Learning, University,2009.
ICML ’05, pages 89–96, 2005. [24] A.Oghina, M. Breuss, M. Tsagkias, and M. de Rijke.
[7] Z. Cao, T. Qin,T.-Y. Liu, M.-F. Tsai, and H.Li. Predicting imdb movieratings usingsocial media. In
Learning torank: From pairwise approach to listwise Proceedings of the 34th European Conference on
approach. InProceedings of the 24th International Advances in Information Retrieval, ECIR’ 12, pages
Conference on Machine Learning, ICML ’07, pages 503–507, 2012.
129–136, 2007. [25] F. Pedregosa, G. Varoquaux,A. Gramfort, V.Michel,
[8] O.Chapelle and Y.Chang. Yahoo! learning to rank B. Thirion, O. Grisel, M. Blondel, P.Prettenhofer,
challenge overview. Journal of Machine Learning R.Weiss, V. Dubourg,J. Vanderplas,A. Passos,
Research - Proceedings Track, 14:1–24, 2011. D.Cournapeau, M. Brucher, M. Perrot, and
[9] S.Dooms, T. DePessemier, and L. Martens. E. Duchesnay.Scikit-learn: Machinelearning in
Movietweetings: a movie rating dataset collected from python.J. Mach. Learn. Res., 12:2825–2830, 2011.
twitter. In Workshop on Crowdsourcing and Human [26] S.Petrovic, M. Osborne, and V.Lavrenko.Rt to win!
Computation for Recommender Systems, CrowdRec at predictingmessage propagation in twitter. In
RecSys 2013, 2013. International AAAI Conference on Weblogs and Social
[10] M. D. Ekstrand,J. T. Riedl, and J. A. Konstan. Media,pages 586–589, 2011.
Collaborative filtering recommendersystems. Found. [27] A.Said, S.Dooms, B. Loni, and D.Tikk.
Trends Hum.-Comput. Interact., 4(2):81–173, 2011. Recommendersystemschallenge 2014. In Proceedings
[11] P. Geurts, D.Ernst, and L. Wehenkel.Extremely of the eighth ACM conference on Recommender
randomized trees. Mach. Learn., 63(1):3–42, 2006. systems, RecSys’14, 2014.
[12] I.Guy and D. Carmel. Social recommender systems. [28] B. Suh,L. Hong,P. Pirolli, andE. H. Chi. Wantto be
In Proceedings of the 20th International Conference retweeted? large scale analytics on factors impacting
Companion on World Wide Web, WWW ’11, pages retweet in twitter network.In Proceedings of the 2010
283–284, 2011. IEEE Second International Conference on Social
[13] L. Hong,O. Dan,and B. D.Davison. Predicting Computing, SOCIALCOM’10, pages 177–184, 2010.
popularmessages in twitter. InProceedings of the 20th [29] G. Szaboand B. A.Huberman.Predicting the
International Conference Companion on World Wide popularity of online content.Commun. ACM,
Web, WWW ’11, pages 57–58, 2011. 53(8):80–88, 2010.
[14] K.J¨arvelin and J. Kek¨al¨ainen. Cumulated gain-based [30] A.Tatar, P.Antoniadis, M. D.de Amorim, and
evaluation of ir techniques.ACM Trans. Inf. Syst., S.Fdida. Rankingnews articles based on popularity
20(4):422–446, 2002. prediction.In Proceedings of the 2012 International
[15] T. Joachims. Optimizing search engines using Conference on Advances in Social Networks Analysis
clickthrough data. In Proceedings of the Eighth ACM and Mining,ASONAM’12, pages 106–110, 2012.
SIGKDD International Conference on Knowledge [31] I.Uysaland W.B. Croft. Useroriented tweet ranking:
Discovery and Data Mining,KDD’02, pages 133–142, A filteringapproach tomicroblogs. In Proceedings of
2002. the 20th ACM International Conference on
[16] A.Karatzoglou, L. Baltrunas, and Y.Shi.Learning to Information and Knowledge Management, CIKM’11,
rank for recommender systems. In Proceedings of the pages 2261–2264, 2011.
7th ACM Conference on Recommender Systems, [32] F. Xia, T.-Y.Liu, J. Wang, W. Zhang,and H. Li.
RecSys’13, pages 493–494, 2013. Listwise approach to learning to rank: Theory and
[17] J. G. Kemeny.Mathematics without numbers. algorithm. InProceedings of the 25th International
Daedalus, 88(4), 1959. Conference on Machine Learning, ICML ’08, pages
[18] M. G. Kendall. A new measure of rank correlation. 1192–1199, 2008.
Biometrika, 30(1/2):81–93, 1938. [33] J. Xu and H.Li. Adarank: A boosting algorithm for
[19] S.M. Kywe,E.-P. Lim, and F. Zhu.A surveyof information retrieval. In Proceedings of the 30th
recommendersystems in twitter. In Proceedings of the Annual International ACM SIGIR Conference on
4th International Conference on Social Informatics, Research and Development in Information Retrieval,
SocInfo ’12, pages 420–433, 2012. SIGIR’07, pages 391–398, 2007.
[20] G. Lebanon and J. D. Lafferty.Cranking: Combining [34] T. R.Zaman, R. Herbrich,J. Van Gael, and D.Stern.
rankingsusing conditional probability models on Predicting information spreading in twitter. In
permutations. InProceedings of the Nineteenth Workshop on Computational Social Science and the
Wisdom of Crowds, NIPS, 2010.