Group-based ranking method for online rating systems with spamming attacks

1.5 MB·English

by  Jian Gao

epl draft
Group-based ranking method for online rating systems with spam-
ming attacks
Jian Gao1, Yu-Wei Dong1, Ming-Sheng Shang1 (a), Shi-Min Cai1 and Tao Zhou1,2 (b)
1 Web Sciences Center, University of Electronic Science and Technology of China - 611731 Chengdu, PRC
2 Big Data Research Center, University of Electronic Science and Technology of China - 611731 Chengdu, PRC
5
1
0
2 PACS 89.65.-s–Social and economic systems
n PACS 89.20.Hh–World WideWeb, Internet
a PACS 89.20.Ff–Computer science and technology
J
Abstract–Rankingproblemhasattractedmuchattentioninrealsystems. Howtodesignarobust
4
ranking method is especially significant for online rating systems under the threat of spamming

] attacks. By building reputation systems for users, many well-performed ranking methods have
R been applied to address this issue. In this Letter, we propose a group-based ranking method
I that evaluates users’ reputations based on their grouping behaviors. More specifically, users are
.
s assigned with high reputation scores ifthey always fall intolarge ratinggroups. Resultson three
c realdatasetsindicatethatthepresentmethodismoreaccurateandrobustthancorrelation-based
[
method in thepresence of spamming attacks.

1
v
7
7
6
0 Introduction. – With the rapid development of the ing objects’ quality (i.e., weighted average rating). The
0 Internet,billionsofservicesandobjectsareonlineforusto reputationandtheestimatedqualityareiterativelycalcu-
.
1 choose [1]. At the same time, the problem of information lated until they become stable. An improved IR method
0 overloadtroublesuseveryday[2–4]. Therefore,manyweb is proposed in [26], by assigning trust to each individual
5 sites (Amazon, Ebay, MovieLens, Netlfix, etc.) introduce rating. Later, Zhou et al. [27] proposed the correlation-
1
online rating systems, where users can give discrete rat- based ranking (CR) method that is robust to spamming
:
v ingstoobjects. Inturn,theratingsofanobjectserveasa attacks,whereauser’sreputationisiterativelydetermined
Xi referenceandlatteraffectotherusers’decisions[5,6]. Ba- by the correlation between his ratings and objects’ esti-
sically, high ratings can promote the consumption, while mated quality. Very recently, by introducing a reputa-
r
a low ratings play the opposite role [7]. In real cases, some tion redistribution process and two penalty factors, Liao
users may give unreasonable ratings since they are sim- et al. [28] further improved the CR method.
ply unfamiliar with the related field [8], and some others Inthe majorityofpreviousworks[29,30],asinglestan-
deliberately give biased ratings for various psychosocial dardobjects’qualityisrequiredindeterminingusers’rep-
reasons [9–13]. These widely existed distort ratings can utations,withanunderlyingassumptionthateveryobject
harm or boost objects’ reputation, mislead others’ judg- isassociatedwithamostobjectiveratingthatbestreflect
ments, and affect the evolution of rating systems [14–16]. its quality. However, in real cases, one object may have
Due to the negative effects of spamming attacks, how to multiplevalidratings,sincetheratingsaresubjectiveand
design a robust method for online rating systems is be- canbe affected by users’background[31–34]. In the pres-
coming an urgent task [17–19]. ence of more than one reasonable answer to a single task,
To solve this problem, normally, building a reputation Tian et al. [29] analyzed the group structure of schools
system for users is a good way [20–28]. Laureti et al. [25] of thought in solving the problem of identifying reliable
proposed an iterative refinement (IR) method, where a workers as well as unambiguous tasks in data collection.
user’sreputationisinverselyproportionaltothedifference Specifically, a worker who is consistent with many other
betweenhis ratings andthe estimation ofthe correspond- workers in most of the tasks is reliable, and a task whose
answers form a few tight clusters is easy and clear. Anal-
(a)E-mail: [email protected] ogously,in the online rating systems, one object’s quality
(b)E-mail: [email protected] is clearif its ratingsare centralized,while it is notclear if
p-1
J. Gao et al.
(a) (b) (c)
O O O O O O O O W
1 2 3 4 1 2 3 4
User U1 U2 U3 U4 U5 é5 5 2 5ù U ì U U - U,U ü 1
1 5 5 3 5
ê- 4 2 5ú U (cid:311) ï U - U,U - ï 2
Rating A=êê2 5 3 1úú U23 G=ïïí -3 - U1 32 - ïïý 3
ê5 4 - 5ú U ï - U ,U - - ï 4
ê ú 4 ï 2 4 ï
Object O1 O2 O3 O4 êë1 1 5 1úû U5 ïîU1,U4 U1,U3 U5 U1,U2,U4ïþ 5
(cid:314) (cid:312)
(g) (f) (e) (d)
R O O O O O O O O W O O O O W
1 2 3 4 1 2 3 4 1 2 3 4
U 3.18 é0.50 0.40 0.50 0.60ù U é0.25 0.20 - 0.40ù 1 é1 1 - 2ù 1
Spammer U5 3.75 (cid:315) ê - 0.40 0.50 0.60ú U1 (cid:314) ê0.25 - 0.50 - ú 2 (cid:313) ê1 - 2 -ú 2
3 ê ú 2 ê ú ê ú
Ri=Lsm=((2AAii¢¢)) UUU124 556...001002 A¢=êêêêë000...522055 000...244000 00..-2255 000...464000úúúúû UUU453 L*=êêêêë0.--50 00..-4400 00..-2255 0.--60úúúúû 453 L=êêêêë--2 -22 -11 --3úúúúû 453
Fig.1: (Coloronline)Illustratingthegroup-basedmethod. Thenumberbesidesthegrayarrowmarksthestepoftheprocedure.
(a) The original weighed bipartite network, G. (b) The corresponding rating matrix, A. The row and column correspond to
usersandobjects,respectively. Thesymbol“-”standsforanon-rating,whichcanbeignoredinthecalculation. (c)Thegroups
of users, Γ, after being grouped according to their ratings. Take O as an example (green vertical box). As U and U rated
2 2 4
4 to O2, they are put into group Γ4,2. (d) The sizes of groups, Λ, e.g. Λ4,2 =2 as Γ4,2 ={U2,U4}. (e) The rating-rewarding
∗ ∗ ′
matrix, Λ , constructed b(cid:6658)yn(cid:1941)orm(cid:2119)aliz(cid:7735)ing Λ(cid:1566)by(cid:7017)colu(cid:17934)mn(cid:17897), e.g, Λ4,2 =2/(1+2+2)=0.40. (f) Therewarding matrix, A,obtained
∗ ′
by mapping matrix A referring to Λ , e.g. A = 0.40. (g) The ranking list of users based on reputation R. Take U as an
4,2 3
example (blue horizontal box in (f)), R = µ(A′)/σ(A′) = 3.75. Setting spam list’s length as L = 2, then U and U (red
3 3 3 5 3
dashed box) are detected as spammers.
the ratings are widely distributed. Under this framework, α are denoted as k and k , respectively. Here, we use
i α
a single estimation of an object’s quality is no longer ap- Greek and Latin letters, respectively, for object-related
plicable [30]. Practically,a randomrating to objects with and user-related indices to distinguish them. Consider-
confusingqualityshouldbeacceptablesincenosinglerat- ing a discrete rating system, the bipartite network can be
ing candominantits true quality,while a biasedrating to represented by a rating matrix A [37], where the element
objects with clear quality is unreasonable. Users who are a ∈ Ω = {ω ,ω ,...,ω } is the weight of the link con-
iα 1 2 z
consistentwith the majorityin mostratings will form big necting node U and node O , i.e., the rating given by
i α
groups and be trusted since herding is a well-documented useritoobjectα. Inareputationsystem,eachuseriwill
featureofhumanbehaviour[6,35]. Userswhoalwaysgive be assigned a reputation, denoted as R . The users with
i
distort ratings will form relatively small groups and be very low reputations are detected as spammers.
highly suspected since unreasonable or biased ratings are The GR method works as follows. Firstly, we group
discordant [29]. These ideas bring us a promising way usersaccordingtotheir ratings. Specifically,foranobject
to build reputation systems based on users’ grouping be- α, we put users who gave the rating ω into group Γ :
s sα
haviour instead of solving the crucial problem of estimat-
Γ ={U |a =ω ,i=1,2,...,m}. (1)
ing objects’ true qualities as before. sα i iα s
In this Letter, we propose a group-based ranking (GR) Obviously, user i belongs to k different groups. Sec-
i
method for online rating systems with spamming attacks. ondly,wecalculatethesizesofallgroupsΛ =|Γ |, i.e.
sα sα
By grouping users according to their ratings, users’ rep- the number of users who gave the rating ω to object α.
s
utations are determined according to the corresponding Thirdly,weestablishamatrixΛ∗,namedrating-rewarding
group sizes. If they always fall into large groups, their matrix, by normalizing Λ per column:
reputations are high, on the contrary, their reputations
Λ
∗ sα
are low. Extensive experiments on three real data sets Λ = . (2)
sα k
(MovieLens, Netflix and Amazon) suggest that the pro- α
posed method outperforms the CR method. Fourthly, referring to Λ∗, we map the original rating ma-
trix A to a matrix A′, named rewarding matrix. More
Method. – The online rating system is naturally de- specifically, the rewarding that a user i obtain from his
scribed by a weighed bipartite network G = {U,O,E}, rating a is defined as A′ = Λ∗ , where a = ω . A′
iα iα sα iα s iα
where U = {U ,U ,...,U }, O = {O ,O ,...,O }, E = is null if user i has not yet rated object α.
1 2 m 1 2 n
{E ,E ,...,E } are sets of users, objects and ratings, re- Then, we assign reputations to users according to their
1 2 l
spectively [36]. The degree of a user i and an object rewarding. On the one hand, if the average of a user’s
p-2
A group-basedranking method for online rating systems

1.0
Table 1: Basic statistics of the three real data sets. m is the
numberofusers,nisthenumberofobjects,hkUiistheaverage 0.8 (a)
degreeofusers,hkOiistheaveragedegreeof objects, andS = R0.6
l/mn is thesparsity of thebipartite network. 0.4
Data set m n hk i hk i S 0.2
U O
0.0
MovieLens 943 1682 106 60 0.063 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Netflix 1038 1215 47 40 0.039 1.0
Amazon 662 1500 36 15 0.023 0.8 (b)
R0.6
0.4
rewarding is small, most of his ratings must be deviated 0.2
from the majority,suggesting that he is highly suspected.
0.0
On the other hand, if the rewarding varies largely, he is 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
alsountrustworthyforhisunstableratingbehavior. Based
MovieLens Netflix Amazon
on these considerations,we defined user i’s reputation as
Fig. 2: (Color online) The relation between reputation R and
R = µ(A′i), (3) rating error δ (bins) in different methods. (a) and (b) are for
i σ(A′) theCRandGRmethod,respectively. δandRarerespectively
i
normalized for comparison underdifferent datasets.
where µ and σ are functions of mean value and standard
deviation, respectively. Specifically, the mean value of A′
i
is defined as objects [9,10]. The random ratings mainly come from
A′
µ(A′)= iα, (4) some naughty users or test engineers who randomly give
i ki meaningless ratings [14,15]. As spammers are unknown
α
X
and the standard deviation of A′ is defined as in real data, to test the method, we manipulate the three
i realdatasets by adding either type ofartificialspammers
(A′ −µ(A′))2 (i.e. malicious or random) at one time.
σ(A′)= α iα i . (5) In the implementation, we randomly select d users and
i sP ki assign them distorted ratings: (i) integer 1 or 5 with the
Infact,R isthesamewiththeinverseofthecoefficientof same probability (i.e., 0.5) for malicious spammers, and
i
variation [38] of vector A′, which shows the dispersion of (ii)randomintegersin{1,2,3,4,5}forrandomspammers.
i
the frequency distribution of user i’s rewardings. Finally, Thus, the ratio of spammers is q =d/m. To study the ef-
we sort users in ascending order by reputation, and deem fectsofspammers’activity,wedefinep=k/ntheactivity
the top-L ones as detected spammers. A visual represen- ofspammers,wherekisthedegreeofeachspammer. Here
tation of GR method is given in fig. 1. k is a tunable parameter, that is, if k is no more than a
spammer’s original degree, we randomly select his/her k
Data and Metrics. – We consider three commonly
ratings and replace them with distorted ratings and the
studiedrealdatasets,MovieLens,NetflixandAmazon,to
un-selected ratings are ignored. Otherwise, after replac-
test the accuracy of GR method. MovieLens and Netflix
ing allthe spammer’s originalratings,we randomly select
containratingsonmovies,providedbyGroupLensproject
remaining number of non-rated objects and assign them
at University of Minnesota (www.grouplens.org) and re-
distorted ratings.
leased by the DVD rental company Netflix for a contest
Metrics for evaluation. To evaluate the performance
on recommender systems (www.netflixprize.com), respec-
ofrankingmethods,weadopttwocommonlyusedmetrics:
tively. Amazon contains ratings on products (e.g. books,
recall[40] andAUC (the area under the ROC curve)[41].
music, etc) crawled from amazon.com [39]. All the three
The recall value measures to what extent the spammers
datasetsuse a5-pointratingscalewith1 being the worst
can be detected in the top-L ranking list,
and 5 being the best. Herein, we sampled and extracted
threesmallerdatasetsfromtheoriginaldatasets,respec- d′(L)
tively, by choosing users who have at least 20 ratings and R (L)= , (6)
c
d
objects having been rated by these users since it’s hard
totellwhethersmall-degreeusersarespammers[28]. The where d′(L) ≤ d is the number of detected spammers in
basic statistics of data sets are summarized in table 1. the top-L list. A higher R indicates a higher accuracy.
c
Generatingartificialspammer. Twotypesofdistorted Notethat, R onlyfocusesonthetop-Lranks,andthus
c
ratings are widely found in real rating systems, namely, we also consider an L-independent metric called AUC.
malicious ratings and random ratings. The malicious rat- Provided the rank of all users, AUC value can be inter-
ingsarefromspammerswhoalwaysgivesminimum(max- pretedastheprobabilitythatarandomlychosenspammer
imum) allowableratingsto push down(up) certaintarget is ranked higher than a randomly chosen non-spammer.
p-3
J. Gao et al.
MovieLens Net flix Ama zon Table2: AUCofdifferentalgorithmsfortherealdatasetswith
d=50 beingfixed. Theparameter p is set as about 0.05, 0.03
and0.02forMovieLens,NetflixandAmazon,respectively. The
results are averaged over 100 independent realizations.
Malicious spamers Random spamers
Data set
(a) (b) (c) CR GR CR GR
MovieLens 0.876 0.994 0.914 0.959
Netflix 0.543 0.977 0.668 0.930
L) Amazon 0.824 0.941 0.877 0.949
R(c
(d) (e) (f)
correlation coefficient ρ [38] between R and δ . Specifi-
i i
L L L cally, ρ = −0.956 (−0.949), −0.906 (−0.872) and −0.966
GR method (−0.816) after applying GR (CR) method to MovieLens,
Fig. 3: (Color online) The recall Rc of different algorithms as NetflixandAmazon,respectively. Thelargernegativecor-
a function of the length of the list L. (a), (b) and (c) are for relation suggest that GR method is better on evaluating
maliciousspammers,(d),(e)and(f)areforrandomspammers users’ reputations.
withd=50beingfixed. Theparameterpissetasabout0.05,
Effectiveness and efficiency. To test the effectiveness
0.03and0.02forMovieLens,NetflixandAmazon,respectively.
of ranking algorithms, based on the three real data sets,
The results are averaged over 100 independent realizations.
wefirstgenerateartificialdatasetswith50spammers(i.e.
d=50). Eachdatasetisonlywithonetypeofspammers:
TocalculateAUC,ateachtimewerandomlypickaspam- malicious or random. On the generateddata sets, we cal-
mer and a non-spammer to compare their reputations, if culate recall of different algorithms as a function of the
among N independent comparisons, there are N′ times spammer list’s length L. As shown in fig. 3, GR method
the spammer has a lower reputation and N′′ times they hasremarkableadvantageoverthe CR methodondetect-
have the same reputation, the AUC value is ing both types of spammers, especially when L is larger
N′+0.5N′′ than d. We also note that Rc of ranking malicious spam-
AUC = . (7) mersisalittlehigherthanthatofrandomspammerswhen
N
L is smaller than d, which implies that to detect random
If all users are ranked randomly, the AUC value should
spammers is relatively harder.
be about 0.5. Therefore, the degree to which the value
Results of AUC values are shown in table 2, where one
exceeds0.5indicateshowbetterthemethodperformsthan
canseethatAUCvaluesofGRmethodishigherthanthat
pure chance [42].
of the CR method for every data set, suggesting that GR
Results. – According to the single standardassump- methodhassignificantadvantagetowardstheCRmethod.
tion, each object α has a true quality, denoted by Q . As It also shows that the CR method is better at detecting
α
the true quality is unknown in reality, taking the average random spammers than malicious spammers, while GR
rating as an estimation of Q is the most straightforward method is inverse. In addition, it is worthy to be noticed
α
way. Then, the rating error of user i is defined as that the AUC is generally lower in Netflix, especially for
theCRmethod. Onepossibleexplanationisthatthereare
|a −Qˆ |
δ = α iα α , (8) more harmful spammers in Netflix and the CR method is
i
P ki verysensitiveto“real”spammers[27,28]. Additionally,in
where Qˆ = 1/ a is the average rating of object Netflix,therearemoresmalldegreeobjectswhosequality
α j∈Γα jα will be considered higher by the CR method [28], which
α and α runs over all objects being rated by user i. A
P may also lead to the biased ranking.
reasonable ranking method should give high reputations
to the users with low rating errors, i.e. R should be Robustness against spammers. We then study the ro-
i
negatively correlated with δ . bustness of different methods by varying p (the ratio of
i
Figure2showsthe relationbetweenusers’ratingerrors objectsratedbyspammers)andq(theratioofspammers).
and reputations evaluated by CR and GR, respectively. In the following, we set the length of detected spam list
As users’ rating errors are continuous and with different being equalto the number ofartificialspammers,namely,
scales, we normalize and divide them into bins with the L=d. Figure4showsthe recallobtainedbyGRmethod.
length 0.05. As one can see, the two methods both as- The ranges of p and q are personalized set for different
sign high reputations for users with small δ , while GR datasetsreferringtotheirsparsity. Onecanobservethat,
i
method outperforms CR method by stably assigning low overall, GR method has better performance on detecting
reputations(see fig.2(b))foruserswithlargeδ . Further- maliciousspammers,especiallywhenpissmall(i.e. spam-
i
more,toquantifythecorrelation,wecalculatethePearson mers are of small degree). Moreover,when q is small, the
p-4
A group-basedranking method for online rating systems
Movie Lens Ne tflix Ama zon MovieLens Netflix Amazon
0.060 0.150 0.150 1.00 0.060 0.150 0.150 0.50
0.90 0.45
q0.035 0.0 85 0.0 80 0.80 q0.035 0.0 85 0.080 0.40
0.70 0.35
0.60 0.30
(a) (b) (c) (a) (b) (c)
00..006100 00..105200 00..011500 0.50 00..001600 00..012500 00..011500 0.25
0.40 0.20
0.30 0.15
q0.035 0.0 85 0.080 0.20 q0.035 0.0 85 0.0 80 0.10
0.10 0.05
0.010 (d) 0.020 (e) 0.010 (f) 0.00 0.010 (d) 0.020 (e) 0.010 (f) 0.00
0.010 0.130 0.2500.010 0.030 0.0500.005 0.018 0.030 0.010 0.130 0.2500.010 0.030 0.0500.005 0.018 0.030
p p p p p p
Fig. 4: (Color online) The effectiveness of GR method. The Fig.5: (Coloronline)ThecomparisonofGRandCRmethods.
colormarksrecallRc. q andpareratioofspammersandratio Thecolormarks∆Rc if∆Rc >0,otherwisethecolorisgreen,
of objects rated by spammers, respectively. (a), (b) and (c) meaningthattheCRmethodisbetter. (a),(b)and(c)arefor
are for malicious spammers. (d), (e) and (f) are for random maliciousspammers. (d),(e)and(f)areforrandomspammers.
spammers. The parameter is set as L = d. The results are The parameter is set as L=d. The results are averaged over
averaged over100 independentrealizations. 100 independent realizations.
cially on resisting small-degree spammers. Interestingly,
recalls of ranking random spammers are low for Movie-
the accuracy of spam detection on Netflix data set is low
Lens and Netflix data sets (see figs. 4(d) and 4(e)), while
for both the two methods, indicating that there are more
for Amazon data set, the recallofranking both two types
original distort ratings in Netflix data sets, which is in
of spammers is low (see figs. 4(c) and 4(f)). In addition,
accordance with some previous studies [27,28].
the recall positively increases with q. These observations
Theproposedmethodhasseveraldistinguishingcharac-
suggest that (i) detecting malicious spammers is easier
teristics, differentiating it from current users’ reputation
than random spammers, (ii) MovieLens and Netflix may
allocation procedures: (i) The method assigns users’ rep-
contain more “real” spammers, and (iii) GR method is
utation by considering the grouping behavior of users in-
powerful to detect spammers who only rate a small num-
steadofbasedonthe estimationofobjects’true qualities.
ber of objects.
(ii)Themethodiswithhighperformanceinbothaccuracy
To comprehensively compare the performance of GR
androbustness,especiallywhendealingwithsmall-degree
and CR methods, we calculate the difference of recall be-
spammers’attacks. (iii)Themethodisveryefficient,asits
tween the two, formulateed as ∆R = RGR −RCR. As
c c c timecomplexity[43]isO(m2),whichissignificantlylower
shown in fig. 5, ∆R is above 0 in most area, suggesting
c
than most of previously proposed iterative methods. As
that the overall performance of GR method is better. In
further improvement, we could consider introducing this
detail,GRmethodhasremarkableadvantageindetecting
methodtoaniterativeprocess[25–28],applyingittocon-
malicious spammers, while the advantage is not obvious
tinuous rating systems [44,45], and considering the effect
for random spammers. In addition, ∆R is big when p
c
of long-term evolution of online rating systems [16].
andq aresmall,implying that GRmethodis morerobust
against a small number of small-degree spammers, which
∗∗∗
are usually difficult to be detected out.
Conclusions and Discussion. – In summary, we This work was partially supported by the National
Natural Science Foundation of China under Grant Nos.
have proposed a group-based (GR) method to solve the
61370150, 91324002, 61433014 and 11222543. J.G. ac-
ranking problem in online rating systems with spamming
knowledges support from Tang Lixin Education Devel-
attacks. By grouping users according to their ratings, we
opment Foundation by UESTC. T.Z. acknowledges the
construct a rating-rewardingmatrix according to the cor-
Program for New Century Excellent Talents in Univer-
responding group sizes, base on which we map a user’s
sityunderGrantNo. NCET-11-0070,andSpecialProject
rating vector to rewarding vector. Then, this user’s repu-
of Sichuan Youth Science and Technology Innovation Re-
tationis assignedaccordingto the inverseof dispersionof
search Team under Grant No. 2013TD0006.
frequency distribution of his rewarding vector. Extensive
experimentsshowedthatthe proposedmethodis effective
on evaluating users’ reputations, especially for those with
REFERENCES
highratingerrors. Intestingwiththegenerateddatawith
two types of artificialspammers,GR method gives higher [1] Lu¨ L., Medo M., Yeung C. H., Zhang Y.-C., Zhang
performance in both accuracy and robustness compared Z.-K. and Zhou T., Phys. Rep., 519 (2012) 1.
with the correlation-based ranking (CR) method, espe- [2] Zhang F., and Zeng A., EPL, 100 (2012) 58005.
p-5
J. Gao et al.
[3] ZengA.,VidmerA.,MedoM.andZhangY.-C.,EPL, ery and Data Mining(ACMPress) 2012 pp.226–234.
105 (2014) 58002. [30] Raykar V. C., Yu S., Zhao L.-H., Valadez G. H.,
[4] Guo Q., Song W.-J. and Liu J.-G., EPL, 107 (2014) Florin C., Bogoni L. and Moy L., J. Mach. Learn.
18001. Res., 11 (2010) 1297.
[5] Jindal N. and Liu B., in Proceedings of the 16th Inter- [31] Shi X., Zhu J., Cai R., and Zhang L., in Proceedings
national Conference on World Wide Web (ACM Press) of the 15th ACM SIGKDD International Conference on
2007 pp.1189–1190. KnowledgeDiscoveryandDataMining(ACMPress)2009
[6] Muchnik L., Aral S., and Taylor S.-J., Science, 341 pp.777–786.
(2013) 647. [32] Welinder P., Branson S., Belongie S. and Perona
[7] Yu X., Liu Y., Huang X. and An A., IEEE Trans. on P.,inAdvancesinNeuralInformationProcessingSystems
Knowl. and Data Eng., 24 (2012) 720. (NIPS) (MIT Press) 2010 pp.2424–2432.
[8] PanW.,XiangE.-W.,LiuN.-N.andYangQ.,inPro- [33] Reed M. S., Evely A. C., Cundill G., Fazey I.,
ceedings of the 24th AAAI Conference on Artificial Intel- Glass J., Laing A., Newig J., Parrish B., Prell C.,
ligence (AAAIPress) 2013 pp.39–55. Raymond C. and Stringer L. C., Ecology and Society,
[9] Wang G., Xie S., Liu B. and Yu P. S., in Proceedings 15 (2010) r1.
of IEEE 11th International Conference on Data Mining [34] Ross J., Irani L., Silberman M., Zaldivar A. and
(ICDM) (IEEE Press) 2011 pp. 1242–1247. Tomlinson B., in CHI ’10 Extended Abstracts on Hu-
[10] ChungC.-Y.,Hsu P.-Y.andHuangS.-H.,Decis.Sup- man Factors in Computing Systems (ACM Press) 2012
port Syst., 55 (2013) 314. pp.2863–2872.
[11] Yang Z., Zhang Z.-K. and Zhou T., EPL, 100 (2012) [35] RaafatR.M.,ChaterN.,andFrithC.,TrendsCogn.
68002. Sci., 13 (2009) 420.
[12] Huang J., Cheng X.-Q., Shen H.-W., Zhou T., and [36] Shang M.-S., Lu¨ L., Zhang Y.-C. and Zhou T., EPL,
JinX.,inProceedingsofthe5thACMInternationalCon- 90 (2010) 48006.
ference onWebSearch andDataMining(WSDM)(ACM [37] Zhou T., Ren J., Medo M. and Zhang Y.-C., Phys.
Press) 2012 pp.573–582. Rev. E, 76 (2007) 46115.
[13] Zhang Y.-L., NiJ., GuoQ.,andLiu J.-G.,Physica A, [38] Lin L. I., Biometrics, 45 (1989) 255.
417 (2015) 261. [39] Jindal N. and Liu B., in Proceedings of the 2008 In-
[14] ZengA.andCiminiG.,Phys.Rev.E,85(2012)036101. ternational Conference on Web Search and Data Mining
[15] Zhang Q.-M., Zeng A. and Shang M.-S., PLoS ONE, (WSDM) (ACM Press) 2008 pp. 219–230.
8 (2013) e62624. [40] Herlocker J. L., Konstan J. A., Terveen L. G.and
[16] Zhao D.-D., Zeng A., Shang M.-S,andGao J.,Chin. Riedl J. T.,ACM Trans. Inf. Syst., 22 (2004) 5.
Phys. Lett., 30 (2013) 8901. [41] HanleyJ.A.,andMcNeilB.J.,Radiology,143(1982)
[17] MukherjeeA.,LiuB.,WangJ., GlanceN.,andJin- 29.
dal N., in Proceedings of the 20th International Confer- [42] Lu¨, L. and Zhou T., Physica A, 390 (2011) 1150.
ence on World Wide Web (ACM Press) 2011 pp. 93–94. [43] Sipser M., Introduction to the Theory of Computation
[18] FeiG., MukherjeeA., LiuB., HsuM., Castellanos (Cengage Learning, Boston) 2012.
M., and Ghosh R., in Proceedings of the 7th Interna- [44] Stanley N. and Jenkins S., in Challanges of a chang-
tional AAAI Conference on Weblogs and Social Media ingworld.Proceedingsofthe5thInternationalConference
(ICWSM) (AAAIPress) 2013 pp. 175–184. of the Association for Survey Computing(Association for
[19] Lin Y., Zhu T., Wang X., Zhang J. and Zhou A., in SurveyComputing, Berkeley) 2007 pp. 81–92.
Proceedings of the Companion Publication of the 23rd In- [45] TreiblmaierH.andFilzmoserP.,inProceedingsofthe
ternational Conference on World Wide Web Companion International Conference on Information Systems (ICIS)
(WWW) 2014 pp.341–342. (Association for Information Systems, Utrecht) 2011 Pa-
[20] Masum H. and Zhang Y.-C., First Monday, 9 (2004) . per 1.
[21] Resnick P., Kuwabara K., Zeckhauser R. and
Friedman E., Commun. ACM, 43 (2000) 45.
[22] Jøsang A., Ismail R. and Boyd C., Decis. Support
Syst., 43 (2007) 618.
[23] Allahbakhsh M., Ignjatovic A., Motahari-Nezhad
H. R. andBenatallah B.,World Wide Web, (2013) 1.
[24] Ling G., King I. and Lyu M. R., in Proceedings of the
23rd International Joint Conference on Artificial Intelli-
gence (AAAIPress) 2013 pp.2670–2676.
[25] Laureti P., Moret L., Zhang Y.-C. and Yu Y.-K.,
EPL, 75 (2006) 1006.
[26] de Kerchove C.andVan Dooren P.,arXiv:0711.3964
(2007)
[27] ZhouY.-B.,LeiT.andZhouT.,EPL,94(2011)48002.
[28] Liao H., Zeng A., Xiao R., Ren Z.-M., Chen D.-B.
and Zhang Y.-C., PLoS ONE,9 (2014) e97146.
[29] Tian Y. and Zhu J., in Proceedings of the 18th ACM
SIGKDDInternationalConferenceonKnowledgeDiscov-
p-6