analyzing image tweets in microblogs

162 Pages·2016·8.29 MB·English

ANALYZING IMAGE TWEETS IN
MICROBLOGS
TAO CHEN
(B.Eng (Hons.), East China Normal University)
A THESIS SUBMITTED
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
SCHOOL OF COMPUTING
NATIONAL UNIVERSITY OF SINGAPORE
2016

DECLARATION
I hereby declare that this thesis is my original work and it has been
written by me in its entirety. I have duly acknowledged all the sources of
information which have been used in the thesis.
This thesis has also not been submitted for any degree in any university
previously.
TAO CHEN
20 January 2016
i
ACKNOWLEDGEMENTS
Thisthesiswouldnothavebeenpossiblewithoutthesupport, direction,
and love of a multitude of people.
Foremost, I would like to express my deepest gratitude to my advisor
Prof. Min-Yen Kan for the continuous support of my Ph.D study and re-
search, for his patience, motivation, enthusiasm and immense knowledge.
Without his generous and unwavering help, I would not have been able to
complete this thesis.
I would also like to thank members of my thesis committee, including
Prof. Tat-Seng Chua, Prof. Shengdong Zhao and Prof. Lexing Xie, who
have put in the time and effort to review and assess this thesis. I also want
to thank Prof. Chew-Lim Tan and Prof. Hwee Tou Ng for their insightful
comments to my thesis proposal and graduate research paper.
My sincere thanks also go to Prof. Peng Cui for his generous help and
guidance during the start of my research in multimedia. I also appreciate
the help from my colleagues including Jin Zhao, Jesse Prabawa Gozali,
Ziheng Lin, Junping Ng, Aobo Wang, Jovian Lin, Xiangnan He, Muthu
Chandrasekaran, Dongyuan Lu, Kazunari Sugiyama, Hany SalahEldeen,
Yi Yu and Yongfeng Zhang.
Althoughnotdirectlyrelatedtothisthesis, IwouldliketothankGoogle
for providing the internship opportunity in NYC. It’s a wonderful three-
month experience for both work and life. I also thank Google for its gen-
erous Anita Borg Memorial Scholarship that covers my tuition fee for one
semester.
My appreciation also goes out to my friends who are either in Singa-
pore or far away, for their help and support for my ups and downs. In
particular, I thank my fellow PhD friends for their companion and help for
accomplishing each milestone. I was also very lucky to have great flatmates
in UTown, who made our dormitory like a home.
ii
Finally, I would like to express my heartful thanks to my parents for
their love, support, encouragement and understanding all the time. With-
out them, I would not have the courage to conquer all the difficulties.
iii

CONTENTS
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Key Contributions . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Background 11
2.1 Image Characteristics . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Sentiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Popularity Prediction . . . . . . . . . . . . . . . . . . . . . . 15
2.4 Events and Trending Topics . . . . . . . . . . . . . . . . . . 17
2.5 Credibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.6 Search and Mining . . . . . . . . . . . . . . . . . . . . . . . 20
2.7 User-centric Studies . . . . . . . . . . . . . . . . . . . . . . . 21
3 Understanding Image Tweets 25
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.3 Image Characteristics . . . . . . . . . . . . . . . . . . . . . . 27
3.4 Access Behavior . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.5 Temporal Behavior . . . . . . . . . . . . . . . . . . . . . . . 32
3.6 Medium and Reaction . . . . . . . . . . . . . . . . . . . . . 34
3.7 Content Analysis . . . . . . . . . . . . . . . . . . . . . . . . 36
3.8 Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
v
3.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4 Identifying and Classifying Image-Text Relations 45
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.3 Image and Text Relation . . . . . . . . . . . . . . . . . . . . 47
4.4 Visual/Non-Visual Classification . . . . . . . . . . . . . . . . 49
4.4.1 Dataset Construction . . . . . . . . . . . . . . . . . . 50
4.4.2 Features . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.5 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5 Modeling Image-Text Relations 59
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.3 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . 63
5.4 Visual-Emotional LDA . . . . . . . . . . . . . . . . . . . . . 66
5.4.1 Model Formulation . . . . . . . . . . . . . . . . . . . 66
5.4.2 Parameter Estimation . . . . . . . . . . . . . . . . . 68
5.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . 71
5.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.5.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.5.2 Feature Extraction . . . . . . . . . . . . . . . . . . . 76
5.5.3 Experimental Settings . . . . . . . . . . . . . . . . . 78
5.5.4 Results and Analysis . . . . . . . . . . . . . . . . . . 79
5.6 Towards Microblog Illustration . . . . . . . . . . . . . . . . . 82
5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6 Mining Contextual Text for Image Tweets 87
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 89
6.2.1 Semantics of Image Tweets . . . . . . . . . . . . . . . 90
6.2.2 Tweet Recommendation . . . . . . . . . . . . . . . . 91
6.3 Context-aware Image Tweets Modeling . . . . . . . . . . . . 93
vi
6.3.1 Four Strategies to Construct Contexts . . . . . . . . 93
1. Hashtag Enhanced Text . . . . . . . . . . . . . . . 93
2. Text in Image . . . . . . . . . . . . . . . . . . . . 94
3. External Webpages . . . . . . . . . . . . . . . . . 97
4. Search Engine as Context Miner . . . . . . . . . . 98
6.3.2 Fusing the Text . . . . . . . . . . . . . . . . . . . . . 99
6.4 Personalized Image Tweet Recommendation . . . . . . . . . 100
6.4.1 Drawbacks of Collaborative Filtering . . . . . . . . . 102
6.4.2 Feature-aware MF Framework . . . . . . . . . . . . . 102
6.4.3 Learning from Implicit Feedback . . . . . . . . . . . 105
Time-aware Negative Sampling . . . . . . . . . . . . 106
6.5 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.5.1 Experimental Settings . . . . . . . . . . . . . . . . . 108
6.5.2 Utility of Proposed Contexts (RQ 1) . . . . . . . . . 109
6.5.3 Effectiveness of Context Fusion (RQ 2) . . . . . . . . 111
6.5.4 Importance of Negative Sampling (RQ 3) . . . . . . . 112
6.5.5 Insufficiency of Visual Objects (RQ 4) . . . . . . . . 113
6.5.6 Case Studies . . . . . . . . . . . . . . . . . . . . . . . 114
6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
7 Conclusion and Future Work 119
7.1 Main Contributions . . . . . . . . . . . . . . . . . . . . . . . 119
7.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
vii
ABSTRACT
Social media platforms now allow users to share images alongside their
textual posts. These image tweets make up a fast-growing percentage of
tweets, but have not been studied in depth unlike their text-only counter-
parts. Mostexistingstudiesonimagetweetstackletasksthatareoriginated
from text tweet domain, and their main effort is to incorporate generic im-
age features (e.g., low-level features, deep learning features) to improve the
performance of using text-only approaches.
In this thesis, we conduct a series of studies to answer four fundamen-
tal questions about image tweets: 1) What are the characteristics of image
tweets? 2) What are the relationships between the image and text in image
tweets? 3) How to model such image-text relationships? and 4) How to
interpret the semantics of an image tweet? Answers to these questions will
not only help us gain a deep understanding of image tweets and the related
user behaviors, but also be beneficial to downstream applications.
Toanswerthefirstquestion, wecollectalargecorpusofmicroblogposts
from (Western) Twitter and (Chinese) Sina Weibo. We find 56.0% of posts
on Weibo and 14.1% on Twitter are image tweets. We then perform a
multipronged analysis of these image tweets from the perspective of image
characteristics, user posting behaviors (e.g., temporal, access) and textual
contents.
As an image tweet usually consists of two media—an image and its ac-
companying text, we naturally ask: what are the relationships between
these two? Using an appropriate corpus analysis, we identify two key
image-text relations for image tweets: visual relevance and emotional rel-
evance. Considering the practical values of visually relevant image tweets,
webuildanautomatedclassifierutilizingtext, imageandsocialcontextfea-
tures to distinguish them from the others, obtaining a macro F of 70.5%.
1
Based on this, a follow-up question arises: can we model these image-
text relations and explain how image tweets are generated? To this end, we
viii