Mining the Social Web: Analyzing Data from Facebook, Twitter, LinkedIn, and Other Social Media Sites

344 Pages·2011·4.81 MB·English

by  Matthew A. Russell

Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
1. Introduction: Hacking on Twitter Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Installing Python Development Tools 1
Collecting and Manipulating Twitter Data 3
Tinkering with Twitter’s API 4
Frequency Analysis and Lexical Diversity 7
Visualizing Tweet Graphs 14
Synthesis: Visualizing Retweets with Protovis 15
Closing Remarks 17
2. Microformats: Semantic Markup and Common Sense Collide . . . . . . . . . . . . . . . . . . 19
XFN and Friends 19
Exploring Social Connections with XFN 22
A Breadth-First Crawl of XFN Data 23
Geocoordinates: A Common Thread for Just About Anything 30
Wikipedia Articles + Google Maps = Road Trip? 30
Slicing and Dicing Recipes (for the Health of It) 35
Collecting Restaurant Reviews 37
Summary 40
3. Mailboxes: Oldies but Goodies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
mbox: The Quick and Dirty on Unix Mailboxes 42
mbox + CouchDB = Relaxed Email Analysis 48
Bulk Loading Documents into CouchDB 51
Sensible Sorting 52
Map/Reduce-Inspired Frequency Analysis 55
Sorting Documents by Value 61
couchdb-lucene: Full-Text Indexing and More 63
Threading Together Conversations 67
Look Who’s Talking 73
ix
Visualizing Mail “Events” with SIMILE Timeline 77
Analyzing Your Own Mail Data 80
The Graph Your (Gmail) Inbox Chrome Extension 81
Closing Remarks 82
4. Twitter: Friends, Followers, and Setwise Operations . . . . . . . . . . . . . . . . . . . . . . . . . 83
RESTful and OAuth-Cladded APIs 84
No, You Can’t Have My Password 85
A Lean, Mean Data-Collecting Machine 88
A Very Brief Refactor Interlude 91
Redis: A Data Structures Server 92
Elementary Set Operations 94
Souping Up the Machine with Basic Friend/Follower Metrics 96
Calculating Similarity by Computing Common Friends and Followers 102
Measuring Influence 103
Constructing Friendship Graphs 108
Clique Detection and Analysis 110
The Infochimps “Strong Links” API 114
Interactive 3D Graph Visualization 116
Summary 117
5. Twitter: The Tweet, the Whole Tweet, and Nothing but the Tweet . . . . . . . . . . . . 119
Pen : Sword :: Tweet : Machine Gun (?!?) 119
Analyzing Tweets (One Entity at a Time) 122
Tapping (Tim’s) Tweets 125
Who Does Tim Retweet Most Often? 138
What’s Tim’s Influence? 141
How Many of Tim’s Tweets Contain Hashtags? 144
Juxtaposing Latent Social Networks (or #JustinBieber Versus #TeaParty) 147
What Entities Co-Occur Most Often with #JustinBieber and #TeaParty
Tweets? 148
On Average, Do #JustinBieber or #TeaParty Tweets Have More
Hashtags? 153
Which Gets Retweeted More Often: #JustinBieber or #TeaParty? 154
How Much Overlap Exists Between the Entities of #TeaParty and
#JustinBieber Tweets? 156
Visualizing Tons of Tweets 158
Visualizing Tweets with Tricked-Out Tag Clouds 158
Visualizing Community Structures in Twitter Search Results 162
Closing Remarks 166
6. LinkedIn: Clustering Your Professional Network for Fun (and Profit?) . . . . . . . . . . 167
Motivation for Clustering 168
x | Table of Contents
Clustering Contacts by Job Title 172
Standardizing and Counting Job Titles 172
Common Similarity Metrics for Clustering 174
A Greedy Approach to Clustering 177
Hierarchical and k-Means Clustering 185
Fetching Extended Profile Information 188
Geographically Clustering Your Network 193
Mapping Your Professional Network with Google Earth 193
Mapping Your Professional Network with Dorling Cartograms 198
Closing Remarks 198
7. Google Buzz: TF-IDF, Cosine Similarity, and Collocations . . . . . . . . . . . . . . . . . . . . . 201
Buzz = Twitter + Blogs (???) 202
Data Hacking with NLTK 205
Text Mining Fundamentals 209
A Whiz-Bang Introduction to TF-IDF 209
Querying Buzz Data with TF-IDF 215
Finding Similar Documents 216
The Theory Behind Vector Space Models and Cosine Similarity 217
Clustering Posts with Cosine Similarity 219
Visualizing Similarity with Graph Visualizations 222
Buzzing on Bigrams 224
How the Collocation Sausage Is Made: Contingency Tables and Scoring
Functions 228
Tapping into Your Gmail 231
Accessing Gmail with OAuth 232
Fetching and Parsing Email Messages 233
Before You Go Off and Try to Build a Search Engine… 235
Closing Remarks 237
8. Blogs et al.: Natural Language Processing (and Beyond) . . . . . . . . . . . . . . . . . . . . . 239
NLP: A Pareto-Like Introduction 239
Syntax and Semantics 240
A Brief Thought Exercise 241
A Typical NLP Pipeline with NLTK 242
Sentence Detection in Blogs with NLTK 245
Summarizing Documents 250
Analysis of Luhn’s Summarization Algorithm 256
Entity-Centric Analysis: A Deeper Understanding of the Data 258
Quality of Analytics 267
Closing Remarks 269
Table of Contents | xi
9. Facebook: The All-in-One Wonder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
Tapping into Your Social Network Data 272
From Zero to Access Token in Under 10 Minutes 272
Facebook’s Query APIs 278
Visualizing Facebook Data 289
Visualizing Your Entire Social Network 289
Visualizing Mutual Friendships Within Groups 301
Where Have My Friends All Gone? (A Data-Driven Game) 304
Visualizing Wall Data As a (Rotating) Tag Cloud 309
Closing Remarks 311
10. The Semantic Web: A Cocktail Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
An Evolutionary Revolution? 313
Man Cannot Live on Facts Alone 315
Open-World Versus Closed-World Assumptions 315
Inferencing About an Open World with FuXi 316
Hope 319
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
xii | Table of Contents
Preface
The Web is more a social creation than a technical one.
I designed it for a social effect—to help people work
together—and not as a technical toy. The ultimate goal
of the Web is to support and improve our weblike exis-
tence in the world. We clump into families, associations,
and companies. We develop trust across the miles and
distrust around the corner.
—Tim Berners-Lee, Weaving the Web (Harper)
To Read This Book?
If you have a basic programming background and are interested in insight surrounding
the opportunities that arise from mining and analyzing data from the social web, you’ve
come to the right place. We’ll begin getting our hands dirty after just a few more pages
of frontmatter. I’ll be forthright, however, and say upfront that one of the chief com-
plaints you’re likely to have about this book is that all of the chapters are far too short.
Unfortunately, that’s always the case when trying to capture a space that’s evolving
daily and is so rich and abundant with opportunities. That said, I’m a fan of the “80-20
rule”, and I sincerely believe that this book is a reasonable attempt at presenting the
most interesting 20 percent of the space that you’d want to explore with 80 percent of
your available time.
This book is short, but it does cover a lot of ground. Generally speaking, there’s a little
more breadth than depth, although where the situation lends itself and the subject
matter is complex enough to warrant a more detailed discussion, there are a few deep
dives into interesting mining and analysis techniques. The book was written so that
you could have the option of either reading it from cover to cover to get a broad primer
on working with social web data, or pick and choose chapters that are of particular
interest to you. In other words, each chapter is designed to be bite-sized and fairly
standalone, but special care was taken to introduce material in a particular order so
that the book as a whole is an enjoyable read.
xiii
Social networking websites such as Facebook, Twitter, and LinkedIn have transitioned
from fad to mainstream to global phenomena over the last few years. In the first quarter
of 2010, the popular social networking site Facebook surpassed Google for the most
*
page visits, confirming a definite shift in how people are spending their time online.
Asserting that this event indicates that the Web has now become more a social milieu
than a tool for research and information might be somewhat indefensible; however,
this data point undeniably indicates that social networking websites are satisfying some
very basic human desires on a massive scale in ways that search engines were never
designed to fulfill. Social networks really are changing the way we live our lives on and
†
off the Web, and they are enabling technology to bring out the best (and sometimes
the worst) in us. The explosion of social networks is just one of the ways that the gap
between the real world and cyberspace is continuing to narrow.
Generally speaking, each chapter of this book interlaces slivers of the social web along
with data mining, analysis, and visualization techniques to answer the following kinds
of questions:
• Who knows whom, and what friends do they have in common?
• How frequently are certain people communicating with one another?
• How symmetrical is the communication between people?
• Who are the quietest/chattiest people in a network?
• Who are the most influential/popular people in a network?
• What are people chatting about (and is it interesting)?
The answers to these types of questions generally connect two or more people together
and point back to a context indicating why the connection exists. The work involved
in answering these kinds of questions is only the beginning of more complex analytic
processes, but you have to start somewhere, and the low-hanging fruit is surprisingly
easy to grasp, thanks to well-engineered social networking APIs and open source
toolkits.
‡
Loosely speaking, this book treats the social web as a graph of people, activities, events,
concepts, etc. Industry leaders such as Google and Facebook have begun to increasingly
push graph-centric terminology rather than web-centric terminology as they simulta-
neously promote graph-based APIs. In fact, Tim Berners-Lee has suggested that perhaps
he should have used the term Giant Global Graph (GGG) instead of World Wide Web
(WWW), because the terms “web” and “graph” can be so freely interchanged in the
context of defining a topology for the Internet. Whether the fullness of Tim Berners-
* See the opening paragraph of Chapter 9.
† Mark Zuckerberg, the creator of Facebook, was named Person of the Year for 2010 by Time magazine (http:
//www.time.com/time/specials/packages/article/0,28804,2036683_2037183_2037185,00.html)
‡ See http://journal.planetwork.net/article.php?lab=reed0704 for another perspective on the social web that
focuses on digital identities.
xiv | Preface
Lee’s original vision will ever be realized remains to be seen, but the Web as we know
it is getting richer and richer with social data all the time. When we look back years
from now, it may well seem obvious that the second- and third-level effects created by
an inherently social web were necessary enablers for the realization of a truly semantic
web. The gap between the two seems to be closing.
Or Not to Read This Book?
Activities such as building your own natural language processor from scratch, venturing
far beyond the typical usage of visualization libraries, and constructing just about any-
thing state-of-the-art are not within the scope of this book. You’ll be really disappointed
if you purchase this book because you want to do one of those things. However, just
because it’s not realistic or our goal to capture the holy grail of text analytics or record
matching in a mere few hundred pages doesn’t mean that this book won’t enable you
to attain reasonable solutions to hard problems, apply those solutions to the social web
as a domain, and have a lot of fun in the process. It also doesn’t mean that taking a very
active interest in these fascinating research areas wouldn’t potentially be a great idea
for you to consider. A short book like this one can’t do much beyond whetting your
appetite and giving you enough insight to go out and start making a difference some-
where with your newly found passion for data hacking.
Maybe it’s obvious in this day and age, but another important item of note is that this
book generally assumes that you’re connected to the Internet. This wouldn’t be a great
book to take on vacation with you to a remote location, because it contains many
references that have been hyperlinked, and all of the code examples are hyperlinked
directly to GitHub, a very social Git repository that will always reflect the most up-to-
date example code available. The hope is that social coding will enhance collaboration
between like-minded folks such as ourselves who want to work together to extend the
examples and hack away at interesting problems. Hopefully, you’ll fork, extend, and
improve the source—and maybe even make some new friends along the way. Readily
accessible sources of online information such as API docs are also liberally hyperlinked,
and it is assumed that you’d rather look them up online than rely on inevitably stale
copies in this printed book.
The official GitHub repository that maintains the latest and greatest
bug-fixed source code for this book is http://github.com/ptwobrussell/
Mining-the-Social-Web. The official Twitter account for this book is
@SocialWebMining.
This book is also not recommended if you need a reference that gets you up to speed
on distributed computing platforms such as sharded MySQL clusters or NoSQL tech-
nologies such as Hadoop or Cassandra. We do use some less-than-conventional storage
technologies such as CouchDB and Redis, but always within the context of running on
Preface | xv