Collective Intelligence in Action

425 Pages·2009·11.39 MB·English

by  Satnam Alag

Collective Intelligence in Action

Collective Intelligence
in Action
SATNAM ALAG
MANNING
Greenwich
(74° w. long.)
To my dear sons, Ayush and Shray,
and my beautiful, loving, and intelligent wife, Alpana

For online information and ordering of this and other Manning books, please visit
www.manning.com. The publisher offers discounts on this book when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
Sound View Court 3B fax: [phone]
Greenwich, CT 06830 email: [email protected]
©2009 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in
any form or by means electronic, mechanical, photocopying, or otherwise, without prior written
permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are
claimed as trademarks. Where those designations appear in the book, and Manning
Publications was aware of a trademark claim, the designations have been printed in initial caps
or all caps.
Recognizing the importance of preserving what has been written, it is Manning’s policy to have
the books we publish printed on acid-free paper, and we exert our best efforts to that end.
Recognizing also our responsibility to conserve the resources of our planet, Manning books are
printed on paper that is at least 15% recycled and processed without the use of elemental chlorine.
Development Editor: Jeff Bleiel
Manning Publications Co. Copyeditor: Benjamin Berg
Sound View Court 3B Typesetter: Gordan Salinovic
Greenwich, CT 06830 Cover designer: Leslie Haimes
ISBN 1933988312
Printed in the United States of America
1 2 3 4 5 6 7 8 9 10 – MAL – 13 12 11 10 09 08
brief contents
P 1 G .....................................1
ART ATHERING DATA FOR INTELLIGENCE
1 ■ Understanding collective intelligence 3
2 ■ Learning from user interactions 20
3 ■ Extracting intelligence from tags 50
4 ■ Extracting intelligence from content 82
5 ■ Searching the blogosphere 107
6 ■ Intelligent web crawling 145
P 2 D ..................................................173
ART ERIVING INTELLIGENCE
7 ■ Data mining: process, toolkits, and standards 175
8 ■ Building a text analysis toolkit 206
9 ■ Discovering patterns with clustering 240
10 ■ Making predictions 274
P 3 A .................... 307
ART PPLYING INTELLIGENCE IN YOUR APPLICATION
11 ■ Intelligent search 309
12 ■ Building a recommendation engine 349

v

contents
foreword xvii
preface xix
acknowledgments xxi
about this book xxiii
P 1 G ...................... 1
ART ATHERING DATA FOR INTELLIGENCE
1 Understanding collective intelligence 3
1.1 What is collective intelligence? 4
1.2 CI in web applications 6
Collective intelligence from the ground up: a sample
application 7 ■ Benefits of collective intelligence 9 ■ CI is the
core component of Web 2.0 10 ■ Harnessing CI to transform from
content-centric to user-centric applications 12
1.3 Classifying intelligence 14
Explicit intelligence 14 ■ Implicit intelligence 15 ■ Derived
intelligence 16
1.4 Summary 18
1.5 Resources 18
vii
viii CONTENTS
2 Learning from user interactions 20
2.1 Architecture for applying intelligence 21
Synchronous and asynchronous services 21 ■ Real-time
learning in an event-driven system 23 ■ Polling services for
non–event-driven systems 24 ■ Advantages and disadvantages
of event-based and non–event-based architectures 25
2.2 Basics of algorithms for applying CI 25
Users and items 26 ■ Representing user information 27
Content-based analysis and collaborative filtering 29
Representing intelligence from unstructured text 30
Computing similarities 31 ■ Types of datasets 32
2.3 Forms of user interaction 34
Rating and voting 35 ■ Emailing or forwarding a
link 36 ■ Bookmarking and saving 36 ■ Purchasing
items 37 ■ Click-stream 37 ■ Reviews 39
2.4 Converting user interaction into collective intelligence 41
Intelligence from ratings via an example 41 ■ Intelligence from
bookmarking, saving, purchasing Items, forwarding, click-stream,
and reviews 46
2.5 Summary 48
2.6 Resources 48
3 Extracting intelligence from tags 50
3.1 Introduction to tagging 51
Tag-related metadata for users and items 52 ■ Professionally
generated tags 52 ■ User-generated tags 53 ■ Machine-generated
tags 54 ■ Tips on tagging 55 ■ Why do users tag? 55
3.2 How to leverage tags 56
Building dynamic navigation 56 ■ Innovative uses of tag clouds 58
Targeted search 59 ■ Folksonomies and building a dictionary 60
3.3 Extracting intelligence from user tagging: an example 60
Items related to other items 61 ■ Items of interest for a
user 61 ■ Relevant users for an item 62
3.4 Scalable persistence architecture for tagging 62
Reviewing other approaches 63 ■ Recommended persistence
architecture 66
3.5 Building tag clouds 69
Persistence design for tag clouds 69 ■ Algorithm for building a tag
cloud 70 ■ Implementing a tag cloud 71 ■ Visualizing a tag
cloud 76
CONTENTS ix
3.6 Finding similar tags 79
3.7 Summary 80
3.8 Resources 81
4 Extracting intelligence from content 82
4.1 Content types and integration 83
Classifying content 83 ■ Architecture for integrating content 85
4.2 The main CI-related content types 86
Blogs 87 ■ Wikis 89 ■ Groups and message boards 91
4.3 Extracting intelligence step by step 93
Setting up the example 94 ■ Naïve analysis 95 ■ Removing
common words 98 ■ Stemming 99 ■ Detecting phrases 100
4.4 Simple and composite content types 102
4.5 Summary 103
4.6 Resources 104
5 Searching the blogosphere 107
5.1 Introducing the blogosphere 108
Leveraging the blogosphere 108 ■ RSS: the publishing
format 109 ■ Blog-tracking companies 111
5.2 Building a framework to search the blogosphere 111
The searcher 113 ■ The search parameters 113 ■ The query
results 114 ■ Handling the XML response 115 ■ Exception
handling 116
5.3 Implementing the base classes 116
Implementing the search parameters 117 ■ Implementing the result
objects 117 ■ Implementing the searcher 119 ■ Parsing XML
response 123 ■ Extending the framework 127
5.4 Integrating Technorati 128
Technorati search API overview 128 ■ Implementing classes for
integrating Technorati 130
5.5 Integrating Bloglines 135
Bloglines search API overview 135 ■ Implementing classes for
integrating Bloglines 136
5.6 Integrating providers using RSS 139
Generalizing the query parameters 139 ■ Generalizing the blog
searcher 140 ■ Building the RSS 2.0 XML parser 141
5.7 Summary 143
5.8 Resources 143