Numbersense : how to use big data to your advantage

214 Pages·2013·3.51 MB·English

by  Fung

Copyright © 2013 by Kaiser Fung. All rights reserved. Except as permitted
under the United States Copyright Act of 1976, no part of this publication may
be reproduced or distributed in any form or by any means, or stored in a database
or retrieval system, without the prior written permission of the publisher.
ISBN: 978-0-07179967-6
MHID: 0-07-1799672
The material in this eBook also appears in the print version of this title: ISBN:
978-0-07-179966-9, MHID: 0-07-179966-4.
All trademarks are trademarks of their respective owners. Rather than put a
trademark symbol after every occurrence of a trademarked name, we use names
in an editorial fashion only, and to the benefit of the trademark owner, with no
intention of infringement of the trademark. Where such designations appear in
this book, they have been printed with initial caps.
McGraw-Hill Education eBooks are available at special quantity discounts to use
as premiums and sales promotions or for use in corporate training programs. To
contact a representative please visit the Contact Us page at
www.mhprofessional.com.
This publication is designed to provide accurate and authoritative information in
regard to the subject matter covered. It is sold with the understanding that neither
the author nor the publisher is engaged in rendering legal, accounting, or other
professional service. If legal advice or other expert assistance is required, the
services of a competent professional person should be sought.
—From a Declaration of Principles Jointly Adopted by a Committee of the
American Bar Association and a Committee of Publishers and Associations
TERMS OF USE
This is a copyrighted work and McGraw-Hill Education and its licensors reserve
all rights in and to the work. Use of this work is subject to these terms. Except as
permitted under the Copyright Act of 1976 and the right to store and retrieve one
copy of the work, you may not decompile, disassemble, reverse engineer,
reproduce, modify, create derivative works based upon, transmit, distribute,
disseminate, sell, publish or sublicense the work or any part of it without
McGraw-Hill Education’s prior consent. You may use the work for your own
noncommercial and personal use; any other use of the work is strictly prohibited.
Your right to use the work may be terminated if you fail to comply with these
terms.
THE WORK IS PROVIDED “AS IS.” McGRAW-HILL EDUCATION AND
ITS LICENSORS MAKE NO GUARANTEES OR WARRANTIES AS TO
THE ACCURACY, ADEQUACY OR COMPLETENESS OF OR RESULTS
TO BE OBTAINED FROM USING THE WORK, INCLUDING ANY
INFORMATION THAT CAN BE ACCESSED THROUGH THE WORK VIA
HYPERLINK OR OTHERWISE, AND EXPRESSLY DISCLAIM ANY
WARRANTY, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR
A PARTICULAR PURPOSE. McGraw-Hill Education and its licensors do not
warrant or guarantee that the functions contained in the work will meet your
requirements or that its operation will be uninterrupted or error free. Neither
McGraw-Hill Education nor its licensors shall be liable to you or anyone else for
any inaccuracy, error or omission, regardless of cause, in the work or for any
damages resulting therefrom. McGraw-Hill Education has no responsibility for
the content of any information accessed through the work. Under no
circumstances shall McGraw-Hill Education and/or its licensors be liable for any
indirect, incidental, special, punitive, consequential or similar damages that
result from the use of or inability to use the work, even if any of them has been
advised of the possibility of such damages. This limitation of liability shall apply
to any claim or cause whatsoever whether such claim or cause arises in contract,
tort or otherwise.
Contents
Acknowledgments
List of Figures
Prologue
PART 1 SOCIAL DATA
1 Why Do Law School Deans Send Each Other Junk Mail?
2 Can a New Statistic Make Us Less Fat?
PART 2 MARKETING DATA
3 How Can Sellouts Ruin a Business?
4 Will Personalizing Deals Save Groupon?
5 Why Do Marketers Send You Mixed Messages?
PART 3 ECONOMIC DATA
6 Are They New Jobs If No One Can Apply?
7 How Much Did You Pay for the Eggs?
PART 4 SPORTING DATA
8 Are You a Better Coach or Manager?
EPILOGUE
References
Index
Acknowledgments
Iowe a great debt to readers of Numbers Rule Your World and my two blogs, and
followers on Twitter. Your support keeps me going. Your enthusiasm has carried
over to the McGraw-Hill team, led by Knox Huston. Knox shepherded this
project while meeting the demands of being a new father. Many thanks to the
production crew for putting up with the tight schedule. Grace Freedson, my
agent, saw the potential of the book.
Jay Hu, Augustine Fou, and Adam Murphy contributed materials that made
their way into the text. They also reviewed early drafts. The following people
assisted me by discussing ideas, making connections or reading parts of the
manuscript: Larry Cahoon, Steven Paben, Darrell Phillipson, Maggie Jordan,
Kate Johnson, Steven Tuntono, Amanda Lee, Barbara Schoetzau, Andrew
Tilton, Chiang-ling Ng, Dr. Cesare Russo, Bill McBride, Annette Fung, Kelvin
Neu, Andrew Lefevre, Patty Wu, Valerie Thomas, Hillary Wool, Tara Tarpey,
Celine Fung, Cathie Mahoney, Sam Kumar, Hui Soo Chae, Mike Kruger, John
Lien, Scott Turner, Micah Burch, and Andrew Gelman. Laurent Lheritier is a
friend whom I inadvertently left out last time. The odds are good that the above
list is not complete, so please accept my sincere apology for any omission.
Double thanks to all who took time out of their busy lives to comment on
chapters. A special nod to my brother Pius for being a willing subject in my
experiment to foist Chapter 8 on non-sports fans.
This book is dedicated to my grandmother, who sadly will not see it come to
print. A brave woman who grew up in tumultuous times, she taught herself to
read and cook. Her cooking honed my appreciation for food, and since the field
of statistics borrows quite a few culinary words, her influence is felt within these
pages.
New York, April 2013
List of Figures
P-1 America West Had a Lower Flight Delay Rate, Aggregate of Five West
Coast Airports
P-2 Alaska Flights Had Lower Flight Delay Rates Than America West Flights at
All Five West Coast Airports
P-3 National Polls on the 2012 U.S. Presidential Election
P-4 Re-weighted National Polls on the 2012 U.S. Presidential Election
P-5 Explanation of Simpson’s Paradox in Flight Delay Data
P-6 The Flight Delay Data
1-1 Components of the U.S. News Law School Ranking Formula
1-2 Faking the Median GPA by Altering Individual Data
1-3 The Missing-Card Trick
1-4 Downsizing
1-5 Unlimited Refills
1-6 Law Schools Connect
1-7 Partial Credits
1-8 Doping Does Not Help, So They Say
2-1 The Curved Relationship between Body Mass Index and Mortality
2-2 Region of Disagreement between BMI and DXA
3-1 The Groupon Deal Offered by Giorgio’s of Gramercy in January 2011
3-2 The Case of the Missing Revenues
3-3 Merchant Grouponomics
3-4 The Official Analysis is Too Simple
4-1 Matching Groupons to Fou’s Interests
4-2 Trend in Deal Types
4-3 Method One of Targeting
4-4 Method Two of Targeting
4-5 Method Three of Targeting
4-6 Conflicting Objectives of Targeting
5-1 The Mass Retailer Target Uses Prior Purchases to Predict Future Purchases
5-2 Evaluating a Predictive Model
5-3 Latent Factors in Modeling Consumer Behavior
6-1 The Scariest Jobs Chart
6-2 Snow Days of February 2010
6-3 The Truth According to Crudele
6-4 Seasonality
6-5 Official Unemployment Rate, Sometimes Known as U-3
6-6 Growth in the Population Considered Not in Labor Force
6-7 The U-5 Unemployment Rate
6-8 Another Unemployment Rate
6-9 Employment-Population Ratio (2002–2012)
7-1 A Sample Consumer Expenditure Basket
7-2 Core versus Headline Inflation Rates
7-3 Major Categories of Consumer Expenditures
7-4 Food and Energy Component CPI
7-5 How Prices of Selected Foods Changed Since 2008—Eggs and Milk
7-6 How Prices of Selected Foods Changed Since 2008—Fruits and Vegetables
7-7 How Prices of Selected Foods Changed Since 2008—Coffee and Bakery
Goods
8-1 Win Total and Points Total of 14 Teams in the Tiffany Victoria Memorial
Fantasy Football League, 2011–2012
8-2 Jean’s Selected Squad, a Modified Squad, and the Optimal Squad for Week
13 in the Tiffany Victoria Memorial Fantasy Football League, 2011–2012
8-3 Coach’s Prafs and Ranking in the Tiffany Victoria Memorial Fantasy
Football League, 2011–2012
8-4 The Points Totals of All 240 Feasible Squads in Week 8 for Perry’s Team in
the Tiffany Victoria Memorial Fantasy Football League, 2011–2012
8-5 The Points Totals of All Feasible Squads in All Weeks for Perry’s Team in
the Tiffany Victoria Memorial Fantasy Football League, 2011–2012
8-6 Manager’s Polac Points and Ranking in the Tiffany Victoria Memorial
Fantasy Football League, 2011–2012
8-7 The 14 Teams in the Tiffany Victoria Memorial Fantasy Football League
Divided into Three Types, According to Coaching and Managerial Skills
8-8 Luck in the Tiffany Victoria Memorial Fantasy Football League, 2011–2012