Statistics Hacks: Tips & Tools for Measuring the World and Beating the Odds

289 Pages·2006·2.24 MB·English

by  Bruce Frey

Statistics Hacks
By Bruce Frey
...............................................< tr> Publisher: O’Reilly
Pub Date: May 2006
Print ISBN-10: 0-596-10164-3
Print ISBN-13: 978-0-59-610164-0
Pages: 356

Table of Contents | Index
Want to calculate the probability that an event will happen? Be able to spot fake data? Prove beyond
doubt whether one thing causes another? Or learn to be a better gambler? You can do that and much
more with 75 practical and fun hacks packed into Statistics Hacks. These cool tips, tricks, and
mind-boggling solutions from the world of statistics, measurement, and research methods will not only
amaze and entertain you, but will give you an advantage in several real-world situations-including
business.
This book is ideal for anyone who likes puzzles, brainteasers, games, gambling, magic tricks, and those
who want to apply math and science to everyday circumstances. Several hacks in the first chapter
alone-such as the "central limit theorem,", which allows you to know everything by knowing just a
little-serve as sound approaches for marketing and other business objectives. Using the tools of
inferential statistics, you can understand the way probability works, discover relationships, predict events
with uncanny accuracy, and even make a little money with a well-placed wager here and there.
Statistics Hacks presents useful techniques from statistics, educational and psychological measurement,
and experimental research to help you solve a variety of problems in business, games, and life. You’ll
learn how to:
Play smart when you play Texas Hold ’Em, blackjack, roulette, dice games, or even the lottery
Design your own winnable bar bets to make money and amaze your friends
Predict the outcomes of baseball games, know when to "go for two" in football, and anticipate the
winners of other sporting events with surprising accuracy
Demystify amazing coincidences and distinguish the truly random from the only seemingly
random--even keep your iPod’s "random" shuffle honest
Spot fraudulent data, detect plagiarism, and break codes
How to isolate the effects of observation on the thing observed
Whether you’re a statistics enthusiast who does calculations in your sleep or a civilian who is entertained
by clever solutions to interesting problems, Statistics Hacks has tools to give you an edge over the
world’s slim odds.

Statistics Hacks
By Bruce Frey
...............................................< tr> Publisher: O’Reilly
Pub Date: May 2006
Print ISBN-10: 0-596-10164-3
Print ISBN-13: 978-0-59-610164-0
Pages: 356

Table of Contents | Index
credits Credits< tr> Preface
Chapter 1. The Basics
Hack 1. Know the Big Secret
Hack 2. Describe the World Using Just Two Numbers
Hack 3. Figure the Odds
Hack 4. Reject the Null
Hack 5. Go Big to Get Small
Hack 6. Measure Precisely
Hack 7. Measure Up
Hack 8. Power Up
Hack 9. Show Cause and Effect
Hack 10. Know Big When You See It
Chapter 2. Discovering Relationships
Hack 11. Discover Relationships
Hack 12. Graph Relationships
Hack 13. Use One Variable to Predict Another
Hack 14. Use More Than One Variable to Predict Another
Hack 15. Identify Unexpected Outcomes
Hack 16. Identify Unexpected Relationships
Hack 17. Compare Two Groups
Hack 18. Find Out Just How Wrong You Really Are
Hack 19. Sample Fairly
Hack 20. Sample with a Touch of Scotch
Hack 21. Choose the Honest Average
Hack 22. Avoid the Axis of Evil
Chapter 3. Measuring the World
Hack 23. See the Shape of Everything
Hack 24. Produce Percentiles
Hack 25. Predict the Future with the Normal Curve
Hack 26. Give Raw Scores a Makeover
Hack 27. Standardize Scores
Hack 28. Ask the Right Questions
Hack 29. Test Fairly
Hack 30. Improve Your Test Score While Watching Paint Dry
Hack 31. Establish Reliability
Hack 32. Establish Validity
Hack 33. Predict the Length of a Lifetime
Hack 34. Make Wise Medical Decisions
Chapter 4. Beating the Odds
Hack 35. Gamble Smart
Hack 36. Know When to Hold ’Em
Hack 37. Know When to Fold ’Em
Hack 38. Know When to Walk Away
Hack 39. Lose Slowly at Roulette
Hack 40. Play in the Black in Blackjack
Hack 41. Play Smart When You Play the Lottery
Hack 42. Play with Cards and Get Lucky
Hack 43. Play with Dice and Get Lucky
Hack 44. Sharpen Your Card-Sharping
Hack 45. Amaze Your 23 Closest Friends
Hack 46. Design Your Own Bar Bet
Hack 47. Go Crazy with Wild Cards
Hack 48. Never Trust an Honest Coin
Hack 49. Know Your Limit
Chapter 5. Playing Games
Hack 50. Avoid the Zonk
Hack 51. Pass Go, Collect $200, Win the Game
Hack 52. Use Random Selection as Artificial Intelligence
Hack 53. Do Card Tricks Through the Mail
Hack 54. Check Your iPod’s Honesty
Hack 55. Predict the Game Winners
Hack 56. Predict the Outcome of a Baseball Game
Hack 57. Plot Histograms in Excel
Hack 58. Go for Two
Hack 59. Rank with the Best of Them
Hack 60. Estimate Pi by Chance
Chapter 6. Thinking Smart
Hack 61. Outsmart Superman
Hack 62. Demystify Amazing Coincidences
Hack 63. Sense the Real Randomness of Life
Hack 64. Spot Faked Data
Hack 65. Give Credit Where Credit Is Due
Hack 66. Play a Tune on Pascal’s Triangle
Hack 67. Control Random Thoughts
Hack 68. Search for ESP
Hack 69. Cure Conjunctionitus
Hack 70. Break Codes with Etaoin Shrdlu
Hack 71. Discover a New Species
Hack 72. Feel Connected
Hack 73. Learn to Ride a Votercycle
Hack 74. Live Life in the Fast Lane (You’re Already In)
Hack 75. Seek Out New Life and New Civilizations
Colophon
Index
Credits
About the Author
Bruce Frey, Ph.D., is a comic book collector and film buff. In his spare time, he teaches statistics to
graduate students and conducts research in his secret identity as an assistant professor in Educational
Psychology and Research at the University of Kansas. He is an award-winning teacher, and his scholarly
research interests are in the areas of teacher-made tests and classroom assessment, the measurement of
spirituality, and program evaluation methods. Bruce’s honors include taking third place in the Kansas
Monopoly Championship as a teenager, second place in the Kansas Film Festival as a college student, and
a respectable third-place finish in the Lawrence, Kansas, Texas Hold ’Em Poker Tournament as a
middle-aged man. He is proudest of two accomplishments: his marriage to his sweet wife, and his
purchase of a low-grade copy of Showcase #4, a comic book wherein the "Silver Age Flash first appears,"
whatever that means.
Contributors
The following people contributed their hacks, writing, and inspiration to this book:
Joseph Adler is the author of Baseball Hacks (O’Reilly), and a researcher in the Advanced Product
Development Group at VeriSign, focusing on problems in user authentication, managed security
services, and RFID security. Joe has years of experience analyzing data, building statistical models,
and formulating business strategies as an employee and consultant for companies including
DoubleClick, American Express, and Dun & Bradstreet. He is a graduate of the Massachusetts
Institute of Technology with an Sc.B. and an M.Eng. in computer science and computer engineering.
Joe is an unapologetic Yankees fan, but he appreciates any good baseball game. Joe lives in Silicon
Valley with his wife, two cats, and a DirecTV satellite dish.
Ron Hale-Evans is a writer, thinker, and game designer who earns his daily sandwich with frequent
gigs as a technical writer. He has a Bachelor’s degree in Psychology from Yale, with a minor in
Philosophy. Thinking a lot about thinking led him to create the Mentat Wiki
(http://www.ludism.org/mentat), which led to his recent book, Mind Performance Hacks (O’Reilly).
You can find his multinefarious [sic] other projects at his home page, http://ron.ludism.org, including
his award-winning board games, a list of his Short-Duration Personal Saviors, and his blog. Ron’s
next book will probably be about game systems, especially since his series of articles on that topic for
the dearly departed The Games Journal (http://www.thegamesjournal.com< /a>) has been relatively
popular among both gamers and academics. If you want to email Ron the names of some gullible
publishers, or if you just want to bug him, you can reach him at [email protected] (rhymes with
nudism and has nothing to do with Luddism).
Brian E. Hansen, 27, grew up in the Dallas, Texas area. After serving a two-year religious mission in
Spain, he attended Texas A&M University and graduated in 2004 with a B.S. degree in Petroleum
Engineering. He currently works as a Reservoir Engineer for a large independent oil and gas exploration
and production company headquartered in Irving, Texas.
Jill H. Lohmeier received her Ph.D. in Cognitive Psychology from The University of Massachusetts,
Amherst. She is currently the Evaluation Director for the School Program Evaluation and Research group
at the University of Kansas. Jill likes outdoor sports, especially running, hiking, and playing soccer with
her kids.
Ernest E. Rothman is a Professor and Chair of the Mathematical Sciences Department at Salve
Regina University (SRU) in Newport, Rhode Island. Ernie holds a Ph.D. in Applied Mathematics from
Brown University and held positions at the Cornell Theory Center in Ithaca, New York before coming to
SRU. His interests are primarily in scientific computing, mathematics and statistics education, and the
Unix underpinnings of Mac OS X. You can keep abreast of his latest activities at
http://homepage.mac.com/samchops< /a>.
Neil J. Salkind is a sometimes faculty member at the University of Kansas with an office opposite
that of Bruce Frey, of Statistics Hacks fame. In addition to being the author of Statistics for People Who
(Think They) Hate Statistics (SAGE), Neil is a developmental psychologist who collects books, cooks,
works on old houses and a p1800 Volvo, and is active in Masters swimming. He has also written over 100
trade books and textbooks, and works with StudioB Literary Agency in New York.
William Skorupski is currently an assistant professor in the School of Education at the University of
Kansas, where he teaches courses in psychometrics and statistics. He earned his Bachelor’s degree in
educational research and psychology from Bucknell University in 2000, and his Doctorate in psychometric
methods from the University of Massachusetts, Amherst in 2004. His primary research interest is in the
application of mathematical models to psychometric data, including the use of Bayesian statistics for
solving practical measurement problems. He also enjoys applying his knowledge of statistics and
probability to everyday situations, such as playing poker against the author of this book!
Acknowledgments
I’d like to thank all the contributors to this book, both those who are listed in the "Contributors" section
and those who helped with ideas, reviewed the manuscript, and provided suggestions of sources and
resources. Thanks in this capacity especially go to Tim Langdon, neon bender, whose gift of Harry
Blackstone, Jr.’s paperback book There’s One Born Every Minute (Jove Publications) provided great
inspiration for many of the hacks herein.
I’d like to thank my editor, Brian Sawyer, who shepherded this project with a strong hand and a strong
vision of what is and is not a hack. He was right most of the time. (Though not all the time, Brian. That
hack about using a monkey to pick the winner of the Kentucky Derby should have made it in. Maybe next
time....) Brian was instrumental in bringing this project to completion, especially during a string of
unlucky rolls where the odds of success looked slim.
I’d like to thank Neil Salkind, statistics writer supreme, for his help with many facets of my professional
life and this book.
Most importantly, thanks to Bonnie Johnson, my sweet wife, whom I vaguely recall, but who I think will
be waiting for me at home when I finally turn in the last revision of this book.
Preface
Chance plays a huge part in your life, whether you know it or not. Your particular genetic makeup mutated
slightly when you were created, and it did so based on specific laws of probability. Performance in school
involves human errors, yours and others’, which tends to keep your actual ability level from being
reflected precisely in your report card or on those high-stakes tests. Research on careers even suggests that
what you do for a living was probably not a result of careful planning and preparation, but more likely due
to happenstance. And, of course, chance determines your fate in games of chance and plays a large role in
the outcome of sporting events.
Fortunately, an entire set of scientific tools, the various applications of statistics, can be used to solve the
problems caused by our fate-influenced system. Inferential statistics, a field of science based entirely on
the nature of probability, allows us to understand the way things work, discover relationships among
variables, describe a huge population by seeing just a small bit of it, make uncannily accurate predictions,
and, yes, even make a little money with a well-placed wager here and there.
This book is a collection of statistical tricks and tools. Statistics Hacks presents useful tools from statistics,
of course, but also from the realms of educational and psychological measurement and experimental
research design. It provides solutions to a variety of problems in the world of social science, but also in the
worlds of business, games, and gambling.
If you are already a top scientist and do statistical calculations in your sleep, you’ll enjoy this book and the
creative applications it finds for those rusty old tools you know so well. If you just like the scientific
approach to life and are entertained by cool ideas and clever solutions to interesting problems, don’t
worry. Statistics Hacks was written with the nonscientist in mind, too, so if that is you, you’ve come to the
right place. It’s written for the nonstatistician as well, so if this still describes you, you’ll feel safe here.
If, on the other hand, you are taking a statistics course or have some interest in the academic nature of the
topic, you might find this book a pleasant companion to the textbooks typically required for those sorts of
courses. There won’t be any contradictions between your textbook and this book, so hearing about
real-world applications of statistical tools that seem only theoretical won’t hurt your development. It’s just
that there are some pretty cool things that you can do with statistics that seem more like fun than like
work.
Why Statistics Hacks?
The term hacking has a bad reputation in the press. They use it to refer to people who break into systems
or wreak havoc, using computers as their weapon. Among people who write code, though, the term hack
refers to a "quick-and-dirty" solution to a problem or a clever way to get something done. And the term
hacker is taken very much as a compliment, referring to someone as being creative, having the technical
chops to get things done. The Hacks series is an attempt to reclaim the word, document the good ways
people are hacking, and pass the hacker ethic of creative participation on to the uninitiated. Seeing how
others approach systems and problems is often the quickest way to learn about a new technology.