Table Of Content

Statistics For Big Data For Dummies® Published by: John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030-5774, www.wiley.com Copyright © 2015 by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the Publisher. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/permissions. Trademarks: Wiley, For Dummies, the Dummies Man logo, Dummies.com, Making Everything Easier, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc., and may not be used without written permission. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc., is not associated with any product or vendor mentioned in this book. LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: WHILE THE PUBLISHER AND AUTHOR HAVE USED THEIR BEST EFFORTS IN PREPARING THIS BOOK, THEY MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS BOOK AND SPECIFICALLY DISCLAIM ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES REPRESENTATIVES OR WRITTEN SALES MATERIALS. THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR YOUR SITUATION. YOU SHOULD CONSULT WITH A PROFESSIONAL WHERE APPROPRIATE. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. For general information on our other products and services, please contact our Customer Care Department within the U.S. at 877-762-2974, outside the U.S. at 317- 572-3993, or fax 317-572-4002. For technical support, please visit www.wiley.com/techsupport. Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of this book may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the version you purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com. Library of Congress Control Number: 2015943222 ISBN 978-1-118-94001-3 (pbk); ISBN 978-1-118-94002-0 (ePub); ISBN 978-1-118- 94003-7 (ePDF) Statistics For Big Data For Dummies Visit http://www.dummies.com/cheatsheet/statisticsforbigdata to view this book’s cheat sheet. Table of Contents Cover Introduction About This Book Foolish Assumptions Icons Used in This Book Beyond the Book Where to Go From Here Part I: Introducing Big Data Statistics Chapter 1: What Is Big Data and What Do You Do with It? Characteristics of Big Data Exploratory Data Analysis (EDA) Statistical Analysis of Big Data Chapter 2: Characteristics of Big Data: The Three Vs Characteristics of Big Data Traditional Database Management Systems (DBMS) Chapter 3: Using Big Data: The Hot Applications Big Data and Weather Forecasting Big Data and Healthcare Services Big Data and Insurance Big Data and Finance Big Data and Electric Utilities Big Data and Higher Education Big Data and Retailers Big Data and Search Engines Big Data and Social Media Chapter 4: Understanding Probabilities The Core Structure: Probability Spaces Discrete Probability Distributions Continuous Probability Distributions Introducing Multivariate Probability Distributions Chapter 5: Basic Statistical Ideas Some Preliminaries Regarding Data Summary Statistical Measures Overview of Hypothesis Testing Higher-Order Measures Part II: Preparing and Cleaning Data Chapter 6: Dirty Work: Preparing Your Data for Analysis Passing the Eye Test: Does Your Data Look Correct? Being Careful with Dates Does the Data Make Sense? Frequently Encountered Data Headaches Other Common Data Transformations Chapter 7: Figuring the Format: Important Computer File Formats Spreadsheet Formats Database Formats Chapter 8: Checking Assumptions: Testing for Normality Goodness of fit test Jarque-Bera test Chapter 9: Dealing with Missing or Incomplete Data Missing Data: What’s the Problem? Techniques for Dealing with Missing Data Chapter 10: Sending Out a Posse: Searching for Outliers Testing for Outliers Robust Statistics Dealing with Outliers Part III: Exploratory Data Analysis (EDA) Chapter 11: An Overview of Exploratory Data Analysis (EDA) Graphical EDA Techniques EDA Techniques for Testing Assumptions Quantitative EDA Techniques Chapter 12: A Plot to Get Graphical: Graphical Techniques Stem-and-Leaf Plots Scatter Plots Box Plots Histograms Quantile-Quantile (QQ) Plots Autocorrelation Plots Chapter 13: You’re the Only Variable for Me: Univariate Statistical Techniques Counting Events Over a Time Interval: The Poisson Distribution Continuous Probability Distributions Chapter 14: To All the Variables We’ve Encountered: Multivariate Statistical Techniques Testing Hypotheses about Two Population Means Using Analysis of Variance (ANOVA) to Test Hypotheses about Population Means The F-Distribution F-Test for the Equality of Two Population Variances Correlation Chapter 15: Regression Analysis The Fundamental Assumption: Variables Have a Linear Relationship Defining the Population Regression Equation Estimating the Population Regression Equation Testing the Estimated Regression Equation Using Statistical Software Assumptions of Simple Linear Regression Multiple Regression Analysis Multicollinearity Chapter 16: When You’ve Got the Time: Time Series Analysis Key Properties of a Time Series Forecasting with Decomposition Methods Smoothing Techniques Seasonal Components Modeling a Time Series with Regression Analysis Comparing Different Models: MAD and MSE Part IV: Big Data Applications Chapter 17: Using Your Crystal Ball: Forecasting with Big Data ARIMA Modeling Simulation Techniques Chapter 18: Crunching Numbers: Performing Statistical Analysis on Your Computer Excelling at Excel Programming with Visual Basic for Applications (VBA) R, Matey! Chapter 19: Seeking Free Sources of Financial Data Yahoo! Finance Federal Reserve Economic Data (FRED) Board of Governors of the Federal Reserve System U.S. Department of the Treasury Other Useful Financial Websites Part V: The Part of Tens Chapter 20: Ten (or So) Best Practices in Data Preparation Check Data Formats Verify Data Types Graph Your Data Verify Data Accuracy Identify Outliers Deal with Missing Values Check Your Assumptions about How the Data Is Distributed Back Up and Document Everything You Do Chapter 21: Ten (or So) Questions Answered by Exploratory Data Analysis (EDA) What Are the Key Properties of a Dataset? What’s the Center of the Data? How Much Spread Is There in the Data? Is the Data Skewed? What Distribution Does the Data Follow? Are the Elements in the Dataset Uncorrelated? Does the Center of the Dataset Change Over Time? Does the Spread of the Dataset Change Over Time? Are There Outliers in the Data? Does the Data Conform to Our Assumptions? About the Authors Cheat Sheet Advertisement Page Connect with Dummies End User License Agreement Introduction Welcome to Statistics For Big Data For Dummies! Every day, what has come to be known as big data is making its influence felt in our lives. Some of the most useful innovations of the past 20 years have been made possible by the advent of massive data-gathering capabilities combined with rapidly improving computer technology. For example, of course, we have become accustomed to finding almost any information we need through the Internet. You can locate nearly anything under the sun immediately by using a search engine such as Google or DuckDuckGo. Finding information this way has become so commonplace that Google has slowly become a verb, as in “I don’t know where to find that restaurant — I’ll just Google it.” Just think how much more efficient our lives have become as a result of search engines. But how does Google work? Google couldn’t exist without the ability to process massive quantities of information at an extremely rapid speed, and its software has to be extremely efficient. Another area that has changed our lives forever is e-commerce, of which the classic example is Amazon.com. People can buy virtually every product they use in their daily lives online (and have it delivered promptly, too). Often online prices are lower than in traditional “brick-and-mortar” stores, and the range of choices is wider. Online shopping also lets people find the best available items at the lowest possible prices. Another huge advantage to online shopping is the ability of the sellers to provide reviews of products and recommendations for future purchases. Reviews from other shoppers can give extremely important information that isn’t available from a simple product description provided by manufacturers. And recommendations for future purchases are a great way for consumers to find new products that they might not otherwise have known about. Recommendations are enabled by one application of big data — the use of highly sophisticated programs that analyze shopping data and identify items that tend to be purchased by the same consumers. Although online shopping is now second nature for many consumers, the reality is that e-commerce has only come into its own in the last 15–20 years, largely thanks to the rise of big data. A website such as Amazon.com must process quantities of information that would have been unthinkably gigantic just a few years ago, and that processing must be done quickly and efficiently. Thanks to rapidly improving technology, many traditional retailers now also offer the option of making purchases online; failure to do so would put a retailer at a huge competitive disadvantage. In addition to search engines and e-commerce, big data is making a major impact in a surprising number of other areas that affect our daily lives: Social media Online auction sites Insurance Healthcare Energy Political polling Weather forecasting Education Travel Finance