The Art of Data Science: A Guide for Anyone Who Works with Data PDF
Preview The Art of Data Science: A Guide for Anyone Who Works with Data
The Art of Data Science A Guide for Anyone Who Works with Data Roger D. Peng and Elizabeth Matsui Thisbookisforsaleat http://leanpub.com/artofdatascience Thisversionwaspublishedon2015-09-04 ThisisaLeanpubbook.Leanpubempowersauthorsand publisherswiththeLeanPublishingprocess.Lean Publishingistheactofpublishinganin-progressebook usinglightweighttoolsandmanyiterationstogetreader feedback,pivotuntilyouhavetherightbookandbuild tractiononceyoudo. ©2015RogerD.PengandElizabethMatsui Also By Roger D. Peng RProgrammingforDataScience ExploratoryDataAnalysiswithR ReportWritingforDataScienceinR Special thanks to Maggie Matsui, who created all of the artwork forthisbook. Contents 1. DataAnalysisasArt . . . . . . . . . . . . . . . . . 1 2. EpicyclesofAnalysis . . . . . . . . . . . . . . . . . 4 3. StatingandRefiningtheQuestion . . . . . . . . 16 4. ExploratoryDataAnalysis . . . . . . . . . . . . . 31 5. UsingModelstoExploreYourData . . . . . . . 55 6. Inference:APrimer . . . . . . . . . . . . . . . . . 78 7. FormalModeling . . . . . . . . . . . . . . . . . . . 92 8. Inferencevs.Prediction:ImplicationsforMod- elingStrategy . . . . . . . . . . . . . . . . . . . . . 112 9. InterpretingYourResults. . . . . . . . . . . . . . 124 10. Communication . . . . . . . . . . . . . . . . . . . 144 11. ConcludingThoughts . . . . . . . . . . . . . . . . 153 1. Data Analysis as Art Data analysis is hard, and part of the problem is that few people can explain how to do it. It’s not that there aren’t any people doing data analysis on a regular basis. It’s that thepeoplewhoarereallygoodatithaveyettoenlightenus aboutthethoughtprocessthatgoesonintheirheads. Imagine you were to ask a songwriter how she writes her songs.Therearemanytoolsuponwhichshecandraw.We have a general understanding of how a good song should be structured: how long it should be, how many verses, maybe there’s a verse followed by a chorus, etc. In other words, there’s an abstract framework for songs in general. Similarly, we have music theory that tells us that certain combinations of notes and chords work well together and other combinations don’t sound good. As good as these toolsmightbe,ultimately,knowledgeofsongstructureand music theory alone doesn’t make for a good song. Some- thingelseisneeded. InDonaldKnuth’slegendary1974essayComputerProgram- mingasanArt1,Knuthtalksaboutthedifferencebetweenart andscience.Inthatessay,hewastryingtogetacrosstheidea that although computer programming involved complex machinesandverytechnicalknowledge,theactofwritinga computerprogramhadanartisticcomponent.Inthisessay, hesaysthat Science is knowledge which we understand so wellthatwecanteachittoacomputer. 1http://www.paulgraham.com/knuth.html DataAnalysisasArt 2 Everythingelseisart. At some point, the songwriter must inject a creative spark intotheprocesstobringallthesongwritingtoolstogether to make something that people want to listen to. This is a key part of the art of songwriting. That creative spark is difficult to describe, much less write down, but it’s clearly essentialtowritinggoodsongs.Ifitweren’t,thenwe’dhave computer programs regularly writing hit songs. For better orforworse,thathasn’thappenedyet. Much like songwriting (and computer programming, for thatmatter),it’simportanttorealizethatdata analysis is an art.Itisnotsomethingyetthatwecanteachtoacomputer. Data analysts have many tools at their disposal, from linear regression to classification trees and even deep learning, andthesetoolshaveallbeencarefullytaughttocomputers. But ultimately, a data analyst must find a way to assemble allofthetoolsandapplythemtodatatoanswerarelevant question—aquestionofinteresttopeople. Unfortunately, the process of data analysis is not one that we have been able to write down effectively. It’s true that there are many statistics textbooks out there, many lining our own shelves. But in our opinion, none of these really addresses the core problems involved in conducting real- world data analyses. In 1991, Daryl Pregibon, a promi- nent statistician previously of AT&T Research and now of Google, said in reference to the process of data analysis2 that “statisticians have a process that they espouse but do notfullyunderstand”. Describing data analysis presents a difficult conundrum. On the one hand, developing a useful framework involves characterizingtheelementsofadataanalysisusingabstract 2http://www.nap.edu/catalog/1910/the-future-of-statistical-software- proceedings-of-a-forum DataAnalysisasArt 3 languageinordertofindthecommonalitiesacrossdifferent kindsofanalyses.Sometimes,thislanguageisthelanguage of mathematics. On the other hand, it is often the very details of an analysis that makes each one so difficult and yet interesting. How can one effectively generalize across many different data analyses, each of which has important uniqueaspects? What we have set out to do in this book is to write down the process of data analysis. What we describe is not a specific “formula” for data analysis—something like “ap- ply this method and then run that test”— but rather is a general process that can be applied in a variety of situ- ations. Through our extensive experience both managing data analysts and conducting our own data analyses, we havecarefullyobservedwhatproducescoherentresultsand whatfailstoproduceusefulinsightsintodata.Ourgoalisto write down what we have learned in the hopes that others mayfindituseful. 2. Epicycles of Analysis To the uninitiated, a data analysis may appear to follow a linear, one-step-after-the-other process which at the end, arrives at a nicely packaged and coherent result. In reality, data analysis is a highly iterative and non-linear process, betterreflectedbyaseriesofepicycles(seeFigure),inwhich information is learned at each step, which then informs whether (and how) to refine, and redo, the step that was justperformed,orwhether(andhow)toproceedtothenext step. An epicycle is a small circle whose center moves around the circumference of a larger circle. In data analysis, the iterative process that is applied to all steps of the data analysis can be conceived of as an epicycle that is repeated for each step along the circumference of the entire data analysisprocess.Somedataanalysesappeartobefixedand linear, such as algorithms embedded into various software platforms, including apps. However, these algorithms are finaldataanalysisproductsthathaveemergedfromthevery non-linearworkofdevelopingandrefiningadataanalysis sothatitcanbe“algorithmized.” EpicyclesofAnalysis 5 EpicyclesofAnalysis 2.1 Setting the Scene Before diving into the “epicycle of analysis,” it’s helpful to pause and consider what we mean by a “data analy- sis.” Although many of the concepts we will discuss in this
The list of books you might like
