Scaling Collaborative Open Data Science Micah J. Smith

141 Pages·2017·3.37 MB·English

Scaling Collaborative Open Data Science
by
Micah J. Smith
B.A., Columbia University (2014)
Submitted to the Department of Electrical Engineering and Computer
Science
in partial fulfillment of the requirements for the degree of
Master of Science in Computer Science
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
June 2018
© Massachusetts Institute of Technology 2018. All rights reserved.
Author ................................................................
Department of Electrical Engineering and Computer Science
May 23, 2018
Certified by............................................................
Kalyan Veeramachaneni
Principal Research Scientist
Laboratory for Information and Decision Systems
Thesis Supervisor
Accepted by ...........................................................
Leslie A. Kolodziejski
Professor of Electrical Engineering and Computer Science
Chair, Department Committee on Graduate Students
2
Scaling Collaborative Open Data Science
by
Micah J. Smith
Submitted to the Department of Electrical Engineering and Computer Science
on May 23, 2018, in partial fulfillment of the
requirements for the degree of
Master of Science in Computer Science
Abstract
Large-scale, collaborative, open data science projects have the potential to address
important societal problems using the tools of predictive machine learning. However,
no suitable framework exists to develop such projects collaboratively and openly, at
scale. In this thesis, I discuss the deficiencies of current approaches and then develop
new approaches for this problem through systems, algorithms, and interfaces. A cen-
tral theme is the restructuring of data science projects into scalable, fundamental
units of contribution. I focus on feature engineering, structuring contributions as the
creation of independent units of feature function source code. This then facilitates
the integration of many submissions by diverse collaborators into a single, unified,
machine learning model, where contributions can be rigorously validated and verified
to ensure reproducibility and trustworthiness. I validate this concept by designing
and implementing a cloud-based collaborative feature engineering platform, Feature-
Hub, as well as an associated discussion platform for real-time collaboration. The
platform is validated through an extensive user study and modeling performance is
benchmarked against data science competition results. In the process, I also collect
and analyze a novel data set on the feature engineering source code submitted by
crowd data scientist workers of varying backgrounds around the world. Within this
context, I discuss paths forward for collaborative data science.
Thesis Supervisor: Kalyan Veeramachaneni
Title: Principal Research Scientist, Laboratory for Information and Decision Systems
3
4
Acknowledgments
I’d like to thank my advisor, Kalyan Veeramachaneni, for invaluable mentorship,
feedback, resources, and support. I’d like to thank all of the members of the Data
To AI Lab, both past and present, who have contributed feedback and valuable
discussions. RoyWedgeandMaxKanterhavebeengreatcollaborators. I’dalsoliketo
thank the members of our FeatureHub platform user study for their participation and
feedback. Finally, I’d like to thank all of my friends and family who have supported
me in my studies.
5
6
Contents
1 Introduction 19
1.1 Motivating open data science . . . . . . . . . . . . . . . . . . . . . . 21
1.2 The modern data science process . . . . . . . . . . . . . . . . . . . . 23
1.3 The state of collaboration . . . . . . . . . . . . . . . . . . . . . . . . 27
1.3.1 Shared notebooks . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.3.2 Vanilla software engineering . . . . . . . . . . . . . . . . . . . 30
1.3.3 Hosted platforms . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.3.4 Data science competitions . . . . . . . . . . . . . . . . . . . . 32
1.3.5 Challenges of open data science . . . . . . . . . . . . . . . . . 34
1.4 Data science abstractions . . . . . . . . . . . . . . . . . . . . . . . . . 35
2 Background and Related Work 39
2.1 Crowdsourcing methods in machine learning . . . . . . . . . . . . . . 39
2.1.1 Crowdsourced data labeling . . . . . . . . . . . . . . . . . . . 39
2.1.2 Crowdsourced data cleaning . . . . . . . . . . . . . . . . . . . 40
2.1.3 Crowdsourced feature engineering . . . . . . . . . . . . . . . . 40
2.2 Automated methods in machine learning . . . . . . . . . . . . . . . . 40
2.2.1 Automated machine learning . . . . . . . . . . . . . . . . . . . 41
2.2.2 Automated feature engineering . . . . . . . . . . . . . . . . . 42
2.3 Programming and data science competitions . . . . . . . . . . . . . . 42
2.4 Collaborative software development . . . . . . . . . . . . . . . . . . . 44
7
3 A collaborative data science platform 45
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.2 Launching a project . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3 Creating new features . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.3.1 Load and explore . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.3.2 Write features . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.3.3 Evaluate and submit . . . . . . . . . . . . . . . . . . . . . . . 53
3.3.4 Collaborate . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.4 Combining contributions . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.5 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4 Assessment 61
4.1 Experimental conditions . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.1.1 Study participants . . . . . . . . . . . . . . . . . . . . . . . . 62
4.1.2 Prediction problems . . . . . . . . . . . . . . . . . . . . . . . 63
4.1.3 Experiment groups . . . . . . . . . . . . . . . . . . . . . . . . 63
4.1.4 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . 64
4.2 Modeling performance . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.3 Evaluating collaboration . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.4 Analysis of contributors . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.4.1 Worker attributes and output . . . . . . . . . . . . . . . . . . 72
4.4.2 Contributions in source code . . . . . . . . . . . . . . . . . . . 74
4.4.3 Educational opportunities . . . . . . . . . . . . . . . . . . . . 74
5 Discussion 85
5.1 Opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.1.1 Flexible data science abstractions . . . . . . . . . . . . . . . . 85
5.1.2 Automated methods to augment collaboration . . . . . . . . . 87
5.1.3 Creating interpretable models . . . . . . . . . . . . . . . . . . 89
5.2 Challenges we faced . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
5.2.1 Maintaining platforms . . . . . . . . . . . . . . . . . . . . . . 90
8
5.2.2 Administering user studies . . . . . . . . . . . . . . . . . . . . 92
5.2.3 Motivating contributions . . . . . . . . . . . . . . . . . . . . . 93
5.2.4 Adversarial behavior . . . . . . . . . . . . . . . . . . . . . . . 95
6 The future of collaborative data science 97
6.1 The problem with platforms . . . . . . . . . . . . . . . . . . . . . . . 97
6.2 A turn towards platformless collaboration . . . . . . . . . . . . . . . 99
6.3 Lightweight collaborative systems . . . . . . . . . . . . . . . . . . . . 100
A Implementation of FeatureHub 109
A.1 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
A.2 User interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
A.3 Feature evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
A.4 Discussion forum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
A.5 Automated machine learning . . . . . . . . . . . . . . . . . . . . . . . 114
A.6 Data management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
A.7 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
B Prediction problem details 119
B.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
B.2 Data schemata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
B.3 Pre-extracted features . . . . . . . . . . . . . . . . . . . . . . . . . . 123
C User study survey 127
C.1 Survey questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
C.1.1 Getting oriented . . . . . . . . . . . . . . . . . . . . . . . . . 128
C.1.2 FeatureHub Platform . . . . . . . . . . . . . . . . . . . . . . . 128
C.1.3 Forum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
C.1.4 Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
C.2 Survey responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
D Listings 135
9
10