Massively Collaborative Machine Learning

31 Pages·2016·1.16 MB·English

Cover Page

The handle http://hdl.handle.net/1887/44814 holds various files of this Leiden University
dissertation

Author: Rijn, Jan van
Title: Massively collaborative machine learning
Issue Date: 2016-12-19
4
Experiment Databases
Manyfieldsofsciencehavemadesignificantbreakthroughsbyadoptingonlinetools
thathelporganize,structureandmineinformationthatistoodetailedtobeprintedin
journals.Inthischapter,weintroduceOpenML,anonlineplatformformachinelearn-
ingresearcherstoshareandorganizedatainfinedetail,sothattheycanworkmore
effectively,bemorevisible,andcollaboratewithotherstotackleharderproblems.We
discuss some important concepts upon which it is built and showcase various types
of Machine Learning studies that can be conducted using information from previous
experiments.
4.1 Introduction
WhenGalileoGalileidiscoveredtheringsofSaturn,hedidnotwriteascientificpaper.
Instead,hewrotehisdiscoverydown,jumbledthelettersintoananagram,andsent
ittohisfellowastronomers.Thiswascommonpracticeamongrespectedscientistsof
theage,includingLeonardo,HuygensandHooke.
The reason was not technological. The printing press was well in use those days
and the first scientific journals already existed. Rather, there was little personal gain
in letting your rivals know what you were doing. The anagrams ensured that the
original discoverer alone could build on his ideas, at least until someone else made
the same discovery and the solution to the anagram had to be published in order to
claimpriority.
Thisbehaviourchangedgraduallyinthelate17thcentury.MembersoftheRoyal
Society realized that this secrecy was holding them back, and that if they all agreed
topublishtheirfindingsopenly,theywouldalldobetter[79].Underthemotto“take
42 4.2. Networkedscience
nobody’s word for it”, they established that scientists could only claim a discovery
if they published it first, if they detailed their experimental methods so that results
couldbeverified,andiftheyexplicitlygavecredittoallpriorworktheybuiltupon.
Moreover,wealthypatronsandgovernmentsincreasinglyfundedscienceasapro-
fession,andrequiredthatfindingsbepublishedinjournals,thusmaximallybenefiting
thepublic,aswellasthepublicimageofthepatrons.Thiseffectivelycreatedaneco-
nomybasedonreputation[79,96].Bypublishingtheirfindings,scientistswereseen
as trustworthy by their peers and patrons, which in turn led to better collaboration,
research funding, and scientific jobs. This new culture continues to this day and has
createdabodyofsharedknowledgethatisthebasisformuchofhumanprogress.
Today,however,theubiquityoftheInternetisallowingnew,morescalableforms
ofscientificcollaboration.Wecannowsharedetailedobservationsandmethods(data
and code) far beyond what can be printed in journals, and interact in real time with
manypeopleatonce,allovertheworld.
Asaresult,manyscienceshaveturnedtoonlinetoolstoshare,structureandana-
lysescientificdataonaglobalscale.Suchnetworkedscienceisdramaticallyspeeding
up discovery because scientists are now able to build directly on each other’s ob-
servations and techniques, reuse them in unforeseen ways, mine all collected data
for patterns, and scale up collaborations to tackle much harder problems. Whereas
the journal system still serves as our collective long-term memory, the Internet in-
creasingly serves as our collective short-term working memory [97], collecting data
and code far too extensive and detailed to be comprehended by a single person, but
instead(re)usedbymanytodrivemuchofmodernscience.
Manychallengesremain,however.Inthespiritofthejournalsystem,theseonline
tools must also ensure that shared data is trustworthy so that others can build on it,
andthatitisinindividualscientists’bestinteresttosharetheirdataandideas.Inthis
chapter,weintroduceOpenML,acollaborationplatformthroughwhichscientistscan
automatically share, organize and discuss machine learning experiments, data, and
algorithms.
4.2 Networked science
Networked science tools are changing the way we make discoveries in several ways.
They allow hundreds of scientists to discuss complex ideas online, they structure in-
formationfrommanyscientistsintoacoherentwhole,andallowanyonetoreuseall
collecteddatainnewandunexpectedways.
Chapter4. ExperimentDatabases 43
4.2.1 Designing networked science
Nielsen [97] reviews many examples of networked science, and explains their suc-
cessesbythefactthat,throughtheinteractionofmanyminds,thereisagoodchance
thatsomeonehasjusttherightexpertisetocontributeatjusttherighttime:
Designedserendipity Because many scientists have complementary expertise, any
sharedidea,question,observation,ortoolmaybenoticedbysomeonewhohas
justtheright(micro)expertisetosparknewideas,answerquestions,reinterpret
observations, or reuse data and tools in unexpected new ways. By scaling up
collaborations,such‘happyaccidents’becomeevermorelikelyandfrequent.
Dynamicdivisionoflabour Because each scientist is especially adept at certain re-
search tasks, such as generating ideas, collecting data, mining data, or inter-
preting results, any seemingly hard task may be routine for someone with just
the right skills, or the necessary time or resources to do so. This dramatically
speedsupprogress.
Designedserendipityandadynamicdivisionoflabouroccurnaturallywhenideas,
questions,data,ortoolsarebroadcasttoalargegroupofpeopleinawaythatallows
everyone in the collaboration to discover what interests them, and react to it easily
and creatively. As such, for online collaborations to scale, online tools must make it
practical for anybody to join and contribute any amount at any time. This can be
expressedinthefollowing‘designpatterns’[97]:
• Encourage small contributions, allowing scientists to contribute in (quasi) real
time.Thisallowsmanyscientiststocontribute,increasingthecognitivediversity
andrangeofavailableexpertise.
• Split up complex tasks into many small subtasks that can be attacked (nearly)
independently. This allows many scientists to contribute individually and ac-
cordingtotheirexpertise.
• Construct a rich and structured information commons, so that people can effi-
ciently build on prior knowledge. It should be easy to find previous work, and
easytocontributenewworktotheexistingbodyofknowledge.
• Humanattentiondoesnotscaleinfinitely.Scientistsonlyhavealimitedamount
ofattentiontodevotetothecollaboration,andshouldthusbeabletofocuson
theirinterestsandfilteroutirrelevantcontributions.
• Establish accepted methods for participants to interact and resolve disputes.
This can be an ‘honour code’ that encourages respectable and respectful beha-
44 4.3. Machinelearning
viour, deters academic dishonesty, and protects the contributions of individual
scientists.
Still, even if scientists have the right expertise or skill to contribute at the right
time,theytypicallyalsoneedtherightincentivetodoso.
This problem was actually solved already centuries ago by establishing a reputa-
tion system implemented using the best medium for sharing information of the day,
the journal. Today, the internet and networked science tools provide a much more
powerfulmedium,buttheyalsoneedtomakesurethatsharingdata,codeandideas
onlineisinscientists’bestinterest.
The key to do this seems to lie in extending the reputation system [97]. Online
tools should allow everyone to see exactly who contributed what, and link valuable
contributions to increased esteem amongst the users of the tools and the scientific
communityatlarge.Thetraditionalapproachtodothisistolinkusefulonlinecontri-
butionstoauthorshipinensuingpapers,ortolinkthereuseofshareddatatocitation
ofassociatedpapersorDOI’s.Scientificcommunitiesaretypicallysmallworlds,thus
scientistsareencouragedtorespectsuchagreements.
Moreover, beyond bibliographic measures, online tools can define new measures
todemonstratethescientific(andsocietal)impactofcontributions.Thesearesome-
timescalledaltmetricsorarticle-levelmetrics[112].AninterestingexampleisArXiv,
anonlinearchiveofpreprints(unpublishedmanuscripts)withitsownreferencetrack-
ingsystem(SPIRES).Inphysics,preprintsthatarereferencedmanytimeshaveahigh
statusamongphysicists.Theyareaddedtoresumesandusedtoevaluatecandidates
for scientificjobs. Thisillustrates thatwhat getsmeasured, gets rewarded,and what
getsrewarded,getsdone[97,100].Ifscholarlytoolsdefineusefulnewmeasuresand
trackthemaccurately,scientistswillusethemtoassesstheirpeers.
4.3 Machine learning
Machine learning is a field where a more networked approach would be particularly
valuable. Machine learning studies typically involve large datasets, complex code,
large-scaleevaluationsandcomplexmodels,noneofwhichcanbeadequatelyrepres-
ented in papers. Still, most work is only published in papers, in highly summarized
formssuchastables,graphsandpseudo-code.Oddlyenough,whilemachinelearning
has proven so crucial in analysing large amounts of observations collected by other
scientists, the outputs of machine learning research are typically not collected and
organizedinanywaythatallowsotherstoreuse,reinterpret,orminetheseresultsto
learnnewthings,e.g.,whichtechniquesaremostusefulinagivenapplication.
Chapter4. ExperimentDatabases 45
4.3.1 Reusability and reproducibility
This makes us duplicate a lot of effort, and ultimately slows down the whole field
of machine learning [63, 153]. Indeed, without prior experiments to build on, each
study has to start from scratch and has to rerun many experiments. This limits the
depthofstudiesandtheinterpretabilityandgeneralizabilityoftheirresults[4,63].It
has been shown that studies regularly contradict each other because they are biased
toward different datasets [80], or because they do not take into account the effects
ofdatasetsize,parameteroptimizationandfeatureselection[71,105].Thismakesit
veryhard,especiallyforotherresearchers,tocorrectlyinterprettheresults.Moreover,
itisoftennotevenpossibletorerunexperimentsbecausecodeanddataaremissing,
orbecausespacerestrictionsimposedonpublicationsmakeitpracticallyinfeasibleto
publish many details of the experiment setup. This lack of reproducibility has been
warned against repeatedly [80, 102, 142], and has been highlighted as one of the
mostimportantchallengesindataminingresearch[67].
4.3.2 Prior work
Manymachinelearningresearchersarewellawareoftheseissues,andhaveworked
toalleviatethem.Toimprovereproducibility,thereexistrepositoriestopubliclyshare
benchmarking datasets, such as UCI [90]. Moreover, software can be shared on the
MLOSSwebsite.TherealsoexistsanopensourcesoftwaretrackintheJournalforMa-
chineLearningResearch(JMLR)whereshortdescriptionsofusefulmachinelearning
softwarecanbesubmitted.Also,somemajorconferenceshavestartedcheckingsub-
missions for reproducibility [91], or issue open science awards for submissions that
arereproducible(e.g.,ECML/PKDD2013).
Moreover, there also exist experiment repositories. First, meta-learning projects
such as StatLog [93] and MetaL [22], and benchmarking services such as MLcomp
runmanyalgorithmsonmanydatasetsontheirservers.Thismakesbenchmarkscom-
parable, and even allows the building of meta-models, but it does require the code
to be rewritten to run on their servers. Moreover, the results are not organized to be
easilyqueriedandreused.
Second,dataminingchallengeplatformssuchasKaggle[29]andTunedIT[162]
collect results obtained by different competitors. While they do scale and offer mon-
etary incentives, they are adversarial rather than collaborative. For instance, code is
typicallynotsharedduringacompetition.
Finally,theexperimentdatabaseformachinelearning[153]wasintroduced,which
organizes results from different users and makes them queryable through an online
interface. Unfortunately, it does not allow collaborations to scale easily. It requires
researchers to transcribe their experiments into XML, and only covers classification
46 4.4. OpenML
experiments.
Whileallthesetoolsareveryvaluableintheirownright,theydonotprovidemany
of the requirements for scalable collaboration discussed above. It can be quite hard
for scientists to contribute, there is often no online discussion, and they are heavily
focusedonbenchmarking,notonsharingotherresultssuchasmodels.Byintroducing
OpenML,weaimtoprovideacollaborativescienceplatformformachinelearning.
4.4 OpenML
OpenMLisanonlineplatformwheremachinelearningresearcherscanautomatically
share data in fine detail and organize it to work more effectively and collaborate on
aglobalscale[124,155,154].
Itallowsanyonetochallengethecommunitywithnewdatatoanalyse,andevery-
one able to mine that data to share their code and results (e.g., models, predictions,
and evaluations). In this sense, OpenML is similar to data mining challenge plat-
forms, except that it allows users to work collaboratively, building on each other’s
work.OpenMLmakessurethateach(sub)taskisclearlydefined,andthatallshared
resultsarestoredandorganizedonlineforeasyaccess,reuseanddiscussion.
Moreover,itisbeingintegratedinpopulardataminingplatformssuchasWeka[61],
R [16, 148], Scikit-learn [28, 103], RapidMiner [151, 121], MOA [13] and Cort-
ana[92].Thismeansthatanyonecaneasilyimportthedataintothesetools,pickany
algorithm or workflow to run, and automatically share all obtained results. Results
are being produced locally: everyone that participates can run experiments on his
own computers and share the results on OpenML. The web-interface provides easy
accesstoallcollecteddataandcode,comparesallresultsobtainedonthesamedata
oralgorithms,buildsdatavisualizations,andsupportsonlinediscussions.Finally,itis
anopensourceproject,invitingscientiststoextenditinwaysmostusefultothem.
OpenMLoffersvariousservicestoshareandfinddatasets,todownloadorcreate
scientifictasks,toshareandfindalgorithms(calledflows),andtoshareandorganize
results.TheseservicesareavailablethroughtheOpenMLwebsite,aswellasthrough
aRESTAPIforintegrationwithsoftwaretools.
4.4.1 Datasets
Anyonecanprovidethecommunitywithnewdatasetstoanalyse.Tobeabletoana-
lysethedata,OpenMLacceptsalimitednumberofformats.Forinstance,currentlyit
requirestheARFFformat[61]fortabulardata,althoughmoreformatswillbeadded
overtime.
Chapter4. ExperimentDatabases 47
Table4.1:StandardMeta-features.
Category Meta-features
Simple # Instances, # Attributes, # Classes, Dimensionality, Default Ac-
curacy, # Observations with Missing Values, # Missing Values,
%ObservationsWithMissingValues,%MissingValues,#Numeric
Attributes,#NominalAttributes,#BinaryAttributes,%Numeric
Attributes, % Nominal Attributes, % Binary Attributes, Majority
ClassSize,%MajorityClass,MinorityClassSize,%MinorityClass
Statistical Mean of Means of Numeric Attributes, Mean Standard Deviation
ofNumericAttributes,MeanKurtosisofNumericAttributes,Mean
SkewnessofNumericAttributes
InformationTheoretic ClassEntropy,MeanAttributeEntropy,MeanMutualInformation,
EquivalentNumberOfAttributes,NoisetoSignalRatio
Landmarkers[108] Accuracy of Decision Stump, Kappa of Decision Stump, Area un-
der the ROC Curve of Decision Stump, Accuracy of Naive Bayes,
KappaofNaiveBayes,AreaundertheROCCurveofNaiveBayes,
Accuracy of k-NN, Kappa of k-NN, Area under the ROC Curve of
k-NN,...
The data can either be uploaded or referenced by a URL. This URL may be a
landing page with further information or terms of use, or it may be an API call to
large repositories of scientific data such as the SDSS [146]. In some cases, such as
Twitter feeds, data may be dynamic, which means that results won’t be repeatable.
However,insuchtasks,repeatabilityisnotexpected.OpenMLwillautomaticallyver-
sioneachnewlyaddeddataset.Optionally,auser-definedversionnamecanbeadded
for reference. Next, authors can state how the data should be attributed, and which
(creativecommons)licencetheywishtoattachtoit.Authorscanalsoaddareference
forcitation,andalinktoapaper.Finally,extrainformationcanbeadded,suchasthe
(default) target attribute(s) in labelled data, or the row-id attribute for data where
instancesarenamed.
For known data formats, OpenML will then compute an array of data character-
istics, also called meta-features. Typical meta-features are often categorized as either
simple,statistical,informationtheoreticorlandmarkers.Table4.1showssomemeta-
featurescomputedbyOpenML.
OpenML indexes all datasets and allows them to be searched through a standard
keyword search and search filters. Each dataset has its own page with all known in-
formation. This includes the general description, attribution information, and data
characteristics, but also statistics of the data distribution and, and all scientific tasks
definedonthisdata(seebelow).Italsoincludesadiscussionsectionwherethedata-
48 4.4. OpenML
setandresultscanbediscussed.
4.4.2 Task types
Adatasetalonedoesnotconstituteascientifictask.Wemustfirstagreeonwhattypes
ofresultsareexpectedtobeshared.Thisisexpressedintasktypes:theydefinewhat
typesofinputsaregiven,whichtypesofoutputareexpectedtobereturned,andwhat
scientific protocols should be used. For instance, classification tasks should include
well-definedcross-validationprocedures,labelledinputdata,andrequirepredictions
asoutputs.
OpenMLcoversthefollowingtasktypes:
SupervisedClassification Givenadatasetwithanominaltargetandasetoftrain/test
splits (e.g., generated by a cross-validation procedure) train a model and re-
turnthepredictionsofthatmodel.Theserverwillevaluatethese,andcompute
standard evaluation measures, such as predictive accuracy, f-measure and area
undertheROCcurve.
SupervisedRegression Givenadatasetwithanumerictargetandasetoftrain/test
splits (e.g., generated by a cross-validation procedure) train a model and re-
turnthepredictionsofthatmodel.Theserverwillevaluatethese,andcompute
standardevaluationmeasures,suchasrootmeansquarederror(RMSE),mean
absoluteerrorandrootrelativesquarederror.
LearningCurveAnalysis A variation of Supervised Classification. Given a dataset
with a nominal target, various data samples of increasing size are defined. A
modelisbuildforeachindividualdatasample.Foreachofthesesamples,vari-
ous evaluation measures are calculated; from these a learning curve can be
drawn.Chapter6willelaborateonthis.
DataStreamClassification Theonlineversionofclassification.Givenasequenceof
observations,buildamodelthatisabletoprocesstheseonebyoneandadapts
topossiblechangesintheinputspace.Chapter5willelaborateonthis.
MachineLearningChallenge AvariationofSupervisedClassification,similartothe
setup of Kaggle [29]. The user is presented with a partly labelled dataset. The
task is to label the unlabelled instances. As a result, there can be no cross-
validationprocedure,astherewillalwaysbeacompletelyhiddentestset.
SubgroupDiscovery Given a dataset, return a conjunction of rules that describes
a subgroup that is interesting with respect to a given quality measure. These
quality measures can be weighted relative accuracy, jaccard measure or chi-
squared.
Chapter4. ExperimentDatabases 49
Given inputs
sourcedata anneal(1) Dataset(required)
estimationprocedure 10-foldCross-validation EstimationProcedure
(required)
evaluationmeasures predictiveaccuracy String(optional)
targetfeature class String(required)
datasplits http://www.openml.org/apisplits/get/1/1/Task1splits.arff TrainTestSplits(hidden)
Expected outputs
model Afilecontainingthemodelbuiltonalltheinputdata. File(optional)
evaluations Alistofuser-definedevaluationsofthetaskaskey-valuepairs. KeyValue(optional)
predictions Anarfffilewiththepredictionsofamodel Predictions(required)
Figure4.1:ExampleofanOpenMLtaskdescription.
4.4.3 Tasks
If scientists want to perform, for instance, classification on a given dataset, they can
createanewmachinelearningtask.Tasksareinstantiationsoftasktypeswithspecific
inputs (e.g., datasets). Tasks are created once, and then downloaded and solved by
anyone.
AnexampleofsuchataskisshowninFigure4.1.Inthiscase,itisaclassification
taskdefinedondataset‘anneal’(version1).Nexttothedataset,thetaskincludesthe
target attribute, the evaluation procedure (here: 10-fold cross-validation) and a file
withthedatasplitsforcross-validation.Thelatterensuresthatresultsfromdifferent
researchers can be objectively compared. For researchers doing an (internal) hyper-
parameter optimization, it also states the evaluation measure to optimize for. The
required outputs for this task are the predictions for all test instances, and option-
ally, the models built and evaluations calculated by the user. However, OpenML will
also compute a large range of evaluation measures on the server to ensure objective
comparison.
Finally,eachtaskhasitsownnumericid,amachine-readableXMLdescription,as
wellasitsownwebpageincludingallrunsuploadedforthattaskandleaderboards.
4.4.4 Flows
Flows are implementations of single algorithms, workflows, or scripts designed to
solve a given task. Flows can either be uploaded directly (source code or binary)