Principles of Big data Preparing, Sharing, and Analyzing Complex Information

288 Pages·2014·15.29 MB·English

by  Jules J. Berman

PRINCIPLES OF BIG DATA
Intentionally left as blank
PRINCIPLES OF
BIG DATA
Preparing, Sharing, and Analyzing
Complex Information
JULES J. BERMAN, Ph.D., M.D.
AMSTERDAM (cid:129) BOSTON (cid:129) HEIDELBERG (cid:129) LONDON
NEW YORK (cid:129) OXFORD (cid:129) PARIS (cid:129) SAN DIEGO
SAN FRANCISCO (cid:129) SINGAPORE (cid:129) SYDNEY (cid:129) TOKYO
Morgan Kaufmann is an imprint of Elsevier
AcquiringEditor:AndreaDierna
EditorialProjectManager:HeatherScherer
ProjectManager:PunithavathyGovindaradjane
Designer:RussellPurdy
MorganKaufmannisanimprintofElsevier
225WymanStreet,Waltham,MA02451,USA
Copyright#2013ElsevierInc.Allrightsreserved
Nopartofthispublicationmaybereproducedortransmittedinanyformorbyanymeans,
electronicormechanical,includingphotocopying,recording,oranyinformationstorage
andretrievalsystem,withoutpermissioninwritingfromthepublisher.Detailsonhowto
seekpermission,furtherinformationaboutthePublisher’spermissionspoliciesandour
arrangementswithorganizationssuchastheCopyrightClearanceCenterandthe
CopyrightLicensingAgency,canbefoundatourwebsite:www.elsevier.com/permissions.
Thisbookandtheindividualcontributionscontainedinitareprotectedundercopyrightby
thePublisher(otherthanasmaybenotedherein).
Notices
Knowledgeandbestpracticeinthisfieldareconstantlychanging.Asnewresearchand
experiencebroadenourunderstanding,changesinresearchmethodsorprofessional
practices,maybecomenecessary.
Practitionersandresearchersmustalwaysrelyontheirownexperienceandknowledgein
evaluatingandusinganyinformationormethodsdescribedherein.Inusingsuch
informationormethodstheyshouldbemindfuloftheirownsafetyandthesafetyofothers,
includingpartiesforwhomtheyhaveaprofessionalresponsibility.
Tothefullestextentofthelaw,neitherthePublishernortheauthors,contributors,or
editors,assumeanyliabilityforanyinjuryand/ordamagetopersonsorpropertyasa
matterofproductsliability,negligenceorotherwise,orfromanyuseoroperationofany
methods,products,instructions,orideascontainedinthematerialherein.
LibraryofCongressCataloging-in-PublicationData
Berman,JulesJ.
Principlesofbigdata:preparing,sharing,andanalyzingcomplexinformation/Jules
JBerman.
pagescm
ISBN978-0-12-404576-7
1.Bigdata.2.Databasemanagement.I.Title.
QA76.9.D32B472013
005.74–dc23
2013006421
BritishLibraryCataloguing-in-PublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
PrintedandboundintheUnitedStatesofAmerica
13 14 15 16 17 10 9 8 7 6 5 4 3 2 1
ForinformationonallMKpublicationsvisitourwebsiteatwww.mkp.com
Dedication
To my father,Benjamin
v
Intentionally left as blank
Contents
Acknowledgments xi 4. Introspection
Author Biography xiii
Background 49
Preface xv
KnowledgeofSelf 50
Introduction xix eXtensibleMarkupLanguage 52
IntroductiontoMeaning 54
NamespacesandtheAggregationofMeaningful
1. Providing Structure to Unstructured
Assertions 55
Data
ResourceDescriptionFrameworkTriples 56
Reflection 59
Background 1
UseCase:TrustedTimeStamp 59
MachineTranslation 2
Summary 60
Autocoding 4
Indexing 9
5. Data Integration and Software
TermExtraction 11
Interoperability
2. Identification, Deidentification,
Background 63
and Reidentification
TheCommitteetoSurveyStandards 64
StandardTrajectory 65
Background 15
SpecificationsandStandards 69
FeaturesofanIdentifierSystem 17
Versioning 71
RegisteredUniqueObjectIdentifiers 18
ComplianceIssues 73
ReallyBadIdentifierMethods 22
InterfacestoBigDataResources 74
EmbeddingInformationinanIdentifier:Not
Recommended 24
One-WayHashes 25 6. Immutability and Immortality
UseCase:HospitalRegistration 26
Deidentification 28 Background 77
DataScrubbing 30 ImmutabilityandIdentifiers 78
Reidentification 31 DataObjects 80
LessonsLearned 32 LegacyData 82
DataBornfromData 83
ReconcilingIdentifiersacrossInstitutions 84
3. Ontologies and Semantics
Zero-KnowledgeReconciliation 86
TheCurator’sBurden 87
Background 35
Classifications,theSimplestofOntologies 36
Ontologies,ClasseswithMultipleParents 39 7. Measurement
ChoosingaClassModel 40
IntroductiontoResourceDescriptionFramework Background 89
Schema 44 Counting 90
CommonPitfallsinOntologyDevelopment 46 GeneCounting 93
vii
viii
CONTENTS
DealingwithNegations 93 Step2.ResourceEvaluation 158
UnderstandingYourControl 95 Step3.AQuestionIsReformulated 159
PracticalSignificanceofMeasurements 96 Step4.QueryOutputAdequacy 160
Obsessive-CompulsiveDisorder:TheMarkofaGreat Step5.DataDescription 161
DataManager 97 Step6.DataReduction 161
Step7.AlgorithmsAreSelected,IfAbsolutely
8. SimplebutPowerfulBigDataTechniques Necessary 162
Step8.ResultsAreReviewedandConclusions
Background 99 AreAsserted 164
LookattheData 100 Step9.ConclusionsAreExaminedandSubjected
DataRange 110 toValidation 164
Denominator 112
FrequencyDistributions 115
12. Failure
MeanandStandardDeviation 119
Estimation-OnlyAnalyses 122
Background 167
UseCase:WatchingDataTrendswithGoogle
FailureIsCommon 168
Ngrams 123
FailedStandards 169
UseCase:EstimatingMoviePreferences 126
Complexity 172
WhenDoesComplexityHelp? 173
9. Analysis
WhenRedundancyFails 174
SaveMoney;Don’tProtectHarmless
Background 129
Information 176
AnalyticTasks 130
AfterFailure 177
Clustering,Classifying,Recommending,and
UseCase:CancerBiomedicalInformaticsGrid,
Modeling 130
aBridgeTooFar 178
DataReduction 134
NormalizingandAdjustingData 137
BigDataSoftware:SpeedandScalability 139 13. Legalities
FindRelationships,NotSimilarities 141
Background 183
10. Special Considerations in Big Data ResponsibilityfortheAccuracyandLegitimacyof
Analysis ContainedData 184
RightstoCreate,Use,andSharetheResource 185
Background 145 CopyrightandPatentInfringementsIncurredby
TheoryinSearchofData 146 UsingStandards 187
DatainSearchofaTheory 146 ProtectionsforIndividuals 188
Overfitting 148 Consent 190
BignessBias 148 UnconsentedData 194
TooMuchData 151 GoodPoliciesAreaGoodPolicy 197
FixingData 152 UseCase:TheHavasupaiStory 198
DataSubsetsinBigData:NeitherAdditivenor
Transitive 153 14. Societal Issues
AdditionalBigDataPitfalls 154
Background 201
11. Stepwise Approach to Big Data HowBigDataIsPerceived 201
Analysis TheNecessityofDataSharing,EvenWhenIt
SeemsIrrelevant 204
Background 157 ReducingCostsandIncreasingProductivitywith
Step1.AQuestionIsFormulated 158 BigData 208
ix
CONTENTS
PublicMistrust 210 Glossary 229
SavingUsfromOurselves 211 References 247
HubrisandHyperbole 213 Index 257
15. The Future
Background 217
LastWords 226