Transparency and Interpretability for Learned Representations of Artificial Neural Networks

230 Pages·2022·5.313 MB·English

by  Richard Meyes

Richard Meyes
Transparency and
Interpretability
for Learned
Representations
of Artificial Neural
Networks
Transparency and Interpretability
for Learned Representations of Artificial
Neural Networks
Richard Meyes
Transparency
and Interpretability
for Learned
Representations
of Artificial Neural
Networks
Dr.-Ing.RichardMeyes,M.Sc.
ResearchGroupLead“InterpretableLearningModels”
InstituteforTechnologiesandManagementofDigital
Transformation
UniversityofWuppertal
Wuppertal,Germany
Thescientificstudiesandcorrespondingresultspresentedinthisbookwereconducted
withintheframeworkofadoctoralprojecttoobtainthedegreeofDr.-Ing.attheSchool
ofElectrical,InformationandMediaEngineeringattheUniversityofWuppertal.
ISBN978-3-658-40003-3 ISBN978-3-658-40004-0 (eBook)
https://doi.org/10.1007/978-3-658-40004-0
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringer
FachmedienWiesbadenGmbH,partofSpringerNature2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,
whetherthewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprint-
ing, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other
physicalway,andtransmissionorinformationstorageandretrieval,electronicadaptation,computer
software,orbysimilarordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthis
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexempt
fromtherelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthis
bookarebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernorthe
authorsortheeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontained
hereinorforanyerrorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwith
regardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
This Springer Vieweg imprint is published by the registered company Springer Fachmedien
WiesbadenGmbH,partofSpringerNature.
Theregisteredcompanyaddressis:Abraham-Lincoln-Str.46,65189Wiesbaden,Germany
Menoughttoknowthatfromthebrain,
andfromthebrainonly,ariseour
pleasures,joys,laughter,andjests,as
wellasoursorrows,pains,griefs,and
tears.Throughit,inparticular,wethink,
see,hear,anddistinguishtheuglyfrom
thebeautiful,thebadfromthegood,the
pleasantfromtheunpleasant,insome
casesusingcustomasatest,inothers
perceivingthemfromtheirutility.Itisthe
samethingwhichmakesusmador
delirious,inspiresuswithdreadorfear,
whetherbynightorbyday,brings
sleeplessness,inopportunemistakes,
aimlessanxieties,absent-mindedness,
andactsthatarecontrarytohabit.These
thingsthatwesufferallcomefromthe
brain,whenitisnothealthy,butbecomes
abnormallyhot,cold,moist,ordry,or
suffersanyotherunnatural affectionto
whichitwasnotaccustomed.Madness
comesfromitsmoistness.Whenthebrain
isabnormallymoist,ofnecessityit
moves,andwhenitmovesneithersight
norhearingarestill,butweseeorhear
nowonethingandnowanother,andthe
tonguespeaksinaccordancewiththe
thingsseenandheardonanyoccasion.
Butallthetimethebrainisstillamanis
intelligent.
—Hippocrates
TheSacredDisease,inHippocrates,
trans.W.H.S.Jones(1923), Vol.2,175
Contents
1 Introduction .................................................. 1
1.1 Object of Investigation .................................... 2
1.2 Research Questions ....................................... 3
1.3 Structure of the Thesis .................................... 6
2 Background&Foundations .................................... 9
2.1 A Short History of AI Research ............................ 9
2.1.1 The Early Years .................................... 10
2.1.2 The Golden Ages ................................... 11
2.1.3 The AI Winter ..................................... 13
2.1.4 The AI Renaissance ................................ 13
2.2 The Modern Era of AI Research ............................ 14
2.2.1 Deep Learning for Computer Vision .................. 15
2.2.2 Computer Vision Beyond the ILSVRC ................ 17
2.2.3 From Supervised Learning to Reinforcement
Learning .......................................... 18
2.2.4 Deep Reinforcement Learning Breakthroughs
for Video and Board Games ......................... 19
2.2.5 Tackling Games with Imperfect Information ........... 23
2.3 Towards Research on Transparency & Interpretability ......... 24
2.3.1 A Shift of Paradigm: From Optimizing
to Understanding ................................... 25
2.3.2 Inspirations from Neuroscience Research .............. 27
3 MethodsandTerminology ..................................... 31
3.1 Learned Representations ................................... 31
3.1.1 Investigating Learned Representations ................. 33
vii
viii Contents
3.1.1.1 Statistical Measures ........................ 34
3.1.1.2 Transformations & Embeddings .............. 35
3.1.2 Visualizing Structure of the Learned Representations ... 46
3.1.2.1 Magnitude of Neuron Activations ............ 47
3.1.2.2 Selectivity of Neuron Activations ............. 47
3.1.2.3 Ablations of Individual Neurons .............. 48
3.1.2.4 Gini Importance ............................ 51
3.2 Delimitation of the Object of Investigation ................... 51
3.2.1 Transparency for Computer Vision Models ............ 52
3.2.2 Transparency for Motion Control Models .............. 53
3.2.3 Transfer to an Industrial Application Scenario .......... 55
4 RelatedWork ................................................. 57
4.1 Relationship Between a Network’s Input Features and its
Output ................................................... 59
4.2 Visualization of Network Properties and Graphical User
Interfaces ................................................ 61
4.3 Investigating the Importance of Individual Network
Components .............................................. 64
4.3.1 Miscellaneous Contributions ......................... 65
4.3.2 Ablations Studies ................................... 67
4.3.3 Reverse Engineering of Neural Networks .............. 69
5 ResearchStudies .............................................. 71
5.1 Investigating Learned Representations in Computer Vision ..... 75
5.1.1 Research Study 1: Characterizing Single Neurons
in a Shallow MLP .................................. 75
5.1.1.1 Key Contributions of the Study .............. 75
5.1.1.2 Methods and Experimental Design ........... 77
5.1.1.3 Results .................................... 78
5.1.1.4 Summary and Contribution of the Results
to the Research Questions ................... 88
5.1.2 Research Study 2: Network Ablations in a Deep
Neural Network .................................... 91
5.1.2.1 Key Contributions of the Study .............. 91
5.1.2.2 Methods and Experimental Design ........... 92
5.1.2.3 Results .................................... 93
5.1.2.4 Summary and Contribution of the Results
to the Research Questions ................... 98
Contents ix
5.1.3 Research Study 3: Functional Neuron Populations
in Custom-made CNNs ............................. 100
5.1.3.1 Key Contributions of the Study .............. 100
5.1.3.2 Methods and Experimental Design ........... 101
5.1.3.3 Results .................................... 105
5.1.3.4 Summary and Contribution of the Results
to the Research Questions ................... 113
5.2 Investigating Learned Representations in Motor Control ....... 114
5.2.1 Research Study 4: Influence of Network Ablations
on Activation Patterns ............................... 115
5.2.1.1 Key Contributions of the Study .............. 116
5.2.1.2 Methods and Experimental Design ........... 117
5.2.1.3 Results .................................... 119
5.2.1.4 Summary and Contribution of the Results
to the Research Questions ................... 128
5.2.2 Research Study 5: Relation Between Neural
Activations and Agent Behavior ...................... 129
5.2.2.1 Key Contributions of the Study .............. 130
5.2.2.2 Methods and Experimental Design ........... 131
5.2.2.3 Results .................................... 135
5.2.2.4 Summary and Contribution of the Results
to the Research Questions ................... 147
6 TransferStudies .............................................. 149
6.1 Transfer Study 1: Network Ablations for Deep Drawing ....... 150
6.1.1 Key Contributions of the Study ...................... 151
6.1.2 Methods and Experimental Design .................... 151
6.1.3 Results ............................................ 159
6.1.4 Summary and Contribution of the Results
to the Research Questions ........................... 164
6.2 Transfer Study 2: Attention Mechanisms for Deep Drawing .... 166
6.2.1 Key Contributions of the Study ...................... 167
6.2.2 Methods and Experimental Design .................... 167
6.2.3 Results ............................................ 173
6.2.4 Summary and Contribution of the Results
to the Research Questions ........................... 181
x Contents
7 CriticalReflection&Outlook .................................. 183
7.1 Reflection of Results & Contribution to Research Questions .... 183
7.1.1 Research Question 1 ................................ 184
7.1.2 Research Question 2 ................................ 185
7.1.3 Research Question 3 ................................ 187
7.1.4 Research Question 4 ................................ 189
7.2 Future Research Directions ................................. 191
8 Summary ..................................................... 195
References ....................................................... 197