Generative adversarial networks as integrated forward and inverse model for motor control

69 Pages·2017·2.91 MB·English

DEGREE PROJECT IN THE FIELD OF TECHNOLOGY
ENGINEERING PHYSICS
AND THE MAIN FIELD OF STUDY
COMPUTER SCIENCE AND ENGINEERING,
SECOND CYCLE, 30 CREDITS
STOCKHOLM, SWEDEN 2017
Generative adversarial networks
as integrated forward and inverse
model for motor control
MOVITZ LENNINGER
KTH ROYAL INSTITUTE OF TECHNOLOGY
SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION
Generative adversarial
networks as integrated
forward and inverse model for
motor control
MOVITZ LENNINGER
Master in Machine Learning
Date: December 22, 2017
Supervisor: Hansol Choi (University of Freiburg),
Jeanette Hellgren Kotaleski (KTH)
Examiner: Erik Fransén
Swedish title: Generativa konkurrerande nätverk som integrerad
framåtriktad och invers modell för rörelsekontroll
School of Computer Science and Communication

Abstract
Internal models are believed to be crucial components in human motor control. It has
beensuggestedthatthecentralnervoussystem(CNS)usesforwardandinversemod-
els as internal representations of the motor systems. However, it is still unclear how
the CNS implements the high-dimensional control of our movements. In this project,
generative adversarial networks (GAN) are studied as a generative model of move-
ment data. It is shown that, for a relatively small number of effectors, it is possible
to train a GAN which produces new movement samples that are plausible given a
simulator environment. It is believed that these models can be extended to generate
high-dimensional movement data. Furthermore, this project investigates the possi-
bility to use a trained GAN as an integrated forward and inverse model for motor
control.
iii
Sammanfattning
Internamodellertrosvaraenviktigdelavmänskligrörelsekontroll.Detharföreslagits
attdetcentralanervsystemet(CNS)användersigavframåtriktademodellerochinver-
samodellerförinternrepresentationavmotorsystemen.Dockärdetfortfarandeokänt
hurdetcentralanervsystemetimplementerardennahögdimensionellakontroll.Detta
examensarbete undersöker användningen av generativa konkurrerande nätverk som
generativ modell av rörelsedata. Experiment visar att dessa nätverk kan tränas till att
generera ny rörelsedata av en tvådelad arm och att den genererade datan efterliknar
träningsdatan. Vi tror att nätverken även kan modellera mer högdimensionell rörel-
sedata. I projektet undersöks även användningen av dessa nätverk som en integrerad
framåtriktadochinversmodell.
iv
Contents
1 Introduction 1
1.1 Introductionoftheproject . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Ethicalconsiderations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Motorcontrol 4
2.1 Coordinationofmovement . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Actionandperception . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 Motorcontroltheory . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.2 Predictivecoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3 Generativemodels 13
3.1 RepresentationlearningandDeeplearning . . . . . . . . . . . . . . . . . 13
3.2 Generativemodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2.1 Populargenerativemodels . . . . . . . . . . . . . . . . . . . . . . 15
3.3 GenerativeAdversarialNetworks-GAN . . . . . . . . . . . . . . . . . . 16
3.3.1 TrainingGANs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.3.2 Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.3.3 GAN-Aspecialcaseoff-GAN . . . . . . . . . . . . . . . . . . . . 19
3.3.4 WassersteinGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3.5 Inferenceinlatentspace-Reconstructingmissingelements . . . 24
4 Relatedwork 27
5 Method 28
5.1 Simulatorenvironment-atoymodel . . . . . . . . . . . . . . . . . . . . . 28
5.1.1 Trainingdata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
5.2 TrainingGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
5.2.1 WassersteinGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
v
5.2.2 Quantifyingtrainingprogression . . . . . . . . . . . . . . . . . . . 32
5.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
5.3.1 Forwardmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3.2 Inversemodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
5.3.3 Inversemodel-Selectingminimalaction . . . . . . . . . . . . . . 35
5.3.4 Exploringthelatentspace . . . . . . . . . . . . . . . . . . . . . . . 35
6 Experimentalresults 36
6.1 Trainingphase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
6.2 Forwardmodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
6.3 Inversemodel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6.4 Choosinggenerativemodel . . . . . . . . . . . . . . . . . . . . . . . . . . 44
6.5 Inversemodel-Selectingminimalaction . . . . . . . . . . . . . . . . . . 45
6.6 Exploringthelatentspace . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7 DiscussionandConclusions 50
7.1 Discussionofexperiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
7.2 Discussionofimplementation . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3 Difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.4 ConnectiontoOptimalcontrol? . . . . . . . . . . . . . . . . . . . . . . . . 53
7.5 Futuredevelopment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
Bibliography 56
vi
Chapter 1
Introduction
The human body consists of many joints and limbs, which our motor systems have
to coordinate across both space and time. Thus, the problem of motor control is in-
herently high-dimensional [9]. Suggestions have been raised that the brain uses di-
mensionality reduction to obtain a low-dimensional control space, however the prin-
ciple of such a dimensionality reduction is unknown [7, 58]. Additionally, the use
of forward and inverse models in human motor control has been extensively studied
[61, 42, 21]. Forward models are specialized at learning the causal dynamics of the
bodily interaction with the external world in order to predict the future state of the
body. Inverse models, on the other hand, are the forward processes in reverse. Given
adesiredfuturestateofthebody,aninversemodelproducesmotorcommandswhich
guides the body towards the desired state. Although it is conjectured that the brain
makes use of forward and inverse models to handle the high-dimensional coordina-
tion of the body, it is not known how the brain implements these models even at an
algorithmiclevel.
In machine learning, generative adversarial networks (GANs) [29] was initially de-
velopedasahigh-dimensionalimagegenerationmodel. Byprovidingalargeamount
of digital images, a generator network can be taught to generate new, original images
similar to those in the training set. In addition, studies have shown that these gen-
erative networks can be used to reconstruct corrupted images. However, the GAN
framework is not limited to images and has been applied to other domains, such as
astrophysics[45]andNLP[62],aswell.
1
1.1 Introduction of the project
Motor control suffers from several problems, including the problem of degrees-of-
freedom. Some of these problems could be solved by GAN. GAN has demonstrated
theabilitytogeneratehigh-dimensionaldataandtoperforminference,propertiesthat
may be related to the analysis-by-synthesis hypothesis. This master thesis aims to in-
vestigatewhetherGANscanbetrainedtogeneratenew,plausiblemovementdataand
subsequently be used as an integrated forward and inverse model for motor control.
In this proof-of-concept study, GANs were trained using movement data from a two-
linked arm in a two-dimensional task space. The training data consisted of samples
ofrandommovement,thusresemblingmotorbabbling. Importantly,thetrainingwas
conducted without providing the networks with any prior information of the prob-
lemathand. Thenetworks’capacitiesasintegratedforwardandinversemodelswere
tested by reconstructing masked (i.e. corrupted) movement samples. The forward
and inverse models were defined as reconstruction tasks where the networks had to
re-create the full movement samples from partially masked samples. The forward
and inverse tasks were distinguished by masking different domains of the movement
samples.
1.2 Contribution
This project contributes to the field of computational neuroscience by suggesting a
newapproachtomodelthegenerationofmovement. Althoughthisprojectonlydeals
with relatively low-dimensional movement data generated from a toy environment,
themethodofthisprojectcouldbeextendedtomodelbothdataofhigherdimensions
anddatageneratedfrommorecomplexenvironments. Thehopeisthatthisapproach
could, in the future, be used to test ideas proposed by the theories of human motor
control,ortobecomparedwithexperimentsconductedwithhumansubjects.
Fromamachinelearningperspective,thethesisdemonstratesanewdomaininwhich
the GAN framework could be useful. Although the GAN framework has been most
intensely researched as an image generation tool, the number of possible areas of ap-
plication is believed to be huge. In addition, high-dimensional control also poses a
probleminrobotics,andtheresultsoftheprojectcouldbeofequalinterestforrobotic
applications.
2
1.3 Delimitations
In this project, only small networks consisting of fully connected layers were trained
and tested. Many standard machine learning methods, such as batch normalization
etc., were not required. The intention was to keep the neural networks as simple as
possible as this was a proof-of-concept study. However, in future research, larger and
moresophisticatednetworksmightbenecessaryifthetaskcomplexityisraised.
Alltrainingdataweresampledfromatoyenvironmentwithlineardynamicswithout
any external forces. Of course, the movements of the human body is more complex
butitisleftasfurtherresearchtomodelmorecomplexdata.
1.4 Ethical considerations
The project was in its entirety carried out using computer experiments. The results
of this project could potentially be of interest for robotic applications. Accurate high-
dimensionalcontrolcouldenablerobotstoperformnewtasks,aidinghumansbothin
private life and via industry. Of course, robots could be designed to perform tasks
which are of morally dubious character. These are, naturally, difficult issues that
should not be ignored. However, the benefits of robots and artificial intelligence
shouldnotbeignoredeither. Inaddition,adeeperunderstandingofhigh-dimensional
motor control could aid computational neuroscience in understanding human motor
control. Anunderstandingwhichcouldalsobenefitindividualsandsocietyatlarge.
1.5 Acknowledgment
Iwouldliketothankeveryonewhohavehelpedandsupportedmeduringthisproject.
A special thank to: Hansol Choi and Carsten Mehring at University of Freiburg for
their invaluable help and advice as supervisor and principle of this project, Jeanette
Hällgren Kotaleski and Erik Fransén for their work as supervisor and examiner at
KTH,SaraLenningerforproofreadingthereport,andJoschkaBoedeckerandManuel
Watter at the Machine Learning Lab at University of Freiburg for their helpful com-
mentsandinterest.
3