Generation of Synthetic Images with Generative Adversarial Networks

67 Pages·2017·2.5 MB·English

Generation of Synthetic Images with
Generative Adversarial Networks
DegreeProjectforMasterofScienceinComputerScienceandEngineering
by
Mousa Zeid Baker
Supervisor: Professor Håkan Grahn
Blekinge Institute of Technology
Department of Computer Science and Engineering
January 2018
SE-371 79 Karlskrona, Sweden

Abstract
Machine Learning is a fast growing area that revolutionizes computer programs by providing
systemswiththe abilitytoautomatically learnandimprovefrom experience. Inmostcases, the
training process begins withextracting patterns from data. Thedata is a key factor for machine
learning algorithms, without data the algorithms will not work. Thus, having sufficient and
relevantdataiscrucialfortheperformance.
In this thesis, the researcher tackles the problem of not having a sufficient dataset, interms of
thenumberoftrainingexamples,foranimageclassificationtask. TheideaistouseGenerative
AdversarialNetworkstogeneratesyntheticimagessimilartothegroundtruth,andinthisway
expandadataset. Twotypesofexperimentswereconducted: thefirstwasusedtofine-tuneaDeep
ConvolutionalGenerativeAdversarialNetworkforaspecificdataset,whilethesecondexperiment
wasusedtoanalyzehowsyntheticdataexamplesaffecttheaccuracyofaConvolutionalNeural
Network in a classification task. Three well known datasets were used in the first experiment,
namelyMNIST,Fashion-MNISTandFlowerphotos,whiletwodatasetswereusedinthesecond
experiment: MNISTandFashion-MNIST.
The results of the generated images of MNIST and Fashion-MNIST had good overall quality.
Some classes had clear visual errors while others were indistinguishable from ground truth
examples. When itcomes to theFlower photos,the generatedimages suffered frompoor visual
quality. One can easily tell the synthetic images from the real ones. One reason for the bad
performance is due to the large quantity of noise in the Flower photos dataset. This made it
difficultforthemodeltospottheimportantfeaturesoftheflowers.
The results from the second experimentshow that the accuracy does not increase when the two
datasets, MNIST and Fashion-MNIST, are expanded with synthetic images. This is not because
thegeneratedimageshadbadvisualquality,butbecausetheaccuracyturnedouttonotbehighly
dependentonthenumberoftrainingexamples.
ItcanbeconcludedthatDeepConvolutionalGenerativeAdversarialNetworksarecapableof
generatingsynthetic images similartothe groundtruthand thuscan beusedto expand adataset.
However, this approach does not completely solve the initial problem of not having adequate
datasetsbecauseDeepConvolutionalGenerativeAdversarialNetworksmaythemselvesrequire,
dependingonthedataset,alargequantityoftrainingexamples.
Keywords: classification,deeplearning,generativeadversarialnetwork,machinelearning
Preface
ThisthesisissubmittedtotheDepartmentofComputerScienceandEngineeringatBlekinge
InstituteofTechnology(BTH)inpartialfulfillmentoftherequirementsforthedegreeofMaster
ofScienceinComputerScienceandEngineering. TheDegreeProgrammeinComputerScience
andEngineeringiscomposedof300ECTSwhereofthethesisisequivalentto30ECTS.This
researchwasconductedincollaborationwithSonyMobileCommunicationsinLund.
I would like to extend my sincerest thanks and appreciation to the people who helped me
accomplishthisstudy. Firstly,IwouldliketogratitudemanythankstomysupervisorfromBTH
ProfessorHåkanGrahnforhisadvices,supportandguidancethatcontributedtothequalityof
this thesis. Secondly, I would like to thank Jim Rasmusson and Daniel Lönnblad from Sony
MobileCommunicationsfortheirexperienceinDeepLearningandguidanceduringtheresearch.
AspecialthanksisalsoextendedtoDrAbbasCheddadfromBTHforhisfeedbackonthereport,
and to Stefan Petersson from BTH for providing me with an office and equipment. Lastly, I
wouldliketothankfriendsandfamilyfortheirsupport.
MousaZeidBaker
Karlskrona,January2018
i

Table of Contents
Abstract
Preface i
TableofContents iii
ListofFigures iv
ListofTables v
1 INTRODUCTION 1
1.1 ObjectivesandThesisQuestion . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 ARTIFICIALNEURALNETWORK 4
2.1 Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 ActivationFunction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.1 LinearFunction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.2 SigmoidandTanh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2.3 ReLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.4 Softmax . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.3 LossFunction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.4 GradientDescent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.5 OverfittingandUnderfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.5.1 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5.2 EarlyStopping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6 ConvolutionalNeuralNetwork . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6.1 ConvolutionalLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.6.2 SharedWeights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.6.3 PoolingLayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 GENERATIVEADVERSARIALNETWORK 14
3.1 TheStructureofaGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.2 TheTrainingProcess . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.3 DeepConvolutionalGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4 RELATEDWORK 17
5 METHOD 19
5.1 ResearchTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.1.1 CUDAandcuDNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.1.2 TensorFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.1.3 Caffe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.1.4 DIGITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.2 DataSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.3 DataPreprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
iii
5.4 DCGANsspecifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.5 LeNetCNNspecifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.6 ExperimentalSetup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.6.1 Generationofsyntheticimages . . . . . . . . . . . . . . . . . . . . . . . 25
5.6.2 Measurementofaccuracy . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6 RESULTS 28
6.1 Generationofsyntheticimages . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.1.1 MNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.1.2 Fashion-MNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6.1.3 Flowerphotos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.2 Measurementofaccuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.2.1 MNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
6.2.2 Fashion-MNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
7 ANALYSISANDDISCUSSION 40
7.1 Generationofsyntheticimages . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
7.2 Measurementofaccuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
7.3 ChallengesandLimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
7.4 ReliabilityandValidity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8 CONCLUSIONSANDFUTUREWORK 44
8.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
8.2 FutureWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
References 46
process_images.py 50
List of Figures
2.1 Anillustrationofafeedforwardfullyconnecteddeepneuralnetwork . . . . . . . . . . . . . 4
2.2 Anexampleofanartificialneuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 AplotofSigmoidandTanh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 AplotofReLU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.5 Dropout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.6 ThearchitectureofaCNNthatperformsclassification. . . . . . . . . . . . . . . . . . . . . 10
2.7 Anillustrationofhowafilterslidesacrossaninput . . . . . . . . . . . . . . . . . . . . . . 11
2.8 Anillustrationofhowzeropaddingisappliedtoan8x8input . . . . . . . . . . . . . . . . . 12
2.9 Featuremap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.10 Anillustrationofhowa2x2max-poolingisappliedona4x4featuremap . . . . . . . . . . 13
2.11 AtypicalcombinationoflayersusedinaCNN . . . . . . . . . . . . . . . . . . . . . . . . 13
3.1 AnexampleofageneratornetworkinaDCGAN . . . . . . . . . . . . . . . . . . . . . . . 16
5.1 ThearchitectureoftheLeNetCNNclassifier . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6.1 SamplesofgroundtruthtrainingexamplesfromMNIST . . . . . . . . . . . . . . . . . . . 28
6.2 SamplesofsyntheticimagesgeneratedbythecDCGAN(MNIST10k) . . . . . . . . . . . . 28
6.3 SamplesofsyntheticimagesgeneratedbythecDCGAN(MNIST20k) . . . . . . . . . . . . 29
6.4 SamplesofsyntheticimagesgeneratedbythecDCGAN(MNIST30k) . . . . . . . . . . . . 29
6.5 SamplesofsyntheticimagesgeneratedbythecDCGAN(MNIST60k) . . . . . . . . . . . . 29
6.6 SamplesofgroundtruthtrainingexamplesfromFashion-MNIST . . . . . . . . . . . . . . . 30
iv
6.7 SamplesofsyntheticimagesgeneratedbythecDCGAN(Fashion-MNIST10k) . . . . . . . 30
6.8 SamplesofsyntheticimagesgeneratedbythecDCGAN(Fashion-MNIST20k) . . . . . . . 30
6.9 SamplesofsyntheticimagesgeneratedbythecDCGAN(Fashion-MNIST30k) . . . . . . . 31
6.10 SamplesofsyntheticimagesgeneratedbythecDCGAN(Fashion-MNIST60k) . . . . . . . 31
6.11 AsampleofgroundtruthexamplesfromFlowerphotos(64x64x3) . . . . . . . . . . . . . . 32
6.12 AsampleofgroundtruthexamplesfromFlowerphotos(128x128x3) . . . . . . . . . . . . . 32
6.13 AsampleofsyntheticimagesgeneratedbytheDCGAN(64x64x3,epoch500,batch64) . . . 33
6.14 AsampleofsyntheticimagesgeneratedbytheDCGAN(64x64x3,epoch1000,batch64) . . 33
6.15 AsampleofsyntheticimagesgeneratedbytheDCGAN(64x64x3,epoch500,batch128) . . 34
6.16 AsampleofsyntheticimagesgeneratedbytheDCGAN(64x64x3,epoch1000,batch128) . . 34
6.17 AsampleofsyntheticimagesgeneratedbytheDCGAN(128x128x3,epoch500,batch64) . . 35
6.18 AsampleofsyntheticimagesgeneratedbytheDCGAN(128x128x3,epoch1000,batch64) . 35
6.19 PlotsoftheaccuraciesofLeNetCNNclassifierswhentrainedonMNIST(30k) . . . . . . . 36
6.20 PlotsofthelossvaluesofLeNetCNNclassifierswhentrainedonMNIST(30k) . . . . . . . 36
6.21 PlotsoftheaccuraciesofLeNetCNNclassifierswhentrainedonMNIST(60k) . . . . . . . 37
6.22 PlotsofthelossvaluesofLeNetCNNclassifierswhentrainedonMNIST(60k) . . . . . . . 37
6.23 PlotsoftheaccuraciesofLeNetCNNclassifierswhentrainedonFashion-MNIST(30k) . . 38
6.24 PlotsofthelossvaluesofLeNetCNNclassifierswhentrainedonFashion-MNIST(30k) . . 38
6.25 PlotsoftheaccuraciesofLeNetCNNclassifierswhentrainedonFashion-MNIST(60k) . . 39
6.26 PlotsofthelossvaluesofLeNetCNNclassifierswhentrainedonFashion-MNIST(60k) . . 39
7.1 PlotsofthelossvaluesofLeNetCNNclassifierswhentrainedonMNIST(real) . . . . . . . 41
7.2 PlotsofthelossvaluesofLeNetCNNclassifierswhentrainedonFashion-MNIST(real) . . 41
7.3 PlotsofthelossvaluesoftwoLeNetCNNclassifierswhentrainedonFashion-MNIST . . . 42
List of Tables
5.1 ThediscriminatorinthecDCGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5.2 ThegeneratorinthecDCGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5.3 ThediscriminatorintheDCGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.4 ThegeneratorintheDCGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.5 Generationofsyntheticimages: MNISTandFashion-MNIST . . . . . . . . . . . . . . . . . 26
5.6 Generationofsyntheticimages: Flowerphotos . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.7 Measurementofaccuracy: MNISTandFashion-MNIST . . . . . . . . . . . . . . . . . . . . 27
v

1 INTRODUCTION
Withinthelasthalf-century,thecomputerhasbeenthemostsuccessfulinventionintheworldandithas
notonlychangedourlivesbutalsocompletelychangedtheworldwelivein. Computersandtheiruses
haverapidlygrewduetothehugepotentialtheyhavetoresolveproblemsthatweencounterdailyinour
lives. Theyaredeeplyinvolvedinalmosteverysocietyandhavecontributedtothedevelopmentofmany
areassuchascommunication,education,healthcare,utilityfacilitiesandmanymore. Thechangesthat
alreadyhavebeenmadeiswithgreatcertaintyknownbymostpeople,it’smoreinterestingtodiscussthe
newchangescomputerscanbringinthenearfuture[1].
Computerscientistsandinventorshavelongdreamedofcreatingmachinesthatcanthinkandbecome
intelligent[2]. ArtificialIntelligence(AI)isanactivefieldthatconcernsthistopic. Itstartedwhenthe
nascentfieldofcomputersciencestartedtoaskifacomputercouldbecomeintelligentormimiccognitive
abilities that lead to knowledge such as learning, problem solving and reasoning. In the beginning of
thedevelopmentofAI,softwarewerehard-codedwithknowledgeabouttheworldwithalistofformal,
mathematical rules. This approach never led to a major success due to the struggle of describing the
complexity of the world with formal rules. Instead of relying on hard-coded knowledge, AI systems
neededacapabilitytoextracttheirownknowledge. Systemsstartedtoextractpatternsfromrawdata,this
capabilitycometobeknownasMachineLearning(ML)[3].
MLisafieldwithmanydifferentlearningcapabilitiesanditisstillexpanding. Therearedifferenttypesof
learningproblems,threepopulartypeswillbedescribedherebutthereadershouldknowthattheyare
nottheonlytypes. Thefirsttypeiscalledsupervisedlearning,averywellstudiedproblemwithmany
successfuloutcomes. Supervisedlearningiswhereallthedataislabeled,thatiswhenforeveryinput
variable(x),theoutputvariable(y)isknown. Analgorithmlearntomaptheinputtotheoutputandsince
theoutput(correctanswer)isknownforeveryinput,thealgorithmissaidtobesupervised. Supervised
learning can further be divided into two sections: the first is when the output is a real value, that is a
regression problem, and the second is when the output variable is a category, that is a classification
problem. Forexample,supposeadatasetwithimagesofcatsanddogs. Everyimageiseitherlabeled
with"cat"or"dog",analgorithmthenlearnstomapeachimagetoitscorrespondinglabelcategory,thisis
aclassificationproblem[4,1].
Another ML problem type is when only the input data (x) is known, this is refereed to unsupervised
learning. The task here is to organize the data or to discover the structure or distribution of the data
in order to learn more about it. Since there are no correct answers (y), algorithms work on their own.
Unsupervisedlearningcanfurtherbedividedintotwosections: clusteringandassociation. Clusteringis
usedtodiscovertheinherentgroupingsinthedatawhileassociationisusedtodiscoverrulesthatdescribe
thedata[4,1].
Thelasttypeiscalledsemi-supervisedmachinelearningandreferstoproblemswhereonepartofthe
datasetislabeledandonepartisunlabeled. Thisisverycommonbecauseitisveryexpensiveandtime
consumingtolabelbigdatasets. Supposeaclassificationproblemwherethedatasetisnotfullylabeled.
Thenunsupervisedlearningtechniquescanbeusedtodiscoverthestructureintheinputvariables. Or
supervisedlearningtechniquescanbeusedtopredictlabelstoeveryunlabeled x [4,1].
OneimportantfactorregardingtheperformanceofmostMLalgorithmsishowthedataisrepresented.
Alogisticregressionalgorithmusedin,let’ssayhealth-care,forcesareanrecommendationswillnotmake
usefulpredictionsifitwasgivenanmagneticresonanceimaging(MRI)scan. TheMLalgorithmwillnot
beabletoextractusefulfeatures(information)fromtheMRIscan. Instead,adoctorcanprovidetheML
algorithmwithsomerelevantfeaturesaboutthepatient. ManyAItaskscanbesolvedifthecorrectsetof
featuresareextractedandthenprovidedtoanMLalgorithm. Forexample,toidentifyifaspeakerisaman,
1