Artificial Neural Networks. A Practical Course

303 Pages·2017·11.882 MB·english

by  Ivan Nunes da Silva, Danilo Hernane Spatti, Rogerio Andrade Flauzino, Luisa Helena Bartocci Liboni, Silas Franco dos Reis Alves

Ivan Nunes da Silva Danilo Hernane Spatti
(cid:129)
Rogerio Andrade Flauzino
Luisa Helena Bartocci Liboni
Silas Franco dos Reis Alves
fi
Arti cial Neural Networks
A Practical Course
123
IvanNunes daSilva LuisaHelena Bartocci Liboni
Department ofElectrical andComputer Department ofElectrical andComputer
Engineering, USP/EESC/SEL Engineering, USP/EESC/SEL
University of SãoPaulo University of SãoPaulo
São Carlos, São Paulo São Carlos, São Paulo
Brazil Brazil
DaniloHernane Spatti Silas Francodos Reis Alves
Department ofElectrical andComputer Department ofElectrical andComputer
Engineering, USP/EESC/SEL Engineering, USP/EESC/SEL
University of SãoPaulo University of SãoPaulo
São Carlos, São Paulo São Carlos, São Paulo
Brazil Brazil
RogerioAndrade Flauzino
Department ofElectrical andComputer
Engineering, USP/EESC/SEL
University of SãoPaulo
São Carlos, São Paulo
Brazil
ISBN978-3-319-43161-1 ISBN978-3-319-43162-8 (eBook)
DOI 10.1007/978-3-319-43162-8
LibraryofCongressControlNumber:2016947754
©SpringerInternationalPublishingSwitzerland2017
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAGSwitzerland
Preface
Whatareartificialneuralnetworks?Whatistheirpurpose?Whataretheirpotential
practical applications? What kind of problems can they solve?
Withthesequestionsinmind,thisbookwaswrittenwiththeprimaryconcernof
answering readers with different profiles, from those interested in acquiring
knowledgeaboutarchitecturesofartificialneuralnetworktothosemotivatedbyits
multiple applications (of practical aspect) for solving real-world problems.
Thisbookaudienceismultidisciplinary,asitwillbeconfirmedbythenumerous
exercisesandexamplesaddressedhere.Itexploresdifferentknowledgeareas,such
as engineering, computer science, mathematics, physics, economics, finance,
statistics, and neurosciences. Additionally, it is expected that this book could be
interestingfor those from many other areas that have been inthe focus ofartificial
neural networks, such as medicine, psychology, chemistry, pharmaceutical sci-
ences, biology, ecology, geology, and so on.
Regarding the academic approach of this book and its audience, the chapters
were tailored in a fashion that attempts to discuss, step-by-step, the thematic con-
cepts, covering a broad range of technical and theoretical information. Therefore,
besidesmeetingtheprofessionalaudience’sdesiretobeginordeepentheirstudyon
artificial neural networks and its potential applications, this book is intended to be
used as a textbook for undergraduate and graduate courses, which address the
subject of artificial neural networks in their syllabus.
Furthermore,thetextwascomposedusinganaccessiblelanguagesoitcouldbe
readbyprofessionals,students,researchers,andautodidactics,asastraightforward
andindependentguideforlearningbasicandadvancedsubjectsrelated toartificial
neuralnetworks.Tothisend,theprerequisitesforunderstandingthisbook’scontent
are basic, requiring only a few elementary knowledge about linear algebra, algo-
rithms, and calculus.
The first part of this book (Chaps. 1–10), which is intended for those readers
who want to begin or improve their theoretical investigation on artificial neural
networks, addresses the fundamental architectures that can be implemented in
several application scenarios.
Thesecondpartofthisbook(Chaps. 11–20)wasparticularlycreatedtopresent
solutions that comprise artificial neural networks for solving practical problems
from different knowledge areas. It describes several developing details considered
toachievethedescribedresults.Suchaspectcontributestomatureandimprovethe
reader’s knowledge about the techniques of specifying the most appropriated
artificial neural network architecture for a given application.
São Carlos, Brazil Ivan Nunes da Silva
Danilo Hernane Spatti
Rogerio Andrade Flauzino
Luisa Helena Bartocci Liboni
Silas Franco dos Reis Alves
Organization
This book was carefully created with the mission of presenting an objective,
friendly, accessible,andillustrative text,whosefundamental concern is, infact, its
didactic format. The book organization, made in two parts, along with its com-
position filled with more than 200 figures, eases the knowledge building for dif-
ferentreaders’profiles.Thebibliography,composedofmorethan170references,is
thefoundationforthethemescoveredinthisbook.Thesubjectsarealsoconfronted
with up-to-date context.
Thefirstpartofthebook(PartI),dividedintotenchapters,coversthetheoretical
features related to the main artificial neural architectures, including Perceptron,
Adaptive Linear Element (ADALINE), Multilayer Perceptron, Radial Basis
Function (RBF), Hopfield, Kohonen, Learning Vector Quantization (LCQ),
Counter-Propagation,andAdaptiveResonanceTheory(ART).Ineachoneofthese
chapters from Part I, a section with exercises was inserted, so the reader can
progressively evaluate the knowledge acquired within the numerous subjects
addressed in this book.
Furthermore, the chapters that address the different architectures of neural net-
worksarealsoprovidedwithsectionsthatdiscusshands-onprojects,whosecontent
assiststhereaderonexperimentalaspectsconcerningtheproblemsthatuseartificial
neural networks. Such activities also contribute to the development of practical
knowledge, which may help on specifying, parameterizing, and tuning neural
architectures.
The second part (Part II) addresses several applications, covering different
knowledge areas, whose solutions and implementations come from the neural
networkarchitecturesexploredinPartI.Thedifferentapplicationsaddressedonthis
partofthebookreflectthepotentialapplicabilityofneuralnetworkstothesolution
of problems from engineering and applied sciences. These applications intend to
drive the reader through different modeling and mapping strategies based on arti-
ficial neural networks for solving problems of diverse nature.
Several didactic materials related to this book, including figures, exercise tips,
anddatasets for training neural networks (in tableformat), arealso available atthe
following Website:
http://laips.sel.eesc.usp.br/
In summary, this book will hopefully create a pleasant and enjoyable reading,
thus contributing to the development of a wide range of both theoretical and
practical knowledge concerning the area of artificial neural networks.
Contents
Part I Architectures of Artificial Neural Networks
and Their Theoretical Aspects
1 Introduction... .... .... ..... .... .... .... .... .... ..... .... 3
1.1 Fundamental Theory..... .... .... .... .... .... ..... .... 5
1.1.1 Key Features.... .... .... .... .... .... ..... .... 5
1.1.2 Historical Overview... .... .... .... .... ..... .... 6
1.1.3 Potential Application Areas. .... .... .... ..... .... 8
1.2 Biological Neuron.. ..... .... .... .... .... .... ..... .... 9
1.3 Artificial Neuron... ..... .... .... .... .... .... ..... .... 11
1.3.1 Partially Differentiable Activation Functions..... .... 13
1.3.2 Fully Differentiable Activation Functions .. ..... .... 15
1.4 Performance Parameters .. .... .... .... .... .... ..... .... 18
1.5 Exercises. .... .... ..... .... .... .... .... .... ..... .... 19
2 Artificial Neural Network Architectures
and Training Processes.. ..... .... .... .... .... .... ..... .... 21
2.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 21
2.2 Main Architectures of Artificial Neural Networks... ..... .... 21
2.2.1 Single-Layer Feedforward Architecture.... ..... .... 22
2.2.2 Multiple-Layer Feedforward Architectures . ..... .... 23
2.2.3 Recurrent or Feedback Architecture .. .... ..... .... 24
2.2.4 Mesh Architectures ... .... .... .... .... ..... .... 24
2.3 Training Processes and Properties of Learning. .... ..... .... 25
2.3.1 Supervised Learning .. .... .... .... .... ..... .... 26
2.3.2 Unsupervised Learning .... .... .... .... ..... .... 26
2.3.3 Reinforcement Learning ... .... .... .... ..... .... 26
2.3.4 Offline Learning . .... .... .... .... .... ..... .... 27
2.3.5 Online Learning.. .... .... .... .... .... ..... .... 27
2.4 Exercises. .... .... ..... .... .... .... .... .... ..... .... 28
3 The Perceptron Network. ..... .... .... .... .... .... ..... .... 29
3.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 29
3.2 Operating Principle of the Perceptron.... .... .... ..... .... 30
3.3 Mathematical Analysis of the Perceptron . .... .... ..... .... 32
3.4 Training Process of the Perceptron.. .... .... .... ..... .... 33
3.5 Exercises. .... .... ..... .... .... .... .... .... ..... .... 38
3.6 Practical Work .... ..... .... .... .... .... .... ..... .... 39
4 The ADALINE Network and Delta Rule. .... .... .... ..... .... 41
4.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 41
4.2 Operating Principle of the ADALINE.... .... .... ..... .... 42
4.3 Training Process of the ADALINE.. .... .... .... ..... .... 44
4.4 ComparisonBetweentheTrainingProcessesofthePerceptron
and the ADALINE. ..... .... .... .... .... .... ..... .... 50
4.5 Exercises. .... .... ..... .... .... .... .... .... ..... .... 52
4.6 Practical Work .... ..... .... .... .... .... .... ..... .... 52
5 Multilayer Perceptron Networks ... .... .... .... .... ..... .... 55
5.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 55
5.2 Operating Principle of the Multilayer Perceptron ... ..... .... 56
5.3 Training Process of the Multilayer Perceptron . .... ..... .... 57
5.3.1 Deriving the Backpropagation Algorithm .. ..... .... 58
5.3.2 Implementing the Backpropagation Algorithm.... .... 69
5.3.3 Optimized Versions of the Backpropagation
Algorithm . ..... .... .... .... .... .... ..... .... 71
5.4 Multilayer Perceptron Applications.. .... .... .... ..... .... 78
5.4.1 Problems of Pattern Classification.... .... ..... .... 78
5.4.2 Functional Approximation Problems
(Curve Fitting)... .... .... .... .... .... ..... .... 86
5.4.3 Problems Involving Time-Variant Systems . ..... .... 90
5.5 Aspects of Topological Specifications for MLP Networks . .... 97
5.5.1 Aspects of Cross-Validation Methods . .... ..... .... 97
5.5.2 Aspects of the Training and Test Subsets .. ..... .... 101
5.5.3 Aspects of Overfitting and Underfitting Scenarios. .... 101
5.5.4 Aspects of Early Stopping.. .... .... .... ..... .... 103
5.5.5 Aspects of Convergence to Local Minima.. ..... .... 104
5.6 Implementation Aspects of Multilayer Perceptron Networks.... 105
5.7 Exercises. .... .... ..... .... .... .... .... .... ..... .... 109
5.8 Practical Work 1 (Function Approximation)... .... ..... .... 110
5.9 Practical Work 2 (Pattern Classification).. .... .... ..... .... 112
5.10 Practical Work 3 (Time-Variant Systems). .... .... ..... .... 114
6 Radial Basis Function Networks.... .... .... .... .... ..... .... 117
6.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 117
6.2 Training Process of the RBF Network ... .... .... ..... .... 117
6.2.1 Adjustment of the Neurons from the Intermediate
Layer (Stage I) .. .... .... .... .... .... ..... .... 118
6.2.2 Adjustment of Neurons of the Output Layer
(Stage II).. ..... .... .... .... .... .... ..... .... 123
6.3 Applications of RBF Networks. .... .... .... .... ..... .... 125
6.4 Exercises. .... .... ..... .... .... .... .... .... ..... .... 130
6.5 Practical Work 1 (Pattern Classification).. .... .... ..... .... 133
6.6 Practical Work 2 (Function Approximation)... .... ..... .... 135
7 Recurrent Hopfield Networks.. .... .... .... .... .... ..... .... 139
7.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 139
7.2 Operating Principles of the Hopfield Network . .... ..... .... 140
7.3 Stability Conditions of the Hopfield Network.. .... ..... .... 142
7.4 Associative Memories.... .... .... .... .... .... ..... .... 145
7.4.1 Outer Product Method. .... .... .... .... ..... .... 146
7.4.2 Pseudoinverse Matrix Method... .... .... ..... .... 147
7.4.3 Storage Capacity of Memories... .... .... ..... .... 148
7.5 Design Aspects of the Hopfield Network . .... .... ..... .... 150
7.6 Hardware Implementation Aspects .. .... .... .... ..... .... 151
7.7 Exercises. .... .... ..... .... .... .... .... .... ..... .... 153
7.8 Practical Work .... ..... .... .... .... .... .... ..... .... 154
8 Self-Organizing Kohonen Networks. .... .... .... .... ..... .... 157
8.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 157
8.2 Competitive Learning Process.. .... .... .... .... ..... .... 158
8.3 Kohonen Self-Organizing Maps (SOM) .. .... .... ..... .... 163
8.4 Exercises. .... .... ..... .... .... .... .... .... ..... .... 170
8.5 Practical Work .... ..... .... .... .... .... .... ..... .... 171
9 LVQ and Counter-Propagation Networks.... .... .... ..... .... 173
9.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 173
9.2 Vector Quantization Process... .... .... .... .... ..... .... 173
9.3 LVQ Networks (Learning Vector Quantization).... ..... .... 176
9.3.1 LVQ-1 Training Algorithm. .... .... .... ..... .... 177
9.3.2 LVQ-2 Training Algorithm. .... .... .... ..... .... 180
9.4 Counter-Propagation Networks. .... .... .... .... ..... .... 182
9.4.1 Aspects of the Outstar Layer.... .... .... ..... .... 183
9.4.2 Training Algorithm of the Counter-Propagation
Network... ..... .... .... .... .... .... ..... .... 184
9.5 Exercises. .... .... ..... .... .... .... .... .... ..... .... 185
9.6 Practical Work .... ..... .... .... .... .... .... ..... .... 186
10 ART (Adaptive Resonance Theory) Networks. .... .... ..... .... 189
10.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 189
10.2 Topological Structure of the ART-1 Network.. .... ..... .... 190
10.3 Adaptive Resonance Principle.. .... .... .... .... ..... .... 192
10.4 Learning Aspects of the ART-1 Network. .... .... ..... .... 193
10.5 Training Algorithm of the ART-1 Network ... .... ..... .... 201
10.6 Aspects of the ART-1 Original Version .. .... .... ..... .... 202
10.7 Exercises. .... .... ..... .... .... .... .... .... ..... .... 205
10.8 Practical Work .... ..... .... .... .... .... .... ..... .... 205
Part II Application of Artificial Neural Networks in Engineering
and Applied Science Problems
11 Coffee Global Quality Estimation Using Multilayer Perceptron ... 209
11.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 209
11.2 MLP Network Characteristics.. .... .... .... .... ..... .... 209
11.3 Computational Results ... .... .... .... .... .... ..... .... 211
12 Computer Network Traffic Analysis Using SNMP Protocol
and LVQ Networks. .... ..... .... .... .... .... .... ..... .... 215
12.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 215
12.2 LVQ Network Characteristics.. .... .... .... .... ..... .... 217
12.3 Computational Results ... .... .... .... .... .... ..... .... 217
13 Forecast of Stock Market Trends Using Recurrent Networks. .... 221
13.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 221
13.2 Recurrent Network Characteristics .. .... .... .... ..... .... 222
13.3 Computational Results ... .... .... .... .... .... ..... .... 223
14 Disease Diagnostic System Using ART Networks .. .... ..... .... 229
14.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 229
14.2 Art Network Characteristics ... .... .... .... .... ..... .... 230
14.3 Computational Results ... .... .... .... .... .... ..... .... 231
15 Pattern Identification of Adulterants in Coffee Powder
Using Kohonen Self-organizing Map.... .... .... .... ..... .... 235
15.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 235
15.2 Characteristics of the Kohonen Network.. .... .... ..... .... 236
15.3 Computational Results ... .... .... .... .... .... ..... .... 239
16 Recognition of Disturbances Related to Electric Power Quality
Using MLP Networks ... ..... .... .... .... .... .... ..... .... 241
16.1 Introduction .. .... ..... .... .... .... .... .... ..... .... 241
16.2 Characteristics of the MLP Network. .... .... .... ..... .... 243
16.3 Computational Results ... .... .... .... .... .... ..... .... 244