Impact of Class Assignment on Multinomial Classification Using Multi-Valued Neurons

89 Pages·2022·2.725 MB·English

by  Julian Knaup

BestMasters
Julian Knaup
Impact of Class
Assignment
on Multinomial
Classification Using
Multi-Valued Neurons
BestMasters
Mit „BestMasters“ zeichnet Springer die besten Masterarbeiten aus, die an renom-
mierten Hochschulen in Deutschland, Österreich und der Schweiz entstanden sind.
Die mit Höchstnote ausgezeichneten Arbeiten wurden durch Gutachter zur Veröf-
fentlichung empfohlen und behandeln aktuelle Themen aus unterschiedlichen
Fachgebieten der Naturwissenschaften, Psychologie, Technik und Wirtschaftswis-
senschaften. Die Reihe wendet sich an Praktiker und Wissenschaftler gleicherma-
ßen und soll insbesondere auch Nachwuchswissenschaftlern Orientierung geben.
Springer awards „BestMasters“ to the best master’s theses which have been com-
pleted at renowned Universities in Germany, Austria, and Switzerland. The studies
received highest marks and were recommended for publication by supervisors.
They address current issues from various fields of research in natural sciences,
psychology, technology, and economics. The series addresses practitioners as well
as scientists and, in particular, offers guidance for early stage researchers.
Julian Knaup
Impact of Class
Assignment on
Multinomial
Classification Using
Multi-Valued Neurons
Julian Knaup
Lemgo, Germany
ISSN 2625-3577 ISSN 2625-3615 (electronic)
BestMasters
ISBN 978-3-658-38954-3 ISBN 978-3-658-38955-0 (eBook)
https://doi.org/10.1007/978-3-658-38955-0
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Fachmedien
Wiesbaden GmbH, part of Springer Nature 2022
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher,
whether the whole or part of the material is concerned, specifically the rights of translation,
reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any
other physical way, and transmission or information storage and retrieval, electronic adaptation,
computer software, or by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in
this book are believed to be true and accurate at the date of publication. Neither the publisher
nor the authors or the editors give a warranty, expressed or implied, with respect to the material
contained herein or for any errors or omissions that may have been made. The publisher remains
neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Responsible Editor: Stefanie Eggert
This Springer Vieweg imprint is published by the registered company Springer Fachmedien
Wiesbaden GmbH, part of Springer Nature.
The registered company address is: Abraham-Lincoln-Str. 46, 65189 Wiesbaden, Germany
Abstract
Multilayerneuralnetworksbasedonmulti-valuedneurons(MLMVNs)havebeenproposed
to combine the advantages of complex-valued neural networks with a plain derivative-
free learning algorithm. In addition, multi-valued neurons (MVNs) offer a multi-valued
threshold logic resulting in the ability to replace multiple conventional output neurons
in classification tasks. Therefore, several classes can be assigned to one output neuron.
ThisthesisintroducesanovelapproachtoassignmultipleclassesonnumerousMVNsin
theoutputlayer. Itwasfoundthatclassesthatpossesssimilarityshouldbeallocatedto
thesameneuronandarrangedadjacenttoeachotherontheunitcircle. SinceMLMVNs
require input data located on the unit circle, two employed transformations are reeval-
uated. The min-max scaler utilizing the exponential function obtains decent results for
numerical data. The 2D discrete Fourier transform restricting to the phase information
was found to be unsuitable for image recognition. Even if this transformation approach
could be improved, it loses key properties such as translational invariance by discarding
themagnitudeinformation. TheevaluationwasperformedontheSDDandtheFashion
MNISTdataset.
V
Contents
Abstract V
Table of Contents VII
List of Abbreviations IX
List of Symbols XI
1 Introduction 1
1.1 ResearchObjectives. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 ScopeofWork. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 StructureoftheWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Preliminaries 4
2.1 MachineLearningBasics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 ArtificialNeuralNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2.2 ArchitectureandTraining . . . . . . . . . . . . . . . . . . . . . . . 8
2.2.3 Multi-ValuedNeuron . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.4 MultilayerNeuralNetworkBasedonMulti-ValuedNeurons. . . . . 13
2.3 LossFunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3 Scientific State of the Art 16
3.1 MultinomialClassificationinReal-ValuedNeuralNetworks . . . . . . . . . 17
3.2 MultinomialClassificationinComplex-ValuedNeuralNetworks . . . . . . 19
3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4 Approach 26
4.1 DataPreparationandProcessing . . . . . . . . . . . . . . . . . . . . . . . 26
4.2 FeatureEngineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3.1 ClassAllocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.3.2 ClassArrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.4 ModelTraining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
VII
Contents
4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5 Evaluation 39
5.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.2 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2.1 SensorlessDriveDiagnosis . . . . . . . . . . . . . . . . . . . . . . . 40
5.2.2 FashionMNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3.1 ClassAllocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3.2 ClassArrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6 Conclusion and Outlook 50
6.1 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
A Results 52
A.1 BinaryClassification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
A.2 ClassAllocation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
A.3 ClassArrangement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
Bibliography 65
List of Figures 72
List of Tables 76
VIII
List of Abbreviations
2D DFT TwoDimensionalDiscreteFourierTransform
Adam AdaptiveMomentEstimation
AI ArtificialIntelligence
ANN ArtificalNeuralNetwork
AutASS AutonomousDriveTechnologybySensorFusionforIntelligent,
Simulation-basedProductionFacilityMonitoringandControl(abbr.
fromGerman)
BP Backpropagation
CNN ConvolutionalNeuralNetwork
CPU CentralProcessingUnit
CVNN Complex-ValuedArtificalNeuralNetwork
DT DecisionTree
FDR FisherDiscriminantRatio
FN FalseNegative
FP FalsePositive
GD GradientDescent
GPU GraphicsProcessingUnit
LDA LinearDiscriminantAnalysis
LSTM LongShort-TermMemory
MLMVN MultilayerFeedforwardNeuralNetworkBasedonMulti-ValuedNeurons
MNIST ModifiedNationalInstituteofStandardsandTechnology
MSE Mean-Square-Error
MVN Multi-ValuedNeuron
NLP NaturalLanguageProcessing
PCA PrincipalComponentAnalysis
ReLU RectifiedLinearUnit
RMSE Root-Mean-Square-Error
RNN RecurrentNeuralNetworks
IX
ListofAbbreviations
RVNN Real-ValuedArtificalNeuralNetwork
SDD SensorlessDriveDiagnosis
SVD SingularValueDecomposition
SVM Support-VectorMachine
t-SNE t-DistributedStochasticNeighborEmbedding
TN TrueNegative
TP TruePositive
UMAP UniformManifoldApproximationandProjectionforDimension
Reduction
XOR ExclusiveDisjunction
X