Deep Learning Applications, Volume 3 (Advances in Intelligent Systems and Computing)

328 Pages·2021·9.767 MB·English

by  M. Arif Wani (editor), Bhiksha Raj (editor), Feng Luo (editor), Dejing Dou (editor)

Advances in Intelligent Systems and Computing 1395
M. Arif Wani
Bhiksha Raj
Feng Luo
Dejing Dou   Editors
Deep
Learning
Applications,
Volume 3
Advances in Intelligent Systems and Computing
Volume 1395
SeriesEditor
JanuszKacprzyk,SystemsResearchInstitute,PolishAcademyofSciences,
Warsaw,Poland
AdvisoryEditors
NikhilR.Pal,IndianStatisticalInstitute,Kolkata,India
RafaelBelloPerez,FacultyofMathematics,PhysicsandComputing,
UniversidadCentraldeLasVillas,SantaClara,Cuba
EmilioS.Corchado,UniversityofSalamanca,Salamanca,Spain
HaniHagras,SchoolofComputerScienceandElectronicEngineering,
UniversityofEssex,Colchester,UK
LászlóT.Kóczy,DepartmentofAutomation,SzéchenyiIstvánUniversity,
Gyor,Hungary
VladikKreinovich,DepartmentofComputerScience,UniversityofTexas
atElPaso,ElPaso,TX,USA
Chin-TengLin,DepartmentofElectricalEngineering,NationalChiao
TungUniversity,Hsinchu,Taiwan
JieLu,FacultyofEngineeringandInformationTechnology,
UniversityofTechnologySydney,Sydney,NSW,Australia
PatriciaMelin,GraduateProgramofComputerScience,TijuanaInstitute
ofTechnology,Tijuana,Mexico
NadiaNedjah,DepartmentofElectronicsEngineering,UniversityofRiodeJaneiro,
RiodeJaneiro,Brazil
NgocThanhNguyen ,FacultyofComputerScienceandManagement,
WrocławUniversityofTechnology,Wrocław,Poland
JunWang,DepartmentofMechanicalandAutomationEngineering,
TheChineseUniversityofHongKong,Shatin,HongKong
Theseries“AdvancesinIntelligentSystemsandComputing”containspublications
on theory, applications, and design methods of Intelligent Systems and Intelligent
Computing.Virtuallyalldisciplinessuchasengineering,naturalsciences,computer
and information science, ICT, economics, business, e-commerce, environment,
healthcare,lifesciencearecovered.Thelistoftopicsspansalltheareasofmodern
intelligentsystemsandcomputingsuchas:computationalintelligence,softcomput-
ingincludingneuralnetworks,fuzzysystems,evolutionarycomputingandthefusion
oftheseparadigms,socialintelligence,ambientintelligence,computationalneuro-
science, artificial life, virtual worlds and society, cognitive science and systems,
Perception and Vision, DNA and immune based systems, self-organizing and
adaptive systems, e-Learning and teaching, human-centered and human-centric
computing, recommender systems, intelligent control, robotics and mechatronics
including human-machine teaming, knowledge-based paradigms, learning para-
digms,machineethics,intelligentdataanalysis,knowledgemanagement,intelligent
agents, intelligent decision making and support, intelligent network security, trust
management,interactiveentertainment,Webintelligenceandmultimedia.
The publications within “Advances in Intelligent Systems and Computing” are
primarily proceedings of important conferences, symposia and congresses. They
cover significant recent developments in the field, both of a foundational and
applicable character. An important characteristic feature of the series is the short
publication time and world-wide distribution. This permits a rapid and broad
disseminationofresearchresults.
Indexed by DBLP, INSPEC, WTI Frankfurt eG, zbMATH, Japanese Science and
TechnologyAgency(JST).
AllbookspublishedintheseriesaresubmittedforconsiderationinWebofScience.
Moreinformationaboutthisseriesathttp://www.springer.com/series/11156
· · ·
M. Arif Wani Bhiksha Raj Feng Luo
Dejing Dou
Editors
Deep Learning Applications,
Volume 3
Editors
M.ArifWani BhikshaRaj
DepartmentofComputerScience LanguageTechnologiesInstitute
UniversityofKashmir CarnegieMellonUniversity
Srinagar,India Pittsburgh,PA,USA
FengLuo DejingDou
ClemsonUniversity UniversityofOregon
Clemson,SC,USA Eugene,OR,USA
ISSN2194-5357 ISSN2194-5365 (electronic)
AdvancesinIntelligentSystemsandComputing
ISBN978-981-16-3356-0 ISBN978-981-16-3357-7 (eBook)
https://doi.org/10.1007/978-981-16-3357-7
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNature
SingaporePteLtd.2022
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsoftranslation,reprinting,reuse
ofillustrations,recitation,broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,and
transmissionorinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilar
ordissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublisher,theauthorsandtheeditorsaresafetoassumethattheadviceandinformationinthisbook
arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor
theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany
errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional
claimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
Machinelearningalgorithmshaveinfluencedmanyaspectsofourday-to-dayliving
and transformed major industries around the world. At the frontier of machine
learning innovation are deep learning systems, a class of multi-layered networks
capable of automatically learning meaningful hierarchical representations from a
variety of structured and unstructured data. Breakthroughs in deep learning allow
us to generate new representations, extract knowledge, and draw inferences from
raw images, video streams, text and speech, time series, and other complex data
types.Thesepowerfuldeeplearningmethodsarebeingappliedtonewandexciting
real-worldproblemsinmedicaldiagnostics,factoryautomation,publicsafety,envi-
ronmental sciences, autonomous transportation, military applications, and much
more.
Thisbookexploressomeofthelatestapplicationsindeeplearningandincludes
avarietyofarchitecturesandnoveldeeplearningtechniques,whicharedescribedin
twelvechapters.EachchapterisaccompaniedbyPythonSoftware/Codealongwith
testdatasets.Asummaryofthesechaptersisgivenbelow.
Chapter1discussesDeepRapidClassAugmentation(DeepRCA)fordeepneural
networks.TheapproachattemptstoeliminatetheissuesassociatedwithCatastrophic
Forgetting. Deep RCA’s ability to train only on the new task data while jointly
optimizingacrossalltasksisshowntoreduceaugmentationtrainingtimesanddata
storagerequirements.ThismakesDeepRCAwellsuitedforapplicationsthatrequire
rapidsequentiallearningandwheredatastoragecapacityislimited.
Chapter2presentsananalysisofusingtheBERTmodelforfivelanguageswith
richmorphology(Finnish,Czech,Hungarian,Turkish,Japanese).Theworkevaluates
cross-lingual, multilingual, and monolingual BERT models and three non-BERT-
basedmodelsonfivemorphologicallyrichlanguagesandcomparestheperformance
ofthemodelsontheEnglishlanguage.
Chapter3discussesfusionofRGBandDepthimagesforimprovingtheperfor-
manceofobjectdetectionincomplexscenes.Earlyfusionarchitectureispresented
thatemploysanunsupervisedlearningdepthestimationtechniquetoautomatically
infer a dense depth image from a single RGB input image. The depth image is
v
vi Preface
concatenated to the RGB image to perform object detection using deep learning
models.
Chapter4presentsdimensionestimationusingAutoencoders(AEs).Dimension
estimationattemptstoestimatetheintrinsicdimensionalityornumberoflatentvari-
ables in a set of measurements of a random vector. The work discusses AE archi-
tectural choices and regularization techniques that allow it to transform AE latent
layerrepresentationsintoestimatesofintrinsicdimension.Theeffectivenessofthe
techniques is demonstrated on benchmark image processing problems, analysis of
financialmarkets,andnetworksecurity.
Chapter 5 discusses the problem of high demand for computing and storing
resources associated with the deep learning approach. Transforming the original
large pre-trained networks into new smaller models, by utilizing Model Compres-
sionandAcceleration(MCA)techniquesisdescribed.WithintheMCAframework,
a clustering-based approach is discussed that is able to increase the number of
employedcentroids/representativesforanaccelerationgain.Thetheoreticalacceler-
ationgainsarepresentedandthekeysystemhyper-parametersthataffectthegainare
identified.Extensiveevaluationstudiesarecarriedoutusingvariousstate-of-the-art
DNN models trained on image classification and object detection tasks to validate
theeffectivenessoftheclusteringmethodinMCA.
Chapter 6 presents Deep Learning-based time series forecasting. The work
describesRNN,CNN,andTransformerbasedstructuresfortimeseriesforecasting.
Experimentsareperformedtocomparethesetechniquesonfourdifferentpublicly
availabledatasets.
Chapter7discussesDeepEvidentialActiveLearning(DEAL)forImageClassifi-
cation.ActiveLearning(AL)ispresentedtomitigatetheproblemoflimitedlabeled
data.AnovelALalgorithmthatefficientlylearnsfromunlabeleddatabycapturing
highpredictionuncertaintyisdiscussed.ByreplacingtheSoftMaxstandardoutput
ofaCNNwiththeparametersofaDirichletdensity,themodellearnstoidentifydata
instancesthatcontributeinimprovingmodelperformanceduringtraining.Results
of several experiments performed on publicly available data are presented. On the
real-worldmedicaldatasetsforpneumoniadetectioninchestX-rayimagesarealso
presented.
Chapter8discussesConvolutionalNeuralNetworkwithLatentBinarizationfor
large-scalemulti-classclassification.Error-correctingoutputcodesareincorporated
intoconvolutionalneuralnetworksbyinsertingalatentbinarizationlayerintheCNN
classification layer. This approach encapsulates both encoding and decoding steps
of error-correcting output codes into a single CNN architecture that is capable of
discoveringanoptimalcodingmatrixduringtraining.Theresultsofapplyingthese
modelstoseveralimagerecognitiontasksarepresented.
Chapter 9 discusses a framework for deployment of pre-trained deep learning
models on robots. An open-source framework, that is easy to deploy and is inde-
pendent of the underlying deep learning models, is employed. The framework is
testedandevaluatedonsevenpre-traineddeeplearningmodelsforhazarddetection
in supermarket floors in a simulated environment. Experimental results of testing
Preface vii
the framework in deploying the deep learning models in Robot Operating System
environmentarepresented.
Chapter10discussesextendedFeatureAugmentedandTransformedGAN(FAT-
GAN)modelforconditionalsimulationofElectron–ProtonScatteringEventswith
Variate Beam Energies. The model produces inclusive event feature distributions
andcorrelationsforacontinuousrangeofreactionenergiesbyautomaticallyinter-
polatingandextrapolatingfromasetoftrainedenergies.Acontinuousenergyfeature
representationisemployedtoenablethenetworkstolearnthedistributionrelation-
shipsbetweendifferentreactionenergies,layingthegroundworkforaccessingevents
atuntrainedenergies.
Chapter 11 discusses the practical challenges of performing multi-agent data
processing. The issues of coordinating connected Distributed Energy Resources
(DERs) in the Internet of Things (IoT) are discussed. The results on case studies
regardingtheissuesofloadforecastingandcoordinationchallengesarepresented.
Chapter12discussestheuseofDeepLearningapproachfordonorjourneytasks.
ExperimentalresultsofusingseveralvariantsofRecurrentNeuralNetworks(RNNs)
andConvolutionalNeuralNetworks(CNNs),aswellaswithvarioussizedwindows,
ontheabovetaskarepresented.
PythonSoftware/CodeAssociatedwitheachchapter
Srinagar,India M.ArifWani
Pittsburgh,USA BhikshaRaj
Clemson,USA FengLuo
Eugene,USA DejingDou
Contents
DeepRapidClassAugmentation;ANewProgressiveLearning
ApproachthatEliminatestheIssueofCatastrophicForgetting ........ 1
HannaWitzgall
AComprehensiveAnalysisofSubwordContextualEmbeddings
forLanguageswithRichMorphology ............................... 31
ArdaAkdemir,TetsuoShibuya,andTungaGüngör
RGBandDepthImageFusionforObjectDetectionUsingDeep
Learning ......................................................... 73
FahimehFarahnakianandJukkaHeikkonen
DimensionEstimationUsingAutoencodersandApplication ........... 95
NitishBahadur,BrianLewandowski,andRandyPaffenroth
ANewClustering-BasedTechniquefortheAccelerationofDeep
ConvolutionalNetworks ............................................ 123
ErionVasilisPikoulis, ChristosMavrokefalidis, StavrosNousias,
andArisS.Lalos
DeepLearning-BasedTimeSeriesForecasting ....................... 151
KushagraAgarwal,LalasaDheekollu,GauravDhama,AnkurArora,
SiddharthaAsthana,andTanmoyBhowmik
DEAL:DeepEvidentialActiveLearningforImageClassification ...... 171
PatrickHemmer,NiklasKühl,andJakobSchöffer
LB-CNN:ConvolutionalNeuralNetworkwithLatentBinarization
forLargeScaleMulti-classClassification ............................ 193
TimothyReeseandYuMichaelZhu
EfficientDeploymentofDeepLearningModelsonAutonomous
RobotsintheROSEnvironment .................................... 215
M.G.SarwarMurshed,JamesJ.Carroll,NazarKhan,andFarazHussain
ix
x Contents
cFAT-GAN:ConditionalSimulationofElectron–ProtonScattering
Events with Variate Beam Energies by a Feature Augmented
andTransformedGenerativeAdversarialNetwork ................... 245
LuisaVelasco, EvanMcClellan, N.Sato, PawelAmbrozewicz,
TianboLiu, W.Melnitchouk, MichelleKuchera, YasirAlanazi,
andYaohangLi
Building Power Grid 2.0: Deep Learning and Federated
ComputationsforEnergyDecarbonizationandEdgeResilience ........ 263
CarolynGoodman,JesseThornburg,ShankarKoduvayurRamaswami,
andJavadMohammadi
Deep Learning the Donor Journey with Convolutional
andRecurrentNeuralNetworks .................................... 295
GregLee,AjithKumarRaghavan,andMarkHobbs
AuthorIndex ...................................................... 321