Table Of Content

Smart Innovation, Systems and Technologies 136 Valentina Emilia Balas Sanjiban Sekhar Roy Dharmendra Sharma Pijush Samui Editors Handbook of Deep Learning Applications Smart Innovation, Systems and Technologies Volume 136 Series editors Robert James Howlett, Bournemouth University and KES International, Shoreham-by-sea, UK e-mail: [email protected] Lakhmi C. Jain, Centre for Artificial Intelligence, Faculty of Engineering and Information Technology, University of Technology Sydney, Broadway, NSW, Australia; Faculty of Science, Technology and Mathematics, University of Canberra, Canberra, ACT, Australia; KES International, UK e-mail: [email protected]; [email protected] The Smart Innovation, Systems and Technologies book series encompasses the topics of knowledge, intelligence, innovation and sustainability. The aim of the seriesistomakeavailableaplatformforthepublicationofbooksonallaspectsof single and multi-disciplinary research on these themes in order to make the latest resultsavailableinareadily-accessibleform.Volumesoninterdisciplinaryresearch combining two or more of these areas is particularly sought. Theseriescoverssystemsandparadigmsthatemployknowledgeandintelligence inabroadsense.Itsscopeissystemshavingembeddedknowledgeandintelligence, whichmaybeappliedtothesolutionofworldproblemsinindustry,theenvironment and the community. It also focusses on the knowledge-transfer methodologies and innovationstrategiesemployedtomakethishappeneffectively.Thecombinationof intelligent systems tools and a broad range of applications introduces a need for a synergy of disciplines from science, technology, business and the humanities. The series will include conference proceedings, edited collections, monographs, hand- books, reference books, and other relevant types of book in areas of science and technology where smart systems and technologies can offer innovative solutions. Highqualitycontentisanessentialfeatureforallbookproposalsacceptedforthe series. It is expected that editors of all accepted volumes will ensure that contributionsaresubjectedtoanappropriatelevelofreviewingprocessandadhere to KES quality principles. More information about this series at http://www.springer.com/series/8767 Valentina Emilia Balas (cid:129) Sanjiban Sekhar Roy Dharmendra Sharma (cid:129) (cid:129) Pijush Samui Editors Handbook of Deep Learning Applications 123 Editors Valentina EmiliaBalas SanjibanSekhar Roy AurelVlaicu University of Arad Schoolof Computer Science Arad,Romania andEngineering VelloreInstitute of Technology Dharmendra Sharma Vellore, Tamil Nadu,India University of Canberra Bruce,ACT, Australia Pijush Samui Department ofCivil Engineering National Institute ofTechnology Patna Patna, Bihar, India ISSN 2190-3018 ISSN 2190-3026 (electronic) Smart Innovation,Systems andTechnologies ISBN978-3-030-11478-7 ISBN978-3-030-11479-4 (eBook) https://doi.org/10.1007/978-3-030-11479-4 LibraryofCongressControlNumber:2018967433 ©SpringerNatureSwitzerlandAG2019 Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar methodologynowknownorhereafterdeveloped. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Contents Designing a Neural Network from Scratch for Big Data Powered by Multi-node GPUs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Alcides Fonseca and Bruno Cabral Deep Learning for Scene Understanding . . . . . . . . . . . . . . . . . . . . . . . . 21 Uzair Nadeem, Syed Afaq Ali Shah, Ferdous Sohel, Roberto Togneri and Mohammed Bennamoun An Application of Deep Learning in Character Recognition: An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Sana Saeed, Saeeda Naz and Muhammad Imran Razzak Deep Learning for Driverless Vehicles . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Cameron Hodges, Senjian An, Hossein Rahmani and Mohammed Bennamoun Deep Learning for Document Representation. . . . . . . . . . . . . . . . . . . . . 101 Mehran Kamkarhaghighi, Eren Gultepe and Masoud Makrehchi Applications of Deep Learning in Medical Imaging . . . . . . . . . . . . . . . . 111 Sanjit Maitra, Ratul Ghosh and Kuntal Ghosh Deep Learning for Marine Species Recognition . . . . . . . . . . . . . . . . . . . 129 Lian Xu, Mohammed Bennamoun, Senjian An, Ferdous Sohel and Farid Boussaid Deep Molecular Representation in Cheminformatics . . . . . . . . . . . . . . . 147 Peng Jiang, Serkan Saydam, Hamed Lamei Ramandi, Alan Crosky and Mojtaba Maghrebi A Brief Survey and an Application of Semantic Image Segmentation for Autonomous Driving . . . . . . . . . . . . . . . . . . . . . . . . . 161 Çağrı Kaymak and Ayşegül Uçar v vi Contents Phase Identification and Workflow Modeling in Laparoscopy Surgeries Using Temporal Connectionism of Deep Visual Residual Abstractions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 Kaustuv Mishra, Rachana Sathish and Debdoot Sheet Deep Learning Applications to Cytopathology: A Study on the Detection of Malaria and on the Classification of Leukaemia Cell-Lines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 G. Gopakumar and Gorthi R. K. Sai Subrahmanyam Application of Deep Neural Networks for Disease Diagnosis Through Medical Data Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Alper Baştürk, Hasan Badem, Abdullah Caliskan and Mehmet Emin Yüksel Why Dose Layer-by-Layer Pre-training Improve Deep Neural Networks Learning?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Seyyede Zohreh Seyyedsalehi and Seyyed Ali Seyyedsalehi Springer: Deep Learning in eHealth. . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Peter Wlodarczak Deep Learning for Brain Computer Interfaces. . . . . . . . . . . . . . . . . . . . 333 Ankita Bose, Sanjiban Sekhar Roy, Valentina Emilia Balas and Pijush Samui Reducing Hierarchical Deep Learning Networks as Game Playing Artefact Using Regret Matching . . . . . . . . . . . . . . . . . . . . . . . . 345 Arindam Chaudhuri and Soumya K. Ghosh Deep Learning in Gene Expression Modeling . . . . . . . . . . . . . . . . . . . . 363 Dinesh Kumar and Dharmendra Sharma Designing a Neural Network from Scratch for Big Data Powered by Multi-node GPUs AlcidesFonsecaandBrunoCabral 1 Introduction Lately,MachineLearninghastakenacrucialroleinthesocietyindifferentvertical sectors.Forcomplexproblemswithhigh-dimensionality,DeepLearninghasbecome an efficient solution for learning in the context of supervisioned learning. Deep Learning[1]consistsinusingArtificialNeuralNetworks(ANNorNN)withseveral hiddenlayers,typicallyalsowithalargenumberofnodesineachlayer. ANNs have initially been proposed in 1943 [2], but only recently have been gainingpopularityduetodecreasingstoragecostsandtheincreaseofcomputational power,bothinCPUandGPUs.Nowadays,ANNsareusedforseveraltasks,suchas imageclassification[3],characterrecognitioninscanneddocuments[4],predicting bankruptcy[5]orhealthcomplications[6].Morerecently,ANNshavebeenthebasis forthesoftwareusedinself-drivingvehicles[7]. In complex problems, both in terms of diversity in instances and in number of features/classes,networksalsohaveamorecomplexstructureandmoreexpensive training process. It is recommended to use a number of instances three orders of magnitude higher than the number of features [8]. Training an ANN consists on applyingtheNNtoseveralbatchesofmultipleinstancesasmanytimesasnecessary untilagood-enoughweightdistributionisobtained.Thus,trainingacomplexANN is a computationally intensive operation in terms of processing, memory and disk usage.Astheamountofdataavailablefortraininggoesaboveaterabyte,itbecomes BigDataproblem[9]. B A.Fonseca( ) LASIGE,FaculdadedeCiênciasdaUniversidadedeLisboa,Lisbon,Portugal e-mail:[email protected] B.Cabral CISUC,DepartamentodeEngenhariaInformática,FaculdadedeCiênciaseTecnologiada UniversidadedeCoimbra,Coimbra,Portugal e-mail:[email protected] ©SpringerNatureSwitzerlandAG2019 1 V.E.Balasetal.(eds.),HandbookofDeepLearningApplications, SmartInnovation,SystemsandTechnologies136, https://doi.org/10.1007/978-3-030-11479-4_1 2 A.FonsecaandB.Cabral Thetwomostcommonandefficientapproachesforperformingthiscomputation in useful time is to distribute work across different machines, and to use GPUs to perform ANN training. GPUs are used in ANN training because they are more efficientthanCPUsformatricialoperations(whattheyweredesignedfor,inthefield ofGraphicalComputation),andANNapplicationandtrainingcanbedescribedin thoseoperations. Distributing the computation of the training is also used to train with more instancesthatfitasinglemachine(usuallyintermsofmemory,asCPUsandGPUs havelimitedvolatilememories).Thisapproachispossiblebecausetrainingcanbe subdividedinembarrassinglyparallelsub-problemsthatcanbecombinedatalatter stage. This chapter will cover the design and implementation of a distributed CPU andGPU-backedDeepArtificialNeuralNetworkforclassificationproblems.This chapter aims to help researchers and practitioners, who are looking to implement alternativeANNconfigurationsormodels,increatingefficientGPU-enabledcode. This approach is an alternative to existing tutorials that focus on the usage of readily-available ANN toolkits, such as as Tensorflow [10], Theano [11] or Torch [12], among many others [13]. While these tools make the task of creating ANNs simple without much effort, the space of possible resulting ANN is limited by the framework.Inthischapter,thegoalistoequippractitionerswiththetoolstodevelop newanddifferentapproachestoANNs,sotheycanexploredifferentresearchlines inANNarchitectureortrainingsystems. ThischapterwillbrieflyintroduceANNsandtheback-propagationtrainingpro- cess (Sect. 2 briefly introduces ANNs and the back-propagation training process) alongwithitsmathematicalbackgroundforANNs(Sect.3).Then,wewillpresent theproblemthatwillserveasanexamplefordevelopingourcustomArtificialNeural Network. Then, we will cover three different phases of the implementation of the ANN: a single-CPU version (Sect. 5); a parallel, distributed version (Sect. 6); and aGPUversion(Sect.7).Finally,wewilldiscussthegoodnessoftheimplemented design (Sect. 8) and conclude this chapter (Sect. 9) with final remarks and future work. Therunnablesourcecodeforthischapterisalsomadeavailableonlineathttps:// github.com/alcides/bigdatagpunn. 2 APrimeronNeuralNetworks Neural Networks are used in classification (and also regression) problems, more frequently in supervisioned problems. In these cases, there is an existing dataset with classification labels (classes) in each instance. From this data set, a learning algorithm should infer how to classify new unlabelled instance from its features. Featuresareasetofinstanceproperties(whenclassifyingthetypeoffruit,thecolor andlengthofthefruitarecandidatefeatures.Orange,appleandbananaaresomeof thepossibleclasses. DesigningaNeuralNetworkfromScratchforBigData… 3 Input Layer Hidden Ouput Layer Layer Fig.1 AnANNwithfourinputneurons,onesinglehiddenlayerwithfourneuronsandtwooutput neurons WhenusingANNsasthelearningalgorithms,featuresaretheinputoftheANNs, andtherecanbeoneoutputforeachclass.Whenfeedingthefeaturesofanunlabelled new fruit through the ANNs, the highest scoring class output node should be the correct one (Assuming a correct prediction, which unfortunately is not always the case). ANNs have a directional architecture, in which the processing flows from the inputlayertotheoutputlayer,passingthroughseveralhiddenlayers.Theinputlayer hasoneneuronforeachconsideredfeature.Hiddenlayerscanhaveanynumberof neuronsthatreceiveasinputtheoutputofthepreviouslayer.Finally,theoutputlayer has as many neurons as desirable for the problem at hand, and receive input from thepreviouslayeroutputs.Anexampleofaverysimpleneural-networkisdepicted inFig.1,consistingoffourinputneurons,fourneuronsinthehiddenlayerandtwo outputneurons. Thisisanexampleofasingle-hidden-layernetwork,butDeepNeuralNetworks havealargenumberofhiddenlayers,representingtheabstractioncapabilityofthe network.Thelayoutofthenetworkconsistsonthenumberoflayersandthenumber ofneuronsoneachlayers. When the number of features increases (high dimensionality), the number of neuronsinthehiddenlayersincreasesaswell,inordertocompensateforthepossible interactions of input neurons [14]. However, a rule of thumb is to use only one hiddenlayer[8]withthesamenumberofhiddenneuronsasthereareinputneurons. Nonetheless, one might prefer to explore more recent approaches to this problem [15, 16], since nowadays computers are much more powerful than they were in 1995,whenthisrulewasfirstpublished.