Analog VLSI Neural Networks: A Special Issue of Analog Integrated Circuits and Signal Processing PDF
Preview Analog VLSI Neural Networks: A Special Issue of Analog Integrated Circuits and Signal Processing
ANALOG VLSI NEURAL NETWORKS edited by Yoshiyasu Takefuji Case Westem Reserve University Keio University, Japan A Special Issue of ANALOG INTEGRATED CIRCUITS AND SIGNAL PROCESSING Reprinted from ANALOG INTEGRATED CIRCurrs AND SIGNAL PROCESSING VoI. 2, No. 4 (1992) Springer-Science+Business Media, B.V. THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE ANALOG CIRCUITS AND SIGNAL PROCESSING Consulting Editor Mohammed Ismail Ohio State University Related tides: ANALOG CMOS FILTERS FOR VERY HIGH FREQUENCIES, Bram Nauta ISBN: 0-7923-9272-8 ANALOG VLSI NEURAL NETWORKS, Yoshiyasu Takefuji ISBN: 0-7923-9273-8 INTRODUCTION TO THE DESIGN OF TRANSCONDUCTOR-CAPACITOR FILTERS, Jaime Kardontchik ISBN: 0-7923-9195-0 VLSI DESIGN OF NEURAL NETWORKS, Ulrich Ramacher, Ulrich Rucken ISBN: 0-7923-9127-6 LOW-NOISE WIDE-BAND AMPLIFIERS IN BIPOLAR AND CMOS TECHNOLOGIES, Z.Y. Chang, Willy Sansen ISBN: 0-7923-9096-2 ANALOG INTEGRATED CIRCUITS FOR COMMUNICATIONS: Principles, Simulation and Design, Donald O. Pederson, Kartikeya Mayaram ISBN: 0-7923-9089-X SYMBOLIC ANALYSIS FOR AUTOMATED DESIGN OF ANALOG INTEGRATED CIRCUITS, Georges Gielen, Willey Sansen ISBN: 0-7923-9161-6 AN INTRODUCTION TO ANALOG VLSI DESIGN AUTOMATION, Mohammed Ismail, Jose Franca ISBN: 0-7923-9071-7 STEADY-STATE METHODS FOR SIMULATING ANALOG AND MICROWAVE CIRCUITS, Kenneth S. Kundert, Jacob White, Alberto Sangiovanni-Vincentelli ISBN: 0-7923-9069-5 MIXED-MODE SIMULATION: Algorithms and Implementation, Reseve A. Saleh, A. Richard Newton ISBN: 0-7923-9107-1 ANALOG VLSI IMPLEMENTATION OF NEURAL NETWORKS, Carver A. Mead, Mohammed Ismail ISBN: 0-7923-9040-7 Contents Special Issue: Analog VLSI Neural Networks Guest Editor: Yoshiyasu Takefuji Guest Editorial Yoshiyasu Takefuji 1 Analog Computational Models of Concept Formation Yoh-Han Pao and Wassim Hafez 3 An Analog BiCMOS Hopfield Neuron Paul W. Hollis and John J. Paulos 11 Full Analog CMOS Integration of Very Large Time Constants for Synaptic Transfer in Neural Networks P Kinget, M. Steyaert and J. Van der Spiegel 19 A Hierarchical Clustering Network Based on a Model of Olfactory Processing PA. Shoemaker, CG. Hutchens and S. Paul 35 CMOS Analog/Digital Circuits of the Hysteresis McCulloch-Pitts Neuron for Ramsey Numbers .... , Yong Beom Cho, Kazuhiro Tsuchiya and Yoshiyasu Takefuji 51 Competitive Learning in Asynchronous-Pulse-Density Integrated Circuits David A. Watola and Jack L. Meador 61 A Programmable Analog CMOS Synapse for Neural Networks Seokjim Kim, Yong-Chul Shin, Naidu C.R. Bogineni and Ramalingam Sridhar 83 Two-Stage Neural Network Architecture for Feedback Control of Dynamic Systems Stephen M. Phillips and Christop Müller-Dott 91 Temporal Signal Processing with High-Speed Hybrid Analog-Digital Neural Networks Mark DeYong, Thomas C Eskridge and Chris Fields 105 A Super Parallel Sorter Using a Binary Neural Network with AND-OR Synaptic Connections Manabu Yamada, Tohru Nakagawa and Hajime Kitagawa 127 Library of Congress Cataloging-in-PubUcation Data Analog VLSI neural networks / edited by Yoshiyasu Thkefuji. p. cm. - (The Kluwer international seriesin engineering and computer science. Analog circuits and signal processing) ISBN 978-1-4613-6592-1 ISBN 978-1-4615-3582-9 (eBook) DOI 10.1007/978-1-4615-3582-9 1. Neural networks (Computer science) 2. Integrated circuits-Very large scale integration. 1. Thkefuji, Yoshiyasu, 1955- ll. Series. QA76.87.A53 1992 92-27780 621.39'S-dc20 CIP Copyright © 1993 by Springer Science+Business Media Dordrecht Originally published by Kluwer Academic Publishers in 1993 Softcover reprint of the hardcover 1s t edition 1993 A11 rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written per mission ofthe publisher, Springer Science+Business Media, LLC. Printed on acid-free paper. Analog Integrated Circuits and SignalProcessing2, 263-264(1992) © 1992 Kluwer Academic Publishers, Boston. Manufactured inThe Netherlands. Guest Editorial Weare happy topresentthespecial issueonanalog VLSIneuralnetworks intheInternationalJournalofAnalog Integrated Circuits and Signal Processing. We have received more than 15 papers, and 10 papers are accepted inthisissue. Wewouldliketoexpressourthankstothefollowing associateeditors: ProfessorJunWang(University of North Dakota), Dr. Paul Hollis (North Carolina State University), Professor Meng Hiot Lim (Nanyang TechnologicalUniversity,Singapore),Dr. KuoChunLee(CirrusLogic), anDr. YongB. Cho(CaseWesternReserve University). Please enjoy this issue. Thank you. Yoshiyasu Takefuji Guest Editor Scanning this issue: Inapaperentitled "Analog computational models ofconceptformation," Paoand Hafezproposean inductive concept learning method which is suitable for parallel computing and analog VLSI neural network circuitry. An analog BiCMOS Hopfield neuron is depicted by Hollis and Paulos. The analog circuit with linear input synapses is implemented for the standard Hopfield neuronmodel. SPICE simulations and large scale integration issues are discussed in the paper. Kinget, Steyaert, and Spiegelpresent a paperentitled "Full analog CMOS integration ofverylarge timecon stantsfor synaptictransferinneuralnetworks." Intheirpaperamethodforthefullon-chipanalogimplementation oflargetimeconstantsinaCMOStechnologyisdescribed. Thetimeconstantsareusedfor delayedsynaptictransfer in neural networks for signal processing. Apaperentitled ''Ahierarchical clustering networkbasedonamode ofolfactory processing," by Shoemaker, Hutchens, and Patildescribesadirectanalog implementationofa neural networkmodelofolfactoryprocessing. Current-mode circuit designs to implementthe required functions in CMOS integrated circuitry, and the use of floating-gate MOS transistors for modifiable, nonvolatile interconnection weights are presented in their paper. CMOSanalog/digitalcircuitsofthehysteresisMcCulloch-PittsneuronarepresentedbyCho,Tsuchiya,anTakefuji. In their paper the hysteresis McCulloch-Pitts neurons are used for finding unknown Ramsey numbers. Watola and Meador present a paper entitled "Competitive learning in asynchronous-pulse-density integrated circuits." They introduce MOS circuits for the integratedimplementationofcompetitive learning and depicts the simulation result ofthe two-input, three-output competitive network. AprogrammableanalogCMOSsynapseispresentedbyKim, Shin,Bogineni,andSridhar. Thedesignedsynapse consists oftwo complementary floating-gate MOSFETs which are programmable in both directions by Fowler Nordheim tunneling. PhillipsandMUller-Dottdescribetwo-stateneuralnetworkarchitectureforfeedbackcontrolofdynamicsystems. In their paper system identification and the design ofa controller using neural networks are investigated. Apaperentitled"Temporalsignalprocessingwithhigh-speedhybridanalog-digitalneuralnetworks," byDeYong, Eskridge, and Fields deals with temporal signal processing problems. The proposed approach using the hybrid analog-digital techniques overcomes some ofthe problems in the conventional approaches including hardware overhead, complexalgorithmicsolutions, orlossofinformationthroughthetransformationoftemporalproperties ofthe input. Finally,apaperentitled ''AsuperparallelsorterusingabinaryneuralnetworkwithAND-ORsynapticconnec tions," by Nakagawaand Yamadapresentsadigitalapproach forimplementingtheneural networkparallelsorting algorithm. The proposed idea will be able to reduce a large amount ofhardware. 1 264 Takefuji YoshiyasuTakefujihasbeenanassociateprofessoronthefaculty ofenvironmentalinfor mation at Keio University since 1992 and has been on the electrical engineering faculty atCaseWesternReserveUniversitysince1988.BeforejoiningCase,hetaughtattheUniver sityofSouth Florida and the University ofSouth Carolina. Hereceived his B.S. (1978), M.S. (1980), and Ph,D. (1983) in electrical engineering from Keio University (Japan). His research interests focus on neural network parallel computing for solving real-world problems. Heis interestedin VLSI applications and siliconarchitecture. He received the National Science Foundation/ResearchInitiation Award in 1989 and is an NSF advisory panelist. AmemberoftheIEEE ComputerSociety, ACM, International Neural Network Society,and AmericanAssociationfortheAdvancementofScience,hereceivedtheInfor mation Processing Society ofJapan's best paper award in 1980. He has written a book entitled Neural Network Parallel Computing, published by Kluwer, and has co-authored twobooks,DigitalCircuits(Ohrn-ShaPublishers)in1984andNeuralNetworkComputing (BaifukanPublishers) in 1992. He was aneditor oftheJournal ofNeural Network Com putingandisanassociateeditorofIEEETransactionsonNeuralNetworksandNeurocom puting. He is the guest editor ofthe Journal Analog Integrated Circuits and Signal Pro cessing in the special issue on analog VLSI neural networks, and the guest editor of Neurocomputinginthespecialissueonneural-network-basedoptimization. Hehaspublished more than 100 papers. 2 Analog IntegratedCircuitsand SignalProcessing2, 265-272 (1992) © 1992 Kluwer Academic Publishers, Boston. Manufactured inThe Netherlands. Analog Computational Models of Concept Formation YOH-HAN PAO AND WASSIM HAFEZ ElectricalEngineeringandAppliedPhysics, Case J#stem Reserve University, Cleveland, OHandAl Hflre, Inc., Cleveland OB44106 Abstract. This paper proposes and describesa method ofinductive concept learning, a method suitablefor im plementation in parallel computationalmode with analog VLSI neural-netcircuitry. Theapproach is consonant with the original Perceptronapproach. However, weights along linearlinks are notlearned adaptively. Instead, the netdepends upon thefrequency ofoccurrenceto adjust the strength ofactivation generated by an inputand theattention paidto the input. Ofcritical importance are the relative magnitudes ofthe information complexity oftheconcepttobelearnedandthe complexityoftheimplementationhardware. Iftheformer exceedsthelatter, the concept cannot be learned. The manner in which failure is signaled and hardware complexity is increased is described in this paper. 1. Introduction previousworkwhichcanbecitedincludesWinston's work on learning the concept of an Arch [1], Aconceptis exemplifiedby a class ofobjects whose Michlalski'sINDUCE[2],thePao-Humethod[3],and attributes obey a set of characterization rules. Evi theID3 algorithm [4]. Allofthesefall inthecategory dently, then, aconceptcannotexistunlessatleastone ofwhat might be called symbolic processing and are other concept exists, whose objects do not obey the ofthenatureofhypothesis testingandconceptforma rulesandformwhatiscalledthenegativeclass. Now, tion through generalization and specialization. theconceptformationtaskcanbedescribedasfollows: Incontrasttothatarethediscriminantformationap given a set ofobjects that are known to belong to a proaches ofpattern recognition, exemplified perhaps certain (positive) concept, and given a setofobjects bythe Perceptron [5], andby themorerecentparallel thatbelongtotheother(negative)concept,theobjec distributedconnectionistcomputingincludingthegen tiveistoinfer(eitherimplicitlyorexplicitly)therules eralized delta rule net[6], the functional-link net [7], that govern the concept. and others. Forourpurpose, anobjectisasetoffeaturenames Thesymbolicversusconnectionistissueremainsac andasetofcorrespondingfeaturevalues. Thefeature tive in the AI community [8]. It would seem to us names are linguistic expressions and the values may howeverthatsomeoftheadvantagesordisadvantages be either numeric or linguistic symbolic. attributedto theconnectionistapproachare really not The subject oflearning is a very rich one indeed, intrinsic to the approach itselfbut more due to inter a subject of interest to philosophers, psychologists, pretationsbythoseaccustomedtosymbolicprocessing. educators,artificialintelligenceresearchersandsoon Forexampleitisoftenarguedthattheconnectionist fordecadesandcenturies,eversincetimeimmemorial. paradigmaddressestheconceptformation, andinduc There is much that can be said, to place our present tivelearningingeneral, fromanarrowangleasitdoes workproperlyin relationshiptocogentpreviouswork. notresultinanexplicitsymbolicrepresentationofthe Itisdifficulttodosoadequately giventhelimitations rulesthatcharacterizetheconcept. Also, thatmostcon ofone briefpaper. nectionist learning schemes are of sequential nature, However, wecansaythatwhatwearedealingwith as they are based on minimizing a centralized error is generally known as the task of learningfrom ex function, andhenceprovidenoappreciableadvantages amples and our bent is that ofBayes' statistics, pat over the search schemes. Moreoever, in the connec tern recognition, AI, and neural-networkcomputing. tionist paradigm, prior knowledge ofall possible ob Within that limited field ofdiscourse, some relevant jects' attributes and their values is required, since 3 266 Pao and Hafez connectionistmodelsareoffIxedstructure;andhence, In Section 2, we introducethe proposed approach incremental learning of any new attributes or values toconceptfonnationwiththehelpofanillustrativeex becomes infeasible. ample described previously in psychology literature. A similar debate, but in different terminology, ex Some aspects ofthe underlying principles ofthis ap istswithinthecognitivesciencecommunity.Thehypoth proacharedescribed in Section3together with some esis-testing theory ofhuman concept formation is in salientcharacteristicsoftheapproach. Otherproperties essenceequivalentto the AI'sproblemsolvingtheory ofthis methodology remain to be explicated in sub based on search. According to the hypothesis-testing sequentreports, Summarizingandconcludingremarks theoryhumansubjectscontinuallyformulateandreject are contained in Section 4. hypothesesabouttherelevantandirrelevantfeaturesof theconcept.Incontrast,thefrequencytheoryofconcept formation suggests that human subjects discriminate 2. AnillustrativeExampleofConceptIdentification between the relevant and irrelevant features ofa con ceptbasedontheirrelativefrequencyofoccurrencein In cognitive psychology research, there is interest in boththenegativeandpositiveexamples. Thethrustof trying to understand how people do conscious thepresentpaperisthatthefrequency theoryofconcept hypothesisformation, Anderson[11]describesatypical formation can lead to parallel distributed models that concept-formation task in the following manner: areanalogous tothoseoftheconnectionistparadigm. "Consider the following: Both the hypothesis-testing and the frequency theoriesaresupportedbylargebodiesofexperiments, A dax can be large, bright, red and square, which suggests that human subjects utilize both par A dax can be large, dull, red and square, adigms and can shift from one strategy to another. A dax cannot be small, dull, red and square, However, it is not clear why and when such shift in A dax cannot belarge, bright, red and triangle, strategy occurs. The thesis that is being proposed in A dax can be large, dull, blue and square. thispaperis thattheanswerstothosequestionsarecon What is a dax? nected to the complexity ofthe concept. The best answer is probably that a dax is a large Briefly,whenthecomplexityoftheconceptisequal square, ..." to that ofthe computational hardware (or procedure) conceptlearningcanbecarriedoutinthehypothesis As an illustrativeexampleofour work, weusethe testingmanner, indeterministicmode, andinmanners samekindofmaterialused byBruner, Goodnow, and compatiblewith symbolicprocessingand stepby step Austin [12], shown in fIgure 1. The stimuli varied search or reasoning. Itis postulated that as the com among themselves along four dimensions: number of plexity oftheconceptis increased, the inadequacy of objects(one, two, orthree); numberofbordersaround the computational procedure manifests itselfin an ir the boxes (one, two, orthree), shape(cross, circle, or reduciblestochasticnatureofoccurrencesandwhatis square), and color (green, black, or red; represented then learned is a measure ofthe relative frequencies in the fIgure by white, black, or gray). Human sub ofoccurrenceswithinaclassandoutsideofthatclass. jects were shown number of such stimuli and were Finally, itis postulated thatoncethe nature ofthe ir asked to discover the concept common to all the in reduciblerandomnesshasbeenidentified,thecomplex stances shown. ityofthecomputationalprocedurecanbeincreasedin Threecolumnsofcardsarealso showninfIgure 1. I, an appropriately directed manner to result in a more Each column is made up of instances identifIed as accurateandmoreefficienthypothesistestingprocedure membersofaconcept(+) or notmembers (-), Each again. Furtherdiscussionontherelationshipbetween column represents a different concept. In the randomness and complexity can be found in articles psychologyexperimentsthe human subjects would be by Abu-Mustafa [9] and Kolrnogorov [10]. presentedwith the instances ina column, onecardat It is the purpose of this paper to describe a a time. Fromtheseinstancesthesubjects wouldtry to distributedparallelprocessingmodelwhichcanaccom determine what the concept was. modate different phases of concept learning, as ex In our concept identifIcation procedure, each perience reveal more and more ofthe complexity of positiveornegativeinstanceofaconceptisrepresented theconcept. Furthermore, themodel issuitableforim in terms ofa pattern of features. As shown in fIgure plementation in analog VLSI. 2,thefeature namesareshape,color,count,andborder 4 Analog Computational Models of Concept Formation 267 + @2] + [Q] + 1<0<01 1§fj] 1 1++1- ~ + 1001 W+ ~ ~ ~... + fj][lJ1J _ m [E+ + ~ ~ Concept1 ~ + 1 1 1FJDfill I 000 + G Concept2 m ImI + Concept3 Fig. 1. Conceptformationtask. Awell-knownconceptformationexperimentfromtheliteratureoncognitivepsychology[11, 12]. Foreach concept,thesubjectswereshownasetoflabeledcards.Cardsthatbelongtotheconcepttobelearnedarelabeled +whileothersarelabeled -. Eachcardhasfourfeatures; shape,color,count,andnumberofborders. Eachfeaturecanassumeonlyoneofthreepossiblevalues. The setofall possiblecards consistsof81 cards. Examplecards for three separateconcepts are shown. number. Eachfeaturecanhaveanyoneofthreevalues helpoftheschematicillustrationoffigure 3. Infigure which are (circle, cross, and square), (white, gray, 2,weseethatthefeaturevalue"cross"isactivatedthree black), (one, two, three) and (single, double, and tri times in the training experience. However, two ofthe ple) respectively. three times belonged to the positive concept whereas Inthisconceptlearningapproach, weutilizeanet thethirdinstancedidnot. Accordingly, theprocessing workofinterconnectedprocessing elements. This net elementsplitsitsoutputactivationinproportiontothe differssignificantlyfrom manyothersinthatprocess relativefrequencyofthepositiveandnegativeclasses. ingisdeterminedbyparameterswhicharestoredloc Therefore, we say that the frequency parameters are ally.Thusalthoughthereisindeedmuchconnectionism, learned and are stored at the output gate. And the eachnode does not need to collecta greatdeal ofin output energy which is finite is split among the out formation onparametervaluesfrom afarbeforeitcan goingconnectivitiesinaccordancewiththefrequency decide on how to proceed computationally. information. Ofprimeimportancearetheideasofactivationand On the input side the processing element needs attention. Figure2 can be understood better with the to divide its attention among the various incoming 5 268 Pao and Hafez Class+ Class- "+" Upperlayer processingnodes Lowerlayer processingnodes ConceptI Pattern CLalabsesl NOeuttwpuotrk PatternType I<0<0 I + + I I I OD - - "Cross" "Square" ~ - - Training Fig. 3. llIustrationofasmallportionofaconceptlearningnet. Fre ~I I~ ODD - - quencyparametersarestoredlocallyattheoutputgatesoftheproc [E]] 1 essingelementsoftheinputlayer,whileattentionparametersarestored + + attheinputgatesoftheoutputlayer. Strengthofactivationfroma processingelementisdistributtedamongoutputgatesinaccordance ~ ++ II + + withfrequency parameters. Spatialintegrationofincomingactiva Consulting tionatinputgatesisgovernedbyhowmuchattentionisdevotedto [I + II + + (Generalization) eachgate. underothercircumstancesitcanbeshownquiteclearly Fig. 2. Networkforconcept1. Thenetworkresponseforthetrain that a linear net such as we advocate would not be ingpatternsandconsultingpatternsindicatethatthenetworkhasac adequate. In fact we are now revisiting the old Per quiredtheconceptof"twocrossesorblackcolor." Originally, in [11, 12]theconceptwasclaimedtobe"twocrosses," however, the ceptrondiscussion[13] underveryinterestingandnew lasttrainingpatternjustifiesthenetworkresponseforthesecondcon circumstances. sultingpattern. Toillustratesomepoints,letusnowmodifyconcept I to "one cross or two squares," a disjunctive state channels which are connected to it. Accordingly, in mentoftwoconjunctions.Weencodetherelevantsubset figures 2 and 3, it so happens that a "positive" node ofcards in terms ofpatterns with two binary-valued isabletodevoteitsentireattentiontothechannelfrom features, asshowninfigure4. Ascanbeseeninfigure "cross" while the "negative" node only devotes half 4, thereisatotalof16conceptsthatcanbeconstructed ofits attention to the "cross" channel. withthefourpatterns(ingeneral,forann-dimensional zn When the cross node is activated, an activation of spaceofbinary-valuedvariables thereare 2 Boolean YJ x 1 = YJ goes to the "positive" node while an ac functions). Itisknownthatifwerepresentthepatterns tivationof YJ x ~ = Y6 goes to the "negative" node, aspointsinatwo-dimensionalspacethentherearetwo indicatingthat "cross" islikelytobeastrongpositive functions, namely the exclusive OR and its comple indicationoftheconcept. Similarly,weseethat"two" ment, forwhichonecannotseparatethe "true" points also sends a strong activation to the positive node. from the "false" by a linear equation [14]. Itisinterestingthatthissituationis signaledbyour network as shown in figure 5. When the complexity 3. Discussion ofthe Method oftheconceptbecomesgreaterthanthatwhichthenet workcanhandle, theentropiesatindividualfeaturenets Bruneretal. [12] statethatconcept1is "twocrosses," attain their maximum values. aconjunctiveconcept. Undersomecircumstancescon In ourapproach we take note ofwhat features are junctiveconcepts canbelearnedwithalinearnetbut nolongereffectiveindicatorsofclassmembershipand 6
Description:The list of books you might like
