Table Of Content

LOCALDYNAMICMODELING WITHSELF-ORGANIZINGFEATUREMAPS By LUDONGWANG ADISSERTATIONPRESENTEDTOTHEGRADUATESCHOOL OFTHEUNIVERSITYOFFLORIDAINPARTIALFULFILLMENTOFTHE REQUIREMENTSFORTHEDEGREEOF DOCTOROFPHILOSOPHY UNIVERSITYOFFLORIDA 1996 ACKNOWLEDGEMENTS IwouldliketothankDr.JoseC.Principe,mymajoradvisor,forhisguidance, support,motivation,andpatience,andforhelpwiththecompletionofthisdissertation.I amextremely grateful tothe members ofmy supervisory committee. Dr. Donald G. Childers,Dr.FredrickJ.Taylor,Dr.JohnM.M.Anderson,Dr.KermitN.Sigmon,Dr.Paul E.Ehrlich,fortheircontributionstomyresearchandtothepreparationandreviewofthis manuscript.ImustalsothankmanyCNELfellowstudentswhohaveenrichedmylife duringthisprogram.SpecialthanksgotoJohnFisherandSamelCelebi. Thanksareduetomyfamilyfortheirencouragement.Finally,Iwouldliketothank mylovingwife.Hong,forhersupport,andbeautifuldaughter,Amy,formakingmylife special. ii TABLEOFCONTENTS page ACKNOWLEDGEMENTS ii ABSTRACT vi CHAPTERS 1 INTRODUCTION 1 1.1Motivations 1 1.2ResearchOutlineandObjectives 2 1.3OverviewoftheDissertation 5 2 NONLINEARDYNAMICMODELING 6 2.1DynamicSystemsandSignals 7 2.2SignalAnalysis:LinearandNonlinearModels 8 2.2.1LinearModelsandLimitations 8 2.2.2AnalysisofLinearandNonlinearSystems 10 2.3NonlinearDynamicModeling 11 2.3.1GlobalModels 13 2.3.2LocalModels 15 2.4DynamicalInvariantsinModeling 17 2.4.1DynamicsReconstructionandCorrelationDimension 18 2.4.2TheLyapunovExponents 19 2.4.3ModelValidation 21 3 SELF-ORGANIZINGFEATUREMAP 23 3.1Introduction 23 3.2FormationofSOFM 24 iii 3.2.1Structures 24 3.2.2TheAlgorithmofKohonenModel 26 3.2.3Convergence 29 3.3LocalizedNeuralRepresentationofSignals 31 3.3.1PropertiesofSOFM 31 3.3.2SimulationsofSOFMwithTemporalSignalProcess 39 3.4ApplicationPotential 44 4 NON-LINEARTIMESERIESMODELINGWITHSOFM 47 4.1Introduction 47 4.2State-DependentARModels 48 4.2.1NonlinearAutoregressiveModels 48 4.2.2State-DependentPredictionofNonlinearProcesses 50 4.3LocalLinearApproximationofGlobalDynamicsandLimitation ... 51 4.4SOFM-BasedLocalLinearModelingNetworks 55 4.4.1Methodology 55 4.4.2Architecture 56 4.4.3NetworkConstruction 57 5 METHODOLOGYOFLEARNINGEQUATIONS 60 5.1PropertiesofSOFM-BasedLocalModeling 60 5.2ImprovedEstimationofLocalLinearModels 62 5.3Dynamic-OrientedRepresentations 63 5.3.1TheConstraintsofSOFMLearningProcess 64 5.3.2DynamicLearningRule 65 5.4TheWeightedLeast-SquaresSolution 69 6 EXPERIMENTALRESULTS 73 6.1ModelingofNumericallyGeneratedTimeSeries 75 6.1.1Mackey-GlassTimeSeries 77 6.1.2LorenzTimeSeries 88 6.1.3ApproximationofSeamlessPatchingofLocalModels 95 6.2ConsistencyoftheConstructedModels 98 6.2.1Consistencyvs.DifferentInitialSOFMStates 98 6.2.2TemporalConsistency 102 6.3ModelingReal-WorldSignals 104 6.3.1LaserTimeSeries 105 iv 6.3.2EEGSignal 108 6.3.3SunspotTimeSeries 108 7 CONCLUSIONSANDFUTURERESEARCH 114 7.1Conclusions 114 7.2FutureResearch 117 REFERENCES 119 BIOGRAPHICALSKETCH 127 V AbstractofDissertationPresentedtotheGraduateSchooloftheUniversityofFloridain PartialFulfillmentoftheRequirementsfortheDegreeofDoctorofPhilosophy LOCALDYNAMICMODELING WITHSELF-ORGANIZINGFEATUREMAP By LudongWang December1996 Chairman:Dr.JoseC.Principe MajorDepartment:ElectricalandComputerEngineering Chaotic signals are associated with autonomous response ofcertain nonlinear dynamicalsystems.Whiletheyaredeterministicwithfewdegreesoffreedom,chaotic signalsarenotpredictableinthelongterm.Theyclearlyhaveconsiderablymorestructure thancanbeinferredfromandexploitedbytraditional stochasticmodelingtechniques. Consequentlyitisimportanttodevelopnewsignalprocessingtechniquesthatarematched tothespecialcharacteristicsofthisclassofsignals. Inthisresearch,afinitesetoflinearlocalmodelsisestablishedasafeasibleand effectiveimplementationofdynamicmodeling.Asaself-organizingfeaturemap(SOFM) isusedasthemodelinginfrastructure, the method iscalled SOFM-based local linear modelingnetwork. Thepreviousworkonlocalmodelsonly focusesoninterpolatinglocaldatafor short-termprediction.Anexampleisstate-dependentfunctionalmappingwhichdoesnot considerdeducingtheequationsofmotion.Theaimoftheproposedmodelingmethodis toderiveafinitesetoflocallinearmodelstoapproximatetheglobaldynamics.These modelsareconstructedwithspatialconstraintswhilematchingthedynamicsofasignalin vi thetemporalsense.Thisspatial-temporalarchitecturerequiresfewerassumptionsaboutthe underlyingdynamics. The SOFM constructed with Kohonen's learning law is a localized neural representationofsignals.WeappendedanewlinearlayertotheSOFMwhichistrainedto approximatethetangentspacetothedynamicsrepresentedbythelocalneuralfield.The useofaneighborhoodfunctionimposesastrongstatisticalconstraintovertheconverged neuralfield,suchthattheirregularspacingoflocaldatacanbesmoothedout.Thisis significanttothelocalmodelingmethodasagivensignalisusuallyoffinitelength.Based onthisstructure,theconstructedlocallinearmodelsarenottotallyindependentofeach other, whichhelps toreduce the discontinuity between them. On the otherhand, the statisticaldensitymatchingpropertymakesSOFMagoodvectorquantizationprocedure. Thisisnotanoptimalstructurefortheoverallpurposeofdynamicmodeling.Thereforea modifiedKohonenlearningruleisproposedasanewtrainingprocedureoftheemployed SOFM.Asthepredictionerrorisinvolvedtoreflectthelocaldynamicfluctuation,this procedureisthuscalleddynamiclearning.Withthisprocedure,theconvergedneuralfield becomesadynamic-orientedrepresentationofthesignal. TheproposedSOFM-basedmodelingschemehasbeenappliedtobothsynthetic andreal-worldsignals.Experimentsdemonstratethatthismethodiscapableoffaithfully capturingtheunderlyingdynamicsofchaoticsignals. Theoverallstructureoftheproposedmethodisasignificantextensionofthelocal linearmodelingtechniques. It also representsa new researchdirectiontoexplore the potentialofthevigorousself-organizingfeaturemap. vii CHAPTER 1 INTRODUCTION 1.1Motivations Temporalpatternrecognition,systemcontrol,signalprediction,andchaoticdata analysisshareacommonproblem:deducingequationsofmotionfromobservationsof time-dependentdata.Eachofthemseekstomodelthephysicalworldwithacertaingoalin mind. These models encapsulate the data complexity in a compact, algorithmic specificationproducingavastdatareduction.Thereforemodelingcanbedefinedasa processtoextracttheunderlyingdynamicmodelfromagivensignal. Inconventionalsignalprocessingawiderangeofapproacheshavebeendeveloped formodelingsignalsthataredeterministicandpredictable.Thesignalsaregenerallytaken astherealizationofsomestochasticprocess,andtheunderlyingsystemismodeledas linear[92].Insuchamodelingscheme,therandominputproducestheuncertaintyinthe systemoutput. Chaoticsignalsareassociatedwiththeautonomousresponseofcertainnonlinear dynamical systems. Without any random input, the output ofsuch systems exhibits complex behavior and possesses noise-like spectra. These systems, named chaotic dynamicalsystems[27],areubiquitous,e.g.,laserbeams,biologicalsystems,astronomical motions,weatherpatterns,fluidflows,andelectriccircuits,etc.[20][32][67][93].Asthe uncertaintyinitsoutputoriginatesfromthesystemdynamicsinsteadofanexternaldriving force,theyclearlypossessconsiderably more structurethancan be inferredfromand exploitedbytraditionalstochasticmodelingtechniques. Tomodelthechaotictimeseries,manytechniqueshavebeendeveloped.Theycan generallybeclassifiedasglobalandlocalmodels.Globalmodelsareconstructedonceand 1 2 fitallthestatespace,whilelocal modelsrepresent local regions. Structureinchaotic attractors tends to be very intricate and nonuniform. Depending on the functional representation, global models have difficulty mimicking such systems adequately. Localizedrepresentationscanchartoutthenuancesofchaoticmorphologyandprovidean atlasfortheentirestatespace.Amongthelocalmethods,linearmodelsareprovedtobe computationallycost-effective. Theself-organizingfeaturemap(SOFM)constructedwithKohonen'slearninglaw [52]isnormallyutilizedforfeatureextraction. Itischaracterizedbytheformationofa topographicmapoftheinputpatterns,inwhichthespatiallocationoftheneuronsinthe outputneuralfieldcorrespondtointrinsicfeaturesoftheinputpatterns.Asacomputational map,itconstitutesabasicbuildingblockforinformationprocessing.Thefeaturemapso derivedprovidesabetterrepresentationofthelocalinformation. Thus,exploringtheapplicationoflocallinearmodelsinglobaldynamicmodeling, andutilizingSOFMasthemodelinginfrastructureforthispurpose,composethemotiva- tionofthisresearch. 1.2ResearchOutlineandObjectives So farthe research on the method and application oflocal linear models is restrictedtotheforecastingofchaoticsignals.Inthisscenario,alinearpredictivemodelis estimatedasthelocaldynamicmapbasedonthenearestneighborsofthecurrentstate, andthepredictionisthusobtainedbysimplefunctionevaluation.Duetothefactthatcha- otictimeseriesisnotpredictableinthelong-termsense,smallshort-termpredictionerror isthegoalofthisscenario.Undertheassumptionthatthestatedynamicsissmoothand sufficientdatasamplesareavailable,thismethodcangenerallyprovideagoodpredicting performance. Capturingthedynamics,i.e.,thelongtermbehaviorofthesystem,isdefinitelya significantextensionofthelocallinearforecastingscenario.Thegoalistofitasetoflocal linearmodelstothegivensignalreconstructedinthestatespacesuchthatautonomous outputofthe constructed structure exhibits similardynamic behavior. However, such attemptshavebeenhinderedbysomeconcerns[23].Dependingonthepartitioningand scales,piecewiselinearequationsofmotionmayexhibitperiodicbehaviorwhentheorig- inaldynamicsischaoticandviceversa.Piecewiselineardynamicsareconsideredasvio- latingthephysicallymotivatedhypothesisofsmoothdynamicalsystems.Mostphysical processesdonotexhibitarbitrarilyfastchangesintheirfirstderivatives.Moreoverthe establishedsetoflocallinearmodelsgenerallymaynotbeadiscontinuousfunctional map,whichcanleadtoundesiredbehavioritisiteratedasanautonomoussystem. Inthisresearch,theaboveconcernsarereconsideredinthefollowingaspects.As longasthelocallinearmodelscanexhibitchaoticbehaviorwhentheoriginaldynamicsis periodicasshowedintheinvestigationbyCrutchfield,etal.[23],itisreasonabletoexpect thatthestructureoflocallinearmodelsiscapableofpossessingchaoticdynamics.Inthe contextofchaoticdynamics,thelackofsmoothcontinuationmaydegradetheshort-term predictionperformance.However,itisnotsufficienttoconcludethatafaithfulapproxima- tionofglobaldynamicsisimpossibleintermsofafinitesetoflocallinearmodels.Finally thediscontinuityproblemmaybereducedtoasufficientextent,ifnoteliminated,such thattheestablishedlocallinearmodelspiecedtogetherasawholemaynotbehinderedto approximatetheoriginaldynamics. Torealizethescenariooflocallinearmodeling,itisproposedheretoutilizethe SOFMasthemodelinginfrastructure.WhileSOFMisalocalizedneuralrepresentationof theinput,itisconstructedintheglobalsenseviathecompetitivelearningprocess.The locallinearmodelsaredirectlyfittedfromtheconstructedSOFMmap.Twomajorissues willbeinvestigated:a)CantheneuronsofeachlocalsegmentintheSOFMneuralfield collectivelybearmorereliableinformationaboutthelocaldynamics?b)Canall local dynamicsextractedfromtheindividuallocalsegmentscollectivelyrepresenttheglobal dynamicsofthegivensignal?