Dedication To Don My husband For yourconstant love, support, and encouragement To do the best I can do in all My endeavors as a Woman and a Scholar! Contents Preface...................................................................................................... xvii Acknowledgments..................................................................................... xxi About the Author.................................................................................... xxiii 1 From Data to Models: Complexity and Challenges in Understanding Biological, Ecological, and Natural Systems................................................................................. 1 1.1: Introduction 1 1.2: LayoutoftheBook 4 References 7 2 Fundamentals of Neural Networks and Models for Linear Data Analysis ................................................................ 11 2.1: IntroductionandOverview 11 2.2: NeuralNetworksandTheirCapabilities 12 2.3: InspirationsfromBiology 16 2.4: ModelingInformationProcessinginNeurons 18 2.5: NeuronModelsandLearningStrategies 19 2.5.1: ThresholdNeuronasaSimpleClassifier 20 2.5.2: LearningModelsforNeuronsandNeuralAssemblies 23 2.5.2.1: HebbianLearning 23 2.5.2.2: UnsupervisedorCompetitiveLearning 26 2.5.2.3: SupervisedLearning 26 2.5.3: PerceptronwithSupervisedLearningasaClassifier 27 2.5.3.1: PerceptronLearningAlgorithm 28 2.5.3.2: APracticalExampleofPerceptrononaLarger RealisticDataSet:IdentifyingtheOrigin ofFishfromtheGrowth-RingDiameterofScales 35 2.5.3.3: ComparisonofPerceptronwithLinear DiscriminantFunctionAnalysisinStatistics 38 viii & 2.5.3.4: Multi-OutputPerceptronforMulticategory Classification 40 2.5.3.5: Higher-DimensionalClassificationUsingPerceptron 45 2.5.3.6: PerceptronSummary 45 2.5.4: LinearNeuronforLinearClassificationandPrediction 46 2.5.4.1: LearningwiththeDeltaRule 47 2.5.4.2: LinearNeuronasaClassifier 51 2.5.4.3: ClassificationPropertiesofaLinearNeuron asaSubsetofPredictiveCapabilities 53 2.5.4.4: Example:LinearNeuronasaPredictor 54 2.5.4.5: APracticalExampleofLinearPrediction: PredictingtheHeatInfluxinaHome 61 2.5.4.6: ComparisonofLinearNeuronModelwith LinearRegression 62 2.5.4.7: Example:MultipleInputLinearNeuron Model—ImprovingthePredictionAccuracy ofHeatInfluxinaHome 63 2.5.4.8: ComparisonofaMultiple-InputLinearNeuron withMultipleLinearRegression 63 2.5.4.9: MultipleLinearNeuronModels 64 2.5.4.10: ComparisonofaMultipleLinearNeuron NetworkwithCanonicalCorrelationAnalysis 65 2.5.4.11: LinearNeuronandLinearNetworkSummary 65 2.6: Summary 66 Problems 66 References 67 3 Neural Networks for Nonlinear Pattern Recognition.............. 69 3.1: OverviewandIntroduction 69 3.1.1: MultilayerPerceptron 71 3.2: NonlinearNeurons 72 3.2.1: NeuronActivationFunctions 73 3.2.1.1: SigmoidFunctions 74 3.2.1.2: GaussianFunctions 76 3.2.2: Example:PopulationGrowthModelingUsing aNonlinearNeuron 77 3.2.3: ComparisonofNonlinearNeuronwithNonlinear RegressionAnalysis 80 3.3: One-InputMultilayerNonlinearNetworks 80 3.3.1: ProcessingwithaSingleNonlinearHiddenNeuron 80 3.3.2: Examples:ModelingCyclicalPhenomenawith MultipleNonlinearNeurons 86 3.3.2.1: Example1:ApproximatingaSquareWave 86 3.3.2.2: Example2:ModelingSeasonalSpeciesMigration 94 3.4: Two-InputMultilayerPerceptronNetwork 98 3.4.1: ProcessingofTwo-DimensionalInputsby NonlinearNeurons 98 3.4.2: NetworkOutput 102 & ix 3.4.3: Examples:Two-DimensionalPrediction andClassification 103 3.4.3.1: Example1:Two-DimensionalNonlinear FunctionApproximation 103 3.4.3.2: Example2:Two-DimensionalNonlinear ClassificationModel 105 3.5: MultidimensionalDataModelingwithNonlinear MultilayerPerceptronNetworks 109 3.6: Summary 110 Problems 110 References 112 4 Learning of Nonlinear Patterns by Neural Networks............ 113 4.1: IntroductionandOverview 113 4.2: SupervisedTrainingofNetworksforNonlinear PatternRecognition 114 4.3: GradientDescentandErrorMinimization 115 4.4: BackpropagationLearning 116 4.4.1: Example:BackpropagationTraining—AHandComputation 117 4.4.1.1: ErrorGradientwithRespecttoOutput NeuronWeights 120 4.4.1.2: TheErrorGradientwithRespecttothe Hidden-NeuronWeights 123 4.4.1.3: ApplicationofGradientDescentin BackpropagationLearning 127 4.4.1.4: BatchLearning 128 4.4.1.5: LearningRateandWeightUpdate 130 4.4.1.6: Example-by-Example(Online)Learning 134 4.4.1.7: Momentum 134 4.4.2: Example:BackpropagationLearning ComputerExperiment 138 4.4.3: Single-InputSingle-OutputNetworkwith MultipleHiddenNeurons 141 4.4.4: Multiple-Input,Multiple-HiddenNeuron,and Single-OutputNetwork 142 4.4.5: Multiple-Input,Multiple-HiddenNeuron, Multiple-OutputNetwork 143 4.4.6: Example:BackpropagationLearningCase Study—SolvingaComplexClassificationProblem 145 4.5: Delta-Bar-DeltaLearning(AdaptiveLearningRate)Method 152 4.5.1: Example:NetworkTrainingwithDelta-Bar-Delta— AHandComputation 154 4.5.2: Example:Delta-Bar-DeltawithMomentum— AHandComputation 157 4.5.3: NetworkTrainingwithDelta-BarDelta— AComputerExperiment 158 4.5.4: ComparisonofDelta-Bar-DeltaMethodwith Backpropagation 159
Description: