Models of Neural Networks: Temporal Aspects of Coding and Information Processing in Biological Systems PDF
Preview Models of Neural Networks: Temporal Aspects of Coding and Information Processing in Biological Systems
Physics of Neural Networks Series Editors: E. Domany J. L. van Hemmen K. Schulten Advisory Board: H. Axelrad R. Eckmiller J. A. Hertz J. J. Hopfield P. I. M. Johannesma D. Sherrington M. A. Virasoro Physics of Neural Networks Models of Neural Networks E. Domany, J.L. van Hemmen, K. Schulten (Eds.) Neural Networks: An Introduction B. Muller, J. Reinhart Models of Neural Networks II: Temporal Aspects of Coding and Information Processing in Biological Systems E. Domany, J.F. van Hemmen, K. Schulten (Eds.) E. Domany J. L. van Hemmen K. Schulten (Eds.) Models of Neural Networks II Temporal Aspects of Coding and Information Processing in Biological Systems With 90 Figures Springer-Verlag New York Berlin Heidelberg London Paris Tokyo Hong Kong Barcelona Budapest Series and Volume Editors: Professor Dr. J. Leo van Hemmen Institut fur Theoretische Physik Technische Universitat Munchen D-85H7 Garching bei Munchen Germany Professor Eytan Domany Professor Klaus Schulten Department of Electronics Department of Physics Weizmann Institute of Science and Beckman Institute 76100 Rehovot University of Illinois Israel Urbana. IL 61801 USA Library of Congress Cataloging-in-Publication Data Models of neural networks II/[edited by] Eytan Domany. J. Leo van Hemmen. Klaus Schulten. p. cm. Includes bibliographical references and index. ISBN-13:978-1-4612-8736-0 e-ISBN-13:978-1-4612-4320-5 DOl: 10.107/978-1-4612-4320-5 1. Neural networks (Computer science) I. Hemmen. J. L. van (Jan Leonard). 1947- . II. Domany. Eytan. 1947- . III. Schulten. K. (Klaus) IV. Title: Models of neural networks 2. V. Title: Models of neural networks three. QA76.87.M58 1994 006.3 -dc20 94-28645 Printed on acid-free paper. © 1994 Springer-Verlag New York. Inc. Reprint of the original edition 1994 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer-Verlag New York. Inc .• 175 Fifth Avenue. New York. NY 10010. USA). except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval. electronic adaptation. computer software. or by similar or dissimilar methodology now known or hereaf ter developed is forbidden. The use of general descriptive names. trade names. trademarks. etc., in this publication, even if the former are not especially identified. is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act. may accordingly be used freely by anyone. Production managed by Natalie Johnson; manufacturing supervised by Genieve Shaw. Photocomposed copy provided from the editors' LaTeX file. 987654321 ISBN 0-387-94362-5 Springer-Verlag New York Berlin Heidelberg ISBN 3-540-94362-5 Springer-Verlag Berlin Heidelberg New York Preface Since the appearance of Vol. 1 of Models of Neural Networks in 1991, the theory of neural nets has focused on two paradigms: information coding through coherent firing of the neurons and functional feedback. Information coding through coherent neuronal firing exploits time as a cardinal degree of freedom. This capacity of a neural network rests on the fact that the neuronal action potential is a short, say 1 ms, spike, localized in space and time. Spatial as well as temporal correlations of activity may represent different states of a network. In particular, temporal correlations of activity may express that neurons process the same "object" of, for example, a visual scene by spiking at the very same time. The traditional description of a neural network through a firing rate, the famous S-shaped curve, presupposes a wide time window of, say, at least 100 ms. It thus fails to exploit the capacity to "bind" sets of coherently firing neurons for the purpose of both scene segmentation and figure-ground segregation. Feedback is a dominant feature of the structural organization of the brain. Recurrent neural networks have been studied extensively in the physical literature, starting with the ground breaking work of John Hop field (1982). Functional feedback arises between the specialized areas of the brain involved, for instance, in vision when a visual scene generates a picture on the retina, which is transmitted to the lateral geniculate body (LGN), the primary visual cortex, and then to areas with "higher" func tions. This sequence looks like a feed-forward structure, but appearances are deceiving, for there are equally strong recurrent signals. One wonders what they are good for and how they influence or regulate coherent spik ing. Their role is explained in various contributions to this volume, which provides an in-depth analysis of the two paradigms. The reader can enjoy a detailed discussion of salient features such as coherent oscillations and their detection, associative binding and segregation, Hebbian learning, and sensory computations in the visual and olfactory cortex. Each volume of Models of Neural Networks begins with a longer pa per that puts together some theoretical foundations. Here the introductory chapter, authored by Gerstner and van Hemmen, is devoted to coding and information processing in neural networks and concentrates on the funda mental notions that will be used, or treated, in the papers to follow. More than 10 years ago Christoph von der Malsburg wrote the mean while classical paper "The correlation theory of brain function." For a long time this paper was available only as an internal report of the Max-Planck Institute for Biophysical Chemistry in Gottingen, Germany, and is here vi Preface made available to a wide audience. The reader may verify that notions which seemed novel 10 years ago still are equally novel at present. The paper "Firing rates and well-timed events in the cerebral cortex" by Moshe Abeles does exactly what its title announces. In particular, Abeles puts forward cogent arguments that the firing rate by itself does not suffice to describe neuronal firing. Wolf Singer presents a careful analysis of "The role of synchrony in neocortical processing and synaptic plasticity" and in so doing explains what coherent firing is good for. This essay is the more interesting since he focuses on the relation between coherence - or synchrony - and oscillatory behavior of spiking on a global, extensive scale. This connection is taken up by Ritz et al. in their paper "Associative binding and segregation in a network of spiking neurons." Here one finds a synthesis of scene segmentation and binding in the associative sense of pattern completion in a network where neural coding is by spikes only. Moreover, a novel argument is presented to show that a hierarchical struc ture with feed-forward and feedback connections may playa dominant role in context sensitive binding. We consider this an explicit example of func tional feedback as a "higher" area provides the context to data presented to several "lower" areas. Coherent oscillations were known in the olfactory system long before they were discovered in the visual cortex. Zhaoping Li describes her work with John Hopfield in the paper "Modeling the sensory computations of the olfactory bulb." She shows that here too it is possible to describe both odor recognition and segmentation by the very same model. Until now we have used the notions "coherence" and "oscillation" in a loose sense. One may ask: How can one attain the goal of "Detecting coher ence in neuronal data?" Precisely this is explained by Klaus Pawelzik in his paper with the above title. He presents a powerful information-theoretic al gorithm in detail and illustrates his arguments by analyzing real data. This is important not only for the experimentalist but also for the theoretician who wants to verify whether his model exhibits some kind of coherence and, if so, what kind of agreement with experiment is to be expected. As is suggested by several papers in this volume, there seems to be a close connection between coherence and synaptic plasticity; see, for exam ple, the essay by Singer (Secs. 13 and 14) and Chap. 1 by Gerstner and van Hemmen. Synaptic plasticity itself, a fascinating subject, is expounded by Brown and Chattarji in their paper "Hebbian synaptic plasticity." By now long-term depression is appreciated as an essential element of the learn ing process or, as Willshaw aptly phrased it, "What goes up must come down." On the other hand, Hebb's main idea, correlated activity of the pre and postsynaptic neuron, has been shown to be a necessary condition for the induction of long-term potentiation but the appropriate time window of synchrony has not been determined unambiguously yet. A small time window in the millisecond range would allow to learn, store, and retrieve Preface vii spatio-temporal spike patterns, as has been pointed out by Singer and im plemented by the Hebbian algorithm of Gerstner et al. Whether or not such a small time window may exist is still to be shown experimentally. A case study of functional feedback or, as they call it, reentry is provided by Sporns, Tononi, and Edelman in the essay "Reentry and dynamical interactions of cortical networks." Through a detailed numerical simulation these authors analyze the problem of how neural activity in the visual cortex is integrated given its functional organization in the different areas. In a sense, in this chapter the various parts of a large puzzle are put together and integrated so as to give a functional architecture. This integration, then, is sure to be the subject of a future volume of Models of Neural Networks. The Editors Contents Preface v Contributors xv 1. Coding and Information Processing in Neural Networks 1 Wulfram Gerstner and J. Leo van Hemmen 1.1 Description of Neural Activity. . . . . . . . . 1 1.1.1 Spikes, Rates, and Neural Assemblies 2 1.2 Oscillator Models . . . . . . . . . . 4 1.2.1 The Kuramoto Model . . . 6 1.2.2 Oscillator Models in Action 23 1.3 Spiking Neurons ......... 35 1.3.1 Hodgkin-Huxley Model 35 1.3.2 Integrate-and-Fire Model 38 1.3.3 Spike Response Model . . 39 1.4 A Network of Spiking Neurons 47 1.4.1 Stationary Solutions - Incoherent Firing 55 1.4.2 Coherent Firing - Oscillatory Solutions . 68 1.5 Hebbian Learning of Spatio-Temporal Spike Patterns. 72 1.5.1 Synaptic Organization of a Model Network 73 1.5.2 Hebbian Learning ........ 74 1.5.3 Low-Activity Random Patterns . 78 1.5.4 Discussion...... 81 1.6 Summary and Conclusions . 83 References .. .. .. .. .. 85 2. The Correlation Theory of Brain Function 95 Christoph von der Malsburg Foreword .......... 95 2.1 Introduction......... 96 2.2 Conventional Brain Theory 96 2.2.1 Localization Theory 96 2.2.2 The Problem of Nervous Integration 98 2.2.3 Proposed Solutions . . . . . . . . . . 100 2.3 The Correlation Theory of Brain Function . 104 2.3.1 Modifications of Conventional Theory 104 2.3.2 Elementary Discussion . . . . . . . . . 106 x Contents 2.3.3 Network Structures. . . . . . . . . . 107 2.3.4 Applications of Correlation Theory . 112 2.4 Discussion ............ . 115 2.4.1 The Text Analogy ... . 115 2.4.2 The Bandwidth Problem 116 2.4.3 Facing Experiment 117 2.4.4 Conclusions. 118 References . . . . . . . . . 118 3. Firing Rates and Well-Timed Events in the Cerebral Cortex 121 M. Abeles 3.1 Measuring the Activity of Nerve Cells . . . . . . . 121 3.2 Rate Functions and Stationary Point Processes . . 124 3.3 Rate Functions for Nonstationary Point Processes 129 3.4 Rate Functions and Singular Events 132 References. . . . . . . . . . . . . . . . . . . . . . . 138 4. The Role of Synchrony in Neocortical Processing and Synaptic Plasticity 141 Wolf Singer 4.1 Introduction........................... 141 4.2 Pattern Processing and the Binding Problem . . . . . . . . 143 4.3 Evidence for Dynamic Interactions Between Spatially Dis- tributed Neurons . . . . . . . . . 147 4.4 Stimulus-Dependent Changes of Synchronization Probability . . 149 4.5 Synchronization Between Areas. 151 4.6 The Synchronizing Connections. 151 4.7 Experience-Dependent Modifications of Synchronizing Con- nections and Synchronization Probabilities. . . . . . . . . . 155 4.8 Correlation Between Perceptual Deficits and Response Syn- chronization in Strabismic Amblyopia ....... 156 4.9 The Relation Between Synchrony and Oscillations 157 4.10 Rhythm Generating Mechanisms 159 4.11 The Duration of Coherent States . . . . . . . 160 4.12 Synchronization and Attention . . . . . . . . 162 4.13 The Role of Synchrony in Synaptic Plasticity 163 4.14 The Role of Oscillations in Synaptic Plasticity 164 4.15 Outlook . . . . . . . 166 4.16 Concluding Remarks 167 References. . . . . . 167
The list of books you might like
