Title Pages Sensory Cue Integration Julia Trommershäuser, Konrad Kording, and Michael S. Landy Print publication date: 2011 Print ISBN-13: 9780195387247 Published to Oxford Scholarship Online: September 2012 DOI: 10.1093/acprof:oso/9780195387247.001.0001 Title Pages (p.i) Sensory Cue Integration (p.ii) Computational Neuroscience (p.iii) Sensory Cue Integration (p.xv) (p.1) Sensory Cue Integration (p.2) SERIES EDITORS Michael Stryker Terrence J. Sejnowski Biophysics of Computation Christof Koch 23 Problems in Systems Neuroscience Edited by J. Leo van Hemmen and Terrence J. Sejnowski Sensory Cue Integration Edited by Julia Trommershäuser, Konrad P. Körding, and Michael S. Landy (p.iv) Oxford University Press, Inc., publishes works that further Page 1 of 2 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Title Pages Oxford University's objective of excellence in research, scholarship, and education. Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Copyright © 2011 by Oxford University Press Published by Oxford University Press, Inc. 198 Madison Avenue, New York, New York 10016 www.oup.com Oxford is a registered trademark of Oxford University Press All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior permission of Oxford University Press. ____________________________________ CIP data on file ISBN-13: 978-0-19-538724-7 ____________________________________ 1 3 5 7 9 8 6 4 2 Printed in the United States of America on acid-free paper Page 2 of 2 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Preface Sensory Cue Integration Julia Trommershäuser, Konrad Kording, and Michael S. Landy Print publication date: 2011 Print ISBN-13: 9780195387247 Published to Oxford Scholarship Online: September 2012 DOI: 10.1093/acprof:oso/9780195387247.001.0001 (p.v) Preface Aristotle distinguished five senses—sight, hearing, touch, smell, and taste—that provide cues about the outside world to the nervous system. The modern perceptual literature has found a host of additional senses and has also found that many senses provide more than one cue. For example, if we want to estimate how heavy an object is, then vision will provide us both with size cues (larger objects tend to be heavier) as well as surface textural cues (objects with visible metal texture tend to be denser). Cues can be thought of as largely independent pieces of sensory information that are used by the nervous system. In most situations we make use of a multitude of cues. Thus, one of the central objectives of the nervous system is the combination of all cues into useful estimates of the properties of the world, a notion often attributed to Helmholtz (1856/1962), but already present in the work of Alhazen (1021/1989) in the 11th century. Based on this information we can then successfully interact with the world. The study of cue combination asks under which circumstances and how cues are combined toward this purpose. There are several reasons why the nervous system needs to combine cues (see Chapter 1). Cues tend to be noisy. For example, the auditory system cannot estimate the location of the source of a sound perfectly. Moreover, cues can be ambiguous. For example, vision alone cannot always discriminate metal from plastic that has been painted to look like metal. If we combine vision with touch, however, we can more readily distinguish between different materials. Cue combination allows the nervous system to reduce noise and ambiguity. This book focuses on the emerging probabilistic way of thinking about cue combination in terms of uncertainty. These probabilistic approaches derive from the realization that all our sensors are noisy and, moreover, are often affected by ambiguity. For example, our mechanoreceptors are noisy and they cannot Page 1 of 3 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Preface distinguish whether a perceived force is caused by the weight of an object or by the force we are producing ourselves. The probabilistic approaches elaborated in this book aim at formalizing the uncertainty of cues. They describe cue combination as the nervous system's attempt to minimize uncertainty in its estimates and to choose successful actions. Some computational approaches described in the chapters of this book are concerned with the application of such statistical ideas to real-world cue-combination problems. Others ask how uncertainty may be represented in the nervous system and used for cue combination. Notably, across behavioral, electrophysiological, and theoretical approaches, Bayesian statistics is emerging as a common language in which cue- combination problems can be expressed. The broadening scope of probabilistic approaches to cue combination is highlighted in the breadth of topics covered in this book. The different chapters summarize and discuss computational approaches and behavioral evidence aimed at understanding the combination of visual, auditory, proprioceptive, and haptic cues. Some chapters address the combination of cues within a single modality, whereas others address the combination across modalities. Neural implementation, (p.vi) behavior, and theory are considered. The unifying aspect of this book is the focus on the uncertainty intrinsic to sensory cues and the underlying question of how the nervous system deals with this uncertainty. Following David Marr's taxonomy of three different levels of modeling of the nervous system (Dayan & Abbott, 2001; Marr, 1982), we can divide models of the nervous system into those that describe the implementation of computation (level 3), the algorithm used (level 2), and the objective of computation (level 1). This book derives from considerations about the computational objective (level 1) of cue combination (primarily dealing with uncertainty). The first section of the book gives an overview of the fundamental concepts and mathematics. Chapters in the subsequent two sections are concerned with the specific algorithms used by the brain and how they are implemented. This book is divided into three sections. The first section, “Theory and Fundamentals, ” introduces the mathematical ideas needed to formalize cues and uncertainty. The second section, “Behavioral Studies, ” asks how human subjects behave, comparing human behavior with the behavior of an optimal cue-integration scheme. The chapters in the final section, “Neural Implementation, ” ask how the nervous system is able to produce behavior that is impressively close to the predictions that arise from optimal algorithms. This book is the result of a workshop that was held in the autumn of 2008 at the beautiful German castle Rauischholzhausen, and we gratefully acknowledge the support we received from the German Science Foundation (DFG) for organizing this workshop. Scientists from many parts of the cue-combination community participated. The discussions we had at that workshop highlighted that scientists Page 2 of 3 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Preface who study a wide range of distinct cue-combination phenomena are converging onto a common set of open questions and are starting to use a common language to frame and analyze experimentally observable phenomena. During the workshop it became clear that this unified quantitative language for talking about cue combination enables researchers from a broad spectrum of backgrounds and research areas to communicate effectively about their research progress and current theoretical issues. While earlier research had already started to move in that direction (Knill & Richards, 1996), recent research has broadened the scope of phenomena that can now be described in a coherent and consistent framework. References Alhazen (1021/1989). The optics of Ibn al-Haytham. Volume I. Translation (Translated by A. I. Sabra), London: Warburg Institute, University of London. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: Computational, mathematical modeling of neural systems. Cambridge, MA: MIT Press. Helmholtz, H. L. F. (1856/1962). Treatise on physiological optics. Translated from the third German edition by J. P. C. Southall. New York: Dover. Knill, D., & Richards, W. (1996). Perception as Bayesian inference. New York: Cambridge University Press. Marr, D. (1982). Vision. San Francisco, CA: W. H. Freeman. Page 3 of 3 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Contributors Sensory Cue Integration Julia Trommershäuser, Konrad Kording, and Michael S. Landy Print publication date: 2011 Print ISBN-13: 9780195387247 Published to Oxford Scholarship Online: September 2012 DOI: 10.1093/acprof:oso/9780195387247.001.0001 (p.ix) Contributors Dora E. Angelaki Washington University School of Medicine Anatomy and Neurobiology St. Louis, MO Gregory Apker School of Biological and Health Systems Engineering Arizona State University Tempe, AZ Benjamin T. Backus Graduate Program in Vision Science SUNY College of Optometry New York, NY Martin S. Banks School of Optometry University of California, Berkeley Berkeley, CA Peter W. Battaglia Brain & Cognitive Sciences Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology Cambridge, MA Jeff Beck Gatsby Computational Neuroscience Unit University College London London, England Ulrik Beierholm Gatsby Computational Neuroscience Unit University College London Page 1 of 6 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Contributors London, England Paola Binda Institute of Neuroscience CNR Pisa, Italy Anne-Marie Brouwer TNO Human Factors Soesterberg, The Netherlands Christopher A. Buneo School of Biological and Health Systems Engineering Arizona State University Tempe, AZ Johannes Burge Center for Perceptual Systems Department of Psychology The University of Texas Austin, TX David Burr Department of Psychology Università degli Studi di Firenze Florence, Italy Corrado Caudek Department of Psychology Università degli Studi di Firenze Florence, Italy Gregory C. DeAngelis Department of Brain & Cognitive Sciences University of Rochester Rochester, NY (p.x) Sophie Denève Group for Neural Theory, DEC Ecole Normale Supérieure Paris, France Massimiliano Di Luca MPI for Biological Cybernetics Tübingen, Germany Fulvio Domini Cognitive and Linguistic Sciences Brown University Providence, RI Marc O. Ernst MPI for Biological Cybernetics Tübingen, Germany Page 2 of 6 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Contributors Christopher R. Fetsch Washington University School of Medicine Anatomy and Neurobiology St. Louis, MO Monica Gori Robotics, Brain and Technical Science Department Italian Institute of Technology University of Genoa Genoa, Italy Yong Gu Washington University School of Medicine Anatomy and Neurobiology St. Louis, MO Adrian Haith Department of Biomedical Engineering Johns Hopkins University Baltimore, MD Robert T. Held Joint Graduate Group in Bioengineering University of California, San Francisco and University of California, Berkeley Berkeley, CA Yun-Xian Ho Department of Biomedical Informatics Vanderbilt University Medical Center Nashville, TN Timothy Hospedales Department of Computer Science Queen Mary University of London London, England Robert A. Jacobs Center for Visual Science Department of Brain & Cognitive Sciences University of Rochester Rochester, NY Daniel Kersten Department of Psychology University of Minnesota Minneapolis, MN David C. Knill Center for Visual Science University of Rochester Rochester, NY Konrad P. Körding Page 3 of 6 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Contributors Department of Physiology, Physical Medicine and Rehabilitation Rehabilitation Institute of Chicago Northwestern University Chicago, IL Michael S. Landy Department of Psychology and Center for Neural Science New York University New York, NY Timm Lochmann Group for Neural Theory, DEC Ecole Normale Supérieure Paris, France Wei Ji Ma Department of Neuroscience Baylor College of Medicine Houston, TX Laurence T. Maloney Department of Psychology and Center for Neural Science New York University New York, NY (p.xi) Melchi M. Michel Center for Perceptual Systems Department of Psychology The University of Texas Austin, TX Rama Natarajan Center for Neural Science New York University New York, NY Alexandre Pouget Center for Visual Sciences Brain and Cognitive Science Department University of Rochester Rochester, NY Pedro Rosas Institute of Biomedical Sciences Faculty of Medicine Universidad de Chile Santiago, Chile Benjamin A. Rowland Department of Neurobiology & Anatomy Wake Forest University School of Medicine Winston-Salem, NC Page 4 of 6 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis Contributors Maneesh Sahani Gatsby Computational Neuroscience Unit University College London London, England Paul Schrater Department of Psychology & Computer Science University of Minnesota Minneapolis, MN Sascha Serwe FernUniversität in Hagen Institute for Psychology Hagen, Germany Anna Seydell Laboratory of Integrative Neuroscience and Cognition Department of Physiology & Biophysics Georgetown University Medical Center Washington, D.C. Ladan Shams Department of Psychology University of California, Los Angeles Los Angeles, CA Ying Shi School of Biological and Health Systems Engineering Arizona State University Tempe, AZ Terrence R. Stanford Department of Neurobiology & Anatomy Wake Forest University School of Medicine Winston-Salem, NC Barry E. Stein Department of Neurobiology & Anatomy Wake Forest University School of Medicine Winston-Salem, NC Julia Trommershäuser Center for Neural Science New York University New York, NY Sethu Vijayakumar 1.28 Informatics Forum School of Informatics University of Edinburgh Edinburgh, Scotland Kunlin Wei Department of Physiology, Physical Medicine and Rehabilitation Page 5 of 6 PRINTED FROM OXFORD SCHOLARSHIP ONLINE (oxford.universitypressscholarship.com). (c) Copyright Oxford University Press, 2021. No Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use. Subscriber: Library genesis