Integrating Face and Voice in Person Perception PDF
Preview Integrating Face and Voice in Person Perception
I ntegrating Face and Voice in Person Perception Pascal Belin Salvatore Campanella Thomas Ethofer Editors Integrating Face and Voice in Person Perception Editors Pascal Belin Salvatore Campanella Voice Neurocognition Laboratory Laboratory of Psychological Medicine Institute of Neuroscience and Psychology Free University of Brussels College of Medical, Veterinary Brussels, Belgium and Life Sciences and University of Glascow Psychiatry Department (EEG) Glascow, UK CHU Brugmann The Belgian Fund for Scientifi c International Laboratories for Brain Research (FNRS) Music and Sound (BRAMS) Brussels, Belgium Université de Montréal & McGill University Montreal, Quebec, Canada Thomas Ethofer Department of General Psychiatry University of Tübingen Tübingen, Germany ISBN 978-1-4614-3584-6 ISBN 978-1-4614-3585-3 (eBook) DOI 10.1007/978-1-4614-3585-3 Springer New York Heidelberg Dordrecht London Library of Congress Control Number: 2012942182 © Springer Science+Business Media New York 2013 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifi c ally the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfi l ms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Exempted from this legal reservation are brief excerpts in connection with reviews or scholarly analysis or material supplied specifi c ally for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of this publication or parts thereof is permitted only under the provisions of the Copyright Law of the Publisher’s location, in its current version, and permission for use must always be obtained from Springer. Permissions for use may be obtained through RightsLink at the Copyright Clearance Center. Violations are liable to prosecution under the respective Copyright Law. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specifi c statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com) F oreword M ost of our social interactions involve combining information from both the face and voice of other persons: speech information, and also crucial nonverbal informa- tion on a person’s identity and affective state. The cerebral bases of the multimodal integration of speech have been intensively investigated; by contrast, only a few studies have focused on n onverbal aspects of face– v oice integration. U ntil recently, the quite different approaches used by investigators in auditory and visual perception have hindered efforts at bringing these two fi e lds together: auditory perception largely concentrated on speech processing, while visual percep- tion essentially investigated object and face recognition. Such emphasis on different types of information in the two modalities has not facilitated the understanding of how social signals from the face and voice are combined and integrated in everyday behaviour. I n an effort towards a broader perspective on these two research fi e lds, we noted, as several other authors before us did, that information carried by faces and voices— speech, affect, identity—is largely similar in kind (if not in the underlying physical signals) and proposed that this similarity could extend to the underlying cerebral processing functional architecture. We suggested that Bruce and Young’s (1986) infl u ential model of face processing could be meaningfully extended to the process- ing of voice information. I n the “auditory face” model of voice processing (Belin, Fecteau, & Bedard, 2004), we proposed that the three main types of vocal information—speech, affect, identity—are processed, as for faces, in three partially independent functional path- ways that interact during normal behaviour but can be impaired selectively from one another. In a subsequent refi n ing of this model (Campanella & Belin, 2007) we proposed that multimodal face–voice integration occurs between corresponding pathways across the visual and auditory modalities (Fig. 1). This model, as most models, is probably wrong, but it was proposed in the hope that it could provide a useful heuristic to guide further research on the way our brain combines signals from the face and voice of other persons. v vi Foreword F ig. 1 T he “auditory face” model of voice perception. The r ight-hand part of the fi g ure is adapted from Bruce and Young’s (1986) model of face perception. The l eft-hand part proposes a similar functional organization for voice processing. D ashed arrows indicate multimodal interactions. Reprinted (permission pending) from Belin, Fecteau, and Bedard (2004) T he present book aims to highlight recent exciting advances in the investigation of the behavioural and cerebral bases of face–voice integration in the context of person perception, focusing on the integration of a ffective and i dentity information. Several research domains are brought together: behavioural and neuroimaging work in healthy adult humans, and also evidence from several other relevant research fi e lds to provide complementary insights. P art I: Evolution and development provides both evolutionary and developmen- tal perspectives on face–voice integration. Do other animals show evidence of face– voice integration? And how early do these abilities develop in human infants? I pek G. Kulahci and A sif A. Ghazanfar review evidence in primates showing that multi- sensory processes enhance multiple types of behaviour and that cortical processes are largely multisensory by default. Akihiro Izumi reviews research paradigms employed in probing auditory–visual conceptual representations and highlights evi- dence for clear multimodal integration processes in non-human primates. M aria M. Diehl and L iz M. Romanski examine integrative properties of neurons in the macaque ventro-lateral prefrontal cortex and show that these cells respond optimally to face and vocalization stimuli, exhibiting multisensory enhancement or suppression when face and vocalization stimuli are combined. R oss Flom reviews evidence on the development face and voice perception and integration and interprets this evidence in the context of the “intersensory redundancy hypothesis”. T obias Grossman pres- ents evidence on the development of face–voice integration with a focus on affec- tive information and shows that at least by the age of 7 months, infants reliably integrate and recognize emotional information across face and voice. Foreword vii P art II: Identity information examines the integration of identity information from faces and voices, which play a central role in our social interactions. Indeed, both faces and voices are rich in information on a person’s identity and gender. S tefan R. Schweinberger reviews evidence illustrating these audiovisual interac- tions during familiar speaker recognition. R ebecca Watson and c olleagues focus on face–voice integrative processes related to gender, using dynamic, ecological “mor- phed video” stimuli. F rederic Joassin and S alvatore Campanella argue that the cross-modal integration of identity and gender information through faces and voices involve similar networks. While the above chapters report results obtained using functional magnetic resonance imaging, A ina Puce reviews evidence on face–voice integrative processes obtained using neurophysiological techniques. P art III: Affective Information is dedicated to the integration of emotional information in voice (affective prosody) and face (emotional facial expressions). G illes Pourtois and M onica Dhar review theoretical models and provide behav- ioural data arguing that perception of emotion can be conceptualized as an object- based multisensory phenomenon. T obias Brosch and D idier Grandjean discuss cross-modal infl u ences of emotion on spatial attention and their neural correlates as determined by electrophysiological and brain imaging methods. B enjamin Kreifelts, D irk Wildgruber and T homas Ethofer provide an overview on methodological approaches for studying audiovisual integration of emotion using functional mag- netic resonance imaging and discuss the advantages and weaknesses of different analysis strategies. B eatrice de Gelder, B ernard M.C. Stienen and J an Van den Stock review the explosion of research on emotional face–voice integration since their pioneering innovative work in this domain. They extend the review to abnor- mal affective processing in schizophrenia and autism—a perfect transition to the last part of the book. P art IV: Impairment illustrates the importance of cross-modal face–voice inter- actions by showing their impact on people’s life when these processes are altered. P ierre Maurage, S cott Love and Fabien D’Hondt provide evidence for a cross- modal defi c it when chronic alcoholic patients are confronted with emotional stim- uli. B arbra Zupan stresses the role of audition in processing of bimodal cues of speech and emotion in individuals with hearing loss. D yna Delle-Vigne, C harles Konreich, P aul Verbank and S alvatore Campanella suggest that emotional cross- modal stimulations, through the use of cognitive event-related potentials, may help discriminate more clearly between different psychiatric populations. T hese contributions refl e ct a dynamic, emerging research fi e ld situated at the conjunction between two active currents of neuroscience and psychology: Social Neuroscience and Multimodal Integration. We hope they illustrate important recent advances in these exciting domains and constitute an interesting reading. G lasgow, UK Pascal Belin Brussels, Belgium Salvatore Campanella Tubingen, Germany Thomas Ethofer viii Foreword R eferences B elin, P., Fecteau, S., & Bedard, C. (2004) Thinking the voice: neural correlates of voice perception. T rends in Cognitive Sciences, 8, 129–135. B ruce, V., & Young, A. (1986) Understanding face recognition. B ritish Journal of Psychology, 77, 305–327. C ampanella, S., & Belin, P. (2007) Integrating face and voice in person perception. Trends in Cognitive Sciences, 11, 535–543. Contents Part I Evolution and Development 1 Multisensory Recognition in Vertebrates (Especially Primates) ........ 3 Ipek G. Kulahci and Asif A. Ghazanfar 2 Cross-Modal Representation in Humans and Nonhuman Animals: A Comparative Perspective .................................................... 29 Akihiro Izumi 3 Representation and Integration of Faces and Vocalizations in the Primate Ventral Prefrontal Cortex ............................................. 45 Maria M. Diehl and Lizabeth M. Romanski 4 Intersensory Perception of Faces and Voices in Infants ...................... 71 Ross Flom 5 The Early Development of Processing Emotions in Face and Voice ..................................................................................... 95 Tobias Grossman Part II Identity Information 6 Audiovisual Integration in Speaker Identification ............................... 119 Stefan R. Schweinberger 7 Audiovisual Integration of Face–Voice Gender Studied Using “Morphed Videos” ....................................................................... 135 Rebecca Watson, Ian Charest, Julien Rouger, Christoph Casper, Marianne Latinus, and Pascal Belin ix
The list of books you might like
