The Psychology of Music in Multimedia PDF
Preview The Psychology of Music in Multimedia
The Psychology of Music in Multimedia “This well-edited, laid out, and contextualized collection of essays provides a much needed resource on a topic whose rigorous examination has, until now, been limited to scholarly articles scattered amongst a variety of academic journals. Its appearance could not be more timely, given the steadily increasing interest in cross modal percep- tion, and, specifi cally, perception of audio-visual composites, as evidenced in univer- sity curricula, relevant interdisciplinary research, and the desire of ‘sound for visual media’ industries to anchor their practice and its effectiveness on empirical research. The book’s contributors and editors represent a ‘who’s who’ in the area and their work provides rigorous substance to the ever-growing realization that the presence of an image changes what we ‘hear’ and the presence of a sound changes what we ‘see.’ A must-have resource for experts, students, and practitioners of the topic alike!” Pantelis Vassilakis, Associate Professor and Chair, Audio Arts and Acoustics Department, Columbia College Chicago, USA “This cutting-edge collection of essays highlights new perspectives, research, and ideas about how music impacts many different kinds of media—from fi lm to video games to television advertisements. Any serious media scholar will want this volume as part of their library.” James C. Kaufman Professor of Psychology University of Connecticut, USA Founding Editor, Psychology of Popular Media Culture The Psychology of Music in Multimedia Edited by Siu-Lan T an Annabel J. Cohen Scott D. Lipscomb and Roger A. K endall 1 3 Great Clarendon Street, Oxford, OX2 6DP, United Kingdom Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide. Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries © Oxford University Press 2013 The moral rights of the authors have been asserted First Edition published in 2013 Impression: 1 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, by licence or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this work in any other form and you must impose this same condition on any acquirer British Library Cataloguing in Publication Data Data available ISBN 978–0–19–960815–7 Printed and bound in Great Britain by CPI Group (UK) Ltd, Croydon, CR0 4YY Oxford University Press makes no representation, express or implied, that the drug dosages in this book are correct. Readers must therefore always check the product information and clinical procedures with the most up-to-date published product information and data sheets provided by the manufacturers and the most recent codes of conduct and safety regulations. The authors and the publishers do not accept responsibility or legal liability for any errors in the text or for the misuse or misapplication of material in this work. Except where otherwise stated, drug dosages and recommendations are for the non-pregnant adult who is not breastfeeding. Links to third party websites are provided by Oxford in good faith and for information only. Oxford disclaims any responsibility for the materials contained in any third party website referenced in this work. Foreword Nowadays music hardly exists in the sense in which it existed, or was thought to exist, a hundred years ago. At that time music was conceived as something exclusively defi ned by and contained within the auditory realm. Music might be combined with words or images, as in song or opera, but this was the bringing together of distinct media, each with their own independent existence and their own meaning. By contrast, to think of music as an integral element of multimedia is to think about it quite differently. Seen—as well as heard—this way, music becomes a dimension of a larger, complex experience, and it acquires its meaning from the context of that experience, whether in the concert hall, at home watching television, or playing a video game. This book rethinks music from the perspective of today’s multimedia-oriented world, explores its psychological underpinnings, and illustrates a wide range of empirical techniques through which it can be investigated. This is a book about music today. It is also a book about how music always was, but was not understood to be. The conception of music as an independent, exclusively auditory medium—what Peter Kivy (1990) calls ‘music alone’—was most perfectly realized in the hi-fi culture of the post-war years, above all in the image of music heard on headphones in a darkened room. But the aesthetic aspiration preceded its tech- nological realization. From the second half of the 19th century, concert halls were designed to minimize social interaction and de-emphasize the visual, fostering instead a direct and purely auditory communion between musical work and listener. This is the aesthetic and conceptual order that is still expressed in the institutions of today’s academia: music, words, and moving images are the business respectively of depart- ments of musicology, literature, and fi lm studies. Outside academia, however, the world moved on. Film and television became the 20th century’s dominant forms of entertainment. In the latter decades of the century, video, MTV, and most decisively the development of digital technology normalized the consumption of music in the form of multimedia: not music and words and moving images ( and dance and tactile stimulation and . . . ), but rather music as one dimension of a holistic experience in which meaning emerges from dynamic interactions between multiple media. Multimedia playback has itself converged with computer, Internet, and telephone technologies, resulting in a proliferation of digitally mediated practices of listening and viewing. All this leads us to think of multimedia as a distinctively con- temporary cultural phenomenon, perhaps a dimension of the postmodern experience in the same way that the de-socialized listening of concert hall and headphones were symptomatic of modernism. But there is a sense in which the world never was quite what aestheticians and aca- demics made it out to be. Even during the half century when audio-only technology dominated—say from the 1920s to the 1970s—music was received alongside such vi FOREWORD paratexts as program notes and LP sleeve images. Concert audiences heard music, but they saw it too, in the sometimes considered and always signifi cant gestures of stage performance. And it goes further than that. Even heard on headphones, music elicits perceptions of space and motion, embodied experience, and emotional empathy: it invokes multiple sensory modalities and engages listeners as inhabitants of a world that is experienced through sight, sound, smell, taste, and touch. In that sense the idea of music as multimedia—the phrase is Zohar Eitan’s (Chapter 8, this volume)—is not so much the recognition of a new condition as a new and more realistic recognition of something that has always been the case. The idea of music as an independent, purely auditory medium was protected by walls, ranging from the physical walls of concerts halls that excluded the sounds and sights of the outside world to the conceptual but no less real walls that divide academic disciplines. Seeking to break down such walls, this book concludes with a call for the formation of multidisciplinary research teams in place of the disciplinary fragmenta- tion bequeathed by the old aesthetic and academic order. Just as multimedia is not music and words and moving images ( and . . . ), but rather subsists in the meaningful experience that results from their interaction, so a multidisciplinary approach to mul- timedia means not psychology a nd musicology and communication studies and fi lm studies and semiotics ( and . . . ), but rather a dynamic process of interaction between different perspectives and methodologies. The book also breaches the walls that have traditionally separated the aesthetic from the applied: as a dimension of multimedia experience, music emerges from the following chapters as a cultural practice as much at home in advertising, data visualization, and the creation of computer and video games as in the concert hall. At the same time, the transition from music as medium to music as multimedia gives a new signifi cance to the empirical orientation that lies at the heart of the book, with its focus on the perceptions and experiences of real, situated listener-viewers (it is a real problem that there is no one word in English to designate how we perceive multime- dia). Under the old order, music was assumed to be always already meaningful: mean- ing was deposited into the musical artwork by its composer, to be communicated by the performer and reconstructed by the listener, and the analysis of such meaning was the province of the musicologist. Within that paradigm, the role of empirical investigation was basically limited to testing the extent to which that meaning was or was not in fact communicated. Consequently musicologists saw the work of music psychologists as peripheral to their core business. But in the multimedia experience, as I said, meaning emerges from dynamic interac- tions between media: it is produced in the real time of listening-viewing. In this context, empirical investigation engages fully with the production of meaning, contributing to an understanding of what music means as well as the processes through which it comes to mean what it means. And that makes it possible for empirical approaches to enter into dialogue with the equally evidence-based but different approaches of historical musicology and of what is sometimes referred to as speculative theory (perhaps more accurately seen as a blending of introspection and systematic analysis). It was through that kind of speculative approach that I attempted to extend tradi- tional music-theoretical thinking into the fi eld of multimedia in my book Analysing FOREWORD vii Musical Multimedia (Cook, 1998), fi rst published 15 years before the present book. My aim was to rescue music from the subordinate position in which it had been placed by writing on fi lm that refl ected the practices and biases of the Hollywood system (where the music was added at the last minute to an otherwise completed fi lm); prompted by the then dominant phenomenon of the music video, I wanted to show that words and images could themselves be deployed in musical ways. Another example of a specu- lative approach, coming from a different perspective and informed by the author’s experience as an electroacoustic composer and fi lm-maker, was Michel Chion’s (1994) infl uential book Audio-Vision . The chapters that follow show how such established approaches can coexist productively with approaches informed by empirical methods: speculative approaches provide material for empirical investigation, while empirical fi ndings prompt new speculation. This kind of dialogue between approaches tradition- ally associated with the humanities on the one hand, and with scientifi c investigation on the other, is full of promise as an instrument of disciplinary renewal. Nicholas Cook University of Cambridge, UK References Chion, P. (1994). Audio-vision: Sound on screen (C. Gorbman, Ed. and Trans.). New York, NY: Columbia University Press. (Originally published 1990.) Cook, N. (1998). Analysing musical multimedia . Oxford: Oxford University Press. Kivy, P. (1990). Music alone: Philosophical refl ections on the purely musical experience. Ithaca, NY: Cornell University Press. Acknowledgements We would like to thank Martin Baum for his support and interest in this book from the beginning and Charlotte Green for illuminating our path through every step of the journey with her expert guidance. Our thanks to all the other splendid staff at Oxford University Press including Emma Lonie, Papitha Ramesh, Fiona Richardson, Lauren Small, Katie Stileman, Nic Williams, Simon Witter, Ben Tiller, and Geraldine Begley. We are also grateful to Nicolas Wehmeier and Sarah Brett at OUP for designing the companion website with multimedia features to supplement the content of this book. At every stage of this venture, we were thoroughly impressed by the warm sup- portive style, clear communication, and attention to detail of all OUP staff. We express our appreciation to anonymous reviewers for their expertise and valuable insights that helped strengthen and refi ne this work. We especially thank all chapter authors for their unfl agging dedication and enthusiastic involvement; this book would not have been possible without their diverse and vibrant contributions. Our gratitude is also extended to Nicholas Cook for presenting the Foreword for this book. Finally, the edi- tors express personal thanks to their wonderfully supportive colleagues, friends, and families. ST, AJC, SDL, RAK Contents Companion Website x i Contributors xiii 1 Introduction: The psychology of music in multimedia 1 Annabel J. Cohen, Scott D. Lipscomb, Siu-Lan Tan, and Roger A. Kendall Part I Models and Multidisciplinary Perspectives 2 Congruence-Association Model of music and multimedia: Origin and evolution 17 Annabel J. Cohen 3 Experimental semiotics applied to visual, sound, and musical structures 48 Roger A. Kendall and Scott D. Lipscomb 4 Integrating media effects research and music psychology 66 Mark Shevy 5 Musical analysis for multimedia: A perspective from music theory 89 David Bashwiner 6 Emotion and music in narrative fi lms: A neuroscientifi c perspective 118 Lars Kuchinke, Hermann Kappelhoff, and Stefan Koelsch Part II Cross-Modal Relations in Multimedia 7 Perceived congruence between auditory and visual elements in multimedia 141 Shin-ichiro Iwamiya 8 How pitch and loudness shape musical space and motion 165 Zohar Eitan 9 Cross-modal alignment of accent structures in multimedia 192 Scott D. Lipscomb
The list of books you might like
