Playing with sound : a theory of interacting with sound and music in video games

198 Pages·2013·6.256 MB·English

by  Collins, Karen

Playing with Sound

Playing with Sound
A Theory of Interacting with Sound and Music in Video Games
Karen Collins
The MIT Press
Cambridge, Massachusetts
London, England
© 2013 Massachusetts Institute of Technology
All rights reserved. No part of this book may be reproduced in any form by any
electronic or mechanical means (including photocopying, recording, or information
storage and retrieval) without permission in writing from the publisher.

MIT Press books may be purchased at special quantity discounts for business or sales
promotional use. For information, please email [email protected] or
write to Special Sales Department, The MIT Press, 55 Hayward Street, Cambridge,
MA 02142.

This book was set in Stone Serif and Stone Sans by Toppan Best-set Premedia Limited,
Hong Kong. Printed and bound in the United States of America.

Library of Congress Cataloging-in-Publication Data
Collins, Karen, 1973–
Playing with sound : a theory of interacting with sound and music in video games /
Karen Collins.
pages cm
Includes bibliographical references and index.
ISBN 978-0-262-01867-8 (hardcover : alk. paper)
1. Interactive multimedia. 2. Video games. I. Title.
QA76.76.I59C653 2013
006.7— dc23
2012025349

10 9 8 7 6 5 4 3 2 1
Contents

Preface: Game Sound Gets Heard vii

Acknowledgments xi
Introduction 1
How Is Interacting w ith Sound Different from Listening t o Sound? 2
The Sound of Music: Musical Sound 3
Interacting with and Listening To 4
Game Players: An Interactive Audience 13
Game Players: An Embodied Cognition Approach to Audience 15
1 Interacting with Sound: A Theory of Action, Image, and Sound 19
Schizophonia: Disembodied Sound? 23
Synchresis: Integrating Sound and Image 26
Kinesonic Synchresis: The Event-Driven Nature of Interactive Sound 32
Kinesonic Congruence and the Player 35
2 Being in the Game: A Sonic Approach 39
Extension and Incorporation 41
Self-Produced Sound 43
Sonic Game Space: Point of Audition in Games 44
Spatial Sonic Embodiment in the Game 56
3 Sound at the Borders: Enacting Game Sound 59
Posing and Playing 60
Voice and Role-Play in Games 68
Alternate-Reality Games 82
New Sonic Boundaries: Identifi cation, Performance, and Cocreativity 87
4 Embodying Game Sound in Performance: The Real and the
Virtual 89
Performing Music in Games 92
Performing to Music in Games 96
vi Contents
Performing Game Music 105
Creating Music from the Game 111
Interacting with the Game as Instrument 119
5 The Second Life of Game Sound: Playing with the Game 121
Sonic Modifi cation and Player-Generated Content 123
Modding Game Sound 126
Art Mods 134
Player-Generated Content: A Fourth Wall of Sound 138
Conclusions 143
Future Directions in Interactive Sound Studies 147

Notes 149

References 155

Audiovisual References 173

Index 179
Preface: Game Sound Gets Heard
A few years ago, I began my previous book, Game Sound: An Introduction to
the History, Theory, and Practice of Video Game Music and Sound Design (MIT
Press, 2008), by discussing the importance of video games and of game
sound in particular. Today, it is taken for granted (at least in some circles)
that the study of sound in games is a valid pursuit. Courses are taught on
campuses everywhere, and research is attracting wider attention. But in
addition to recent changes in the ways in which video games are perceived
publicly and in the academy, there have been a variety of changes in the
industry, in the technology, within game studies, and in my own personal
thinking about game sound that have inspired me to write this follow-up
book.
The video game industry has grown tremendously in the past half-
decade, and most major game development companies have expanded
their sound departments. The roles played by sound professionals have
become increasingly specialized as production becomes more complex for
“ triple A” games— large-scale, epic games with multimillion dollar budgets.
But another significant development in the industry is the rise of indepen-
dent companies, particularly new game companies that have flourished
in the social media space. Zynga ’ s FarmVille game, for example, managed
to attract over 80 million active monthly Facebook users at its peak. Such
games have dominated the social networking world and become a ubiqui-
tous form of casual gaming. 1 Multiplayer online gaming has expanded as
Internet speeds have improved and social networking and smart phones
(notably Apple ’ s iPhone) have gained popularity. Independent game devel-
opers from all over the world have found a foothold in the industry by
creating game applications ( “ apps ” ). The accessibility and affordability of
apps has resulted in the growing popularity of casual gaming in general —
on smart phones, tablets, netbooks, and other portable devices. To a signifi-
cant extent, the success of these devices has depended on the availability
viii Preface
of free or cheap game apps. Social networking and mobile media have had
interesting consequences for game sound, since both forms of casual games
are often played with the sound turned off. The eventual outcome of this
fact for game sound practitioners remains to be seen, but the history of
games is repeating itself once again. Small developers have a space where
they can sell and share their games, and the “ one-man band” game devel-
oper has returned.
Technological changes in the game industry in the past few years have
also affected the development of game sound. Audio middleware engines
such as Wwise and Fmod enable practitioners to demonstrate the interac-
tive elements of their sound without having to program code. 2 New game
sound software such as PsaiCore is opening up audio artificial intelligence
and computer-aided approaches to composition in games. As artificial intel-
ligence gains ground in the game development community (and as the
predictability and planning of narrative and action becomes more diffi-
cult), designers are returning to sound synthesis as they explore new
methods in procedural generation to be more responsive to the player ’ s
actions. Spatial sound has also become more important to console play as
home surround-sound systems have become increasingly affordable. And
more important than ever before is the increased role that gesture and the
player ’ s body play in gaming. With the massive success of the Wii as well
as the introduction of Microsoft ’ s Kinect and Sony ’ s Move, gaming is no
longer about sitting relatively still in front of a screen but now incorporates
whole-body interaction with the game. This gestural involvement is a recur-
ring theme in this book as I explore the implications for the player’ s
embodied experience of game sound.
As the popularity of games continues to grow, scholars have entered
game studies from a wide variety of disciplines, which has resulted in interest-
ing interdisciplinary conferences where computer scientists both clash and
collaborate with cultural theorists. The study of game sound has seen
remarkable growth in just a few years as the literature in this area has
increased, the scope of research has widened beyond what was previously
available, and new ideas and perspectives have been brought in from dif-
ferent disciplines. With this rise in scholarly interest, game sound design
and composition programs and courses have formed in universities and
technical colleges around the world, with a need for literature to support
these courses.
Finally, my own theoretical perspective has changed during my time
at the University of Waterloo, a university known for its strong computer
science and engineering programs. I moved out of a music department
Preface ix
and into a digital arts program where I now teach games and sound design.
Along with my collaborative work with people from the humanities, social
sciences, and natural sciences at Waterloo and beyond, this disciplinary
shift has altered my own thinking about interactivity and sound through
my exploration of less conventional video games (such as slot machines).
One of these research collaborations resulted in several trips to Japan,
where I wandered the otaku alleys of Akihabara and Harajuku and became
fascinated with the role that fandom plays in Japanese video game con-
sumption. This helped develop my interest in the player ’ s experience of
games.
All of these changes have led me to develop a new approach to thinking
about interactive sound that I present here and that in some ways repre-
sents a shift from my previous thinking. I still integrate recent research
from music, film, and cultural studies, but I expand this disciplinary back-
ground to include more work from computer science and psychology. In
short, I build on my previous thinking by offering a companion book
to Game Sound , which focused on the creative, production side of game
sound. In this book, I seek to understand games not as texts but as sites
of participation and practice where players construct meanings. Here, then,
I explore game sound from the other side of the console, focusing on the
player ’ s experience of sound.