Soma: live performance where congruent musical, visual, and proprioceptive stimuli fuse to form a

245 Pages·2011·7.62 MB·English

by  Bergstrom, I.

Soma: live performance where congruent musical,
visual, and proprioceptive stimuli fuse to form a
combined aesthetic narrative

Ilias Bergstrom

UCL

A thesis submitted for the degree of
Doctor of Philosophy

2010

1
I, Ilias Bergstrom, confirm that the work presented in this thesis is my own. Where
information has been derived from other sources, I confirm that this has been
indicated in the thesis.
2
Abstract
Artists and scientists have long had an interest in the relationship between music and
visual art. Today, many occupy themselves with correlated animation and music,
called ‗visual music‘. Established tools and paradigms for performing live visual
music however, have several limitations:
Virtually no user interface exists, with an expressivity comparable to live musical
performance.
Mappings between music and visuals are typically reduced to the music‘s beat and
amplitude being statically associated to the visuals, disallowing close audiovisual
congruence, tension and release, and suspended expectation in narratives.
Collaborative performance, common in other live art, is mostly absent due to
technical limitations.
Preparing or improvising performances is complicated, often requiring software
development.
This thesis addresses these, through a transdisciplinary integration of findings from
several research areas, detailing the resulting ideas, and their implementation in a
novel system:
Musical instruments are used as the primary control data source, accurately encoding
all musical gestures of each performer. The advanced embodied knowledge musicians
have of their instruments, allows increased expressivity, the full control data
bandwidth allows high mapping complexity, while musicians‘ collaborative
performance familiarity may translate to visual music performance.
The conduct of Mutable Mapping, gradually creating, destroying and altering
mappings, may allow for a narrative in mapping during performance.
The art form of Soma, in which correlated auditory, visual and proprioceptive
stimulus form a combined narrative, builds on knowledge that performers and
audiences are more engaged in performance requiring advanced motor knowledge,
and when congruent percepts across modalities coincide.
Preparing and improvising is simplified, through re-adapting the Processing
programming language for artists to behave as a plug-in API, thus encapsulating
complexity in modules, which may be dynamically layered during performance.
3
Design research methodology is employed during development and evaluation, while
introducing the additional viewpoint of ethnography during evaluation, engaging
musicians, audience and visuals performers.

4
Acknowledgments
Firstly, I wish to thank my supervisors, Beau Lotto and Anthony Steed, for taking on
my project, and for helping me through with the work presented in this thesis.
I am further grateful to have been surrounded by very interesting people at the
Lottolab. Without Lefkothea Andreou, Daniel Hulme, David Malkin, David Corney,
Erwan LeMartelot, Martina Wicklein, Rich Clarke, Madi Boyd, Emma Byrne, Sam
Walker, and Katherine Bash, the time spent working in the lab would have been far
less stimulating.
Many thanks are due to Nadia Berthouze, Nick Bryan-Kinns, Alan Blackwell,
Marcelo Wanderley, Katherine Stevens, Nigel Jenkins, Olivier Ruellet, Mauritius
Seeger, Mat Hourteillan, Alison Hardcastle, Anne-Marie Preston, Illi Adatto, Dor
Wand, the members of Accretion and of the Gathering, for their invaluable advice and
assistance towards completing the work presented here.
I owe my gratitude to my family, without the support of which I would never have
had the chance of writing these lines.
And last but not least, Giulia!

5
Table of Contents
Soma: live performance where congruent musical, visual, and proprioceptive stimuli fuse to
form a combined aesthetic narrative .........................................................................................1
Ilias Bergstrom ............................................................................................................................1
Abstract ..............................................................................................................................3
Acknowledgments ..............................................................................................................5
Table of Contents ...............................................................................................................6
Table of Figures ............................................................................................................... 11
1. Introduction ..................................................................................................................... 15
1.1 Limitations with current practice ............................................................................ 17
1.2 Proposed system ..................................................................................................... 21
1.2.1 Increasing expressivity ..................................................................................... 22
1.2.2 Simplifying preparation for performance ........................................................ 23
1.2.3 Increasing achievable mapping complexity and variability ............................. 24
1.2.4 Allowing for collaborative performance .......................................................... 25
1.2.5 Soma: a new artistic practice .......................................................................... 25
1.2.6 A new performer role ...................................................................................... 27
1.3 Methodology ........................................................................................................... 29
1.4 Summary of contributions, and how each is evaluated .......................................... 33
1.5 Thesis structure ....................................................................................................... 36
2 Background ...................................................................................................................... 39
2.1 Relating visual arts to music .................................................................................... 39
2.1.1 Visual Music, Intermedia, Abstract film, and Synesthetic Art ......................... 39
2.1.2 Colour Organs and Lumia ................................................................................ 43
2.1.3 Audiovisual Composition ................................................................................. 49
2.2 Musical Instruments ................................................................................................ 54
2.2.1 Electronic musical instruments and the interface using which these are played
55
6
2.2.2 Analysis of Digital Musical Instruments ........................................................... 59
2.2.3 Mapping ........................................................................................................... 61
2.2.4 Embodied / Enactive knowledge, and music virtuosity................................... 63
2.2.5 Collaborative performance, mutual engagement and group flow.................. 64
2.2.6 The mixing engineer ........................................................................................ 65
2.2.7 Performance using Conducting Gesture DMIs ................................................ 66
2.3 Musical Narrative .................................................................................................... 69
2.3.1 Absolute music ................................................................................................ 69
2.3.2 Narrative in visual music and audiovisual art .................................................. 71
2.4 Programming as art: Creative Code ........................................................................ 73
2.4.1 New Media art ................................................................................................. 74
2.4.2 Demo Scene ..................................................................................................... 75
2.4.3 Live Coding ....................................................................................................... 77
2.5 Programming for non-computer scientists ............................................................. 78
2.5.1 Visual programming languages ....................................................................... 79
2.5.2 Procedural languages, and the Processing language ...................................... 81
2.6 Review of related research and evaluation methodology ...................................... 85
2.6.1 Evaluation of Music Technologies ................................................................... 86
2.6.2 Human Computer Interaction ......................................................................... 89
2.6.3 Review of evaluation methodology employed in recent PhD theses analogous
to present work ............................................................................................................. 105
2.7 Recent related work .............................................................................................. 108
3 Mutable Mapping .......................................................................................................... 112
3.1 Related work to mutable mapping ........................................................................ 114
3.2 The mutable mapping performer .......................................................................... 114
4 The Trinity system ......................................................................................................... 117
4.1 Mother ................................................................................................................... 117
4.2 Live Input Processor .............................................................................................. 119
7
4.3 Mediator ................................................................................................................ 120
4.4 Hardware ............................................................................................................... 127
4.5 Supported modes of performance ........................................................................ 129
5 Process........................................................................................................................... 131
5.1 First prototyping stage .......................................................................................... 131
5.2 Second prototyping stage ...................................................................................... 133
5.3 Third prototyping stage: the birth of Mother ....................................................... 136
5.4 Fourth prototyping stage: Mutable Mapping and the Mediator .......................... 138
5.5 Fifth prototyping stage: non-musical control ........................................................ 139
6 Experiments ................................................................................................................... 141
6.1 Online audience survey ......................................................................................... 141
6.2 Rehearsals and live performances with groups of musicians ............................... 144
6.3 Interviews with VJs and live visuals performers .................................................... 146
7 Evaluation ...................................................................................................................... 149
7.1 Results from online survey .................................................................................... 149
7.2 Results from rehearsals with musicians ................................................................ 151
2.7.1 Setup of conducted rehearsals ...................................................................... 151
7.2.1 Summary of conclusions from rehearsals ..................................................... 153
7.3 Results from interviews with VJs and live visuals performers .............................. 155
7.3.1 Olivier Ruellet, AKA Ctrl-N ............................................................................. 155
7.3.2 Mauritius Seeger, AKA Dr. Mo ....................................................................... 158
7.3.3 Mat Hourteillan, AKA Matsai, VJ Om boy ...................................................... 160
7.3.4 Nigel Jenkins, AKA Nebulus ........................................................................... 161
7.3.5 Conclusions from interviews with VJs and live visuals performers ............... 165
7.4 Further observed impact of present work ............................................................ 167
7.5 Technical evaluation .............................................................................................. 167
7.6 Evaluation discussion............................................................................................. 169
8 Further work .................................................................................................................. 172
8
8.1 Experiments ........................................................................................................... 172
8.2 Development ......................................................................................................... 173
8.3 Content .................................................................................................................. 174
9 Conclusion ..................................................................................................................... 176
9.1 Summary of contribution ...................................................................................... 176
9.2 Closing remarks ..................................................................................................... 179
Appendix A: Online survey questionnaire & responses ........................................................ 181
Appendix B: Reports from rehearsals and performances ..................................................... 195
The Gathering, St Pancras Old Church .............................................................................. 195
Guitarist 1: ..................................................................................................................... 196
Guitarist 2: ..................................................................................................................... 197
Percussion synthesizer player: ...................................................................................... 197
Accretion, EC1 Music Project ............................................................................................. 198
Guitarist: ........................................................................................................................ 199
Singer: ............................................................................................................................ 199
Drummer: ...................................................................................................................... 200
Live performance: The Gathering, St Pancras Old Church, 16/12/09 ............................... 201
Rehearsal towards live performance at Inspiral Lounge, 08/01/10 .................................. 202
Percussion synthesizer player: ...................................................................................... 203
Maschine and synthesizer player: ................................................................................. 203
Live performance: Inspiral Lounge, Camden Town, 26/01/10 .......................................... 204
Appendix C: Call for musicians .............................................................................................. 206
I am looking for musicians to accompany with live visuals!.................................................. 206
Appendix D: Non-exhaustive list of visual synthesizers created ........................................... 207
Appendix E: Example code of Processing sketch turned into visual synths for Mother ....... 215
Gradient Processing Sketch ............................................................................................... 215
Gradient Visual Synth for Mother ..................................................................................... 216
Appendix F: Presence research ............................................................................................. 219
9
Relevance to Presence field .......................................................................................... 220
Appendix G: Physiological Measurements for Detecting Enjoyment and Engagement in Video
Games, Affective Computing ................................................................................................. 222
Appendix H: Allowing for multiple interpretations during evaluation process .................... 224
Appendix I: HCI, Interaction Design, and present work ........................................................ 229
References ............................................................................................................................. 230

10