Development and Exploration of a Timbre Space Representation of Audio Craig Andrew Nicol

310 Pages·2006·1.53 MB·English

Development and Exploration of a Timbre
Space Representation of Audio
Craig Andrew Nicol
Submitted for the degree of Doctor of Philosophy
University of Glasgow
Department of Computing Science
September, 2005
(cid:176)c Craig Andrew Nicol, 2005
Abstract
Sound is an important part of the human experience and provides valuable infor-
mation about the world around us. Auditory human-computer interfaces do not
have the same richness of expression and variety as audio in the world, and it has
been said that this is primarily due to a lack of reasonable design tools for audio
interfaces.
Thereareanumberofgoodguidelinesforaudiodesignandastrongpsychoacoustic
understanding of how sounds are interpreted. There are also a number of sound
manipulation techniques developed for computer music. This research takes these
ideas as the basis for an audio interface design system. A proof-of-concept of this
system has been developed in order to explore the design possibilities allowed by
the new system.
The core of this novel audio design system is the timbre space. This provides a
multi-dimensional representation of a sound. Each sound is represented as a path
inthetimbrespaceandthispathcanbemanipulatedgeometrically. Severaltimbre
spaces are compared to determine which amongst them is the best one for audio
interface design. The various transformations available in the timbre space are
discussed and the perceptual relevance of two novel transformations are explored
by encoding \urgency" as a design parameter.
This research demonstrates that the timbre space is a viable option for audio inter-
face design and provides novel features that are not found in current audio design
systems. A number of problems with the approach and some suggested solutions
are discussed. The timbre space opens up new possibilities for audio designers to
explore combinations of sounds and sound design based on perceptual cues rather
than synthesiser parameters.
Dedication
To Dad.
1950-2002
Acknowledgments
First of all, my heartfelt and sincere thanks to Mary for her love, encouragement,
understanding and assistance.
Thanks to Stephen and Phil for their supervision, inspiration and motivation.
Thanks to David for his help with LaTeX, and thanks to Lorna for the software to
run the experiments in Chapter 6.
Thanks to my family and friends for their support. Thanks to everyone who par-
ticipated in the perception experiment described in Chapter 6.
The Wave writing component of the code was written by Don Cross[1] and was
downloaded from his web-site.
Thanks to Ron Kuper of Cakewalk, who has made available the source to the DXi
host and has helped in understanding the technical details of the speciflcation.
TheMatlabimplementationoftheEM-Kalmanalgorithm,onwhichtheEMversion
of PCA was based, was developed by A-V.I. Rosti [2].
The Matlab implementation of the CQT algorithm used in this research was writ-
ten by Judy Brown [3] with an improved inverse by Derry Fitzgerald at the Cork
Institute of Technology.
Thek-dtreeimplementationusedhereisfromtheANNlibrarybyMountandAnil
[4].
This PhD was funded from an EPSRC studentship.
Declaration
Thematerialpresentedinthisthesisistheresultofmyownresearchcarriedoutat
the Department of Computing Science at the University of Glasgow working under
thesupervisionofProfessorStephenBrewsterandPhilipGray. Someoftheresults
presented in Chapters 4 and 5 have previously been presented at CADUI 2004 [5]
and ICAD 2004 [6].
Contents
Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii
Dedication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv
Declaration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 General overview of the fleld . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Aims and objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Typographic conventions . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2 Audio perception, interfaces and synthesis 10
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Perception of sound . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 What do we hear? . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.2 Equal-loudness curves . . . . . . . . . . . . . . . . . . . . . . 14
2.2.3 Perception of phase . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.4 Situated audio . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.2.5 Analytical listening . . . . . . . . . . . . . . . . . . . . . . . . 18
2.2.6 Ecological listening . . . . . . . . . . . . . . . . . . . . . . . . 19
2.2.7 Ecological design . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.8 Representing perception of timbres . . . . . . . . . . . . . . . 22
2.2.9 Implications for audio design . . . . . . . . . . . . . . . . . . 23
2.3 Interfaces and design . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.3.1 Cultural independence . . . . . . . . . . . . . . . . . . . . . . 25
vi
2.3.2 Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3.3 Auditory Icons . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.3.4 Earcons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.3.5 Comparison of Auditory Icons and Earcons . . . . . . . . . . 31
2.3.6 Combining Auditory Icons and Earcons . . . . . . . . . . . . 32
2.3.7 Other audio interactions . . . . . . . . . . . . . . . . . . . . . 34
2.3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4 Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.4.1 MIDI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4.2 Wavetable synthesis . . . . . . . . . . . . . . . . . . . . . . . 41
2.4.3 Synthesis using oscillators . . . . . . . . . . . . . . . . . . . . 43
2.4.4 Additive synthesis . . . . . . . . . . . . . . . . . . . . . . . . 45
2.4.5 Frequency Modulation (FM) synthesis . . . . . . . . . . . . . 46
2.4.6 Subtractive synthesis . . . . . . . . . . . . . . . . . . . . . . . 47
2.4.7 Granular synthesis . . . . . . . . . . . . . . . . . . . . . . . . 48
2.4.8 Physical modelling . . . . . . . . . . . . . . . . . . . . . . . . 50
2.4.9 Justiflcation for choice of synthesiser . . . . . . . . . . . . . . 50
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3 Theory of timbre spaces 55
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
3.2 Perceptual mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.2.1 Methods of mapping . . . . . . . . . . . . . . . . . . . . . . . 59
3.2.2 Map from perceptual to synthesis . . . . . . . . . . . . . . . . 60
3.2.3 Map from synthesis to perceptual . . . . . . . . . . . . . . . . 61
3.2.4 Choosing the direction for the mapping . . . . . . . . . . . . 62
3.3 Construction of timbre spaces . . . . . . . . . . . . . . . . . . . . . . 62
3.3.1 Signal analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.3.2 Dimensionality reduction . . . . . . . . . . . . . . . . . . . . 83
3.4 Perception in timbre spaces . . . . . . . . . . . . . . . . . . . . . . . 90
3.5 Synthesis from timbre spaces . . . . . . . . . . . . . . . . . . . . . . 92
3.5.1 Additive synthesis . . . . . . . . . . . . . . . . . . . . . . . . 93
3.5.2 Non-linear mapping . . . . . . . . . . . . . . . . . . . . . . . 94
3.5.3 Synthesiser choice . . . . . . . . . . . . . . . . . . . . . . . . 102
3.6 Alternatives to timbre spaces . . . . . . . . . . . . . . . . . . . . . . 103
3.6.1 Suitability of timbre spaces . . . . . . . . . . . . . . . . . . . 103
3.6.2 Alternative technologies for audio design . . . . . . . . . . . . 107
3.6.3 Discussion of alternatives . . . . . . . . . . . . . . . . . . . . 109
3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4 Overall system design 114
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.3 Data (cid:176)ow overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.4 Conceptual model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
4.4.1 Object parameters . . . . . . . . . . . . . . . . . . . . . . . . 120
4.4.2 Timbre space . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.4.3 Synthesis engines . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.5 Mapping of space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4.6 Studio technology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.6.1 VSTi from Steinberg . . . . . . . . . . . . . . . . . . . . . . . 125
4.6.2 DXi from Cakewalk . . . . . . . . . . . . . . . . . . . . . . . 126
4.6.3 Cross-platform support . . . . . . . . . . . . . . . . . . . . . 128
4.7 Storage and manipulation of audio data . . . . . . . . . . . . . . . . 130
4.7.1 Parameter flle format . . . . . . . . . . . . . . . . . . . . . . 132
4.7.2 Timbre space flle format . . . . . . . . . . . . . . . . . . . . . 135
4.7.3 Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
4.7.4 Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.7.5 DXi parameter data format . . . . . . . . . . . . . . . . . . . 141
4.8 Current implementation status . . . . . . . . . . . . . . . . . . . . . 145
4.8.1 Current status of the DXi . . . . . . . . . . . . . . . . . . . . 145
4.8.2 Current status of the synthesiser host . . . . . . . . . . . . . 147
4.9 Extending the system . . . . . . . . . . . . . . . . . . . . . . . . . . 148
4.9.1 How to add a new analysis technique . . . . . . . . . . . . . . 148
4.9.2 TFR matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
4.9.3 How to add a new synthesiser . . . . . . . . . . . . . . . . . . 149
4.10 A simple GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.10.1 How to add GUI components . . . . . . . . . . . . . . . . . . 150
4.10.2 Extending the GUI . . . . . . . . . . . . . . . . . . . . . . . . 152
4.11 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
5 Evaluation of timbre spaces 155
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.2 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
5.2.1 Testing of hypotheses . . . . . . . . . . . . . . . . . . . . . . 159
5.3 Choosing a timbre space . . . . . . . . . . . . . . . . . . . . . . . . . 163
5.3.1 Test suite of timbre spaces. . . . . . . . . . . . . . . . . . . . 165
5.3.2 Speed and size of spaces . . . . . . . . . . . . . . . . . . . . . 167
5.3.3 Accuracy of sound reproduction . . . . . . . . . . . . . . . . 169
5.3.4 General method for timbre space comparisons . . . . . . . . . 169
5.3.5 Efiect of technique . . . . . . . . . . . . . . . . . . . . . . . . 171
5.3.6 Efiect of phase . . . . . . . . . . . . . . . . . . . . . . . . . . 175
5.3.7 Efiect of windows . . . . . . . . . . . . . . . . . . . . . . . . . 177
5.3.8 Efiect of window length . . . . . . . . . . . . . . . . . . . . . 180
5.3.9 Efiect of window ofiset . . . . . . . . . . . . . . . . . . . . . . 183
5.3.10 Chosen timbre space . . . . . . . . . . . . . . . . . . . . . . . 185
5.4 Extending the timbre space with new sounds . . . . . . . . . . . . . 186
5.4.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.4.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
5.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
5.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
5.5 Comparison of FM and additive synthesis . . . . . . . . . . . . . . . 190
5.5.1 Smoothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
5.5.2 Additive synthesis . . . . . . . . . . . . . . . . . . . . . . . . 192
5.5.3 FM synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
5.5.4 Gaver’s Auditory Icon synthesiser . . . . . . . . . . . . . . . 194
5.5.5 Earcons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
5.5.6 Path morphing . . . . . . . . . . . . . . . . . . . . . . . . . . 197
5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
6 Perception of urgency in timbre space 202
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
6.2 Two methods for modelling urgency . . . . . . . . . . . . . . . . . . 203
6.3 Hypotheses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
6.3.1 Consistency of rank . . . . . . . . . . . . . . . . . . . . . . . 205
6.3.2 Comparison of vector and gravity models . . . . . . . . . . . 206
6.3.3 Comparison of musicians and non-musicians . . . . . . . . . . 206
6.4 Experimental design . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
6.4.1 Experiment implementation . . . . . . . . . . . . . . . . . . . 207
6.4.2 Statistical analysis . . . . . . . . . . . . . . . . . . . . . . . . 209