Table Of ContentTowards Autonomous Agents for Live Computer Music: Realtime
Machine Listening and Interactive Music Systems
Nicholas M. Collins
St.John’s College
Centre for Music and Science
Faculty of Music
University of Cambridge
This dissertation is submitted for the degree of Doctor of Philosophy
PhD supervisors Dr. Ian Cross and Dr. Alan Blackwell
External examiners Dr. Michael Casey and Dr. Jonathan Impett
The research described in this dissertation was carried out between October 2003 and August
2006. This dissertation is the result of my own work and includes nothing which is the
outcome of work done in collaboration except where specifically indicated in the text. No part
of this has been submitted to any other university. This dissertation contains no more than
80000 words.
For a world full of composers and the artificial intelligences which might supplant them
Acknowledgements
With a huge number of people to thank, my first acknowledgement is to all those I’m about to
miss out for reasons of space and my ailing memory! Your support is immeasurable and cannot
be put into words, especially where I have forgotten you.
There are a vast array of friends from Cambridge and London life who have contributed
through their support and as healthily as not, distractions. Sorry I cannot make a huge list of
you all, but then, perhaps it’s not so healthy for you to read my thesis anyway?
The musicians I collaborated with for concert performances deserve special mention. Dave
Ellis played drums, Nikki Moran coaxed the Sitar, Ian Cross guided the guitar, Dan Tidhar
tinkled the harpsichord and Inga Maria Klaucke excited baroque recorders. Thankyou to West
Road Concert Hall and Kettles’ Yard, and to my fellow composer Julio d’Escrivan and all
involved with electronic music at Anglia Ruskin University.
A number of electronica artists gave their time to correspond. From London I have to thank
Alex McLean, a TOPLAP founder member and still my favourite laptop perfomer. And big-
up arrow to the live code massive. Chris Jeffs demoed his Cylob Music System, Matt Black
(Coldcut) allowed himself to be interviewed and Tom Jenkinson (Squarepusher) simply gave
permission for a computational analysis that appears in this PhD.
The audiovisual duo klipp av have been running around the world during the last three
years, and whilst I have not gone overboard in describing our activities in this thesis (indeed
I’ve almost suppressed them!), much of our work has grown out of research I’ve been able to
carry out in my time in Cambridge. So first, a big thankyou to Fredrik Olofsson, whose name I
always spell correctly even when others fail, and whose brilliant desire for visual excellence and
coffee/alcohol have been a driving force behind our tours. But most importantly, thankyou to
all we met on our travels, all who helped organise events for us and those who we met on the
way.
Many individual researchers have supported this thesis through the exchange of publications
and data. I wish to thank the Queen Mary University of London group for extremely useful
exchanges. In particular Juan Bello provided onset annotation data and detection code, and
Matthew Davies kept me informed about his beat tracking research and provided a MATLAB
prototype of one of his models. Before leaving Cambridge, Stephen Hainsworth provided his
beat induction test suite and MATLAB code; internationally, Tristan Jehan and Anssi Klapuri
made their algorithms available for testing. Dirk Moelants and Martin McKinney provided
tapping data for reaction times. Joseph Timoney supplied MATLAB code and Brian Glasberg
and Michael Stone provided loudness model code and ISO2003 equal loudness contour data.
Many thanks are due to the MIREX2005 testing group and co-ordination team for all their
hard work in overseeing this contest. Also, Adam Lindsay organised a workshop on feature
based editing for ICMC2005, and even covered the fee so I could more easily attend it!
A consideration of the cognitive basis of beat induction was prompted by the musical entrain-
ment conference series co-organised between Cambridge and the Open University. I attended
meetings at Ohio State and Florida Atlantic thanks to funding from the Faculty of Music here,
and the Entrainment Network itself.
I must mention the SuperCollider list. A big thankyou to James McCartney for writing the
software in the first place and the developers for maintaining it in a usable condition! And in
particular, cheers to Julian Rohrhuber, Scott Wilson and John Eacott for collaboration running
SuperCollider summer schools in recent years.
For immense time and efforts in discussion I have to thank my fellow Centre for Music
and Science graduate students. Chapter 2 of this thesis (and indeed, issues relating to many
other parts) were openly discussed in a student seminar series organised by Tommi Himberg.
For chapter proof-reading in particular I acknowledge John Bispham, Taylan Cemgil, Mark
d’Inverno, Jessica Grahn, Justin London, Martin Rohrmeier, Dan Tidhar, Nick Whiteley and
Matthew Woolhouse, and Roger Dannenberg, Rudi Villing and those anonymous reviewers who
gave comments on papers relating to these research projects.
Thankyou to my external examiners for agreeing to take the time in their busy schedules for
assessing this work. A great deal of thanks must also be conferred on my two supervisors, Ian
and Alan, who were always there to advise me when I needed them, and who still gave me the
independence to research creatively. One of the joys of being supervised by such widely read
and accomplished people has been the broadening of my own knowledge and outlook.
Conference funding was provided by the AHRC, the Digital Music Research Network, St.John’s
College and the Faculty of Music. This research was generously supported throughout by AHRC
grant 2003/104481.
Finally, with love to my family.
3
Related Publications
Some of the work contained within this thesis has appeared in the following publications:
Material from chapter 2 was first presented at the Rhythm Perception and Production Workshop
(RPPW10) in Bilzen, Belgium in July 2005.
Work on perceptual attack time and the psychology of beat tracking appeared in the proceedings
of ICMPC06:
Nick Collins (2006) Investigating computational models of perceptual attack time.
Nick Collins (2006) Towards a style-specific basis for beat tracking.
Chapter 3 contains results also presented in:
Nick Collins (2005) A Change Discrimination Onset Detector with Peak Scoring Peak Picker
and Time Domain Correction. Music Information Retrieval Exchange (MIREX2005) http://
www.music-ir.org/evaluation/mirex-results/audio-onset/index.html
Nick Collins (2005) Using a Pitch Detector for Onset Detection. International Conference
on Music Information Retrieval (ISMIR2005), London, September 11-15th 2005.
Nick Collins (2005) A Comparison of Sound Onset Detection Algorithms with Emphasis
on Psychoacoustically Motivated Detection Functions. Proceedings of AES118 Convention,
Barcelona.
The beat tracking model and interactive music system DrumTrack first appeared in:
Nick Collins (2005) DrumTrack: Beat Induction from an Acoustic Drum Kit with Synchro-
nised Scheduling. Proceedings of the International Computer Music Conference, Barcelona.
Earlier realtime on-the-fly event analysis work was presented at conferences:
Nick Collins (2005) An Automated Event Analysis System with Compositional Applications.
Proceedings of the International Computer Music Conference, Barcelona.
Nick Collins (2004) On Onsets On-the-fly: Real-time Event Segmentation and Categorisation
as a Compositional Effect. Proceedings of Sound and Music Computing (SMC04), IRCAM,
Paris.
A few technical results and descriptions of BBCut overlap with:
Nick Collins and Fredrik Olofsson (2006) klipp av: Live Algorithmic Splicing and Audiovisual
Event Capture. Computer Music Journal 30(2).
Nick Collins (2006) BBCut2: Incorporating Beat Tracking and On-the-fly Event Analysis.
Journal of New Music Research 35(1).
Abstract
Musical agents which can interact with human musicians in concert situations are a real-
ity, though the extent to which they themselves embody human-like capabilities can be called
into question. They are perhaps most correctly viewed, given their level of artificial intelligence
technology, as ‘projected intelligences’, a composer’s anticipation of the dynamics of a concert
setting made manifest in programming code. This thesis will describe a set of interactive sys-
tems developed for a range of musical styles and instruments, all of which attempt to participate
in a concert by means of audio signal analysis alone. Machine listening, being the simulation
of human peripheral auditory abilities, and the hypothetical modelling of central auditory and
cognitive processes, is utilised in these systems to track musical activity. Whereas much of this
modelling is inspired by a bid to emulate human abilities, strategies diverging from plausible hu-
man physiological mechanisms are often employed, leading to machine capabilities which exceed
or differ from the human counterparts. Technology is described which detects events from an
audio stream, further analysing the discovered events (typically notes) for perceptual features of
loudness, pitch, attack time and timbre. In order to exploit processes that underlie common mu-
sical practice, beat tracking is investigated, allowing the inference of metrical structure which
can act as a co-ordinative framework for interaction. Psychological experiments into human
judgement of perceptual attack time and beat tracking to ecologically valid stimuli clarify the
parameters and constructs that should most appropriately be instantiated in the computational
systems. All the technology produced is intended for the demanding environment of realtime
concert use. In particular, an algorithmic audio splicing and analysis library called BBCut2
is described, designed with appropriate processing and scheduling faculties for realtime opera-
tion. Proceeding to outlines of compositional applications, novel interactive music systems are
introduced which have been tested in real concerts. These are evaluated by interviews with
the musicians who performed with them, and an assessment of their claims to agency in the
sense of ‘autonomous agents’. The thesis closes by considering all that has been built, and the
possibilities for future advances allied to artificial intelligence and signal processing technology.
Contents
1 Introduction 7
1.1 Personal Motivations and Thesis Structure . . . . . . . . . . . . . . . . . . . . . . 9
1.2 Interactive Instrument Research in Computer Music . . . . . . . . . . . . . . . . 12
1.2.1 The Current Use of Computers in Concerts . . . . . . . . . . . . . . . . . 14
1.2.2 Accompaniment Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.3 Interactive Improvisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.2.4 Musical Artificial Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.3 Psychological Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.3.1 Rhythm and Metre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.3.2 Expressive Timing and Movement . . . . . . . . . . . . . . . . . . . . . . 31
1.4 Signal Processing Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.4.1 The Difficulty of Automatic Transcription . . . . . . . . . . . . . . . . . . 33
1.4.2 Computational Beat Tracking . . . . . . . . . . . . . . . . . . . . . . . . . 36
1.4.3 Cognitive Processing of Music and Computational Modelling . . . . . . . 38
1.5 Aims and Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
1.5.1 The BBCut Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
1.5.2 Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
1.5.3 Evaluation Criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
1.5.4 Implementation Technologies . . . . . . . . . . . . . . . . . . . . . . . . . 42
2 Beat Tracking and Reaction Time 44
2.1 Beat Tracking and Reaction Time . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.1.1 Published Results on Reaction Time . . . . . . . . . . . . . . . . . . . . . 46
2.1.2 Measurements and Analysis of Reaction Time and Phase Error . . . . . . 48
2.1.3 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.2 Experiment 1: Phase Determination and Reaction Time From Degraded Signals 50
2.2.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.2.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
2.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3 Experiment 2: Reaction Time After Phase Jumps on Polyphonic Audio . . . . . 54
2.3.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
2.3.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
2.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2
3 Automatic Segmentation 59
3.1 Onset Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.1.1 How do Humans Detect Events? . . . . . . . . . . . . . . . . . . . . . . . 61
3.1.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.2 A Comparison of Onset Detectors With Emphasis on Psychoacoustically Relevant
Detection Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.1 Detection Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
3.2.2 Psychoacoustically Motivated Models . . . . . . . . . . . . . . . . . . . . 65
3.2.3 A Detection Function Based on Equal Loudness Contours . . . . . . . . . 66
3.2.4 Peak Picking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2.6 First Comparison – NPP . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.2.8 Second Comparison – PNP . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.10 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3 A Realtime Onset Detector and the MIREX05 Evaluation . . . . . . . . . . . . . 74
3.3.1 Peak Picker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.2 Time Domain Correction . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.3.4 Evaluation Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.3.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.4 Using a Pitch Detector as an Onset Detector . . . . . . . . . . . . . . . . . . . . 78
3.4.1 Algorithm Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.4.2 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.5 Neural Net Based Onset Detection . . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.5.1 An Onset Detector for Baroque Recorder . . . . . . . . . . . . . . . . . . 89
3.5.2 Predominant Event Analysis in Polyphonic Audio . . . . . . . . . . . . . 91
3.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4 Realtime Beat Tracking Algorithms 94
4.1 Evaluation of Beat Tracking Algorithms . . . . . . . . . . . . . . . . . . . . . . . 95
4.1.1 Evaluation Metrics in the Beat Tracking Literature . . . . . . . . . . . . . 96
4.1.2 Evaluation Metrics in this Thesis . . . . . . . . . . . . . . . . . . . . . . . 99
4.2 Earlier Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.3 DrumTrack: Combining the Laroche and Goto Beat Trackers . . . . . . . . . . . 101
4.3.1 Cross Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.3.2 Detecting Drum Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.3.3 Low Frequency Evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3.4 Dynamic Programming Step . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.3.5 Consistency Check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.3.6 Implementation as a SuperCollider UGen . . . . . . . . . . . . . . . . . . 105
4.3.7 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
3
4.4 AutoTrack: A Realtime Adaptation of the Davies Beat Tracker . . . . . . . . . . 107
4.5 A Comparison of Humans and Computational Models on Transitions . . . . . . . 110
4.5.1 Improving Computational Algorithms on Transitions . . . . . . . . . . . . 111
4.6 Beat Tracking of a Harpsichord and Recorder Duo . . . . . . . . . . . . . . . . . 115
4.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
5 Automated Event Analysis 118
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
5.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.2.1 Event Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.2.2 Event Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
5.2.3 Analysing Event-wise Features . . . . . . . . . . . . . . . . . . . . . . . . 122
5.3 Perceptual Attack Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.3.1 Experiments on Perceptual Attack Time . . . . . . . . . . . . . . . . . . . 127
5.3.2 Modelling Ground Truth Data . . . . . . . . . . . . . . . . . . . . . . . . 131
5.4 Timbre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5.4.1 A Prototype for Categorisation On-the-fly . . . . . . . . . . . . . . . . . . 135
5.4.2 A Percussion Sound Classifier for Event Analysis . . . . . . . . . . . . . . 136
5.5 Heuristics for Event Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
5.6 Implementation and Compositional Applications . . . . . . . . . . . . . . . . . . 139
5.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
6 BBCut 2 141
6.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
6.1.1 Academic Engagement with Electronica . . . . . . . . . . . . . . . . . . . 142
6.1.2 Automated DJing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
6.1.3 Algorithmic Composition and Electronica . . . . . . . . . . . . . . . . . . 144
6.2 Algorithmic Splicing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
6.2.1 An Introduction to Breakbeat Cutting . . . . . . . . . . . . . . . . . . . . 146
6.2.2 The Benefits of Automation . . . . . . . . . . . . . . . . . . . . . . . . . . 147
6.2.3 Breakbeat Cutting Procedures . . . . . . . . . . . . . . . . . . . . . . . . 148
6.2.4 A Case Study – A Squarepusher Inspired Cut Procedure . . . . . . . . . . 151
6.2.5 Rendering Strategies for Realtime Audio Splicing . . . . . . . . . . . . . . 154
6.3 BBCut1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
6.4 BBCut2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
6.4.1 Scheduling Synchronised to an External Clock . . . . . . . . . . . . . . . 157
6.4.2 Time Representations in BBCut2 . . . . . . . . . . . . . . . . . . . . . . . 160
6.4.3 BBCut2 Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
6.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
7 Interactive Music Systems 166
7.1 Precursors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
7.2 Machine Enhanced Improvisation . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
7.2.1 Sat at Sitar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
7.2.2 Free Improvisation Simulation . . . . . . . . . . . . . . . . . . . . . . . . 178
7.2.3 DrumTrack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
4
7.3 Baroqtronica: The Art of Machine Listening . . . . . . . . . . . . . . . . . . . . . 191
7.3.1 Substituet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
7.3.2 Ornamaton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
7.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
8 Conclusions 206
8.1 Intelligent Agents? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
8.1.1 Autonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
8.1.2 The Shape of Musical Actions . . . . . . . . . . . . . . . . . . . . . . . . . 209
8.1.3 Interactive Music Systems as Agents . . . . . . . . . . . . . . . . . . . . . 211
8.2 Machine Listening Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
8.2.1 Event Detection and Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 217
8.2.2 Beat Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
8.2.3 BBCut3? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
8.3 Research Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
8.4 Compositional Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
5
