A Visual Programming Language for Live Video Sonification

152 Pages·2008·6.28 MB·English

UNIVERSITY OF CALGARY

A Visual Programming Language for Live Video Sonification

by

Roberto Arturo Diaz-Marino

A THESIS SUBMITTED TO THE FACULTY OF GRADUATE STUDIES
IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE
DEGREE OF MASTER OF SCIENCE

DEPARTMENT OF COMPUTER SCIENCE

CALGARY, ALBERTA
MARCH, 2008

© Roberto Arturo Diaz-Marino 2008
THE UNIVERSITY OF CALGARY
FACULTY OF GRADUATE STUDIES

The undersigned certify that they have read, and recommend to the Faculty of Graduate
Studies for acceptance, a thesis entitled “A Visual Programming Language for Live
Video Sonification” submitted by Roberto Arturo Diaz-Marino in partial fulfillment of
the requirements for the degree Master of Science.

Supervisor, Saul Greenberg
Department of Computer Science

Rob Kremer
Department of Computer Science

Ehud Sharlin
Department of Computer Science

External Examiner, David Eagle
Faculty of Music

Date

- i -
Abstract
The problem explored in this thesis is how to allow a person to map features from
multiple video streams, or parts of those video streams, into an audio space, and how they
can then monitor the resulting soundscape. I consider the requirements, user interface
and architecture of a system that lets an end-user configure a video-only stream so that it
generates an audio soundscape. In particular, I present a programming by demonstration
environment called Cambience. Using this system, end users can map visual
characteristics collected from video regions onto audio samples, and assemble the audio
stream in a way that creates a meaningful soundscape. Cambience is used to mitigate a
simple scenario between two distance separated collaborators and a shared resource. The
wide range of visual programming elements allow for complex sonification behavior.
Informal evaluations of the system show that Cambience has real usefulness with mostly
minor considerations.
- iii -
Publications
Diaz-Marino, R. and Greenberg, S. (2006) Cambience: A Video-Driven Sonic Ecology
for Media Spaces. Video Proceedings of ACM CSCW'06 Conference on Computer
Supported Cooperative Work, November, ACM Press. Video and two-page
summary. Duration 3:52
Diaz-Marino, R. and Greenberg, S. (2006) Demonstrating How to Construct a Sonic
Ecology for Media Spaces through Cambience. Demo, Adjunct Proc ACM CSCW
2006.

- iv -
Acknowledgements
First, I would like to give a huge thank-you to my supervisor Saul Greenberg who noticed
my talents and encouraged me to pursue this Master’s degree. Without his continued
encouragement I might not have finished it with all the influences in my life that were
pulling me away. Even though our areas of interest were very different and at first we
had a tough time finding something that satisfied us both, I am proud of the very unique
project we went ahead with. Furthermore, I am proud to have seen this project through
until the end, as I am someone who typically has too many ideas to stick with one thing
for very long.
I would also like to thank my fellow ILab members for their help and interest in this
project. My regret is that I couldn’t get more involved with group activities and get to
know everyone even better. Nevertheless, it didn’t stop anyone from welcoming me in,
on the rare occasion I was able to pull myself away from my other responsibilities.
I would also like to thank:
• Leila Sujir for facilitating my session at the Banff centre for my dance case study.
• Jim Jewitt of the former QUAB Gallery for his patience, cooperation, and helpful
advice in the art gallery case study.
• Alan Dunning for giving me a crash course in Isadora and Eyesweb.
- v -
Dedication
I dedicate this thesis to my dad, who probably had no idea what he was starting when he
gave me my first BASIC programming book. When I got the wild idea to create my own
video game, he taught me from the engineering calculation programs he created on our
Tandy 1000, and within a month he couldn’t help me anymore because I had gone
beyond what he knew. Still he found ways to nab programming reference manuals from
work to keep me going. Until I got my own computer we would often fight for time on
his. When I had my own computer, he would fight for my time off it! Nevertheless, he
put me on my direction in life and I felt very fortunate to have that kind of certainty as I
went on to University.
- vi -
Table of Contents
Approval Page ..................................................................................................................... ii
Abstract .............................................................................................................................. iii
Publications ........................................................................................................................ iv
Acknowledgements ..............................................................................................................v
Dedication .......................................................................................................................... vi
Table of Contents .............................................................................................................. vii
List of Tables ..................................................................................................................... xi
List of Figures .................................................................................................................. xiii
Chapter 1. Introduction ....................................................................................................... 1 
1.1  Motivations........................................................................................................... 2 
1.1.1  Video and Privacy ......................................................................................... 2 
1.1.2  Video as a Foreground Display ..................................................................... 3 
1.1.3  Video as an Interactive Installation ............................................................... 5 
1.2  Goals..................................................................................................................... 6 
1.3  Organizational Outline ......................................................................................... 6 
Chapter 2. Background and Related Work ......................................................................... 8 
2.1  Media Spaces........................................................................................................ 8 
2.1.1  Casual Interaction and Informal Awareness ................................................. 9 
2.1.2  What is a Media Space? .............................................................................. 10 
2.1.3  Sound in Media Spaces ............................................................................... 12 
2.1.4  Abstract Media Spaces ................................................................................ 14 
2.1.5  Abstract Mappings ...................................................................................... 16 
2.2  Sound .................................................................................................................. 17 
2.2.1  Acoustics ..................................................................................................... 17 
2.2.2  Psychoacoustics .......................................................................................... 21 
- vi -
2.2.3  Changing Sound .......................................................................................... 22 
2.2.4  Sound as an interface .................................................................................. 23 
2.3  Visual Programming .......................................................................................... 24 
2.3.1  VPLs for Mapping Video to Audio ............................................................ 26 
2.4  Summary ............................................................................................................ 28 
Chapter 3. Introducing Cambience through Scenarios of Use .......................................... 29 
3.1  Scenario 1 ........................................................................................................... 30 
3.1.1  Step 1 – Physical Setup ............................................................................... 30 
3.1.2  Step 2 – Configuring Regions of Interest ................................................... 31 
3.1.3  Step 3 – Monitoring Activities in Regions ................................................. 34 
3.1.4  Step 4 – Mapping Activities to Sounds ....................................................... 36 
3.1.5  Step 5 – Sharing Mappings with Others ..................................................... 41 
3.2  Scenario 2 ........................................................................................................... 41 
3.2.1  Step 1 – Physical Setup ............................................................................... 42 
3.2.2  Step 2 – Connecting to the Server............................................................... 43 
3.2.3  Step 3 – Loading an Ecology Share ............................................................ 43 
3.3  Scenario 3 ........................................................................................................... 44 
3.3.1  Step 1 – Configuring Regions in Private Space .......................................... 46 
3.3.2  Step 2 – Monitoring and Mapping Private Activities ................................. 47 
3.4  Scenario 4 ........................................................................................................... 50 
3.4.1  Configuring Regions in Public Space ......................................................... 50 
3.4.2  Step 2 – Monitoring and Mapping Public Activities .................................. 50 
3.5  Scenario Summary ............................................................................................. 51 
3.6  Conclusion .......................................................................................................... 52 
Chapter 4. Shared Media and Live Visual Programming ................................................. 54 
4.1  Sharing Media .................................................................................................... 54 
4.1.1  Shared Data ................................................................................................. 55 
4.1.2  Cambience as a Distributed System ............................................................ 56 
4.2  Visual Programming .......................................................................................... 59 
4.2.1  Items, Attributes, and Functions ................................................................. 62 
- viii -
4.2.2  Sources, Sinks, and Patches ........................................................................ 63 
4.2.3  Signals ......................................................................................................... 64 
4.2.4  Live Programming ...................................................................................... 67 
4.3  Summary ............................................................................................................ 69 
Chapter 5. Mapping Video to Audio ................................................................................. 70 
5.1  Visual Items ........................................................................................................ 70 
5.1.1  Analyzing Regions ...................................................................................... 70 
5.1.2  Change Detection through Frame Differencing .......................................... 71 
5.1.3  Other Change Metrics ................................................................................. 74 
5.1.4  Ambiguity ................................................................................................... 75 
5.1.5  Signaling Change with Region Items ......................................................... 76 
5.1.6  Camera Items .............................................................................................. 77 
5.2  Sound Items ........................................................................................................ 77 
5.2.1  Sonic Ecologies ........................................................................................... 78 
5.2.2  Selecting Sounds ......................................................................................... 80 
5.2.3  The Wave3D Item ....................................................................................... 82 
5.2.4  The Wave Item ............................................................................................ 83 
5.2.5  The Audio Item ........................................................................................... 84 
5.2.6  The Playlist Item ......................................................................................... 84 
5.2.7  MIDI Instrument and MIDI Track Items .................................................... 85 
5.2.8  The Cambience SFX Library ...................................................................... 86 
5.2.9  Audio Configuration ................................................................................... 87 
5.3  Signal Items ........................................................................................................ 87 
5.3.1  Monitoring and Testing............................................................................... 89 
5.3.2  Signal Transformation ................................................................................ 90 
5.3.3  Logical Operations ...................................................................................... 95 
5.3.4  Mathematical Operations ............................................................................ 97 
5.3.5  History and Memory ................................................................................... 99 
5.3.6  Time and Date ........................................................................................... 102 
5.3.7  Composite Items ....................................................................................... 102 
- ix -
5.4  Extensibility ..................................................................................................... 104 
5.5  Summary .......................................................................................................... 107 
Chapter 6. Applications and Case Studies ...................................................................... 109 
6.1  Case Studies ..................................................................................................... 109 
6.1.1  Workgroup Space...................................................................................... 110 
6.1.2  Dance Performance ................................................................................... 113 
6.1.3  Art Gallery ................................................................................................ 117 
6.2  Applications to Prior Research ......................................................................... 120 
6.2.1  Very Nervous System ............................................................................... 120 
6.2.2  Iamascope ................................................................................................. 121 
6.3  Summary .......................................................................................................... 122 
Chapter 7. Conclusion and Future Work ........................................................................ 123 
7.1  Contributions .................................................................................................... 123 
7.2  Future Work ..................................................................................................... 124 
7.2.1  Revision and Evaluation of Cambience .................................................... 124 
7.2.2  Sound as an Interface ................................................................................ 125 
7.2.3  Social Protocols of Public Audio .............................................................. 126 
7.3  Possible Future Applications ............................................................................ 126 
7.3.1  Store Security ............................................................................................ 126 
7.3.2  Interactive Music in Malls ........................................................................ 127 
7.3.3  Interactive Museum Exhibits .................................................................... 127 
7.3.4  Aging in Place ........................................................................................... 128 
7.3.5  Home Security and Notification ............................................................... 128 
7.3.6  Abstract Art Piece ..................................................................................... 129 
7.4  Conclusion ........................................................................................................ 129 
- x -