Data Sonification Artworks

101 Pages·2014·9.85 MB·English

by  Constantina Caldis

Title

Data Sonification Artworks:
A Music and Design Investigation of Multi-modal
Interactive Installations

A Research Report submitted to the University of the Witwatersrand, Wits School of Arts.
In partial fulfilment of the requirements for the Degree of Masters of Digital Arts

Constantina Caldis
February 2014

Supervised by
Tegan Bristow
Declaration

I declare that this is my own unaided work. It is submitted for the Degree of Masters of
Digital Arts (Interactive Media) at the University of the Witwatersrand, Johannesburg. It has
not been submitted before for any degree of examination in any other university.

______________________________________
Constantina Caldis
February 2014

  2
Acknowledgements

I would like to especially thank my supervisor, Tegan Bristow for all her support, patience
and guidance. I thank Christo Doherty and Cameron Harris for always making themselves
available and providing foundational advice towards this research report. Thank you to my
husband, Aidan Roberts for providing constant motivation and love. My magical parents,
Stathy and Maria Caldis for their constant support in every aspect of my life and last but not
least, my good friends Donato Somma and Nicola Cloete for their solid academic advice and
encouragement.

  3
Abstract

Through three case studies this research report will explore the design and multi-
modal attributes of interactive installations that feature characteristics comparable to those of
musical instruments. It will briefly outline and define data sonification and the five
sonification techniques: Audification, Auditory Icons, Earcons, Parameter-mapping
sonification and Model-based sonification. Model-based data sonification will be more
closely explored, as interactivity and the incorporation of multiple modes are key elements
within these types of sonifications. Pre-existing knowledge of musical instruments, their
multi-modal attributes and functionality will be analyzed in relation to the case studies. The
design analysis will focus on the interaction and interface design while incorporating the
three modes, namely: visual guidance, real-time physical/gestural interaction, and
instantaneous acoustic feedback. To conclude, this research report will demonstrate, through
the case studies, that the function and design of musical instruments can be suggestive in
realizing a potential way forward for digital artists and their multi-modal interactive
installations.

  4
Table of Contents
TITLE
  1

DECLARATION
  2

ACKNOWLEDGEMENTS
  3

ABSTRACT
  4

LIST
 OF
 ILLUSTRATIONS
  6

INTRODUCTION
  10

CHAPTER ONE: DATA SONIFICATION
  14

DATA SONIFICATION
  14

SONIFICATION TECHNIQUES
  17

MODEL-BASED SONIFICATION
  23

CONCLUSION
  25

CHAPTER TWO: INTERACTION AND MUSICAL INSTRUMENTS
  26

ORGANOLOGY
  27

ELECTROPHONES
  32

HUMAN-OBJECT INTERACTION
  35

CONCLUSION
  37

CHAPTER THREE: INTERACTIVITY
  39

HUMAN COMPUTER INTERACTION: PHYSICALITY
  39

INTERACTIVITY AND ARTISTIC OUTCOMES
  41

INTERACTIVE SONIFICATION
  47

MULTI-MODAL INTERACTION
  50

CONCLUSION
  52

CHAPTER FOUR: DESIGN FOR DIGITAL INTERACTIVE WORKS
  53

INTERACTION DESIGN
  53

THE VIOLIN AND DESIGN
  61

INTERFACE DESIGN
  64

CONCLUSION
  67

CHAPTER FIVE: CASE STUDIES
  68

FIREWALL
  69

THE MANUAL INPUT WORKSTATION
  77

MODEL HAT DIGIPHONY
  83

CONCLUSION
  90

WORKS
 CITED
  95

  5
List of Illustrations

Figure 1: A Sine Wave. Source: “Audio Test Tone” Wavelength Media, 2012.
http://www.mediacollege.com/audio/tone/. Web. 10 August 2013.

Figure 2: Dynamic Model for Model-based Sonifications. Source: Hermann, T., Hunt, A. &
Neuhoff, G. eds. The Sonification Handbook. Berlin: Logos Publishing House. 2011.
Print.

Figure 3: Scope of the Study of Instruments as Objects and as Aspects of Biological and
Social Musical Activity. Source: Wachsmann, K. et al. Oxford Music Online. Grove
Music Online. Oxford University Press. Web. 2 October 2013.

Figure 4: Acoustic Pressure Waveforms at 440Hz of a Violin, Trumpet, Flute and Oboe.
Source: Feilding, C. Lecture 009 Hearing IV. College of Santa Fe Auditory Theory.
2013. http://www.feilding.net/sfuad/musi301201/html/lectures/009_hearing_IV.htm.
Web. 27 October 2013.

Figure 5: Topic Web of Interactive Sonification. Source: Hermann, T., Hunt, A. & Neuhoff,
G. eds. The Sonification Handbook. Berlin: Logos Publishing House. 2011. Print.

Figure 6: The Overlapping Disciplines of Interactive Design. Source: Saffer, D. Designing
for Interaction. Berkeley: AIGA Design Press. 2007. Print.

  6
Figure 7: Very Nervous System. Source: Rokeby, D. Works. 2013.
http://www.davidrokeby.com/installations.html. Web. 18 May 2013.

Figure 8.1: Firewall Soft Curves. Source: Sherwood, A. Firewall. Works. 2013.
http://aaron-sherwood.com/works/firewall/. Web. 28 August 2013.

Figure 8.2: Firewall Sharp Angular. Source: Sherwood, A. Firewall. Works. 2013.
http://aaron-sherwood.com/works/firewall/. Web. 28 August 2013.

Figure 8.3: Firewall Depth. Source: Sherwood, A. Firewall. Firewall. 2012.
http://www.flickr.com/photos/asherwood/8292630863/in/set-72157632301614586
Web. 20 January 2014.

Figure 8.4: Mizaru Head Interaction. Source: Sherwood, A. Mizaru performance. 2013.
http://aaron-sherwood.com/works/MIZARUperformance/. Web. 28 August 2013.

Figure 8.5: Mizaru Hand-to-Shoulder Interaction. Source: Sherwood, A. Mizaru
performance. Works. 2013.
http://aaron-sherwood.com/works/MIZARUperformance/. Web. 28 August 2013.

Figure 8.6: Mizaru Visual Mapping. Source: Sherwood, A. Mizaru performance. 2013.
http://aaron-sherwood.com/works/MIZARUperformance/. Web. 28 August 2013.

  7
Figure 9.1: Negdrop High frequencies. Source: Levin, G. et al. “The Manual Input
Workstation - Interactive Art by Golan Levin and Collaborators.” Flong. Web. 26
May 2013.

Figure 9.2: Negdrop Low frequency. Source: Levin, G. et al. “The Manual Input
Workstation - Interactive Art by Golan Levin and Collaborators.” Flong. Web. 26
May 2013.

Figure 9.3: Innerstamp Creation. Source: Levin, G. et al. “The Manual Input Workstation -
Interactive Art by Golan Levin and Collaborators.” Flong. Web. 26 May 2013.

Figure 9.4: Innerstamp Continuation. Source: Levin, G. et al. “The Manual Input Workstation
- Interactive Art by Golan Levin and Collaborators.” Flong. Web. 26 May 2013.

Figure 9.5: Rotuni Many objects. Source: Levin, G. et al. “The Manual Input Workstation -
Interactive Art by Golan Levin and Collaborators.” Flong. Web. 26 May 2013.

Figure 9.6: Rotuni Hand with spike. Source: Levin, G. et al. “The Manual Input Workstation
- Interactive Art by Golan Levin and Collaborators.” Flong. Web. 26 May 2013.

Figure 10.1: Layout of Model Hat Digiphony. Source: Caldis, M. Layout of Model Hat
Digiphony. 2012. JPEG.

Figure 10.2: Multiple Users Wearing Hats in the Interactive Space. Source: Caldis, M.
Multiple users wearing hats in the interactive space. 2012. JPEG.

  8
Figure 10.3: Visual Projection. Source: Caldis, M. Visual Projection. 2012. JPEG.

Figure 10.4: Interaction, Projection and Users. Source: Caldis, C. Interaction, Projection and
Users. 2012. JPEG.

  9
Introduction

In a paper entitled The Discipline of Interactive Sonification, Hermann and Hunt
(2004) express ideas and trends in the field of data sonification for future developments that
explore real-time, multi-modal interactive systems (3). One such idea evaluates the design
and central workings of traditional musical instruments that exemplify multiple modes of
interactivity. This research report will aim to investigate the characteristics of interaction and
interface design aspects observed in traditional musical instruments and compare them to
those observed in multi-modal interactive installations.
Interactive installations form part of the genre of interactive art. Interactive art is a
broad field, but this research report will focus on interactive installations that are sound
reactive in digitally responsive environments. The designs of multi-modal interactive
installations provide an interactive space for user involvement. The term multi-modal refers
to the interaction possibilities within this space as including real-time tactile or gestural
action, visual guidance and more importantly instantaneous acoustic feedback. The physical,
visual and auditory modes are pertinent to this study as they will be observed and compared
alongside musical instruments, which have been utilizing these modes for centuries.
When discussing musical instruments, I am primarily referring to acoustic instruments
and how the physical object i.e. the instrument (e.g. flute, violin, piano) functions as a means
for the player to interact in a tangible and visual manner that bestows upon the player an
instantaneous acoustic result. The sounds emitted from these instruments are purely acoustic
i.e. without electrical amplification. When dealing with sound emitted from a multi-modal

  10