Table Of Content

Human-Computer Interaction Series Michele Geronazzo Stefania Serafin Editors Sonic Interactions in Virtual Environments Human–Computer Interaction Series Editors-in-Chief JeanVanderdonckt,LouvainSchoolofManagement,Universitécatholiquede Louvain,Louvain-La-Neuve,Belgium Q.VeraLiao,MicrosoftResearchCanada,Montréal,Canada TheHuman–ComputerInteractionSeries,launchedin2004,publishesbooksthat advancethescienceandtechnologyofdevelopingsystemswhichareeffectiveand satisfying for people in a wide variety of contexts. Titles focus on theoretical perspectives (such as formal approaches drawn from a variety of behavioural sciences), practical approaches (such as techniques for effectively integrating user needsinsystemdevelopment),andsocialissues(suchasthedeterminantsofutility, usabilityandacceptability). HCI is a multidisciplinary field and focuses on the human aspects in the development of computer technology. As technology becomes increasingly more pervasive the need to take a human-centred approach in the design and developmentofcomputer-basedsystemsbecomesevermoreimportant. TitlespublishedwithintheHuman–ComputerInteractionSeriesareincludedin Thomson Reuters’ Book Citation Index, The DBLP Computer Science BibliographyandTheHCIBibliography. · Michele Geronazzo Stefania Serafin Editors Sonic Interactions in Virtual Environments Editors MicheleGeronazzo StefaniaSerafin DepartmentofEngineering AalborgUniversity andManagement KøbenhavnSV,Denmark UniversityofPadova Padova,Italy DysonSchoolofDesignEngineering ImperialCollegeLondon London,UK DepartmentofHumanitiesandCultural Heritage UniversityofUdine Udine,Italy NordicSMCUniversityHub,NordForsk,Stensberggata27,0170Oslo,Norwayrepresented by:HansJørgenAndersen,Professor UniversitàdegliStudidiUdine,DipartimentodiStudiUmanisticiedelPatrimonioCulturale, VicoloFlorion.2/B,33100Udine,Italyrepresentedby:Prof.AndreaZannini,Department Director EU project SONICOM, funded by the Horizon 2020 research and innovation programme (grantagreementNo101017743),representedby:Dr.LorenzoPicinali,Reader ISSN 1571-5035 ISSN 2524-4477 (electronic) Human–ComputerInteractionSeries ISBN 978-3-031-04020-7 ISBN 978-3-031-04021-4 (eBook) https://doi.org/10.1007/978-3-031-04021-4 ©TheEditor(s)(ifapplicable)andTheAuthor(s)2023.Thebookisanopenaccesspublication. OpenAccessThisbookislicensedunderthetermsoftheCreativeCommonsAttribution4.0International License(http://creativecommons.org/licenses/by/4.0/),whichpermitsuse,sharing,adaptation,distribu- tionandreproductioninanymediumorformat,aslongasyougiveappropriatecredittotheoriginal author(s)andthesource,providealinktotheCreativeCommonslicenseandindicateifchangeswere made. Theimagesorotherthirdpartymaterialinthisbookareincludedinthebook’sCreativeCommonslicense, unlessindicatedotherwiseinacreditlinetothematerial.Ifmaterialisnotincludedinthebook’sCreative Commonslicenseandyourintendeduseisnotpermittedbystatutoryregulationorexceedsthepermitted use,youwillneedtoobtainpermissiondirectlyfromthecopyrightholder. Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant protectivelawsandregulationsandthereforefreeforgeneraluse. Thepublisher,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthisbook arebelievedtobetrueandaccurateatthedateofpublication.Neitherthepublishernortheauthorsor theeditorsgiveawarranty,expressedorimplied,withrespecttothematerialcontainedhereinorforany errorsoromissionsthatmayhavebeenmade.Thepublisherremainsneutralwithregardtojurisdictional claimsinpublishedmapsandinstitutionalaffiliations. ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSwitzerlandAG Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland Betweenstimulusandresponsethereisa space.Inthatspaceisourpowertochoose ourresponse.Inourresponseliesourgrowth andourfreedom. —ViktorE.Frankl Preface SonicInteractionDesign(SID)isthestudyandexploitationofsoundbeingoneofthe principalchannels conveying information,meaning,esthetic,andemotionalquali- tiesininteractivecontexts.ThefieldofSonicInteractionsinVirtualEnvironments (SIVE)extendsSIDtoimmersivemedia,i.e.,virtual/augmented/mixedreality(XR). Consideringavirtualitycontinuum,thisbookmainlyfocusedonvirtualreality(VR) alsofacingoccasionallymixedandhybridrealitysettings. Thebasicandmostobviousassumptionthatmotivatesthisvolumeis:itishard toliveinaworldwithoutsoundanditishardinvirtualenvironments(VE)too.VR without plausible and convincing sounds feels unnatural to users. Auditory information is a powerful omnidirectional source of learning for our interaction in real and virtual environments. The good news brought by this book is that VR finally soundsplausible.Advancesinseveralfieldsarenowabletoprovideanimmersive listeningexperiencethatisperceptuallyindistinguishablefromrealitywhichmeans thatimmersivesoundscouldmakeinteractionintrinsicallynatural.Auralizationand spatialaudiotechnologiesplayafundamentalroleinprovidingimmersionandpres- enceinVRapplicationsatanunprecedentedlevel.Thecombinationofrecentdevel- opmentsinVRheadsetsandearablesfurtherstrengthenstheperceptualvalidityof multimodalvirtualenvironmentsandexperiences. We can therefore promote a true audio-centered and audio-first design for VR withlevelsofrealismandimmersivenessthatcanevensurpassthevisualcounterpart. Visuals,althoughrightlyemphasizedbymanystudiesandproducts,areoftennotvery effectivelyenhancedandstrengthenedbysound.Thefinalresultisaweakeningof multisensoryintegrationandthecorrespondingVRpotentialsthatstronglydetermine thequalityanddurabilityoftheexperience. Theeditorswouldliketoidentifytwostartingpointsinthepast10yearsthathave given rise and awareness to the SIVE research area and studies. The first episode is symbolic: we would like to anecdotally bring back from our memories the first meetingbetweenus,thetwoeditorsofthebook.Theyearwas2011,exactly10years ago. Michele had recently started his Ph.D. at the Sound and Music Computing Group of the Department of Information Engineering at the University of Padua, underthesupervisionofDr.Avanzini.TheItalianAssociationofMusicalInformatics vii viii Preface (AIMI)organizedtheworkshop“SoundandMusicComputingforHuman-Computer Interaction” at the ninth edition of the Biannual Conference of the Italian ACM SIGCHIChapter(CHItaly)atthebeautifulAlgheroinSardiniainearlySeptember. Agreatperiodfortheseaside. Michelewasaskedtowritehisfirstconferencepapertobepresentedatthework- shopentitled“Customized3DSoundforInnovativeInteractionDesign,”Anarticle withahigh-soundingtitlethatpromisesalotbutprovideslittle:inshort,anarticleof whichnottobeproud.Ontheotherhand,thereweresomevaluablereferencestothe egocentricaudioperspectivethatwillbeformalizedintheintroductorychapterof thisbook.However,thereasonwhywetellthisanecdoteisthatathisfirstpresentation atascientificconferenceforthePh.D.studentMicheleGeronazzo,amongthevery smallaudience,therewasDr.StefaniaSerafin.Tenyearsago,webegantodiscuss issuesthatconnectedsonicinteractiondesignwithimmersive3DaudioinVR.The AIMI president of that time failed to get the workshop’s contributions included in the official ACM CHItaly proceedings despite a regular peer-review process. The poorPh.D.studentMichelefoundhimselfwithoutanofficialpublication,athisfirst conference,inanunknownscientificcommunity.Weliketothinkthatatthatevent andwiththatmeetingstartedsomethingmuchmorerelevantandimpactful:SIVE. Weareheretogiveitashapeinthisbookeditedandstructuredtogether. Anothertemporalcoincidencebringsustoconnectthisstorywiththesecondand officialstartingpointofthisadventure.Michele’sunpublishedconferencepaperwas finallypublishedwithinhisdoctoralthesis,defendedin2014,theyearinwhichthe IEEE Virtual Reality workshop series “Sonic Interactions in Virtual Environments (SIVE)”started(https://sive.create.aau.dk/).ThemissionofIEEEVRSIVEwasto increaseamongthevirtualrealitycommunityandjuniorresearcherstheawareness oftheimportanceofsonicelementswhendesigningimmersiveXRenvironments. However, we can also identify a certain degree of reciprocity while considering thefragmentednatureandspecificityofthosestudiesaimatdevelopingimmersive XR environments for sound and music. First, we, therefore, refer to our beloved Sound and Music Computing (SMC) network, and then we consider the Interna- tional Community for Auditory Display (ICAD), the Audio Engineering Society (AES), and the communities linked to the International Conference on New Inter- faces for Musical Expression (NIME), the Digital Audio Effects (DAFX), and the SonicInteractionDesignCOSTAction(COST-SIDIC601,endedin2012).Allthese communitiesaddressaspectsoftheSIVEtopicsaccordingtotheirspecificities.No institutionalnorcontextualreferencesthatcollecttechnologicaldevelopments,best practices,andcreativeeffortsrelatedtothepeculiaritiesofimmersiveVEsexisted before the SIVE workshop. The book follows a similar philosophy trying to give anexhaustiveviewofthosemultidisciplinarytopicsalreadymentionedinourtwo recentreviews.1 Itfeaturesstate-of-the-artresearchonreal-timeauralization,sonic 1S.Serafin,M.Geronazzo,N.C.Nilsson,C.Erkut,andR.Nordahl,“Sonicinteractionsinvirtual reality:stateoftheart,currentchallengesandfuturedirections,”IEEEComputerGraphicsand Applications,vol.38,no.2,pp.31–43,2018. S. Serafin et al., “Reflections from five years of Sonic Interactions in Virtual Environments workshops,”JournalofNewMusicResearch,vol.49,no.1,pp.24–34,Jan.2020. Preface ix interactiondesigninVR,qualityoftheexperienceinmultimodalenvironments,and applications. We aim to provide an organized starting point on which to develop a new generation of immersive experiences and applications. Since the editors are awareoftheveryfastsocialtransformationbytheaccelerationinthedevelopment ofdigitaltechnologies,allchaptersshouldbereadasentrypoints.Futurescenarios andsolutionswillnecessarilyevolvebycombiningemergingresearchareassuchas artificial intelligence, ubiquitous and pervasive computing, quantum technologies, as well as continuous discoveries in the neuroscientific field and anthropological reflectionsontheauthenticityoftheexperienceinVR. Forthisreason,contributingauthorsandeditorsincludeinterdisciplinaryexperts from the fields of computer science, engineering, acoustics, psychology, design, humanities,andbeyond.Sothatwecangivetothereaderabroadviewandaclear introduction to the state-of-the-art technologies and design principles, and to the challengesthatmightbeawaitingusinthefuture. Throughanoverviewofemergingtopics,theories,methods,tools,andpractices insonicinteractionsinvirtualenvironmentsresearch,thebookaimstoestablishthe basisforfurtherdevelopmentofthisnewresearcharea.Theauthorswereinvitedto contributetospecifictopicsaccordingtotheirwell-knownexpertise.Theyfollowed apredefinedstructureoutlinedbytheeditors. Thebookisdividedintofourparts: Part I, Introduction: this theoretical part frames the background and the key themesinSIVE.Theeditorsaddressseveralphenomenologicalfoundationalissues intendingtoshapeanewresearchfieldfromanarchipelagoofstudiesscatteredin differentresearchcommunities. Part II, Interactive and Immersive Audio: we cover the system requirement partwithfourchaptersintroducingandanalyzingaudio-relatedtechnologicalaspects andchallenges.Withsomeoverlapsandconnections,thefourchaptersdealwiththe plausibilityofanimmersiverenderingabletotacklethecomputationalburden.Todo so,wedealwithmethodsandalgorithmsforreal-timerenderingconsideringsound production, propagation, and spatialization, respectively. Finally, the reproduction andevaluationphaseallowsclosingthedevelopmentloopofnewaudiotechnologies. PartIII,SonicInteractions:asonicinteractiondesignpartdevotedtoempha- sizingthepeculiaraspectsofsoundinimmersivemedia.Inparticular,spatialinterac- tionsareimportantwherewewouldliketoproduceandtransformideasandactions to create meaning with VR, as well as the virtual auditory space is an information container that could be shaped by users. As the VR systems enter people’s lives,manufacturers,developers,andcreatorsshouldcarefullyconsideranembodied experiencereadytoshareacommonspacewithpeers,collaboratively. Part IV, Sonic Experiences: the last part focuses on multimodal integration for sonic experiences in VR with the help of several case studies. Starting from a literature review of multimodal experiments and experiences with sound, this last part offers some reflections on the concept of audio-visual immersion and audio- hapticintegrationabletoformourecologyofeverydayormusicalsounds.Finally, thepotentialsofVRtotransportartistsandspectatorsintoaworldofimaginationand x Preface unprecedentedexpressionistakenasanexemplarofwhatmultimodalandimmersive experiencescanelicitintermsofemotionalandrationalengagement. In the following, a summary for each chapter is provided to help the reader to followtheproposednarrativestructure. PartI Chapter1illustratestheeditors’visionoftheSIVEresearchfield.Themainconcept introduced here is the egocentric audio perspective in a technologically mediated environment.Thelistenersshouldbeentangledwiththeirauditorydigitaltwinsin aparticipatoryandenactedexplorationforsense-makingcharacterizedbyaperson- alized and multisensory first-person spatial reference frame. Intra-actions between humansandnon-humanagents/actorsdynamicallyandfluidlydetermineimmersion andcoherenceoftheexperience,participatively.SIDaimstofacilitatethediffraction ofknowledgeindifferenttasksandcontexts. PartII Chapter2addressesthefirstbuildingblockofSIVE,i.e.,themodelingandsynthesis ofsoundsources,focusingonproceduralapproaches.Specialemphasisisplacedon physics-basedsoundsynthesismethodsandtheirpotentialforimprovedinteractivity concerningthesenseofpresenceandembodimentofauserinavirtualenvironment. InChap.3,criticalchallengesinauralizationsystemsinvirtualrealityandgames areidentified,includingprogressingfrommodelingenclosurestocomplex,general scenessuchasacityblockwithbothindoorandoutdoorareas.Theauthorsprovide a general overview of real-time auralization systems, their historical design and motivations,andhownovelsystemshavebeendesignedtotacklethenewchallenges. Chapter4dealswiththeconceptsofadaptationinabinauralaudiocontext,consid- eringfirsttheadaptationoftherenderingsystemtotheacousticandperceptualprop- ertiesoftheuser,andsecondtheadaptationoftheusertotherenderingqualityofthe system. The authors introduce the topics of head-related transfer function (HRTF) selection (system-to-user adaptation) and HRTF accommodation (user-to-system adaptation). Finally, Chap. 5 concludes the second part of the book by introducing audio reproductiontechniquesforvirtualreality,theconceptsofaudioquality,andquality oftheexperienceinVR.