Eye-Tracking in Interaction: Studies on the Role of Eye Gaze in Dialogue PDF
Preview Eye-Tracking in Interaction: Studies on the Role of Eye Gaze in Dialogue
Eye-tracking A d v a n c in Interaction e s i n I n t e r a c ti edited by o n Geert Brône and Bert Oben S t u d i e s John Benjamins Publishing Company Eye-tracking in Interaction Advances in Interaction Studies (AIS) issn 1879-873X Advances in Interaction Studies (AIS) provides a forum for researchers to present excellent scholarly work in a variety of disciplines relevant to the advancement of knowledge in the field of interaction studies. The book series accompanies the journal Interaction Studies: Social Behaviour and Communication in Biological and Artificial Systems. The book series allows the presentation of research in the forms of monographs or edited collections of peer-reviewed material in English. For an overview of all books published in this series, please see http://benjamins.com/catalog/ais Editors Kerstin Dautenhahn Angelo Cangelosi University of Waterloo & The University of Manchester University of Hertfordshire Editorial Board Henrik Christensen Chrystopher L. Nehaniv Georgia Institute of Technology University of Hertfordshire Harold Gouzoules Stefano Nolfi Emory University CNR, Rome Takayuki Kanda Pierre-Yves Oudeyer ATR, Kyoto INRIA, Bordeaux Tetsuro Matsuzawa Irene M. Pepperberg Kyoto University Harvard University & Brandeis University Giorgio Metta Kerstin Severinson Eklundh IIT, Genoa KTH, Stockholm Adam Miklosi Stefan Wermter Eötvös Loránd University University of Hamburg Robert W. Mitchell Eastern Kentucky University Volume 10 Eye-tracking in Interaction. Studies on the role of eye gaze in dialogue Edited by Geert Brône and Bert Oben Eye-tracking in Interaction Studies on the role of eye gaze in dialogue Edited by Geert Brône Bert Oben University of Leuven John Benjamins Publishing Company Amsterdam / Philadelphia TM The paper used in this publication meets the minimum requirements of 8 the American National Standard for Information Sciences – Permanence of Paper for Printed Library Materials, ansi z39.48-1984. doi 10.1075/ais.10 Cataloging-in-Publication Data available from Library of Congress. isbn 978 90 272 0152 2 (Hb) isbn 978 90 272 6346 9 (e-book) © 2018 – John Benjamins B.V. No part of this book may be reproduced in any form, by print, photoprint, microfilm, or any other means, without written permission from the publisher. John Benjamins Publishing Company · https://benjamins.com Table of contents Chapter 1 Introduction: Gaze, interaction and eye-tracking: A multidisciplinary endeavor 1 Geert Brône and Bert Oben Part 1. Theoretical considerations Chapter 2 Eye gaze as a cue for recognizing intention and coordinating joint action 21 Franco Amati and Susan E. Brennan Chapter 3 Effects of a speaker’s gaze on language comprehension and acquisition 47 Pia Knoeferle, Helene Kreysa and Martin J. Pickering Chapter 4 Weaving oneself into others: Coordination in conversational systems 67 Rick Dale and Michael J. Spivey Chapter 5 On the role of gaze for successful and efficient communication 91 Maria Staudte and Matthew W. Crocker Part 2. Methodological considerations Chapter 6 Quantifying the interplay of gaze and gesture in deixis using an experimental-simulative approach 109 Thies Pfeiffer and Patrick Renner Chapter 7 Gaze and face-to-face interaction: From multimodal data to behavioral models 139 Gérard Bailly, Alaeddine Mihoub, Christian Wolf and Frédéric Elisei vi Eye-tracking in Interaction Chapter 8 Automatic analysis of in-the-wild mobile eye-tracking experiments using object, face and person detection 169 Stijn De Beugher, Geert Brône and Toon Goedemé Part 3. Case studies Chapter 9 Gaze, addressee selection and turn-taking in three-party interaction 197 Peter Auer Chapter 10 Gaze as a predictor for lexical and gestural alignment 233 Bert Oben Chapter 11 Mobile dual eye-tracking in face-to-face interaction: The case of deixis and joint attention 265 Anja Stukenbrock Chapter 12 Displaying recipiency in an interpreter-mediated dialogue: An eye-tracking study 301 Jelena Vranjes, Hanneke Bot, Kurt Feyaerts and Geert Brône Index 323 Chapter 1 Introduction Gaze, interaction and eye-tracking: A multidisciplinary endeavor Geert Brône and Bert Oben 1. Eye gaze in interaction Several subdisciplines and programs in linguistics and psychology have demon- strated a long-standing interest in the study of non-verbal communication in re- lation to speech. The most widely studied form of non-vocal communication is by far (co-speech) gesture, defined as the use of hands, arms and other body parts, typically in conjunction with speech. Within the broadly defined category of ges- ture, hand gestures have received most attention because of their broad semiotic potential (see Müller et al., 2013, 2014 for a state-of-the-art). This volume, however, zooms in on a different semiotic channel within the visual modality that has been shown to have multiple communicative functions as well, viz. eye gaze in face-to- face interaction. Ever since the publication of early pioneering work by Kendon (1967), Argyle & Cook (1976) and Goodwin (1980, 1981), different disciplines have shown an interest in the role of eye gaze in conversation, including conversation analysis, psycholinguistics and research on human-computer interaction. One of the main reasons for this multidisciplinary interest is the key role that eye gaze plays in establishing successful communication as an essentially joint action (Clark, 1996; Pickering & Garrod, 2004, 2006; Linell, 2009; Zima and Brône, 2015; Feyaerts, Brône and Oben, 2017). Communication partners coordinate their production and interpretation, that is, they need to be able to react or adjust to their partners as much as they need to be able to anticipate their partners’ actions. Establishing and managing this interaction typically requires information from different signal sys- tems simultaneously, and eye gaze seems to play a constitutive role in the different phases of this complex process, including signaling attention and interest, achieving joint attention, organizing the sequential structure of the interaction, reference identification and disambiguation, feedback and feedback elicitation, etc. Teasing apart these different functions, and designing methods to study or implement https://doi.org/10.1075/ais.10.01bro © 2018 John Benjamins Publishing Company 2 Geert Brône and Bert Oben them (e.g. in human-computer interaction) is a complex undertaking that – not surprisingly – requires a multidisciplinary approach. This volume aims to present a state-of-the-art collection of chapters on eye gaze in interaction, focusing on theo- retical and methodological issues, presenting critical overviews of key phenomena and introducing original empirical studies. What unites the different takes on the phenomenon of eye gaze in interaction in this volume, is a methodological focus on the potential of (mobile) eye-tracking as a tool for gaining high-quality information on the gaze behavior of participants engaged in human-human or human-computer interaction. It is an explicit aim of this volume to bring together different dis- ciplines, which to some extent have an altogether different approach to the use of eye-tracking in interaction. We hope that crossing the disciplinary boundaries may be fruitful and inspire the further development of this new technology-driven endeavor. Before we discuss the introduction of eye-tracking to interaction research in Section 2, the remainder of this section examines, in somewhat more detail, the body of literature that has dealt with the role of eye gaze as a communicative resource in interaction. The above-mentioned pioneering work is generally referred to as the first truly empirical research on eye gaze in interaction, based on the detailed analysis of video data (and its transcription), and focusing on both the speakers’ and hearers’ gaze direction while engaging in face-to-face dialogue. These studies provided compel- ling evidence for the multifunctionality of eye gaze as a communicative instrument and have, in fact, paved the way for more recent work. This work can, broadly speaking, be subdivided into two clusters, relating to the research fields involved. One research line focuses on gaze behavior as an interactional resource in conver- sation (functioning as a display of (dis)engagement, participation, affiliation, turn allocation, etc.) and is mainly inspired by conversation analysis and interactional linguistics. Another line zooms in on the cognitive import of speakers’ and listeners’ gaze on language processing and production, spatial mapping (including deixis) and the coordination of joint tasks. Needless to say, there is no strict divide between the two lines, but for organizational reasons we discuss them separately here. In the more strongly interactional line of research on eye gaze, two key topics can be identified (drawing on Rossano’s (2012a) overview), viz. (i) the relation between eye gaze and participation roles in the interaction, and (ii) the regulatory function of eye gaze. The first refers to the relationship among the participants engaged in an interaction, including the distribution and negotiation of speaker and hearer roles (referred to as the participation framework by Goffman, 1981). The early empirical observations on gaze behavior in interaction, in fact, related to this participation framework, as they showed that gaze behavior while speaking differs substantially from gaze while listening. Kendon (1967) and Argyle & Cook (1976) showed that hearers typically displayed longer sequences of uninterrupted Chapter 1. Introduction 3 gaze towards the speakers, while speakers tended to shift their gaze towards and away from the hearer more frequently. This basic observation has been confirmed in several later studies (Goodwin, 1981, Vertegaal, 1999, Hirvenkari et al., 2013, Brône et al., 2017). Rossano (2012b), however, adds that this distributional pattern is partly dependent on the specific interactional activities of the participants. For instance, during the activities of tellings and questions, a more sustained gaze by the recipient is required, as it is viewed as a display of attention and engagement. Apart from the speakers and their directly addressed recipients, other unaddressed par- ticipants engaged in the interaction may display interesting gaze behavior as well. For instance, Holler and Kendrick (2015) show that unaddressed participants seem to anticipate turn shifts between the primary participants: in question-response sequences the unaddressed participants typically shift their gaze towards the pro- jected next speaker before the ongoing turn has been completed. As for the regulatory functions of eye gaze, i.e. its role in the sequential organ- ization of the ongoing interaction, several studies have shown that speaker gaze can have a ‘floor apportionment’ function: speakers tend to display gaze aversion briefly after taking the turn and then shift their gaze back to the (primary) recipient shortly before turn completion as a signal of potential turn transition (Duncan, 1975, Kendon, 1967, Streeck, 2014, Rossano, 2010, Jehoul et al., 2017, but also De Ruiter, 2005 for a somewhat different position). Auer (this volume) takes this basic insight and extends it to the interactionally more complex constellation of triadic (and thus multi-party) conversations. He shows that speaker gaze may serve both addressee selection and next-speaker selection, and the sequential position is con- stitutive in determining this function. Thus, while speakers may shift their gaze between different addressees while speaking, the gaze target at the TCU end seems to determine who will be the next speaker. Apart from its function in the distri- bution of turns, gaze may also be used by speakers to monitor or elicit responses by the recipients (Goodwin & Goodwin, 1986, Sweetser & Stec, 2016). Face-to- face interactions typically involve gaze windows, i.e. moments during which both participants look at each other, and which allow speakers to elicit and recipients to realize a form of minimal response (which can be verbal back channels such as mh, nonverbal signals such as headnods, or combinations of these) (Bavelas et al., 2002). Vranjes et al. (this volume) show, for the specific case of interpreter-mediated interaction, that this multimodal display of recipiency involving gaze may differ depending on the specific role of the participant in the interaction. What the studies on the relationship between speaker-hearer gaze and feedback show, is that interactants display a synchronized system of multimodal behavior to achieve successful communication. And this brings us to the more cognitively oriented line of work on eye gaze in interaction, which has focused strongly on joint attention and the construction of shared situation models. One correlate of
The list of books you might like
