Table Of Content

Spoken Multimodal Human-Computer Dialogue in Mobile Environments Text, Speech and Language Technology VOLUME 28 Series Editors Nancy Ide, Vassar College, New York Jean Véronis, Université de Provence and CNRS, France Editorial Board Harald Baayen, Max Planck Institute for Psycholinguistics, The Netherlands Kenneth W. Church, AT & T Bell Labs, New Jersey, USA Judith Klavans, Columbia University, New York, USA David T. Barnard, University ofRegina, Canada Dan Tufis, Romanian Academy of Sciences, Romania Joaquim Llisterri, Universitat Autonma de Barcelona, Spain Stig Johansson, University of Oslo, Norway Joseph Mariani, LIMSI-CNRS, France The titles published in this series are listed at the end of this volume. Spoken Multimodal Human-Computer Dialogue in Mobile Environments Edited by W. Minker University of Ulm, Germany Dirk ühler University of Ulm, Germany and LailaDybkjræ University of Southern Denmark, Odense, Denmark <£J Springer A C.I.P. Catalogue record for this book is available from the Library of Congress. ISBN 1-4020-3074-6 (PB) ISBN 1-4020-3073-8 (HB) ISBN 1-4020-3075-4 (e-book) Published by Springer, P.O. Box 17, 3300 AA Dordrecht, The Netherlands. Sold and distributed in North, Central and South America by Springer, 101 Philip Drive, Norwell, MA 02061, U.S.A. In all other countries, sold and distributed by Springer, P.O. Box 322, 3300 AH Dordrecht, The Netherlands. Printed on acid-free paper All Rights Reserved © 2005 Springer No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written permission from the Publisher, with the exception of any material supplied specifically for the purpose of being entered and executed on a computer system, for exclusive use by the purchaser of the work. Printed in the Netherlands Contents Preface xi Contributing Authors xiii Introduction xxi Part I Issues in Multimodal Spoken Dialogue Systems and Components 1 Multimodal Dialogue Systems Alexander I. Rudnicky 1. Introduction 3 2. Varieties of Multimodal Dialogue 4 3. Detecting Intentional User Inputs 6 4. Modes and Modalities 7 5. History and Context 7 6. Domain Reasoning 8 7. Output Planning 9 8. Dialogue Management 9 9. Conclusion 10 References 11 2 Speech Recognition Technology in Multimodal/Ubiquitous Com- 13 puting Environments Sadaoki Furui 1. Ubiquitous/Wearable Computing Environment 13 2. State-of-the-Art Speech Recognition Technology 14 3. Ubiquitous Speech Recognition 16 4. Robust Speech Recognition 18 5. Conversational Systems for Information Access 21 6. Systems for Transcribing, Understanding and Summarising Ubiquitous Speech Documents 24 7. Conclusion 32 References 33 vi SPOKEN MULTIMODAL HUMAN-COMPUTER DIALOGUE 3 A Robust Multimodal Speech Recognition Method using Optical 37 Flow Analysis Satoshi Tamura, Koji Iwano, Sadaoki Furui 1. Introduction 38 2. Optical Flow Analysis 39 3. A Multimodal Speech Recognition System 40 4. Experiments for Noise-Added Data 43 5. Experiments for Real-World Data 48 6. Conclusion and Future Work 49 References 52 4 Feature Functions for Tree-Based Dialogue Course Management 55 Klaus Macherey, Hermann Ney 1. Introduction 55 2. Basic Dialogue Framework 56 3. Feature Functions 59 4. Computing Dialogue Costs 63 5. Selection of Dialogue State/Action Pairs 64 6. XML-based Data Structures 65 7. Usability in Mobile Environments 68 8. Results 69 9. Summary and Outlook 74 References 74 5 A Reasoning Component for Information-Seeking and Planning 77 Dialogues Dirk Biihler, Wolfgang Minker 1. Introduction 77 2. State-of-the-Art in Problem Solving Dialogues 80 3. Reasoning Architecture 81 4. Application to Calendar Planning 85 5. Conclusion 88 References 90 6 A Model for Multimodal Dialogue System Output Applied to an 93 Animated Talking Head Jonas Beskow, Jens Edlund, Magnus Nordstrand 1. Introduction 93 2. Specification 97 3. Interpretation 103 4. Realisation in an Animated Talking Head 105 5. Discussion and Future Work 109 References 111 Contents vii Part II System Architecture and Example Implementations 7 Overview of System Architecture 117 Andreas Kellner 1. Introduction 117 2. Towards Personal Multimodal Conversational User Interface 118 3. System Architectures for Multimodal Dialogue Systems 122 4. Standardisation of Application Representation 126 5. Conclusion 129 References 130 XISL: A Modality-Independent MMI Description Language 133 Kouichi Katsurada, Hirobumi Yamada, Yusaku Nakamura, Satoshi Kobayashi, Tsuneo Nitta 1. Introduction 133 2. XISL Execution System 134 3. Extensible Interaction Scenario Language 136 4. Three Types of Front-Ends and XISL Descriptions 140 5. XISL and Other Languages 146 6. Discussion 147 References 148 9 A Path to Multimodal Data Services for Telecommunications 149 Georg Niklfeld, Michael Pucher, Robert Finan, Wolfgang Eckhart 1. Introduction 149 2. Application Considerations, Technologies and Mobile Termi- nals 150 3. Projects and Commercial Developments 154 4. Three Multimodal Demonstrators 156 5. Roadmap for Successful Versatile Interfaces in Telecommuni- cations 161 6. Conclusion 163 References 164 10 Multimodal Spoken Dialogue with Wireless Devices 169 Roberto Pieraccini, Bob Carpenter, Eric Woudenberg, Sasha Caskey, Stephen Springer, Jonathan Bloom, Michael Phillips 1. Introduction 169 2. Why Multimodal Wireless? 171 3. Walking Direction Application 172 4. Speech Technology for Multimodal Wireless 173 5. User Interface Issues 174 6. Multimodal Architecture Issues 179 7. Conclusion 182 References 184 viii SPOKEN MULTIMODAL HUMAN-COMPUTER DIALOGUE 11 The SmartKom Mobile Car Prototype System for Flexible Human- 185 Machine Communication Dirk Biihler, Wolfgang Minker 1. Introduction 185 2. Related Work 186 3. SmartKom - Intuitive Human-Machine Interaction 189 4. Scenarios for Mobile Use 191 5. Demonstrator Architecture 193 6. Dialogue Design 194 7. Outlook - Towards Flexible Modality Control 197 8. Conclusion 199 References 200 12 LARRI: A Language-Based Maintenance and Repair Assistant 203 Dan Bohus, Alexander I. Rudnicky 1. Introduction 203 2. LARRI - System Description 204 3. LARRI - Hardware and Software Architecture 208 4. Experiments and Results 213 5. Conclusion 215 References 217 Part III Evaluation and Usability 13 Overview of Evaluation and Usability 221 Laila Dybkjeer, Niels Ole Bernsen, Wolfgang Minker 1. Introduction 221 2. State-of-the-Art 223 3. Empirical Generalisations 227 4. Frameworks 234 5. Multimodal SDSs Usability, Generalisations and Theory 236 6. Discussion and Outlook 238 References 241 14 Evaluating Dialogue Strategies in Multimodal Dialogue Systems 247 Steve Whittaker, Marilyn Walker 1. Introduction 247 2. Wizard-of-Oz Experiment 251 3. Overhearer Experiment 262 4. Discussion 266 References 267 Contents ix 15 Enhancing the Usability of Multimodal Virtual Co-drivers 269 Niels Ole Bernsen, Laila Dybkjtsr 1. Introduction 269 2. The VICO System 271 3. VICO Haptics - How and When to Make VICO Listen? 272 4. VICO Graphics - When might the Driver Look? 274 5. Who is Driving this Time? 278 6. Modelling the Driver 280 7. Conclusion and Future Work 284 References 285 16 Design, Implementation and Evaluation of the SENECA Spoken 287 Language Dialogue System Wolfgang Minker, Udo Haiber, Paul Heisterkamp, Sven Scheible 1. Introduction 288 2. The SENECA SLDS 290 3. Evaluation of the SENECA SLDS Demonstrator 301 4. Conclusion 308 References 309 17 Segmenting Route Descriptions for Mobile Devices 311 Sabine Geldof, Robert Dale 1. Introduction 311 2. Structured Information Delivery 315 3. Techniques 315 4. Evaluation 322 5. Conclusion 326 References 327 18 Effects of Prolonged Use on the Usability of a Multimodal Form- 329 Filling Interface Janienke Sturm, Bert Cranen, Jacques Terken, Use Bakx 1. Introduction 329 2. The Matis System 332 3. Methods 335 4. Results and Discussion 337 5. Conclusion 345 References 346 19 User Multitasking with Mobile Multimodal Systems 349 Anthony Jameson, Kerstin Klockner 1. The Challenge of Multitasking 350 2. Example System 354 3. Analyses of Single Tasks 354 4. Analyses of Task Combinations 359 5. Studies with Users 364 x SPOKEN MULTIMODAL HUMAN-COMPUTER DIALOGUE 6. The Central Issues Revisited 371 References 375 20 Speech Convergence with Animated Personas 379 Sharon Oviatt, Courtney Darves, Rachel Coulston, Matt Wesson 1. Introduction to Conversational Interfaces 379 2. Research Goals 382 3. Method 383 4. Results 387 5. Discussion 391 6. Conclusion 393 References 394 Index 399