Cognition, Communication and Interaction: Transdisciplinary Perspectives on Interactive Technology

609 Pages·2008·10.591 MB·English

by  Satinder P. Gill (auth.), Satinder Gill (eds.)

Human-Computer Interaction Series
Editors-in-chief
John Karat
IBM Thomas Watson Research Center (USA)
Jean Vanderdonckt
Université catholique de Louvain (Belgium)
Editorial Board
Gregory Abowd, Georgia Institute of Technology (USA)
Gaëlle Calvary, LIG-University of Grenoble I (France)
John Carroll, School of Information Sciences & Technology, Penn State University (USA)
Gilbert Cockton, University of Sunderland (UK)
Mary Czerwinski, Microsoft Research (USA)
Steve Feiner, Columbia University (USA)
Elizabeth Furtado, University of Fortaleza (Brazil)
Kristiana Höök, SICS (Sweden)
Robert Jacob, Tufts University (USA)
Robin Jeffries, Google (USA)
Peter Johnson, University of Bath (UK)
Kumiyo Nakakoji, University of Tokyo (Japan)
Philippe Palanque, Université Paul Sabatier (France)
Oscar Pastor, University of Valencia (Spain)
Fabio Paternò, ISTI-CNR (Italy)
Costin Pribeanu, National Institute for Research & Development in Informatics (Romania)
Marilyn Salzman, Salzman Consulting (USA)
Chris Schmandt, Massachussetts Institute of Technology (USA)
Markus Stolze, IBM Zürich (Switzerland)
Gerd Szwillus, Universität Paderborn (Germany)
Manfred Tscheligi, University of Salzburg (Austria)
Gerrit van der Veer, University of Twente (The Netherlands)
Schumin Zhai, IBM Almaden Research Center (USA)
Human-Computer Interaction is a multidisciplinary field focused on human aspects of the development
of computer technology. As computer-based technology becomes increasingly pervasive – not just in
developed countries, but worldwide – the need to take a human-centered approach in the design and
development of this technology becomes ever more important. For roughly 30 years now, researchers and
practitioners in computational and behavioral sciences have worked to identify theory and practice that
influences the direction of these technologies, and this diverse work makes up the field of human-
computer interaction. Broadly speaking it includes the study of what technology might be able to do for
people and how people might interact with the technology.
In this series we present work which advances the science and technology of developing systems which
are both effective and satisfying for people in a wide variety of contexts. The human-computer interaction
series will focus on theoretical perspectives (such as formal approaches drawn from a variety of
behavioral sciences), practical approaches (such as the techniques for effectively integrating user needs
in system development), and social issues (such as the determinants of utility, usability and
acceptability).
Author guidelines: springer.com > Authors > Author Guidelines
Also in this series
Rossi, G., Pastor, O., Schwabe, D., Olsina, L. (Eds.) Karat, C.-M., Blom, J.O., Karat, J. (Eds.)
Web Engineering – Modelling and Implementing Designing Personalized User Experiences in
Web Applications eCommerce
ISBN 978-1-84628-922-4, 2008 Vol. 5, ISBN 978-1-4020-2147-3, 2004
Law, E., Hvannberg, E., Cockton, G. (Eds.) Ivory, M.Y.
Maturing Usability – Quality in Software, Automating Web Site Evaluation – Researchers’
Interaction and Value and Practitioners’ Perspectives
ISBN 978-1-84628-940-8, 2008 Vol. 4, ISBN 978-1-4020-1672-1, 2004
Lieberman, H., Paternò, F., Wulf, V. (Eds.) Blythe, M.A., Overbeeke, K., Monk, A.F., (et al.)
End User Development (Eds.)
Vol. 9, ISBN 978-1-4020-4220-1, 2006 Funology – From Usability to Enjoyment
Vol. 3, ISBN 978-1-4020-2966-0, 2004 (softcover)
Lieberman, H., Paternò, F., Wulf, V. (Eds.)
End User Development Blythe, M.A., Overbeeke, K., Monk, A.F., (et al.)
Vol. 9, ISBN 978-1-4020-5309-2, 2006 (softcover) (Eds.)
Funology – From Usability to Enjoyment
Seffah, A., Gulliksen, J., Desmarais, M.C. (Eds.) Vol. 3, ISBN 978-1-4020-1252-5, 2003
Human-Centred Software Engineering –
Integrating Usability in the Software Development Schreck, J.
Lifecycle Security and Privacy in User Modeling
Vol. 8, ISBN 978-1-4020-4027-6, 2005 Vol. 2, ISBN 978-1-4020-1130-6, 2003
Ruttkay, Z., Pelachaud, C., (Eds.) Chi, E.H.
From Brows to Trust – Evaluating Embodied A Framework for Visualizing Information
Conservational Agents Vol 1, ISBN 978-1-4020-0589-2, 2002
Vol. 7, ISBN 978-1-4020-2729-1, 2004
Ardissono, L., Kobsa, A., Maybury, M.T. (Eds.)
Personalized Digital Television – Targeting
Programs to Individual Viewers
Vol. 6, ISBN 978-1-4020-2147-3, 2004
Satinder Gill
Editor
Cognition, Communication
and Interaction
Transdisciplinary Perspectives on Interactive
Technology
123
Satinder Gill
Senior Research Fellow
School of Computing
Middlesex University
UK
British Library Cataloguing in Publication Data
A catalogue record for this book is available from the British Library
Library of Congress Control Number: 2007936184
Human-Computer Interaction Series ISSN 1571-5035
ISBN: 978-1-84628-926-2 e-ISBN: 978-1-84628-927-9
Printed on acid-free paper
© Springer-Verlag London Limited 2008
Apart from any fair dealing for the purposes of research or private study, or criticism or
review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may
only be reproduced, stored or transmitted, in any form or by any means, with the prior
permission in writing of the publishers, or in the case of reprographic reproduction in
accordance with the terms of licences issued by the Copyright Licensing Agency. Enquiries
concerning reproduction outside those terms should be sent to the publishers.
The use of registered names, trademarks, etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the relevant laws and
regulations and therefore free for general use.
The publisher makes no representation, express or implied, with regard to the accuracy of the
information contained in this book and cannot accept any legal responsibility or liability for
any errors or omissions that may be made.
9 8 7 6 5 4 3 2 1
Springer Science+Business Media
springer.com
Preface
This book examines the theoretical and methodological research issues that underlie
the design and use of interactive technology. Present interactive designs are address-
ing the multi-modality of human interaction and the multi-sensory dimension of how
we engage with each other. The writings contribute to the growing trans-disciplinary
research for interaction design. The analysis directs attention to three human capaci-
ties that our engagement with interactive technology has made salient and open to
constant redefinition. These capacities are human cognition, communication, and
interaction.
The collection is a celebration of the 21 years of the foundation of the AI & Soci-
ety Journal. Over the years, a wide horizon of researchers, practitioners and scholars
have contributed to shaping a transdisciplinary debate on cognition, communication
and interaction. The discussions represent this ongoing debate by bringing together
timeless and historical papers on some of the fundamental issues that the journal's
life has travelled through.
It brings together work by researchers and practitioners from a wide range of dis-
courses that are pertinent to understanding the boundaries and possibilities produced
by the intervention of interactive technology in all spheres of human social life. The
authors come from philosophy, human-centred design, interactive art, participatory
design, computer supported cooperative working (CSCW), human-computer interac-
tion (HCI), dance, opera, theatre, interactive multi-modal design, cognitive technol-
ogy, knowledge management, ambient interactive design, immersive and responsive
technology, presence research, communications, linguistics, social science, psychol-
ogy, engineering, and computer science.
2 Navigating the Transdisciplinary Debate
Interactive technology has now entered every sphere of human life, from the most
private (e.g. the home, the ‘presence’ Amigo Project) to the most public spaces (e.g.
v
RCA’s Presence Project, Buxton 2001; Immersive responsive media projects, Topo-
logical Media Lab, Concordia University, Sha 2005). The conceptual separation of
mind and body, so prevalent in early work on AI and information systems (Fodor,
1976, 1981, Simon, 1985), has given way to the concepts of embodied cognition
(Varela, Thomson, Rosch, 1993). The user model is questioned with a shift from the
user being an individual, to the user being part of a community of practice (Wenger
and Lave, 1991). Designers now include performance artists, interactive artists, as
well as social scientists, philosophers, engineers, physicists, mathematicians, and
computer scientists. This is a far cry from the days of computer scientists trying to
elicit and explicate expertise to create autonomous expert systems. These changes of
the cross-disciplinary make-up of designers (their motivations, perceptions, back-
grounds, cultures) and the adaptation and sometimes unexpected use of their applica-
tions across contexts, necessitates taking a step back to make sense of this complex-
ity and its implications. The writings in this book are a collection of reflections,
concepts, methodologies, and applications, that address the fundamental bases on
which we can understand these changes and harness the issues they raise for design-
ing technologies to be symbiotic with human capacities and qualities.
The book is divided into three parts: Part I, on "Communication and Interaction",
provides an analysis of those aspects of human communication and interaction that
computer mediated communication, immersive, and interactive technology impacts
upon. Part II of the book covers the impact of interactive technologies on the concep-
tions and performance of human knowledge and cognition. It presents issues pertain-
ing to this impact and provides design concepts for interaction design. In Part III, on
Ethics, Aesthetics, and Design, the understanding of the tacit dimension of human
communication and human knowledge for designing interactive technologies is ex-
tended to consider the ethical and aesthetic aspects.
3 Part I: Communication and Interaction
Part I, on "Communication and Interaction", provides an analysis of those aspects of
human communication and interaction that computer mediated communication,
immersive, and interactive technology impacts upon. These impacts in turn cause us
to rethink the distinctions between human-human communication and that impacted
upon by the machine artifacts we increasingly use to mediate or even substitute that
communication. The effects on our interaction practices also asks us to consider the
nature of our relation with the machine and try to define what 'machine' means (Ne-
grotti).
- whether it is a tool and what does 'tool' mean. (Mey), on the relationship be-
tween cognition and tool in "Cognitive Technology")
- whether it is an augmentation and what does that mean in the context of com-
munication.
- whether it is a simulation and what does that mean in contrast to say going to
the theatre and engaging with a simulation of everyday life.
This analysis unravels fundamentals of human communication that are necessary to
take into account when considering design and application of such technologies. For
example:
vi
- the effect upon how we negotiate and make decisions.
- on how we relate to others and perceive ourselves.
- on the rhythmic synchrony of our engagement with each other and our envi-
ronment. (SP Gill on communication and knowledge transformation in "Cognition,
Communication, and Interaction")
- on conceptions of collaboration, cooperation, and coordination (Fruchter re-
flects on "Degrees of Engagement in Interactive Workspaces" with practical applica-
tions of collaborative interfaces).
- on conceptions of learning (Binder "Designing for Workplace Learning")
- on what it means to be in a community (Memmi proposes that virtual communi-
ties of abstract relationships are evolution of modern society).
In considering these fundamentals of communication, design itself involves these
very aspects in its process of creating the machine artefact. The complexity of de-
signing the shape and possibilities of communication and interaction necessitates a
complex of perspectives that can encompass it. In the 1980's the complexity of in-
formation or expert systems threw up a concern for participation in the design proc-
ess by potential users to ensure that 'design' and 'use' would meet. The present day
concern with interactive multi-modal and multi-sensory technology is to involve
numerous disciplines, kinds of expertise, as well as potential users, to ensure that
'design' and 'use' meet for human communication. The multiplicity of perspectives
becomes problematic when commonly held conceptions of say, 'use', are rooted in
different cultural traditions of design practice (Allwood and Hakken on "Use Dis-
courses in System Development"). This added complexity of perspectives within the
design process engages a multi-layering of language games that need to find com-
mon ground (Borchers, on "Pattern Approach to Interaction Design"). This multi-
layering is intricate in global communication where interactive technologies increas-
ingly intervene. Their intervention lies outside of cultural practices that need to find
some translation. Addressing this in design of interactive technology involves the
consideration of one's relation to others and perception of self, and one's existence in
a community.
The nature of global communications is not just about individuals communicat-
ing with other individuals. It is about how we, as individuals, communicate from
within our communities of practice or cultures (Banerjee, on cross-cultural commu-
nication and the management of language, "Narration, Discourse, and Dialogue".).
This conception of global communication is described as the communication be-
tween cultural architectures or holonic structures within which we as individuals
engage (KS Gill, on "The User and the Interaction Architecture"). Cross-cultural
communication becomes a communication matrix of cultural holons (e.g. cultures,
for instance, Japanese and British). Inevitably, as different cultural holons have dif-
ferent architectures, there are gaps in their communication. KS Gill proposes that
such gaps in global communication necessitate 'cross-appropriation' and not just
'common ground', which is a Western concept of mutual understanding.
The design of interactive multi-modal and multi-sensory technology has the chal-
lenge to both sustain and shape a concept of the global that is sustainable at the local
community level, in order that individuals can sustain their everyday communica-
tions with those they live, walk, play, learn, and work with.
vii
The shape that such designs take will be determined by the motivations of those
involved in their design. Negrotti, in "Towards a Theory of the Artificial", takes us
through the possible motivations through history of man's creation of the artificial,
which are to simulate, reproduce (imitate), and control nature, and now human be-
haviour. He proposes that 'control', rather than simulation or reproduction, offers the
possibility for technology to be adaptive to human purpose.
4 Part II: Knowledge and Cognition
Part II of the book covers the impact of interactive technologies on the conceptions
and performance of human knowledge and cognition. It presents issues pertaining to
this impact and provides design concepts for interaction design. In particular it cov-
ers the relationship between the tacit and explicit dimensions of 'knowing' in human
skill acquisition, multi-modal communication, and multi-sensory human engage-
ment.
The tacit-explicit knowledge discussion was salient in the 1980's amongst propo-
nents of knowledge based interactive systems who,
- assumed implicit knowledge to be something that could be made explicit
- designed structures of definition and causation for its representation and appli-
cation.
- created masses of data and faced problems of processing it.
Tacit/implicit knowledge became termed a bottleneck in knowledge based interactive
systems design. The model of the neuron as an alternative to causal propositions, for
moving and processing knowledge, seemed to alleviate the 'bottleneck' problem as
technologists built hybrid neural-network/knowledge based interactive technologies.
However, the explication of the tacit was to simulate human knowledge in order to
achieve one of many possible functions of technology. Some basic ones are cited in
Part I, namely, to augment, to simulate, to replicate (imitate) human action, human
decision-making, human judgement, and human communication. In Part II, Dreyfus
on "The Socratic and Platonic Basis of Cognitivism", and Noble (Cockpit Cognition)
provide us with the historical and conceptual bases of the Cognitivist paradigm of
human knowledge and skill, and give insights into the problems of representation.
Noble, directs our attention to the evolving notions of 'intelligence' and education
technologies. FT Evans provides a historical understanding of how man's creativity
shapes technology and reflects on how this technology can in turn affect human
creative capacity.
The discussions here on the tacit in relation to the explicit provide definitions,
frameworks, and contextual exemplars to explain what constitutes the tacit and why
it is related to the explicit. One reason that technologists faced the 'bottleneck' prob-
lem above, is that they separated the tacit from the explicit by assuming the latter
subsumes the former. This is to assume that you can separate the wood from the trees
and focus on the trees and still see the wood. As Cooley puts it in Part III, ("On
Human Machine Symbiosis), these are mutually exclusive. It is the problem Polanyi
addressed in the Tacit Dimension, when he said, to paraphrase, that you can attend
from the picture to the parts that constitute it, but you cannot attend from the parts
and see the picture. A bottleneck was bound to happen if you did as you would never
reach the whole.
viii
But the design problem has two dimensions to it, of 'design' meeting 'use'. The
design involves assumptions about what it is to 'know' and 'have knowledge'. The use
part is where the humans interact with the representations of 'knowing' and 'having
knowledge' as embedded in the machine interaction. In her work, Josefson ("The
Nurse as an Engineer") presents concerns for nurses' tacit knowing when the lan-
guage they are required to use in their practice is imported from medical practice, in
order to make their 'profession' scientific. In this example, language becomes tech-
nology that is not rooted in experiential and personal knowing, leading to a gap be-
tween knowing and having knowledge.
Early work on interactive technology focused on the written word and mathe-
matical representations and sometimes included speech. These representations im-
pacted on human cognition, one of the most troubling outcomes being that skilled
experts could doubt their own judgements and pass on decisions to machines (Go-
ranzon, "The Practice and Use of Computers"). This is of concern as skill and inno-
vation in any organization is dependent on 'tacit knowledge' (Senker, on the "Contri-
bution of Tacit Knowledge to Innovation") that is rooted in personal interaction,
mutual construction, and managing uncertainty. It is necessary for us to understand
the cognitive impact of our engagement with representations of our skilled perform-
ance and the designs of interactive interfaces that are based on these.
Present interactive designs are addressing the multi-modality of human interac-
tion and the multi-sensory dimension of how we engage with each other. The illusion
of 'intelligence' now extends to the sphere of human communication where machines
such as interactive embodied virtual agents and robots, are assigned human-like
qualities that cause in us affective responses, and a perception of a 'social intelli-
gence' (SP Gill, "On Tacit Knowing").
At the heart of both the present and previous forms of interactive technologies
lies the conception of 'rule' but not 'rule-following'. The latter lies in the performance
of human knowledge and the former the abstraction of it (Johannessen, "Tacit Know-
ing and Rule-Following"). The former is embedded in the machine and guides the
machine interaction. The latter lies in human interaction and guides human dialogue.
The performance domains such as dance, opera, theatre, music, are helpful, al-
most vital, for analyzing and experiencing how we achieve rule-following or 'practi-
cal knowledge', giving us awareness that this is constituted by the human imagina-
tion, human senses, the body, and personal and experiential knowing. Ikuta ("The
Role of Craft Language in Learning 'Waza'") provides an example of this with the
embodiment of the traditional Japanese artistic performance, in this case Noh or
Kabuki, by the performer through metaphor, imagination, experience, and the im-
mersion of self. Interactive technologies can place an emphasis on the visual and
textual representations of human knowledge and cognition, but if one is blind, such
emphasis is of little 'use' and it may be more appropriate and more universal to de-
velop conceptions of 'knowing how', 'knowing when' and 'knowing what', around
action, the body, and emotion (Saha and Gangopadhyay, on "Building a Pedagogy
around Action and Emotion: Experiences of Blind Opera of Kolkata"). Develop-
ments in responsive media technologies, (Sha, in Part III) take us in this direction,
and the enrichment of the performance arts is opening up the possibilities for the
multi-modal and multi-sensory dimensions of human knowing. This perspective of
human knowing as skilled performance allows for the movement from 'seeing-as' (an
instruction in a Noh dance lesson) to 'seeing' (embodied skilled performance)
(Tilghman).
ix
As well as looking closely into design assumptions and design processes, the
technics of design, it is important to reflect back on the design motivations behind
the representations of human knowing, as raised by Negrotti and KS Gill in Part I.
5 Part III: Ethics, Aesthetics and Design
And this takes us to Part III, on Ethics, Aesthetics, and Design, where an understand-
ing of the tacit dimension of human communication, human knowledge, for design-
ing interactive technologies, is extended to consider the ethical and aesthetic aspects.
If we synthesise the essence of the tacit as being, 'we know more than we can tell'
(Polanyi, 1966), then ethics and aesthetics are part of the tacit dimension and it is not
insignificant that as we enter the complexity of human communication and interac-
tion, that these aspects of human knowledge are becoming of greater importance as
interactive design criteria. The writings in Part III unravel the critical ethical ques-
tions that target both design and use, and provide examples and analyses of how the
narrative domains redefine the aesthetic quality of both the design experience and
applications of design. The challenge for interactive technology designers is to
achieve both the ethical and the aesthetic requirements for a symbiosis of human and
machine which ensures the human capacity for being together with other humans, in
social and cultural co-existence, as discussed in Parts I and Part II. This suggests that
aesthetics within the sphere of interaction design would need to be in balance with
the ethics of sociality.
The ethics of interactive design are concerned with the following conceptual and
methodological issues:
- the distinction between causality and purpose - the constraints of defining the
complexity of human communication in terms of 'one-best-way' (Rosenbrock, "Eth-
ics and Intellectual Structures")
- defining human relations with machine metaphors (Hirose, "Organisational
Spaces and Intelligent Machines: A Metaphorical Approach to Ethics")
- understanding the human social capacity of 'normal responsible behaviour', es-
sential for the formation and functioning of any community (Leal, "Ethics is Fragile,
Goodness is Not")
These methodological issues underlie the design considerations for human-
machine symbiotic interaction, and are developed by Cooley ("On Human Machine
Symbiosis"). Understanding how a designer thinks and being able to shape that for
ethical design is part of the methodological framework (Ostberg, "What Goes on
When a Designer Thinks?").
The ways designers think will also be determined by who the designers are, what
their backgrounds are, their skills. An artist will have a different way of thinking
about design than a computer scientist. In Part I we discussed the complexity of
finding shared ground in the communication amongst multi-disciplinary designers
that is now essential because of the complexity of the arena of interactive technology
that impinges on the relations between human cognition, communication, and inter-
action. In this section, we are given an insight into how artists conceive and create
databases and how their aesthetics and artistic practices shape these processes of
design meeting use (Vesna, "Databases are Us").
x