Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction

287 Pages·2010·3.07 MB·English

by  Desney Tan

Human-Computer Interaction Series
Editors-in-chief
JohnKarat
IBMThomasJ.WatsonResearchCenter,YorktownHeights,USA
JeanVanderdonckt
UniversitécatholiquedeLouvain,Louvain-la-Neuve,Belgium
EditorialBoard
GaëlleCalvary,LIG-UniversityofGrenoble1,Grenoble,France
JohnCarroll,SchoolofInformationSciences&Technology,PennStateUniversity,
UniversityPark,USA
GilbertCockton,NorthumbriaUniversity,Newcastle,UK
LarryConstantine,UniversityofMadeira,Funchal,Portugal,andConstantine&Lockwood
Ltd,Rowley,MA,USA
StevenFeiner,ColumbiaUniversity,NewYork,USA
PeterForbrig,UniversitätRostock,Rostock,Germany
ElizabethFurtado,UniversityofFortaleza,Fortaleza,Brazil
HansGellersen,LancasterUniversity,Lancaster,UK
RobertJacob,TuftsUniversity,Medford,USA
HilaryJohnson,UniversityofBath,Bath,UK
KumiyoNakakoji,UniversityofTokyo,Tokyo,Japan
PhilippePalanque,UniversitéPaulSabatier,Toulouse,France
OscarPastor,UniversityofValencia,Valencia,Spain
FabioPianesi,BrunoKesslerFoundation(FBK),Trento,Italy
CostinPribeanu,NationalInstituteforResearch&DevelopmentinInformatics,Bucharest,
Romania
GerdSzwillus,UniversitätPaderborn,Paderborn,Germany
ManfredTscheligi,UniversityofSalzberg,Salzburg,Austria
GerritvanderVeer,UniversityofTwente,Enschede,TheNetherlands
ShuminZhai,IBMAlmadenResearchCenter,SanJose,USA
ThomasZiegert,SAPResearchCECDarmstadt,Darmstadt,Germany
Human-Computer Interaction is a multidisciplinary field focused on human aspects of the
developmentofcomputertechnology.Ascomputer-basedtechnologybecomesincreasingly
pervasive – not just in developed countries, but worldwide – the need to take a human-
centered approach in the design and development of this technology becomes ever more
important.Forroughly30yearsnow,researchersandpractitionersincomputationalandbe-
havioralscienceshaveworkedtoidentifytheoryandpracticethatinfluencesthedirectionof
thesetechnologies,andthisdiverseworkmakesupthefieldofhuman-computerinteraction.
Broadlyspeaking,itincludesthestudyofwhattechnologymightbeabletodoforpeople
andhowpeoplemightinteractwiththetechnology.
Inthisseries,wepresentworkwhichadvancesthescienceandtechnologyofdeveloping
systems which are both effective and satisfying for people in a wide variety of contexts.
Thehuman-computerinteractionserieswillfocusontheoreticalperspectives(suchasformal
approachesdrawnfromavarietyofbehavioralsciences),practicalapproaches(suchasthe
techniquesforeffectivelyintegratinguserneedsinsystemdevelopment),andsocialissues
(suchasthedeterminantsofutility,usabilityandacceptability).
Forfurthervolumes:
http//www.springer.com/series/6033
Desney S. Tan (cid:2) Anton Nijholt
Editors
Brain-Computer
Interfaces
Applying our Minds to
Human-Computer Interaction
Editors
DesneyS.Tan AntonNijholt
MicrosoftResearch Fac.ElectricalEngineering,
OneMicrosoftWay Mathematics&ComputerScience
Redmond UniversityofTwente
WA98052 Enschede
USA TheNetherlands
[e-mail] [e-mail]
ISSN1571-5035
ISBN978-1-84996-271-1 e-ISBN978-1-84996-272-8
DOI10.1007/978-1-84996-272-8
SpringerLondonDordrechtHeidelbergNewYork
BritishLibraryCataloguinginPublicationData
AcataloguerecordforthisbookisavailablefromtheBritishLibrary
LibraryofCongressControlNumber:2010929774
©Springer-VerlagLondonLimited2010
Apartfromanyfairdealingforthepurposesofresearchorprivatestudy,orcriticismorreview,asper-
mittedundertheCopyright,DesignsandPatentsAct1988,thispublicationmayonlybereproduced,
storedortransmitted,inanyformorbyanymeans,withthepriorpermissioninwritingofthepublish-
ers,orinthecaseofreprographicreproductioninaccordancewiththetermsoflicensesissuedbythe
CopyrightLicensingAgency.Enquiriesconcerningreproductionoutsidethosetermsshouldbesentto
thepublishers.
Theuseofregisterednames,trademarks,etc.,inthispublicationdoesnotimply,evenintheabsenceofa
specificstatement,thatsuchnamesareexemptfromtherelevantlawsandregulationsandthereforefree
forgeneraluse.
Thepublishermakesnorepresentation,expressorimplied,withregardtotheaccuracyoftheinformation
containedinthisbookandcannotacceptanylegalresponsibilityorliabilityforanyerrorsoromissions
thatmaybemade.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Human-Computer Interaction (HCI) research used to be about the ergonomics of
interfacesand,interfacesusedtoconsistofakeyboard,amouseandwhatevercould
bedisplayedonthescreenofamonitor,thatis,thegraphicaluserinterface.Nowa-
days, when we talk about Human-Computer Interaction research, we are talking
about multimodal interaction in environments where we research natural human
behavior characteristics in general, rather than looking at keyboard and mouse in-
teraction.Theenvironmentsweliveinsupportusinouractivities.Sensor-equipped
environmentsknowaboutus,ouractivities,ourpreferences,andaboutourinterac-
tionsinthepast.Thisknowledgeisobtainedfromourinteractionbehavior,behavior
that can be observed and interpreted using knowledge that becomes available and
that can be fused from cameras, microphones, and position sensors. This allows
the environment to not only be reactive, but also proactive, anticipating the user’s
activities,needsandpreferences.
LesstraditionalsensorsarenowbeingintroducedintheHuman-ComputerInter-
actionfield.Theaimistogatherasmuchinformationaspossiblefromthehuman
interaction partner and the context, including the interaction history, that can be
sensed,interpreted,andstored.Thisinformationmakesitpossiblefortheenviron-
ment to improve its performance when supporting its users or inhabitants in their
daily activities. These sensors detect our activities, whether we move and how we
moveandtheycanbeembeddedinourclothesandindeviceswecarrywithus.In
thepast,physiologicalsensorshavebeenusedtoevaluateuserinterfaces.Howdoes
theuserexperienceaparticularuserinterface?Whatcanwelearnfrominformation
aboutheartrate,bloodpressureandskinconductivityabouthowauserexperiences
aparticularinterface?Suchinformationcanhelpinimprovingthedesignofaninter-
face.Atpresentweseetheintroductionofthesephysiologicalsensorsindeviceswe
carrywithusorthatareembeddedindevicesthatallowexplicitcontrolofcomputer
orcomputercontrolledenvironments.Hence,thisinformationcanbeused‘on-line’,
thatis,toimprovethereal-timeinteraction,ratherthan‘off-line’,thatis,toimprove
thequalityoftheinterface.Thisinformationgivesinsightintheuser’saffectiveand
cognitivestateandithelpsustounderstandtheutterancesandactivitiesoftheuser.
Itcanbeusedtoprovideappropriatefeedbackortoadapttheinterfacetotheuser.
v
vi Preface
Now we see the introduction of sensors that provide us with information that
comes directly from the human brain. As in the case of the physiological sensors
mentionedabove,informationfromtheseneuro-physiologicalsensorscanbeused
toprovidemorecontextthathelpsustointerpretauser’sactivitiesanddesires.In
addition,brainactivitycanbecontrolledbytheuseranditcanbeusedtocontrolan
application.Hence,ausercandecidetousehisorherbrainactivitytoissuecom-
mands.Oneexampleismotorimagery,wheretheuserimaginesacertainmovement
inorderto,forexample,navigateinavirtualorphysicalenvironment.Ontheother
hand,anenvironmentcanattempttoissuesignalsfromwhichitcanbecomeclear,
by looking at the initiated brain activity, what the user is interested in or wants to
achieve.
Theadvancesincognitiveneuroscienceandbrainimagingtechnologiesprovide
us with the increasing ability to interface directly with activity in the brain. Re-
searchershavebeguntousethesetechnologiestobuildbrain-computerinterfaces.
Originally,theseinterfacesweremeanttoallowpatientswithseveremotordisabil-
ities to communicate and to control devices by thought alone. Removing the need
formotormovementsincomputerinterfacesischallengingandrewarding,butthere
is also the potential of brain sensing technologies as input mechanisms that give
access to extremely rich information about the state of the user. Having access to
thisinformationisvaluabletoHuman-ComputerInteractionresearchersandopens
upatleastthreedistinctareasofresearch:controllingcomputersbyusingthought
aloneorasacomplementaryinputmodality,evaluatingsystemsandinterfaces,and
buildingadaptiveuserinterfaces.
Specifically,thisbookaimstoidentifyanddiscuss
• Brain-computer interface applications for users with permanent and situational
physicaldisabilities,aswellasforable-bodiedusers;thisincludesapplicationin
domainssuchastraditionalcommunicationandproductivitytasks,aswellasin
gamesandentertainmentcomputing;
• Sensingtechnologiesanddataprocessingtechniquesthatapplywelltothesuite
ofapplicationsinwhichHCIresearchersareinterested;
• Techniques for integrating brain activity, whether induced by thought or by
performing a task, in the palette of input modalities for (multimodal) Human-
ComputerInteraction
The Human-Computer Interaction field has matured much in the last several
decades.Itisnowfirmlyrootedasafieldthatconnectsmoretraditionalfieldssuch
ascomputerscience,design,andpsychologyinsuchawayastoallowustolever-
ageandsynthesizeworkinthesespacestobuildtechnologiesthataugmentourlives
in some way. The field has also built up well-defined methodologies for repeating
thisworkacrossaseriesofdisciplines.Simultaneously,neurosciencecontinuesto
advancesufficientlyfastandbrain-computerinterfacesarestartingtogainenough
traction so that we believe it is a field ripe for collaboration with others such as
HCI.Infact,wearguethatthespecificpropertiesofthetwofieldsmakethemex-
tremely well suited to cross-fertilization, and that is the intent of this book. That
said,wehopethatthespecificwaywehavecraftedthisbookwillalsoprovidebrain-
Preface vii
computerinterfaceresearcherswiththeappropriatebackgroundtoengagewithHCI
researchersintheirwork.
Acknowledgements TheeditorsaregratefultoHendriHondorpforhishelpwitheditingthis
book.
Redmond/Enschede DesneyTan
AntonNijholt
Contents
PartI OverviewandTechniques
1 Brain-ComputerInterfacesandHuman-ComputerInteraction . . . 3
DesneyTanandAntonNijholt
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.1.1 The Evolution of BCIs and the Bridge with Human
ComputerInteraction . . . . . . . . . . . . . . . . . . . . 5
1.2 BrainImagingPrimer . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 ArchitectureoftheBrain . . . . . . . . . . . . . . . . . . 7
1.2.2 GeographyofThought. . . . . . . . . . . . . . . . . . . . 7
1.2.3 MeasuringThoughtwithBrainImaging . . . . . . . . . . 8
1.2.4 BrainImagingTechnologies . . . . . . . . . . . . . . . . . 8
1.3 BrainImagingtoDirectlyControlDevices . . . . . . . . . . . . . 10
1.3.1 BypassingPhysicalMovementtoSpecifyIntent . . . . . . 10
1.3.2 LearningtoControlBrainSignals . . . . . . . . . . . . . . 10
1.3.3 EvaluationofPotentialImpact . . . . . . . . . . . . . . . 11
1.4 BrainImagingasanIndirectCommunicationChannel . . . . . . . 12
1.4.1 ExploringBrainImagingforEnd-UserApplications . . . . 12
1.4.2 UnderstandingCognitionintheRealWorld. . . . . . . . . 13
1.4.3 CognitiveStateasanEvaluationMetric. . . . . . . . . . . 14
1.4.4 AdaptiveInterfacesBasedonCognitiveState. . . . . . . . 14
1.5 TheRestoftheBook . . . . . . . . . . . . . . . . . . . . . . . . 16
Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2 NeuralControlInterfaces . . . . . . . . . . . . . . . . . . . . . . . . 21
MelodyMooreJacksonandRudolphMappus
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 Background-Biofeedback . . . . . . . . . . . . . . . . . . . . . . 22
2.3 ControlTasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
ix
x Contents
2.3.1 ExogenousControlTaskParadigms . . . . . . . . . . . . . 23
2.3.2 EndogenousControlTaskParadigms . . . . . . . . . . . . 24
2.4 CognitiveModelsofInteraction . . . . . . . . . . . . . . . . . . . 25
2.5 InteractionTaskFrameworks . . . . . . . . . . . . . . . . . . . . 26
2.5.1 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.5.2 TextandQuantify . . . . . . . . . . . . . . . . . . . . . . 28
2.5.3 Position . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6 DialogInitiative . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.6.1 SynchronousInterfaces . . . . . . . . . . . . . . . . . . . 29
2.6.2 AsynchronousInterfaces . . . . . . . . . . . . . . . . . . 29
2.6.3 UserAutonomy . . . . . . . . . . . . . . . . . . . . . . . 29
2.7 ImprovingBCIControlInterfaceUsability . . . . . . . . . . . . . 30
2.7.1 UserTraining . . . . . . . . . . . . . . . . . . . . . . . . 31
2.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3 CouldAnyoneUseaBCI? . . . . . . . . . . . . . . . . . . . . . . . . 35
BrendanZ.AllisonandChristaNeuper
3.1 WhyBCIs(Sometimes)Don’tWork . . . . . . . . . . . . . . . . 35
3.2 IlliteracyinDifferentBCIApproaches . . . . . . . . . . . . . . . 37
3.2.1 IlliteracyinERDBCIs. . . . . . . . . . . . . . . . . . . . 37
3.2.2 IlliteracyinSSVEPBCIs . . . . . . . . . . . . . . . . . . 39
3.2.3 IlliteracyinP300BCIs . . . . . . . . . . . . . . . . . . . 40
3.3 ImprovingBCIFunctionality . . . . . . . . . . . . . . . . . . . . 42
3.3.1 ImproveSelectionand/orClassificationAlgorithms . . . . 42
3.3.2 ExploreDifferentNeuroimagingTechnologies . . . . . . . 43
3.3.3 ApplyErrorCorrectionorReduction . . . . . . . . . . . . 44
3.3.4 GenerateBrainSignalsthatareEasiertoCategorize . . . . 44
3.3.5 PredictingIlliteracy . . . . . . . . . . . . . . . . . . . . . 46
3.4 TowardsStandardizedTerms,Definitions,andMeasurement
Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
3.4.1 TheRelativeSeverityofIlliteracy . . . . . . . . . . . . . . 49
3.4.2 (Re)Defining“BCIIlliteracy”. . . . . . . . . . . . . . . . 50
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
4 Using Rest Class and Control Paradigms for Brain Computer
Interfacing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
SiamacFazli,MártonDanóczy,FlorinPopescu,BenjaminBlankertz,
andKlaus-RobertMüller
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.1.1 ChallengesinBCI . . . . . . . . . . . . . . . . . . . . . . 56
4.1.2 BackgroundonRestClassandControllerConcepts . . . . 58
4.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2.1 ExperimentalParadigm . . . . . . . . . . . . . . . . . . . 59
4.2.2 FeatureExtraction . . . . . . . . . . . . . . . . . . . . . . 60