Expanding the Vocabulary of Multitouch Input using Magnetic Fingerprints

2.7 MB·English

by  Halim Cagri Ates

Expanding the Vocabulary of Multitouch Input
using Magnetic Fingerprints
HalimC¸ag˘rıAtes¸ IliasApostolopoulous EelkeFolmer
[email protected] [email protected] [email protected]
ComputerScienceandEngineering
UniversityofNevada
5
1
0
2

n
a
J

4 Figure1. Left: Adevice’smagnetometer(bluearrow)pointstowardstheearth’smagneticfield(North). Center: Asmallnon-obtrusivemagnetis
1 attachedtotheuser’sindexfingerwiththeSouthPoleup. Whenthisfingertouchesthescreen,themagnetometerreportsthedirectionandstrength
ofthemagnet’sfieldandwhichallowsforidentifyingtheindexfinger. Themagnet’sorientationcanbeusedtodefinedifferentfingerprintsasto

] dinstinguishdifferentfingers.Right:ThemiddlefingerisequippedwithamagnetwiththeNorthPoleup.
C
H
ABSTRACT Toincreasethevocabularyoftouchinteraction,anumberof
.
s Wepresentmagneticfingerprints;aninputtechniqueformo- techniques build on or enhance single touch events. Single
c biletouchscreendevicesthatusesasmallmagnetattachedto touch events can be combined into multi-gestures, such as a
[
auser’sfingernailinordertodifferentiatebetweenanormal pinch, but these do not scale up very well due to technical
1 touchandamagnetictouch. Thepolarityofthemagnetcan and physiological constraints and do not offer the same ac-
v be used to create different magnetic fingerprints where this curacy as a single touch input. Temporal features of single
8 technique takes advantage of the rich vocabulary offered by toucheventscanbetakenintoaccounttodefineuniqueges-
1 theuseofmultitouchinput.Userstudiesinvestigatetheaccu- tures,suchasmulti-tapsordwelling.Differenttypesoftouch
2 racyofmagneticfingerprintrecognitioninrelationtomagnet eventscanbedefinedbasedonthesizeofatouchevent[5].
3 size, number of magnetic fingerprints used; and size of the Temporalfeaturesalsocanbecombinedwithspatialinforma-
0
touchscreen. Studiesfoundourtechniquetobelimitedtous- tion,e.g.,socalledmicro-gestures[22],butthistypeofinput
.
1 inguptotwofingerprintsnon-simultaneously,whileachiev- is considered less efficient and more error prone than when
0 ing a high classification accuracy (95%) but it nearly triples usingsingletouch[11].
5 the number of distinguishable multi touch events. Potential
1 usefulapplicationsofthistechniquearepresented. Beyondcountingandtiming,asignificantlylargertouchvo-
: cabulary could be created if we could distinguish between
v
differenttypesofsingletouchevents. Somerecentworkhas
i
X AuthorKeywords already explored this idea by creating distinct touch events
Magneticfingerprinting,touch-screen,fingers,touch, through the incorporation of additional sensing information
r
a multi-user,collaborative,security,input. acquiredwithanaccelerometer[12]ormicrophone[10](see
Related work following) A limitation of these techniques is
that they cannot be used as part of multi-touch input, where
ACMClassificationKeywords
thelargestincreaseintouchvocabularymaybeachieved.
H.5.2. HCI:UserInterfaces;Inputdevicesandstrategies
Though magnets have been previously explored as a mobile
inputtechniquethoseonlyinvolvearound-the-deviceinterac-
INTRODUCTION
tion [9,18] or input with a magnet embedded in an object,
Touch-screens have become a de facto standard of input for
suchasastylus[20]. Ourapproachisnovelasweinstrument
mobile devices as they most optimally use the limited input
afingerwithasmallmagnet, whichallowsustodistinguish
andoutputspacethatisimposedbytheirformfactor.Asingle
magnetic fingers from non-magnetic fingers. Our approach
toucheventistypicallyinterpretedusingspatialinformation,
requiresthesmallestamountofuserinstrumentation. Unlike
but precisely manipulating content is often challenging due
existing approaches, out technique can be used to augment
tosmallscreens,occlusion,variationbetweenusers,andfin-
multitouch input and therefore significantly increase current
gerorientation[13];whichrequiresvirtualbuttonsontouch-
touchvocabulary.
screenstobelargerthanthephysicalbuttonstheyreplace.
1
RELATEDWORK cannotbeusedonexistingmobiletouchscreendevices. Due
Anumberofinputtechnologiesarerelatedtoourapproach. totherearprojectionrequirement,thistechniqueisconfined
We differentiate approaches that augment single touch input tointeractivetables.
fromthosethatexploreusingmagnetsforinput.
A limitation of the aforementioned approaches is that it is
Augmenting single touch input: The spatial and temporal challengingtousethemaspartofamulti-touchgesturemade
featuresofatoucheventcanbeusedtodefinemicro-gestures. withasinglehand. Pressurebasedapproachesonlyseemto
Microrolls [22], e.g., unidirectional or circular rolls or rubs workwithasinglefinger[7,12].Athumbroll[22]oraknock
madewiththethumbcanbeusedtodefinenewtypesofin- withaknuckle[10]areimpracticaltouseaspartofamulti-
put, such as opening up a contextual menu. Sixteen differ- touch gesture, though this is an area in which the largest in-
entmicrogesturescanbedefinedwithanoverallrecognition creaseintouchvocabularymaybeachieved.
accuracy of 95%, though this requires per-user calibration.
Magnetbasedinput: Amagnetometermeasurestheearth’s
Thumbrock[4]improvesupontheaforementionedapproach
magnetic field, and -when combined with an accelerometer-
andachievesa96%accuracyinrecognizingthumbrollswith-
canbeusedtodeterminetheabsolute3Dorientationofamo-
outcalibration. SimPress[2]usesthecontactareaofatouch
biledevice. Magnetometersaretypicallyusedasacompass
event generated by the index finger as a form of simulated
fornavigationortopointoutcontextualgeographicinforma-
pressure to generate different types of single touch events.
tion and are thus widely available in current mobile touch-
Fatthumb[5]extendsthisapproachtothethumbwhichhas
screendevices. Amagnetometeralsosensesthestrengthand
alargersurfacearea. Thisapproachlendsitselfwellforsin-
direction of the magnetic field of a magnet. Because mag-
glehandmobileinputwherepressingthethumbmorefirmly
nets are cheap and do not require an external power source
to the touchscreen can be used for zooming in and pressing
a number of approaches have been explored that use mag-
softlyforzoomingout. Thetotalnumberofcontactsizelev-
nets for wireless around-the-device input. Abracadabra [9],
elsthatcanbefeasiblyusedorrecognizedisnotinvestigated.
Nenya [1] and MagiTact [18] use magnet rings to allow for
Abenefitoftheseapproachesisthattheydonotrequireany
wireless analog input on mobile devices. Abracadabra and
form of instrumentation though it does require per user cal-
Magitactusethemagnet’sdistancefromthemagnetometerto
ibration. Though these approaches work on existing touch-
controlacursororscrollthroughalist. Byrotatingthering,
screendevices,alimitationofcombiningspatialandtempo-
Nenya uses changes in the direction of the magnetic field to
ral information of touch events is that recognition is slower
scroll through a list. MagiWrite [17] enables text input by
andmoreerror-prone[11]thanusingsingletouchevents.
allowing its users to write digits in the air using a handheld
Hinckley and Song [12] present a technique that combines magnet.
capacitivetouchsensingwithaccelerationdata,whichallows
A different set of input techniques embed the magnet into
for distinguishing “hard” from “soft” taps and swipes. This
an object. GaussSense [20] embeds a magnet into a stylus,
approachoffersanaturalsemanticsasahardtapcanbeused
which then facilitates advanced stylus interaction on touch-
to drill down into a menu and a soft tap to go down a sin-
screens where the orientation and tip pressure of a stylus
gle level. No results are presented on the accuracy of this
can be accurately sensed and used for input. GaussSense
approach, though qualitative results are reported. A similar
requires an external sensor grid that is attached to the back
approach is Gripsense [7] which uses inertial sensors and a
of the mobile device. User studies investigate the accuracy
vibrator to measure pressure on a touchscreen. When the
with which finger touches can be distinguished from stylus
user touches the screen the vibrator is briefly activated and
touches. Magpen[15]isasimilartechniquethatusesamag-
the damping of vibrations is measured using the accelerom-
neticstylusbutworkswithadevice’sinternalmagnetometer.
eter and gyroscope. This information can then be used to
Asetofrotarygestureswiththepenareproposed.
distinguishthreelevelsoffingerpressurewithhighaccuracy
(95%)anddistinguishdifferenthandpostures, e.g., athumb Appcessories [3] presents a number of tangible objects with
from an index finger with 84% accuracy. Both approaches magnets embedded in these object. A number of novel in-
workoncurrentmobiletouchscreendeviceswithoutanyex- teraction options are presented, e.g., objects placed on the
trasensors. touchscreencanbeidentifiedusingthedirectionofthemag-
net’smagneticfield. Whencombinedwiththeirspatialloca-
Tapsense[10]combinescapacitivetouchsensingwithacous-
tion, this information can be used to activate different com-
ticinformationacquiredwithanexternalhighfidelitystetho-
mands. By rotating an object the change in the direction of
scope. Capacitivetouchesmadebydifferentobjectsorparts
the projected magnetic field is interpreted to allow for pro-
of the finger make different sounds, which can be classified
vidinganalogformsofinput,suchasscrollingthroughalist.
accordingly. Forsingle-handedmobilefingerinput,fourdif-
GaussBits [19] is a similar approach that uses objects with
ferenttypesoftouchesmadebythetip,pad,nailandknuckle
magnets embedded in it. GaussBits uses an external mag-
can be recognized with a high 95% accuracy, but it requires
neticsensorgridandthereforeoffersamuchhigheraccuracy
per user calibration. This technique supports multiuser in-
thanwhenusingadevice’sinternalmagnetometer.GaussBits
put, though touch events with different objects cannot over-
supportswireless3Dinteraction.MagGetz[16]embedsmag-
lap. Fiberio [14] a rear projected multitouch table that can
netsinphysicalcontrollerslikeswitches,buttons,slidersand
opticallysenseauser’sfingerprints. Thistechniquerelieson
joysticksandchangesinthemagneticfieldareinterpretedby
a novel fiber optic plate and unlike magnetic fingerprints, it
adevice’smagnetometer.
2
Figure2.Asinglemagneticfingerprintdoublesthenumberofdistinguishablesingleandmulti-touchevents,i.e.,wecandistinguishtwo
uniquesingletoucheventsandtwodifferentpinchgestures.
DESIGNOFMAGNETICFINGERPRINTS netembeddedinit.Ourpreliminaryexperimentsshowedthat
We developed a novel input technique, called magnetic fin- wearingamagneticring[9,18]doesnotallowforaccurately
gerprints, that is based on the following observations. For distinguishing individual fingers. A capacitive glove could
input,existingmagnetinputtechniqueshaveeitherexplored: accommodatemultiplemagnetsandisavailableatlowcost.
• Around-the-deviceinteractionwheretheusereitherwears A major consideration in the definition of magnetic finger-
the magnet as a ring [1,9,18] or holds a magnet in their printsisitsformfactor;e.g.,thesizeandtheshapeofamag-
hand[17]. netthatneedstobeattachedtotheuser’sfingernail. Amag-
• On-screen interaction either with magnet embedded in an net’s strength depends –among other factors– on its size. A
object[3,19]orstylus[15,16,20,21]. larger magnet is more easily detected, especially when the
magnet is held further away from the magnetometer. For
Ratherthanusingamagnetasastandaloneinputtechnique,
weuseamagnettoaugmentfingerbasedtouchinput. Asig- practicalandaestheticreasonsitismoredesirabletousethe
smallestpossiblesizemagnet. Achallengewithusingmag-
nificantbenefitofthisapproachisthatmagneticfingerprints
netstrengthisthatdifferentsizedmagnetsmaygeneratethe
can be used as part of a multitouch gesture, such as a pinch
same value for (cid:126)c depending on the distance of the magnet
(see Figure 2). Another benefit is that it doesn’t require the
from the magnetometer. To accurately discriminate differ-
user to learn new gestures. Though current mobile touch-
ent sized magnets, we need to incorporate where the finger
screendevicestypicallyrelyoncapacitivesensing,magnetic
touches the screen and a classifier needs to be trained using
fingerprints could in theory work with any type of touch-
measuredvaluesof(cid:126)cfordifferentmagnetsizesandlocations.
screen technology (resistive/optical) as long as it features a
magnetometer,whichexcludesinteractivetabletabletops. If we define magnetic fingerprints based on the direction of
themagneticfield,theimplementationissignificantlysimpler
Apermanentmagnetcanprovidetwotypesofinput: (1)the
aswedonotneedtocollecttrainingdataandahigheraccu-
sensedstrengthof themagnetdependson itsdistancetothe
racymaybeachievedasweonlyneedtolookatthedirection
magnetometerandthispropertycanbeusedtofacilitateana-
of change of the components of (cid:126)c. Because this approach
logformsofinput,suchasscrolling[9,17];or(2)thedirec-
doesn’t require incorporating where the finger touches the
tion of its magnetic field can be interpreted to provide dis-
screen, touch less input is one possible application of mag-
crete forms of input, i.e., selecting an item [3,9]. A mag-
netometer reports a 3D vector (cid:126)v, which –when there is no netic fingerprints (see Example Applications). A previous
magnetic interference– points towards North (cid:126)n (see Figure studyfoundthatdiscerningmagnetsbasedontheirstrengthis
verydifficultunlesstheyarereallydifferentinsize[3],which
1:left). Whenamagnetapproachesthescreen,thevectorre-
leads to form factor issues. Given the simpler implementa-
ported by the magnetometer will change to the value of the
magneticfieldprojectedbythemagnetm(cid:126). Thedirectionand tion and possible higher accuracy, we only explore defining
magnitude of m(cid:126) differs significantly from the earth’s mag- fingerprintsusingthedirectionofthemagneticfield.
netic field, i.e., m(cid:126) (cid:54)= (cid:126)n and |m(cid:126)| (cid:29) |(cid:126)n| (for strong enough Becausewecombinetouchsensingwithsensingthepresence
magnets). The strength of the magnet or the orientation of ofamagneticfingerprint,weonlyrequiren−1magneticfin-
itsprojectedfieldwillgenerateuniquechangesinthemagne- gerprintstodistinguishndifferentfingers,assensingtheab-
tometer,i.e.,(cid:126)c = (cid:126)n+m(cid:126) whenthemagnetismovedtowards senceofamagneticfingerprintinatoucheventalsodefinesa
themagnetometer. uniqueinputevent. Mobiletouchscreenscandetectuptofive
differenttypesoftouchgestures, withsingletouch, twoand
It is this property that our approach exploits by instrument-
threefingermultitouchgesturesbeingmostpractical.Amag-
ing a fingernail with a small magnet, where the strength of
netic fingerprint may be chorded with other non-magnetic
themagnetorthedirectionofthemagneticfieldcanbeused
fingers to form multi-gestures. Using a single magnetic fin-
to define unique “magnetic fingerprints” that can be distin-
guishedthroughuniquevaluesof(cid:126)c. Variousnovelinputtech- gerprint, nine unique gestures (two single touch, and seven
multitouchgestures)canbedefined(seeFigure2). Withtwo
niques [6,24] have explored instrumenting the user’s finger
magnetic fingerprints, this number theoretically increases to
withsensorsorevenanon-obtrusivemagnet[23]. Different
16uniquegestures,morethanthreetimestheamountofwhat
approaches could be used for attaching a magnet to a user’s
currenttouchscreendevicesareabletodetect. Besidesprac-
finger,suchaswitharubberband[23],thoughitseemsmost
tical issues, e.g., fingers could get stuck to each other when
practicalfortheusertowearacapacitivethimblewithamag-
3
Figure 3. Left: for our studies a finger is instrumented with a small Figure4. Left:the4x3gridusedforsingletargetselectiontask. Right:
neodymiummagnet(7.9mmdiam.) usinganadhesive. Right: alterna- coloredtargetsthatarerenderedforthemulti-touchselectiontask.
tivelytheuserwearsacapacitivethimblewithanembeddedmagnet.
Participants
magnets attract each other, another potential problem with
We recruited 10 participants (3 female, average age 27.5,
chording magnetic fingerprints could be magnetic interfer-
SD = 3.5). All subjects were right handed and none had
ence.Currentmagnetometersonlyreportthesumofthecom-
any self-reported non-correctible impairments in perception
plexmagneticfieldbetweenmultiplemagnets,whichmakes
or impairments in motor control. All subjects were familiar
itdifficulttorecognizethisgesture. Userstudiesprovidein-
withtouchscreeninput,astheyallownedasmartphone.
sight into how many fingerprints can be used and whether
chordingisfeasible.
Procedure
The strength of the magnet’s magnetic field sensed by the
STUDY1: FORMFACTORANDSCALABILITY
magnetometervariesdependingonthedistancebetweenthe
Forthisstudy,weareinterestedinunderstandinghowtheac-
magnetandthemagnetometer. Tounderstandwhatsizemag-
curacyofrecognizingmagneticfingerprintsvariesdepending
netisrequiredtoaccuratelydetectthemagneticfingerprintat
onmagnetstrengthandthenumberofmagneticfingerprints
everylocationonthescreen,wehadsubjectsperformasingle
used. A larger magnet is detected more easily but magnetic
targetselectiontask. AniOSapplicationwaswrittenthatre-
interferencemayincreasewhenusingmultiplemagnets.
quiressubjectstoperformatargetselectiontaskona4x3grid
(seeFigure4).Atargetisrenderedusinganorangecirclethat
Instrumentation
measured220by200pixels(seeFigure4left.) The(X,Y,Z)
An Apple iPhone 4, with a 1.94” x 2.91” (640 x 960 pixel)
valuesofthemagnetometerwererecordedinalogfilewitha
displaywasusedforourexperiment. Forthisstudy,welimit
samplerateof40Hz. Werecordthetimeandlocation(X,Y)
ourselvestotouchscreeninputonsmartphones,asanincrease
ofeachtouchevent.
ininputspaceoffersthelargestbenefitforthisspecificplat-
form due to their limited screen real estate. When held in We used a repeated measures within-subjects design. Inde-
the portrait position, the iPhone’s magnetometer is located pendentvariablesweremagnetsizeandnumberofmagnetic
in the upper right corner. We used N40 grade, disc-shaped fingerprints. For one magnetic fingerprint, we instrumented
Neodymiummagnets,whichisthestrongesttypeofcommer- the middle finger with a magnet (North pole up) and used
ciallyavailablepermanentmagnetatalowcost(<$0.10). A the middle and index finger for target selection. This setup
disc-shapedmagnetwaschosenasitisleastobtrusiveandbe- wasusedastheindexfingerismostfrequentlyusedformo-
causethisshapehasbeensuccessfullyusedinarelatedhaptic bile input. For two magnetic fingerprints, we instrumented
outputtechnique[23]. Thisshapeismoreeasilyattachedtoa the index (North pole up) and ring finger (South pole up)
nailthan,forexample,acubeshapedmagnet. Discmagnets and used these fingers and the middle finger for the target
are axially magnetized with poles located on the opposing selectiontask. Thissetupwaschosentominimizemagnetic
flatcircularsurfacesofthedisc. Preliminarytrialsconfirmed interference by having the largest possible distance between
that the classification accuracy of using magnet strength for them. Anadditionalconstraintthatweidentifiedusingapre-
definingmagneticfingerprintswasmuchlowerthanwhenus- liminarytrialisthatbetterresultswereachievedforwhenthe
ingthedirectionofthemagneticfield. Whenusingmultiple userretractsamagneticfinger(placeitinthepalm)whenthis
magnets, they need to differ in size significantly as to accu- fingerisnotinvolvedinthetargetselection,asthisminimizes
rately classify them, which is detrimental to the form factor falserecognitions. Thoughthesespecificconditionsoptimize
of this technique. When using the direction of the magnetic the use of our technique, it helps establish an upper bound
field,amagnetofthesamesizecanbeused.Forthesereasons onthebestperformance. Wecounter-balancedtheconditions
our experiment is limited to evaluating up to two magnetic among subjects. The phone was placed on a flat surface in
fingerprintsthataredefinedbythedirectionofthemagnetic the landscape orientation and subjects used their dominant
field(eachpolecanonlybewornfaceup). Asmobileinput handfortargetselection. Eachfingerselected24targetswith
is typically limited to single hand input, we deem a poten- two targets per cell in randomized order. Subjects did not
tiallythree-foldincreaseintouchvocabularybyexploringthe switch between fingers, as we were primarily interested in
useoftwomagneticfingerprintslargeenoughforourstudy. understanding how the accuracy varies depending on where
Basedonpreliminarytrials,wedecidedtoexplorethefollow- thescreenistouched.
ing diameters for our magnets (12.7, 7.9 and 3.2 mm with a
magnet height of 0.8mm). We marked the north pole of the Results
magnet with an X and attached the magnet to the subject’s Todetermine theaccuracywith whichmagneticfingerprints
fingernailusingareusabledotadhesive(seeFigure3:left). canbedistinguishedfromeachotherandfromnon-magnetic
4
Table1.Classifieraccuracy(standarddev) ingthreeuniquepinchgestures(thumb-index,thumb-middle,
magnet #magneticfingerprints index-middle). Threesubjectsperformedanumberofdiffer-
diameter(mm) 1 2 entpincheswheretheyhadtopinchtwotargetstogether. We
12.7 98.47(1.23) 95.71(1.68) used fixed targets that were defined around the center of the
7.9 99.02(0.95) 94.88(2.73) screen. AnSVMwastrainedusingthesamefeaturesasour
3.2 92.18(3.49) 80.74(3.68) first study, which yielded an overall classification accuracy
of 74.44% (SD=12.88) with an accuracy of 60.00% for the
pinchgestureinvolvingbothmagneticfingerprints. Basedon
touches,wetrainedaclassifier,i.e.,asupportvectormachine
thisresult,chordingmagneticfingerprintsdoesnotseemfea-
(SVM) using the LibSVM library. To avoid over-fitting our
sibleandwelimitourstudytomulti-gesturesthatuseasingle
SVM to the training data, we used ten-fold cross-validation
magneticfingerprintatatime.
usingallourdataforatotalof480datapointsforonemag-
neticfingerprintand720datapointsfortwofingerprints.The
Instrumentation
datawaslabeledandpartitionedintotenequalsizesubsam-
Basedonpreviousresults,weusedthemagnetwitha7.9mm
ples. A single subsample is retained for testing where the
diameterandweusedthesameiPhone4forourexperiment.
other nine subsamples are used as training data. The cross-
validationprocessisrepeated10timeswitheachofthesub-
Participants
samples used as the validation data, and a single estimate We recruited eight people (1 female, average age 25.8,
is achieved by averaging the results from each fold. Vari- SD=4.7) with four people having participated in our first
ous features were explored to train the SVM, with the best study. All subjects were right handed, owned a touchscreen
performanceachievedfordatapointscontaining62features, deviceandnonereportedanynon-correctableimpairment.
whichincluded: (1)touchlocation(X,Y);and(2)differential
magnetometer data (0.5 seconds) preceding the touch event, Procedure
which included 20 vectors (X,Y, Z) of (cid:126)c with (cid:126)c = m(cid:126) − v(cid:126)t Weusedthesameapplicationthatwedevelopedforourfirst
where m(cid:126) is measured at the touch event t and v(cid:126)t in 25ms experiment, but we modified it so that two targets are ren-
decrements. Parameters were determined experimentally to dered that need to be moved together using a pinch gesture
yield the best results with no filtering applied. Differential (seeFigure4:right).Forthesamereasonasourfirststudy,we
magnetometer values are used as raw magnetometer values leave the index finger free and instrument the middle finger
vary depending on how the phone is oriented with regard to withamagnettoallowfordefiningtwouniquepinchgestures
theearth’smagneticfield,whichcouldvarybetweentrials. (seeFigure2). Whatpinchgesturetoprovideisindicatedus-
ingvisualcues,e.g.,abluetargetindicatesusingamagnetic
Table 1 lists the classification accuracy for each condition –
fingerprint where an orange target a non-magnetized finger.
includingtheirstandarddeviation–basedontheclassification
Tenuniquecombinationsoftargetsweredefinedwithonecell
accuracy of each fold. We performed a two-way ANOVA
distance between the targets. These combinations consisted
and found a significant interaction for size and number of
of six horizontal pinches and four diagonal pinches. To es-
magneticfingerprintsregardingclassifieraccuracy (F =
2,54
tablishanupperboundontheperformanceofthissetup,sub-
16.926, p = .00, partial η2 = .385). Post-hoc analysis re-
jectswereaskedtoretracttheirmagneticfingerwhenitwas
vealedthat3.2mmmagnetdiffersinclassifieraccuracyfrom
not involved in the target selection to minimize false recog-
the other sizes (p = .00) with no difference between the
nitions. Subjectsusedtheirdominanthandforgestureinput.
7.9mm and 12.7mm magnets (p = 0.99). A single magnet
Foreachtypeofpinch,subjectsselected20targetsforatotal
yields the best performance (p = .00) with an accuracy of
of40targetswiththeorderoftargetsandtypeofgesturebe-
99.02% (SD=.95) for the 7.9 mm magnet. All errors were
ingrandomized.Wealsorecordthenumberofattemptsmade
duetonotcorrectlyrecognizingthemagneticfingerprintwith
foreachcorrectpinchgesture.
71%oftheerrorsmadebyasinglesubject. Fortwomagnets,
thebestperformanceis95.71%(SD=1.68)usingthe12.7mm
Results
magnet,witherrorsuniformlydistributedamongsubjectsand
We employed an SVM using ten-fold cross-validation for a
typesoffingers. Errorsweremorelikelytooccurincellsfar-
total of 160 data points per type of pinch gesture. Using
thestawayfromthemagnetometer.
64 features, e.g., 4 for the touch locations and 60 for dif-
ferential magnetometer data, we achieved an overall classi-
STUDY2: MULTI-TOUCHGESTURERECOGNITION fication accuracy of 97.19% (SD=3.11) with an accuracy of
Where our first study focused on single finger target selec- 96.88% (SD=4.15) for the non-magnetic pinch and 97.50%
tion, our second study evaluates the classification accuracy (SD=3.23) for the magnetic pinch, with no significant dif-
ofmulti-touchgestures. Wefocusontwo-fingergesturesand ference between them (t = .172, p = .693). An anal-
18
requiresubjectstoswitchbetweendifferentfingers,whichre- ysis of errors found a relatively uniform distribution of er-
semblesamorepracticalusageofourtechniquethatwedid rors among subjects, with slightly higher error rate for the
not evaluate in our first study. Using a preliminary experi- cell farthest away from the magnetometer. There was no
ment,weanalyzedwhetheritwasfeasibletochordtwomag- significantdifferenceinclassificationaccuracywithourfirst
netic fingerprints. The thumb and index finger were instru- study(t =.401,p=.093),whichindicatesthatswitching
18
mentedwiththe12.7mmmagnet,whichyieldedthebestper- between fingers for input does not negatively affect perfor-
formanceinourfirststudy. Twofingerprintsallowfordefin- mance. Usersrequiredanaverageof1.12attempts(SD=.18)
5
Table2.Classifieraccuracy(standarddev) theresults. Forthe8x6grid,aone-wayANOVAfoundasta-
magnet gridsize tisticallysignificantdifferenceinaccuracybetweenmagnets
height(mm) 8x6 61/2x5 4x3 (F = 7.108, p < .05) where a post-hoc analysis showed
4,7
0.8 79.40(4.99) 81.98(5.50) 93.68(5.45) thatthe2.4mmmagnetoutperformsthe1.6mm(p = .049)
1.6 81.64(4.76) 81.33(2.77) 93.75(5.25) and the .8 mm (p = .00) magnets. An accuracy of 86.85%
2.4 86.85(3.70) 88.11(4.08) 98.43(2.59) (SD=3.70)isachievedforthe9.7”iPadwithaslightlyhigher
accuracy (88.11%) for 8” tablets (iPad Mini). Errors were
uniformlydistributedamongsubjectsandamongtypesoffin-
foreachtypeofpinchwithnosignificantdifferenceinnum-
gers. Similar to our prior results, errors were more likely to
berofattemptsbetweenbothpinches(t = .00,p = 1.00).
7 occurincellsfarthestawayfromthemagnetometer;however,
Besides demonstrating the feasibility of using a single mag-
wealsoobservedfalserecognitionsinthecellsclosesttothe
netic fingerprint to augment multitouch input we found that
magnetometer. Though we ensured subjects retracted their
usingthemiddlefingerforapinchgestureisjustaseffective
magnetized finger, it seems due to the larger screen and the
asusingtheindexfinger.
size of the magnet used, false recognitions become difficult
toavoid. Thisalsoexplainstheloweraccuracyfoundforthe
STUDY3: TOUCHSCREENSIZE .8mmmagnetforthe4x3grid.
Tablets have become a popular mobile computing platform
but they typically feature larger touchscreens than smart-
EXAMPLEAPPLICATIONS
phones. For this study, we evaluate whether magnetic fin-
We illustrate the usefulness of magnetic fingerprints using a
gerprints can be used on tablets. Due to their larger screen
numberofexampleapplications.
size, stronger and larger magnets need to be used. We limit
ourexperimenttoevaluatingasinglemagneticfingerprint.
ModeSwitching
Instrumentation
Forthisexperiment,weusedthepopularAppleiPad4tablet,
whichfeaturesa9.50”x7.31”(2,048x1,536pixel)display
(9.7” diagonal). We were unable to verify whether the iPad
featuresthesamemagnetometerastheiPhone. Weusedthe
largestdiametermagnetfromourfirstexperiment(12.7mm).
As the 12.7 mm magnet is already at the size of a nail, we Figure5.Left:regularcharactersareprovidedwiththeindexfingerand
explore magnets with larger height. Based on experiments, specialcharacterswithamagneticfingerprint. Middle: Beforetouch-
weevaluatedthefollowingheights(0.8,1.6and2.4mm). ingthescreen,wecansensethepresenceofamagneticfingerprintand
switchtothenumericalkeyboard.Right:aspecialcharacterisselected.
Participants
Providingspecialcharactersonmobilevirtualkeyboardsof-
We recruited eight subjects (1 female, average age 27.0,
tenrequiresswitchingtoaspecialkeyboard,whichisineffi-
SD=4.6). Foursubjectshadparticipatedinpriorstudies. All
cientasitrequiresmultipleuserinputstoselecttherightkey.
subjects were right handed, owned a smartphone and none
To address this problem, Harrison et al. presents a solution
reportedanynon-correctableimpairment.
usingtheirTapsensetechnique[9]inwhichspecialcharacters
canbeprovidedusingdifferentpartsofthefinger(pad,nail).
Procedure Thisapproachintegratesmultiplekeyboardsintoasinglekey-
Subjects performed the same target selection task as in our boardandrendersmultiplecharactersoneachkey,whichmay
first study. Instead of using a 4x3 grid, we use an 8x6 grid causekeystobecomeillegible. Magneticfingerprintscanbe
toallow fora comparisonofresults usinga similarcellsize used for efficient mode switching. For a mobile keyboard,
(accommodatedfordifferencesinscreenresolution).Though userstyperegularcharacterswiththeirindexfingerandusea
theiPhone4andiPad4haveslightlydifferentaspectratios, magneticfingerprintontheirmiddlefingertoprovideaspe-
we did not deem this to be a significant problem. For a sin- cial character. What is novel about this approach is that we
gle magnetic fingerprint, we instrumented the middle finger can already sense when a magnetized finger is approaching
withamagnetandusedthemiddleandindexfingerfortarget beforeittouchesthescreenandswitchtothecorresponding
selection. Each finger selects 48 targets in the grid in ran- keyboard. This allows users to hover with their magnetized
domized order for a total of 96 targets per magnet size. We fingeroverthekeystofindthespecialcharacterbeforeselect-
counter-balancedtheconditionsamongsubjects. ingit. Ourapproachcouldsupportthreedifferentkeyboards
whileallowingforlegiblecharactersonkeys.
Results
AnSVMwastrainedusingthesamefeaturesasforourfirst RotaryGestures
studyforatotalof384datapointspertypeoffinger. Toun- Knobs or sliders are GUI elements that can be used for ad-
derstandtheeffectthattouchscreensizehasonclassification justingcontinuousinputvalues,suchasvolumeorpitchina
accuracy, we trained SVMs using subsets of our grid. We musicapplication. Itismoredesirabletouseaknobformo-
usedagridof61/2x5(iPadmini)and4x3(iPhone4)toallow biletouchscreeninput,astheyrequirelessscreenrealestate
foracomparisonwithresultsfromthefirststudy.Table2lists thanaslider[8].Knobsaredifficulttomanipulateusingtouch
6
Figure6. Left: aknobistouchedwithamagneticfingerprint. Right:
Whenthefingerrotates,changesinthemagneticfieldadjusttheknob.
Figure8.Byremovingthespacebarandusingamagneticfingerprintto
providespaces,virtualkeyboardscouldbemade25%smaller.
as this cannot be achieved with a single touch event and re-
quiresinterpretingatwofingerrotate. Twofingerrotatesare
difficulttoperformwithasinglehandandrequiresincreasing used key but nearly takes up an entire row. We developed
thesizeoftheknobtoallowforbimanualaccess. Bianchiet a simple text editor application that uses a smaller custom
al.[3]demonstratesprovidingrotaryinputwithamagnetem- keyboard with no spacebar and only three rows of keys us-
beddedinanobject. Theobjectisplacedonthetouchscreen ing a minor rearrangement of keys. With a magnetic finger-
andwhenitisrotatedchangesinthedirectionofitsmagnetic print attached to the thumb users can activate the space bar
field are interpreted to adjust a value. A magnet attached to by briefly tapping the screen or even using a wireless flick
auser’sfingercanbeinterpretedinthesamewayandallows gesture. Backspace could be activated using a magnetic fin-
usersfortopreciselymanipulateaknobbyrotatingtheirfin- gerprintontheotherthumb.
ger instead of an object or using two fingers. Experiments
showafeasibleinputrangeofnearly180◦.Thisgesturecould DISCUSSIONANDFUTUREWORK
alsoexploitthemetaphororscrew/unscrewgestureswherea Scalability. Ourexperimentsrevealthatourtechniqueislim-
value,e.g.,asliderbarhandle,canbefixedtemporarily. itedtousinguptotwomagneticfingerprintsatthesametime
Multi2-TouchGestures but magnetic fingerprints cannot be chorded. This limita-
tion is due to magnetic interference, as current magnetome-
ters are unable to resolve the complex magnetic fields that
appear between using multiple magnets. Despite this limi-
tation magnetic fingerprint significantly increases touch vo-
cabulary. Without chording up to 13 unique (multi) touch
eventscanbedefinedfortwomagneticfingerprintsandupto
9unique(multi)toucheventsforusingonemagneticfinger-
print,whichnearlydoubling(one)ortriplesl(two)thecurrent
Figure7.Left:anon-magnetizedpinchscalesaphoto.Right:apinch-in touchvocabularyofmobiletouchdevices.
withamagneticfingerprintpicksupthephotowhereitcanbedragged
toanewlocationandapinch-outdropsthephoto. Comparison. Wecompareourresultstoanumberofclosely
Multi-touch gestures, such as pinches, are typically associ- related approaches. A precise comparison is difficult due to
ated with a single type of functionality, such as zooming. differences in the increase in touch vocabulary and experi-
In real life, a pinch gesture could be used for different ac- mentalconditions. Ourapproachoffersaslightlysmallerin-
tions, e.g., pickingupanobjectorsqueezinganobject. Our crease in input space as Microrolls [22] (16 micro gestures,
study did not find a significant difference in proficiency be- 95% accuracy) but offers a slightly higher accuracy. Thum-
tween making pinches with the index or middle finger. We brock [4] offers a similar accuracy (96%) but only adds a
created a simple mobile photo manipulation application that single thumb micro gesture. In general, microgestures are a
usesMulti2-touchgestures,e.g.,multi-touchgesturesthataf- slowermethodofinputthanourapproach[11]. Unlikemicro
forddifferenttypesofnaturalinteractionsbyusingdifferent gestures, our technique does require a small amount of user
fingers in the gesture. A pinch gesture made with the index instrumentation. Tapsense[10]achievesanaccuracyof95%
andthumbisusedforscalingaphotobutapinch-ingesture and can distinguish four different types of finger touches.
madewithamagneticfingerprintonthemiddlefingerpicks Gripsense [7] identifies three levels of finger pressure with
up the photo, upon which the user can drag the photo to a 95%accuracy. Weachieveanaccuracyof96%forthreedif-
new location and drop the photo with a pinch-out. Though ferenttypesoffingers,butourtechniquecanbeusedtoaug-
thisgesturemaybelessefficientthanusingasinglefingerit mentmultitouchinput(nochording). Tapsensealsorequires
doesn’t require mode switching and allows for two different anexternalacousticsensorandperusercalibration. Otherre-
pinchgesturesthatbothresemblenaturalinteractions. latedapproaches[2,5,12]donotreportaccuracy. Lookingat
otherfactors,Magneticfingerprintsdonotrequireanyexter-
TouchscreenTyping
nalsensorsoranyformofusercalibrationanditislow-cost
Virtual keyboards on tablet devices (in landscape mode)
withneodymiummagnetsretailingforlessthan$0.10.
nearly take up half of the screen; which leaves little screen
realestateavailableforapplicationsthatrelyontyping,such Limitations. Finger instrumentation is used in a number of
as word processors. The spacebar is the most frequently novelinputtechniques[6,23,24]. Ourapproachusesexisting
7
sensors and requires the smallest amount of user instrumen- 4. Bonnet,D.,Appert,C.,andBeaudouin-Lafon,M.Extending
tation (an untethered small magnet) which seems like a rea- thevocabularyoftoucheventswithThumbRock.InProc.of
GI’13,221-228.
sonableusabilitytradeofftoachieveasignificantincreasein
inputspacewithahighaccuracy.Acurrentlimitationofmag- 5. Boring,S.,Ledo,D.,Chen,X.A.,Marquardt,N.,Tang,A.,
andGreenberg,S.Thefatthumb:usingthethumb’scontact
neticfingerprintsisthatfalserecognitionsmayoccurwhena
sizeforsingle-handedmobileinteraction.InProc.of
userdoesnotfullyretracttheirmagnetizedfingerifthatfinger
MobileHCI’12,207–208.
isnotinvolvedinthegestureprovision. Wedidnotobserve
6. Chan,L.,andLiang,R-H.,andTsai,M-C.,andCheng,K-Y.,
thisinterferenceinourfirststudyforusingasinglemagnetic
andSu,C.H.FingerPad:PrivateandSubtleInteractionUsing
fingerprint,butthisproblemmanifesteditselfonlywhenus- Fingertips.InProc.ofUIST’13,255–260.
ing two and larger sized magnets on the tablet sized touch-
7. Goel,M.,andWobbrock,J.,andPatel,S.GripSense:using
screen, as due to the larger screen users had to reach over built-insensorstodetecthandpostureandpressureon
the magnetometer to provide input. We observed that some commoditymobilephones.InProc.ofUIST’12,545–554.
usersalreadyretractthosefingersnotinvolvedininputespe- 8. Gelineck,S.,andSerafin,S.quant.eval.ofthedifferences
ciallywhenusingsmartphonebaseddevices,sothisseemsa betweenknobsandsliders.Proc.ofNIME’09,13–18.
feasible requirement. Individual classifiers could be used to 9. Harrison,C.,andHudson,S.E.Abracadabra:wireless,
mitigatefalserecognitions,whichmaydependonthesizeof high-precision,andunpoweredfingerinputforverysmall
theuser’shandorthepositionoftheirfingers,butthiswould mobiledevices.InProc.ofUIST’09,121–124.
limitthegeneralapplicabilityofourtechnique. Tokeepcon- 10. Harrison,C.,Schwarz,J.,andHudson,S.E.Tapsense:
ditions balanced between trials, our experiments were per- enhancingfingerinteractionontouchsurfaces.InProc.of
UIST’11,627–636.
formedwiththedevicelyingflatonthetable. Inpreliminary
trials, we did not observe a difference in performance when 11. Hinckley,K.,Baudisch,P.,Ramos,G.,andGuimbretiere,F.
Designandanalysisofdelimitersforselection-actionpen
holdingthedeviceinthehand.
gesturephrasesinscriboli.InProc.ofCHI’05,451–460.
Futureresearch.Acubeshapedmagnetmayallowsfordefin- 12. Hinckley,K.,andSong,H.Sensorsynaesthesia:touchin
ing six unique magnetic fingerprints based on the available motion,andmotionintouch.InProc.ofCHI’11,801–810.
orientations of the magnet. It may be challenging however 13. Holz,C.,andBaudisch,P.Thegeneralizedperceivedinput
to fit this magnet on a finger and it may be harder to clas- pointmodelandhowtodoubletouchaccuracybyextracting
fingerprints.InProc.ofCHI’10,581–590.
sify magnetic fingerprints. To optimize form factor we will
explore the use of magnetic nail polish though this may not 14. Holz,C.,andBaudisch,P.Fiberio:Atouchscreenthatsenses
fingerprints.InProc.ofUIST’13,41–50.
generatedastrongenoughmagneticfield. Ourstudydemon-
strated that using larger magnets allows our technique to be 15. Hwang,S.andBianchi,A.,andAhn,M.,andWohn,K.
MagPen:magneticallydrivenpeninteractionsonandaround
used on tablets, though false recognitions may increase. To
conventionalsmartphones.InProc.ofMobileHCI’13,
circumventthisproblem, wewillexplorebimanualusewith
412–415.
thehandclosesttothemagnetometerequippedwithasmaller
16. Hwang,S.,andAhn,M.,andWohn,K.MagGetz:
magnetandtheotherhandwithlargermagnet.
User-ConfigurableMagneticTangibleControllersOnand
AroundMobileDevices.InProc.ofUIST’13,411-416.
CONCLUSION 17. Ketabdar,H.,Roshandel,M.,andYu¨ksel,K.A.Magiwrite:
Thispaperpresentsmagneticfingerprintsamobileinputtech- towardstouchlessdigitentryusing3dspacearoundmobile
nique that instruments the user’s finger with a small non- devices.InProc.ofMobileHCI’10,443–446.
obtrusivemagnet.Byincorporatingsensinginformationfrom 18. Ketabdar,H.,Yu¨ksel,K.A.,andRoshandel,M.Magitact:
the device’s magnetometer, touches made with a magnetic interactionwithmobiledevicesbasedoncompass(magnetic)
fingerprintcanbedistinguishedfromanon-instrumentedfin- sensor.InProc.ofIUI’10,413–414.
ger. Unlikeexistingmagneticinputtechnique,magneticfin- 19. Liang,R.-H.,Cheng,K.-Y.,Chan,L.,Peng,C.-X.,Chen,
gerprints take advantage of the rich vocabulary offered by M.Y.,Liang,R.-H.,Yang,D.-N.,andChen,B.-Y.Gaussbits:
magnetictangiblebitsforportableandocclusion-free
multitouchinput. Magneticfingerprintsarelow-cost,require
near-surfaceinteractions.InProc.ofCHI’13,1391–1400.
no user calibration and can implemented on current touch-
20. Liang,R.-H.,Cheng,K.-Y.,Su,C.-H.,Weng,C.-T.,Chen,
screen devices. User studies investigate the accuracy, scala-
B.-Y.,andYang,D.-N.Gausssense:attachablestylussensing
bilityandlimitationsofthistechnique,whichfoundthatmag- usingmagneticsensorgrid.InProc.ofUIST’12,319–326.
netic fingerprints cannot be chorded. Useful applications of
21. Liang,R.-H.,Su,C.-H.,Weng,C.-T.,Cheng,K.-Y.,Chen,
magneticfingerprintsarepresented. B.-Y.,andYang,D.-N.Fluxsketch:sketchingusingmagnetic
stylus.InProc.ofSIGGRAPHAsia2012Emerging
REFERENCES Technologies,11:1–11:2.
1. Ashbrook,D.andBaudisch,P.,andWhite,S.Nenya:subtle 22. Roudaut,A.,Lecolinet,E.,andGuiard,Y.Microrolls:
andeyes-freemobileinputwithamagnetically-trackedfinger expandingtouch-screeninputvocabularybydistinguishing
ring.InProcofCHI’11,2043-2046. rollsvs.slidesofthethumb.InProc.ofCHI’09,927–936.
2. Benko,H.,Wilson,A.D.,andBaudisch,P.Preciseselection 23. Weiss,M.,Wacharamanotham,C.,Voelker,S.,andBorchers,J.
techniquesformulti-touchscreens.InProc.ofCHI’06, Fingerflux:near-surfacehapticfeedbackontabletops.InProc.
1263–1272. ofUIST’11,615–620.
3. Bianchi,A.,andOakley,I.Designingtangiblemagnetic 24. Yang,X-D,andGrossman,T.,andWigdor,D.,and
appcessories.InProc.ofTEI’13,255–258. Fitzmaurice,G.MagicFinger:Always-availableInput
ThroughFingerInstrumentation.InProc.ofUIST’12,
147–156.
8