Drawing on the World: Sketch in Context Andrew Correa

111 Pages·2014·3.15 MB·English

Drawing on the World: Sketch in Context
by
Andrew Correa
Submitted to the Department of Electrical Engineering and
Computer Science
in partial fulfillment of the requirements for the degree of
Master of Science
at the
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
September 2009
(cid:13)
c Massachusetts Institute of Technology 2009. All rights reserved.
Author.............................................................
Department of Electrical Engineering and Computer Science
September 04, 2009
Certified by........................................................
Randy Davis
Professor of Computer Science and Electrical Engineering
Thesis Supervisor
Accepted by .......................................................
Terry P. Orlando
Chair, Department Committee on Graduate Students

Drawing on the World: Sketch in Context
by
Andrew Correa
SubmittedtotheDepartmentofElectricalEngineeringandComputerScience
onSeptember04,2009,inpartialfulfillmentofthe
requirementsforthedegreeof
MasterofScience
Abstract
This thesis introduces the idea that combining sketch recognition with contextual
data—informationaboutwhatisbeingdrawnon—canimprovetherecognitionof
meaninginsketchandenrichtheuserinteractionexperience.
I created a language called StepStool that facilitates the description of the re-
lationship between digital ink and contextual data, and wrote the corresponding
interpreter that enables my system to distinguish between gestural commands is-
suedtoanautonomousforklift. Auserstudywasdonetocomparethecorrectness
ofasketchinterfacewithandwithoutcontextonthecanvas.
Thisthesiscoinsthephrase“DrawingontheWorld”tomeancontextualsketch
reconition, describes the implementation and methodology behind “Drawing on
theWorld”,describestheforklift’sinterface,anddiscussesotherpossibleusesfora
contextualgesturerecognizer. Samplecodeisprovidedthatdescribesthespecifics
of the StepStool engine’s implementation and the implementation of the forklift’s
interface.
ThesisSupervisor: RandyDavis
Title: ProfessorofComputerScienceandElectricalEngineering
3
4
Acknowledgments
I would like to thank Professor Randy Davis for his insight and feedback. I thank
SethTellerforhissupport. IthankthemembersoftheMultimodalUnderstanding
group and the Agile Robotics group for their feedback and support. I thank my
parents and Ashley Murray for their encouragement, without which I would not
havehadthestrengthtocarrythisworkthroughtocompletion.
Also, I thank Ashley Murray, Prof. Randy Davis, and Matthew Walter, for
proofreading my thesis and I thank Ali Mohammad for his help. They deserve all
thecreditforanyerrors.
5
6
Contents
1 Introduction 13
1.1 SketchOntology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.2 TaskDomain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2.1 Pallets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.2.2 SupplySupportActivities . . . . . . . . . . . . . . . . . . . . . 17
1.3 ForkliftWorldModel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2 DrawingontheWorld: AContextualGestureRecognizer 21
2.1 VocabularyofGestures . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2 ResolvingAmbiguity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.1 CircleAmbiguity . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2.2 DotAmbiguity . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.3 XAmbiguity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.3 GroundingGestures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4 ExampleUseCase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3 StepStool 35
3.1 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.1.1 ScenicItemDescriptionGrammar . . . . . . . . . . . . . . . . 37
3.1.2 ShapeDescriptionGrammar . . . . . . . . . . . . . . . . . . . 38
3.1.3 GestureDescriptionGrammar . . . . . . . . . . . . . . . . . . 38
3.2 SyntaxandSemantics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3.2.1 DescriptionBoundaries . . . . . . . . . . . . . . . . . . . . . . 40
7
3.2.2 ShapeandScenicItemAttributes . . . . . . . . . . . . . . . . . 40
3.2.3 GestureConditions . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.4 Constants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.2.5 OtherGestureStatements . . . . . . . . . . . . . . . . . . . . . 41
3.2.6 GestureModifiers . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.3 ForkliftStepStoolFileExample . . . . . . . . . . . . . . . . . . . . . . 43
3.3.1 ScenicItemDescriptions . . . . . . . . . . . . . . . . . . . . . . 43
3.3.2 ShapeDescriptions . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3.3 GestureDescriptions . . . . . . . . . . . . . . . . . . . . . . . . 48
4 SystemArchitecture 57
4.1 StepStoolEngineImplementation . . . . . . . . . . . . . . . . . . . . . 57
4.1.1 TheFileInputModule . . . . . . . . . . . . . . . . . . . . . . . 58
4.1.2 TheSceneBuilder . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.1.3 TheEvaluationModule . . . . . . . . . . . . . . . . . . . . . . 59
4.2 StepStoolEngineAPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2.1 FileInput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.2.2 SceneBuilding . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.2.3 ClassificationInvocation . . . . . . . . . . . . . . . . . . . . . . 61
4.3 AnExampleApplication . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.1 ControlFlow . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
4.3.2 LoadingtheStepStoolDescriptionFile . . . . . . . . . . . . . 63
4.3.3 HandlingtheWorldModel . . . . . . . . . . . . . . . . . . . . 64
4.3.4 HandlingStrokeRecognition . . . . . . . . . . . . . . . . . . . 64
5 UserStudy 67
5.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.1.1 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.1.2 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.1.3 Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2 AccuracyMeasurements . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8
5.2.1 FunctionalAccuracy . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.2 TheoreticalAccuracy . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
6 FutureDirections 77
6.1 StepStoolImprovements . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.1.1 MoreBaseShapes . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.1.2 AdditionalFunctionality . . . . . . . . . . . . . . . . . . . . . . 78
6.1.3 LiftingImplementationLimitations . . . . . . . . . . . . . . . 79
6.2 OtherRoboticDomains . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
6.3 ImprovedUsabilityofStepStoolEngine . . . . . . . . . . . . . . . . . 80
6.3.1 OpenGL-likeInterface . . . . . . . . . . . . . . . . . . . . . . . 80
6.3.2 SQL-likeInterface . . . . . . . . . . . . . . . . . . . . . . . . . . 80
7 Conclusion 83
A TheStepStoolFileUsedintheForkliftProject 85
B TheForkliftProject’sCInterfacetotheStepStoolFile 93
C DefinitionofTerms 107
10