DTIC ADA532600: A Hierarchical and Contextual Model for Learning and Recognizing Highly Variant Visual Categories

5.1 MB·English

by  Defense Technical Information Center

UNIVERSITY OF CALIFORNIA
LosAngeles
A Hierarchical and Contextual Model for Learning and
Recognizing Highly Variant Visual Categories
Adissertationsubmittedinpartialsatisfaction
oftherequirementsforthedegree
DoctorofPhilosophyinStatistics
by
Jacob Matthew Porway
2010
Report Documentation Page Form Approved
OMB No. 0704-0188
Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington
VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it

does not display a currently valid OMB control number.
1. REPORT DATE 3. DATES COVERED
2010 2. REPORT TYPE 00-00-2010 to 00-00-2010
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER
A Hierarchical and Contextual Model for Learning and Recognizing
5b. GRANT NUMBER
Highly Variant Visual Categories
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
University of California, Los Angeles,Los Angeles,CA,90095 REPORT NUMBER
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION/AVAILABILITY STATEMENT
Approved for public release; distribution unlimited
13. SUPPLEMENTARY NOTES
14. ABSTRACT
In this dissertation we present a hierarchical and contextual model for representing image patterns
(manmade objects and aerial images) that are highly variant from instance to instance. These types of
patterns are difficult to model because objects within the same class may have very different photometric
and geometric properties and/or compositions of parts, e.g. teapots may have very different colors, shapes,
and locations of their spouts and handles. We hypothesize that these varied visual patterns can be captured
by using a novel representation that arranges common primitives of the patterns in a probabilistic
hierarchy, thus compactly capturing possible compositional variations, and then enforces contextual
constraints on the appearances of the parts, thus modeling the conditional photometric and geometric
relationships of the object parts. We combine a Stochastic Context Free Grammar (SCFG), which captures
the long-range compositional variations of a pattern, with a Markov Random Field (MRF), which captures
the short-range constraints between neighboring pattern primitives, to create our model. We also present a
minimax entropy framework for automatically learning which contextual constraints are most relevant for
modeling a type of pattern and estimating their parameters. Finally, we present a novel Markov Chain
Monte Carlo (MCMC) algorithm called Clustering Cooperative and Competitive Constraints (C4 ) for
efficiently performing Bayesian inference with our model. C4 is a method for minimizing energy functions
defined on graphs that we will use to combine bottom-up and top-down information to find the best
interpretation of an image. We show experiments on learning models of a number of manmade object
categories and of aerial images and demonstrate that our algorithms automatically learn models that
accurately capture the statistical nature of the patterns we are modeling. We also show that our model can
be used for inference in new images, allowing it to identify objects in challenging scenarios.
15. SUBJECT TERMS
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF
ABSTRACT OF PAGES RESPONSIBLE PERSON
a. REPORT b. ABSTRACT c. THIS PAGE Same as 169
unclassified unclassified unclassified Report (SAR)
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std Z39-18
(cid:13)c Copyrightby
JacobMatthewPorway
2010
ThedissertationofJacobMatthewPorwayisapproved.
YingNianWu
AlanYuille
DemetriTerzopoulos
SongChunZhu,CommitteeChair
UniversityofCalifornia,LosAngeles
2010
ii
Tomyparents,
whotaughtmenevertopassupanopportunitytolearnsomething.
iii
TABLE OF CONTENTS
1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 OurApproachandTechnicalContributions . . . . . . . . . . . . . . 4
1.3 OverviewoftheDissertation . . . . . . . . . . . . . . . . . . . . . . 6
2 RelatedWork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1 Appearance-BasedMethods . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Geometric-BasedMethods . . . . . . . . . . . . . . . . . . . . . . . 10
2.3 Grammar/Compositional-BasedMethods . . . . . . . . . . . . . . . 12
2.4 Domain-specificMethods . . . . . . . . . . . . . . . . . . . . . . . 17
3 TheHierarchicalContextualModel . . . . . . . . . . . . . . . . . . . . 19
3.1 StochasticContextFreeGrammars . . . . . . . . . . . . . . . . . . 19
3.2 MarkovRandomFields . . . . . . . . . . . . . . . . . . . . . . . . 23
3.3 CreatingaContextualHierarchy . . . . . . . . . . . . . . . . . . . . 25
3.4 MathematicalFormulationfortheContextualHierarchy . . . . . . . 30
3.5 ApplicationtoObjectModeling . . . . . . . . . . . . . . . . . . . . 33
3.6 ApplicationtoAerialImageModeling . . . . . . . . . . . . . . . . . 35
4 LearningviaMinimaxEntropy . . . . . . . . . . . . . . . . . . . . . . 39
4.1 MaximumLikelihoodEstimation . . . . . . . . . . . . . . . . . . . 39
4.2 Learning(λ(α),λ(β)) . . . . . . . . . . . . . . . . . . . . . . . . . . 40
iv
4.3 RelationshipPursuit . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.4 SummaryofParameterLearningandRelationshipPursuitAlgorithms 47
5 ExperimentsonLearningandSampling . . . . . . . . . . . . . . . . . 50
5.1 ExperimentsonObjectLearning . . . . . . . . . . . . . . . . . . . . 50
5.1.1 DataCollection . . . . . . . . . . . . . . . . . . . . . . . . 50
5.1.2 ObjectRepresentation . . . . . . . . . . . . . . . . . . . . . 51
5.1.3 RelationshipFunctions . . . . . . . . . . . . . . . . . . . . 54
5.1.4 ParseGraphConstruction . . . . . . . . . . . . . . . . . . . 54
5.1.5 Learningtheλ()Parameters . . . . . . . . . . . . . . . . . 55
5.1.6 RelationshipPursuit . . . . . . . . . . . . . . . . . . . . . . 57
5.1.7 AnalysisbySynthesis . . . . . . . . . . . . . . . . . . . . . 58
5.1.8 SmallSampleSetGeneralization . . . . . . . . . . . . . . . 59
5.2 LearningResultsforAerialImages . . . . . . . . . . . . . . . . . . 61
5.2.1 DataCollection . . . . . . . . . . . . . . . . . . . . . . . . 62
5.2.2 ObjectRepresentation . . . . . . . . . . . . . . . . . . . . . 62
5.2.3 RelationshipFunctions . . . . . . . . . . . . . . . . . . . . 63
5.2.4 DeterministicallyFormingParseGraphs . . . . . . . . . . . 64
5.2.5 Learningtheλ()Parameters . . . . . . . . . . . . . . . . . . 67
5.2.6 AnalysisbySynthesis . . . . . . . . . . . . . . . . . . . . . 68
5.3 ConclusionsonLearningExperiments . . . . . . . . . . . . . . . . . 69
6 InferenceWithTheC4 Algorithm . . . . . . . . . . . . . . . . . . . . 71
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
v
6.1.1 MotivationsandObjective . . . . . . . . . . . . . . . . . . . 71
6.1.2 RelatedWorkintheLiterature . . . . . . . . . . . . . . . . . 74
6.1.3 OverviewofthemajorconceptsofC4 . . . . . . . . . . . . 76
6.2 Graphs,CouplingandClustering . . . . . . . . . . . . . . . . . . . 80
6.2.1 AdjacencyandCandidacyGraphs . . . . . . . . . . . . . . . 80
6.2.2 PositiveandNegativeEdges . . . . . . . . . . . . . . . . . . 82
6.2.3 TheNeckerCubeExample . . . . . . . . . . . . . . . . . . 83
6.2.4 EdgeProbabilityforClustering . . . . . . . . . . . . . . . . 86
6.3 C4 algorithmonflatgraphs . . . . . . . . . . . . . . . . . . . . . . 89
6.3.1 Outlineofthealgorithm . . . . . . . . . . . . . . . . . . . . 89
6.3.2 CalculatingtheAcceptanceProbability . . . . . . . . . . . . 91
6.3.3 Specialcase: Pottsmodelwith+/-edges . . . . . . . . . . . 93
6.4 ExperimentsonFlatGraphs . . . . . . . . . . . . . . . . . . . . . . 95
6.4.1 CheckerboardIsingModel . . . . . . . . . . . . . . . . . . . 95
6.4.2 CheckerboardPottsModelwith7Labels . . . . . . . . . . . 97
6.4.3 LineDrawingInterpretation . . . . . . . . . . . . . . . . . . 99
6.4.4 LabelingMan-madeStructuresonCRFs . . . . . . . . . . . 100
6.5 C4 onHierarchicalGraphs . . . . . . . . . . . . . . . . . . . . . . 102
6.5.1 ConditionforGraphConsistency . . . . . . . . . . . . . . . 102
6.5.2 FormulationofHierarchicalC4 . . . . . . . . . . . . . . . . 104
6.5.3 ExperimentsonHierarchicalC4 . . . . . . . . . . . . . . . 106
6.6 ConclusionsonC4 . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
vi
7 ExperimentsonInference . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.1 ObjectInference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
7.2 AerialImageInference . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.2.1 Bottom-UpDetections . . . . . . . . . . . . . . . . . . . . . 114
7.2.2 Top-DownPredictionofMissingObjects . . . . . . . . . . . 120
7.2.3 ExperimentalResults . . . . . . . . . . . . . . . . . . . . . 121
7.3 ConclusionsonInferenceExperiments . . . . . . . . . . . . . . . . . 130
8 DiscussionandFutureWork . . . . . . . . . . . . . . . . . . . . . . . 133
8.1 SummaryofMajorContributions . . . . . . . . . . . . . . . . . . . . 135
vii