Pattern theory : from representation to inference

609 Pages·2009·8.599 MB·English

by  Ulf Grenander, Michael I. Miller

PATTERNTHEORY
This page intentionally left blank
PATTERN THEORY: FROM
REPRESENTATION
TO INFERENCE
UlfGrenanderandMichaelI.Miller
1
3
GreatClarendonStreet,OxfordOX26DP
OxfordUniversityPressisadepartmentoftheUniversityofOxford.
ItfurtherstheUniversity’sobjectiveofexcellenceinresearch,scholarship,
andeducationbypublishingworldwidein
Oxford NewYork
Auckland CapeTown DaresSalaam HongKong Karachi
KualaLumpur Madrid Melbourne MexicoCity Nairobi
NewDelhi Shanghai Taipei Toronto
Withofficesin
Argentina Austria Brazil Chile CzechRepublic France Greece
Guatemala Hungary Italy Japan Poland Portugal Singapore
SouthKorea Switzerland Thailand Turkey Ukraine Vietnam
OxfordisaregisteredtrademarkofOxfordUniversityPress
intheUKandincertainothercountries
PublishedintheUnitedStates
byOxfordUniversityPressInc.,NewYork
©OxfordUniversityPress,2007
Themoralrightsoftheauthorshavebeenasserted
DatabaserightOxfordUniversityPress(maker)
Firstpublished2007
Allrightsreserved.Nopartofthispublicationmaybereproduced,
storedinaretrievalsystem,ortransmitted,inanyformorbyanymeans,
withoutthepriorpermissioninwritingofOxfordUniversityPress,
orasexpresslypermittedbylaw,orundertermsagreedwiththeappropriate
reprographicsrightsorganization.Enquiriesconcerningreproduction
outsidethescopeoftheaboveshouldbesenttotheRightsDepartment,
OxfordUniversityPress,attheaddressabove
Youmustnotcirculatethisbookinanyotherbindingorcover
andyoumustimposethesameconditiononanyacquirer
BritishLibraryCataloguinginPublicationData
Dataavailable
LibraryofCongressCataloginginPublicationData
Dataavailable
TypesetbyNewgenImagingSystems(P)Ltd.,Chennai,India
PrintedinGreatBritain
onacid-freepaperby
AntonyRoweLtd.,Chippenham,Wiltshire
ISBN0–19–850570–1 978–0–19–850570–9
ISBN0–19–929706–1 978–0–19–929706–1
1 3 5 7 9 10 8 6 4 2
CONTENTS
1 Introduction 1
1.1 Organization 3
2 TheBayesParadigm,EstimationandInformationMeasures 5
2.1 BayesPosteriorDistribution 5
2.1.1 MinimumRiskEstimation 6
2.1.2 InformationMeasures 7
2.2 MathematicalPreliminaries 8
2.2.1 ProbabilitySpaces,RandomVariables,Distributions,
Densities,andExpectation 8
2.2.2 TransformationsofVariables 10
2.2.3 TheMultivariateNormalDistribution 10
2.2.4 CharacteristicFunction 11
2.3 MinimumRiskHypothesisTestingonDiscreteSpaces 12
2.3.1 MinimumProbabilityofErrorviaMaximumAPosteriori
HypothesisTesting 13
2.3.2 Neyman–PearsonandtheOptimalityoftheLikelihoodRatioTest 14
2.4 MinimumMean-SquaredErrorRiskEstimationinVectorSpaces 16
2.4.1 NormedLinearandHilbertSpaces 17
2.4.2 Least-SquaresEstimation 20
2.4.3 ConditionalMeanEstimationandGaussianProcesses 22
2.5 TheFisherInformationofEstimators 24
2.6 Maximum-Likelihoodanditsconsistency 26
2.6.1 ConsistencyviaUniformConvergenceofEmpiricalLog-likelihood 27
2.6.2 AsymptoticNormalityand√nConvergenceRateoftheMLE 28
2.7 Complete–IncompleteDataProblemsandtheEMAlgorithm 30
2.8 HypothesisTestingandModelComplexity 38
2.8.1 Model-OrderEstimationandthed/2logSample-SizeComplexity 38
2.8.2 TheGaussianCaseisSpecial 41
2.8.3 ModelComplexityandtheGaussianCase 42
2.9 BuildingProbabilityModelsviathePrincipleofMaximumEntropy 43
2.9.1 PrincipleofMaximumEntropy 44
2.9.2 MaximumEntropyModels 45
2.9.3 ConditionalDistributionsareMaximumEntropy 47
3 ProbabilisticDirectedAcyclicGraphsandTheirEntropies 49
3.1 DirectedAcyclicGraphs(DAGs) 49
3.2 ProbabilitiesonDirectedAcyclicGraphs(PDAGs) 51
3.3 FiniteStateMarkovChains 54
3.4 Multi-typeBranchingProcesses 56
3.4.1 TheBranchingMatrix 59
3.4.2 TheMoment-GeneratingFunction 60
3.5 ExtinctionforFinite-StateMarkovChainsandBranchingProcesses 62
3.5.1 ExtinctioninMarkovChains 62
3.5.2 ExtinctioninBranchingProcesses 63
3.6 EntropiesofDirectedAcyclicGraphs 64
3.7 CombinatoricsofIndependent,IdenticallyDistributedStringsviathe
AymptoticEquipartitionTheorem 65
v
vi CONTENTS
3.8 EntropyandCombinatoricsofMarkovChains 66
3.9 EntropiesofBranchingProcesses 68
3.9.1 TreeStructureofMulti-TypeBranchingProcesses 69
3.9.2 EntropiesofSub-Critical,Critical,andSuper-CriticalProcesses 70
3.9.3 TypicalTreesandtheEquipartitionTheorem 71
3.10 FormalLanguagesandStochasticGrammars 74
3.11 DAGsforNaturalLanguageModelling 81
3.11.1 MarkovChainsandm-Grams 81
3.11.2 Context-FreeModels 82
3.11.3 HierarchicalDirectedAcyclicGraphModel 84
3.12 EMAlgorithmsforParameterEstimationinHidden
MarkovModels 87
3.12.1 MAPDecodingoftheHiddenStateSequence 88
3.12.2 MLEstimationofHMMparametersviaEMForward/Backward
Algorithm 89
3.13 EMAlgorithmsforParameterEstimationinNatural
LanguageModels 92
3.13.1 EMAlgorithmforContext-FreeChomskyNormalForm 93
3.13.2 GeneralContext-FreeGrammarsandtheTrellisAlgorithm
ofKupiec 94
4 MarkovRandomFieldsonUndirectedGraphs 95
4.1 UndirectedGraphs 95
4.2 MarkovRandomFields 96
4.3 GibbsRandomFields 101
4.4 TheSplittingPropertyofGibbsDistributions 104
4.5 BayesianTextureSegmentation:Thelog-NormalizerProblem 110
4.5.1 TheGibbsPartitionFunctionProblem 110
4.6 Maximum-EntropyTextureRepresentation 112
4.6.1 EmpiricalMaximumEntropyTextureCoding 113
4.7 StationaryGibbsRandomFields 116
4.7.1 TheDobrushin/Lanford/RuelleDefinition 116
4.7.2 GibbsDistributionsExhibitMultipleLawswiththeSameInteractions
(PhaseTransitions):TheIsingModelatLowTemperature 117
4.8 1DRandomFieldsareMarkovChains 119
4.9 MarkovChainsHaveaUniqueGibbsDistribution 120
4.10 EntropyofStationaryGibbsFields 121
5 GaussianRandomFieldsonUndirectedGraphs 123
5.1 GaussianRandomFields 123
5.2 DifferenceOperatorsandAdjoints 124
5.3 GaussianFieldsInducedviaDifferenceOperators 126
5.4 StationaryGaussianProcessesonZdandtheirSpectrum 133
5.5 Cyclo-StationaryGaussianProcessesandtheirSpectrum 134
5.6 Thelog-DeterminantCovarianceandtheAsymptoticNormalizer 137
5.6.1 AsymptoticsoftheGaussianprocessesandtheirCovariance 138
5.6.2 TheAsymptoticCovarianceandlog-Normalizer 142
5.7 TheEntropyRatesoftheStationaryProcess 142
5.7.1 Burg’sMaximumEntropyAuto-regressiveProcessesonZd 143
5.8 GeneralizedAuto-RegressiveImageModellingviaMaximum-Likelihood
Estimation 144
5.8.1 AnisotropicTextures 147
CONTENTS vii
6 TheCanonicalRepresentationsofGeneralPatternTheory 154
6.1 TheGenerators,Configurations,andRegularityofPatterns 154
6.2 TheGeneratorsofFormalLanguagesandGrammars 158
6.3 GraphTransformations 162
6.4 TheCanonicalRepresentationofPatterns:DAGs,MRFs,GaussianRandomFields 166
6.4.1 DirectedAcyclicGraphs 167
6.4.2 MarkovRandomFields 169
6.4.3 GaussianRandomFields:Generatorsinducedviadifferenceoperators 170
7 MatrixGroupActionsTransformingPatterns 174
7.1 GroupsTransformingConfigurations 174
7.1.1 SimilarityGroups 174
7.1.2 GroupActionsDefiningEquivalence 175
7.1.3 GroupsActionsonGeneratorsandDeformableTemplates 177
7.2 TheMatrixGroups 177
7.2.1 LinearMatrixandAffineGroupsofTransformation 177
7.2.2 MatrixgroupsactingonRd 179
7.3 TransformationsConstructedfromProductsofGroups 181
7.4 RandomRegularityontheSimilarities 184
7.5 CurvesasSubmanifoldsandtheFrenetFrame 190
7.6 2DSurfacesinR3andtheShapeOperator 195
7.6.1 TheShapeOperator 196
7.7 FittingQuadraticChartsandCurvaturesonSurfaces 198
7.7.1 GaussianandMeanCurvature 198
7.7.2 SecondOrderQuadraticCharts 200
7.7.3 IsosurfaceAlgorithm 201
7.8 RidgeCurvesandCrestLines 205
7.8.1 DefinitionofSulcus,Gyrus,andGeodesicCurveson
TriangulatedGraphs 205
7.8.2 DynamicProgramming 207
7.9 BijectionsandSmoothMappingsforCoordinatizing
ManifoldsviaLocalCoordinates 210
8 Manifolds,ActiveModels,andDeformableTemplates 214
8.1 ManifoldsasGenerators,TangentSpaces,andVectorFields 214
8.1.1 Manifolds 214
8.1.2 TangentSpaces 215
8.1.3 VectorFieldsonM 217
8.1.4 CurvesandtheTangentSpace 218
8.2 SmoothMappings,theJacobian,andDiffeomorphisms 219
8.2.1 SmoothMappingsandtheJacobian 219
8.2.2 TheJacobianandLocalDiffeomorphicProperties 221
8.3 MatrixGroupsareDiffeomorphismswhichareaSmoothManifold 222
8.3.1 Diffeomorphisms 222
8.3.2 MatrixGroupActionsareDiffeomorphismsonthe
BackgroundSpace 223
8.3.3 TheMatrixGroupsareSmoothManifolds(LieGroups) 224
8.4 ActiveModelsandDeformableTemplatesasImmersions 226
8.4.1 SnakesandActiveContours 226
8.4.2 DeformingClosedContoursinthePlane 226
8.4.3 NormalDeformableSurfaces 227
8.5 ActivatingShapesinDeformableModels 229
8.5.1 LikelihoodofShapesPartitioningImage 229
8.5.2 AGeneralCalculusforShapeActivation 229
viii CONTENTS
8.5.3 ActiveClosedContoursinR2 232
8.5.4 ActiveUnclosedSnakesandRoads 234
8.5.5 NormalDeformationofCirclesandSpheres 236
8.5.6 ActiveDeformableSpheres 236
8.6 LevelSetActiveContourModels 237
8.7 GaussianRandomFieldModelsforActiveShapes 240
9 SecondOrderandGaussianFields 244
9.1 SecondOrderProcesses(SOP)andtheHilbertSpaceofRandomVariables 244
9.1.1 Measurability,Separability,Continuity 244
9.1.2 Hilbertspaceofrandomvariables 247
9.1.3 CovarianceandSecondOrderProperties 249
9.1.4 QuadraticMeanContinuityandIntegration 251
9.2 OrthogonalProcessRepresentationsonBoundedDomains 252
9.2.1 CompactOperatorsandCovariances 253
9.2.2 OrthogonalRepresentationsforRandomProcessesandFields 257
9.2.3 StationaryPeriodicProcessesandFieldsonBoundedDomains 258
9.3 GaussianFieldsontheContinuum 262
9.4 SobolevSpaces,Green’sFunctions,andReproducing
KernelHilbertSpaces 264
9.4.1 ReproducingKernelHilbertSpaces 265
9.4.2 SobolevNormedSpaces 266
9.4.3 RelationtoGreen’sFunctions 267
9.4.4 GradientandLaplacianInducedGreen’sKernels 267
9.5 GaussianProcessesInducedviaLinearDifferentialOperators 271
9.6 GaussianFieldsintheUnitCube 274
9.6.1 MaximumLikelihoodEstimationoftheFields:GeneralizedARMA
Modelling 278
9.6.2 SmallDeformationVectorFieldsModelsinthePlaneandCube 280
9.7 DiscreteLatticesandReachabilityofCyclo-StationarySpectra 283
9.8 StationaryProcessesontheSphere 285
9.8.1 LaplacianOperatorInducedGaussianFieldsontheSphere 289
9.9 GaussianRandomFieldsonanArbitrarySmoothSurface 293
9.9.1 Laplace-BeltramiOperatorwithNeumannBoundaryConditions 293
9.9.2 SmoothinganArbitraryFunctiononManifoldsbyOrthonormalBases
oftheLaplace-BeltramiOperator 297
9.10 SamplePathPropertiesandContinuity 299
9.11 GaussianRandomFieldsasPriorDistributionsinPoint
ProcessImageReconstruction 303
9.11.1 TheNeedforRegularizationinImageReconstruction 304
9.11.2 SmoothnessandGaussianPriors 304
9.11.3 Good’sRoughnessasaGaussianPrior 305
9.11.4 ExponentialSplineSmoothingviaGood’sRoughness 306
9.12 Non-CompactOperatorsandOrthogonalRepresentations 309
9.12.1 CramerDecompositionforStationaryProcesses 311
9.12.2 OrthogonalScaleRepresentation 312
10 MetricsSpacesfortheMatrixGroups 316
10.1 RiemannianManifoldsasMetricSpaces 316
10.1.1 MetricSpacesandSmoothManifolds 316
10.1.2 RiemannianManifold,GeodesicMetric,andMinimumEnergy 317
10.2 VectorSpacesasMetricSpaces 319
10.3 CoordinateFramesontheMatrixGroupsand
theExponentialMap 320
CONTENTS ix
10.3.1 LeftandRightGroupAction 320
10.3.2 TheCoordinateFrames 321
10.3.3 LocalOptimizationviaDirectionalDerivativesandthe
ExponentialMap 323
10.4 MetricSpaceStructurefortheLinearMatrixGroups 324
10.4.1 GeodesicsintheMatrixGroups 324
10.5 ConservationofMomentumandGeodesicEvolutionofthe
MatrixGroupsviatheTangentattheIdentity 326
10.6 MetricsintheMatrixGroups 327
10.7 ViewingtheMatrixGroupsinExtrinsicEuclideanCoordinates 329
10.7.1 TheFrobeniusMetric 329
10.7.2 ComparingintrinsicandextrinsicmetricsinSO(2,3) 330
11 MetricsSpacesfortheInfiniteDimensionalDiffeomorphisms 332
11.1 LagrangianandEulerianGenerationofDiffeomorphisms 332
11.1.1 OnConditionsforGeneratingFlowsofDiffeomorphisms 333
11.1.2 ModelingviaDifferentialOperatorsandtheReproducing
KernelHilbertSpace 335
11.2 TheMetricontheSpaceofDiffeomorphisms 336
11.3 MomentumConservationforGeodesics 338
11.4 ConservationofMomentumforDiffeomorphismSplines
SpecifiedonSparseLandmarkPoints 340
11.4.1 AnODEforDiffeomorphicLandmarkMapping 343
12 MetricsonPhotometricandGeometricDeformableTemplates 346
12.1 MetricsonDenseDeformableTemplates:GeometricGroups
ActingonImages 346
12.1.1 GroupActionsontheImages 346
12.1.2 InvariantMetricDistances 347
12.2 TheDiffeomorphismMetricfortheImageOrbit 349
12.3 NormalMomentumMotionforGeodesicConnectionVia
InexactMatching 350
12.4 NormalMomentumMotionforTemporalSequences 354
12.5 MetricDistancesBetweenOrbitsDefinedThrough
InvarianceoftheMetric 356
12.6 FiniteDimensionalLandmarkedShapeSpaces 357
12.6.1 TheEuclideanMetric 357
12.6.2 Kendall’sSimilitudeInvariantDistance 359
12.7 TheDiffeomorphismMetricandDiffeomorphismSplinesonLandmarkShapes 361
12.7.1 SmallDeformationSplines 361
12.8 TheDeformableTemplate:OrbitsofPhotometricand
GeometricVariation 365
12.8.1 MetricSpacesforPhotometricVariability 365
12.8.2 TheMetricsInducedviaPhotometricandGeometricFlow 366
12.9 TheEulerEquationsforPhotometricandGeometricVariation 369
12.10 MetricsbetweenOrbitsoftheSpecialEuclideanGroup 373
12.11 TheMatrixGroups(EuclideanandAffineMotions) 374
12.11.1 ComputingtheAffineMotions 376
13 EstimationBoundsforAutomatedObjectRecognition 378
13.1 TheCommunicationsModelforImageTransmission 378
13.1.1 TheSourceModel:ObjectsUnderMatrixGroupActions 379
13.1.2 TheSensingModels:ProjectiveTransformationsinNoise 379
13.1.3 TheLikelihoodandPosterior 379