3-D Computer Vision: Principles, Algorithms and Applications

453 Pages·2023·10.365 MB·English

by  Yu-Jin Zhang

Yu-Jin Zhang
3-D Computer
Vision
Principles, Algorithms and Applications
3-D Computer Vision
Yu-Jin Zhang
3-D Computer Vision
Principles, Algorithms and Applications
Yu-JinZhang
DepartmentofElectronicEngineering
TsinghuaUniversity
Beijing,China
ISBN978-981-19-7579-0 ISBN978-981-19-7580-6 (eBook)
https://doi.org/10.1007/978-981-19-7580-6
JointlypublishedwithPublishingHouseofElectronicsIndustry
TheprinteditionisnotforsaleinChina(Mainland).CustomersfromChina(Mainland)pleaseorderthe
printbookfrom:PublishingHouseofElectronicsIndustry.
©TheEditor(s)(ifapplicable)andTheAuthor(s),underexclusivelicensetoSpringerNatureSingapore
PteLtd.2023
JointlypublishedwithPublishingHouseofElectronicsIndustry,Beijing,China
Thisworkissubjecttocopyright.AllrightsaresolelyandexclusivelylicensedbythePublisher,whether
thewholeorpartofthematerialisconcerned,specificallytherightsofreprinting,reuseofillustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or
dissimilarmethodologynowknownorhereafterdeveloped.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
Thepublishers,theauthors,andtheeditorsaresafetoassumethattheadviceandinformationinthis
book are believed to be true and accurate at the date of publication. Neither the publishers nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publishers remain neutral with
regardtojurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
ThisSpringerimprintispublishedbytheregisteredcompanySpringerNatureSingaporePteLtd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface
This book is a specialized textbook that introduces the basic principles, typical
methods, and practical techniques of 3-D computer vision. It can provide an
advanced/secondcourseserviceofcomputervisionforgraduatestudentsofrelated
majorsinhigherengineeringcollegesanduniversities,afterstudying2DComputer
Vision:Principles,AlgorithmsandApplications.
Thisbookmainlycoversthehigherlevelofcomputervisionfromtheselectionof
materials.Thisbookisself-contained,mainlyforinformationmajors,butalsotakes
into account learners of different professional backgrounds, and also considers the
needs of self-study readers. After learning the fundamental content of computer
visioninthisbook,readerscancarryoutscientificresearchandsolvemoreandeven
difficultspecificproblemsinpracticalapplications.
This book pays more attention to practicality in writing. Considering that com-
puter vision technology has been involved in many professional fields in recent
years,butmanyworkingpeoplearenotspecializedincomputervisiontechnology,
this book does not emphasize the theoretical system too much, minimizes the
formula derivation, and focuses on commonly used techniques. This book has
many sample questions and uses intuitive explanation to help readers understand
abstract concepts. A subject index list is given at the end of the book, and those
correspondingtermsaremarkedinboldinthetext.
This book provides a large number of self-test questions (including hints and
answers). In terms of purpose: on the one hand, it is convenient for self-study to
judge whether they have mastered the key content; on the other hand, it is also
convenient forteachers tocarry out online teaching and strengthen teacher-student
interaction during lectures. The types of questions are multiple-choice questions,
which can be easily judged by a computer. In terms of content, many questions
expressthebasicconceptsinadifferentway,supplementingthetext,sothatlearners
candeepentheirunderstanding.Somequestionslistdescriptionsthataresimilarbut
notthesame,orevenhaveoppositemeanings.Throughdialecticalthinkingofpros
and cons, learners can also deeply understand the essence. The hints have been
provided for all self-test questions, allowing readers to obtain more information to
v
vi Preface
furtheridentifythemeaningofthequestions.Atthesametime,eachquestioncanbe
saidtobedividedintotwolevelsinthisway.Readerscancompletetheself-testafter
readingthehintstoshowthattheybasicallyunderstandit,andtocompletetheself-
testwithoutlookingatthehintsindicatesthattheyhaveanevenbettergrasp.
Fromthestructureofthisbook,thereare12chaptersintotal,plusoneappendix,
answerstoself-testquestions,andsubjectindex.Underthese15first-levelheadings,
thereareatotalof103second-levelheadings(sections),andthereare141third-level
headings (subsections) underneath. The book has a total of about 500,000 words
(including pictures, drawings, tables, formulas, etc.) and a total of (numbered)
228figures,22tables,and566formulas.Inordertofacilitateteachingandlearning,
thisbookprovidesatotalof68examplesofvarioustypesand157self-testquestions
(all with hints and answers). In addition, there are a list of more than 100 directly
relatedreferencesandalistofmorethan500subjecttermsforindexing,attheendof
thebook.
This book can consider three aspects from the knowledge requirements of the
prerequisitecourses:(1)Mathematics:includinglinearalgebraandmatrixtheory,as
wellasbasicknowledgeaboutstatistics,probabilitytheory,andrandommodeling;
(2) Computer science: including the mastery of computer software technology, the
understanding of computer structure system, and the application of computer pro-
gramming methods; (3) Electronics: On the one hand, the characteristics and prin-
ciplesofelectronicequipment;ontheotherhand,circuitdesignandothercontent.In
addition, the book 2D Computer Vision: Principles, Algorithms and Applications
canbecountedasthedisciplineprerequisiteofthisbook.
Thanks to the editors of the publisher for carefully composing the manuscript,
seriouslyreviewing,andattentivelymodifying.
Finally, the author thanks his wife Yun HE and daughter Heming ZHANG for
theirunderstandingandsupportinallaspects.
Beijing,China Yu-JinZhang
Contents
1 ComputerVisionOverview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 HumanVisionandCharacteristics. . . . . . . . . . . . . . . . . . . . . . 2
1.1.1 VisualCharacteristics. . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 BrightnessPropertiesofVision. . . . . . . . . . . . . . . . . . 3
1.1.3 SpatialPropertiesofVision. . . . . . . . . . . . . . . . . . . . . 6
1.1.4 TemporalPropertiesofVision. . . . . . . . . . . . . . . . . . . 8
1.1.5 VisualPerception. . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.2 ComputerVisionTheoryandFramework. . . . . . . . . . . . . . . . . 13
1.2.1 ReaserchGoals,Tasks,andMethodsofComputer
Vision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.2.2 VisualComputationalTheory. . . . . . . . . . . . . . . . . . . 14
1.2.3 FrameworkProblemsandImprovements. . . . . . . . . . . 20
1.3 Three-DimensionalVisionSystemandImageTechnology. . . . . 23
1.3.1 Three-DimensionalVisionSystemProcess. . . . . . . . . . 23
1.3.2 ComputerVisionandImageTechnologyLevels. . . . . . 24
1.3.3 ImageTechnologyCategory. . . . . . . . . . . . . . . . . . . . 25
1.4 OverviewoftheStructureandContentofThisBook. . . . . . . . . 27
1.4.1 StructuralFrameworkandContentofThisBook. . . .. 27
1.4.2 ChapterOverview. . . . . . . . . . . . . . . . . . . . . . . . . . . 28
1.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 29
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
2 CameraCalibration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.1 LinearCameraModel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.1.1 CompleteImagingModel. . . . . . . . . . . . . . . . . . . . . . 38
2.1.2 BasicCalibrationProcedure. . . . . . . . . . . . . . . . . . . . 40
2.1.3 InternalandExternalParameters. . . . . . . . . . . . . . . . . 41
2.2 Non-LinearCameraModel. . . . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.1 TypeofDistortion. . . . . . . . . . . . . . . . . . . . . . . . . . . 44
vii
viii Contents
2.2.2 CalibrationSteps. . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.2.3 ClassificationofCalibrationMethods. . . . . . . . . . . . . 48
2.3 TraditionalCalibrationMethods. . . . . . . . . . . . . . . . . . . . . . . . 49
2.3.1 BasicStepsandParameters. . . . . . . . . . . . . . . . . . . . . 49
2.3.2 Two-StageCalibrationMethod. . . . . . . . . . . . . . . . . . 51
2.3.3 PrecisionImprovement. . . . . . . . . . . . . . . . . . . . . . . . 56
2.4 Self-CalibrationMethods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
2.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 61
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
3 Three-DimensionalImageAcquisition. . . . . . . . . . . . . . . . . . . . . . . 67
3.1 High-DimensionalImage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.2 DepthImage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.2.1 DepthImageandGrayscaleImage. . . . . . . . . . . . . . . 69
3.2.2 IntrinsicImageandNon-IntrinsicImage. . . . . . . . . . . 70
3.2.3 DepthImagingModes. . . . . . . . . . . . . . . . . . . . . . . . 72
3.3 DirectDepthImaging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.3.1 Time-of-FlightMethod. . . . . . . . . . . . . . . . . . . . . . . . 74
3.3.2 StructuredLightMethod. . . . . . . . . . . . . . . . . . . . . . . 77
3.3.3 MoiréContourStripesMethod. . . . . . . . . . . . . . . . . . 79
3.3.4 SimultaneousAcquisitionofDepthandBrightness
Images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.4 StereoVisionImaging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.4.1 BinocularHorizontalMode. . . . . . . . . . . . . . . . . . . . . 85
3.4.2 BinocularConvergenceHorizontalMode. . . . . . . . . . . 89
3.4.3 BinocularAxialMode. . . . . . . . . . . . . . . . . . . . . . . . 92
3.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 95
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4 VideoImageandMotionInformation. . . . . . . . . . . . . . . . . . . . . . . 101
4.1 VideoBasic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.1.1 VideoExpressionandModel. . . . . . . . . . . . . . . . . . . 102
4.1.2 VideoDisplayandFormat. . . . . . . . . . . . . . . . . . . . . 104
4.1.3 ColorTVSystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.2 MotionClassificationandRepresentation. . . . . . . . . . . . . . . . . 108
4.2.1 MotionClassification. . . . . . . . . . . . . . . . . . . . . . . . . 109
4.2.2 MotionVectorFieldRepresentation. . . . . . . . . . . . . . 109
4.2.3 MotionHistogramRepresentation. . . . . . . . . . . . . . . . 111
4.2.4 MotionTrackDescription. . . . . . . . . . . . . . . . . . . . . . 113
4.3 MotionInformationDetection. . . . . . . . . . . . . . . . . . . . . . . . . 114
4.3.1 MotionDetectionBasedonCameraModel. . . . . . . .. 114
4.3.2 FrequencyDomainMotionDetection. . . . . . . . . . . . . 118
4.3.3 DetectionofMovementDirection. . . . . . . .... ..... 121
Contents ix
4.4 Motion-BasedFiltering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
4.4.1 MotionDetection-BasedFiltering. . . . . . . . . . . . . . . . 123
4.4.2 MotionCompensation-BasedFiltering. . . . . . . . . . . . . 124
4.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 130
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
5 MovingObjectDetectionandTracking. . . . . . . . . . . . . . . . . . . . . . 135
5.1 DifferentialImage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
5.1.1 CalculationofDifferenceImage. . . . . . . . . . . . . . . . . 136
5.1.2 CalculationofAccumulativeDifferenceImage. . . . . . . 138
5.2 BackgroundModeling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
5.2.1 BasicPrinciple. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.2.2 TypicalPracticalMethods. . . . . . . . . . . . . . . . . . . . . . 140
5.2.3 EffectExamples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
5.3 OpticalFlowFieldandMotion. . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.1 OpticalFlowEquation. . . . . . . . . . . . . . . . . . . . . . . . 145
5.3.2 OpticalFlowEstimationwithLeastSquareMethod. . . 146
5.3.3 OpticalFlowinMotionAnalysis. . . . . . . . . . . . . . . . . 147
5.3.4 DenseOpticalFlowAlgorithm. . . . . . . . . . . . . . . . . . 150
5.4 MovingObjectTracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
5.4.1 KalmanFilter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
5.4.2 ParticleFilter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.4.3 MeanShiftandKernelTracking. . . . . . . . . . . . . . . . . 160
5.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 164
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
6 BinocularStereoVision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.1 StereoVisionProcessandModules. . . . . . . . . . . . . . . . . . . . . 170
6.1.1 CameraCalibration. . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.1.2 ImageAcquisition. . . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.1.3 FeatureExtraction. . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.4 StereoMatching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
6.1.5 3-DInformationRecovery. . . . . . . . . . . . . . . . . . . . . 172
6.1.6 Post-Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
6.2 Region-BasedStereoMatching. . . . . . . . . . . . . . . . . . . . . . . . 173
6.2.1 TemplateMatching. . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.2.2 StereoMatching. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
6.3 Feature-BasedStereoMatching. . . . . . . . . . . . . . . . . . . . . . . . 185
6.3.1 BasicStepsandMethods. . . . . . . . . . . . . . . . . . . . . . 186
6.3.2 ScaleInvariantFeatureTransformation. . . . . . . . . . . . 190
6.3.3 DynamicProgrammingMatching. . . . . . . . . . . . . . . . 192
6.4 ErrorDetectionandCorrectionofParallaxMap. . . . . . . . . . . . 194
6.4.1 ErrorDetection. . ...... ...... ....... ...... .. 194
x Contents
6.4.2 ErrorCorrection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196
6.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 198
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7 MonocularMultipleImageRecovery. . . . . . . . . . . . . . . . . . . . . . . 205
7.1 PhotometricStereo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206
7.1.1 LightSource,Scenery,Lens. . . . . . . . . . . . . . . . . . . . 207
7.1.2 SceneBrightnessandImageBrightness. . . . . . . . . . . . 208
7.1.3 SurfaceReflectionCharacteristicsandBrightness. . . . . 213
7.2 ShapefromIllumination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.2.1 RepresentationoftheSurfaceOrientation
ofaScene. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
7.2.2 ReflectanceMapandBrightnessConstraint
Equation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
7.2.3 SolutionofPhotometricStereo. . . . . . . . . . . . . . . . . . 222
7.3 OpticalFlowEquation. . . . .. . . .. . . .. . . .. . . . .. . . .. . . .. 225
7.3.1 OpticalFlowandMotionField. . . . . . . . . . . . . . . . . . 226
7.3.2 SolvingOpticalFlowEquation. . . . . . . . . . . . . . . . . . 229
7.4 ShapefromMotion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
7.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 240
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
8 MonocularSingleImageRecovery. . . . . . . . . . . . . . . . . . . . . . . . . 247
8.1 ShapefromShading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
8.1.1 ShadingandOrientation. . . . . . . . . . . . . . . . . . . . . . . 248
8.1.2 GradientSpaceMethod. . . . . . . . . . . . . . . . . . . . . . . 250
8.2 SolvingBrightnessEquation. . . . . . . . . . . . . . . . . . . . . . . . . . 253
8.2.1 LinearityCase. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
8.2.2 RotationalSymmetryCase. . . . . . . . . . . . . . . . . . . . . 256
8.2.3 TheGeneralCaseofSmoothnessConstraints. . . . . . . . 258
8.3 ShapefromTexture. . . . .. . . . . . .. . . . . . .. . . . . .. . . . . . .. 260
8.3.1 MonocularImagingandDistortion. . . . . . . . . . . . . . . 260
8.3.2 OrientationRestorationfromtheChangeofTexture. . . 263
8.4 DetectionofTextureVanishingPoints. . . . . . . . . . . . . . . . . . . 271
8.4.1 DetectingtheVanishingPointofLineSegment
Texture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
8.4.2 DeterminetheVanishingPointOutsidetheImage. . . . 273
8.5 KeyPointsandReferencesforEachSection. . . . . . . . . . . . . . . 276
Self-TestQuestions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
9 Three-DimensionalSceneryRepresentation. . . . . . . . . . . . . . . . . . 283
9.1 LocalFeaturesoftheSurface. . . . . . . . . . . . . . . . . . . . . . . . . . 284
9.1.1 SurfaceNormalSection..... ...... ...... ...... 284