High accuracy metrology using low-resolution cameras

219 Pages·2007·8.27 MB·English

High accuracy metrology using
low-resolution cameras
D.Phil Thesis
Robotics Research Group
Department of Engineering Science
University of Oxford
David Claus
New College
March 8, 2007
David Claus Doctor of Philosophy
New College Hilary Term 2007
High accuracy metrology using
low-resolution cameras
Abstract
Camera based localization can provide extremely accurate 3D pose information, even
from consumer grade video lenses. Advances in lens distortion correction, pose com-
putation and feature detection would permit low cost cameras to be used in many
applications that currently require more expensive equipment. I show how: (1) careful
modelling and (2) careful fitting of these models to data; provides increased camera
accuracy from the same camera equipment with little or no additional computational
overhead.
Theprimarycontributiontowardscameramodellingisalensdistortionmodelbasedon
rational functions that can represent standard, fisheye and catadioptric lens systems.
Three separate calibration methods are demonstrated, making this a useful technique
that can be implemented in a wide range of applications. Evaluation of calibration pre-
cisionindicatesthattheproposedmodelaccuratelyrepresentsreal-worldlensdistortion
andprovideslowererrorsthanothermodelsincommonuse. Althoughsensitivitytoim-
age noise can be a problem with such flexible models, several techniques are presented
here that yield robust calibration in the midst of image uncertainty. I demonstrate
multiple view camera auto-calibration on fisheye lens sequences using point correspon-
dences alone, without first requiring the removal of lens distortion.
Fitting of the camera model is improved by including a non-linear optimization to tune
the model parameters against a known error measure. Careful optimizer construction
is shown to avoid local minima, converge in realtime and achieve very high levels of
precision. Image feature detection error is transmitted through the entire calibration
process, so a robust exemplar based learning scheme is proposed to accurately detect
known fiducial markers. This efficient classification approach handles the challenges of
changing scene conditions (lighting variation, motion blur, clutter) without the large
increase in false detections that plague other detection algorithms.
i
Acknowledgements
I would like to thank Andrew Fitzgibbon for his patient instruction, guidance, and
assistance throughout the course of my research. I could not have asked for a better
supervisor.
Thanks to Jamie Paterson, Nicholas Apostoloff, Aeron Buchanan and the others in the
Visual Geometry Group for all your assistance and input.
My research was made possible through the funding of the Rhodes Trust; I would like
tothanktheWardenandallofthestaffatRhodesHouseformakingmystayinOxford
that much more enjoyable.
Finally, I would like to thank my wife, Krista, for all her support and encouragement
during both the research and the writing phases of this project. I did finish it in the
new year; it is just a different year.
ii
For Apsley
iii
Contents
1 Introduction 1
1.1 Why Localize? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Notational Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Authorship . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2 Definitions 5
2.1 Imaging Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.1.1 Pinhole projection . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 Aberrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1.3 Non-central cameras . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.4 Cubic Polynomial Camera . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Camera Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 Internal Calibration . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 External Calibration . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.3 Nonlinear Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.3.1 Levenberg-Marquardt . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.2 Bundle Adjustment . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4 Miscellaneous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4.1 Sampson Distance . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.4.2 Camera ringing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4.3 Direct Linear Transform . . . . . . . . . . . . . . . . . . . . . . . 29
2.4.4 Centre Shift Under Projection . . . . . . . . . . . . . . . . . . . 30
2.4.5 Pattern Classification . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4.6 Receiver Operator Characteristic Curves . . . . . . . . . . . . . . 35
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3 Lens Distortion: Modelling 37
3.1 Distortion Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.1.1 Forward and Reverse Camera Models . . . . . . . . . . . . . . . 39
3.2 Rational Function Model. . . . . . . . . . . . . . . . . . . . . . . . . . . 40
3.2.1 General Mathematical Framework . . . . . . . . . . . . . . . . . 40
3.2.2 Physical Interpretation of A . . . . . . . . . . . . . . . . . . . . . 42
3.2.3 Back-projection and Projection . . . . . . . . . . . . . . . . . . . 43
3.2.4 Canonicalization of A . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.5 Parametrizations for Specific Lenses . . . . . . . . . . . . . . . . 45
3.3 Two-view geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
3.3.1 Epipolar curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
iv
CONTENTS v
4 Lens Distortion: Calibration 51
4.1 Linear Calibration from an Arbitrary Planar Grid . . . . . . . . . . . . 51
4.2 Calibration by Plumb-line Constraints . . . . . . . . . . . . . . . . . . . 54
4.2.1 Linear Factorization Method . . . . . . . . . . . . . . . . . . . . 55
4.2.2 Optimization Method . . . . . . . . . . . . . . . . . . . . . . . . 61
4.2.3 Condensed vs. Full Parametrization . . . . . . . . . . . . . . . . 63
4.2.4 Plumbline data from multiple views . . . . . . . . . . . . . . . . 65
4.3 Multiview Calibration from Epipolar Constraints . . . . . . . . . . . . . 66
4.3.1 Linear method for G . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.3.2 Rank 2 nonlinear optimization method for G . . . . . . . . . . . . 68
4.3.3 Recovery of A from G . . . . . . . . . . . . . . . . . . . . . . . . . 69
4.3.4 Parameterizing G using the reduced RF model. . . . . . . . . . . 70
4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5 Lens Distortion: Evaluation 73
5.1 Approximation of existing distortion models . . . . . . . . . . . . . . . . 73
5.2 Planar Grid Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.3 Plumbline Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
5.3.1 Reduced RF consistency . . . . . . . . . . . . . . . . . . . . . . . 82
5.3.2 RF vs. the Matlab Calibration Toolbox . . . . . . . . . . . . . 83
5.4 Multiview results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.4.1 Noise sensitivity of computing G . . . . . . . . . . . . . . . . . . 90
5.4.2 Results on image sequences . . . . . . . . . . . . . . . . . . . . . 97
5.5 Three-Dimensional Reconstruction . . . . . . . . . . . . . . . . . . . . . 101
5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6 Camera Localization 104
6.1 World Points and Corresponding Image Locations . . . . . . . . . . . . 105
6.2 Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.2.1 Camera Intrinsics . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
6.2.2 Extrinsics from coplanar data . . . . . . . . . . . . . . . . . . . . 107
6.2.3 Extrinsics from general point data . . . . . . . . . . . . . . . . . 108
6.3 Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
6.3.1 Analytic Derivatives for Reprojection Error . . . . . . . . . . . . 110
6.3.2 Timing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
6.3.3 Zoom lenses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.4 Surveying Fiducial Positions Optically . . . . . . . . . . . . . . . . . . . 112
6.4.1 Structure from Motion . . . . . . . . . . . . . . . . . . . . . . . . 113
6.4.2 Surveying procedure . . . . . . . . . . . . . . . . . . . . . . . . . 114
6.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.5.1 Coordinate Measurement System Ground Truth . . . . . . . . . 115
6.5.2 ARToolkit comparison . . . . . . . . . . . . . . . . . . . . . . . . 130
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
7 Fiducial Detection 136
7.1 Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.1.1 Target Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.1.2 Training Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.1.3 Cascading Classifier . . . . . . . . . . . . . . . . . . . . . . . . . 142
7.1.4 Cascade Stage One: Ideal Bayes . . . . . . . . . . . . . . . . . . 143
7.1.5 Cascade Stage Two: Nearest Neighbour . . . . . . . . . . . . . . 145
7.2 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.2.1 Training Data Filtering . . . . . . . . . . . . . . . . . . . . . . . 147
7.2.2 Target verification . . . . . . . . . . . . . . . . . . . . . . . . . . 149
CONTENTS vi
7.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.3.1 Engineered Detector . . . . . . . . . . . . . . . . . . . . . . . . . 152
7.3.2 Adaptive Thresholding . . . . . . . . . . . . . . . . . . . . . . . . 152
7.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
7.4 Locating a Circle’s Projected Centre . . . . . . . . . . . . . . . . . . . . 156
7.4.1 Homography from target coordinates . . . . . . . . . . . . . . . . 157
7.4.2 Homography Calculation from Four Circles . . . . . . . . . . . . 159
7.5 Target Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7.6 Fiducial Localization Trials . . . . . . . . . . . . . . . . . . . . . . . . . 161
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
8 Photometric Stereo Application 166
8.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
8.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168
8.2.1 Lens Distortion Correction . . . . . . . . . . . . . . . . . . . . . 170
8.2.2 Light position . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
8.2.3 Cone Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
8.2.4 Camera pose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
8.2.5 Light Attenuation . . . . . . . . . . . . . . . . . . . . . . . . . . 177
8.2.6 Light Fall-off Correction . . . . . . . . . . . . . . . . . . . . . . . 181
8.2.7 Surface Integration . . . . . . . . . . . . . . . . . . . . . . . . . . 186
8.2.8 Parallax Correction . . . . . . . . . . . . . . . . . . . . . . . . . 188
8.3 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
8.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
9 Conclusion 194
9.1 Modelling lens distortion . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
9.2 Camera calibration and localization . . . . . . . . . . . . . . . . . . . . 196
9.3 Reliable fiducial detection . . . . . . . . . . . . . . . . . . . . . . . . . . 196
9.4 Application areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
9.6 Further extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
Appendices 199
A Pinhole Camera Calibration Methods 199
A.1 Extrinsics from Coplanar Data . . . . . . . . . . . . . . . . . . . . . . . 199
A.2 Extrinsics from General Point Data . . . . . . . . . . . . . . . . . . . . . 200
B Rotations 202
B.1 Quaternions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
C Computing A from G 204
Bibliography 212
Chapter 1
Introduction
This thesis topic was born out of necessity. The original topic was to be surveying
(in a building construction sense) with a video camera: merging structure-from-motion
(SfM)andsimultaneouslocalizationandmapping(SLAM)toaccuratelymeasurethree
dimensional positions over an area the size of a football pitch. It seemed a reasonable
starting point to pick up the nearest camera and go outside to record some video of
buildings — “Whatever you do, avoid shooting your own video footage” and “Try to
get some professional video clips to analyze” were two pieces of advice that came too
late in my introduction to computer vision.
Three things became apparent as I processed that early video: 1) detecting and
locatingfeaturesinvideorecordedunderwidelyvaryingsceneconditionsisharderthan
it looks; 2) precise camera calibration is essential, and often not fully taken advantage
of; and finally 3) lens distortion must be carefully compensated for when using low-cost
consumer grade lenses. But I’m getting ahead of myself; let us first take a step back
and discuss measurements in general.
Measurements are important, particularly in engineering. If an engineer is to apply
scientific theory to the solution of a problem he or she will often need to measure some
physical quantity to provide an input to theoretical equations or methods. Many times
it is also essential that these measurements be taken without disturbing the process
under observation.
A camera is a measurement device; it records a projection of both the wavelength
and intensity of light. Specifically, it measures the light falling upon its sensor from
directions in space as dictated by the lens system. Because light can travel through a
vacuum it is a truly non-contact measurement instrument. In practical terms this per-
mitsobjectpropertiestoberecordedatadistance, andthephysicsoflensconstruction
provides the relationship between image locations and directions in space. Thus the
camera is a remote measurement device. This angle measurement function is exploited
1
CHAPTER 1. INTRODUCTION 2
in both SfM and vision-based SLAM. However, angle measurement depends on camera
calibration: the properties of the lens must be known in order to relate image positions
to directions in space. Furthermore, surveying requires the camera location be known
so that all the measurements can be tied in (or referenced) to some physical location
or object.
1.1 Why Localize?
What are the potential uses for a precise camera localization algorithm? First, let
us define what we mean by camera localization: camera localization is the process of
computing the position and orientation of a camera from the images it recorded.
The most obvious application is as a position sensor. A camera that can report
where it is can be attached to robots, aircraft, or even held by the user. Such a camera
then provides a visual record of its surroundings and a position measurement for each
frame. This has uses in robot navigation, but can also be employed for object tagging
inventory systems, mapping and even simply keeping track of where one’s holiday
photos were taken.
Asecondapplicationforcameralocalizationisincinemaspecialeffects. Acomputer
rendered object that is to be added into footage of a real scene must be registered
relative to the background. If the camera does not move this is a job for artists; the
objects are painted in front of a static background. A virtual object inserted into
a sequence from a moving camera, however, will appear to float through the scene
unless it is aligned correctly in each frame. Re-drawing the scene for each frame is
too much for artists, and even slight alignment errors are noticeable. This type of
virtual reality requires that the camera location be known so that the virtual object
can be rendered from the right viewpoint and then placed at the correct position in
the image. In this case the camera could be tracked using other types of sensors
(inertial, ultrasonic, etc.) but it is simpler to use the camera alone and the resulting
image/virtualobjectregistrationismuchtighter. Athirdapplicationareaissurveying.
If the camera location is known then the angle formed by any two points in an image
can be determined.
This list of potential applications for camera localization techniques is quite long;
here we will close with one final example. Where the relative positions of several
cameras must be determined it can be faster and more accurate to localize each of the
cameras in a common reference frame than it would be to physically measure their
positions. This sort of self-calibration could be used to set up surveillance systems
or rigs for studio filming. This is the basis for the single moving camera photometric
CHAPTER 1. INTRODUCTION 3
stereo system described in Chapter 8.
1.2 Outline
So if camera localization is such a useful technology, how does one go about it? Or,
sincethetheoryiswellknownandnottoodifficult,perhapsthemorepertinentquestion
is “What are the tricks to include and the pitfalls to avoid so that my localization
algorithmisasuccess?” ThatiswhatIdescribeinthisthesis. Thebackgroundchapter
deals with some of the physics of image formation and provides technical details on
several of the concepts employed later on. After that is established we shall look at
a method for correcting lens distortion. The rational function model can render the
images from a wide variety of lenses as pinhole images. This greatly simplifies the
mathematics of localization, and even permits Structure from Motion autocalibration
from distorted image correspondences (please refer to Chapter 3 for details on what
that means). A chapter is then devoted to each of distortion model calibration and
evaluation: a model is only useful if a lens can be calibrated, and this calibration must
alsobecomparedwithexistingmodels. Theproposedmodelisshowntobebothsimple
to calibrate and extremely accurate for modelling a wide range of lens distortion. The
theory is presented first, and then followed in a later chapter by the description of
experimental results. This structure was adopted to help the reader; hopefully it is
easy to follow.
With the distortion out of the way Chapter 6 is able to focus on localization itself.
This is best handled with a nonlinear optimizer, and I present methods for assembling
fast and precise fitting algorithms. These were tested both in a registration framework
for placing virtual objects in cinema productions and against absolute position values
obtainedbyphysicallymeasuringthecamerapath. Centraltolocalizationistheability
to reliably and repeatedly detect the same features in an image. Chapter 7 presents
a method for detecting known markers (fiducials) in video. It is an exemplar-based
classifier that locates fiducials which are similar in appearance to those it was shown
beforehand. This allows us to deal with detection challenges such as motion blur,
variable illumination and oblique viewpoints.
To bring it all together we will examine a system that relies upon accurate camera
localization. The photometric stereo application of Chapter 8 uses a single camera
as input. Although the camera is free to move, the application requires images that
were all recorded from a common viewpoint. For this we use distortion correction
and localization from fiducial markers to determine the camera position of each image.
The images are re-rendered and parallax corrected to yield the required fronto-parallel