Table Of ContentNATO ASI Series
Advanced Science Institutes Series
A series presenting the results of activities sponsored by the NA TO Science
Committee, which aims at the dissemination of advanced scientific and
technological knowledge, with a view to strengthening links between scientific
communities.
The Series is published by an international board of publishers in conjunction with
the NATO Scientific Affairs Division
A Life Sciences Plenum Publishing Corporation
B Physics London and New York
C Mathematical and Kluwer Academic Publishers
Physical Sciences Dordrecht, Boston and London
o
Behavioural and
Social Sciences
E Applied Sciences
F Computer and Springer-Verlag
Systems Sciences Berlin Heidelberg New York
G Ecological Sciences London Paris Tokyo Hong Kong
H Cell Biology Barcelona Budapest
I Global Environmental
Change
NATO-PCO DATABASE
The electronic index to the NATO ASI Series provides full bibliographical
references (with keywords and/or abstracts) to more than 30000 contributions
from international scientists published in all sections of the NATO ASI Series.
Access to the NATO-PCO DATABASE compiled by the NATO Publication
Coordination Office is possible in two ways:
-via online FILE 128 (NATO-PCO DATABASE) hosted by ESRIN, Via Galileo
Galilei, 1-00044 Frascati, Italy.
-via CD-ROM "NATO-PCO DATABASE" with user-friendly retrieval software
in English, French and German (© WN GmbH and DATAWARE
Technologies Inc. 1989).
The CD-ROM can be ordered through any member of the Board of Publishers or
through NATO-PCO, Overijse, Belgium.
Series F: Computer and Systems Sciences Vol. 99
The ASI Series Books Published as a Result of
Activities of the Special Programme on
SENSORY SYSTEMS FOR ROBOTIC CONTROL
This book contains the proceedings of a NATO Advanced Research Workshop held
within the activities of the NATO Special Programme on Sensory Systems for Robotic
Control, running from 1983 to 1988 under the auspices of the NATO Science
Committee.
The books published so far as a result of the activities of the Special Programme are:
Vol. F25: Pyramidal Systems for Computer Vision. Edited by V. Cantoni and S. Levialdi. 1986.
Vol. F29: Languages for Sensor-Based Control in Robotics. Edited by U. Rembold and K. Hormann.
1987.
Vol. F33: Machine Intelligence and Knowledge Engineering for Robotic Applications. Edited by
A K. C. Wong and A Pugh. 1987.
Vol. F42: Real-Time Object Measurement and Classification. Edited by A K. Jain. 1988.
Vol. F43: Sensors and Sensory Systems for Advanced Robots. Edited by P. Dario. 1988.
Vol. F44: Signal Processing and Pattern Recognition in Nondestructive Evaluation of Materials.
Edited by C. H. Chen.1988.
Vol. F45: Syntactic and Structural Pattern Recognition. Edited by G. Ferrate, T. Pavlidis, A Sanfeliu
and H. Bunke. 1988.
Vol. F50: CAD Based Programming for Sensory Robots. Edited by B. Ravani. 1988.
Vol. F52: Sensor Devices and Systems for Robotics. Edited by A Casals. 1989.
Vol. F57: Kinematic and Dynamic Issues in Sensor Based Control. Edited by G. E. Taylor. 1990.
Vol. F58 Highly Redundant Sensing in Robotic Systems. Edited by J. T. Tou and J. G. Balchen.
1990.
Vol. F63: Traditional and Non-Traditional Robotic Sensors. Edited by T. C. Henderson. 1990.
Vol. F64: Sensory Robotics for the Handling of Limp Materials. Edited by P M. Taylor. 1990.
Vol. F65: Mapping and Spatial Modelling for Navigation. Edited by L. F. Pau. 1990.
Vol. F66: Sensor-Based Robots: Algorithms and Architectures. Edited by C. S. G. Lee. 1991.
Vol. F99: Multisensor Fusion for Computer Vision. Edited by J. K. Aggarwal. 1993.
Multisensor Fusion
for Computer Vision
Edited by
J. K. Aggarwal
Department of Electrical and Computer Engineering
The University of Texas at Austin
Austin, TX 78712-1084, USA
Springer-Verlag Berlin Heidelberg GmbH
PPrroocceeeeddiinnggss ooff tthhee NNAATTOO AAddvvaanncceedd RReesseeaarrcchh WWoorrkksshhoopp oonn MMuullttiisseennssoorr FFuussiioonn ffoorr
CCoommppuutteerr VViissiioonn,, hheelldd iinn GGrreennoobbllee,, FFrraannccee,, JJuunnee 2266--3300,, 11998899
CCRR SSuubbjjeecctt CCllaassssiiffiiccaattiioonn ((11999911)):: 11..22..99,, 11..22..1100,, 11..44..88,, 11..55..44
ISBN 978-3-642-08135-4 ISBN 978-3-662-02957-2 (eBook)
DOI 10.1007/978-3-662-02957-2
TThhiiss wwoorrkk iiss ssuubbjjeecctt ttoo ccooppyyrriigghhtt.. AAllll rriigghhttss aarree rreesseerrvveedd,, wwhheetthheerr tthhee wwhhoollee oorr ppaarrtt o01f tthhee mmaatteerriiaall iiss ccoonncceerrnneedd,,
ssppeecciifliiccaallllyy tthhee rriigghhttss o01f ttrraannssllaattiioonn,, rreepprriinnttiinngg,, rreeuussee o01f iilllluussttrraattiioonnss,, rreecciittaattiioonn,, bbrrooaaddccaassttiinngg,, rreepprroodduuccttiioonn oonn
mmiiccrroofliillmmss oorr iinn aannyy ootthheerr wwaayy,, aanndd ssttoorraaggee iinn ddaattaa bbaannkkss.. DDuupplliiccaattiioonn o01f tthhiiss ppuubblliiccaattiioonn oorr ppaarrttss tthheerreeoofl iiss
ppeerrmmiitttteedd oonnllyy uunnddeerr tthhee pprroovviissiioonnss o01f tthhee GGeerrmmaann CCooppyyrriigghhtt LLaaww o01f SSeepptteemmbbeerr 99,, 11996655,, iinn iittss ccuurrrreenntt vveerrssiioonn,,
aanndd ppeerrmmiissssiioonn lfoorr uussee mmuusstt aallwwaayyss bbee oobbttaaiinneedd lfrroomm SSpprriinnggeerr--VVeerrllaagg.. VViioollaattiioonnss aarree lliiaabbllee lfoorr pprroosseeccuuttiioonn
uunnddeerr tthhee GGeerrmmaann CCooppyyrriigghhtt LLaaww..
©© SSpprriinnggeerr--VVeerrllaagg BBeerrlliinn HHeeiiddeellbbeerrgg 11999933
OOrriiggiinnaallllyy ppuubblliisshheedd bbyy SSpprriinnggeerr--VVeerrllaagg BBeerrlliinn HHeeiiddeellbbeerrgg NNeeww YYoorrkk iinn 11999933
SSooffttccoovveerr rreepprriinntt o01f tthhee hhaarrddccoovveerr 11ss tt eeddiittiioonn 11999933
TTyyppeesseettttiinngg:: CCaammeerraa rreeaaddyy bbyy aauutthhoorrss
4455//33114400 --55 443322 11 00 --PPrriinntteedd oonn aacciidd--fIrreeee ppaappeerr
Preface
This volume contains revised papers based on contributions to the NATO
Advanced Research Workshop on Multisensor Fusion for Computer Vision, held in
Grenoble, France, in June 1989. The 24 papers presented here cover a broad
range of topics, including the principles and issues in multisensor fusion,
information fusion for navigation, multisensor fusion for object recognition,
network approaches to multisensor fusion, computer architectures for
multi sensor fusion, and applications of multisensor fusion.
The participants met in the beautiful surroundings of Mont Belledonne in
Grenoble to discuss their current work in a setting conducive to interaction and the
exchange of ideas. Each participant is a recognized leader in his or her area in the
academic, governmental, or industrial research community. The workshop focused
on techniques for the fusion or integration of sensor information to achieve the
optimum interpretation of a scene. Several participants presented novel points of
view on the integration of information. The 24 papers presented in this volume are
based on those collected by the editor after the workshop, and reflect various
aspects of our discussions. The papers are organized into five parts, as follows.
Part I, Principles and Issues in Multisensor Fusion, covers the general
issues in multisensor fusion. Since there is no broadly accepted definition of multi
sensor fusion, these papers provide an interesting starting point for discussion. J an
Olof Eklundh presents the importance of model selection for information inte
gration with illustrative examples. James L. Crowley presents the general
principles for integrating perceptual information into a coherent description of the
world. J. K. Aggarwal and Chen-Chau Chu discuss the broad issues in multisensor
fusion for computer vision and give an overview of recent results for scene
segmentation and interpretation using images acquired via diverse sensing modali-
VI
ties. Demitri Terzopoulos surveys the physically based approach to data fusion
using deformable models and simulated forces. Finally, Gerald T. McKee discusses
a broad framework for fusion of information.
Part II, Information Fusion for Navigation, presents four papers on fusing
information for computer vision problems in navigation. Takeo Kanade et al.
introduce a pixel-based algorithm that estimates depth and its uncertainty at each
pixel. The algorithm combines Kalman filtering with iconic descriptions of depth
that can serve as a general framework for low-level dynamic vision. Greg Hager
considers a framework for describing sensor data fusion problems in terms of data
representations, sensors, and sensing tasks, and its decision theoretical
interpretation. He further presents an analysis of linear updating rules as well as a
method of solving this class of problems. Hugh Durrant-Whyte describes
techniques employed in geometric sensor integration from extracting initial de
scriptions to integrating geometrically and geographically disparate sensory infor
mation. G. Toscani and R. Deriche consider the problem of correspondence using
token trackers for two similar tasks, calibration of a stereo set of cameras and fast
motion detection.
Part III, Multisensor Fusion for Object Recognition, presents experi
mental results obtained from using multisensor fusion for object recognition. Uri
Rembold et ai. discuss results on combining intensity and range image data for 3D
object recognition. This timely research uses several range scanners to measure
range and intensity simultaneously. Fusion is performed through the computation
of edge and curvature information. In the second paper, Alberto Sanfeliu et ai.
present a method for recognizing partially occluded three-dimensional objects
from intensity images, using depth at a sparse set of points to reduce the number of
candidate models in the initial hypothesis. X. Lebegue et al. discuss the use of color
and geometric clues for recognition of man-made objects (concrete bridges) in
outdoor scenes. The next paper, by R. M. Bolle et aI., describes a framework of
VII
fusion for visual recognition through the use of constraint satisfaction networks. In
the final paper of this section, Amar Mitiche et al. compare methods for
multi sensor object identification based upon statistical pattern classifiers, neural
networks, and knowledge-based systems. This paper contains an extensive
bibliography for the interested reader.
Part IV, Computer Architectures for Multisensor Fusion, contains five
papers on architectures for multi sensor fusion. First, V. Caglioti and M. Somalvico
present a distributed architecture for fusing information that is capable of parallel
execution of many activities. The second paper, by Andre Ayoun et aI., describes a
general purpose architecture for multi sensor perception known as SKIDS - Signal
and Knowledge Interaction with Decisional Control. Next, L. O. Hertzberger et al.
describe algorithms for a SIMD processor array to achieve real time applications.
Harry Stephanou et al. then present a classification algorithm using the theory of
fractal sets in conjunction with the Dempster-Shafer theory. The final paper, by
L. F. Pau et aI., presents a knowledge-based sensor fusion editor that uses a library
of object-oriented methods and classes to achieve standard sensor fusion tasks.
The papers in Part V, Applications of Multisensor Fusion, present five
applications from diverse environments in which multisensor fusion for computer
vision plays an important role. In the first, Rick Holben discusses the task of
detecting man-made objects in a natural environment based on the analysis of image
sequences. This computer vision system fuses thermal and video sensor data, and is
oriented toward automatic surveillance applications. The second paper, by Paul
Schenker et aI., reviews multi sensor techniques that are being used in space
robotics. Application scenarios include telerobots for in-orbit spacecraft assembly
and service, as well as autonomous roving vehicles for planetary exploration and
sample collection. Next, Avi Kak et al. describe an approach to sensing strategies in
a robot workcell with multi sensor capabilities. This system uses information from
one sensor to develop a set of initial hypotheses and then selects the second sensing
VIII
modality that can best disambiguate the initial hypotheses. Sheldon Gruber presents
a computer vision system that uses multiple sensors to measure the quality of
surfaces. In this system, a hierarchical neural network that has been trained to
recognize a selected set of machined surfaces is used to examine the sensor output.
Finally, Clay Spence et al. detail results on the adaptive visuaVauditory fusion in the
target location system of the barn owl. This paper highlights both the simplicity of
such natural systems and the difficulty in understanding biological sensory systems.
Altogether, these 24 papers cover a broad spectrum of topics and give a repre
sentative picture of the current progress in multisensor fusion for computer vision
among the leading research groups in Europe, Canada, and the United States.
Finally, it is a pleasure to acknowledge the support of Norman Caplan of the
National Science Foundation, whose active encouragement and support nourished
the idea of a workshop on multi sensor fusion for computer vision from the initial
proposal to NATO to its fruition at the Grenoble workshop. A number of individ
uals contributed to the success of the workshop. Local arrangements were handled
by Jim Crowley and Mirella Bello of LIFIA (IMAG), Grenoble, France. The
workshop program committee consisted of Norman Caplan, Jim Crowley, Takeo
Kanade (Carnegie Mellon University, U.S.A.), Alan Pugh (University of Hull,
U.K.), and J. K. Aggarwal. Administrative details at The University of Texas at
Austin were handled by J. K. Aggarwal and Debi Paxton. Most of all, it was the
workshop participants who most contributed to its success, and special thanks are
due to them for their enlightening and informative presentations at the workshop
and in the following papers.
Austin, Texas, October 1992 1. K. Aggarwal
Table of Contents
Preface .................................................. VII
I. Principles and Issues in Multisensor Fusion
Information Integration and Model Selection in Computer Vision . . . . . . . . . . . . . . 3
lan-Olof Eklundh
Principles and Techniques for Sensor Data Fusion. . . . . . . . . . . . . . . . . . . . . . .. 15
lames L. Crowley
The Issues, Analysis, and Interpretation of Multisensor Images. . . . . . . . . . . . . .. 37
I. K. Aggarwal and Chen-Chau Chu
Physically-Based Fusion of Visual Data over Space, Time, and Scale . . . . . . . . . .. 63
Demetri Terzopoulos
What Can Be Fused? 71
Gerard T. McKee
II. Information Fusion for Navigation
Kalman Filter-Based Algorithms for Estimating Depth from Image Sequences 87
Larry Matthies, Richard Szeliski, and Takeo Kanade
Robust Linear Rules for Nonlinear Systems . . . . . . . . . . . . . . . . . . . . . . . . . . " 131
Greg Hager
Geometric Sensor Fusion in Robotics (Abstract) . . . . . . . . . . . . . . . . . . . . . . . .. 151
Hugh F. Durrant -Whyte
Cooperation between 3D Motion Estimation and Token Trackers (Abstract) . . . . . .. 153
G. Toscani andR. Deriche
Three-Dimensional Fusion from a Monocular Sequence of Images . . . . . . . . . . . .. 155
I. L. lezouin and N. Ayache
III. Multisensor Fusion for Object Recognition
Fusion of Range and Intensity Image Data for Recognition of 3D Object
Surfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 171
lianchi Wei, Paul Levi, and Ulrich Rembold
Integrating Driving Model and Depth for Identification of Partially Occluded
3D Models ............................................... 195
A. Sanfeliu, M. Miaiios, and M. I. Dunjo
x
Fusion of Color and Geometric Information ........................... , 213
Xavier F. Lebegue, David C. Baker, and J. K. Aggarwal
Evidence Fusion Using Constraint Satisfaction Networks. . . . . . . . . . . . . . . . . .. 239
Andrea Califano, Ruud M. Bolle, Rick Kjeldsen, and Russell W. Taylor
Multisensor Information Integration for Object Identification . . . . . . . . . . . . . . . .. 255
A. Mitiche, R. Laganiere, and T. Henderson
IV. Computer Architectures for Multisensor Fusion
Distributing Inferential Activity for Synchronic and Diachronic Data Fusion. . . . . .. 279
V. Caglioti and M. Somalvico
Real-Time Perception Architectures: The SKIDS Project . . . . . . . . . . . . . . . . . .. 293
A. Ayoun, C. Bur, R. Havas, N. Touitou, and J.-M. Valade
Algorithms on a SIMD Processor Array. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 307
A. van Inge, L. O. Hertzberger, A. G. Starreveld, and F. C. A. Groen
Shape and Curvature Data Fusion by Conductivity Analysis (Abstract). . . . . . . . .. 323
H. E. Stephanou and A. M. Erkmen
A Knowledge-Based Sensor Fusion Editor. . . . . . . . . . . . . . . . . . . . . . . . . . . .. 325
L. F. Pau, X. Xiao, and C. Westphal
V. Applications of Multisensor Fusion
Multisensor Change Detection for Surveillance Applications . . . . . . . . . . . . . . . .. 345
Rick Holben
Multisensor Techniques for Space Robotics .................. ......... 367
P. Schenker, B. Wilcox, D. Gennery, and C. Anderson
Coordinated Use of Multiple Sensors in a Robotic Workcell . . . . . . . . . . . . . . . .. 395
A. C. Kak, S. A. Hutchinson, C. H. Chen, S. N. Gottschlich, and K. D. Smith
Neural Network Based Inspection of Machined Surfaces Using Multiple Sensors . .. 421
Sheldon Gruber
Adaptive Visual! Auditory Fusion in the Target Localization System
of the Barn Owl ............................................ 439
Clay D. Spence and John C. Pearson
Index of Key Terms 451
VVorkshop Speakers 455
