Learning-Based Robot Vision: Principles and Applications

291 Pages·2001·4.299 MB·English

by  Josef Pauli (auth.)

Lecture Notes in Computer Science 2048
EditedbyG.Goos,J.HartmanisandJ.vanLeeuwen
3
Berlin
Heidelberg
NewYork
Barcelona
HongKong
London
Milan
Paris
Singapore
Tokyo
Josef Pauli
Learning-Based
Robot Vision
Principles and Applications
1 3
SeriesEditors
GerhardGoos,KarlsruheUniversity,Germany
JurisHartmanis,CornellUniversity,NY,USA
JanvanLeeuwen,UtrechtUniversity,TheNetherlands
Author
JosefPauli
Christian-AlbrechtUniversia¨tzuKiel
Institutfu¨rInformatikundPraktischeMathematik
Preusserstr.1-9,24105Kiel,Germany
E-mail:[email protected]
Cataloging-in-PublicationDataappliedfor
DieDeutscheBibliothek-CIP-Einheitsaufnahme
Pauli,Josef:
Learning-basedrobotvision:principlesandapplications/Josef
Pauli.-Berlin;Heidelberg;NewYork;Barcelona;HongKong;
London;Milan;Paris;Singapore;Tokyo:Springer,2001
(Lecturenotesincomputerscience;2048)
ISBN3-540-42108-4
CRSubjectClassification(1998):I.4,I.2.9-11,I.2.6
ISSN0302-9743
ISBN3-540-42108-4Springer-VerlagBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerialis
concerned,specificallytherightsoftranslation,reprinting,re-useofillustrations,recitation,broadcasting,
reproductiononmicrofilmsorinanyotherway,andstorageindatabanks.Duplicationofthispublication
orpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLawofSeptember9,1965,
initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer-Verlag.Violationsare
liableforprosecutionundertheGermanCopyrightLaw.
Springer-VerlagBerlinHeidelbergNewYork
amemberofBertelsmannSpringerScience+BusinessMediaGmbH
http://www.springer.de
©Springer-VerlagBerlinHeidelberg2001
PrintedinGermany
Typesetting:Camera-readybyauthor,dataconversionbyBollerMediendesign
Printedonacid-freepaper SPIN10781470 06/3142 543210
Preface
Industrialrobotscarryoutsimpletasksincustomizedenvironmentsforwhich
it is typical that nearly all effector movements canbe planned during an off-
linephase.Acontinualcontrolbasedonsensoryfeedbackisatmostnecessary
at effector positions near target locations utilizing torque or haptic sensors.
It is desirable to develop new-generation robots showing higher degrees of
autonomyforsolvinghigh-leveldeliberatetasksinnaturalanddynamicenvi-
ronments.Obviously,camera-equippedrobotsystems,whichtakeandprocess
imagesandmakeuseofthe visualdata,cansolvemoresophisticatedrobotic
tasks.Thedevelopmentofa(semi-)autonomouscamera-equippedrobotmust
be grounded on an infrastructure, based on which the system can acquire
and/or adapt task-relevant competences autonomously. This infrastructure
consists of technical equipment to support the presentation of real world
training samples, various learning mechanisms for automatically acquiring
function approximations, and testing methods for evaluating the quality of
the learnedfunctions. Accordingly,to develop autonomouscamera-equipped
robotsystemsonemustfirstdemonstraterelevantobjects,criticalsituations,
and purposive situation-action pairs in an experimental phase prior to the
application phase. Secondly, the learning mechanisms are responsible for ac-
quiring image operatorsand mechanisms ofvisual feedback controlbasedon
supervised experiences in the task-relevant, real environment.
This paradigm of learning-based development leads to the concepts of
compatibilities andmanifolds. Compatibilities are generalconstraintson the
process of image formation which hold more or less under task-relevant
or accidental variations of the imaging conditions. Based on learned de-
grees of compatibilities, one can choose those image operators together with
parametrizations, which are expected to be most adequate for treating the
underlying task. On the other hand, significant variations of image features
arerepresentedasmanifolds.Theymayoriginatefromchangesinthespatial
relationamongroboteffectors,cameras,andenvironmentalobjects.Learned
manifoldsarethebasisforacquiringimageoperatorsfortask-relevantobject
orsituationrecognition.Theimageoperatorsareconstituentsoftask-specific,
behavioralmodules whichintegratedeliberate strategiesandvisualfeedback
control.The guidinglineforsystemdevelopmentisthattheresultingbehav-
iors should meet requirements such as task-relevance, robustness, flexibility,
VI Preface
timelimitation,etc.simultaneously.Allprinciplestobepresentedinthework
are based on real scenes of man-made objects and a multi-component robot
systemconsistingofrobotarm,head,andvehicle.Ahigh-levelapplicationis
presented that includes sub-tasks such as localizing, approaching, grasping,
and carrying objects.
Acknowledgements
Since 1993 I have been a member of the Kognitive Systeme Gruppe in Kiel.
I am most grateful to G. Sommer, the head of this group, for the contin-
ual advice and support. I have learned so much from his wide spectrum of
scientific experience.
A former version of this book has been submitted and accepted as ha-
bilitation thesis at the Institut fu¨r Informatik und Praktische Mathematik,
Technische Fakulta¨tofthe Christian-Albrechts-Universita¨t,in Kiel.I’mvery
grateful to the six persons who have been responsible for assessing the
work. These are: V. Hlavac from the Technical University of Prague in the
Czech Republic, R. Klette from the University of Auckland in New Zealand,
C.-E. Liedtke from the Universit¨at Hannover, G. Sommer and A. Srivastav
from the Christian-Albrechts-Universit¨at Kiel, and F. Wahl from the Tech-
nische Universita¨t Braunschweig. Deepest thanks also for the great interest
in my work.
IappreciatethediscussionswithformerandpresentcolleaguesJ.Bruske,
T. Bu¨low, K. Daniilidis, M. Felsberg, M. Hansen, N. Kru¨ger, V. Kru¨ger,
U. Mahlmeister, C. Perwass, B. Rosenhahn, W. Yu, and my students
M. Benkwitz, A. Bunten, S. Kunze, F. Lempelius, M. Pa¨schke, A. Schmidt,
W. Timm, and J. Tro¨ster.
TechnicalsupportwasprovidedbyA.Bunten,G.Diesner,andH.Schmidt.
My thanks to private individuals have been expressed personally.
March 2001 Josef Pauli
Contents
1. Introduction.............................................. 1
1.1 Need for New-Generation Robot Systems.................. 1
1.2 Paradigms of Computer Vision (CV) and Robot Vision (RV). 5
1.2.1 Characterization of Computer Vision ............... 5
1.2.2 Characterization of Robot Vision................... 8
1.3 Robot Systems versus Autonomous Robot Systems ......... 10
1.3.1 Characterization of a Robot System ................ 10
1.3.2 Characterization of an Autonomous Robot System.... 11
1.3.3 Autonomous Camera-Equipped Robot System ....... 14
1.4 Important Role of Demonstration and Learning ............ 15
1.4.1 Learning Feature Compatibilities under Real Imaging . 15
1.4.2 Learning Feature Manifolds of Real World Situations . 18
1.4.3 Learning Environment-Effector-ImageRelationships .. 20
1.4.4 Compatibilities, Manifolds, and Relationships ........ 21
1.5 Chapter Overview of the Work ........................... 23
2. Compatibilities for Object Boundary Detection........... 25
2.1 Introduction to the Chapter ............................. 25
2.1.1 General Context of the Chapter .................... 25
2.1.2 Object Localization and Boundary Extraction........ 27
2.1.3 Detailed Review of Relevant Literature.............. 28
2.1.4 Outline of the Sections in the Chapter .............. 31
2.2 Geometric/Photometric Compatibility Principles ........... 32
2.2.1 Hough Transformationfor Line Extraction........... 32
2.2.2 Orientation Compatibility between Lines and Edges .. 34
2.2.3 Junction Compatibility between Pencils and Corners.. 41
2.3 Compatibility-Based Structural Level Grouping ............ 46
2.3.1 Hough Peaks for Approximate Parallel Lines......... 47
2.3.2 Phase Compatibility between Parallels and Ramps.... 49
2.3.3 Extraction of Regular Quadrangles ................. 54
2.3.4 Extraction of Regular Polygons .................... 61
2.4 Compatibility-Based Assembly Level Grouping............. 69
2.4.1 Focusing Image Processing on PolygonalWindows.... 70
2.4.2 Vanishing-Point Compatibility of Parallel Lines ...... 74
VIII Contents
2.4.3 Pencil Compatibility of Meeting Boundary Lines ..... 76
2.4.4 Boundary Extraction for Approximate Polyhedra..... 78
2.4.5 Geometric Reasoning for Boundary Extraction ....... 79
2.5 Visual Demonstrations for LearningDegrees ofCompatibility. 85
2.5.1 LearningDegreeofLine/EdgeOrientationCompatibility 85
2.5.2 LearningDegreeofParallel/RampPhaseCompatibility. 90
2.5.3 Learning Degree of Parallelism Compatibility ........ 95
2.6 Summary and Discussion of the Chapter .................. 96
3. Manifolds for Object and Situation Recognition .......... 101
3.1 Introduction to the Chapter ............................. 101
3.1.1 General Context of the Chapter .................... 101
3.1.2 Approach for Object and Situation Recognition ...... 102
3.1.3 Detailed Review of Relevant Literature.............. 103
3.1.4 Outline of the Sections in the Chapter .............. 108
3.2 Learning Pattern Manifolds with GBFs and PCA........... 108
3.2.1 Compatibility and Discriminability for Recognition ... 108
3.2.2 Regularization Principles and GBF Networks ........ 111
3.2.3 Canonical Frameswith PrincipalComponent Analysis . 116
3.3 GBF Networks for Approximation of Recognition Functions.. 122
3.3.1 Approach of GBF Network Learning for Recognition.. 122
3.3.2 Object Recognition under Arbitrary View Angle ..... 124
3.3.3 Object Recognition for Arbitrary View Distance...... 129
3.3.4 Scoring of Grasping Situations ..................... 131
3.4 SophisticatedManifoldApproximationforRobustRecognition. 133
3.4.1 Making Manifold Approximation Tractable .......... 134
3.4.2 Log-PolarTransformation for Manifold Simplification . 137
3.4.3 Space-Time Correlations for Manifold Refinement .... 145
3.4.4 Learning Strategy with PCA/GBF Mixtures......... 154
3.5 Summary and Discussion of the Chapter .................. 168
4. Learning-Based Achievement of RV Competences ........ 171
4.1 Introduction to the Chapter ............................. 171
4.1.1 General Context of the Chapter .................... 171
4.1.2 Learning Behavior-BasedSystems .................. 174
4.1.3 Detailed Review of Relevant Literature.............. 178
4.1.4 Outline of the Sections in the Chapter .............. 182
4.2 Integrating Deliberate Strategies and Visual Feedback....... 183
4.2.1 Dynamical Systems and Control Mechanisms ........ 183
4.2.2 Generic Modules for System Development ........... 197
4.3 Treatment of an Exemplary High-Level Task............... 206
4.3.1 Description of an Exemplary High-Level Task........ 206
4.3.2 Localization of a Target Object in the Image......... 208
4.3.3 Determining and Reconstructing Obstacle Objects.... 213
4.3.4 Approaching and Grasping Obstacle Objects......... 219
Contents IX
4.3.5 Clearing Away Obstacle Objects on a Parking Area... 225
4.3.6 Inspection and/or Manipulation of a Target Object... 231
4.3.7 Monitoring the Task-Solving Process................ 237
4.3.8 Overall Task-Specific Configuration of Modules....... 238
4.4 Basic Mechanisms for Camera–RobotCoordination ......... 240
4.4.1 Camera–Manipulator Relation for One-Step Control .. 240
4.4.2 Camera–Manipulator Relation for Multi-step Control . 245
4.4.3 Hand Servoing for Determining the Optical Axis ..... 248
4.4.4 Determining the Field of Sharp View ............... 250
4.5 Summary and Discussion of the Chapter .................. 252
5. Summary and Discussion ................................. 255
5.1 Developing Camera-Equipped Robot Systems .............. 255
5.2 Rationale for the Contents of This Work .................. 258
5.3 Proposals for Future Research Topics ..................... 260
Appendix 1: Ellipsoidal Interpolation ......................... 263
Appendix 2: Further Behavioral Modules..................... 265
Symbols ...................................................... 269
Index......................................................... 273
References.................................................... 277
1. Introduction
The first chapter presents an extensive introduction to the book by start-
ing with the motivation. Next, the Robot Vision paradigm is characterized
and confrontedwith the field of Computer Vision. RobotVision is the indis-
putable kernel of Autonomous Camera-Equipped Robot Systems. For the de-
velopmentofsuchnew-generationrobotsystemsthe importantroleofvisual
demonstration and learning is explained. The final section gives an overview
to the chapters of the book.
1.1 Need for New-Generation Robot Systems
We briefly describe present state and problems of robotics, give an outlook
on trends of research and development, and summarize the specific novelty
contributed in this book.
Present State of Robotics
Industrial robots carry out recurring simple tasks in a fast, accurate and
reliablemanner.Thisistypicallythecaseinapplicationsofseriesproduction.
The environment is customized in relation to a fixed location and volume
occupied by the robot and/or the robot is built such that certain spatial
relationswithafixedenvironmentarekept.Task-relevanteffectortrajectories
must be planned perfectly during an offline phase and unexpected events
must not occur during the subsequent online phase. Close-range sensors are
utilized (if at all)for a carefulcontrolofthe effectorsatthe targetpositions.
Generally,sophisticatedperceptiontechniquesandlearningmechanisms,e.g.
involving Computer Vision and Neural Networks, are unnecessary due to
customized relations between robot and environment.
Inthenineteeneightiesandninetiesimpressiveprogresshasbeenachieved
in supporting the development and programming of industrial robots.
• CAD (Computer Aided Design) tools are used for convenient and rapid
designing of the hardware of robot components, for example, shape and
size of manipulator links, degrees-of-freedomof manipulator joints, etc.
J. Pauli: Learning-Based Robot Vision, LNCS 2048, pp. 1-24, 2001.
 Springer-Verlag Berlin Heidelberg 2001