Motion Analysis and Image Sequence Processing

498 Pages·1993·25.206 MB·English

by  P. Anandan, J. R. Bergen, K. J. Hanna (auth.), M. Ibrahim Sezan, Reginald L. Lagendijk (eds.)

MOTION ANALYSIS AND IMAGE
SEQUENCE PROCESSING
THE KLUWER INTERNATIONAL SERIES
IN ENGINEERING AND COMPUTER SCIENCE
VLSI, COMPUTER ARCHITECTURE AND
DIGITAL SIGNAL PROCESSING
Consulting Editor
Jonathan Allen
Latest Titles
Introduction to the Design of Transconductor-Capacitor Fjlte~,
1. E. Kardonlchik
ISBN: 0-7923-9195-0
The Synthesis Approach to Digital System Design, P. Michel, U. Laulher, P. Duzy
ISBN: 0-7923-9199-3
Fauit COI'ering Problems in ReconJigurable VLSI Systems, R.Libeskind-Hadas,
N. Hassan, J. Cong, P. McKinley, C. L Liu
ISBN: Q. 7923-9231-0
High Level Synthesis oJ ASICs Under Timing and Synchronization Constraints
D.C. Ku, G. De Micheli
ISBN: Q. 7923-9244-2
The SECD Microprocessor, A VeriJication Case Study, B.T. Graham
ISBN: Q. 7923-9245-0
Field-Programmable Gate Arrays, S.D. Brown, R. J. Francis, 1. Rose,
Z.G. Vranesic
ISBN: 0-7923-9248-5
Anatomy ofA Silicon Compiler, R.W. Brodersen
ISBN: Q. 7923-9249-3
Electronic CAD Frameworks, TJ. Barnes, D. Harrison, A.R. Newton,
R.L Spickelmier
ISBN: Q. 7923-9252-3
VHDL Jor Simulation, Synthesis and Formal Proofs oJH ardware, J. Mcrmel
ISBN: Q. 7923-9253-1
Wavelet Theory and its Applications, R. K. Young
ISBN: 0-7923-9271-X
Digital mCMOS Integrated Circuit Design, S.H.K. Embabi, A. Bellaouar,
M.I Elmasry
ISBN: Q. 7923-9276-0
Design Automation for Timing-Dril'en Layout Synthesis, S. S. Sapalnekar, S. Kang
ISBN: 0-7923-9281-7
Acoustical and Em'ironmental Robustness in Automatic Speech Recognition,
A. Acero
ISBN: Q. 7923-9284-1
Logic Synthesis and Optimization, T. Sasao
ISBN: 0-7923-9308-2
Sigma Della M()(II/Iators: Nonlinear Decoding Algorithms and Stability Analysis,
S. Hein, A. Zakhor
ISBN: 0-7923-9309-0
High-Lel'el Synthesis for Real-Time Digital Signal Processing: The
Cathedral-II Silicon Compiler
J. Vanhoof, K. Van Rompaey, I. Boisens, G. Goosens, H. De Man
ISBN; Q. 7923-9313-9
MOTION ANALYSIS AND IMAGE
SEQUENCE PROCESSING
edited by
M. Ibrahim Sezan
Eastman Kodak Company
Reginald L Lagendijk
Delfi University of Technology
"~.
SPRINGER SCIENCE+BUSINESS MEDIA, LLC
Library of Congrcss Cataloglng-in-Publlcatlon Data
Motion analysis and image sequence processing / edited by M. Ibrahim
Sezan, Reginald L Lagendijk.
p. cm. -- (The Kluwer international series in engineering and
computer science ; SECS 0220. VLBI, computer architecture and
digital signal processing)
Includes bibliographical references and index.
ISBN 978-1-4613-6422-1 ISBN 978-1-4615-3236-1 (eBook)
DOI 10.1007/978-1-4615-3236-1
1. Image processing. 2. Motion perception (Vis ion) 1. Sezan, M.
Ibrahim. Il. Lagendijk, Reginald L. III. Series: Kluwer
international series in engineering and computer science ; SECS
0220. IV. Series: Kluwer international series in engineering and
computer science. VLSI, computer architecture and digital signal
processing.
TA1632 . M68 1993
621 . 39 '9--dc20 92-46330
CIP
Copyright © 1993 by Springer Science+Business Media New York
Originally published by Kluwer Academic Publishers in 1993
Softcover reprint ofthe hardcover Ist edition 1993
AII rights reserved. No part of this publication may be reproduced, stored in a retrieval
system or transmitted in any form or by any means, mechanical, photo-copying, record ing,
or otherwise, without the prior written permission of the publisher, Springer Science
+Business Media, LLC.
Printed on acid-free paper.
Contents
List of Contributors vii
Preface xi
Chapter 1
Hierarchical Model-Based Motion Estimation
P. Anandan, J.R. Bergen, KJ. Hanna and R. Hingomni 1
Chapter 2
An Estimation Theoretic Perspective on Image Processing
and the Calculation of Optical Flow
T.M. Chin, M.R. Luettgen, W.C. Karl and A.S. Willsky 23
Chapter 3
Estimation of 2-D Motion Fields from Image Sequences
with Application to Motion-Compensated Processing
E. Dubois and J. Konrad 53
Chapter 4
Edge-Based 3-D Camera Motion Estimation with
Application to Video Coding
A. Zakhor and F. Lad 89
Chapter 5
Motion Compensation: Visual Aspects, Accuracy,
and Fundamental Limits
B. Girod 125
Chapter 6
Motion Field Estimators and Their Application to Image
Interpolation
S. Tubaro and F. Rocca 153
Chapter 7
Subsampling of Digital Image Sequences Using Motion
Information
R.A.F. Bel/or, R.L. Lagelldijk and J. Biemond 189
vi
Chapter 8
Image Sequence Coding Using Motion-Compensated
Subband Decomposition
A. Nicoulin, M. Mattavelli, W. Li, A. Basso, A. Popat,
and M. [(unt 225
Fourth
Chapter 9
Vector Quantization for Video Data Compression
R.M. Mersereau, M.J. T. Smith, C.S. [(im, F. [(ossentini,
and K.K. Truong 257
Chapter 10
Model-Based Image Sequence Coding
M. Buck and N. Diehl 285
Chapter 11
Human Facial Motion Analysis and Synthesis
with Applications to Model-Based Coding
K. Aizawa, C.S. Choi, H. Harashima and T.s. Huang 317
Chapter 12
Motion Compensated Spatiotemporal Kalman Filter
J. W. Woods and J. [(im 349
Chapter 13
Multiframe Wiener Restoration of Image Sequences
M.K. Ozkan, M.l. Sezan, A. T. Erdem, and A.M. Tekalp 375
Chapter 14
3-D Median Stru~tures for Image Sequence Filtering and
Coding
T. Viera and Y. Neuvo 411
Chapter 15
Video Compression for Digital ATV Systems
J.G. Apostolopoulos and J.S. Lim 447
Index 483
List of Contributors
K. Aizawa C-S. Choi
Electrical Engineering Department Dept. of Information Communication
University of Tokyo Myong Ji University
Tokyo, 113 Japan Kyunggido, 499-728, Korea
P. Anandan N. Diehl
David Sarnoff Research Center Daimler-Benz AG
Princeton NJ 08543-5300, USA D-7900 Ulm, Germany
J. G. Apostolopoulos E. Dubois
Advanced Television Signal Proc. Group INRS-Telecommunications
MIT Verdun, Canada H3E IH6
Cambridge MA 02139, USA
A. T. Erdem
A. Basso Eastman Kodak Company
Signal Processing Laboratory Rochester NY 14650-1816, USA
EPFL-Ecublens
CH-1015 Lausanne, Switzerland B. Girod
Academy of Media Arts Cologne
R. A. F. Belfor W-5000 Koln 1, Germany
Department of Electrical Engineering
Delft University of Technology K. Hanna
2600 GA Delft, The Netherlands David Sarnoff Research Center
Princeton NJ 08543-5300, USA
J. R. Bergen
David Sarnoff Research Center H. Harashima
Princeton NJ 08543-5300, USA Electrical Engineering Department
University of Tokyo
J. Biemond Tokyo, 113 Japan
Department of Electrical Engineering
Delft University of Technology R. Hingorani
2600 GA Delft, The Netherlands AT&T Bell Laboratories
Murray Hill NJ 08544, USA
M. Buck
Daimler-Benz AG T. S. Huang
D-7900 Ulm, Germany Coordinated Science Laboratory
University of Illinois
T. M. Chin Urbana IL 61801, USA
Rosenstiel School of Marine Science
University of Miami
Miami FL 33149, USA
viii
W. C. Karl J. Lim
Laboratory for Information and Decision Advanced Television Signal Proc. Group
Systems MIT
MIT Cambridge MA 02139, USA
Cambridge MA 02139, USA
M. Luettgen
C.S. Kim Laboratory for Information and Decision
Department of Electrical Engineering Systems
University of Washington MIT
Seattle WA 98195, USA Cambridge MA 02139, USA
J. Kim M. Mattavelli
ECSE Department Signal Processing Laboratory
Rensselaer Polytechnic Institute EPFL-Ecublens
Troy NY 12180-3590, USA CH-1015 Lausanne, Switzerland
J. Konrad R. M. Mersereau
INRS-Telecommunications School of Electrical Engineering
Verdun, Canada II3E 1H6 Georgia Institute of Technology
Atlanta, GA 30332-0250, USA
F. Kossentini
School of Electrical Engineering Y. Neuvo
Georgia Institute of Technology Signal Processing Laboratory
Atlanta, GA 30332-0250, USA Tampere University of Technology
SF-33101 Tampere, Finland
M. Kunt
Signal Processing Laboratory A. Nicoulin
EPFL-Ecublens Signal Processing Laboratory
CII-IOI5 Lausanne, Switzerland EPFL-Ecublens
CII-1015 Lausanne, Switzerland
R. L. Lagendijk
Department of Electrical Engineering 1\-1. K. Ozkan
Delft University of Technology Thomson Consumer Electronics
2600 GA Delft, The Netherlands Indianapolis IN 46201-2598, USA
F.Lari A. Popat
Electrical Engineering Department The Media Laboratory
University of California MIT
Berkeley CA 94720, USA Cambridge MA 02139, USA
W.Li F. Rocca
Signal Processing Laboratory Dipartimento di Elettronica
EPFL-Ecublens e Informazione
CII-IOI5 Lausanne, Switzerland Politecnico di Milano
20133 Milano, Italy
IX
M. I. Sezan
Eastman Kodak Company
Rochester NY 14650-1816, USA
M. J. T. Smith
School of Electrical Engineering
Georgia Institute of Technology
Atlanta, GA 30332-0250, USA
A. M. Tekalp
Department of Electrical Engineering
University of Rochester
Rochester NY 14627, USA
K. K. Truong
Atlanta Signal Processors
Atlanta GA 30332, USA
S. Tubaro
Dipartimento di Elettronica
e Informazione
Politecnico di Milano
20133 Milano, Italy
T. Viero
Signal Processing Laboratory
Tampere University of Technology
SF-33101 Tampere, Finland
A. Willsky
Laboratory for Information and Decision
Systems
MIT
Cambridge MA 02139, USA
J. W. Woods
ECSE Department
Rensselaer Polytechnic Institute
Troy NY 12180-3590, USA
A. Zakhor
Electrical Engineering Department
University of California
Berkeley CA 94720, USA
Preface
An image or video sequence is a series of two-dimensional (2-D) images sequen
tially ordered in time. Image sequences can be acquired, for instance, by video,
motion picture, X-ray, or acoustic cameras, or they can be synthetically gen
erated by sequentially ordering 2-D still images as in computer graphics and
animation. The use of image sequences in areas such as entertainment, visual
communications, multimedia, education, medicine, surveillance, remote control,
and scientific research is constantly growing as the use of television and video
systems are becoming more and more common. The boosted interest in digital
video for both consumer and professional products, along with the availability
of fast processors and memory at reasonable costs, has been a major driving
force behind this growth.
Before we elaborate on the two major terms that appear in the title of
this book, namely motion analysis and image sequence processing, we like to
place them in their proper contexts within the range of possible operations that
involve image sequences. In this book, we choose to classify these operations
into three major categories, namely (i) image sequence processing, (ii) image
sequence analysis, and (iii) visualization. The interrelationship among these
three categories is pictorially described in Figure 1 below in the form of an "image
sequence triangle". Image sequence processing, which is essentially a sequence-in
and sequence-out operation, refers to the operations of filtering, spatiotemporal
interpolation and subsampling, and compression of image sequences, aimed at
improving the (visual) image quality, conversion between different video formats,
and bandwidth-efficient representation of image sequences, respectively.
The second category is image sequence analysis, referring to those opera
tions that generate some type of data from image sequences for the purpose of
information retrieval or interpretation. Motion analysis is an important branch
of image sequence analysis. It refers to estimation of the image motion (the pro
jection of the 3-D motion onto the 2-D image plane, or optical flow), inference
of the 3-D object motion, depth information, as well as the surface characteris
tics of objects present in the actual 3-D scene. Motion information is utilized
not only in various analysis tasks such as segmentation, pattern recognition and
tracking, and scene interpretation, but is also instrumental in image sequence
processing, as pointed out in Figure 1. The third category of operations that
involves image sequences is visualization. It is concerned with generating image
sequences on the basis of data which is not readily in the form of an image
sequence. Computer graphics, virtual reality, and several emerging methods for
scientific data visualization belong to this category. This book focuses on mo
tion analysis and image sequence processing, and the use of motion analysis in
developing efficient and powerful image sequence processing algorithms.
At this point, one may raise the following questions: Since image sequences