Statistical Methods for Image and Signal Processing by PHILIP ANDREW SALLEE BS

140 Pages·2004·3.62 MB·English

Statistical Methods for Image and Signal Processing
by
PHILIP ANDREW SALLEE
B.S. (Biola University) 1993
M.S. (University of California, Davis) 2002
DISSERTATION
Submitted in partial satisfaction of the requirements for the degree of
DOCTOR OF PHILOSOPHY
in
Computer Science
in the
OFFICE OF GRADUATE STUDIES
of the
UNIVERSITY OF CALIFORNIA
DAVIS
Approved:
Professor Bruno A. Olshausen (Chair)
Professor Zhaojun Bai
Professor Naoki Saito
Committee in Charge
2004
i
Statistical Methods for Image and Signal Processing
Copyright c 2004
°
by
Philip Andrew Sallee
ii
To Trisha, my love, who always encourages me to follow my
dreams. Thanks for listening to my wild ideas for countless hours.
Without your constant love and support, this work could not have
happened. You are my true companion.
And to my daughters Charisse and Karina. May you dream big
and never let go, and never stop wondering.
iii
Acknowledgements
So many people have impacted this work, that I cannot begin to thank them all.
But I want to particularly thank my advisor, Bruno Olshausen, for countless hours
spent mentoring me, listening to my ideas, guiding me and reviewing my work. This
work is a result of his vision which I caught and hope to pass on to others. I also
want to thank the rest of my thesis committee, Naoki Saito and Zhaojun Bai, for their
patience, support and encouragement until this was completed. Thanks also, to those
in the Computer Science department who have been so supportive of my efforts. A
specialthankyouisduetoTyeStallardforthatemailwhichinspiredmysteganography
work, and to Eero Simoncelli for providing source code, ideas and discussions.
Many thanks to those at the Center for Neuroscience who helped out on many
occasions, reviewed my work, contributed ideas, and sometimes just kept things in-
teresting: Issac Trotts, Maysha Mohamedi, Scott Murray, and Jeff Colombe. Special
thanks go to Jeff Johnson for his help with the EEG data, Matthew Godwin for inves-
tigating matching pursuit algorithms and Surya De who helped to write JPEG tools.
Kevin O’Conner, thank you for many discussions and for providing the natural sound
database.
ManythanksareduetomyfamilywithoutwhosesupportIwouldnothavefinished.
To my wife and children, thank you for your sacrifices and your encouragement. Mom
and Dad, thank you for your loving support and advice through all these years. And
my brother, Greg, thanks for reading my papers and for lots of interesting discussions.
Finally, thanks be to God, author and designer of all things. You have given us more
to explore than we could begin to understand after many lifetimes, and I am richly
blessed to have such wonderful people in my life to explore it with.
iv
Abstract
Statisticalmethodsprovideaprincipledmeansforsolvingmanytypesofprob-
lems which require the estimation of missing or uncertain information. This dis-
sertation discusses methods for adapting statistical models to images, sounds and
other types of signals for applications in image and signal processing. Wavelets
provide a multi-scale representation which has been shown to be well suited for
describing many naturally occurring signals. These are typically designed by
hand based on certain mathematical properties and may not achieve the best
match to the data. We describe an approach for using an overcomplete wavelet
framework as part of a generative statistical model with a sparse prior placed on
the wavelet coefficients. The wavelet functions are adapted to a given dataset
by maximizing the average log likelihood of the model. This is demonstrated for
natural images, sounds, and EEG data. The learned representations are shown to
have a higher degree of sparsity than other wavelet bases. This statistical frame-
work also provides a principled approach for performing certain types of signal
estimation, such as denoising, in terms of a statistical inference process. We ex-
plore two inference methods for the overcomplete wavelet models presented: A
Gibbssamplingmethod, andagreedyoptimizationprocedureknownasmatching
pursuit. We also demonstrate how a statistical model may be applied to a form of
secure information hiding, known as steganography, in which the objective is to
hide information in an image or some other media so that it cannot be detected
without a cryptographic “key”. This model-based approach provides a means
for maximizing the capacity of stored information while obtaining provably se-
cure steganography insofar as the model is accurate. Using this methodology, a
steganography method is proposed for JPEG images which achieves higher em-
bedding efficiency and message capacity than previous methods, while remaining
secure against first order statistical attacks. Methods for applying statistical
models for steganalysis, the art of detecting steganographic messages, are also
presented.
v
Contents
Contents vi
List of Tables ix
List of Figures x
1 Introduction 1
1.1 Generative models . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Sparse coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2.1 Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.2.2 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.2.3 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2.4 Relation to other methods . . . . . . . . . . . . . . . . . . 17
1.3 Wavelets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1.4 Information Hiding . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.5 Outline of Dissertation . . . . . . . . . . . . . . . . . . . . . . . . 23
2 Adapting Wavelets to Natural Images 25
2.1 Wavelet image model . . . . . . . . . . . . . . . . . . . . . . . . . 32
2.1.1 Delta-plus-Gaussian prior . . . . . . . . . . . . . . . . . . 35
2.2 Sampling and Inference . . . . . . . . . . . . . . . . . . . . . . . . 36
2.3 Adapting the model to images . . . . . . . . . . . . . . . . . . . . 39
2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.4.1 One octave scaling . . . . . . . . . . . . . . . . . . . . . . 40
vi
2.4.2 Two octave scaling . . . . . . . . . . . . . . . . . . . . . . 42
2.4.3 Sparsity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.4.4 Denoising . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3 Adapting Wavelet Dictionaries to 1D Signals 52
3.1 Overcomplete wavelet model for 1D signals . . . . . . . . . . . . . 56
3.2 Inference via matching pursuit . . . . . . . . . . . . . . . . . . . . 58
3.2.1 Standard Matching pursuit algorithm . . . . . . . . . . . . 59
3.2.2 Fast matching pursuit algorithm . . . . . . . . . . . . . . . 60
3.2.3 Gibbs sampling versus matching pursuit . . . . . . . . . . 65
3.3 Adapting the model to natural sounds . . . . . . . . . . . . . . . 67
3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4.1 Combined scales . . . . . . . . . . . . . . . . . . . . . . . 70
3.4.2 Separate scales . . . . . . . . . . . . . . . . . . . . . . . . 70
3.4.3 Sound textures . . . . . . . . . . . . . . . . . . . . . . . . 73
3.5 Extending the model to EEG . . . . . . . . . . . . . . . . . . . . 77
3.5.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4 Statistical Methods for Information Hiding 84
4.1 General methodology . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.1.1 Compression and steganography . . . . . . . . . . . . . . . 87
4.1.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.1.3 Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
4.1.4 Implicit models used by current methods . . . . . . . . . . 93
4.1.5 Steganalysis . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.2 Application to JPEG steganography . . . . . . . . . . . . . . . . 96
4.2.1 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.2.2 Embedding method MB1 . . . . . . . . . . . . . . . . . . . 100
4.2.3 Defending against blockiness attacks: Method MB2 . . . . 105
vii
4.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.3 Application to JPEG steganalysis . . . . . . . . . . . . . . . . . . 108
4.3.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
5 Conclusions 117
5.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.2 Implications for future work . . . . . . . . . . . . . . . . . . . . . 119
Bibliography 123
viii
List of Tables
2.1 λ and λ for the learned ψ function in figure 2.6 . . . . . . . . 44
si ui i
2.2 SNR values (in dB) for noisy and denoised images with additive
i.i.d. Gaussiannoiseofstd.dev. σ. “D+G”=Gibbssamplingwith
Delta-plus-Gaussian prior, “S6” = 6-Band Steerable basis, “L6” =
6-Band Learned basis. . . . . . . . . . . . . . . . . . . . . . . . . 50
4.1 Results from embedding maximal length messages with MB1 into
several 512x512 grayscale JPEG images with an embedding step
size of 2. Files were compressed using a JPEG quality factor of 80
and optimized Huffman tables. . . . . . . . . . . . . . . . . . . . . 107
4.2 Results for 22 of the 440 tested stego images. Shown are the rela-
tive number of modifications β, the corresponding message length
ˆ
m, and their estimated values: β and mˆ. . . . . . . . . . . . . . . 113
ˆ
4.3 Percentofimageswithmessagelengthestimationerrorsβ β <Tol,
−
for Tol = .02, .018, .016, .014, .012, and .010. Equivalent percent-
age of total hiding capacity is also shown. . . . . . . . . . . . . . 115
ix
List of Figures
1.1 Relationship of Principal Components Analysis (PCA), Indepen-
dentComponentsAnalysis(ICA),FactorAnalysis(FA),andSparse
Coding (SC) in terms of their assumed generative models. . . . . 18
1.2 A pictoral representation of information hiding problems. Points
of the tetrahedron represent basic competing objectives, forming
a volume of possible trade-off points in which steganography and
digital watermarking exist as points on different faces of the tetra-
hedron. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.1 Wavelet image model. Shown are the coefficients of the first three
levels of a pyramid (l = 0,1,2), with each level split into a number
of different bands (i = 1...B). The highest level (l = 3) is not
shown and contains only one low-pass band. . . . . . . . . . . . . 34
2.2 System diagram for wavelet pyramid decomposition. . . . . . . . . 35
2.3 Prior distribution (dashed), and histogram of samples taken from
the posterior (solid) for a single coefficient. The y-axis is plotted
on a log scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.4 Wavelet functions ψ (x,y) for varying degrees of overcompleteness,
i
and corresponding spectra showing power as a function of spatial
frequency in the 2D Fourier plane. (a) B = 2, (b) B = 4, (c) B = 6. 42
2.5 (a) Wavelet functions ψ (x,y) for 6 bands (B=6) with correspond-
i
ing 2-D spectra. Line plot depicts the rotational average of the
spectra for each filter. (b) Equivalent basis functions for the Steer-
able Pyramid, when constructed for a single octave scaling and 6
bands, their spectra, and rotational averages. . . . . . . . . . . . . 43
x