Table Of ContentStatistics and Computing
Series Editors:
J. Chambers
W.Eddy
W. Hărdle
S. Sheather
L. Tierney
Springer Science+Business Media, LLC
Statistics and Computing
HardlelKlinke/Turlach: XploRe: An Interactive Statistical
Computing Environment
Venables/Ripley: Modern Applied Statistics with S-Plus
6 Ruanaidh/Fitzgerald: Numerical Bayesian Methods Applied
to Signal Processing
6
Joseph 1.K. Ruanaidh
William 1. Fitzgerald
Numerical
Bayesian Methods
Applied
to Signal Processing
With 118 Illustrations
i
Springer
Joseph 1.K. 6 Ruanaidh
William 1. Fitzgerald
Department of Engineering
University of Cambridge
Trumpington Street
Cambridge CB2 lPZ
United Kingdom
Series Editors:
1. Chambers W.Eddy W. HărdJe
AT&T BeII Laboratories Department of Statistics Institut filr Statistik und Okonometrie
Murray HiII, NJ 07974 Carnegie Mellon University Humboldt-Universităt zu Berlin
USA Pittsburgh, PA 15213, USA D-I 0178 Berlin, Germany
S. Sheather L. Tiemey
Australian Graduate School School of Statistics
of Management University of Minnesota
Kensington, New South WaJes 2033 Minneapolis, MN 55455
Australia USA
Library of Congress Cataloging-in-Publication Data
6 Ruanaidh, Joseph J. K.
Numerical Bayesian methods applied to signal processing / Joseph
J.K. 6 Ruanaidh, William J. Fitzgerald.
p. cm.
Includes bibliographical references and index.
1. Signal processing-Statistical methods. 2. Bayesian
statistical decision theory. 1. Fitzgerald, William 1. II. Title.
TK5102.9.078 1996
621.382'2'OI51954-dc20 95-44635
Printed on acid-free paper.
© 1996 Springer Science+Business Media New York
Originally published by Springer-Verlag New York, Inc.in 1996
Softcover reprint of the hardcover 1s t edition 1996
AII rights reserved. This work may not be translated or copied in whole or in part without the
written permission of the publisher Springer Science+BusÎness Media, LLC,
except for brief excerpts in connection with reviews or scholarly
analysis. Use in connection with any form of information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known or hereaf
ter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even
if the former are not especially identified, is not to be taken as a sign that such names, as
understood by the Trade Marks and Merchandise Marks Act, may accordingly be used freely
byanyone.
Production managed by Frank Ganz; manufacturing supervised by Joe Quatela.
Photocomposed pages prepared from the author's LATEX files.
987654321
ISBN 978-1-4612-6880-2 ISBN 978-1-4612-0717-7 (eBook)
DOI 10.1007/978-1-4612-0717-7
Dar muintir
To our families
Acknowledgments
"Ar scath a cheile a mhaireann na daoine"
Thanks are due to Jebu Rajan, Robin Morris, Simon Godsill, Miao Dan
Wu, Jacek Noga, Teng Joon Lim and Ani! Kokaram for proofreading sec
tions of this book. We would also like to thank the network team - Anil
Kokaram, Pete Wilson, Ray Auchterlounie and Ian Calderbank for doing
such a sterling job in managing the lab Novell network and Alex "The
Jb.TEX Baron" Stark for maintaining the document preparation software so
well.
We also wish to acknowledge Radford Neal, Sibusiso Sibisi, Simon God
sill, Thomas Reiss, William Graham, Anthony Quinn and David MacKay
for their help and advice.
We are delighted to thank the Engineering Department, University of
Cambridge for providing such a wonderful working environment where all
of this work was carried out. We would like to thank the Department of
Electronic and ElectricalEngineering, Trinity College, UniversityofDublin
for the generous use of facilities during a critical stage in the final phases
ofthe book's preparation.
6
Joseph J.K. Ruanaidh
William J. Fitzgerald
Cambridge 1995
Glossary
AIC Akaike's Information Criterion
AR Autoregressive
ARMA Autoregressive Moving Average
BFGS Broyden Fletcher Goldfarb Shanno
cdf Cumulative Distribution Function
CG Condensed Gibbs Sampler
cpdf Conditional Probability Density Function
DFP Davidon Fletcher Powell
DVM Dummy Variable Method
EM Expectation Maximisation
FG Full Gibbs Sampler
HMC Hybrid Monte Carlo
GA Genetic Algorithm
GPL General Piecewise Linear
i.i.d. Independent Identically Distributed
LS Least Squares
MA Moving Average
MAP Maximum a Posteriori
MCMC Markov Chain Monte Carlo (MC2)
MCMCMC Markov Chain Monte Carlo Model Comparison(MC3)
MDL Minimum Description Length
ML Maximum Likelihood
PC Programmable Computer
SIR Sampling Importance Resampling
SNR Signal to Noise Ratio
pdf Probability Density Function
VFSR Very Fast Simulated Reannealing
VM Variable Metric
Notation
The following notational conventions are used in the main text:
a scalar
b column vector
bT transpose of b
th
b i element of b
i
R matrix
R-1 inverse ofmatrix R
IRI determinant ofmatrix R
R jth element ofthe ith row of R
ij
I identity matrix
E(.) expectation operator
p(x) joint pdffor the elements ofx
p(x, y) joint pdffor the elements ofx and the elements ofy
P(x Iy) joint cpdffor the elements ofx given the elements ofy
P(A) probability ofevent A
I prior information
N number ofdata points
N! N!=N.(N - l).(N - 2)···3.2.1 where N is an integer
V' gradient operator
o(.)
order ofapproximation
Ixl x scalar; absolute value ofx
~M M dimensional space of real numbers
f (x)lx function f (x) evaluated at the supremum x
(0,1] real interval from 0 to 1, including 1 but excluding 0
x (- f (x) x is random number drawn from pdf f (x)
Contents
Dedication v
Acknowledgments vi
Glossary vii
Notation viii
1 Introduction 1
2 Probabilistic Inference in Signal Processing 6
2.1 Introduction. . . . . . . . . 6
2.2 The likelihood function. . . 7
2.2.1 Maximum likelihood 8
2.3 Bayesian data analysis 9
2.4 Prior probabilities ... 10
2.4.1 Flat priors. . . . . 10
2.4.2 Smoothness priors 11
2.4.3 Convenience priors 12
2.5 The removal of nuisance parameters 12
2.6 Model selection using Bayesian evidence 13
2.6.1 Ockham's razor .......... 14
2.7 The general linear model . ........ 15
2.8 Interpretations ofthe general linear model . 17
x Contents
2.8.1 Features . 17
2.8.2 Orthogonalization . 17
2.9 Example ofmarginalization 18
2.9.1 Results . 19
2.10 Example ofmodel selection 20
2.10.1 Closed form expression for evidence 21
2.10.2 Determining the order ofa polynomial 22
2.10.3 Determining the order of an AR process 22
2.11 Concluding remarks . 24
3 Numerical Bayesian Inference 26
3.1 The normal approximation 27
3.1.1 Effect of number ofdata on the likelihood function 27
3.1.2 Taylor approximation . . . 28
3.1.3 Reparameterization 29
3.1.4 Jacobian oftransformation 31
3.1.5 Normal approximation to evidence 31
3.1.6 Normal approximation to the marginal density 32
3.1.7 The delta method 33
3.2 Optimization . . . . . . . 34
3.2.1 Local algorithms . 35
3.2.2 Global algorithms 38
3.2.3 Concluding remarks 41
3.3 Integration 42
3.4 Numerical quadrature . . . 43
3.4.1 Multiple integrals. . 44
3.5 Asymptotic approximations 46
3.5.1 The saddlepoint approximation and Edgeworth series 47
3.5.2 The Laplace approximation 47
3.5.3 Moments and expectations 48
3.5.4 Marginalization....... 49
3.6 The Monte Carlo method . . . . . 51
3.7 The generation of random variates 54
3.7.1 Uniform variates . . . . . . 54
3.7.2 Non-uniform variates. . . . 55
3.7.3 Transformation ofvariables 55
3.7.4 The rejection method ... 56
3.7.5 Other methods . . . . . . . 56
3.8 Evidence using importance sampling 57
3.8.1 Choice ofsampling density . 57
3.8.2 Orthogonalization using noise colouring 60
3.9 Marginal densities . . 61
3.9.1 Histograms . . . . . . . . . . 61
3.9.2 Jointly distributed variates . 62
3.9.3 The dummy variable method 62
