Random Processes for Engineers: A Primer

208 Pages·2017·4.983 MB·English

by  Arthur David Snider

Random Processes
for Engineers
A P R I M E R

Random Processes
for Engineers
A P R I M E R
Arthur David Snider
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Taylor & Francis Group
Boca Raton, FL 33487-2742
© 2017 by Taylor & Francis Group, LLC
MATLAB® is a trademark of The MathWorks, Inc. and is used with permission. The MathWorks does not war-
rant the accuracy of the text or exercises in this book. This book’s use or discussion of MATLAB® software or
related products does not constitute endorsement or sponsorship by The MathWorks of a particular pedagogical
approach or particular use of the MATLAB® software.
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
Version Date: 20161111
International Standard Book Number-13: 978-1-4987-9903-4 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have
been made to publish reliable data and information, but the author and publisher cannot assume responsibility
for the validity of all materials or the consequences of their use. The authors and publishers have attempted to
trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if
permission to publish in this form has not been obtained. If any copyright material has not been acknowledged
please write and let us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system, with-
out written permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com
(http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive,
Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration
for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate
system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents
Preface .......................................................................................................................ix
Author .......................................................................................................................xi
Chapter 1 Probability Basics: A Retrospective .....................................................1
1.1 What Is “Probability”? ..............................................................1
Online Sources .....................................................................................2
1.2 The Additive Law ......................................................................3
1.3 Conditional Probability and Independence ...............................5
Summary: Important Laws of Probability ............................................8
1.4 Permutations and Combinations ................................................9
1.5 Continuous Random Variables ................................................10
Summary: Important Facts about Continuous Random Variables......15
1.6 Countability and Measure Theory ...........................................16
1.7 Moments ..................................................................................18
Summary: Important Facts about Expected Value and Moments .......21
1.8 Derived Distributions ...............................................................21
Summary: Important Facts about Change of Variable ........................24
1.9 The Normal or Gaussian Distribution .....................................24
Summa ry: Important Equations Involving the Normal
(Gaussian) Distribution ..........................................................28
1.10 Multivariate Statistics ..............................................................28
1.11 The Bivariate Probability Density Functions ..........................30
Online Sources ...................................................................................34
Summary: Important Equations for Bivariate Random Variables ......35
1.12 The Bivariate Gaussian Distribution........................................35
Online Sources ...................................................................................38
Summary of Important Equations for the Bivariate Gaussian ............39
1.13 Sums of Random Variables ......................................................39
Online Sources ...................................................................................43
Summary of Important Equations for Sums of Random Variables ....44
1.14 The Multivariate Gaussian .......................................................44
1.15 The Importance of the Normal Distribution ............................46
Exercises .............................................................................................47
Chapter 2 Random Processes ..............................................................................55
2.1 Examples of Random Processes ..............................................55
2.2 The Mathematical Characterization of Random Processes .....61
Summary: The First and Second Moments of Random Processes .....64
v
vi Contents
2.3 Prediction: The Statistician’s Task ...........................................67
Exercises .............................................................................................69
Chapter 3 Analysis of Raw Data .........................................................................75
3.1 Stationarity and Ergodicity ......................................................75
3.2 The Limit Concept in Random Processes ...............................77
3.3 Spectral Methods for Obtaining Autocorrelations ...................79
3.4 Interpretation of the Discrete Time Fourier Transform ...........82
3.5 The Power Spectral Density ....................................................83
3.6 Interpretation of the Power Spectral Density ..........................89
3.7 Engineering the Power Spectral Density .................................91
3.8 Back to Estimating the Autocorrelation ..................................95
Online Sources ...................................................................................99
3.9 Optional Reading the Secret of Bartlett’s Method ..................99
3.10 Spectral Analysis for Continuous Random Processes ...........104
Summary: Spectral Properties of Discrete and Continuous
Random Processes .................................................................105
Exercises ...........................................................................................105
Chapter 4 Models for Random Processes .........................................................111
4.1 Differential Equations Background .......................................111
4.2 Difference Equations .............................................................112
4.3 ARMA Models ......................................................................115
4.4 The Yule–Walker Equations...................................................116
Online Sources .................................................................................118
4.5 Construction of ARMA Models ............................................118
4.6 Higher-Order ARMA Processes ............................................119
4.7 The Random Sine Wave.........................................................122
Online Sources .................................................................................124
4.8 The Bernoulli and Binomial Processes ..................................125
Summary: Bernoulli Process ............................................................125
Online Sources .................................................................................126
Summary: Binomial Process ............................................................128
4.9 Shot Noise and the Poisson Process ......................................128
Online Sources and Demonstrations ................................................136
4.10 Random Walks and the Wiener Process ................................136
Online Sources .................................................................................138
4.11 Markov Processes ..................................................................139
Online Sources .................................................................................144
Summary: Common Random Process Models .................................144
Exercises ...........................................................................................146
Contents vii
Chapter 5 Least Mean-Square Error Predictors ................................................151
5.1 The Optimal Constant Predictor ............................................151
5.2 The Optimal Constant-Multiple Predictor .............................152
5.3 Digression: Orthogonality .....................................................152
5.4 Multivariate LMSE Prediction: The Normal Equations ........154
5.5 The Bias .................................................................................156
Online Sources .................................................................................157
5.6 The Best Straight-Line Predictor ...........................................157
5.7 Prediction for a Random Process ..........................................159
5.8 Interpolation, Smoothing, Extrapolation,
and Back-Prediction ..............................................................160
5.9 The Wiener Filter ...................................................................161
Online Sources .................................................................................166
Exercises ...........................................................................................166
Chapter 6 The Kalman Filter ............................................................................169
6.1 The Basic Kalman Filter ........................................................169
6.2 Kalman Filter with Transition: Model and Examples ...........171
Digression: Examples of the Kalman Model .........................172
Online Sources .................................................................................173
6.3 Scalar Kalman Filter with Noiseless Transition ....................176
6.4 Scalar Kalman Filter with Noisy Transition ..........................177
6.5 Iteration of the Scalar Kalman Filter .....................................179
6.6 Matrix Formulation for the Kalman Filter .............................182
Online Sources .................................................................................188
Exercises ...........................................................................................189
Index ......................................................................................................................193

Preface
There are a lot of authoritative, comprehensive, and axiomatically correct books on
random processes, but they all suffer from lack of accessibility for engineering stu-
dents starting out in the area. This book fills the gap between the undergraduate engi-
neering statistics course and the rigorous approaches. A refresher on the prerequisite
topics from basic statistics is given in the first chapter.
Some of the features that distinguish the book from other resources are the
following:
1. “Probability spaces” based on measure theory and sigma-algebras are
appropriate for addressing some sticky philosophical questions (“How can
every possible outcome have probability zero while one of them certainly
occurs?”), but this esoteric machinery is not necessary for solving practical
problems. This book only discusses them sufficiently to introduce the issues
and whet the readers’ appetite for rigorous approaches (Section 1.6).
2. The Kalman filter is regarded as formidable by most engineers because it is
traditionally expostulated in full-blown matrix form. This book introduces
it in a very simple scalar context, where the basic strategy is transparent, as
is its extension to the general case (Chapter 6).
3. The book is exceptional in that it distinguishes between the science of
extracting statistical information from raw data (Chapter 3)—for example,
a time series about which nothing is known a priori—and that of analyzing
specific statistical models (Chapter 4). The former motivates the concepts
of statistical spectral analysis (such as the Wiener–Khintchine theory), and
the latter applies and interprets them in specific physical contexts.
4. The book’s premise, throughout, is that new techniques are best introduced
by specific, low-dimensional examples, rather than attempting to strive for
generality at the outset; the mathematics is easier to comprehend and more
enjoyable. Specific instances are the derivations of the Yule–Walker equa-
tions (Section 4.4), the normal equations (Section 5.4), and the Wiener filter
(Section 5.9).
In short, this book is not comprehensive and not rigorous, but it is unique in its
simplified approach to the subject. It “greases the skids” for students embarking on
advanced reading, while it provides an adequate one-semester survey of random pro-
cesses for the nonspecialist.
Supplementary material—selected answers, examples, exercises, insights, and
errata—will be made available as they are generated, at the author’s web site: http://
ee.eng.usf.edu/people/snider2.html.
ix