An Introduction to Information Theory

508 Pages·1961·47.313 MB·English

by  Fazlollah M. Reza

Fazlollah M. Reza
DOVER PUBLICATIONS, INC.
New York
Statistical theory of communication is a broad new field comprised of
methodsfor the studyofthe statisticalproblemsencounteredin all types
of communications. The field embodies many topics such as radar
detection, sources of physical noise in linear and nonlinear systems,
filtering and prediction, information theory, coding, and decision theory.
The theory of probability provides the principal tools for the study of
problems in this field.
Information theory as outlined in the present work is a part of this
broader body of knowledge. This theory, originated by C. E. ~n~tnn.on9
introduced several important new concepts and, although a of
applied communications sciences, has acquired the unique distinction of
opening a new path ofresearch in pure mathematics.
The communication ofinformation is generally of a statistical nature,
and a current theme of information theory is the study of ideal
statistical communication models. The first objective of information
theory is to define different types of sources and channels and to devise
statisticalparametersdescribingtheirindividualandensembleoperations.
The concept ofShannon's communication entropy ofa source and the
transinformation of a channel provide most useful means for studying
simple communication models. In this respect it appears that the con
cept of communication entropy is a type of describing function that is
most appropriate for the statistical models of communications. This is
similar in principle to the way that an impedance function describes a
linear network, or a moment indicates certain properties of a random
variable. The introduction of the concepts of communication entropy,
transinformation, andchannelcapacityisabasic contributionofinforma
tion theory, and these conceptsare ofsuch fundamental significancethat
they may parallel in importance the concepts of power, impedance, and
moment.
Perhaps the most important theoretical result ofinformation theory is
Shannon's fundamental theorem, which implies that it is possible to
communicate information at an ideal rate with utmost reliability in the
presence of "noise." This succinct but deep statement and its conse
quences unfold the limitation and complexity of present and future
iii
iv PREFACE
methods of communications. of the offers many
benefitsto thoseinterestedintheanalysisandsynthesisofcommunication
networks. For this reason we have included several methods of
ofthis theorem (see Chapters4and 12). the reader
is forewarned that the entails much preparation which may prove
to be burdensome.
This book originated about five years ago from the author's lecture
notes on information theory. In presenting the subject to engineers.
the need for preliminary lectures on probability theory was observed.
A course in probability, even now, is not included in the curriculum of a
majority ofengineeringschools. Thisfact motivated the inclusion ofan
introductory treatment of probability for those who wish to pursue the
general study of statistical theory of communications.
The present book, directed toward an engineering audience, has a
threefold purpose:
1. To present elements of modern probability theory (discrete, con
tinuous, and stochastic)
2. To present elements of information theory with emphasis on its
basic roots in probability theory
3. To present elements of coding theory
Thusthisbookisoffered as an introductionto probability,mtormation.
and coding theory. It also provides an adequate treatment of proba
bility theory for those who wish to pursue topics other than information
theory in the field of statistical theory of communications.
One feature of the book is that it requires no formal prerequtsites
except the usual undergraduate mathematics included in an engineering
or science program. Naturally, a willingness to consult other references
or authorities, as necessary, is presumed. The subject is presented in
the light of applied mathematics. The immediate involvement in tech
nological specialities that may solve specific problems at the expense of
a less thorough basic understanding of the theory is thereby avoided.
A most important, though indirect, application of information theory
has been the development of codes for transmission and detection of
information. Codingliterature has grown very rapidly since it, presum
ably, applies to the growing field of data processing. Chapters 4 and
13 present an introduction to coding theory without recourse to the use
of codes.
The book has been divided into four parts: (1) memorylessdiscrete
schemes, (2) memoryless continuum, (3) schemes with memory, and (4)
an outline of some of the recent developments. The appendix contains
some notes which may help to familiarize the reader with some of the
literature in the field. The inclusion of many reference tables and a
bibliography with some 200 entries may also prove to be useful.
PREFACE
The the book is on such basic as
the probability measure associated with sets, space, random
ables, information measure, and capacity. These concepts from
set theory to probability theory and then to information and
theories. The of the to such as
detection, optics, and linguistics was not undertaken. We make no
pretension for usefulness" and immediate of information
H
theory. From an educational standpoint, it appears, however, that the
topics discussed should provide a suitable training ground for communi
cation scientists.
The most rewarding aspect of this undertaking has been the pleasure
oflearningabout a new andfascinating frontier in communications.
working on this book, I came to appreciate fully many subtle points and
ingenious procedures set forth in the papers of the original contributors
to the literature. I trust this attempt to integratethese many contribu
tions will prove of value. Despite pains taken by the author, inac
curacies, original or inherited, may be found. Nevertheless, I the
reader will find this work an existence proof of Shannon's fundamental
theorem; that "information" can be transmitted with a
reliability at a rate close to the channel capacity despite all forms of
"noise."
At anyrate, thereisan eternalseparationbetweenwhatonestrivesfor
and what one actually achieves. AsLeon von Montenaeken
La vie est breve,
Un peu
Un peu de reve,
Et puis-bonsoir.
Fazlollah M. Reza
The author wishes to acknowledge indebtedness to all those who have
directly or indirectly contributed to this book. Special tribute is due to
Dr. C. E. Shannon who primarily initiated information theory.
Dr. P. Elias of the Massachusetts Institute of Technology, has been
generous in undertaking a comprehensive reading and reviewing of the
manuscript. Hiscomments,helpfulcriticism,andstimulatingdiscussions
have been of invaluable assistance.
Dr. L. A. Cote of Purdue University has been very kind to read and
criticize the manuscript with special emphasis upon the material
probability theory. His knowledge of technical Russian literature and
his unlimited patience in reading the manuscript in its various stages of
development have provided a depth that otherwise would not have been
attained.
Thanks are due to Dr. N. Gilbert of Bell Telephone Laboratories
and Dr. J. P. Costas of General Electric Companyfor comments
on the material on coding theory; to Prof. W. W. Harman of Stanford
University for reviewing an early draft of the manuscript; to Mr. L.
Zafiriu of Syracuse University who accepted the arduous task of proof
reading and who provided many suggestions.
In addition numerous scientists have generously provided reprints,
reports, or drafts of unpublished manuscripts. The more recent mate
rial on information theory and coding has been adapted from these cur
rent sources but integrated in our terminology and frame of reference.
An apology is tendered for any omission or failure to reflect fully the
thoughts of the contributors.
the past four years, I had the opportunityto teach and lecture
in this field at Syracuse University, International Business Machines
Corp., General Electric Co., and the Rome Air Development Center..
The keen interest, stimulating discussions, and friendships of the scien
tists of these centers have been most rewarding.
Special acknowledgment is due the United States Air Force
Air Development Center and the Cambridge Research Center for sup
porting several related research projects.
vii
viii ACKNOWLEDGMENTS
am indebted to my colleagues in the ofElectrical
neering at Syracuse University for many helpful discussions and to Mrs.
H. F. Laidlaw and Miss M. J. Phillips for their patient typing of the
manuscript.
I am particularlygratefulto mywifeand family for the patience which
they have shown.
PREFACE iii
CHAPTER 1 Introduction
1-1. Communication Processes.
1-2. A Model for a Communication System . 3
1-3. A Quantitative Measure of Information. 5
1-4. A Binary Unit of Information. 7
1-5. Sketch ofthe Plan. 9
1-6. Main Contributors to Information Theory . 11
1-7. An Outline of Information Theory
Part 1: Discrete Schemes without "" ...
CHAPTER 2 Basic Concepts of Probability
2-1. Intuitive Background . 19
2-2. Sets .
2-3. Operations on Sets . 23
2-4. Algebra of Sets . 24
2-5. Functions 30
2-6. Sample Space 34
2-7. Probability Measure 36
2-8. Frequency of Events . 38
2-9. Theorem of Addition . 40
2-10. Conditional Probability 42
2-11. Theorem of Multiplication. 44
2-12. Bayes's Theorem 46
2-13. Combinatorial Problems in Probability . 49
2-14. Treesand State Diagrams. 52
2-15. Random Variables . 58
2-16. Discrete Probability Functions and Distribution . 59
2-17. Bivariate Discrete Distributions . 61
2-18. Binomial Distribution . 63
2-19. Poisson's Distribution . 65
2-20. Expected Value of a Random Variable . 67
ix
x CONTENTS
3 Basic Concepts of Information
3-1. AMeasureofUncertainty.
3-2. An Intuitive 78
3-3. Formal Requirementsfor theAverage 80
3-4. II Function as a Measure of Uncertainty 82
3-5. An Alternative Proof That the Function
Possesses a Maximum 86
3-6. Sources and Binary Sources 89
3-7. Measure of Information for Two-dimensional Discrete
Finite Probability Schemes. 91
3-8. Conditional Entropies . 94
3-9. ASketchofa Communication Network. 96
3-10. Derivation of the Noise Characteristics ofa Channel. 99
3-11. Some Basic Relationships among Different Entropies.
3-12. A Measure of Mutual Information 104
3-13. Set-theory Interpretation of Shannon's Fundamental
Inequalities 106
3-14. Efficiency, and Channel 108
3-15. CapacityofChannelswith NoiseStructures
3-16. BSC and BEC
3-17. Capacity of Binary Channels. 115
3-18. Binary Pulse Width Communication Channel 122
3-19. Uniqueness of the Entropy Function. 124
CHAPTER 4 Elements of
1kl''ll''\A..",rII,.,..,,1'!I'
4-1. The Purpose of Encoding . 131
4-2. Separable Binary Codes 137
4-3. Shannon-Fane Encoding 138
4-4. Necessary and Sufficient Conditions for Noiseless
Coding. 142
4-5. A Theorem on Decodability 147
4-6. Average Length of Encoded Messages 148
4-7. Shannon's Binary Encoding 151
4-8. Fundamental Theorem of Discrete Noiseless 154
4-9. Huffman's Minimum-redundancy Code. 155
4-10. Gilbert-Moore Encoding 158
4-11. FundamentalTheoremofDiscreteEncodinginPresence
of Noise 160
4-12. Error-detectingand Error-correcting Codes. 166
4-13. Geometryofthe Binary CodeSpace . 168
4-14. Hamming's Single-error Correcting Code 171
CONTENTS
4-15. Elias's Iteration
4-16. AMathematical Proof ofthe Fundamental Theorem of
Information Theory for Discrete BSC. 180
4-17. Encoding the English Alphabet .
:Continuum without
.a,.".....".ll.JII..I.,Jf.l.
CHAPTER 5 Continuous Probability Distribution and Density
5-1. Continuous Sample Space. 191
5-2. Probability Distribution Functions .
5-3. ProbabilityDensity Function. 194
5-4. Normal Distribution 196
5-5. Cauchy's Distribution . 198
5-6. Exponential Distribution . 199
5-7. Multidimensional Random Variables 200
5-8. Joint Distribution of Two Variables: Marginal
Distribution . 202
5-9. Conditional Probability Distribution and Density .
5-10. Bivariate Normal Distribution 206
5-11. Functions of Random Variables . 208
5-12. Transformation from Cartesian to Polar Coordinate
System.
CHAPTER 6 Statistical Averages
6-1. Expected Values; Discrete Case 220
6-2. Expectation of Sums and Products of a Finite Number
of Discrete Random Variables . 222
6-3. Moments of a Univariate Random Variable. 224
6-4. Two Inequalities 227
6-5. Moments of Bivariate Random Variables 229
6-6. Correlation Coefficient. 230
6-7. Linear Combination of Random Variables . 232
6-8. Moments of Some Common Distribution Functions 234
6-9. Characteristic Functionofa RandomVariable . 238
6-10. Characteristic Function and Moment-generating
Function of Random Variables. 239
6-11. Density Functions of Sum of Two Random
Variables . 242
CHAPTER 7 Normal Distributions and Limit Theorems
7-1. Bivariate Normal Consideredas an Extension of One-
dimensional Normal Distribution . 248
7-2. Multinormal Distribution. 250
7-3. Linear Combination of Normally Distributed Inde-
pendent RandomVariables . 252
xii CONTENTS
Central-limit Theorem .
7-5. Random-walk 258
7-6. of the Binomial Distribution the
Normal Distribution 259
7-7. Approximation of Poisson Distribution a Normal
Distribution .
7-8. The Laws ofLarge Numbers . 263
8 Continuous Channel without Memory
8-1. Definition of Different Entropies. 267
8-2. The NatureofMathematicalDifficulties Involved. 269
8-3. Infiniteness of Continuous Entropy . 270
8-4. The Variability ofthe Entropy in the Continuous Case
with Coordinate Systems 273
8-5. A Measure of Information in the Continuous Case. 275
8-6. Maximization ofthe Entropyofa Continuous Random
Variable . 278
8-7. EntropyMaximization Problems. 279
8-8. Gaussian Noisy Channels . 282
8-9. Transmission of Information in Presence of Additive
Noise . 283
8-10. Channel Capacity in Presence of Gaussian Additive
Noise and Specified Transmitter and Noise Average
Power. 285
8-11. Relation between the of Two Related
Random Variables 287
8-12. Note on the Definition of Mutual Information. 289
CHAPTER 9 Transmission of Band-limited Signals
9-1. Introduction 292
9-2. Entropies of Continuous Multivariate Distributions . 293
9-3. Mutual Information ofTwo Gaussian Random Vectors 295
9-4. A Channel-capacity Theorem for Additive Gaussian
Noise . 297
9-5. Digression 299
9-6. Sampling Theorem . 300
9-7. A Physical Interpretation of the Sampling Theorem . 305
9-8. The Concept ofa Vector Space . 308
9-9. Fourier-series Signal Space 313
9-10. Band-limited Signal Space 315
9-11. Band-limited Ensembles . 317
9-12. Entropies of Band-limited Ensemble in Signal Space. 320