Basic Concepts in Information Theory and Coding: The Adventures of Secret Agent 00111

441 Pages·1994·7.23 MB·English

by  Solomon W. Golomb

Basic Concepts in Information
Theory and Coding
The Adventures of Secret Agent 00111
Applications of Communications Theory
Series Editor: R. W • Lucky , Bellcore
Recent volumes In the series:
BASIC CONCEPTS IN INFORMATION THEORY AND CODING:
The Adventures of Secret Agent 00111
Solomon W. Golomb, Robert E. Peile, and Robert A. Scholtz
COMPUTER COMMUNICATIONS AND NETWORKS
John R. Freer
COMPUTER NETWORK ARCHITECTURES AND PROTOCOLS
Second Edition. Edited by Carl A. Sunshine
DATA COMMUNICATIONS PRINCIPLES
Richard D. Gitlin, Jeremiah F. Hayes, and Stephen B. Weinstein
DATA TRANSPORTATION AND PROTECTION
John E. Hershey and R. K. Rao Yarlagadda
DIGITAL PHASE MODULATION
John B. Anderson, Tor AuIin, and Carl-Erik Sundberg
DIGITAL PICTURES: Representation and Compression
Arun N. NetravaIi and Barry G. Haskell
FUNDAMENTALS OF DIGITAL SWITCHING
Second Edition. Edited by John C. McDonald
AN INTRODUCTION TO BROADBAND NETWORKS: LANs, MANs, ATM,
B-ISDN, and Optical Networks for Integrated Multimedia Telecommunications
Anthony S. Acampora
AN INTRODUCTION TO PHOTONIC SWITCHING FABRICS
H. Scott Hinton
OPTICAL CHANNELS: Fibers, Clouds, Water, and the Atmosphere
Sherman Karp, Robert M. Gagliardi, Steven E. Moran, and Larry B. Stotts
PRACTICAL COMPUTER DATA COMMUNICATIONS
William J. Barksdale
SIMULATION OF COMMUNICATIONS SYSTEMS
Michel C. Jeruchim, Philip Balaban, and K. Sam Shanmugan
A Continuation Order Plan is available for this series. A continuation order will bring delivery of each
new volume immediately upon publication. Volumes are billed only upon actual shipment. For further
information please contact the publisher.
Basic Concepts in Information
Theory and Coding
The Adventures of Secret Agent 00111
Solomon W. Golomb
Departments of Electrical Engineering and Mathematics
University of Southern California
Los Angeles, California
Robert E. Peile
Racal Research, Limited
Reading, Berkshire, United Kingdom
Robert A. Scholtz
Department of Electrical Engineering
University of Southern California
Los Angeles, California
Springer Science+Business Media, LLC
LIbrary of Congress CatalogIng-In-PublIcatIon Data
Golomb, Solomon W. (Solomon Wolf)
Basic concepts in inforMation theory and coding, the adventures
of secret agent 00111 / SoloMon W. GOlOMb, Robert E. Peile, Robert
A. Scholtz.
p. eM. -- (Applications of cOMmunications theory)
Includes bibliographical references and index.
1. Coding theory. 2. InforMation theory. I. Peile, Robert E.
II. Scholtz, Robert A. III. Title. IV. Series.
CA268.G575 1994
003'.54--dc20 93-48869
CIP
ISBN 978-1-4419-3236-5 ISBN 978-1-4757-2319-9 (eBook)
DOl 10.1007/978-1-4757-2319-9
© 1994 Springer Science+Business Media New York
Originally published by Plenum Press, New York in 1994.
Softcover reprint of the hardcover I st edition 1994
All rights reserved
No part of this book may be reproduced, stored in a retrieval system, or transmitted in any
form or by any means, electronic, mechanical, photocopying, microfilming, recording, or
otherwise, without written permission from the Publisher
Preface
Basic Concepts in Information Theory and Coding is an outgrowth of a one
semester introductory course that has been taught at the University of Southern
California since the mid-1960s. Lecture notes from that course have evolved
in response to student reaction, new technological and theoretical develop
ments, and the insights of faculty members who have taught the course (in
cluding the three of us). In presenting this material, we have made it accessible
to a broad audience by limiting prerequisites to basic calculus and the ele
mentary concepts of discrete probability theory. To keep the material suitable
for a one-semester course, we have limited its scope to discrete information
theory and a general discussion of coding theory without detailed treatment
of algorithms for encoding and decoding for various specific code classes.
Readers will find that this book offers an unusually thorough treatment
of noiseless self-synchronizing codes, as well as the advantage of problem
sections that have been honed by reactions and interactions of several gen
erations of bright students, while Agent 00111 provides a context for the
discussion of abstract concepts.
Information theory and coding has progressed from its earliest beginnings
in Shannon's epochal paper, "A Mathematical Theory of Communication,"
and the initial work on error-correcting codes by Hamming and Golay (all
in the late 1940s) to revolutionize all aspects of information handling, storage,
and communication and transform how information is viewed in fields as
v
vi Preface
diverse as biology, linguistics, and philosophy. It is our hope that this book
will introduce the fascinating subject of information theory to many new
readers.
Solomon W. Golomb
Los Angeles
Robert E. Peile
Reading, England
Robert A. Scholtz
Los Angeles
Contents
CHAPTER 1. Introduction
1.1. Agent 00 III ........................................ 1
l.2. Uncertainty and Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . 2
l.2.l. Agent 00111 Ponders Pricing .................... 2
l.2.2. Axioms for Uncertainty and the Entropy Function ... 4
l.3. Information Gain .................................... 13
1.3.1. Agent 00 Ill's Sliding Scale of Success Charges ...... 13
1.3.2. Mutual Information and Equivocation . . . . . . . . . . . . . 16
1.4. Handling Large Amounts of Information ................. 21
l.4.l. The Problem for Agent 00 111 .................... 21
1.4.2. A General Model for an Information Source ........ 22
1.5. Tutorial on Homogeneous Markov Sources ............... 27
l.6. The Number of Typical Sequences ...................... 45
l.6.l. Agent 00111 Uncovers a Puzzle .................. 45
1.6.2. List Length and Entropy ........................ 46
l.7. The Utility ofInformation Source Models ................ 53
l. 7.l. Agent 00 III and Language Generation ............ 53
l.7.2. Language Models and Generation ................ 53
1.8. Notes.............................................. 59
References .......................................... 61
CHAPTER 2. Coding for Discrete Noiseless Channels
2.1. The Problem ........................................ 63
2.1.1. Agent 00111 's Problem ......................... 63
2.1.2. Problem Statement ............................ 64
vii
viii Contents
2.2. An Algorithm for Determining Unique Decodability
in the U and Us Cases ................................ 67
F
2.3. A Simple Coding Theorem for Fixed-Rate Sources ......... 74
2.4. The Significance of Information Theory .................. 80
2.5. Tree Codes ......................................... 83
2.6. A Coding Theorem for Controllable Rate Sources .......... 91
2.7. Huffman's Coding Procedure. . . . . . . . . . . . . . . . . . . . . . . . . . . 95
2.8. Efficiently Encoding Markov Sources .................... 10 1
2.9. Variable Symbol Duration Channels ..................... 105
2.10. Lempel-Ziv Coding Procedure ......................... 116
2.10.1. Agent 00111 's Problem ......................... 116
2.10.2. The Algorithm ................................ 116
2.10.3. The Lempel-Ziv Algorithm and Entropy ........... 123
2.10.4. The Lempel-Ziv Approach and Sequence
Complexity Results ............................ 127
2.11. Notes.............................................. 128
References .......................................... 129
CHAPTER 3. Synchronizable Codes
3.1. An Untimely Adventure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 131
3.2. Identifying U Dictionaries ............................. 132
I
3.3. The Hierarchy ofSynchronizable Codes .................. 140
3.4. A Bound on U Dictionary Size ......................... 143
I
3.5. Fixed-Word-Length U Dictionaries. . . . . . . . . . . . . . . . . . . . .. 147
I
3.5.1. Maximal Comma-Free Codes .................... 148
3.5.2. Prefixed Comma-Free Codes ..................... 151
3.5.3. Path-Invariant Comma-Free Codes. . . . . . . . . . . . . . .. 153
3.5.4. Lexicographic U Codes .. . . . . . . . . . . . . . . . . . . . . . .. 155
I
3.6. Comparing Fixed-Word-Length Synchronizable Codes ...... 160
3.7. Variable-Word-Length Synchronizable Codes. . . . . . . . . . . . .. 164
3.8. Necessary Conditions for the Existence of U Dictionaries .... 170
I
3.9. Cyclic Equivalence Class Occupancy and the Sufficiency of
Iteratively Constructed Codes. . . . . . . . . . . . . . . . . . . . . . . . . .. 174
3.10. Constructing Maximal Comma-Free Codes of Odd Word Length 182
3.11. Automating Binary Bounded Synchronization Delay Codes .. 189
3.11.1. Cyclic Equivalence Class Representations .......... 189
3.11.2. Encoding and Decoding ........................ 196
3.12. Notes .............................................. 199
Appendix: The Mobius Inversion Formula ................ 200
References .......................................... 204
Contents ix
CHAPTER 4. Infinite Discrete Sources
4.1. Agent 00 III Meets the Countably Infinite ................ 207
4.2. The Leningrad Paradox ............................... 208
4.3. Mean vs. Entropy in Infinite Discrete Distributions ......... 211
4.4. Run-Length Encodings ................................ 219
4.5. Decoding Run-Length Codes ........................... 222
4.6. Capacity-Attaining Codes .............................. 223
4.7. The Distribution Waiting Times and Performance of Elias-
Shannon Coding ..................................... 226
4.8. Optimal, Asymptotically Optimal, and Universal Codes ..... 230
4.9. The Information-Generating Function of a Probability
Distribution .................... . . . . . . . . . . . . . . . . . . . .. 237
4.9.1. Uniqueness ofthe Inverse ....................... 238
4.9.2. Composition of Generating Functions ............. 240
4.10. Notes .............................................. 241
References .......................................... 241
CHAPTER 5. Error Correction I: Distance Concepts and Bounds
5.1. The Heavy-Handed Cancellation Problem ................ 243
5.2. Discrete Noisy Channels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 244
5.3. Decoding Algorithms ................................. 248
5.4. A Hamming Distance Design Theorem . . . . . . . . . . . . . . . . . .. 256
5.5. Hamming Bound .................................... 260
5.6. Plotkin's Bound ..................................... 265
5.7. Elias Bound. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 269
5.8. Gilbert Bound ....................................... 273
5.9. Perfect Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 275
5.10. Equidistant Codes .................................... 280
5.11. Hamming Distance Enumeration ....................... 284
5.12. Pless Power Moment Identities and the Welch, McEliece,
Rodemich, and Rumsey (WMR) Bound .................. 293
5.13. Finite State Channels ................................. 295
5.14. Pure Error Detection ................................. 302
5.15. Notes .............................................. 306
References .......................................... 307
CHAPTER 6. Error Correction II: The Information-Theoretic
Viewpoint
6.1. Disruption in the Channel ............................. 309
6.2. Data-Processing Theorem and Estimation Problems ........ 310
x Contents
6.3. An Upper Bound on Information Rate for Block-Coding Schemes 315
6.4. The Chernoff Bound .................................. 323
6.5. Linked Sequences .................................... 331
6.6. Coding Theorem for Noisy Channels. . . . . . . . . . . . . . . . . . . .. 336
6.7. The Situation for Reliable Communication ............... 346
6.8. Convex Functions and Mutual Information Maximization ... 348
6.9. Memoryless Channel Capacity Computations. . . . . . . . . . . . .. 358
6.10. Notes .............................................. 367
References .......................................... 368
CHAPTER 7. Practical Aspects of Coding
7.1. Agent 00111 Is Not Concerned ......................... 369
7.2. Types of Practical Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 369
7.2.1. Convolutional Codes ........................... 370
7.2.2. Block Codes: A General Overview ................ 374
7.2.3. Reed-Solomon Codes .......................... 377
7.2.4. Interleaving .................................. 382
7.2.5. Concatenated Codes. . . . . . . . . . . . . . . . . . . . . . . . . . .. 383
7.3. Coding and Modulation ............................... 388
7.4. Hybrid Forward Error Correction (FEC) and Retransmission
(ARQ) Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 389
7.4.1. General Comments ............................ 390
7.4.2. Some Basic ARQ Strategies. . . . . . . . . . . . . . . . . . . . .. 390
7.4.3. Type-1 Hybrid FECI ARQ ....................... 391
7.4.4. Type-2 Hybrid ARQ/FEC ....................... 393
7.4.5. Chase Code Combining ......................... 395
7.4.6. Coding and Future Networks .................... 395
7.A. Appendix: Lattices and Rings . . . . . . . . . . . . . . . . . . . . . . . . . .. 397
References .......................................... 416
Author Index ............................................. 419
Subject Index ............................................ 423