Table Of Content

Introduction to Information Theory and Data Compression Second Edition © 2003 by CRC Press LLC DISCRETE MATHEMATICS and ITS APPLICATIONS Series Editor Kenneth H. Rosen, Ph.D. AT&T Laboratories, Middletown, New Jersey Abstract Algebra Applications with Maple, Richard E. Klima, Ernest Stitzinger, and Neil P. Sigmon Algebraic Number Theory, Richard A. Mollin An Atlas of the Smaller Maps in Orientable and Nonorientable Surfaces, David M. Jackson and Terry I. Visentin An Introduction to Crytography, Richard A. Mollin Combinatorial Algorithms: Generation Enumeration and Search, Donald L. Kreher and Douglas R. Stinson The CRC Handbook of Combinatorial Designs, Charles J. Colbourn and Jeffrey H. Dinitz Cryptography: Theory and Practice, Second Edition, Douglas R. Stinson Design Theory, Charles C. Lindner and Christopher A. Rodgers Frames and Resolvable Designs: Uses, Constructions, and Existence, Steven Furino, Ying Miao, and Jianxing Yin Fundamental Number Theory with Applications, Richard A. Mollin Graph Theory and Its Applications, Jonathan Gross and Jay Yellen Handbook of Applied Cryptography, Alfred J. Menezes, Paul C. van Oorschot, and Scott A. Vanstone Handbook of Constrained Optimization, Herbert B. Shulman and Venkat Venkateswaran Handbook of Discrete and Combinatorial Mathematics, Kenneth H. Rosen Handbook of Discrete and Computational Geometry, Jacob E. Goodman and Joseph O’Rourke Introduction to Information Theory and Data Compression, Darrel R. Hankerson, Greg A. Harris, and Peter D. Johnson Network Reliability: Experiments with a Symbolic Algebra Environment, Daryl D. Harms, Miroslav Kraetzl, Charles J. Colbourn, and John S. Devitt RSA and Public-Key Cryptography Richard A. Mollin Quadratics, Richard A. Mollin Verification of Computer Codes in Computational Science and Engineering, Patrick Knupp and Kambiz Salari © 2003 by CRC Press LLC Darrel Hankerson Greg A. Harris Peter D. Johnson, Jr. Introduction to Information Theory and Data Compression Second Edition CHAPMAN & HALL/CRC A CRC Press Company Boca Raton London New York Washington, D.C. © 2003 by CRC Press LLC C3138-discl. Page 1 Friday, January 17, 2003 1:19 PM Library of Congress Cataloging-in-Publication Data Hankerson, Darrel R. Introduction to information theory and data compression / Darrel R. Hankerson, Greg A. Harris, Peter D. Johnson.--2nd ed. p. cm. (Discrete mathematics and its applications) Includes bibliographical references and index. ISBN 1-58488-313-8 (alk. paper) 1. Information theory. 2. Data compression (Computer science) I. Harris, Greg A. II. Johnson, Peter D. (Peter Dexter), 1945- III. Title. IV. Series. Q360.H35 2003 005.74¢6—dc21 2002041506 CIP This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior permission in writing from the publisher. The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying. Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe. Visit the CRC Press Web site at www.crcpress.com © 2003 by CRC Press LLC No claim to original U.S. Government works International Standard Book Number 1-58488-313-8 Library of Congress Card Number 2002041506 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 Printed on acid-free paper © 2003 by CRC Press LLC Preface This textbook is aimed at graduate students and upper level undergraduates in mathematics, engineering,and computerscience. The materialand the ap- proachofthetextweredevelopedoverseveralyearsatAuburnUniversityintwo independentcourses,InformationTheoryandDataCompression.Althoughthe materialinthetwocoursesisrelated,wethinkitunwiseforinformationtheory tobeaprerequisitefordatacompression,andhavewrittenthedatacompression sectionofthetextsothatitcanbereadbyorpresentedtostudentswithnoprior knowledgeofinformationtheory.Therearereferencesinthedatacompression partto results and proofsin the informationtheorypart of the text, and those who are interested may browse over those references, but it is not absolutely necessaryto doso. Infact, perhapsthe bestpedagogicalorderofapproachto these subjects is the reverse of the apparentlogical order: students will come toinformationtheorycuriousandbetterpreparedforhavingseensomeofthe definitionsandtheoremsofthatsubjectplayingaroleindatacompression. Our main aim in the data compression part of the text, as well as in the course it grew from, is to acquaint the students with a number of significant lossless compression techniques, and to discuss two lossy compression methods.Ouraimisforthestudentstoemergecompetentinandbroadlyconversant with a large range of techniques. We have striven for a “practical” style of presentation: here is what you do and here is what it is good for. Nonethe- less,proofsareprovided,sometimesinthetext,sometimesintheexercises,so thattheinstructorcanhavetheoptionofemphasizingthemathematicsofdata compressiontosomedegree. Informationtheoryis ofa moretheoreticalnaturethandata compression. It providesa vocabulary and a certain abstraction that can bring the power of simplificationtomanydifferentsituations. Wethoughtitreasonabletotreatit as a mathematical theory and to present the fundamental definitions and ele- mentaryresultsofthattheoryinutterabstractionfromtheparticularproblems ofcommunicationthroughnoisychannels,whichinspiredthetheoryinthefirst place.WebringthetheorytobearonnoisychannelsinChapters3and4. The treatment of information theory given here is extremely elementary. The channels are memoryless and discrete, and the sources are all “zeroth- order,” one-state sources (although more complicated source models are dis- cussedinChapter7). We feelthatthiselementaryapproachisappropriatefor thetargetaudience,andthat,byleavingmorecomplicatedsourcesandchannels outofthepicture,wemoreeffectivelyimpartthegraspofInformationTheory thatwehopeourstudentswilltakewiththem. The exercises range from the routine to somewhat lengthy problems that introduceadditionalmaterialorestablishmoredifficultresults. Anasteriskby v © 2003 by CRC Press LLC vi Preface anexerciseorsectionindicatesthatthematerialisoffthemainroad,sotospeak, andmightreasonablybeskipped. Inthecaseofexercises,itmayalsoindicate thattheproblemishardand/orunusual. Inthedatacompressionportionofthebook,anumberofprojectsrequire theuseofacomputer. AppendixAdocumentsOctaveandMatlabscriptswrit- tenbytheauthorsthatcanbeusedonsomeoftheexercisesandprojectsinvolv- ingtransformmethodsand images, andthat can also serve as buildingblocks for other explorations. The software can be obtained from the authors’ site, listedinAppendixC. Inaddition,thesitecontainsinformationaboutthebook, anonlineversionofAppendixA,andlinkstoothersitesofinterest. Organization Here’sabriefsynopsisofeachchapterandappendix. Chapter1 containsanintroductionto the languageandresults ofprobability theory. Chapter2 presentstheelementarydefinitionsofinformationtheory,ajustifi- cationofthequantificationofinformationonwhichthetheoryisbased, and the fundamental relations among various sorts of information and entropy. Chapter3 isaboutinformationflowthroughdiscretememorylessnoisychan- nels. Chapter4 is about coding text from a discrete source, transmitting the en- coded text through a discrete memoryless noisy channel, and decoding theoutput. The“classical”fundamentaltheoremsofinformationtheory, includingtheNoisyChannelTheorem,appearinthischapter. Chapter5 begins the material of the data compression portion of this book. Replacementschemesare discussed and the chapter concludeswith the Noiseless CodingTheorem, provedhere for a binary code alphabet. (It appearsinChapter4inmoregeneralform.) Chapter6 discussesarithmeticcoding,whichisofconsiderableinterestsince itisoptimalinacertainwaythatthereplacementschemesarenot. Con- siderationsforbothan “ideal”schemeandforpracticalimplementation onacomputerarepresented. Chapter7 focuses on the modeling aspects of Chapters 5 and 6 (Chapter 8 continuesthediscussion). Sincecodingmethodssuchasthosepresented in Chapter 6 can (in theory) produce optimal-length output for a given model of the source, much of the interest in improvingcompression in statistical schemes lies in improving the model of the source. Higher- ordermodelsattempttouselargercontextsforpredictions.Inthesecond © 2003 by CRC Press LLC Preface vii edition, a section on probabilistic finite state source automata has been added. Chapter8 considers another approach to modeling, using statistics that are updatedasthesourceisreadandencoded.Thesehavetheadvantagethat nostatisticalstudyneedstobedoneinadvanceandtheschemecanalso detectchangesinthenatureofthesource. Chapter9 discusses popular dictionary methods. These have been widely used,inpartduetotheirsimplicity,speed,andrelativelygoodcompres- sion.ApplicationssuchasRossWilliams’LZRW1algorithm,Unixcom- press,andGNUzip(gzip)areexamined. Chapter10 develops the Fourier, cosine, and wavelet transforms, and dis- cussestheiruseincompressionofsignalsorimages. Thelossy scheme in JPEG is presented as a widely-usedstandardthat relies on transform techniques.Thechapterconcludeswithanintroductiontowavelet-based compression. AppendixA documents the use of the “JPEGtool” collection of Octave and MatlabscriptsinunderstandingJPEG-likeimagecompression. AppendixB contains the source listing for Ross Williams’ LZRW1-A algorithm,whichratherconciselyillustratesaviabledictionarycompression method. AppendixC containsmaterialthatdidn’tfitelsewhere. The firstsection lists sources for information and code for many areas of data compression. The second section contains a few notes on patents affecting the field. The final section contains a semi-famous story illustrating some of the misunderstandingsaboutcompression. AppendixD offerssolutionsandnotesontheexercises. Acknowledgments We’d like to thank Ross Williams for permission to reprinthis LZRW1-A algorithm, and for notes on his sources. Alistair Moffat providedpreprintsand alertedustootherinformationconcerningarithmeticcoding.IanH.Wittenwas kindenoughto respondto ourquestionsconcerninga detailin Text Compres- sion. Weespeciallywishtoacknowledgethehelpoffourreviewers: Jean-loup GaillyofferedmanyimportantsuggestionsconcerningChapter9andAppendix C,andgrantedpermissiontouseportionsofthe“FrequentlyAskedQuestions” documentthatheauthors;TomLanesuggestedanumberofimprovementsand clarifications in Chapter 10; we are grateful to James R. Wall and to Isidore FleischerforreviewingportionsofChapters1–5. There are many folks who have made it easier for the community to un- derstand the subject; some of their names are in this book. Others, working © 2003 by CRC Press LLC viii Preface on“GNUProject”andotherfreelydistributablesoftware,madethisbookpos- sible. The list of major contributors to this software is lengthy, and includes those involved with AUC TeX, dvips[k], Emacs, Ghostview and Ghostscript, GNU/Linux,theIndependentJPEG Group,Info-ZIP,LATEX, Netpbm, PICTEX, PortableNetworkGraphics,TEX,xdvi[k],xfig,xv,XY-pic,andmanyGNUutil- itiessuchasbash, gawk,gccandgdb, gzip,andmake. We wish to especially thanktheprincipaldeveloperofOctave,JohnEaton. Thanks are due to A. Scottedward Hodel, Alfred Menezes, Stan Reeves, andGregRoelofsfor someearly adviceonthe datacompressioncourse. Our studentswerealsogenerouswiththeiradvice. DouglasLeonardandLucTeir- linckprovidedsomeinsightfulsuggestionsandclarifications. AlfredMenezes gets the credit (and the blame) for setting us on the road to a course in data compressionandthisbook. Someofthecomputerresourcesforthecourseandbookweremadepossi- blebyagrantfromtheNationalScienceFoundation,forwhichwearegrateful. Our direct contacts at CRC Press were Bob Stern, Nora Konopka, Suzanne Lassandro,TimPletscher,JamieSigal,MimiWilliams, andSylviaWood,and itwasapleasureworkingwiththem. © 2003 by CRC Press LLC Contents Preface v PartI:InformationTheory 1 ElementaryProbability 1 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Conditionalprobability . . . . . . . . . . . . . . . . . . . . . . 7 1.4 Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1.5 Bernoullitrials . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.6 Anelementarycountingprinciple . . . . . . . . . . . . . . . . 15 1.7* Ondrawingwithoutreplacement . . . . . . . . . . . . . . . . . 17 1.8 Randomvariablesandexpected,oraverage,value . . . . . . . . 18 1.9 TheLawofLargeNumbers. . . . . . . . . . . . . . . . . . . . 22 2 InformationandEntropy 25 2.1 Howisinformationquantified? . . . . . . . . . . . . . . . . . . 25 2.1.1 Namingtheunits . . . . . . . . . . . . . . . . . . . . . 27 2.1.2 Informationconnectingtwoevents . . . . . . . . . . . . 29 2.1.3 The inevitability of Shannon’s quantification of information . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.2 Systemsofeventsandmutualinformation . . . . . . . . . . . . 33 2.3 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 2.4 Informationandentropy . . . . . . . . . . . . . . . . . . . . . 43 3 ChannelsandChannelCapacity 47 3.1 Discretememorylesschannels . . . . . . . . . . . . . . . . . . 47 3.2 Transitionprobabilitiesandbinarysymmetricchannels . . . . . 50 3.3 Inputfrequencies . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.4 Channelcapacity . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.5* ProofofTheorem3.4.3,onthecapacityequations . . . . . . . . 67 4 CodingTheory 71 4.1 Encodinganddecoding . . . . . . . . . . . . . . . . . . . . . . 71 4.2 Prefix-conditioncodesandtheKraft-McMillaninequality. . . . 75 4.3 AveragecodewordlengthandHuffman’salgorithm . . . . . . . 79 4.3.1 ThevalidityofHuffman’salgorithm . . . . . . . . . . . 86 4.4 Optimizingtheinputfrequencies . . . . . . . . . . . . . . . . . 90 4.5 Error correction, maximum likelihood decoding, nearest code worddecoding,andreliability . . . . . . . . . . . . . . . . . . 95 ix © 2003 by CRC Press LLC x CONTENTS 4.6 Shannon’sNoisyChannelTheorem . . . . . . . . . . . . . . . 106 4.7 Errorcorrectionwithbinarysymmetricchannelsandequalsource frequencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 4.8 Theinformationrateofacode . . . . . . . . . . . . . . . . . . 115 PartII:Data Compression 5 LosslessDataCompressionbyReplacementSchemes 119 5.1 Replacementviaencodingscheme . . . . . . . . . . . . . . . . 120 5.2 Reviewoftheprefixcondition . . . . . . . . . . . . . . . . . . 123 5.3 Choosinganencodingscheme . . . . . . . . . . . . . . . . . . 126 5.3.1 Shannon’smethod . . . . . . . . . . . . . . . . . . . . 127 5.3.2 Fano’smethod . . . . . . . . . . . . . . . . . . . . . . 130 5.3.3 Huffman’salgorithm . . . . . . . . . . . . . . . . . . . 131 5.4 TheNoiselessCodingTheoremandShannon’sbound . . . . . . 134 6 ArithmeticCoding 141 6.1 Purezeroth-orderarithmeticcoding:dfwld . . . . . . . . . . . 142 6.1.1 Rescalingwhileencoding . . . . . . . . . . . . . . . . 146 6.1.2 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . 150 6.2 What’sgoodaboutdfwldcoding:thecompressionratio . . . . . 155 6.3 What’sbadaboutdfwldcodingandsomewaystofixit . . . . . 160 6.3.1 Supplyingthesourcewordlength . . . . . . . . . . . . 161 6.3.2 Computation . . . . . . . . . . . . . . . . . . . . . . . 162 6.3.3 Mustdecodingwaituntilencodingiscompleted? . . . . 164 6.4 Implementingarithmeticcoding . . . . . . . . . . . . . . . . . 167 6.5 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 7 Higher-orderModeling 181 7.1 Higher-orderHuffmanencoding . . . . . . . . . . . . . . . . . 182 7.2 TheShannonboundforhigher-orderencoding . . . . . . . . . . 186 7.3 Higher-orderarithmeticcoding . . . . . . . . . . . . . . . . . . 191 7.4 Statisticalmodels,statistics,andthepossiblyunknowabletruth . 193 7.5 Probabilisticfinitestatesourceautomata . . . . . . . . . . . . . 197 8 AdaptiveMethods 205 8.1 AdaptiveHuffmanencoding . . . . . . . . . . . . . . . . . . . 206 8.1.1 Compressionandreadjustment . . . . . . . . . . . . . . 209 8.1.2 Higher-orderadaptiveHuffmanencoding . . . . . . . . 210 8.2 MaintainingthetreeinadaptiveHuffmanencoding:themethod ofKnuthandGallager . . . . . . . . . . . . . . . . . . . . . . 212 8.2.1 Gallager’smethod . . . . . . . . . . . . . . . . . . . . 215 8.2.2 Knuth’salgorithm . . . . . . . . . . . . . . . . . . . . 216 8.3 Adaptivearithmeticcoding . . . . . . . . . . . . . . . . . . . . 219 8.4 Intervalandrecencyrankencoding . . . . . . . . . . . . . . . . 221 © 2003 by CRC Press LLC