(cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) (cid:13) UPC FACULTATDEMATEMÀTIQUESIESTADÍSTICA A computational primer on BLOCK ERROR-CORRECTING CODES Sebastià Xambó i Descamps January 2003 January2003 (cid:13)c SebastiàXambóiDescamps DepartamentdeMatemàticaAplicadaII UniversitatPolitècnicadeCatalunya email: [email protected] Preface Thereisauniquewaytoteach:toleadtheotherpersonthrough thesameexperiencewithwhichyoulearned. ÓscarVillarroya,cognitivescientist.1 Inthisbook,themathematicalaspectsinourpresentationofthebasictheory of block error-correcting codes go together, in mutual reinforcement, with computationaldiscussions,implementationsandexamplesofalltherelevant concepts,functionsandalgorithms. Wehopethatthisapproachwillfacilitate the reading and be serviceable to mathematicians, computer scientists and engineersinterestedinblockerror-correctingcodes.2 Inthishypertexpdfversion, theexamplescanberunwithjustamouse click. Moreover, the examples can be modified by users and saved to their localfacilitiesforlaterwork. The program that handles the computations in the examples, together with the interface that handles the editing (mathematics and text), here will be called WIRIS/cc. More specifically, WIRIS stands for the interface, the (remote) computational engine and the associated language, and cc stands for the extension of WIRIS that takes care of the computational aspects that aremorespecificoferror-correctingcodes. WIRIS/cc is an important ingredient in our presentation, but we do not presuppose any knowledge of it. As it is very easy to learn and use, it is introduced gradually, according to the needs of our presentation. A succint description of the functions used can be found in the Index-Glossary at the end. TheAppendixisasummaryofthemainfeaturesofthesystem. This book represents the author’s response to the problem of teaching a one-semester course in coding theory under the circumstances that will be explainedinamoment. Hopefullyitwillbeusefulaswellforotherteachers 1Cognitive Research Center at the Universitat Autònoma de Barcelona, interviewed by LluísAmiguetin“lacontra”,LAVANGUARDIA,22/08/2002. 2Aforerunnerofthisapproachwasoutlinedin[34]. Preface iii facedwithsimilarchallenges. ThecoursehasbeentaughtattheFacultatdeMatemàtiquesiEstadística (FME) of the Universitat Politècnica de Catalunya (UPC) in the last few years. Ofthesixtysessionsoffiftyminuteseach, nearlyhalfaredevotedto problemdiscussionsandproblemsolving. Thestudentsarejuniororsenior mathematicsmajors(thirdandfourthyear)andsomeofthemarepursuinga doubledegreeinmathematicsandtelecommunicationsengineering. The nature of the subject, the curriculum at the FME and the context at the UPC advise that, in addition to sound and substantial mathematical conceptsandresults,areasonableweightshouldbegiventoalgorithmsand the effective programming of them. In other words, learning should span a wide spectrum ranging from the theoretical framework to aspects that may belabeledas‘experimental’and‘practical’. Allthesevariousboundaryconditions,especiallythestringenttimecon- straintsandthehugesizeofthesubject,aretobeponderedverycarefullyin the design of the course. Given the prerequisites that can be assumed (say linear and polynomial algebra, some knowledge of finite fields, and basic factsaboutprobabilitytheory),itisreasonabletoaimatagoodunderstand- ing of some of the basic algebraic techniques for constructing block error- correcting codes, the corresponding coding and decoding algorithms and a goodexperimentalandpracticalknowledgeoftheirworking. Togiveamore concrete idea, this amounts to much of the material on block error correct- ing codes included in, say, chapters 3-6 in [29] (or chapters 4 to 8 in [24]), supplementedwithafewtopicsthatarenotcoveredinthesebooks,plusthe correspondingalgorithmicsandprogramminginthesenseexplainedabove. Of course, this would be impossible unless a few efficiency principles are obeyed. The basic one, some sort of Ockam’s razor, is that in the theo- reticalpresentationmathematicsisintroducedonlywhenneeded. Asimilar principleisappliedtoallthecomputationalaspects. The choice of topics is aimed at a meaningful and substantial climax in eachsectionandchapter,andinthebookasawhole,ratherthanatanobvi- ouslyfutileattemptatcompleteness(inanyform),whichwouldbesenseless anywayduetotheimmensesizeofthefieldsinvolved. WIRIS/ccmakesitpossibletodropmanyaspectsofthecurrentpresenta- tionsasdefinitelyunnecessary,likethecompilationsofdifferentsortsofta- bles(asforexamplethoseneededforthehandcomputationsinfinitefields). Asaconsequence,moretimeislefttodealwithconceptualmatters. Finally it should be pointed out that this digital version can be a key tool for the organization of laboratory sessions, both for individual work assignments and for group work during class hours. The system, together with the basic communications facilities that today can be assumed in most universities,makesiteasierforstudentstosubmittheirhomeworkindigital iv Preface form, and for teachers to test whether the computational implementations runproperly. The basic pedagogical assumptions underlying the whole approach are thatthestudyofthealgorithmsleadstoabetterunderstandingofthemathe- maticsinvolved,andthatthedisciplineofprogrammingtheminaneffective way promotes a better understanding of the algorithms. It could be argued that the algorithms and programs are not necessary to solve problems, at leastnotinprinciple,butitcanalsobearguedthattakingthemintoaccount reinforceslearning,becauseitintroducesanexperimentalcomponentinthe mathematical practice, and because it better prepares learners for future ap- plications. Ofcourse,wedonotactuallyknowwhetherthesenewtoolsandmethods will meet the high expectations of their capacity to strengthen the students’ understanding and proficiency. But part of the beauty of it all, at least for the author, stems precisely from the excitement of trying to find out, by ex- perimentingtogetherwithourstudents,howfarthosetoolsandmethodscan advancetheteachingandlearningofmathematics,algorithmsandprograms. Thegrowingneedformathematiciansandcomputerscientists in industry will lead to an increase in courses in the area of discretemathematics. Oneofthemostsuitableandfascinating is,indeed,codingtheory. J.H.vanLint,[29],p.ix. Acknowledgements Thecharacterofthisbook,andespeciallythetoolsprovidedat http://www.wiris.com/cc/, would hardly be conceivable without the WIRIS system, which would not have been produced without the sustained, industrious and unflappable col- laborationofDanielMarquèsandRamonEixarch,firstintheProjectOMEGA at the FME and afterwards, since July 1999, as partners in the firm Maths forMore. On the Springer-Verlag side, the production of this book would never have been completed without the help, patience and know-how of Clemens Heine. Several other people at Springer have also been supportive in dif- ferent phases of the project. My gratitude to all, including the anonymous peoplethatprovidedEnglishcorrectionsinthefinalstage. It is also a pleasant duty to thank the FME and the Departament de Matemàtica Aplicada II (MA2) of the UPC for all the support and encour- agement while in charge of the Coding Theory course, and especially the Preface v studentsandcolleaguesfortheeverydaygive-and-takethatcertainlyhasin- fluencedalotthefinalformofthistext. JoanBruna,forexample,suggested animprovementintheimplementationoftheMeggittdecoderthathasbeen includedinSection3.4. Partofthematerialinthistext,andespeciallythesoftwareaspects,were presented and discussed in the EAGER school organized by Mina Teicher and Boris Kunyavski from the Emmy Noether Institute (German Minerva Center) at Eilat (Israel, January 12-16, 2003). I am grateful to the organiz- ers for this opportunity, and to all the participants for their eager inquiring about all aspects of the course. In particular, I have to thank Shmulik Ka- planforsuggestinganimprovementofthealternantdecoderimplementation presentedinSection4.3. Grateful thanks are due to Thomas Hintermann for sharing his enlight- eningviewsonseveralaspectsofwriting,andespeciallyonEnglishwriting. Theyhavesurelycontributedtoimprovethiswork,butofcourseonlytheau- thoristobefoundresponsibleforthemistakesandblundersthathavegone throughundetected. And thanks to my wife, Elionor Sedó. Without her enduring support andloveitwouldnothavebeenpossibletodedicatethisworktoheronthe occasionofourthirtiethweddinganniversary,foritwouldhardlyhavebeen finished. Theauthor L’Escala 20/1/03 Introduction Error-correctingcodeshavebeenincorporatedinnumerous workingcommunicationandmemorysystems. W.WesleyPetersonandE.J.Weldon,Jr.,[19],p.v. This chapter is meant to be an informal introduction to the main ideas of error-correctingblockcodes. We will see that if redundancy is added in a suitable form to the infor- mation to be sent through a communications channel (this process is called coding),thenitispossibletocorrectsomeoftheerrorscausedbythechan- nelnoiseifthesymbolsreceivedareprocessedappropriately(thisprocessis calleddecoding). ThetheoreticallimitofhowgoodcodingcanbeiscapturedbyShannon’s conceptof(channel)capacityandShannon’sepoch-makingchannelcoding theorem.3 Attheendwewillalsoexplainhowarewegoingtopresentthecompu- tationaldiscussionsandmaterials. Howdowecorrecttextmistakes? ‘Atrapdfondmilriancileqbegenswifhasinqlesoep’, Chinesesaying M.Bossert,[5],p.xiii. Assumethatyouareaskedtomakesenseofa‘sentence’likethis: “Coding thearyzsbodhwntelestingandchallenning”(totakeasimplerexamplethan theChinesesayinginthequotation). 3“Whilehisincredibleinventivemindenrichedmanyfields,ClaudeShannon’senduring famewillsurelyrestonhis1948paperAmethematicaltheoryofCommunicationandtheon- goingrevolutionininformationtechnologyitengendered”(SolomonW.Golomb,in“Claude ElwoodShannon(1916-2001)”,NoticesoftheAMS,volume49,number1,p.8). 2 Introduction Yourealizethatthisproposedtexthasmistakesandafteralittleinspec- tion you conclude that most likely the writer meant “Coding theory is both interestingandchallenging”. Why is this so? Look at ‘challenning’, for example. We recognize it is a mistake because it is not an English word. But it differs from the English word ‘challenging’ in a single letter. In fact, we do not readily find other English words that differ from ‘challenning’ in a single letter, and thus any other English word that could be meant by the writer seems much more unlikely than ‘challenging’. So we ‘correct’ ‘challenning’ to ‘challenging’. Asimilarsituationoccurswiththereplacementof‘theary’by‘theory’. Now in ‘bodh’, if we were to change a single letter to get an English word, we would find ‘bode, ‘body” or ‘both’, and from the context we would choose ‘both’,becausetheothertwodonotseemtomakesense. Therehavebeenseveralimplicitprinciplesatworkinourcorrectionpro- cess, including contextual information. For the purposes of this text, how- ever, the main idea is that most English words contain redundancy in the sense that altering one letter (or sometimes even more) will usually still al- lowustoidentifytheoriginalword. Wetakethiswordtobethecorrectone, because,underreasonableassumptionsonthesourceoferrors,wethinkthat itisthemostlikely. TherepetitioncodeRep(3) Similar principles are the cornerstones of the theory and practice of error- correctingcodes. Thiscanbeillustratedwithatoyexample,which,despite itssimplicity,hasmostofthekeyfeaturesofchannelcoding. Suppose we have to transmit a stream of bits u u u ··· (thus u ∈ 1 2 3 i {0,1} for all i) through a ‘noisy channel’ and that the probability that one bitisalteredisp. Weassumethatpisindependentofthepositionofthebit in the stream, and of whether it is 0 or 1. This kind of channel is called a binarysymmetricchannelwithbit-errorratep. Inordertotrytoimprovethequalityoftheinformationatthereceiving end,wemaydecidetorepeateachbitthreetimes(thisistheredundancywe addinthiscase). Sinceforeachinformationbittherearethreetransmission bits,wesaythatthiscodingschemehasrate1/3(ittakesthreetimeslonger totransmitacodedmessagethantheuncodedone). Thuseveryinformation bit u is coded into the code word uuu (or [uuu] if we want to represent it as a vector). In particular, we see that there are two code words, 000 and 111. Since they differ in all three positions, we say that the minimum distanceofthecodeis3. Thefactthatthecodewordshave3bits,thateach correspondstoasingleinformationbit,andthatthetwocode-wordsdifferin Introduction 3 3positionsissummarizedbysayingthatthisrepetitioncode,whichwewill denoteRep(3),hastype[3,1,3]. If we could assume that at most one error is produced per each block of three bits, then the redundancy added by coding suggests to decode each block of three bits by a ‘majority vote’ decision: 000, 100, 010, 001 are decoded as 0 and 111, 011, 101, 110 are decoded as 1. Of course, if two or threeerrorsoccurred—apossibilitythatusuallyismuchmoreunlikelythan havingatmostoneerror—,thenthedecoderwouldgivethewronganswer. All seems very good, but nevertheless we still have to ask whether this schemereallyimprovesthequalityofthereceivedinformation. Sincethere arethreetimesmorebitsthatcanbeinerror,arewesurethatthiscodingand decodingishelping? To decide this, notice that p represents the proportion of error bits if no codingisused,whileifweuseRep(3)theproportionofbiterrorswillequal the proportion p0 of received 3-bit blocks with 2 or 3 errors (then we may callthedecodedbitacode-error). Since p0 = 3p2(1−p)+p3, weseethatp0/p,whichcanbecalledtheerror-reductionfactor ofthecode, isequalto 3p(1−p)+p2 = 3p−2p2 = 3p(1− 2p) 3 If p is small, as it usually is, this expression is also small, and we can usethe(excess)approximation3pasitsvalue. Forexample,3p = 0.003for p = 0.001, which means that, on average, for every 1000 errors produced withoutcodingwewillhavenotmorethan3withcoding. This is definitely a substantial improvement, although we are paying a priceforit: thenumberoftransmittedbitshasbeentripled,andwealsohave addedcomputingcostsduetothecodinganddecodingprocesses. CanwedobetterthanintheRep(3)example? Yes,ofcourse,andthisis whatthetheoryoferror-correctingcodesisabout. Remarks. Thereisnolossofgeneralityinassuming,forabinarysymmet- ric channel, that p 6 1/2 (if itwere p > 1/2, we could modifythe channel byreplacingany0by1andany1by0atthereceivingend). Ifthiscondition issatisfied,thenp0 6 p,withequalityonlyforp = 0andp = 1/2(seeFig- ure1). Itisclear, therefore, thatwecanalwaysassumethatRep(3)reduces the proportion of errors, except for p = 1/2, in which case it actually does not matter because the channel is totally useless. In Figure 1 we have also drawn the graph of p0/p (the error reduction factor). Note that it is < 1 for p < 1/2and< 1/2forp < 0.19. 4 Introduction 1 1/2 0 0.19 1/2 Figure 1: The graph of p0 = 3p2 −2p3 as a function of p in the interval [0,1] 2 is the concave arc. The convex arc is the graph in the same interval of p0/p (the error-reductionfactor). Itis<1forp<1/2and<1/2forp<0.19. Notethatin theinterval[0,0.19]itdecreasesalmostlinearlyto0. ChannelcapacityandShannon’schannelcodingtheorem Shannon, in his celebrated paper [25], introduced the notion of capacity of a channel — a number C in the interval [0,1] that measures the maximum fraction of the information sent through the channel that is available at the receivingend. Inthecaseofabinarysymmetricchannelitturnsoutthat C = 1+plog (p)+(1−p)log (1−p), 2 2 where log is the base 2 logarithm function. Notice that C = C(p) is 2 a strictly decreasing function in the interval [0,1/2], with C(0) = 1 and C(1/2) = 0,andthatC(p) = C(1−p)(seeFigure2). 1 0 1/2 1 Figure2: GraphofthecapacityC(p),p∈[0,1],forabinarysymmetricchannel Shannon’schannelcodingtheoremstatesthatifRisapositiverealnum-