Decrypted Secrets: Methods and Maxims of Cryptology

554 Pages·2007·12.344 MB·English

by  Friedrich L. Bauer

Decrypted Secrets
Friedrich L. Bauer
Decrypted Secrets
Methods and Maxims
of Cryptology
Fourth, Revised and Extended Edition
With 191 Figures, 29 Tables,
and 16 Color Plates
123
Dr.rer.nat.Dr.èssc.h.c.Dr.rer.nat.h.c.mult.FriedrichL.Bauer
ProfessorEmeritusofMathematicsandComputerScience
MunichInstituteofTechnology
DepartmentofComputerScience
Boltzmannstr.3
85748Garching,Germany
ACMComputingClassification(1998):E.3,D.4.6,K.6.5,E.4
MathematicsSubjectClassification(1991):94A60,68P25
LibraryofCongressControlNumber:2006933429
ISBN-10 3-540-24502-2 SpringerBerlinHeidelbergNewYork
ISBN-13 978-3-540-24502-5 SpringerBerlinHeidelbergNewYork
ISBN 3-540-42674-4 3rded.SpringerBerlinHeidelbergNewYork
Thisworkissubjecttocopyright.Allrightsarereserved,whetherthewholeorpartofthematerial
isconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,broad-
casting,reproductiononmicrofilmorinanyotherway,andstorageindatabanks.Duplicationof
thispublicationorpartsthereofispermittedonlyundertheprovisionsoftheGermanCopyrightLaw
ofSeptember9,1965,initscurrentversion,andpermissionforusemustalwaysbeobtainedfrom
Springer.ViolationsareliableforprosecutionundertheGermanCopyrightLaw.
SpringerisapartofSpringerScience+BusinessMedia
springer.com
©Springer-VerlagBerlinHeidelberg1997,2000,2002,2007
Theuseofgeneraldescriptivenames,registerednames,trademarks,etc.inthispublicationdoesnot
imply, even in the absence of a specific statement, thatsuch names are exempt from the relevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
CoverDesign:Design&ConceptE.Smejkal,Heidelberg
ColorPhotos:ReinhardKrause,DeutschesMuseumMünchen
Typesetting:BytheauthorinTEX
Production:LE-TEX,Jelonek,Schmidt&VöcklerGbR,Leipzig
Printedonacid-freepaper 33/3100YL 543210
Preface
Towards the end of the 1960s, under the influence of the rapid development
of microelectronics, electromechanical cryptological machines began to be
replaced by electronic data encryption devices using large-scale integrated
circuits. This promised more secure encryption at lower prices. Then, in
1976, DiffieandHellmanopenedupthenewcryptologicalfieldofpublic-key
systems. Cryptography, hithertocloakedinobscurity, wasemergingintothe
publicdomain. Additionally,ENIGMArevelationsawokethepublicinterest.
Computer science was a flourishing new field, too, and computer scientists
became interested in several aspects of cryptology. But many of them were
notwellenoughinformedaboutthecenturies-longhistoryofcryptologyand
the high level it had attained. I saw some people starting to reinvent the
wheel, and others who had an incredibly naive belief in safe encryption,
and I became worried about the commercial and scientific development of
professional cryptology among computer scientists and about the unstable
situation with respect to official security services.
ThispromptedmetoofferlecturesonthissubjectattheMunichInstituteof
Technology. The first series of lectures in the winter term 1977/78, backed
by the comprehensive and reliable book The Codebreakers (1967) by David
Kahn, was held under the code name ‘Special Problems of Information
Theory’ and therefore attracted neither too many students nor too many
suspicious people from outside the university.
Next time, in the summer term of 1981, my lectures on the subject were
announced under the open title ‘Cryptology’. This was seemingly the first
publicly announced lecture series under this title at a German, if not indeed
a Continental European, university.
Theseriesoflectureswasrepeatedafewtimes,andin1986/87lecturenotes
wereprintedwhichfinallydevelopedintoPartIofthisbook. Activeinterest
on the side of the students led to a seminar on cryptanalytic methods in the
summer term of 1988, from which Part II of the present book originated.
The1993firstedition(inGerman)ofmybookKryptologie,althoughwritten
mainly for computer science students, found lively interest also outside the
field. It was reviewed favorably by some leading science journalists, and
the publisher followed the study book edition with a 1995 hardcover edition
under the title Entzifferte Geheimnisse [Decrypted Secrets], which gave me
the opportunity to round out some subjects. Reviews in American journals
recommendedalsoanEnglishversion,whichledin1997tothepresentbook.
It has become customary among cryptologists to explain how they became
acquainted with the field. In my case, this was independent of the Second
World War. In fact, I was never a member of any official service—and I
VI Preface
consider this my greatest advantage, since I am not bound by any pledge of
secrecy. Ontheotherhand,keepingeyesandearsopenandreadingbetween
the lines, I learned a lot from conversations (where my scientific metier was
a good starting point), although I never know exactly whether I am allowed
to know what I happen to know.
Itallstartedin1951,whenItoldmyformerprofessor
of formal logic at Munich University, Wilhelm Brit-
zelmayr, of my invention of an error-correcting code
forteletypelines1. Thiscausedhimtomakeawrong
association, and he gave me a copy of Sacco’s book,
whichhadjustappeared2. Iwaslucky, foritwasthe
best book I could have encountered at that time—
although I didn’t know that then. I devoured the
book. Noticing this, my dear friend and colleague
Paul August Mann, who was aware of my acquain-
tance with Shannon’s redundancy-decreasing encod-
LuigiSacco(1883–1970)
ing, gave me a copy of the now-famous paper by
Claude Shannon called Communication Theory of Secrecy Systems3 (which
in those days as a Bell Systems Technical Report was almost unavailable in
Germany). I was fascinated by this background to Shannon’s information
theory, which I was already familiar with. This imprinted my interest in
cryptology as a subfield of coding theory and formal languages theory, fields
that held my academic interest for many years to come.
Strange accidents—or maybe sharper observation—then brought me into
contact with more and more people once close to cryptology, starting with
Willi Jensen (Flensburg) in 1955, Karl Stein (Munich) in 1955, Hans Rohr-
bach,mycolleagueatMainzUniversity,in1959,aswellasHelmutGrunsky,
Gisbert Hasenja¨ger, and Ernst Witt. In 1957, I became acquainted with
Erich Hu¨ttenhain (Bad Godesberg), but our discussions on the suitability of
certain computers for cryptological work were in the circumstances limited
by certain restrictions. Among the American and British colleagues in nu-
merical analysis and computer science I had closer contact with, some had
been involved with cryptology in the Second World War; but no one spoke
aboutthat,particularlynotbefore1974,theyearwhenWinterbotham’sbook
TheUltraSecretappeared. In1976,IheardB.RandallandI.J.Goodreveal
somedetailsabouttheColossiinasymposiuminLosAlamos. Asascience-
oriented civilian member of the cryptology academia, my interest in cryp-
tology was then and still is centered on computerized cryptanalysis. Other
aspectsofsignalsintelligence(‘SIGINT’),forexample,trafficanalysisanddi-
rectionfinding,arebeyondthescopeofthisbook;thesameholdsforphysical
devices that screen electromechanical radiation emitted by cipher machines.
1 DBPNo.892767,applicationdateJanuary21,1951.
2 G´en´eralLuigiSacco,ManueldeCryptographie. Payot,Paris1951.
3 BellSystemsTechnicalJournal28,Oct.1949,pp.656–715.
Preface VII
Cryptology is a discipline with an international touch and a particular ter-
minology. It may therefore be helpful sometimes to give in this book some
explanations of terms that originated in a language other than English.
Thefirstpartofthisbookpresentscryptographicmethods. Thesecondpart
coverscryptanalysis, aboveallthefactsthatareimportantforjudgingcryp-
tographic methods and for saving the user from unexpected pitfalls. This
follows from Kerckhoffs’ maxim: Only a cryptanalyst can judge the secu-
rityofacryptosystem. Atheoreticalcourseoncryptographicmethodsalone
seems to me to be bloodless. But a course on cryptanalysis is problematic:
Either it is not conclusive enough, in which case it is useless, or it is conclu-
sive,buttouchesasensitivearea. Thereislittleclearanceinbetween. Ihave
triedtocoveratleastalltheessentialfactsthatareintheopenliteratureor
can be deduced from it. No censorship took place.
Certaindifficultiesarecausedbythefactthatgovernmentalrestrictionsdur-
ing and after World War II, such as the ‘need to know’ rule and other gim-
micks,misledevenpeoplewhohadbeenclosetothecentersofcryptanalysis.
Examples include the concept of Banburismus and the concept of a ‘cilli’.
ThewordBanburismus—thenamewascoinedinBritain—wasmentionedin
1985 by Deavours and Kruh in their book, but the method was only vaguely
described. Likewise, the description Kahn gave in 1991 in his book is rather
incomplete.On the other hand, in Kozaczuk’s book of 1979 (English edi-
tion of 1984),Rejewski gave a description of Ro´z˙ycki’s‘clock method’, which
turned out to be the same—but most of the readers could not know of this
connection. Then, in 1993, while giving a few more details on the method,
Good (in ‘Codebreakers’) confirmed that “Banburism was an elaboration
of...the clock method...[of]...Ro´z˙ycki”. He also wrote that this elabora-
tion was ‘invented at least mainly by Turing’, and referred to a sequential
Bayesianprocessasthe“methodofscoring”. Forlackofdeclassifiedconcrete
examples, the expositioninSect.19.4.2 of the present book, based on the re-
cently published postwar notes of Alexander and of Mahon and articles by
ErskineandbyNoskwithintherecentbookActionThisDay, cannotyetbe
a fully satisfactory one. And as to cillies, even Gordon Welchman admitted
that he had misinterpreted the origin of the word, thinking of ‘silly’. Other
publicationsgaveotherspeculations, seeSect.19.7, fn.29. RalphErskine, in
Action This Day, based on the recently declassified ‘Cryptanalytic Report
on the Yellow Machine’, 71-4 (NACP HCC Box1009, Nr.3175), gives the
following summary of the method:
‘Discovered by Dilly Knox in late January 1940, cillies reduced enormously
the work involved in using the Zygalski sheets, and after 1 May, when the
Zygalskisheetsbecameuseless,theybecameavitalpartofbreakingEnigma
by hand during most of 1940. They were still valuable in 1943.
Cillies resulted from a combination of two different mistakes in a multi-part
message by some Enigma operators. The first was their practice of leaving
therotorsuntouchedwhentheyreachedtheendofsomepartofthemessage.
Sincethelettercountofeachmessagepartwasincludedinthepreamble,the
messagekeyoftheprecedingpartcouldbecalculatedwithinfinelimits. The
seconderrorwastheuseofnon-randommessagekeys—stereotypedkeyboard
touches and 3-letter-acronyms. In combination, and in conjunction with the
differentturnoverpointsofrotorsItoV,theyallowedonetodeterminewhich
rotors could, and which could not, be in any given position in the machine.’
Although Banburismus and cillies were highly important in the war, it is
hard to understand why Derek Taunt in 1993 was prevented by the British
censor from telling the true story about cillies. Possibly, the same happened
to Jack Good about Banburismus.
***
My intellectual delight in cryptology found an application in the collection
‘Informatik’ of the Deutsches Museum in Munich which I built up in 1984
–1988, where there is a section on cryptological devices and machines. My
thanksgototheDeutschesMuseumforprovidingcolorplatesofsomeofthe
pieces on exhibit there.
And thanks go to my former students and co-workers in Munich, Manfred
Broy,HerbertEhler,andAntonGeroldforcontinuingsupportovertheyears,
moreover to Hugh Casement for linguistic titbits, and to my late brother-
in-law Alston S.Householder for enlightenment on my English. Karl Stein
andOttoLeiberichgavemedetailsontheENIGMAstory,andIhadfruitful
discussionsandexchangesofletterswithRalphErskine,HeinzUlbricht,Tony
Sale,FrodeWeierud,Kjell-OveWidman,OttoJ.Horak,GilbertBloch,Arne
Frans´en, and Fritz-Rudolf Gu¨ntsch. Great help was given to me by Kirk
H.Kirchhofer from Crypto AG, Zug (Switzerland). Hildegard Bauer-Vogg
suppliedtranslationsofdifficultLatintexts,MartinBauer,UlrichBauerand
Bernhard Bauer made calculations and drawings. Thanks go to all of them.
The English version was greatly improved by J.Andrew Ross, with whom
working was a pleasure. In particular, my sincere thanks go to David Kahn
who encouraged me (“The book is an excellent one and deserves the widest
circulation”)andmadequiteanumberofproposalsforimprovementsofthe
text. For the present edition, additional material that has been made public
recently has been included, among others on Bletchley Park, the British at-
tackonTunny,ColossusandMaxNewman’spioneeringwork. Moreover,my
particular thanks go to Ralph Erskine who indefatigably provided me with
a lot of additional information and checked some of the dates and wordings.
In this respect, my thanks also go to Jack Copeland, Heinz Ulbricht, and
Augusto Buonafalce. Finally, I have to thank once more Hans Wo¨ssner for
a well functioning cooperation of long standing, and the new copy editor
Ronan Nugent for very careful work. The publisher is to be thanked for the
finepresentationofthebook. AndIshallbegratefultoreaderswhoarekind
enough to let me know of errors and omissions.
Grafrath, Spring 2006 F.L.Bauer
Contents
Part I: Cryptography—The People ................................. 1
1 Introductory Synopsis ...........................................9
1.1 Cryptography and Steganography...................................9
1.2 Semagrams....................................................... 10
1.3 Open Code: Masking............................................. 13
1.4 Cues.............................................................. 17
1.5 Open Code: Veiling by Nulls...................................... 19
1.6 Open Code: Veiling by Grilles .................................... 23
1.7 Classification of Cryptographic Methods.......................... 24
2 Aims and Methods of Cryptography......................... 26
2.1 The Nature of Cryptography...................................... 26
2.2 Encryption....................................................... 32
2.3 Cryptosystems.................................................... 34
2.4 Polyphony........................................................ 36
2.5 Character Sets.................................................... 39
2.6 Keys............................................................. 41
3 Encryption Steps: Simple Substitution...................... 44
3.1 Case V(1) −−−(cid:1)W (Unipartite Simple Substitutions) ............. 44
3.2 Special Case V ≺−−−−(cid:1)V (Permutations).......................... 46
3.3 Case V(1)−−−(cid:1)Wm (Multipartite Simple Substitutions)........... 53
3.4 The General Case V(1) −−−(cid:1)W(m), Straddling ................... 55
4 Encryption Steps: Polygraphic Substitution and Coding . 58
4.1 Case V2 −−−(cid:1) W(m) (Digraphic Substitutions) .................. 58
4.2 Special Cases of Playfair and Delastelle: Tomographic Methods.... 64
4.3 Case V3 −−−(cid:1) W(m) (Trigraphic Substitutions) .................. 68
4.4 The General Case V(n) −−−(cid:1)W(m) : Codes ...................... 68
5 Encryption Steps: Linear Substitution........................80
5.1 Self-reciprocal Linear Substitutions................................ 82
5.2 Homogeneous Linear Substitutions................................ 82
5.3 Binary Linear Substitutions....................................... 86
5.4 General Linear Substitutions...................................... 86
5.5 Decomposed Linear Substitutions................................. 87
X Contents
5.6 Decimated Alphabets............................................ 90
5.7 Linear Substitutions with Decimal and Binary Numbers ......... 91
6 Encryption Steps: Transposition ............................ 93
6.1 Simplest Methods................................................ 93
6.2 Columnar Transpositions........................................ 98
6.3 Anagrams...................................................... 102
7 Polyalphabetic Encryption: Families of Alphabets.........106
7.1 Iterated Substitutions............................................106
7.2 Cyclically Shifted and Rotated Alphabets........................107
7.3 Rotor Crypto Machines..........................................110
7.4 Shifted Standard Alphabets: Vigen`ere and Beaufort .............127
7.5 Unrelated Alphabets.............................................131
8 Polyalphabetic Encryption: Keys ...........................139
8.1 Early Methods with Periodic Keys ..............................139
8.2 ‘Double Key’ ...................................................141
8.3 Vernam Encryption..............................................142
8.4 Quasi-nonperiodic Keys..........................................144
8.5 Machines that Generate Their Own Key Sequences...............145
8.6 Off-Line Forming of Key Sequences..............................156
8.7 Nonperiodic Keys................................................158
8.8 Individual, One-Time Keys......................................161
8.9 Key Negotiation and Key Management...........................165
9 Composition of Classes of Methods ........................169
9.1 Group Property.................................................169
9.2 Superencryption.................................................171
9.3 Similarity of Encryption Methods................................173
9.4 Shannon’s ‘Pastry Dough Mixing’................................174
9.5 Confusion and Diffusion by Arithmetical Operations..............180
9.6 DES and IDEA(cid:3)R ................................................184
10 Open Encryption Key Systems .............................193
10.1 Symmetric and Asymmetric Encryption Methods.................194
10.2 One-Way Functions..............................................196
10.3 RSA Method....................................................203
10.4 Cryptanalytic Attack upon RSA.................................205
10.5 Secrecy Versus Authentication...................................208
10.6 Security of Public Key Systems..................................210
11 Encryption Security..........................................211
11.1 Cryptographic Faults............................................211
11.2 Maxims of Cryptology...........................................220
11.3 Shannon’s Yardsticks............................................225
11.4 Cryptology and Human Rights...................................226
Contents XI
Part II: Cryptanalysis—The Machinery ............................233
12 Exhausting Combinatorial Complexity .....................237
12.1 Monoalphabetic Simple Encryptions.............................238
12.2 Monoalphabetic Polygraphic Encryptions........................239
12.3 Polyalphabetic Encryptions......................................241
12.4 General Remarks on Combinatorial Complexity..................244
12.5 Cryptanalysis by Exhaustion.....................................244
12.6 Unicity Distance ................................................246
12.7 Practical Execution of Exhaustion...............................248
12.8 Mechanizing the Exhaustion.....................................251
13 Anatomy of Language: Patterns ............................252
13.1 Invariance of Repetition Patterns................................252
13.2 Exclusion of Encryption Methods................................254
13.3 Pattern Finding.................................................255
13.4 Finding of Polygraphic Patterns.................................259
13.5 The Method of the Probable Word...............................259
13.6 Automatic Exhaustion of the Instantiations of a Pattern.........264
13.7 Pangrams.......................................................266
14 Polyalphabetic Case: Probable Words .....................268
14.1 Non-Coincidence Exhaustion of Probable Word Position .........268
14.2 Binary Non-Coincidence Exhaustion .............................271
14.3 The De Viaris Attack............................................272
14.4 Zig-Zag Exhaustion of Probable Word Position...................280
14.5 The Method of Isomorphs .......................................281
14.6 A clever brute force method: EINSing............................287
14.7 Covert Plaintext-Cryptotext Compromise........................288
15 Anatomy of Language: Frequencies ........................ 290
15.1 Exclusion of Encryption Methods................................290
15.2 Invariance of Partitions..........................................291
15.3 Intuitive Method: Frequency Profile..............................293
15.4 Frequency Ordering..............................................294
15.5 Cliques and Matching of Partitions...............................297
15.6 Optimal Matching ..............................................303
15.7 Frequency of Multigrams........................................305
15.8 The Combined Method of Frequency Matching...................310
15.9 Frequency Matching for Polygraphic Substitutions...............316
15.10 Free-Style Methods..............................................317
15.11 Unicity Distance Revisited.......................................318
16 Kappa and Chi .............................................. 320
16.1 Definition and Invariance of Kappa.............................. 320
16.2 Definition and Invariance of Chi................................. 323
16.3 The Kappa-Chi Theorem........................................ 325
16.4 The Kappa-Phi Theorem........................................ 326
16.5 Symmetric Functions of Character Frequencies.................. 328