Collected Papers of Claude E. Shannon

954 Pages·1993·19.206 MB·English

by  Claude E. Shannon, N.J.A. Sloane, Aaron D. Wyner

Claude Elwood Shannon
Claude Elwood Shannon
Collected Papers
Edited by
N.J.A. Sloane
Aaron D. Wyner
Mathematical Sciences Research Dept.
AT&T Bell Laboratories
Murray Hill, NJ 07974
USA
IEEE Information Theory Society, Sponsor
+IEEE
TheInstituteofElectricaland ElectronicsEngineers.Inc.,NewYork
mWILEY-
~INTERSCIENCE
AJOHNWILEY &SONS,INC.,PUBLICATION
IEEEPRESS
445 Hoes Lane,PO Box1331
Piscataway,NJ 08855-1331
1992EditorialBoard
William Perkins, Editorin Chief
K.K.Agarwal G. F. Hoffnagle A. C.Schell
R.S.Blicq J. D.Irwin L.Shaw
R. C. Dorf A. Michel M.Simaan
D.M. Etter E.K. Miller Y.Sunahara
J.J. Farrell III J.M. F.Moura D.J. Wells
K. Hess J. G. Nagle
DudleyR. Kay,ExecutiveEditor
Carrie Briggs,AdministrativeAssistant
Karen G. Miller, ProductionEditor
IEEEInformationTheorySociety, Sponsor
G. DavidForney,Jr.
President
© 1993THE INSTITUTEOF ELECTRICALANDELECTRONICS
ENGINEERS, INC. 3Park Avenue, 17thFloor, New York, NY 10016-5997
No partofthis publication may bereproduced, stored inaretrievalsystemor transmitted
inany Iorm or by any means, electronic, mechanical,photocopying,recording, scanning
or otherwise,exceptas permittedunder Sections 107or 108of the 1976 United States
CopyrightAct, withouteithertheprior written permissionof the Publisher,or
authorization through paymentoftheappropriateper-copy feetotheCopyright
ClearanceCenter, 222 RosewoodDrive, Danvers,MA01923,[phone], fax
[phone]. Requests tothePublisherforpermissionshouldbe addressed tothe
PermissionsDepartment,John Wiley& Sons, Inc., 111River Street, Hoboken, NJ 07030,
(20) 748-6011, fax [phone].
LibraryofCongressCataloging-In-Publication Data
Shannon,Claude Elwood,
[Works]
ClaudeElwood Shannon:collectedpapers/ edited byN.J.A.
Sloane,A.D. Wyner.
p. em.
Includesbibliographical references.
ISBN0-7803-0434-9
1. Telecommunication. 2. Information theory. 3. Computers.
I. Sloane, N.J. A. (Neil James Alexander). II. Wyner,A.
D.(Aaron D.) III. Title.
TK5101.S448 1993
621.382-dc20 92-26762
CIP
Contents
Introduction ix
BiographyofClaude Elwood Shannon xi
Profile ofClaude Shannon- Interview by Anthony Liversidge xix
BibliographyofClaude ElwoodShannon xxxv
Part A: Communication Theory, Information Theory, Cryptography
[Bracketednumbersrefertothe Bibliography]
Preface to Part A 3
PAPERS
[37] A mathematical theory ofcommunication 5
[251 Communicationtheory ofsecrecysystems 84
[15] Analogueofthe Vemam system for continuoustime series 144
[22] The bestdetectionofpulses 148
[40] (with B. M. OliverandJ. R. Pierce)The philosophy ofpeM 151
[43] Communication in the presenceofnoise 160
[60] Communication theory- exposition offundamentals 173
[61f General treatmentofthe problemofcoding 177
[62] The lattice theoryofinformation 180
[63] Discussion ofprecedingthreepapers 184
[65] Recent developments in communication theory 190
[69] Prediction and entropy ofprinted English 194
[86] Efficientcodingofabinary source with onevery infrequentsymbol 209
[100] Informationtheory 212
[1091 The zeroerrorcapacityofa noisychannel 221
[t11] Certain results in codingtheory for noisy channels 239
[113] Somegeometrical results in channelcapacity 259
[It5] A note on a partial orderingfor communicationchannels 265
[116] Channels with side informationat the transmitter 273
[117] Probability oferrorfor optimal codes in aGaussianchannel 279
[118] Coding theorems for a discrete source with a fidelity criterion 325
[119] Two-way communication channels 351
[122] (with R. G. Gallagerand E. R. Berlekamp)Lowerbounds to error 385
probabilityforcodingon discretememorylesschannels I
[123] (with R. G. Gallagerand E. R. Berlekamp)Lowerbounds to error 424
probability for codingon discrete memorylesschannels II
v
VI Contents
ABSTRACTS, ETC.
[2] Letterto VannevarBush 455
[21] (with B. M. Oliver)Circuitsfor aP.C.M. transmitterand receiver 457
[68] Sometopics in informationtheory 458
[95] Concavityoftransmission rate as afunction ofinput probabilities 460
[102] The rate ofapproach to ideal coding 461
[103J The bandwagon 462
Notes to Part A 463
Part B: Computers, Circuits, Games
Prefaceto Part B 469
PAPERS
[1] Asymbolic analysisofrelay and switchingcircuits 471
[6] Mathematical theoryofthedifferential analyzer 496
[13] The theoryand design oflineardifferentialequationmachines 514
[14] (WithJohn Riordan)The numberoftwo-terminal series-parallel networks 560
[42] Network rings 571
[44] Atheoremon coloring the lines ofanetwork 584
[50] The synthesisoftwo-terminal switchingcircuits 588
[51] (with H. W. Bode) A simplifiedderivationoflinearleastsquare smoothing 628
and prediction theory
[54] Programming acomputerfor playingchess 637
[55J Achess-playing machine 657
[56] Memory requirements inatelephoneexchange 667
[57] Asymmetrical notationfornumbers 674
[67] Amethodofpoweror signal transmission toa moving vehicle 678
[70] Presentationofamaze solvingmachine 681
[73] Amind-reading (1)machine 688
[75] The potentialitiesofcomputers 691
[76] Throbac I 695
[80] (with E. F. Moore) Machineaid for switchingcircuitdesign 699
[82J Computersand automata 703
[83] Realizationofall 16switchingfunctions oftwo variables requires 18contacts 711
[85] (with D. W. Hagelbarger)Arelay laboratoryoutfitfor colleges 715
[91] (editedjointly with John McCarthy)AutomataStudies (Preface,etc.) 727
Contents vii
[931 Auniversal Turing machine withtwointernalstates 733
[94] (with Karelde Leeuw, EdwardF. MooreandN. Shapiro)Computability by 742
probabilistic machines
[96] Some resultson idealrectifiercircuits 772
[97] The simultaneous synthesis ofsswitchingfunctionsofnvariables 780
[98] (with D. W. Hagelbarger) Concavity ofresistancefunctions 784
[99] Game playing machines 786
[110] (with Peter Eliasand Amiel Feinstein) A noteonthemaximum flowthrough 793
anetwork
[89] (with Edward F.Moore) Reliablecircuits usinglessreliablerelays I 796
[90] (with Edward F. Moore)Reliablecircuitsusinglessreliablerelays II 814
[114J VonNeumann'scontributions toautomata theory 831
[120) Computers andautomation-Progressandpromiseinthetwentieth century 836
[125J Claude Shannon's no-dropjuggling diorama 847
(126] Scientificaspectsofjuggling 850
ABSTRACTS, ETC.
[5] The useoftheLakatos-Hickman relayinasubscribersender 865
[7J Astudyof thedeflection mechanism andsomeresultsonratefinders 866
[8] Backlash inoverdamped systems 868
[1JJ Someexperimental resultson thedeflectionmechanism 869
[30] (withC. L.Dolph)The transient behaviorofalargenumberoffour-terminal 870
unilateral linearnetworks connected intandem
[52] Review ofTransformations on Latticesand Structureso.fLogic 871
byStephen A. Kiss
[53] Review ofCybernetics,or Controland Communication inthe Animal 872
andthe Machine by NorbertWiener
(64] ReviewofDescription ofaRelayCalculator, bythestaffofthe [Harvard] 874
Computation Laboratory
[74] (withE.F.Moore)The relaycircuit analyzer 875
[84J (withE.F.Moore)The relaycircuit synthesizer 876
Notesto PartB 877
Part C: Genetics
PrefacetoPartC 889
DOCTORAL DISSERTATION
[3] Analgebra fortheoretical genetics 891
Notesto PartC 921
Permissions 923
Introduction
The publication of Claude Shannon's collected papers is long overdue. A substantial
collection ofhis papers was published in Russian in 1963(see item [121] of his Bibliography),
but no English edition has ever appeared. The editors were therefore commissioned by the
Information Theory Society of the Institute of Electrical and Electronics Engineers to collect
and publish his papers.
Since much of Shannon's work was never published, our first task was to assemble a
complete bibliography. We did this by consulting Claude and Betty Shannon, who have been
extremely helpful throughout this project and supplied us with copies of a number of
unpublished items; many of Claude's friends and colleagues; the Bell Laboratories Archives;
the National Archives in Washington; the National Security Agency; the patent records;
Mathematical Reviews; and other sources. We believe the resulting bibliography of 127 items
isreasonably complete.
The second step was to decide what to include. Our policy has been to include everything
of importance. We have includedall the publishedpapers, and all the unpublishedmaterial that
seemedof lastingvalue. Some war-time reports of very limited interesthave been excluded, as
well as the M.LT. seminar notes. Ifan excluded paperhas an abstract, we have printed it. We
have made several sets of copies of the excluded material, and plan to deposit copies in the
AT&T Bell Laboratories library at Murray Hill, New Jersey, the M.LT. library, and the Library
ofCongress and the BritishLibrary.
The papers fall naturally into three groups: (A) communication theory, information theory
and cryptography; (8)computers, circuits and games; (C) the hitherto unpublished doctoral
dissertation on population genetics. Inside each group the papers are, with some exceptions,
arranged in chronological order. Minor items (abstracts, book reviews, and so on) have been
placedatthe end ofeach section.
Most of the published works have been photographically reproduced from the originals,
whilethe others have been typesetbycomputerat AT&T Bell Labs.
The "Notes" following each section give references to more recent work.
We should like to thank R. B. Blackman, P. Elias, E. N. Gilbert, R. Gnanadesikan,
R. L.Graham, D. W. Hagelbarger, T. T. Kadota, H. O. Pollak, D. Slepian, E. Wolman and
R. Wright for supplying us with copies of Shannon's papers. R. A. Matula, of the AT&T Bell
Laboratories library staff, has been extremely helpful to us throughout this project. J. P.
Buhler,J. H. Conway,J. F. Crow, R. L.Graham, D. S.Johnson, T. Nagylaki and K. Thompson
kindly provided comments on some of the papers. We are very grateful to Susan Marko
(sometimes assisted by Sue Pope), whoexpertlyretypedmanyofShannon'spapers for us.
ix
Biography ofClaude Elwood Shannon
Claude Elwood Shannon was born in Petoskey, Michigan, on Sunday, April 30, 1916. His
father, Claude Sr. (1862-1934), a descendant of early New Jersey settlers, was a businessman
and, for a period, Judge ofProbate. His mother, Mabel Wolf Shannon (1880-1945), daughter
ofGerman immigrants, was a language teacher and for a numberofyears Principal ofGaylord
High School, inGaylord, Michigan.
The first sixteen years of Shannon's life were spent in Gaylord, where he attended the
Public School, graduating from Gaylord High School in 1932. As a boy, Shannon showed an
inclination toward things mechanical. His best subjects in school were science and
mathematics, and at home he constructed such devices as model planes, a radio-controlled
model boat and a telegraph system to a friend's house halfa mile away. The telegraph made
opportunistic use of two barbed wires around a nearby pasture. He earned spending money
from a paper route and delivering telegrams, as well as repairing radios for a local department
store. His childhood hero was Edison, who he later learned was a distant cousin. Both were
descendants of John Ogden, an important colonial leader and the ancestor of many
distinguished people. Shannon's recent hero list, without deleting Edison, includes more
academic types such as Newton, Darwin, Einstein and Von Neumann.
In 1932 he entered the University ofMichigan, following his sisterCatherine, who had just
received a master's degree in mathematics there. While a senior, he was elected a member of
Phi Kappa Phi and an associate member of SigmaXi. In 1936 he obtained the degrees of
Bachelor of Science in Electrical Engineering and Bachelor of Science in Mathematics. This
dual interest inmathematicsand engineeringcontinuedthroughout his career.
In 1936 he accepted the position of research assistant in the Department of Electrical
Engineering at the Massachusetts Institute of Technology. The position allowed him to
continue studying toward advanced degrees while working part-time for the department. The
work in question was ideally suited to his interests and talents. It involved the operation of the
Bushdifferential analyzer, the most advanced calculating machine ofthat era, which solved by
analog means differential equations of up to the sixth degree. The work required translating
differential equations into mechanical terms, setting up the machine and running through the
needed solutions for various initial values. In some cases as many as four assistants would be
needed tocrank infunctions by following curvesduringthe process ofsolution.
Also of interest was a complex relay circuit associated with the differential analyzer that
controlled its operation and involved over one hundred relays. In studying and servicing this
circuit, Shannon became interested in the theory and design ofrelay and switching circuits. He
had studied symbolic logic and Boolean algebra at Michigan in mathematics courses, and
realized that this was the appropriate mathematics for studying such two-valued systems. He
developed these ideas during the summer of 1937~ which he spent at Bell Telephone
Laboratories in New York City, and, back at M.LT., in his master's thesis, where he showed
how Boolean algebra could be used in the analysis and synthesis of switching and computer
circuits. The thesis, his first published paper, aroused considerable interest when it appeared in
1938 in the A.I.E.E. Transactions [I].* In 1940 it was awarded the Alfred Noble Prize of the
ThenumbersinsquarebracketsrefertoitemsinShannon'sbibliography.
xi
xii C.E.Shannon
combined engineering societies ofthe United States, an award given each year to a person not
over thirty for a paper published in one ofthe journals ofthe participating societies. A quarter
of a century later H. H. Goldstine, in his book The Computerfrom Pascal to Von Neumann,
called this work''one ofthe most important master's theses ever written...a landmark in that it
helped tochange digital circuitdesign from an art to ascience."
During the summerof 1938 he did research work at M.I.T. on the design ofthe Bush Rapid
Selector, and was mainly involved with the vacuum tube circuits employed in this device. In
Septemberof 1938, at the suggestion ofVannevar Bush, Shannon changed from the Electrical
Engineering Department to the Mathematics Department. He was awarded the Bolles
Fellowshipand was also a teaching assistant while working toward a doctorate in mathematics.
Bush had just been made President of the Carnegie Institution in Washington, one of whose
branches, in Cold Spring Harbor, N.Y., dealt with the science of genetics. He suggested to
Shannon that algebra might be as useful in organizing genetic knowledge as it was in
switching, and Shannon decided to look into this matter with a view toward using it for a
doctoral thesis in mathematics. He spent the summer of 1939 at Cold Spring Harbor working
undergeneticist Dr. Barbara Burks exploring the possibility, and found it a suitable subject for
a dissertation under the title "An Algebra for Theoretical Genetics" [3]. His Ph.D. supervisor
at M.I.T. was ProfessorFrank L. Hitchcock, an algebraist. In the Springof 1939 he was elected
to full membership inSigmaXi.
At about this time Shannon was also developing ideas both in computers and
communications systems. In a letter of February 16, 1939 now in the Library of Congress
archives ([2], included in Part A), he writes to Bush about trading relations between time,
bandwidth, noise and distortion in communication systems, and also about a computer design
for symbolic mathematical operations.
As the Springof 1940 approached, Shannon had passed all requirements for both a master's
in electrical engineering and a doctorate in mathematics - except for satisfying the language
requirements, always his weakest subjects. Facing reality, he buckled down in the last few
months, hired a French and German tutor and repeatedly worked his way through stacks of
flash cards. He finally passed the language exams (it took two tries with German) and in the
Spring of 1940 received the S.M. degree in Electrical Engineering and the degree ofDoctorof
Philosophy in Mathematics at the same commencement. His Ph.D. dissertation, [3], is
published here for the firsttime (in Part D).
The Summer of 1940 was spent at Bell Telephone Laboratories doing further research on
switching circuits. A new method ofdesign was developed which greatly reduced the number
of contacts needed to synthesize complex switching functions from earlier realizations. This
was laterpublished ina paper, "TheSynthesisofTwo-Terrninal Switching Circuits" [50].
The academic year 1940-1941 was spent on a National Research Fellowship at the Institute
for Advanced Study in Princeton working under Hermann Weyl. It was during this period that
Shannon began to work seriously on his ideas relating to information theory and efficient
communication systems.
Thornton C. Fry, head of the mathematics department at Bell Labs, was in charge of a
committee on fire control systems for anti-aircraft use - the country was arming up at the time
because of the spreading European war threats - and asked Shannon to join in this effort.
Returning to Bell Labs, Shannon joined a team working on anti-aircraft directors - devices to
observe enemy planes or missiles and calculate the aiming ofcounter missiles. This problem
became crucial with the developmentofthe German VI and V2 rockets. Withoutthe American
anti-aircraftdirectors, the ravaging ofEngland, bad as it was, would have been vastly worse.
BiographyofClaudeElwoodShannon xiii
Shannon spent fifteen years at BeJl Laboratories in a very fruitful association. Many first-
rate mathematicians and scientists were at the Labs - men such as John Pierce, known for
satellite communication; Harry Nyquist, with numerous contributions to signal theory; Hendrik
Bode of feedback fame; transistor inventors Brattain, Bardeen and Shockley; George Stibitz,
who built an early (1938) relay computer; Barney Oliver, engineer extraordinaire; and many
others.
During this period Shannon worked in many areas, most notably in information theory, a
development which was published in 1948 as "A Mathematical Theory of Communication"
[37]. In this paper it was shown that all information sources- telegraph keys, people speaking,
television cameras and so on - have a "source rate" associated with them which can be
measured in bits per second. Communication channels have a "capacity" measured in the
same units. The information can be transmitted over the channel if and only if the source rate
does not exceedthe channelcapacity (see the PrefacetoPartA).
This work on communication is generally considered to be Shannon's most important
scientific contribution. In 1981 Professor Irving Reed, speaking at the International
Symposium on Information Theory in Brighton, England, said, "It was thirty-fouryears ago, in
1948, that Professor Claude E. Shannon first published his uniquely original paper, 'A
Mathematical Theory of Communication,' in the Bell System TechnicalJournal. Few other
works ofthis century have had greater impact on science and engineering. By this landmark
paper and his several subsequent papers on information theory he has altered most profoundly
all aspectsofcommunicationtheory and practice."
Shannon has rung many changes on the problems of information and noise. In a paper
"Communication Theory ofSecrecy Systems" [25] cryptography is related to communication
in a noisy channel, the "noise" being in this case the scrambling by the key of the
cryptographic system. This work later led to his appointment as a consultant on cryptographic
matters to the United StatesGovernment.
Another problem, which he investigated jointly with E. F. Moore [88]-[90], was that' of
increasing the reliability ofrelay circuits by redundant use ofcontacts, each of which may be
unreliable. Again this isa problem related to transmission in noisy channels.
Shannon has also applied these concepts to the problem of optimal investment strategies.
The "noisy signal" is the stock marketand related time series, and the problem is to maximize
a utility function by properchoice and adjustmentofa portfolio.
In a lighter vein and in the computer and artificial intelligence area, Shannon wrote a paper
"Programming a Computer for Playing Chess" in 1950 [54]. At that time computers were
slow, inept and very difficult to program. Since then, many chess-playing programs have been
written,most ofthem following quiteclosely the systemdescribed in that early paper.
In 1965he was invitedto Russiato give lectures at anengineeringconference. While there,
he took the opportunity to meet Mikhail Botvinnik,for many yearsthe WorldChessChampion.
Botvinnik, also an electrical engineer, had become interested in the chess programming
problem. Shannon remembers the discussion as interesting but carried on through a somewhat
noisy channel since the interpretersknew little ofeitherchess or computers.
After the discussion, he asked Botvinnik for the pleasure of a chess game. Translators,
guides and members of the American party watched with rapt attention as the epic battle
unfolded. At one point Shannon had a slightedge (a rook for a knight and pawn), but alas the
result was foregone - after forty-two moves Shannon tipped over his king - a message that
needed no translation.