Selected Works of A.N. Kolmogorov. Volume III: Information Theory and the Theory of Algorithms

300 Pages·1993·9.65 MB·English

by  Shiryayev A. N.

Selected Works of A. N. Kolmogorov
Mathematics and Its Applications (Soviet Series)
Managing Editor:
M.HAZEWINKEL
Centrefor Mathematics and Computer Science, Amsterdam, The Netherlands
Editorial Board:
A. A. KIRILLOV, MGU, Moscow, Russia
Yu. I. MAN IN, Steklov Institute of Mathematics, Moscow, Russia
N. N. MOISEEV, Computing Centre, Academy ofSciences, Moscow, Russia
S. P. NOVIKOV, Landau Institute ofTheoretical Physics, Moscow, Russia
Yu. A. ROZANOV, Steklov Institute of Mathematics, Moscow. Russia
Volume27
Selected Wo rks of
A. N. Kolmogorov
Volume 111
Information Theory and
the Theory of Algorithms
edited by
A. N. Shiryayev
Translatedfrom the Russian by A. B. Sossinsky
..
SPRINGER-SCIENCE+BUSINESS MEDIA, B.V.
ISBN 978-90-481-8456-9 ISBN 978-94-017-2973-4 (eBook)
DOI 10.1007/978-94-017-2973-4
Printed on acid-free paper
This is an annotated translation from the original work
TEOPHß HH4POPMAQHH H TEOPHß A.lIrOPHTMOB
Published by Nauka, Moscow, © 1987
All Rights Reserved
© 1993 Springer Science+Business Media Dordrecht
Originally published by Kluwer Academic Publishers in 1993
No part of the material protected by this copyright notice may be reproduced or
utilized in any fonn or by any means, electronic or mechanical,
incJuding photocopying, recording or by any infonnation storage and
retrieval system, without written pennission from the copyright owner.
SERIES EDITOR 'S PREFACE
'BI moi. ...• si j'aYlliI SII COIIIIDeIIt CD n:valir. je ODe semce m"'-""ies bas rendered tbe
u'y serais point aD6.· bamIIII 18Ile. It bas pat COIIIIDOD _ bei<
JaJesVeme wbae it beJcJup, Oll tbe topmosl sbeIf _ In
tbe dnsty c:aniater labeIIccl ·clilcardecllIIlIISeIIIe'.
1be series is cIiverp:nt; tberefOle _ may be BrieT.BeO
lIbIe In do SOIIleIbiDg wilh it
O. Heaviside
Mathematics is a tool for thought. A highly necessary tool in a world whem both feedback and nonlincari
ties abound. Similarly, aIl kinds of parts of mathematics serve as tools for othcr parts and for othcr sci
enccs.
Applying a simple mwriting ruIe to the quotc on the right above one finds such statements as: 'One ser
vice topology has mndemd mathematical physics ... '; 'One sCIVicc logic has mndcmd computer science
.. .'; 'One service category thcory has rendcmd mathematics .. .'. All arguably true. And aIl statements
obtainable this way form part of the raison d'etm of this scrics.
This scrics. Mathematics and Its Applications, startcd in 1977. Now that ovcr one hundrcd volumes have
appcarcd it sccms opportune to rcexamine its scope. At thc time I wrote
"Growing specialization and divcrsüication have brought a host of monographs and textbooks
on incmasingly specialized topics. Howevcr. thc '!me' of knowlcdge of mathematics and
relatcd fields docs not grow only by putting forth new branchcs. It also happens, quite often in
fact, that branches which wem thought to be complctcly disparate arc suddcnly seen to be
related. Furthcr. the kind and level of sophistication of mathematics appüed in various sci
ences has changcd drasticaIly in rcc:cnt years: measum thcory is uscd (non-trivia1ly) in
regional and thcorctica1 cconomics; a1gebraic gcomctry intcracts with physics; the Minkowsky
lemma, coding theory and the structurc of watcr mcct onc anothcr in packing and covering
thcory; quantum fields. aystal defccts and mathematical programming profit from homotopy
theory; Lie algebras arc relevant to filtcring; and pmdiction and elcctrical engineering can use
Stein spaccs. And in additioo to this there arc such ncw emcrging subdiscipÜDCS as 'cxperi
mental mathematics', 'CPO', 'completely integrable systems', 'chaos, syncrgetics and large
scale order', which arc aImost impossible to fit into the cxisting classificatioo schemes. Thcy
draw upon widely different scctions of mathematics."
By and large, aIl this still applies today. It is still truc that at first sight mathematics sccms rathcr frag
mented and that to find. sec. and exploit the dcepcr undcr1ying intcrrclations morc cffon is nccdcd and so
arc books that can help mathematicians and scientists do so. Accordingly MIA will continue to try to make
such books available.
If anything. the description I gave in 1977 is now an understatement. To the examples of interaction
areas one should add string theory whem Riemann surfaccs, algcbraic gcomctry, modular functions, knots,
quantum field theory. Kac-Moody algebras, monstrous moonshine (and morc) aIl come togethcr. And to
the examples ofthings which can be uscfully applied let me add the topic 'finite gcomctry'; a combination
of words which sounds Iikc it might not even exist, let alonc bc appücable. And yct it is being applied: to
statistics via designs, to radar/sonar dctection arrays (via finite projcctive planes), and to bus connections
of VLSI chips (via difference sets). Therc sccms to be no part of (so-caIled pure) matbcmatics that is not
in immediate danger of being applied. And, accordingly, the appüed mathcmatician nceds to be awarc of
much more. Besidcs analysis and nummcs, the traditional workhorscs, he may nccd aIl kinds of combina
torlcs. algebra, probability , and so on.
In addition. the appücd scientist nccds to cape incrcasingly with thc nonlinear world and the extra
vi
mathematical sophistication that this requires. For that is where the rewards are. Linear models are honest
and a bit sad and depressing: proportional efforts and results. It is in the nonIinear world that infinitesimal
inputs may result in macroscopic outputs (or vice versa). To appreciate wbat I am bioting at: if electronics
were linear we would have no f1m with transistors and computers; we would have no TV; in fact you
would not be reading these lines.
There is also no safety in ignoring such oudandish things as nonstandaId analysis, superspace and
anticommuting integration, p-adic and ultrametrie space. All three have applications in both electrical
engineering and physics. Once, complex numbers were equally outlandish, but they frequendy proved the
shortest path between 'real' results. Similarly, the first two topics named have already provided a number
of 'wormhole' paths. Tbere is no telling where alI this is leading -fonunately.
Tbus the original scape of the series, which for various (sound) reasons now comprises five subseries:
white (Japan), yellow (Olina), red (USSR), blue (Bastern Europe), and green (everything else), still
applies. It has been enlarged a bit to include books treating of the tools from one subdiscipline which are
used in others. Thus the series still aims at books dealing with:
a central concept which plays an imponant role in seveml different mathematical anellor scientific
specialization areas;
new applications of the results and ideas from one area of scientific endeavour into another;
in1luences which the results, problems and concepts of ODe field of enquiry have, and have had, on the
development of another.
Tbe roots of much that is now possible using mathematies, the stock it grows on, much of that goes back to
A.N. Kolmogorov, quite possibly the finest mathematician ofthis century. He solved outstanding problems
in established fields, and created whole new ones; the word 'specialism' did not exist for him.
A main driving idea behind this series is the deep interconnectedness of alI things mathematical (of
which much remains to be discovered). Such interconnectedness can be found in specially written mono
graphs, and in selected proceedings. It can also be found in the werk of a single seientist, especially one
like A.N. Kolmogorov in whose mind the dividing Iines between specialisms did not even exist.
Tbe present volume ia the third of a three volume collection of selected scientific papers of A.N. Kolmo
gorov with added commentary by the author himself, and additional surveys by others on the many
developments staned by Kolmogorov. His papers are scattered far and wide over many different journals
and they are in several languages; many have not been available in Fnglish before. If you can, as Abel
recommended, read and study the masters themselves; this collection makes that possible in the case of one
ofthe masters, A.N. Kolmogorov.
1be sbonest path betwec:D \WO IlUIbs in 1be teal Never leIId boob, for DO ODe ever _ tbcm;
cIomain passes tbrougb 1be compJex domain. 1be oaJy boob I bave in my h1nly _ boob
J.Hadamard Ibat 0Iber folk bave leDt me.
ADaIoIe Prmc:e
La physique De DOUS doooe PIS seaIemmt
I'occasioo de raoudn: des probImIes ... eile 1be fuIIc:tioa of an expert ja not tu be more rigbI
IIOUS fait presseDIir Ja solDlioD. Ibm otber people, bat tu be WIOD8 for moft:
H.PoiDcaR sopbistlcaled ft:8SODS.
DaYid Butler
Amsterdam, September 1992 Michiel Hazewinkel
CONTENTS
Series Editor's Preface ...... . . . . . . . . . . . . . .. . v
Greetings to A. N. Kolmogorov from the Moscow mathematical society . ix
Andrei Nikolaevich Kolmogorov . . . . . . . . . . . . . . . . . xii
Papers by A. N. Kolmogorov
1. On the notion of algorithm . . . . . . . . . . . . . 1
2. On the general definition of the quantity of information . 2
3. The theory of transmission of information. . . . . . . 6
4. Amount of information and entropy for continuous distributions 33
5. New metric invariant of transitive dynarnical systems and automorphisms
of Lebesgue spaces . . . . . . . . . . . 57
6. To the definition of algorithms. . . . . . . . . . . . . . . . . .. 62
7. c-entropy and c-capacity of sets in functional spaces. . . . . . . .. 86
8. Various approaches to estimating the complexity of approximate represen-
tation and calculation of functions . . . . . . . . . . . . . . . . 171
9. On tables of random numbers . . . . . . . . . . . . . . . . . . . 176
10. Three approaches to the definition of the notion of arnount of information 184
11. On the realization of networks in three - dimensional space . . . .. 194
12. To the logical foundations of the theory of information and probability
theory . . . . . . . . . . . . . . . . . . . . . . . . . . .. 203
13.The combinatorial foundations of information theory and the prob ability
calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208
Comments and addenda
On works in information theory and some of its applications 219
Information theory . . . . . 222
Algorithmic information theory 226
c-entropy and c:-capacity 231
vii
viii CONTENTS
Tables of random numbers 240
Realization of networks in 3-dimensional space 245
Ergodie theory . . . . . . . . . . 247
Kolmogorov's algorithms or machines .... 251
From A. N. Kolmogorov's recollections
Appendix 1. Report to the mathematical circle about square pavings 261
Appendix 2. On operations on sets. II . . . . . . . . . . . . . . 266
Afterword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
GREETINGS TO A. N. KOLMOGOROV
FROM THE MOSCOW MATHEMATICAL SOCIETY
Dear Andrei Nikolayevieh!
The Managing Board of the Moscow Mathematical Society, members of the Society
heartily congratulate you on the occasion of your 80th anniversary. Your entire sci
entific life has been most closely connected with the Moscow Mathematieal Society.
You first spoke at a session of the Society on October 8, 1922; your talk then, on
"An Example of a Fourier-Lebesgue Series Diverging Almost Everywhere", made
you, a 19-year-old student, renowned throughout the mathematical world. Begin
ning with that moment, you have given important reports on very varied topies at
sessions of the Society no less than 98 times.
The list of your reports strikes by the wide range of topics considered. It suffices
to enumerate only some of the titles of your reports:
"On Fourier series diverging everywhere" (11.16.1926);
"Ab out a general scheme for the theory of probability" (03.20.1928);
"A new interpretation of intuitionist logie" (12.18.1928);
"On the geometrie ideas of Plücker and Klein" (12.11.1932);
"Markov chains and the reversibility of the laws of nature" (01.05.1935);
"Statistieal theory of the crystallization of solidified met als" (02.16.1937);
"Current questions of point set geometry" (12.22.1938);
"Stationary sequences of elements of Hilbert space" (06.04.1939);
"On two types ofaxiomatic methods" (03.18 and 04.01.1941);
"On measures invariant with respect to transformation groups" (04.16.1941);
"Unitary representations of unitary groups" (02.02.1944);
"The mathematieal theory of turbulence" (10.03.1944);
"The structure of complete metric Boolean algebras" (12.09.1947);
"Best approximations of complex functions" (02.24.1948);
"On certain mathematieal problems related to production control" (01.09.1951);
"Two-valued functions of two-valued variables and their applications to switching
circuits" (11.27.1951);
"On the spectra of dynamical systems on the torus" (09.30.1952);
"On almost periodic motion of asolid about a fixed point" (05.26.1953);
"Estimates of the minimal number of elements in e-nets in various functional
classes and their applications to the represention of functions of several variables
as a superposition of functions of a lesser number of variables" (04.27.1954);
ix
x GREETINGS TO A. N. KOLMOGOROV FROM THE MOSCOW SOCIETY
"'On certain asymptotic characteristics of completely bounded metric space"
(06.05.1956);
"Uniform limiting theorems for sums of independent summands" (12.18.1956);
"SmalI denominators in problems of mechanics and analysis" (01.13.1956);
"Wh at is information" (06.04.1961);
"Self-constructing devices" (11.21.1961);
"Computable functions and the foundations of information theory and probabil-
ity theory" (19.11.1963);
"Experiments and mathematical theory in the study of turbulence" (05.18.1965);
"Statistical hydrodynamics of the ocean" (02.24.1970);
"Complexity of definition and complexity of construction for mathematical ob
jects" (11.23.1971);
"On statistical solutions of the Navier-Stokes equations" (01.18.1978).
You have delivered many survey lectures which served as the source of ideas for
further research:
"Measures and probability distributions in function spaces" (11.30.1948);
"Solved and unsolved problems related to the 13th Hilbert problem" (05.17.1960);
"Mathematical methods in the study of Russian verse" (12.27.1960);
"On uniform limiting theorems for sums of independent variables" (03.26.1963);
"The complexity of algorithms and an objective definitions of randomness"
(04.16.1974).
Andrei Nikolayevich, you have been the initiator and lecturer at numerous ses
sions of the Society devoted to discussions on projects of articles prepared for the
Great Soviet Encyclopaedia. In 1938 (on March 10) you participated in the session
devoted to a discussion of your famous article for the encyclopaedia called "Math
ematics" . Several other times you made reports on your articles for the Great
Soviet Encyclopaedia: "Axioms", "Infinitely Small Magnitudes" (18th and 19th of
October, 1949) and others.
Of special interest to the society were your reports on the following topics:
"The development in the USSR of mathematical methods for the study of nature"
(11.10.1937);
"On Ostrogradsky's criticism of the works of Lobachevsky" (09.29.1948);
"The development of mathematics in the Soviet period" (12.23.1949);
"Elements of Mathematics by Nicholas Bourbaki" (05.29.1956);
"On certain traits of the current stage in the development of mathematics"
(04.21.1959);
"From my experience" (04.25.1963);
"On the International Congress of Mathematicians in Amsterdam" (09.28.1954);
"On my scientific trip to France, GOR and Poland" (02.28.1956).
You have always given great importance to discussions with the mathematical
community of questions related to secondary school education.
On November 22 and 28, 1937, a session of the Society was devoted to discussing
a new outline of a textbook in elementary algebra prepared by yourself and P.S.
Alexandrov. You also made the following reports:
"On a project of a syllabus in mathematics for secondary schools" (03.03.1946);
"On a project of a syllabus for secondary schools" (02.17.1948);