Table Of Content

A Statistical Mechanical Interpretation of Instantaneous Codes Kohtaro Tadaki Research and Development Initiative, Chuo University 1-13-27 Kasuga, Bunkyo-ku, Tokyo 112-8551, Japan [email protected] Abstract—In this paper we develop a statistical mechanical II. INSTANTANEOUSCODES 9 interpretationofthenoiselesssourcecodingschemebasedonan 0 We start with some notation on instantaneous codes from absolutely optimal instantaneous code. The notions in statistical 0 mechanics such as statistical mechanical entropy, temperature, information theory [9], [1], [3]. 2 and thermal equilibrium are translated into the context of For any set S, #S denotes the number of elements in S. n noiseless source coding. Especially, it is discovered that the We denote the set of all finite binary strings by {0,1}∗. For a temperature1correspondstotheaverage codewordlengthofan any s∈{0,1}∗, |s| is the length of s. We define an alphabet J instantaneous code in this statistical mechanical interpretation to be any nonempty finite set. of noiseless source coding scheme. This correspondence is also 3 Let X be an arbitrary random variable with an alphabet H verifiedbytheinvestigationusingbox-countingdimension.Using 1 the notion of temperature and statistical mechanical arguments, andaprobabilitymassfunctionpX(x)=Pr{X =x},x∈H. some information-theoretic relations can be derived in the man- Then the entropy H(X) of X is defined by ] T ner which appeals to intuition. I H(X)≡−XpX(x)logpX(x), . s x∈H c [ I. INTRODUCTION where the log is to the base 2. We will introduce the notion of a statistical mechanical entropy later. Thus, in order to 1 distinguish H(X) from it, we particularly call H(X) the v We introduce a statistical mechanical interpretation to the ShannonentropyofX.AsubsetS of{0,1}∗iscalledaprefix- 8 noiselesssourcecodingschemebasedonanabsolutelyoptimal 0 instantaneous code. The notions in statistical mechanics such freesetifnostringinS isaprefixofanyotherstringinS.An 7 instantaneouscodeC fortherandomvariableX isaninjective as statistical mechanical entropy, temperature, and thermal 1 mappingfromHto {0,1}∗ suchthatC(H)≡{C(x)|x∈H} equilibrium are translated into the context of noiseless source 1. is a prefix-free set. For each x ∈ H, C(x) is called the coding. 0 codewordcorrespondingtoxand|C(x)|isdenotedbyl(x).A 9 Weidentifyacodedmessagebyaninstantaneouscodewith sequencex ,x ,...,x withx ∈Hiscalledamessage.On 1 2 N i 0 anenergyeigenstateofa quantumsystem treatedin statistical theotherhand,thefinitebinarystringC(x )C(x )···C(x ) : mechanics, and the length of the coded message with the 1 2 N v is called the coded message for a message x ,x ,...,x . 1 2 N i energy of the eigenstate. The discreteness of the length of An instantaneous code play an important role in the X coded message naturally corresponds to statistical mechanics noiseless source coding problem described as follows. Let r based on quantum mechanics and not on classical mechanics. a X ,X ,...,X be independent identically distributed ran- 1 2 N Thisisbecausetheenergyofaquantumsystem takesdiscrete dom variables drawn from the probability mass func- value while an energy takes continuous value in classical tion p (x). The objective of the noiseless source coding X physics in general. Especially, in this statistical mechanical problem is to minimize the length of the binary string interpretation of noiseless source coding, the energy of the C(x )C(x )···C(x ) for a message x ,x ,...,x gener- 1 2 N 1 2 N corresponding quantum system is bounded to the above, and ated by the random variables {X } as N → ∞. For that i therefore the system has negative temperature. We discover purpose, it is sufficient to consider the average codeword that the temperature 1 corresponds to the average codeword length L (C) of an instantaneous code C for the random X length of an instantaneous code in the interpretation. This variable X, which is defined by correspondence is also verified by the investigation based on box-counting dimension. LX(C)≡ XpX(x)l(x) Note that, we do not stick to the mathematical strictness x∈H of the argument in this paper. We respect the statistical me- independently on the value of N. We can then show that chanicalintuition in order to shed light on a hidden statistical L (C) ≥ H(X) for any instantaneous code C for the X mechanical aspect of information theory, and therefore make random variable X. Hence, the Shannon entropy gives the an argument on the same level of mathematical strictness as datacompressionlimitforthenoiselesssourcecodingproblem statistical mechanics. basedoninstantaneouscodes.Thus,itisimportanttoconsider the notion of absolutely optimality of an instantaneous code, in statistical mechanics, the entropy S(E,N) is normally where we say that an instantaneous code C for the random estimated to first order in N and E. Thus the magnitude of variable X is absolutely optimal if L (C)=H(X). We can the indeterminacy δE of the energy does not matter unless it X see that an instantaneous code C is absolutely optimal if and is too small. The temperature T(E,N) of the system S is total only if p (x)=2−l(x) for all x∈H. defined by X Finally, for each xN = (x ,x ,...,x ) ∈ HN, we define 1 ∂S 1 2 N ≡ (E,N). p (xN) as p (x )p (x )···p (x ). T(E,N) ∂E X X 1 X 2 X N III. STATISTICALMECHANICAL INTERPRETATION Thus the temperature is a function of E and N. The average energy ε per one subsystem is given by E/N. In this section, we develop a statistical mechanical inter- Now we give a statistical mechanical interpretation to the pretation of the noiseless source coding by an instantaneous noiseless source coding scheme based on an instantaneous code. In what follows, we assume that an instantaneous code code. Let X be an arbitrary random variable with an al- C for a random variable X is absolutely optimal. phabet H, and let C be an absolutely optimal instantaneous Instatisticalmechanics[7],[11],[8],weconsideraquantum code for the random variable X. Let X ,X ,...,X be system S which consists in a large number of identical 1 2 N total independent identically distributed random variables drawn quantumsubsystems.LetN bea numberof suchsubsystems. from the probability mass function p (x) for a large N, say For example, N ∼ 1022 for 1cm3 of a gas at room tem- X N ∼1022. We relate the noiseless source coding based on C perature. We assume here that each quantum subsystem can to the above statistical mechanics as follows. The sequence be distinguishable from others. Thus, we deal with quantum X ,X ,...,X corresponds to the quantum system S , particles which obey Maxwell-Boltzmann statistics and not 1 2 N total where each X corresponds to the ith quantum subsystem Bose-Einstein statistics or Fermi-Dirac statistics. Under this i S . We relate x ∈ H, or equivalently, C(x) to an energy assumption, we can identify the ith quantum subsystem S i i eigenstate of a subsystem, and we relate l(x)=|C(x)| to an for each i = 1,...,N. In quantum mechanics, any quantum energy E of the energy eigenstate of the subsystem. Then a system is described by a quantum state completely. In statis- n sequence (x ,...,x )∈HN, or equivalently, a finite binary tical mechanics, amongall quantumstates, energyeigenstates 1 N stringC(x )···C(x )correspondstoanenergyeigenstateof are of particular importance. Any energy eigenstate of each 1 N S specified by (n ,...,n ). Thus, l(x )+···+l(x )= subsystem S can be specified by a number n = 1,2,3,..., total 1 N 1 N i |C(x )···C(x )|correspondstotheenergyE +···+E called a quantum number, where the subsystem in the energy 1 N n1 nN of the energy eigenstate of S . eigenstate specified by n has the energy E . Then, any total n We define a subset C(L,N) of {0,1}∗ as the set of all energy eigenstate of the system S can be specified by an total coded messages C(x )···C(x ) whose length lies between N-tuple (n ,n ,...,n ) of quantum numbers. If the state 1 N 1 2 N L and L + δL. Then Ω(L,N) is defined as #C(L,N). of the system S is the energy eigenstate specified by total Therefore Ω(L,N) is the total number of coded messages (n ,n ,...,n ), then the state of each subsystem S is the 1 2 N i whose length lies between L and L+δL. We can see that energy eigenstate specified by n and the system S has i total if C(x )···C(x ) ∈ C(L,N), then 2−L ≤ p(xN) ≤ the energy E +E +···+E . Then, the fundamental 1 N n1 n2 nN 2−(L+δL). This is because C is an absolutely optimal instan- postulate of statistical mechanics is stated as follows. taneous code. Thus all coded messages C(x )···C(x ) ∈ 1 N Fundamental Postulate: If the energy of the system Stotal C(L,N) occur with the probability 2−L. Note here that we is known to have a constant value in the range between E care nothing about the magnitude of δL, as in the case of and E+δE, where δE is the indeterminacy in measurement statisticalmechanics.Thus,giventhatthelengthofcodedmes- of the energy of the system Stotal, then the system Stotal is sageisL, allcodedmessagesoccurwith thesame probability equally likely to be in any energy eigenstate specified by 1/Ω(L,N). We introduce a micro-canonicalensemble on the (n1,n2,...,nN) such that E ≤ En1 +En2 +···+EnN ≤ noiseless source coding in this manner. Thus we can develop E+δE. a certain sort of statistical mechanics on the noiseless source coding scheme. Let Ω(E,N) be the total number of energy eigenstates of The statisticalmechanicalentropy S(L,N)of the instanta- S specifiedby(n ,n ,...,n )suchthatE ≤E +E + total 1 2 N n1 n2 neous code C is defined by ···+E ≤ E +δE. The above postulate states that any nN energy eigenstate of Stotal whose energy lies between E and S(L,N)≡logΩ(L,N). E+δE occurswith the probability1/Ω(E,N).This uniform distributionofenergyeigenstateswhoseenergyliesbetweenE The temperature T(L,N) of C is then defined by andE+δE iscalledamicrocanonicalensemble.Instatistical 1 ∂S ≡ (L,N). mechanics, the entropy S(E,N) of the system Stotal is then T(L,N) ∂L defined by Thus the temperature is a function of L and N. The average S(E,N)≡klnΩ(E,N), lengthλofcodedmessageperonecodewordisgivenbyL/N. where k is a positive constant, called the Boltzmann Con- The average length λ corresponds to the average energy ε in stant, and the ln denotes the natural logarithm. Note that, the statistical mechanics above. IV. PROPERTIES OF STATISTICALMECHANICAL ENTROPY the average codeword length LX(C), which is equal to the average length λ of coded message per one codeword at the In statistical mechanics, it is important to know the values temperature 1. oftheenergyE ofsubsystemS forallquantumnumbersn, n i since the values determine the entropy S(E,N) of the quan- V. THERMAL EQUILIBRIUM BETWEENTWO tum system S . Corresponding to this fact, the knowledge total INSTANTANEOUSCODES of l(x) for all x ∈ H is important to calculate S(L,N). We Let XI be an arbitrary random variable with an alphabet investigatesomepropertiesofS(L,N)andT(L,N)basedon HI, and let CI be an absolutely optimal instantaneous code l(x) in the following. fortherandomvariableXI.LetXI,XI,...,XI beindepen- Asiswellknowninstatisticalmechanics,iftheenergyofa 1 2 NI dent identically distributed random variables drawn from the quantumsystemS isboundedtotheabove,thenthesystem total probabilitymass functionp (x) for a largeNI. On the other can have negativetemperature.The same situation happensin XI hand,letXII beanarbitraryrandomvariablewithanalphabet our statistical mechanics developed on an instantaneous code HII, and let CII be an absolutely optimal instantaneous code C, since there are only finite codewords of C. We define for the random variable XII. Let XII,XII,...,XII be inde- lmin and lmax as min{l(x) | x ∈ H} and max{l(x) | x ∈ 1 2 NII pendent identically distributed random variables drawn from H}, respectively. Given N, the statistical mechanical entropy the probability mass function p (x) for a large NII. S(L,N)is a unimodalfunctionof L and takes nonzerovalue XII Consider the following problem: Find the most probable only between Nl and Nl . Let L be the value L which min max 0 values LI and LII, given that the sum LI+LII of the length maximizes S(L,N). If L < L then T(L,N) > 0. On the 0 LI of coded message by CI for the random variables {XI} other hand, if L > L then T(L,N) < 0. The temperature i 0 and the length LII of coded message by CII for the random T(L,N) takes ±∞ at L=L . 0 variables {XII} is equal to L. According to the method of Boltzmann and Planck (see j In order to solve this problem, the statistical mechanical e.g. [11]), we can show that notionof“thermalequilibrium”canbeused.Wefirstnotethat S(L,N)=NH(G(C,T(L,N))), (1) aparticularcodedmessagebyCI oflengthL andaparticular I coded message by CII of length L occur with probability II whereG(C,T)istherandomvariablewiththealphabetHand 2−LI2−LII = 2−L, since the instantaneous codes CI and CII theprobabilitymassfunctionp (x)=Pr{G(C,T)=x} G(C,T) are absolutely optimal. Thus, any particular pair of coded defined by messagesbyCIandCIIoccurswithanequalprobability,given 2−l(x)/T that the total length of coded messages for {XI} and {XII} i j pG(C,T)(x)≡ 2−l(a)/T. is L. Therefore, the most probable allocation LI∗ and LII∗ of Pa∈H L = L +L maximizes the product Ω(L,N)Ω (L ,N ). I II I I I II II II ThetemperatureT(L,N)isimplicitlydeterminedthroughthe We see that this condition is equivalent to the equality: equation L T(L∗,N)=T (L∗,N ), N = Xl(x)pG(C,T(L,N))(x) (2) I I I II II II x∈H where the functions TI and TII are the temperature of CI and as a function of L and N. These properties of S(L,N) and CII, respectively. This equality corresponds to the condition on the thermal equilibrium between two systems, given a T(L,N) are derived only based on a combinatorial aspect of total energy, in statistical mechanics. Using (2), the value of S(L,N). Now,letustakeintoaccounttheprobabilisticissuegivenby TI(L∗I,NI)=TII(L∗II,NII) is obtained by solving the equation therandomvariablesX ,X ,...,X .Sincetheinstantaneous on T: 1 2 N code C is absolutely optimal, a particular coded message of NI length L occurs with probability 2−L. Thus the probability L X (cid:12)(cid:12)CI(x)(cid:12)(cid:12)pG(CI,T)(x)+ that some coded message of length L occurs is given by x∈HI 2se−ttLinΩg(Lth,eNr)e.sHuletntcoe,0b,ywdeiffcearnendtieatteinrmgi2n−eLtΩhe(Lm,Nos)topnroLbaabnlde NLII X (cid:12)(cid:12)CII(x)(cid:12)(cid:12)pG(CII,T)(x) x∈HII length L∗ of coded message, given N. Thus we have the =1. relation Then, again by (2), the most probable values L∗ and L∗ are ∂ I II {−L+S(L,N)}(cid:12) =0, determined. ∂L (cid:12)(cid:12)(L,N)=(L∗,N) which is satisfied by L∗. It follows that T(L∗,N)=1, Thus, VI. DIMENSION OF CODED MESSAGES the temperature 1 corresponds to the most probable length The notion of dimension plays an important role in fractal L∗.On theotherhand,p (x)=2−l(x) atT(L∗,N)=1, geometry [6]. In this section, we investigate our statistical G(C,1) and therefore, by (2), we have L∗/N = H(X) = L (C). mechanicalinterpretation of the noiseless source coding from X SinceC isabsolutelyoptimal,thisresultisconsistentwiththe the point of view of dimension. Let F be a bounded subset law of large numbers.Thus, the temperature1 correspondsto of R, and let N (F) be the number of 2−n-mesh cubes that n intersectF,where2−n-meshcubeisasubsetofRintheform We can show that nlogn< l(x) unless all codewords Px∈H of[m2−n,(m+1)2−n]forsomeintegerm.Thebox-counting have the same length, and obviously logd /l < 1 and min min dimension dim F of F is then defined by logd /l < 1 except for such a trivial case. Thus, B max max in general, the dimension dim F(T) is maximized at the logN (F) B dim F ≡ lim n . temperature T = 1. This can be checked using (3) based on B n→∞ n thedifferentiationofdim F(T).Thatis,wecanshowthat,if B Let {0,1}∞ = {b1b2b3··· | bi = 0,1 for all i = 1,2,3,...} allcodewordsdonothavethesamelength,thenthefollowing be the set of all infinite binary strings. In [10] we investigate hold: the dimension of sets of coded messages of infinite length, d where the number of distinct codewords is finite or infinite. (i) dim F(T)(cid:12) =0 if and only if T =1, In a similar manner,we investigatethe set of coded messages dT B (cid:12)(cid:12)T=T0 0 of infinite length by an absolutely optimal instantaneouscode d2 (ii) dim F(T)(cid:12) <0. C. dT2 B (cid:12)(cid:12)T=1 By(2),theratioL/N isuniquelydeterminedbytemperature Note that all coded messages C(x )C(x )··· of infinite 1 2 T. Thus, by letting L,N → ∞ while keeping the ratio length form the set {0,1}∞ and therefore the interval [0,1], L/N constant, we can regard the set C(L,N) as a subset of since C is an absolutely optimal instantaneous code. Thus, {0,1}∞. This kind of limit is called the thermodynamic limit since dim F(1) is equal to dim [0,1], the set F(1) is as B B in statistical mechanics. Taking the thermodynamic limit, we rich as the set [0,1] in a certain sense. This can be explained denoteC(L,N)byF(T),whereT isrelatedtothelimitvalue as follows. Since L/N = L (C) at the temperature T = 1, of L/N through (2). Although F(T) is a subset of {0,1}∞, X asseen inSectionIV, bythe lawof largenumbers,thelength we can regard F(T) as a subset of [0,1] by identifying of coded message for a message of length N is likely to α∈{0,1}∞withtherealnumber0.α.Inthismanner,wecan equalNL (C),forasufficientlylargeN.ThusF(1)contains X consider the box-counting dimension dim F(T) of F(T). B almostallfinitebinarystringoflengthL.Inotherwords,F(1) We investigate the dependency of dim F(T) on tempera- B consists in coded messages for all messages which form the ture T with −∞≤T ≤∞. First it can be shown that typical set in a sense. logΩ(L,N) dim F(T) = lim VII. CONCLUSION B L,N→∞ L In this paper we have developed a statistical mechanical S(L,N) = lim , interpretation of the noiseless source coding scheme based L,N→∞ L on an absolutely optimal instantaneous code. The notions in where the limits are taken while satisfying (2) for each T. statistical mechanics such as statistical mechanical entropy, Thus the statistical mechanicalentropy S(L,N) and the box- temperature, and thermal equilibrium are translated into the counting dimension dim F(T) of F(T) are closely related. context of information theory. Especially, it is discovered B By (1) and (2), we can obtain, as an explicit formula of T, that the temperature 1 corresponds to the average codeword length L (C) in this statistical mechanical interpretation of 1 1 X dimBF(T)= + logX2−l(x)/T, (3) informationtheory.Thiscorrespondenceisalsoverifiedbythe T λ(T) x∈H investigation using box-counting dimension. The argument is where λ(T) is defined by not necessarily mathematically rigorous. However, using the notion of temperature and statistical mechanical arguments, λ(T)≡ Xl(x)pG(C,T)(x). several information-theoretic relations can be derived in the x∈H manner which appeals to intuition. A statistical mechanical interpretation of the general case Wedefinethe“degeneracyfactors”d andd ofthelowest min max where the underlying instantaneous code is not necessarily andhighest“energies”byd ≡#{x∈H|l(x)=l }and min min absolutely optimal is reported in another work. d ≡ #{x ∈ H | l(x) = l }, respectively. Note here that max max since C is assumed to be absolutely optimal, Px∈H2−l(x) = ACKNOWLEDGMENTS 1 and therefore d can be shown to be an even number. In max Theauthorisgratefultothe21stCenturyCOEProgramand theincreasingorderoftheratioL/N (i.e.λ(T)),wesee from the Research and Development Initiative of Chuo University (3) that for the financial supports. logd min lim dimBF(T) = , REFERENCES T→+0 lmin dim F(1) = 1, [1] R. B. Ash, Information Theory. Dover Publications, Inc., New York, B 1990. nlogn [2] C.S.CaludeandM.A.Stay,“Naturalhaltingprobabilities, partialran- lim dim F(T) = , T→±∞ B Px∈Hl(x) 1d7o3m9n,e2s0s,0a6n.dzetafunctions,”Inform.andComput.,vol.204,pp.1718– logd max [3] T.M. Cover and J. A.Thomas, Elements ofInformation Theory. John lim dim F(T) = . T→−0 B lmax Wiley&Sons,Inc.,NewYork,1991. [4] A.DemboandO.Zeitouni, LargeDeviations Techniques andApplica- tions,2nded.Springer, NewYork,1998. [5] P.A.M.Dirac, ThePrinciples ofQuantumMechanics, 4thed.Oxford University Press,London,1958. [6] K. Falconer, Fractal Geometry, Mathematical Foundations and Appli- cations. JohnWiley&Sons,Inc.,Chichester, 1990. [7] F.Reif,FundamentalsofStatisticalandThermalPhysics.McGraw-Hill, Inc.,Singapore, 1965. [8] D. Ruelle, Statistical Mechanics, Rigorous Results, 3rd ed. Imperial CollegePressandWorldScientificPublishingCo.Pte.Ltd.,Singapore, 1999. [9] C. E. Shannon, “A mathematical theory of communication,” Bell Syst. Tech.J.,vol.27,pt.I,pp.379–423,1948;pt.II,pp.623–656, 1948. [10] K. Tadaki, “A generalization of Chaitin’s halting probability Ω and halting self-similar sets,” Hokkaido Math. J., vol. 31, pp. 219–253, 2002. Electronic Version Available: http://arxiv.org/abs/nlin/0212001 [11] M. Toda, R. Kubo, and N. Saitoˆ, Statistical Physics I. Equilibrium Statistical Mechanics, 2nded.Springer, Berlin, 1992.