BSTJ 30: 1. January 1951: Prediction and Entropy of Printed English. (Shannon, C.E.)

7.8 MB·English

Prediction and Rntropy of Printed English
By €., SHANNON
(atasarige Red ap 5, 05

1. Teese

PX A ones ae! the sottny and sedondeaey of a tengoase base
St elie ie entry it tactical parameter hich menses,
fn a cera see Toe mth iofonnatin 4 peat nthe aver fot
ach ester = text the ana Fhe ge erate in nary
Aigte {or Ty's the most efor: way, tae enteany Histo uvecaze he
lig ig quid pe este ote orga arg, The ela,
tale her and, rasan te amount of ensaint pose on test in
the saage tae tn te sansa strctone elas be hig fe
‘qe ofthe lerter Bt tong tendency of Ht follow Tore (to fll
(OTe we irate thas wien ststcl eflets extending oes net rane
{an eight tlers are caiderel the entepy i oapnly 2 Tos pr ten,
{a vedundaney alan $0 pr vert
Sine tok wat ly be fon fei he sai,
‘whic enor seaicve ad iakes aout of ang age satis, inuences
fxtending aver pases sexy cs TV's mba fy awe an ny
{se putty of Kagishs how well an tens eer fa texte
ft wien the peeling 3 letter ase awn, The rel of seme ex
‘nents in nee wl be given, ead n hectic aaelysis of some ef che
Sroperie of seal pectin, Dy embning te expermens! al thee
resales pee ets ale appr an hooe sal for He ent
find uedundanes. From thi analyse 1c aypeare Baty it onliney Btorary
lish, the long rnge statistic effects fap to 109 leer reuse he
cextopy to someting of the urd of ome it plete, with x cme
sdundancy of soy ly FE. The edumlaey ty eal higher when
Strocivesstening ver paeugraph, claps, et deed. Horses a
‘he lena involve ave teed, the passes in question become mine

saa HET RIS Sante! Be sree Pe
tematic abd uncertain, a they sep more ily on the sre of tet
involve,

2 Eyrany Ceceruaniay reew ie Besta Exes

‘One method of eau he en:rape Bs hea eves of appsoeations
ve Pay Foes thi suecessily tke ware aed rie el he x isin
of th larguage ito cecount acd approach #7 a Lisi. Fy vice be call
The grav igs moses the aroun of comation wenn
lala eateny ver ljcent eters of text, Ps sven by

Fee HE phi lop nd

AE pip saath + 99 ig 08
sve vt Bh 9 1 letters [OYUN
1) on cableny Leas Cllong 3:

th, ie ehe nobly of she Xana ej

the cation! subi of eter ate she block

i is ver by

he yaton 0c he interpneted a tneasutng the average ucts
feorlnal entemy) a Ie nett eter j when the preceding 1-1 laters ane
Known. A5.0Is Ines” Fy inelaces lager and lenge” range attics
fom che enteou, Hen Uy the Uaiting ave ul Py

The pram ertmpies Py fo sel vl of ean Be saute rss
standand tables of leer, gram snd tigama frequencies! Espace al

vs eter alplabet oul Fs ay
“otaken ‘by Cena) se ag 26, 9° LY ite per ete, vo es eta
shecusncies wd i gives Ty

na Spo mms

Tin ig appretination

ive nave weer

(0) = 4b bie poe etn 61

ave he rl
= — Eli) ben

~ = E oi ms 6.9 +E £0) os a

= 0 — 44 = 56 bite por tes
SE wee Brus evotta neemmcar gous, stan 1954

The xa gen by
fra Eh om mith
== Lp 8 twee FE wae atid 8)

=n hr = 83
Tn ths ese trig table! Sot eke it acco ie
ims bridging "wn wonky such as WOW and OW ir TWO WORDS. To
ernpensnte actly forts omen, coveted ceigram prabites 0

eve ain Gant pobabiii f.,80 of he table the Fellow
Fg mag fon

5 225 capt
b= BS. 4 Arta + 3

show if te ait of eter fo the teen etter of nd ad
[NB isthe probity of as om iit eter. Toe the tigams wiehin
fronds Gan average of 25 per wor ate counted acenng the tabs; the
Ten tigcame (ae ofeach tose pes Gord exe tonne approsinately
Tip weun'ngincepencence ofthe lain eter lene word thei
igs inthe pees av we vere, Becguee of “he appeoczations inva
Tete and also Beene the facz deat te eesalng tor in ently
prot wth sump feucncy fe mee serious, the ve of Py 6 Ds
lathe thaw the preiaus numbers.

Since tables Ngan euenie wer vel avaiable for > 3, Hey
et, cowl not be valeste the ame way. Hoeewer, wor feguencie
Fave re tale? aa an be soe! Io obtain 9 farer aporodinavon

fk loging mip af the potsities of won agvid
iy

Tiere Tn
Ieequcnyvank,'The mist frevant Exglsh od ibe” Has a pre
Ff and chit plotted gals. The sext mom request word “et
poabity of Ost ard b poll aguint 2 te Celng hyarnic sees
Fork fr peobahty and ean, eke eure i approsimately a aight line
with slope — 1; thus gis te probaly of the mth mos Srequen wo
tre base, ogy

be o

Zi has poinsedont ta chic type of Form, fr
Ipaination $0 the wort prose in ean lille igus, The

in givesa rather gat

12 gos Mee Tego of Fgh Sc Same aad Cy
formula (@ clely cannot hold indiniely since the total probity 24.
‘oust be unity while 3 1) sintnite af we essome fin the absoce of any

Lote etme) thas she forma fy = 1/n be the wat which the

mk nl
ol TR
accel mal i || any

loa probability unity, end that p= 0 for lager, oe id that the
‘i is Ge noe of eank 8727, The entapy is then

AE ows po — 11.49 bis per wort, o

or 1183/45 = 200 its pr leer since the woerage won ong n English
13 lester Oe righ be tert to enti thie vale with Pas Bt
sell Ihe anise of the Fy curve at ~ 48 il be above this val
Te setsot shat Pyne Ps involves gloup of oat we ive lees rarest
uf onl ison A won 6 seine geoup of lst i ston Tera
Str vey seer eaRTCUT.JoumNAL, Nua 195

atin ndences an ceequety the Negra wa words ee more
tested than show ich Teg wot Tees o hin i that we hove
otaied, in 82 te pr Utero eatin ek comeepont mre sy
to, at, Be

‘A siilar tof eakalsions wa coved oat eluding the space a an
dsitonal lest, giving 27 let apa. Fhe esate of hot 20, and

Procter elaine ete saree bel

“The eatinate of 2. fr Faille to above, was found by avers methods,
Gn of whieh fe the entapelati of tive 2iletie series sbove ant

point. Since the sce bol iz abet svowpotely redundant shen se
fruecee ef one wr ture wore iro, the Yalu of in he 27 eller

cose wil be £3 we of forthe 2 ter abba how Ss cesnaly

dace

The new moth! of cimatingeattopy exlelts the fact that eayone
speaking 8 ingnvesrsencete, xpi, an enornous know of Ue
“Taitce ofthe languages Faiity ws he nds idioms, eke aml
[aan hs bn nang or erect letersi ooo! slings
Ero complete value payee sn convertion, Aa experimental eran
Station el the extent to aiesHnghnh is pretctsble can be given follows
Soi srt pasaege uote nthe erson whois to the prafikng.
Te streaked intestine iat Iles in the pose UE Use cess oeect
hp is 20 seer wl rues to ques the sean ete. I ot he fs ohh
the omee: fst Iter an puocads to bs next ques, This i continued
Thrace tert, Ae the expert prwrses, the abject wiles he
formet tov pt he cate pon for ven prodicng fata eters The
Fest of typed experiment of this type i ven below. Spores were i
‘ilo as on slitions letter, oasing «2 letter alphabet. The tr ie is
the origin tet; the secre fine cotaine x dat fo cach letter wnrecty
fveved, Tn the case of incorcet meses Cae sveet letter is role! he
econ se

(ay oe new a su ne ase 5 a
f) ceeenn ROMS on dneen eel ©
1) Mae LM CF 386 og) HM

f) ice benbeeaed

{a} rive cD AS8oF aR suas veo cer
(ete ene
(fatal of 129 letters, 8 or 69% mere guste cnet. The eros, as
sonido expetes, ocene most frequent at the tering a mbes
Splables wher the linen shout ss mone post Brn hig outs Te
ght be thowent tha Ce second nets (9), whic we wil eal the reed

ten eons muck less information tin the fest Acta, both ins eo

Iain Ue at nor atha i the ste a i yroaiky Test inp
Cpl, te rosner the Girl Ene fom the son. To aor Ms we need
veel tao the ivan en prin le ean, The Sin

{rk met he there, ne eat Wally nha
the sume way when feed wih the sume prublre, Supoose, max, we hv
ny the naval text of (8). We ak the tka to guas the ooesige, tench
te wl oy whather hi guts isooctane be guessing the eee
25 te Fst sin and the presonee ofa das in thereto correspon
toa conect guse The eters ne guesses wrong ae also avilable so tat
eh stage he eas e suppl with preialy the same inforacion the is
fin bad veda

ore coweaen coupon eggs

= sg

te)
ca et er Galena aes Sas
56 rue mtu. este wwemECAL OLENA JANUARY 1954

fire lne sth vigil et td the naman the sro” ie ince he
spss a wie the core letter was obtained

()tenoe 13 Na RP T22eE UK A MerogorouE &
Ghinteausiegangscailiaugmaastitieliiiyg
(yraiiea ey urs Togee 2azs oe
Ghegtotivsautitiaegayanaye sila

i) nasing PRawayreasor rn OrgeR o4y
WMaisssumediigii@isueraiiitiiis @

(xc 192 gymbols he sject guess sight on Li st gust 79 cine,
ce ta econ ce reo shin ues Tne, ta fourth wo AL
Pesss? each ally elt Exes require mee chive guess, Keslts
[tise av i of prodition by a zond abject with onary ieany
Engl Newopeser wting, eiemite work au poetry generally Tel
Smenbat poorer scores.

"Toe sehen! tent ia tie cave eho cousins the same information as the
rg Again ting ie ial Unig we ae i at exc sage to guess
Stmmny tines as the rember ive inthe reduced ext an cove in ie
thay eons To eliminate the Erar-ramaont Lee se 0st aek one

jet foreach 3-gram of tet, tp the mos: probable next
Tester, the secon ttle nest eer, Tis st of dsta can then
sve bth for proton and recovery

“st as bere, he seduced tert cine madre! an encoded weson af
the orginch Te original language, with an alphabet of 29 symbols,
Ry vvr Za spase hasbeen trate ska 2 new language wt the alpck
172,005 27 The traaaatng es ren sl thas the spol 1 mana
xtreme’ hig tequeney ‘The sperbole 2, 34 baow suresively aroler
Sreguencs and shel spi 20,2, »=2 27 ean wry tay. Tes the
Trica ple consierale extent she lane of te ett

il strutuceinvoived, The relwadsacy wie ergialy appeared in com=
ested conatsinlsurvny onupe of tte, bs, hy The arsatng proces,
Tren nade expt fo hye extent te yery oneal probabilities ofthe
rw ayaole Is thi wil sppent Inter, whi enables na t ete
{he earn fru thee experiment,

Sh ener tn detenaine how preity depends on the numbes of
proeosing letters known to the subject, « more involved experinent We
Ferra nt, ne tuzdzed sore of Pagistext were selec trans
From ek cach ten fester in ler Te scboc: as equine to gees
the sent, leer Ly ltter, foreach seme a3 io ee peeling experiment,
"hoes awed eaapls ee tad ie whi he subjet Wl weile
Oto 0+, 1 preeding letters Th al in prditin the sabject made
Sb woes wshel of varia taistial ables, lester, ign aad gea
uosenis aso asTaonY oF rene eon s

‘bles, a tebe ofthe fouonc fini etn in won, ist of the fe
quencies carenon sous an leary. The sarees in tie exporaeh
were fmm afzna tie Prato” ly Dats Matone, These tel,
reer th shar zest ia hil, 100 eters were hove t the subject ate
Sumnatze in Table. The rine owrmapens othe nares of preceding
letsces koowa to the sub’ees ph ones the ror the mb of the gsr
‘The ancy in oalmnn a row 8 the number of tines the subjsct guessed
the gh letter StF goes when ters ere i eae,

the ent 19 in cok 6, 0 2, rac at ith ive esters hon the cor
toc eter waa obtined om the socom guess niuetoea ties oat ute har
tre The Fst te corre thie tle were nok obtained by the exes
rental procmtin online above but were lead dely fon Le
Tinow ldar and diggnn frequencies with ro Kn lertre tho mart
absble symbol isthe space dprclbilty 192), tie nat goese Hh
‘wrong, ould be A ipzebubilty 107}, el. Thase smb ore the
frequencies with hich the righe gus Would oecar a the ra, son
‘was sith best predcsion, Sziayy, a simple ealeslaion fom te dinars
table she eres clin 1 when the slew che table fo best
58 us wun svare mueustou jouer, ASTARY 1951

sdvanoge Siac the frequency tabs are deteminl fron long smpes of
English, ea to colurge se eject to les ating tor chon tebe
Teo seen chat she poem gadsliy irre, apart from nme
eatin) Tactuation, wh Inereasug koowrge nthe pac: os eae
wears a the emer neers of

bye lnger numbers 9 ore Suet.
nigh rank gies

Mh exptiment wats acral out with reverse presction, in which the
subject guessed the as pocating thee alzeady snow. ANDeugh the
task ieuboctvely much maze ciel, tue cones were ony slighly Poors.
‘Thus, its tr I leller amps fr the eam source, the subject
tained the following esas:

Tucientaliy, she ean energy Fr or gneve guage is yal
hat fa te forwae language 8¢ may’ 2 se fom De eon Zo ne
ian (0) et arms ew the sume ae nthe forsanl and eversed se,

A toss

he Gata of Teble Tran lewd to stair ser and le: Bou a the
iygram eateopes Pe. Tones €o do thi, iis asc hr to develop
ne peeval salts coacraing the es. passe pedi ow angus
‘when the prenslng letters arelerowt There wil be or she laaguage set
‘kona yile .yy--> > 1 O)Thiiateprobbiy wien
the I) gray fey ory Sony oceure that the vest ter wl hej. The
best gues forthe pee beer, whet thi 1) wr is krowa t ve 0
fare vile thet letierMaviag the bien eonional pray. The
ecuad guess shout br Cat with the wena highest probity, ee A
chine or geson goes i Ure bess wy” woul gussets inthe onder
oF eras conationalyculsbity. Uwe the procos of relucing text
Shih suchen Heal predictor conse of a mapsing of “yer ato che
uubers from Thr ZT iz sgea a woy tht the most probable ext letler
{eoudiions oe the known poeseing (Y-1) a) is mapped ina 1, ee
"The frequency of sin the reduced txt wil shu he give br

fe oe) ony

‘vhs the sam iste ome (1) rn Sg 2 fs Le sing the
Dre snbich maxinives for that pasteuar (11) gram, Sins, the fre
‘Quenry of 2,2 given By tho sme formula wick jean to be thas
Titer having the areal Fighest vane af ee

‘he che basin of Nogame, «diferent sot of prohiiles for the yrbols
Irmevi0N AND ENTROPY OF PRtIRED eNHIE 0

in the alae eat "4, 8", woul normaly result, Since his
Dredctian fon she has of greater knowledge ofthe past one would ex
post the palit w’ Inv rarnhers 0 be green, and i fact one ets
Domo the ng egies

Eee de sang. an
‘This mens thatthe proheiit of beng right in he Frat S gues when
che oneceding 3 Ietas ase known gear Cia o® equ lat when
only 0-1) are known, forall To prove tis, agin the pdb
ivy feyoee Fe) arunged isa table ih j eening hurionlty ne al
‘he grams vertzally. The tle wil heeire have 27 caine an 27
‘na: The term on the el of (1) these ofthe Slivgest ext in cach
on, eumimed overall he rows. The ight hel member of (1) ele 95
‘fete fou thi abe inh ents ane takes reac eb
beceeay the lages. Ths follve foun the Stet that he vishal
Freleenowld be exelt frm a rie tbe with (WL) grame rater
"ham Nagra a’ very: Rach rm the Vl gram table ete sum
O13 zone of Ue eum abl, since

ee

lig iy 2+ nd

‘The sau of the Slant enries ina rw af he A gram ae wil equ
sun af the 278 select eis fry the corroding 2 roe the

rr able ory the Iter fl ie ems, Far tie ual fe 10)
Tove fora pe ran 8 this rest be oor eee rw of the 8 gram.
Ist Ta thie ave, Pe eller Pie gram fare ret te so the
'S most savbable shoes forthe nxt letter, though the emer. shin
fhe set nny Be afects. However, the ety i (11) hols Foes,
folosthat the oudering as well le alee he fra le othe
N-gram. Th roducod test tained eam a ileal 1 gram praitr i then
ieeosce with that obained Sons 24 eal N-gram prion,

Sine ta pail wus

oe Sa san, a

ste raeaotnisinteaingSunetinns of 8, <1 forall Ny they mos ll a
pages Fite ae 37> 4, Their rt dieses seat terre approach
Fmt e =>, ic, they? approach Lis, 4 Tae may be interpreted
seth relative fequone’ af cavect et tcl, ~~~ gues wilh kl
gs of the See finite} pase history uf Ue Lest,