Information theory and statistics: a tutorial

122 Pages·2004·1.03 MB·english

by  Imre Csisz´ar

Information Theory
and Statistics: A
Tutorial

Information Theory and
Statistics: A Tutorial
Imre Csisz´ar
R´enyiInstituteofMathematics,
HungarianAcademyofSciences
POB127,H-1364Budapest,
Hungary
[email protected]
Paul C. Shields
ProfessorEmeritusofMathematics,
UniversityofToledo,
Ohio,
USA
[email protected]
Boston – Delft
Foundations and TrendsR in
(cid:1)
Communications and Information Theory
Published, sold and distributed by:
now Publishers Inc.
PO Box 1024
Hanover, MA 02339
USA
Tel. [phone]
www.nowpublishers.com
[email protected]
Outside North America:
now Publishers Inc.
PO Box 179
2600 AD Delft
The Netherlands
Tel. [phone]
A Cataloging-in-Publication record is available from theLibrary of Congress
Printed on acid-free paper
ISBN: 1-933019-05-0; ISSNs: Paper version 1567-2190; Electronic
version 1567-2328
c 2004 I. Csisz´ar and P.C. Shields
(cid:1)
All rights reserved. No part of this publication may be reproduced,
stored in a retrieval system, or transmitted in any form or by any
means,mechanical,photocopying,recordingorotherwise,withoutprior
written permission of the publishers.
now Publishers Inc. has an exclusive license to publish this mate-
rial worldwide. Permission to use this content must be obtained from
the copyright license holder. Please apply to now Publishers, PO Box
179,2600ADDelft,TheNetherlands,www.nowpublishers.com;e-mail:
[email protected]
Contents
1 Preliminaries 3
2 Large deviations, hypothesis testing 11
2.1 Large deviations via types 11
2.2 Hypothesis testing 16
3 I-projections 23
4 f-Divergence and contingency tables 31
5 Iterative algorithms 43
5.1 Iterative scaling 43
5.2 Alternating divergence minimization 47
5.3 The EM algorithm 55
6 Universal coding 59
v
vi Contents
6.1 Redundancy 60
6.2 Universal codes for certain classes of processes 65
7 Redundancy bounds 77
7.1 I-radius and channel capacity 78
7.2 Optimality results 85
8 Redundancy and the MDL principle 91
8.1 Codes with sublinear redundancy growth 92
8.2 The minimum description length principle 98
A Summary of process concepts 107
References 113
Contents 1
Preface
This tutorial is concerned with applications of information theory con-
cepts in statistics. Itoriginated as lectures given by ImreCsisz´ar at the
University of Maryland in 1991 with later additions and corrections by
Csisz´ar and Paul Shields.
Attention isrestrictedtofinitealphabetmodels.Thisexcludessome
celebrated applications such as the information theoretic proof of the
dichotomy theorem for Gaussian measures, or of Sanov’s theorem in
a general setting, but considerably simplifies the mathematics and ad-
mits combinatorial techniques. Even within the finite alphabet setting,
noefforts were made at completeness. Rather, some typical topics were
selected,accordingtotheauthors’research interests.Inallofthem,the
informationmeasureknownasinformationdivergence(I-divergence)or
Kullback–Leibler distance or relative entropy plays a basic role. Sev-
eral of these topics involve “information geometry”, that is, results of
a geometric flavor with I-divergence in the role of squared Euclidean
distance.
In Chapter 2, a combinatorial technique of major importance in
information theory is applied to large deviation and hypothesis test-
ing problems. The concept of I-projections is addressed in Chapters
3 and 4, with applications to maximum likelihood estimation in ex-
ponential families and, in particular, to the analysis of contingency
tables.Iterative algorithms basedoninformationgeometry, tocompute
I-projectionsandmaximumlikelihoodestimates,areanalyzedinChap-
ter 5. The statistical principle of minimum description length (MDL)
is motivated by ideas in the theory of universal coding, the theoret-
ical background for efficient data compression. Chapters 6 and 7 are
devoted to the latter. Here, again, a major role is played by concepts
with a geometric flavor that we call I-radius and I-centroid. Finally,
the MDL principle is addressed in Chapter 8, based on the universal
coding results.
Reading this tutorial requires no prerequisites beyond basic proba-
bility theory. Measure theory is needed only in the last three Chapters,
dealing with processes. Even there, no deeper tools than the martin-
gale convergence theorem areused.Tokeep thistutorial self-contained,
2 Contents
the information theoretic prerequisites are summarized in Chapter 1,
and the statistical concepts are explained where they are first used.
Still, while prior exposure to information theory and/or statistics is
not indispensable, it is certainly useful. Very little suffices, however,
say Chapters 2 and 5 of the Cover and Thomas book [7] or Sections
1.1,1.3,1.4oftheCsisz´ar-Ko¨rnerbook[14],forinformationtheory,and
Chapters 1–4 and Sections 9.1–9.3 of the book by Cox and Hinckley
[8], for statistical theory.
1
Preliminaries
ThesymbolA= a ,a ,...,a denotes afinitesetof cardinality
1 2 |A|
{ }
A; xn denotes the sequence x ,x ,...,x , where each x A; An
| | m m m+1 n i ∈
denotes the set of all xn; A∞ denotes the set of all infinite sequences
1
x = x∞, with x A,i 1; and A∗ denotes the set of all finite
1 i ∈ ≥
sequences drawn from A. The set A∗ also includes the empty string Λ.
The concatenation of u A∗ and v A∗ A∞ is denoted by uv. A
∈ ∈ ∪
finite sequence u is a prefix of a finite or infinite sequence w, and we
write u w, if w = uv, for some v.
≺
The entropy H(P) of a probability distribution P = P(a),a A
{ ∈ }
is defined by the formula
H(P) = P(a)logP(a).
−
a∈A
(cid:1)
Here, as elsewhere in this tutorial, base two logarithms are used and
0log0 is defined to be 0. Random variable notation is often used in
this context. For a random variable X with values in a finite set, H(X)
denotes the entropy of the distribution of X. If Y is another random
variable, not necessarily discrete, the conditional entropy H(X Y) is
|
defined as the average, with respect to the distribution of Y, of the
entropy of the conditional distribution of X, given Y = y. The mutual
3