Abstract Methods in Information Theory PDF
Preview Abstract Methods in Information Theory
AABBSSTTRRAACCTT MMEETTHHOODDSS IINN IINNFFOORRMMAATTIIOONN TTHHEEOORRYY SERIES ON MULTIVARIATE ANALYSIS Editor: M M Rao Published Vol. 1: Martingales and Stochastic Analysis J. Yeh Vol. 2: Multidimensional Second Order Stochastic Processes Y. Kakihara Vol. 3: Mathematical Methods in Sample Surveys H. G. Tucker Vol. 4: Abstract Methods in Information Theory Y. Kakihara Forthcoming Convolution Structures and Stochastic Processes R. Lasser Topics in Circular Statistics S. R. Jammalamadaka and A. SenGupta AABBSSTTRRAACCTT MMEETTHHOODDSS IINN IINNFFOORRMMAATTIIOONN TTHHEEOORRYY Yuichiro Kakihara Department oj Mathematics University oj California, Riverside USA World Scientific SSiinngaaappoorree *• NNeeww JJeersrseeyy ,L ondon • Hong Kong Published by World Scientific Publishing Co. Pte. Ltd. P O Box 128, Farrer Road, Singapore 912805 USA office: Suite IB, 1060 Main Street, River Edge, NJ 07661 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE Library of Congress Cataloging-in-Publication Data Kakihara, Yuichiro. Abstract methods in information theory / Yuichiro Kakihara. p. cm. — (Series on multivariate analysis ; v. 4) Includes bibliographical references. ISBN 9810237111 (alk. paper) 1. Information theory. 2. Functional analysis. I. Title. II. Series. Q360.K35 1999 003'.54~dc21 99-31711 CIP British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. Copyright © 1999 by World Scientific Publishing Co. Pte. Ltd. All rights reserved. This book, or parts thereof, may not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system now known or to be invented, without written permission from the Publisher. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. Printed in Singapore by Uto-Print. Dedicated to Professor Hisaharu Umegaki TThhiiss ppaaggee iiss iinntteennttiioonnaallllyy lleefftt bbllaannkk. PREFACE Half a century has passed since C. E. Shannan published his epoch-making paper entitled "A mathematical theory of communication" in 1948. Thereafter the so- called "information theory" began to grow and has now established a firm and broad field of study. Viewing from a mathematical angle, information theory might be thought of having the following four parts: (1) the mathematical structure of information sources, (2) the theory of entropy as amounts of information, (3) the theory of information channels, and (4) the theory of coding. Probabilistic and algebraic methods have mainly been used to develop information theory. Since the early stage of the expansion of information theory, however, measure theoretic and functional analysis methods have also been applied and are providing a powerful tool to obtain rigorous results in this theory. The purpose of this book is to present the first three parts of information theory, mentioned above, in the environment of functional analysis, in addition to probability theory. Here are a couple of examples in each of which functional analysis played a cru cial role obtaining important results in information theory. The coincidence of the ergodic capacity C and the stationary capacity C for a certain channel was one e s of the most important problems in the late 1950s. L. Breiman (1960) showed that for a finite memory channel the equality C = C holds and, moreover, C is at e s e tained by some ergodic input source (= measure) invoking Krein-Milman's theorem to the weak* compact convex set P(X) of all stationary input sources. Another S such example appeared in a characterization of ergodic channels. In the late 1960s H. Umegaki and Y. Nakamura independently proved that a stationary channel is er godic if and only if it is an extreme point of the convex set of all stationary channels. Umegaki observed a one-to-one correspondence between the set of channels and a set of certain averaging operators from the set of bounded measurable functions on the compound space to the set of those functions on the input. Then a channel is identified with an operator, called a channel operator, and hence we can make a full use of functional analysis in studying channels. In this book, readers will find how functional analysis helps to describe information theory, especially the mathematical structure of information souces and channels, in an effective way. Here is a brief summary of this book. In Chapter I, entropy is considered as the amount of information. Shannon's entropy for finite schema is defined and its basic properties are examined together with its axioms. After collecting fundamental prop- vii viii Preface erties of conditional expectation and probability, Kolmogorov-Sinai's entropy is then obtained for a measure preserving transformation. Some fundamental properties of the Kolmogorov-Sinai entropy are presented along with the Kolmogorov-Sinai theo rem. Algebraic models are introduced to describe probability measures and measure preserving transformations. Some conjugacy problems are studied using algebraic models. When we fix a measurable transformation and a finite partition, we can consider Kolmogorov-Sinai's entropy as a functional on the set of invariant (with respect to the transformation) probability measures, called an entropy functional. This functional is extended to be the one defined on the set of all complex valued invariant measures, and its integral representation is obtained. Relative entropy and Kullback-Leibler information are also studied in connection with sufficiency which is one of the most important notions in statistics, and with hypothesis testing. In Chapter II, information sources are considered. Using an alphabet message space as a model, we describe information sources on a compact Hausdorff space. Mean and Pointwise Ergodic Theorems are stated and proved. Ergodicity is one of the important concepts and its characterization is presented in detail. Also strong and weak mixing properties are examined in some detail. Among the nonstation- ary sources, AMS (= asymptotically mean stationary) sources are of interest and the structure of this class is studied. Shannon-McMillan-Breiman Theorem is then formulated for a stationary and an AMS source, which is regarded as the ergodic theorem in information theory. Ergodic decomposition of a stationary source is established and is applied to obtain another type of integral representation of an entropy functional. Chapter III, the main part of this book, is devoted to the information channels. After defining channels, a one-to-one correspondence between a set of channels and a set of certain averaging operators is established, as mentioned before. Strongly and weakly mixing channels are defined as a generalization of finite dependent channels and their basic properties are obtained. Ergodicity of stationary channels is discussed and various necessary and sufficient conditions for it are given. For AMS channels, absolute continuity plays a special role in characterizing ergodicity. Capacity and transmission rate are defined for stationary channels. Coincidence of ergodic and stationary capacities is proved under certain conditions. Finally, Shannon's coding theorems are stated and proved. Special topics on channels are considered in Chapter IV. When a channel has a noise source, some properties of such a channel are studied. If we regard a channel to be a vector (or measure) valued function on the input space, then its measurabilities are clarified. Some approximation problems of channels are treated. When the output space is a (locally) compact abelian group, a harmonic analysis method can be applied to channel theory. Some aspects of this viewpoint are presented in detail. Finally, a noncommutative channel theory is introduced. We use a C*-algebra approach to formulate channel operators as well as other aspects of noncommutative Preface IX extension. Another purpose of this book is to present contributions of Professor Hisaharu Umegaki and his school on information theory. His selected papers is published under the title of "Operator Algebras and Mathematical Information Theory," Kaigai, Tokyo in 1985. As one of his students, the author is pleased to have a chance to write this monograph. In the text III.4.5 denotes the fifth item in Section 4 of Chapter III. In a given chapter, only the section and item number are used, and in a given section, only the item number is used. The author is grateful to Professor M. M. Rao at University of California, River side (UCR) for reading the manuscript and for the valuable suggestions. UCR has provided the author with a very fine environment, where he could prepare this mono graph. He is also grateful for the hospitality of UCR. Yuichiro Kakihara Riverside, California April, 1999 TT hh ii ss Tppaahggisee piissa giinnett eeisnn ittniiootnnenaalltlliyyon llaeefflltty bb llleaafnntkk b.lank.
The list of books you might like
