Table Of ContentUnderstanding Complex Systems
Bernhard C. Geiger
Gernot Kubin
Information Loss
in Deterministic
Signal Processing
Systems
Springer Complexity
Springer Complexity is an interdisciplinary program publishing the best research and
academic-level teaching on both fundamental and applied aspects of complex systems—
cuttingacrossalltraditionaldisciplinesofthenaturalandlifesciences,engineering,economics,
medicine,neuroscience,socialandcomputerscience.
Complex Systems are systems that comprise many interacting parts with the ability to
generate a new quality of macroscopic collective behavior the manifestations of which are
the spontaneous formation of distinctive temporal, spatial or functional structures. Models
of such systems can be successfully mapped onto quite diverse “real-life” situations like
theclimate,thecoherentemissionoflightfromlasers,chemicalreaction-diffusionsystems,
biological cellular networks, the dynamics of stock markets andof the Internet, earthquake
statistics and prediction, freeway traffic, the human brain, or the formation of opinions in
social systems, toname just some ofthe popular applications.
Although their scope and methodologies overlap somewhat, one can distinguish the
following main concepts and tools: self-organization, nonlinear dynamics, synergetics,
turbulence,dynamicalsystems,catastrophes,instabilities,stochasticprocesses,chaos,graphs
and networks, cellular automata, adaptive systems, genetic algorithms and computational
intelligence.
ThethreemajorbookpublicationplatformsoftheSpringerComplexityprogramarethe
monograph series “Understanding Complex Systems” focusing on the various applications
of complexity, the “Springer Series in Synergetics”, which is devoted to the quantitative
theoreticalandmethodologicalfoundations,andthe“SpringerBriefsinComplexity”which
are concise and topical working reports, case studies, surveys, essays and lecture notes of
relevance to the field. In addition to the books in these two core series, the program also
incorporates individual titles ranging from textbooks tomajor reference works.
Editorial and Programme Advisory Board
HenryAbarbanel,InstituteforNonlinearScience,UniversityofCalifornia,SanDiego,USA
DanBraha,NewEnglandComplexSystemsInstituteandUniversityofMassachusettsDartmouth,USA
Péter Érdi, Center for Complex Systems Studies, Kalamazoo College, USA and Hungarian Academy of
Sciences,Budapest,Hungary
KarlFriston,InstituteofCognitiveNeuroscience,UniversityCollegeLondon,London,UK
HermannHaken,CenterofSynergetics,UniversityofStuttgart,Stuttgart,Germany
ViktorJirsa,CentreNationaldelaRechercheScientifique(CNRS),UniversitédelaMéditerranée,Marseille,
France
JanuszKacprzyk,SystemResearch,PolishAcademyofSciences,Warsaw,Poland
KunihikoKaneko,ResearchCenterforComplexSystemsBiology,TheUniversityofTokyo,Tokyo,Japan
ScottKelso,CenterforComplexSystemsandBrainSciences,FloridaAtlanticUniversity,BocaRaton,USA
Markus Kirkilionis, Mathematics Institute and Centre for Complex Systems, University of Warwick,
Coventry,UK
JürgenKurths,NonlinearDynamicsGroup,UniversityofPotsdam,Potsdam,Germany
RonaldoMenezes,FloridaInstituteofTechnology,ComputerScienceDepartment,150W.UniversityBlvd,
Melbourne,FL32901,USA
AndrzejNowak,DepartmentofPsychology,WarsawUniversity,Poland
HassanQudrat-Ullah,SchoolofAdministrativeStudies,YorkUniversity,Toronto,ON,Canada
LindaReichl,CenterforComplexQuantumSystems,UniversityofTexas,Austin,USA
PeterSchuster,TheoreticalChemistryandStructuralBiology,UniversityofVienna,Vienna,Austria
FrankSchweitzer,SystemDesign,ETHZürich,Zürich,Switzerland
DidierSornette,EntrepreneurialRisk,ETHZürich,Zürich,Switzerland
StefanThurner,SectionforScienceofComplexSystems,MedicalUniversityofVienna,Vienna,Austria
Understanding Complex Systems
Founding Editor: S. Kelso
Future scientific and technological developments in many fields will necessarily
depend uponcomingtogripswithcomplexsystems.Such systems arecomplex in
both their composition–typically many different kinds of components interacting
simultaneouslyandnonlinearlywitheachotherandtheirenvironmentsonmultiple
levels–and in the rich diversity of behavior of which they are capable.
TheSpringerSeriesinUnderstandingComplexSystemsseries(UCS)promotes
new strategies and paradigms for understanding and realizing applications of
complex systems research in a wide variety of fields and endeavors. UCS is
explicitlytransdisciplinary.Ithasthreemaingoals:First,toelaboratetheconcepts,
methodsandtoolsofcomplexsystemsatalllevelsofdescriptionandinallscientific
fields,especiallynewlyemergingareaswithinthelife,social,behavioral,economic,
neuro-andcognitivesciences(andderivativesthereof);second,toencouragenovel
applicationsoftheseideasinvariousfieldsofengineeringandcomputationsuchas
robotics, nano-technology,and informatics; third, to provide a single forum within
which commonalities and differences in the workings of complex systems may be
discerned,henceleadingtodeeperinsightandunderstanding.
UCS will publish monographs, lecture notes, and selected edited contributions
aimed at communicating new findings to a large multidisciplinary audience.
More information about this series at http://www.springer.com/series/5394
Bernhard C. Geiger Gernot Kubin
(cid:129)
Information Loss
in Deterministic Signal
Processing Systems
123
Bernhard C.Geiger Gernot Kubin
Institute for Communications Engineering Signal ProcessingandSpeech
Technical University of Munich Communication Lab
Munich Graz University of Technology
Germany Graz
Austria
ISSN 1860-0832 ISSN 1860-0840 (electronic)
Understanding ComplexSystems
ISBN978-3-319-59532-0 ISBN978-3-319-59533-7 (eBook)
DOI 10.1007/978-3-319-59533-7
LibraryofCongressControlNumber:2017943204
©SpringerInternationalPublishingAG2018
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpart
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
orinformationstorageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilar
methodologynowknownorhereafterdeveloped.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publicationdoesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfrom
therelevantprotectivelawsandregulationsandthereforefreeforgeneraluse.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authorsortheeditorsgiveawarranty,expressorimplied,withrespecttothematerialcontainedhereinor
for any errors or omissions that may have been made. The publisher remains neutral with regard to
jurisdictionalclaimsinpublishedmapsandinstitutionalaffiliations.
Printedonacid-freepaper
ThisSpringerimprintispublishedbySpringerNature
TheregisteredcompanyisSpringerInternationalPublishingAG
Theregisteredcompanyaddressis:Gewerbestrasse11,6330Cham,Switzerland
to our families
Acknowledgements
Wethankallourco-authorswehadthepleasuretoworkwithduringthelastyears.
In particular, we thank Christian Feldbauer, for developing first results together
withus,andTobiasKoch,UniversidadCarlosIIIdeMadrid,forhelpingusmaking
some of our results mathematically more rigorous. We are indebted to Rana Ali
Amjad,ClemensBlöchl,OnurGünlü,KairenLiu,AndreiNedelcu,andLarsPalzer,
all from Technical University of Munich, for reading and suggesting changes in
early drafts of this book.
This book grew out of the first author’s Ph.D. thesis and of course notes of the
course “Information-Theoretic System Analysis and Design” that he gave at the
Technical University of Munich in 2016. We thus want to thank the students that
attended this course with great interest, asked many smart questions, helped
improving the course notes, and kept the lecturer motivated to invent challenging
end-of-chapter problems. Finally, we thank Gerhard Kramer, Technical University
of Munich, who made this lecture possible.
The book has been written during the first author’s stay at the Institute for
Communications Engineering, Technical University of Munich. The stay has been
funded by the Erwin Schrödinger Fellowship J 3765 of the Austrian Science Fund
and by the German Ministry of Education and Research in the framework of an
Alexander von Humboldt Professorship.
vii
Contents
1 Introduction .... .... .... ..... .... .... .... .... .... ..... .. 1
1.1 Related Work.... .... ..... .... .... .... .... .... ..... .. 6
1.2 Outline and Prerequisites.... .... .... .... .... .... ..... .. 7
1.3 Motivating Example: Analysis of Digital Systems. .... ..... .. 8
1.4 Motivating Example: Quantizer Design. .... .... .... ..... .. 11
Part I Random Variables
2 Piecewise Bijective Functions and Continuous Inputs .... ..... .. 17
2.1 The PDF of Y ¼gðXÞ. ..... .... .... .... .... .... ..... .. 18
2.2 The Differential Entropy of Y ¼gðXÞ .. .... .... .... ..... .. 20
2.3 Information Loss in PBFs ... .... .... .... .... .... ..... .. 21
2.3.1 Elementary Properties. .... .... .... .... .... ..... .. 21
2.3.2 Upper Bounds on the Information Loss ... .... ..... .. 26
2.3.3 Computing Information Loss Numerically . .... ..... .. 29
2.3.4 Application: Polynomials .. .... .... .... .... ..... .. 30
3 General Input Distributions..... .... .... .... .... .... ..... .. 35
3.1 Information Loss for Systems with General Inputs .... ..... .. 36
3.2 Systems with Infinite Information Loss . .... .... .... ..... .. 37
3.3 Alternative Proof of Theorem 2.1 . .... .... .... .... ..... .. 40
4 Dimensionality-Reducing Functions... .... .... .... .... ..... .. 43
4.1 Information Dimension ..... .... .... .... .... .... ..... .. 44
4.1.1 Properties and Equivalent Definitions of Information
Dimension .... ..... .... .... .... .... .... ..... .. 45
4.1.2 dðXÞ-dimensional Entropy and Mixture
of Distributions. ..... .... .... .... .... .... ..... .. 49
4.1.3 Operational Characterization of Information
Dimension .... ..... .... .... .... .... .... ..... .. 53
ix
x Contents
4.2 Relative Information Loss ... .... .... .... .... .... ..... .. 54
4.2.1 Elementary Properties. .... .... .... .... .... ..... .. 55
4.2.2 Bounds on the Relative Information Loss.. .... ..... .. 56
4.2.3 Relative Information Loss for System Reducing
the Dimensionality of Continuous Random Variables.. .. 57
4.2.4 Relative Information Loss and Perfect Reconstruction . .. 60
4.2.5 Outlook: Relative Information Loss
for Discrete-Continuous Mixtures.... .... .... ..... .. 62
4.3 Application: Principal Components Analysis. .... .... ..... .. 64
4.3.1 The Energy-Centered Perspective.... .... .... ..... .. 65
4.3.2 PCA with Given Covariance Matrix.. .... .... ..... .. 67
4.3.3 PCA with Sample Covariance Matrix. .... .... ..... .. 68
5 Relevant Information Loss. ..... .... .... .... .... .... ..... .. 73
5.1 Definition and Properties .... .... .... .... .... .... ..... .. 73
5.1.1 Elementary Properties. .... .... .... .... .... ..... .. 75
5.1.2 An Upper Bound on Relevant Information Loss ..... .. 78
5.2 Signal Enhancement and the Information Bottleneck Method . .. 80
5.3 Application: PCA with Signal-and-Noise Models . .... ..... .. 82
Part II Stationary Stochastic Processes
6 Discrete-Valued Processes . ..... .... .... .... .... .... ..... .. 93
6.1 Information Loss Rate for Discrete-Valued Processes .. ..... .. 94
6.2 Information Loss Rate for Markov Chains... .... .... ..... .. 96
6.3 Outlook: Systems with Memory... .... .... .... .... ..... .. 99
6.3.1 Partially Invertible Systems .... .... .... .... ..... .. 100
6.3.2 Application: Fixed-Point Implementation of a Linear
Filter. .... .... ..... .... .... .... .... .... ..... .. 102
7 Piecewise Bijective Functions and Continuous Inputs .... ..... .. 105
7.1 The Differential Entropy Rate of Stationary Processes.. ..... .. 105
7.2 Information Loss Rate in PBFs ... .... .... .... .... ..... .. 106
7.2.1 Elementary Properties. .... .... .... .... .... ..... .. 106
7.2.2 Upper Bounds on the Information Loss Rate... ..... .. 108
7.2.3 Application: AR(1)-Process in a Rectifier.. .... ..... .. 109
7.3 Outlook: Systems with Memory... .... .... .... .... ..... .. 111
8 Dimensionality-Reducing Functions... .... .... .... .... ..... .. 115
8.1 Relative Information Loss Rate ... .... .... .... .... ..... .. 115
8.2 Application: Downsampling.. .... .... .... .... .... ..... .. 118
8.2.1 The Energy-Centered Perspective.... .... .... ..... .. 119
8.2.2 Information Loss in a Downsampling Device... ..... .. 119
8.3 Outlook: Systems with Memory... .... .... .... .... ..... .. 123
Contents xi
9 Relevant Information Loss Rate . .... .... .... .... .... ..... .. 127
9.1 Definition and Properties .... .... .... .... .... .... ..... .. 127
9.1.1 Upper Bounds on the Relevant Information Loss Rate. .. 129
9.2 Application: Anti-aliasing Filter Design for Downsampling... .. 130
9.2.1 Anti-aliasing Filters for Non-Gaussian Processes ..... .. 132
9.2.2 FIR Solutions for Information Maximization ... ..... .. 135
10 Conclusion and Outlook... ..... .... .... .... .... .... ..... .. 137
References.. .... .... .... .... ..... .... .... .... .... .... ..... .. 141
