Statistical physics of spin glasses and information processing

252 Pages·2001·1.978 MB·English

by  Hidetoshi Nishimori

Statistical Physics of Spin Glasses and
Information Processing
An Introduction
HIDETOSHI NISHIMORI
Department of Physics
Tokyo Institute of Technology
.
CLARENDON PRESS OXFORD
2001
Statistical Physics of Spin Glasses and
Information Processing
An Introduction

Hidetoshi Nishimori,
Department of Physics, Tokyo Institute of Technology, Japan

• One of the few books in this interdisciplinary area

• Rapidly expanding field

• Up-to-date presentation of modern analytical techniques

• Self-contained presentation

Spin glasses are magnetic materials. Statistical mechanics has been a powerful tool to
theoretically analyse various unique properties of spin glasses. A number of new analytical
techniques have been developed to establish a theory of spin glasses. Surprisingly, these
techniques have offered new tools and viewpoints for the understanding of information
processing problems, including neural networks, error-correcting codes, image restoration, and
optimization problems.

This book is one of the first publications of the past ten years that provides a broad overview of
this interdisciplinary field. Most part of the book is written in a self-contained manner, assuming
only a general knowledge of statistical mechanics and basic probability theory. It provides the
reader with a sound introduction to the field and to the analytical techniques necessary to follow
its most recent developments.

Contents: Mean-field theory of phase transitions; Mean-field theory of spin glasses; Replica
symmetry breaking; Gauge theory of spin glasses; Error-correcting codes; Image restoration;
Associative memory; Learning in perceptron; Optimization problems; A. Eigenvalues of the
Hessian; B. Parisi equation; C. Channel coding theorem; D. Distribution and free energy of K-
Sat; References; Index.

International Series of Monographs on Physics No.111, Oxford University Press
Paperback, £24.95, 0-19-850941-3
Hardback, £49.50, 0-19-850940-5
August 2001, 285 pages, 58 line figures, 2 halftones
PREFACE
Thescopeofthetheoryofspinglasseshasbeenexpandingwellbeyonditsorigi-
nalgoalofexplainingtheexperimentalfactsofspinglassmaterials.Forthefirst
time in the history of physics we have encountered an explicit example in which
the phase space of the system has an extremely complex structure and yet is
amenable to rigorous, systematic analyses. Investigations of such systems have
openedanewparadigminstatisticalphysics.Also,theframeworkoftheanalyti-
caltreatmentofthesesystemshasgraduallybeenrecognizedasanindispensable
tool for the study of information processing tasks.
One of the principal purposes of this book is to elucidate some of the im-
portant recent developments in these interdisciplinary directions, such as error-
correctingcodes,imagerestoration,neuralnetworks,andoptimizationproblems.
In particular, I would like to provide a unified viewpoint traversing several dif-
ferent research fields with the replica method as the common language, which
emerged from the spin glass theory. One may also notice the close relationship
between the arguments using gauge symmetry in spin glasses and the Bayesian
methodininformationprocessingproblems.Accordingly,thisbook isnot neces-
sarilywrittenasacomprehensiveintroductiontosingletopicsintheconventional
classification of subjects like spin glasses or neural networks.
In a certain sense, statistical mechanics and information sciences may have
been destined to be directed towards common objectives since Shannon formu-
latedinformationtheoryaboutfiftyyearsagowiththeconceptofentropyasthe
basic building block. It would, however, have been difficult to envisage how this
actually would happen: that the physics of disordered systems, and spin glass
theory in particular, at its maturity naturally encompasses some of the impor-
tant aspects of information sciences, thus reuniting the two disciplines. It would
then reasonably be expected that in the future this cross-disciplinary field will
continue to develop rapidly far beyond the current perspective. This is the very
purpose for which this book is intended to establish a basis.
The book is composed of two parts. The first part concerns the theory of
spin glasses. Chapter 1 is an introduction to the general mean-field theory of
phase transitions. Basic knowledge of statistical mechanics at undergraduate
level is assumed. The standard mean-field theory of spin glasses is developed
in Chapters 2 and 3, and Chapter 4 is devoted to symmetry arguments using
gauge transformations. These four chapters do not cover everything to do with
spinglasses.Forexample,hotlydebatedproblemslikethethree-dimensionalspin
glassandanomalouslyslowdynamicsarenot includedhere.Thereaderwillfind
relevant references listed at the end of each chapter to cover these and other
topics not treated here.
v
vi PREFACE
The second part deals with statistical-mechanical approaches to information
processingproblems.Chapter5isdevotedtoerror-correctingcodesandChapter
6 to image restoration. Neural networks are discussed in Chapters 7 and 8, and
optimization problems are elucidated in Chapter 9. Most of these topics are
formulated as applications of the statistical mechanics of spin glasses, with a
few exceptions. For each topic in this second part, there is of course a long
history, and consequently a huge amount of knowledge has been accumulated.
The presentation in the second part reflects recent developments in statistical-
mechanicalapproachesanddoesnotnecessarilycoveralltheavailablematerials.
Again,thereferencesattheendofeachchapterwillbehelpfulinfillingthegaps.
The policy for listing up the references is, first, to refer explicitly to the original
papers for topics discussed in detail in the text, and second, whenever possible,
to refer to review articles and books at the end of a chapter inorder to avoidan
excessively long list of references. I therefore have to apologize to those authors
whose papers have only been referred to indirectly via these reviews and books.
The reader interested mainly in the second part may skip Chapters 3 and 4
in the first part before proceeding to the second part. Nevertheless it is recom-
mendedtobrowsethroughtheintroductorysectionsofthesechapters,including
replica symmetry breaking (§§3.1 and 3.2) and the main part of gauge theory
(§§4.1 to 4.3 and 4.6), for a deeper understanding of the techniques relevant to
the second part. It is in particular important for the reader who is interested in
Chapters 5 and 6 to go through these sections.
The present volume is the English edition of a book written in Japanese
by me and published in 1999. I have revised a significant part of the Japanese
edition and added new material in this English edition. The Japanese edition
emerged from lectures at Tokyo Institute of Technology and several other uni-
versities. I would like to thank those students who made useful comments on
the lecture notes. I am also indebted to colleagues and friends for collabora-
tions, discussions, and comments on the manuscript: in particular, to Jun-ichi
Inoue, Yoshiyuki Kabashima, Kazuyuki Tanaka,Tomohiro Sasamoto, Toshiyuki
Tanaka, Shigeru Shinomoto, Taro Toyoizumi, Michael Wong, David Saad, Peter
Sollich, Ton Coolen, and John Cardy. I am much obliged to David Sherrington
forusefulcomments,collaborations, andasuggestionto publishthepresentEn-
glishedition.Ifthisbook isusefultothereader,agoodpartofthecreditshould
be attributed to these outstanding people.
H. N.
Tokyo
February 2001
CONTENTS
1 Mean-field theory of phase transitions 1
1.1 Ising model 1
1.2 Order parameter and phase transition 3
1.3 Mean-field theory 4
1.3.1 Mean-field Hamiltonian 4
1.3.2 Equation of state 5
1.3.3 Free energy and the Landau theory 6
1.4 Infinite-range model 7
1.5 Variational approach 9
2 Mean-field theory of spin glasses 11
2.1 Spin glass and the Edwards–Anderson model 11
2.1.1 Edwards–Anderson model 12
2.1.2 Quenched system and configurational average 12
2.1.3 Replica method 13
2.2 Sherrington–Kirkpatrick model 13
2.2.1 SK model 14
2.2.2 Replica average of the partition function 14
2.2.3 Reduction by Gaussian integral 15
2.2.4 Steepest descent 15
2.2.5 Order parameters 16
2.3 Replica-symmetric solution 17
2.3.1 Equations of state 17
2.3.2 Phase diagram 19
2.3.3 Negative entropy 21
3 Replica symmetry breaking 23
3.1 Stability of replica-symmetric solution 23
3.1.1 Hessian 24
3.1.2 Eigenvalues of the Hessian and the AT line 26
3.2 Replica symmetry breaking 27
3.2.1 Parisi solution 28
3.2.2 First-step RSB 29
3.2.3 Stability of the first step RSB 31
3.3 Full RSB solution 31
3.3.1 Physical quantities 31
3.3.2 Order parameter near the critical point 32
3.3.3 Vertical phase boundary 33
3.4 Physical significance of RSB 35
vii
viii CONTENTS
3.4.1 Multivalley structure 35
3.4.2 q and q 35
EA
3.4.3 Distribution of overlaps 36
3.4.4 Replica representation of the order parameter 37
3.4.5 Ultrametricity 38
3.5 TAP equation 38
3.5.1 TAP equation 39
3.5.2 Cavity method 41
3.5.3 Properties of the solution 43
4 Gauge theory of spin glasses 46
4.1 Phase diagram of finite-dimensional systems 46
4.2 Gauge transformation 47
4.3 Exact solution for the internal energy 48
4.3.1 Application of gauge transformation 48
4.3.2 Exact internal energy 49
4.3.3 Relation with the phase diagram 50
4.3.4 Distribution of the local energy 51
4.3.5 Distribution of the local field 51
4.4 Bound on the specific heat 52
4.5 Bound on the free energy and internal energy 53
4.6 Correlation functions 55
4.6.1 Identities 55
4.6.2 Restrictions on the phase diagram 57
4.6.3 Distribution of order parameters 58
4.6.4 Non-monotonicity of spin configurations 61
4.7 Entropy of frustration 62
4.8 Modified ±J model 63
4.8.1 Expectation value of physical quantities 63
4.8.2 Phase diagram 64
4.8.3 Existence of spin glass phase 65
4.9 Gauge glass 67
4.9.1 Energy, specific heat, and correlation 67
4.9.2 Chirality 69
4.9.3 XY spin glass 70
4.10 Dynamical correlation function 71
5 Error-correcting codes 74
5.1 Error-correcting codes 74
5.1.1 Transmission of information 74
5.1.2 Similarity to spin glasses 75
5.1.3 Shannon bound 76
5.1.4 Finite-temperature decoding 78
5.2 Spin glass representation 78
5.2.1 Conditional probability 78
CONTENTS ix
5.2.2 Bayes formula 79
5.2.3 MAP and MPM 80
5.2.4 Gaussian channel 81
5.3 Overlap 81
5.3.1 Measure of decoding performance 81
5.3.2 Upper bound on the overlap 82
5.4 Infinite-range model 83
5.4.1 Infinite-range model 84
5.4.2 Replica calculations 84
5.4.3 Replica-symmetric solution 86
5.4.4 Overlap 87
5.5 Replica symmetry breaking 88
5.5.1 First-step RSB 88
5.5.2 Random energy model 89
5.5.3 Replica solution in the limit r →∞ 91
5.5.4 Solution for finite r 93
5.6 Codes with finite connectivity 95
5.6.1 Sourlas-type code with finite connectivity 95
5.6.2 Low-density parity-check code 98
5.6.3 Cryptography 101
5.7 Convolutional code 102
5.7.1 Definition and examples 102
5.7.2 Generating polynomials 103
5.7.3 Recursive convolutional code 104
5.8 Turbo code 106
5.9 CDMA multiuser demodulator 108
5.9.1 Basic idea of CDMA 108
5.9.2 Conventional and Bayesian demodulators 110
5.9.3 Replica analysis of the Bayesian demodulator 111
5.9.4 Performance comparison 114
6 Image restoration 116
6.1 Stochastic approach to image restoration 116
6.1.1 Binary image and Bayesian inference 116
6.1.2 MAP and MPM 117
6.1.3 Overlap 118
6.2 Infinite-range model 119
6.2.1 Replica calculations 119
6.2.2 Temperature dependence of the overlap 121
6.3 Simulation 121
6.4 Mean-field annealing 122
6.4.1 Mean-field approximation 123
6.4.2 Annealing 124
6.5 Edges 125
x CONTENTS
6.6 Parameter estimation 128
7 Associative memory 131
7.1 Associative memory 131
7.1.1 Model neuron 131
7.1.2 Memory and stable fixed point 132
7.1.3 StatisticalmechanicsoftherandomIsingmodel 133
7.2 Embedding a finite number of patterns 135
7.2.1 Free energy and equations of state 135
7.2.2 Solution of the equation of state 136
7.3 Many patterns embedded 138
7.3.1 Replicated partition function 138
7.3.2 Non-retrieved patterns 138
7.3.3 Free energy and order parameter 140
7.3.4 Replica-symmetric solution 141
7.4 Self-consistent signal-to-noise analysis 142
7.4.1 Stationary state of an analogue neuron 142
7.4.2 Separation of signal and noise 143
7.4.3 Equation of state 145
7.4.4 Binary neuron 145
7.5 Dynamics 146
7.5.1 Synchronous dynamics 147
7.5.2 Time evolution of the overlap 147
7.5.3 Time evolution of the variance 148
7.5.4 Limit of applicability 150
7.6 Perceptron and volume of connections 151
7.6.1 Simple perceptron 151
7.6.2 Perceptron learning 152
7.6.3 Capacity of a perceptron 153
7.6.4 Replica representation 154
7.6.5 Replica-symmetric solution 155
8 Learning in perceptron 158
8.1 Learning and generalization error 158
8.1.1 Learning in perceptron 158
8.1.2 Generalization error 159
8.2 Batch learning 161
8.2.1 Bayesian formulation 162
8.2.2 Learning algorithms 163
8.2.3 High-temperatureandannealedapproximations 165
8.2.4 Gibbs algorithm 166
8.2.5 Replica calculations 167
8.2.6 Generalization error at T =0 169
8.2.7 Noise and unlearnable rules 170
8.3 On-line learning 171
CONTENTS xi
8.3.1 Learning algorithms 171
8.3.2 Dynamics of learning 172
8.3.3 Generalization errors for specific algorithms 173
8.3.4 Optimization of learning rate 175
8.3.5 Adaptivelearningrateforsmoothcostfunction 176
8.3.6 Learning with query 178
8.3.7 On-line learning of unlearnable rule 179
9 Optimization problems 183
9.1 Combinatorial optimization and statistical mechanics 183
9.2 Number partitioning problem 184
9.2.1 Definition 184
9.2.2 Subset sum 185
9.2.3 Number of configurations for subset sum 185
9.2.4 Number partitioning problem 187
9.3 Graph partitioning problem 188
9.3.1 Definition 188
9.3.2 Cost function 189
9.3.3 Replica expression 190
9.3.4 Minimum of the cost function 191
9.4 Knapsack problem 192
9.4.1 Knapsack problem and linear programming 192
9.4.2 Relaxation method 193
9.4.3 Replica calculations 193
9.5 Satisfiability problem 195
9.5.1 Random satisfiability problem 195
9.5.2 Statistical-mechanical formulation 196
9.5.3 Replica symmetric solution and its interpreta-
tion 199
9.6 Simulated annealing 201
9.6.1 Simulated annealing 202
9.6.2 Annealing schedule and generalized transition
probability 203
9.6.3 Inhomogeneous Markov chain 204
9.6.4 Weak ergodicity 206
9.6.5 Relaxation of the cost function 209
9.7 Diffusion in one dimension 211
9.7.1 Diffusion and relaxation in one dimension 211
A Eigenvalues of the Hessian 214
A.1 Eigenvalue 1 214
A.2 Eigenvalue 2 215
A.3 Eigenvalue 3 216
B Parisi equation 217
xii CONTENTS
C Channel coding theorem 220
C.1 Information, uncertainty, and entropy 220
C.2 Channel capacity 221
C.3 BSC and Gaussian channel 223
C.4 Typical sequence and random coding 224
C.5 Channel coding theorem 226
D Distribution and free energy of K-SAT 228
References 232
Index 243