Table Of ContentModel Calibration and Parameter Estimation
Ne-Zheng Sun • Alexander Sun
Model Calibration and
Parameter Estimation
For Environmental and Water Resource
Systems
1 3
Ne-Zheng Sun Alexander Sun
Department of Civil and Environmental Bureau of Economic Geology, Jackson
Engineering School of Geosciences
University of California at Los Angeles University of Texas at Austin
Los Angeles Austin
California Texas
USA USA
ISBN 978-1-4939-2322-9 ISBN 978-1-4939-2323-6 (eBook)
DOI 10.1007/978-1-4939-2323-6
Library of Congress Control Number: 2015931547
Mathematics Subject Classification (2010): 97Mxx, 93A30, 93B11, 93B30, 90Cxx, 65Kxx, 86A05,
15A29, 35R30, 86A22, 65M32, 65N21, 81T80, 62P12, 65Cxx
Springer New York Heidelberg Dordrecht London
© Springer Science+Business Media, LLC 2015
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publi-
cation does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, express or implied, with respect to the material contained herein or for any
errors or omissions that may have been made.
Printed on acid-free paper
Springer New York is part of Springer Science+Business Media (www.springer.com)
To
Rachel, Albert, Adam, and Jacob
Preface
A mathematical model constructed for a real system must be calibrated by data
and its uncertainty must be assessed before used for prediction, decision making,
and management purposes. After more than a half century of study, however, the
construction of a reliable model for complex systems is still a challenging task.
In mathematical modeling, the “prediction problem” or the “forward problem”
(inputs → outputs) uses as model inputs a fixed model structure, known model pa-
rameters, given system controls, and other necessary information to find the sys-
tem states (as model outputs). Unless all properties of the modeled system can be
measured directly, model inputs tend to always contain unknowns or uncertainties
that have to be determined indirectly. Model calibration (outputs → inputs) uses the
measured system states and other available information to identify or estimate the
unknown model inputs. Thus, in a certain sense it is the “inverse problem” of model
prediction. The history of studying model calibration is probably as long as the his-
tory of forward modeling, but the progress of study had been slow due to the very
nature of inversion: identifying the causes from results is always more difficult than
predicting the results on the basis of known causes. Various optimization-based
data-fitting methods developed for solving the classical inverse problem in math-
ematics have been proven to be successful for model calibration, only if a system’s
structure is simple and well-defined, and both the number and dimensions of un-
known parameters are low. When these assumptions do not hold, the use of data-
fitting may produce an unacceptable model.
With the advances in computing, instrumentation, and information technologies,
more and more sophisticated numerical models have been developed for simulating
complicated physical, chemical, and biological processes observed in environment,
energy, water resources, and other scientific and engineering fields. The advent of
highly sophisticated software packages has made solution to the “forward problem”
much easier, but, at the same time, calibrating the resulting model becomes more
difficult due to the increase of model complexity. Modelers gradually realize that:
(1) the requirement of model uniqueness has to be given up; (2) the classical con-
cept of model inversion must be extended to include model structure identification,
model reduction, and model error quantification; and (3) five fundamental prob-
lems of modeling technology (i.e., the model selection problem, model calibration
problem, model reliability problem, model application problem, and data collection
vii
viii Preface
problem) must be considered systematically rather than separately or sequentially
during the model construction stage. The model calibration problem has thus be-
come more challenging but, at the same time, the process of solving the problem
becomes more interesting and rewarding than the simple data-fitting exercise. Em-
powered by these methodological understandings and newly developed tools in
mathematics and statistics, the research on model construction has made significant
progress in recent years. As a result, existing methods have been improved and new
and promising approaches have emerged.
This book provides a comprehensive introduction on all aspects of constructing
useful models: from the deterministic framework to the statistical frameworks; from
the classical inverse problem of parameter estimation to the extended inverse prob-
lem of system structure identification; from physical-based models to data-driven
models; from model reduction to model uncertainty quantification; from data suffi-
ciency assessment to optimal experimental design; and from basic concepts, theory,
and methods to the state-of-the-art approaches developed for model construction.
A central problem to be considered in this book is how to find surrogate models for
predetermined model applications.
Chapter 1 is a general description of the modeling technology. Models that are
often seen in environmental and water resources fields are introduced here and are
used to exemplify different methods throughout the book. Based on different crite-
ria of model calibration, three kinds of inverse problem are defined: the classical in-
verse problem (CIP) for parameter estimation, the extended inverse problem (EIP)
for system identification, and the goal-oriented inverse problem (GIP) for model
application. After reading this chapter, readers will be able to get a holistic picture
of mathematical modeling, know the difficulties and problems of model construc-
tion, and learn how this book is organized.
Part I of this three-part book (Chapter 2−5) is contributed to the solution of CIP.
Basic concepts and methods on linear model inversion and single-state nonlinear
model inversion are given in Chapter 2; singular value decomposition and various
nonlinear optimization algorithms are introduced briefly. In Chapter 3, the multi-
state model inversion is cast into a multi-objective optimization problem and solved
by the evolutionary algorithms. Regularization is also introduced in this chapter
from the point of view of multi-criterion inversion. The inverse problem is refor-
mulated and resolved in the statistical framework in Chapter 4. Monte Carlo based
sampling methods, including the Markov Chain Monte Carlo method, are intro-
duced for finding the posterior distribution. Various methods of model differentia-
tion are given in Chapter 5. Model differentiation is a necessary tool for almost all
topics covered in this book.
Part II of this book (Chapter 6−8) is dedicated to the solution of EIP. In Chapter
6, various methods for parameterizing deterministic functions or random fields are
introduced. Principal component analysis and other linear and nonlinear dimension
reduction methods, as well as their applications to inverse solution, are covered.
Model structure identification and hyperparameter estimation are the main topics
of Chapter 7, in which various adaptive parameterization approaches, the level set
method, multiscale inversion, and geostatistical inversion are introduced. Methods
for constructing data-driven models are given in Chapter 8, including linear regres-
Preface ix
sion and various machine learning methods such as artificial neural networks, sup-
port vector machine, and Gaussian process regression.
Part III of the book (Chapter 9−12) is contributed to the topic of model reliability.
Chapter 9 introduces various data assimilation methods for inverse solution that al-
low us to update a model continuously to improve its reliability whenever new data
become available. Methods used for uncertainty quantification, including Monte
Carlo simulation, global sensitivity analysis, stochastic response surface, are intro-
duced systematically in Chapter 10. The effects of model parameter uncertainty and
model structure uncertainty on model outputs are assessed. To construct a more reli-
able model, more data are needed. Design of informative and cost-effective data col-
lection strategies is the subject of Chapter 11, in which, optimal experimental design
is formulated into a multi-objective optimization problem. The criteria of optimal
design for linear model inversion are derived. For nonlinear model inversion, Bayes-
ian and robust design methods, especially, the interval-identifiability-based robust
design, are introduced. In Chapter 12, after the goal-oriented forward problem is
described, the GIP is formulated and solved in both the deterministic and statistical
frameworks. When the existing data are insufficient, a cost-effective experimental
design method is given. Finally, the goal-oriented pilot-point method is described.
Preliminary mathematics required for reading this book is reviewed in details
in three Appendices. To help readers better understand the text, review questions
are given at the end of each chapter. All major methods introduced in this book are
illustrated with numerical examples created by the authors, including the informa-
tion on available toolboxes. Alex Sun authored Chapters 6, 8, 9, and 10 and edited
the whole book. Other chapters are authored by Ne-Zheng Sun. This book can be
used as a textbook for graduate and upper-level undergraduate students majoring in
environmental engineering, hydrology, or geosciences. It also serves as an essential
reference book for petroleum engineers, mining engineers, chemists, mechanical
engineers, biologists, medical engineers, applied mathematicians, and others who
perform mathematical modeling. Much of the research conducted by the authors
over the years has been made possible by the support from U.S. National Science
Foundation (NSF), National Aeronautics and Space Administration (NASA), De-
partment of Energy (DOE), Environmental Protection Agency (EPA), and Nucle-
ar Regulatory Commission (NRC). We are grateful to all of our current and past
collaborators for insightful discussions. Ne-Zheng Sun would like to give special
thanks to Drs. Jacob Bear and William Yeh for their guidance, support, collabora-
tion, and long-term friendship. Alex Sun would like to thank Drs. Yoram Rubin and
Dongxiao Zhang, and his colleagues at the University of Texas at Austin for their
advice and collaboration.
We are grateful to the editors Achi Dosanjh, Donna Chernyk, and Danielle Walk-
er at Springer for their support and guidance in every step of the process. Finally,
we would like to thank Fang and Zhenzhen for their endless love, sacrifice, and
patience during this multiyear-long book project.
Santa Monica, CA Ne-Zheng Sun
Austin, TX Alex Sun
Contents
1 Introduction ................................................................................................ 1
1.1 Mathematical Modeling ..................................................................... 2
1.1.1 Modeling an Open System ..................................................... 2
1.1.2 Examples of EWR Models ..................................................... 2
1.1.3 General Form and Classification ............................................ 7
1.1.4 Model Construction Process ................................................... 8
1.2 Forward Solution ................................................................................ 10
1.2.1 T he Forward Problem ............................................................. 10
1.2.2 Solution Methods .................................................................... 10
1.2.3 W ell-Posedness of the Forward Problem ................................ 13
1.3 Model Calibration and Parameter Estimation .................................... 14
1.3.1 Data Availability ..................................................................... 14
1.3.2 Methods and Criteria .............................................................. 15
1.3.3 T he Inverse Problem ............................................................... 15
1.3.4 Model Structure Identification ............................................... 19
1.4 Model Reliability ............................................................................... 20
1.4.1 Model Uncertainty Analysis ................................................... 20
1.4.2 Problems and Difficulties in Model Construction .................. 21
1.4.3 Goal-Oriented Modeling ........................................................ 23
1.5 Review Questions ............................................................................... 24
2 The Classical Inverse Problem .................................................................. 25
2.1 Approximate Solutions ....................................................................... 26
2.1.1 T he Direct Method for Inverse Solution ................................. 26
2.1.2 T he Indirect Method of Inversion ........................................... 28
2.1.3 W ell-Posedness of the Quasi-solution .................................... 30
2.1.4 Parameterization and Discretization ....................................... 32
2.2 Linear Model Identification ............................................................... 34
2.2.1 Linear Model and Normal Equation ....................................... 34
2.2.2 Estimation Using Singular Value Decomposition .................. 38
2.2.3 T runcated Singular Value Decomposition .............................. 41
2.2.4 Linearization ........................................................................... 43
2.3 Nonlinear Model Identification .......................................................... 46
2.3.1 Inverse Solution and Optimization ......................................... 46
xi
xii Contents
2.3.2 Basic Concepts of Numerical Optimization ......................... 48
2.3.3 A lgorithms for Local Optimization ...................................... 51
2.3.4 Norm Selection ..................................................................... 58
2.4 The Gauss–Newton Method ............................................................. 61
2.4.1 T he Gauss–Newton Method for Least Squares .................... 61
2.4.2 Modified Gauss–Newton Methods ....................................... 63
2.4.3 A pplication to Inverse Solution ............................................ 65
2.5 Review Questions ............................................................................. 66
3 Multiobjective Inversion and Regularization ........................................ 69
3.1 Multiobjective Optimization ............................................................ 70
3.2 T he Second Criterion of Inverse Problem Formulation ................... 71
3.2.1 Inversion with Prior Information .......................................... 71
3.2.2 Formulation of the Bicriterion Inverse Problem .................. 72
3.2.3 Inversion with Constrained Optimization ............................ 76
3.3 Regularization .................................................................................. 78
3.3.1 T ikhonov Regularization ...................................................... 78
3.3.2 Regularization of Linear Models .......................................... 79
3.3.3 Selection of Regularization Coefficients .............................. 80
3.4 Parameter Identification of Multistate Models ................................ 86
3.4.1 Multistate Modeling ............................................................. 86
3.4.2 Coupled Inverse Problems .................................................... 88
3.4.3 Solution of Coupled Inverse Problems ................................. 90
3.5 Parameter Identification with Multiobjectives ................................. 93
3.6 A lgorithms for Multiobjective Optimization ................................... 96
3.6.1 Deterministic Methods ......................................................... 96
3.6.2 Genetic Algorithm ................................................................ 99
3.6.3 Multiobjective Evolutionary Algorithm ............................... 103
3.7 Review Questions .............................................................................. 105
4 Statistical Methods for Parameter Estimation ...................................... 107
4.1 T he Statistical Inverse Problem ........................................................ 108
4.1.1 Statement of the Statistical Inverse Problem ........................ 108
4.1.2 Information Content and Uncertainty ................................... 109
4.1.3 Bayesian Inference for Inverse Solution .............................. 111
4.1.4 Probability Distribution of Observation Error ...................... 114
4.1.5 Probability Distribution of Prior Information ....................... 116
4.2 Point Estimation ............................................................................... 117
4.2.1 Maximum a Posteriori Estimate ........................................... 117
4.2.2 Estimators With Uniform Prior Distribution ........................ 119
4.2.3 Estimators With Gaussian Prior Distribution ....................... 121
4.2.4 Minimum Relative Entropy Estimator ................................. 123
4.2.5 Bayesian Inversion for Multistate Models ........................... 124
4.3 Monte Carlo Methods for Bayesian Inference ................................. 125
4.3.1 Exploring the Posterior Distribution .................................... 125
4.3.2 Markov Chain Monte Carlo Sampling Techniques .............. 126
