Convex Optimization for Inequality Constrained Adjustment Problems

138 Pages·2015·3.65 MB·English

Institut für Geodäsie und Geoinformation
Theoretische Geodäsie
Convex Optimization for
Inequality Constrained Adjustment Problems
Inaugural-Dissertation
zur
Erlangung des Grades
Doktor-Ingenieur
(Dr.-Ing.)
der
Landwirtschaftlichen Fakultät
der
Rheinischen Friedrich-Wilhelms-Universität Bonn
von
Lutz Rolf Roese-Koerner
aus
Bad Neuenahr-Ahrweiler
Referent: Prof. Dr. techn. Wolf-Dieter Schuh
Korreferent: Prof. Dr. Hans-Peter Helfrich
Korreferent: Prof. Dr.-Ing. Nico Sneeuw
Tag der mündlichen Prüfung: 10. Juli 2015
Erscheinungsjahr: 2015
Convex Optimization for
Inequality Constrained Adjustment Problems
Summary
Whenever a certain function shall be minimized (e.g., a sum of squared residuals) or maximized
(e.g., profit) optimization methods are applied. If in addition prior knowledge about some of the
parameters can be expressed as bounds (e.g., a non-negativity bound for a density) we are dealing
with an optimization problem with inequality constraints. Although, common in many economic
and engineering disciplines, inequality constrained adjustment methods are rarely used in geodesy.
Within this thesis methodology aspects of convex optimization methods are covered and analogies
toadjustmenttheoryareprovided.Furthermore,threeshortcomingsareidentifiedwhichare—inthe
opinion of the author—the main obstacles that prevent a broader use of inequality constrained ad-
justment theory in geodesy. First, most optimization algorithms do not provide quality information
oftheestimate.Second,mostoftheexistingalgorithmsfortheadjustmentofrank-deficientsystems
either provide only one arbitrary particular solution or compute only an approximative solution.
Third, the Gauss-Helmert model with inequality constraints was hardly treated in the literature
so far. We propose solutions for all three obstacles and provide simulation studies to illustrate our
approach and to show its potential for the geodetic community.
Thus, the aim of this thesis is to make accessible the powerful theory of convex optimization with
inequality constraints for classic geodetic tasks.
Konvexe Optimierung für
Ausgleichungsaufgaben mit Ungleichungsrestriktionen
Zusammenfassung
Methoden der konvexen Optimierung kommen immer dann zum Einsatz, wenn eine Zielfunk-
tion minimiert oder maximiert werden soll. Prominente Beispiele sind eine Minimierung der
Verbesserungsquadratsumme oder eine Maximierung des Gewinns. Oft liegen zusätzliche Vorinfor-
mationen über die Parameter vor, die als Ungleichungen ausgedrückt werden können (beispielsweise
eine Nicht-Negativitätsschranke für eine Dichte). In diesem Falle erhält man ein Optimierungspro-
blem mit Ungleichungsrestriktionen. Ungeachtet der Tatsache, dass Methoden zur Ausgleichungs-
rechnung mit Ungleichungen in vielen ökonomischen und ingenieurwissenschaftlichen Disziplinen
weit verbreitet sind, werden sie dennoch in der Geodäsie kaum genutzt.
In dieser Arbeit werden methodische Aspekte der konvexen Optimierung behandelt und Analogien
zur Ausgleichungsrechnung aufgezeigt. Desweiteren werden drei große Defizite identifiziert, die –
nach Meinung des Autors – bislang eine häufigere Anwendung von restringierten Ausgleichungs-
techniken in der Geodäsie verhindern. Erstens liefern die meisten Optimierungsalgorithmen aus-
schließlicheineSchätzungderunbekanntenParameter,jedochkeineAngabeüberderenGenauigkeit.
ZweitensistdieBehandlungvonrangdefektenSystemenmitUngleichungsrestriktionennichttrivial.
Bestehende Verfahren beschränken sich hier zumeist auf die Angabe einer beliebigen Partikulär-
lösung oder ermöglichen gar keine strenge Berechnung der Lösung. Drittens wurde das Gauß-
Helmert-Modell mit Ungleichungsrestriktionen in der Literatur bisher so gut wie nicht behandelt.
LösungsmöglichkeitenfürallegenanntenDefizitewerdenindieserArbeitvorgeschlagenundkommen
in Simulationsstudien zum Einsatz, um ihr Potential für geodätische Anwendungen aufzuzeigen.
Diese Arbeit soll somit einen Beitrag dazu leisten, die mächtige Theorie der konvexen Optimierung
mit Ungleichungsrestriktionen für klassisch geodätische Aufgabenstellungen nutzbar zu machen.

I
Contents
1 Introduction 1
1.1 Motivation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Scientific Context and Objectives of this Thesis . . . . . . . . . . . . . . . . . . . . . 3
1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
I Fundamentals 7
2 Adjustment Theory 8
2.1 Mathematical Foundations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3 Convex Optimization 23
3.1 Convexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Minimization with Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3 Quadratic Program (QP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.4 Duality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.5 Active-Set Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.6 Feasibility and Phase-1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
II Methodological Contributions 43
4 A Stochastic Framework for ICLS 44
4.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.2 Quality Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.3 Analysis Tools for Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4 The Monte Carlo Quadratic Programming (MC-QP) Method . . . . . . . . . . . . . 53
4.5 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
II Contents
5 Rank-Deficient Systems with Inequalities 59
5.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5.2 Extending Active-Set Methods for Rank-Deficient Systems . . . . . . . . . . . . . . . 61
5.3 Rigorous Computation of a General Solution. . . . . . . . . . . . . . . . . . . . . . . 65
5.4 A Framework for the Solution of Rank-Deficient ICLS Problems . . . . . . . . . . . . 71
5.5 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
6 The Gauss-Helmert Model with Inequalities 77
6.1 State of the Art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.2 WLS Adjustment in the Inequality Constrained GHM . . . . . . . . . . . . . . . . . 78
6.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
III Simulation Studies 93
7 Applications 94
7.1 Robust Estimation with Inequality Constraints . . . . . . . . . . . . . . . . . . . . . 94
7.2 Stochastic Description of the Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.3 Applications with Rank-Deficient Systems . . . . . . . . . . . . . . . . . . . . . . . . 104
7.4 The Gauss-Helmert Model with Inequalities . . . . . . . . . . . . . . . . . . . . . . . 108
IV Conclusion and Outlook 113
8 Conclusion and Outlook 114
8.1 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
8.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
A Appendix i
A.1 Transformation of the Dual Function of an ICLS Problem . . . . . . . . . . . . . . . i
A.2 Deactivating Active Constraints in the Active-Set Algorithm . . . . . . . . . . . . . . ii
A.3 Reformulation of the Huber Loss Function . . . . . . . . . . . . . . . . . . . . . . . . iv
A.4 Data of the Positive Cosine Example . . . . . . . . . . . . . . . . . . . . . . . . . . . v
Tables of Symbols vii
List of Abbreviations viii
List of Figures viii
List of Tables ix
References xv
1
1. Introduction
1.1 Motivation
Many problems in both science and society can be mathematically formulated as optimization
problems.Thetermoptimization alreadyimpliesthatthetaskistofindasolutionthatisoptimal—
i.e.better(oratleastnotworse)thananyotherexistingsolution.Thus,optimizationmethodsalways
come into play if a cost function should be minimized or a profit function should be maximized.
Prominent examples in geodesy include the minimization of the sum of squared residuals or the
maximization of a likelihood function.
In many cases additional information on (or restrictions of) the unknown parameters exist that
could be expressed as constraints. Equality constraints can be easily incorporated in the adjustment
process to take advantage of as much information as possible. However, it is not always possible
to express knowledge on the parameters as equalities. Possible examples are a lower bound of zero
for non-negative quantities such as densities or repetition numbers, or an upper bound for the
maximal feasible slope of a planned road. In these cases, inequalities are used to express the addi-
tional knowledge on the parameters which leads to an optimization problem with inequality
constraints.
If in addition it is known that only a global minimum exists, a convex optimization problem
with inequality constraints has to be solved.
Despitethefactthatthereisarichbodyofworkonconvexoptimizationwithinequalityconstraints
in many engineering and economic disciplines, its application in classic geodetic adjustment tasks
is rather scarce.
Within this thesis methodological aspects from the field of convex optimization with inequality
constraints are described and analogies to classic adjustment theory are pointed out. Furthermore,
we identified the following three shortcomings, which are—in our opinion—the main obstacles that
prevent a broader use of inequality constrained adjustment theory in geodesy:
Quality description. It is a fundamental concept in geodesy that not only the value of an es-
timated quantity is of interest but also its accuracy. Information on the accuracy is usually
provided as a variance-covariance matrix, which allows us to extract standard deviations and
information on the correlation of different parameters. However, this is no longer possible in
the presence of inequality constraints as their influence on the parameters cannot be described
analytical. Furthermore, symmetric confidence regions are no longer sufficient to describe the
accuracy as the parameter space is truncated by inequality constraints.
Rank-deficient systems. Many applications in geodesy lead to a rank-deficient system of normal
equations. Examples are the second order design of a geodetic network with more weights to
be estimated than entries in the criterion matrix, or the adjustment of datum-free networks.
However, most state-of-the-art optimization algorithms for inequality constrained problems
are either not capable of solving a singular system of linear equations or provide only one of
an infinite number of solutions.
Gauss-Helmert model. Oftentimes, not only the relationship between a measured quantity
(i.e.anobservation)andtheunknownparametershastobemodeled,butalsotherelationship
between two or more observations themselves. In this case, it is not possible to perform an ad-
justmentintheGauss-Markovmodel,andtheGauss-Helmertmodelisusedinstead.However,
itisnotstraightforwardtoperformaninequalityconstrainedestimationintheGauss-Helmert
model.
2 1. Introduction
1.2 State of the Art
A rich literature on convex optimization exists, including textbooks such as Gill et al. (1981),
Fletcher (1987) or Boyd and Vandenberghe (2004).
Thesameholdstrueforaspecialcaseofconvexoptimizationproblem:thequadraticprogram.Here,
a quadratic objective function should be minimized subject to some linear constraints. Two out of
many works on this topic are Simon and Simon (2003) and Wong (2011). The former proposed
a quadratic programming approach to set up an inequality constrained Kalman Filter for aircraft
turbofan engine health estimation. The latter examined certain active-set methods for quadratic
programming in positive (semi-)definite as well as in indefinite systems.
Not only in the geodetic community, there exist many articles on the solution of an Inequality Con-
strained Least-Squares (ICLS) problem as a quadratic program. Stoer (1971) for example proposed,
what he defined as “a numerically stable algorithm for solving linear least-squares problems with
linear equalities and inequalities as additional constraints”. Klumpp (1973) formulated the problem
of estimating an optimal horizontal centerline in road design as a quadratic program.
Fritsch and Schaffrin (1980) and Koch (1981) were the first to address inequality constrained least-
squares problems in geodesy. While the former formulated the design of optimal FIR filters as
ICLS problem, the latter examined hypothesis testing with inequality constraints. Later on Schaf-
frin (1981), Koch (1982, 1985) and Fritsch (1982) transformed the quadratic programming problem
resulting from the first and second order design of a geodetic network into a linear complementarity
problem and solved it via Lemke’s algorithm. Fritsch (1983, 1985) examined further possibilities
resulting from the use of ICLS for the design of FIR filters and other geodetic applications. Xu et al.
(1999) proposed an ansatz to stabilize ill-conditioned LCP problems.
A more recent approach stems from Peng et al. (2006) who established a method to express many
simple inequality constraints as one intricate equality constraint in a least-squares context. Koch
(2006) formulated constraints for the semantical integration of two-dimensional objects and digital
terrain models. Kaschenz (2006) used inequality constraints as an alternative to the Tikhonov reg-
ularization leading to a non-negative least-squares problem. She applied her proposed framework to
the analysis of radio occultation data from GRACE (Gravity Recovery And Climate Experiment).
Tangetal.(2012)usedinequalitiesassmoothnessconstraintstoimprovetheestimatedmasschanges
in Antarctica from GRACE observations, which leads again to a quadratic program.
Much less literature exists on the quality description of inequality constrained estimates. The prob-
ably most cited work in this area is the frequentist approach of Liew (1976). He first identified the
active constraints and used them to approximate an inequality constrained least-squares problem
by an equality constrained one. Geweke (1986) on the other hand suggested a Bayesian approach,
which was further developed and introduced to geodesy by Zhu et al. (2005).
However, both approaches are incomplete. While in the first ansatz, the influence of inactive con-
straints is neglected, the second ansatz ignores the probability mass in the infeasible region. Thus,
we propose the MC-QP method in Chap. 4 which overcomes both drawbacks.
The probably most import contributions to the area of rank-deficient systems with inequality con-
straints are the works of Werner (1990), Werner and Yapar (1996) and Wong (2011). In the two
former articles a projector theoretical approach for the rigorous computation of a general solution
of ICLS problems with a possible rank defect is proposed using generalized inverses. In the latter
an extension of classic active-set methods for quadratic programming is described, which enables us
to compute a particular solution despite a possible rank defect.
However, the ansatz from Werner (1990) and Werner and Yapar (1996) is only suited for small-scale
problems, and the approach of Wong (2011) lacks a description of the homogeneous solution. Thus,
we propose a framework for rank-deficient and inequality constrained problems in Chap. 5 that is
applicable to larger problems and provides a particular as well as a homogeneous solution.
1.3. Scientific Context and Objectives of this Thesis 3
Tothebestofourknowledge,nearlynoliteratureontheinequalityconstrainedGauss-Helmertmodel
exists. The works which come closest treat the mixed model—which can be seen as a generalization
of the Gauss-Helmert model. Here, the works of Famula (1983), Kato and Hoijtink (2006), Davis
et al. (2008) and Davis (2011) should be mentioned.
In Chap. 6 we describe two approaches to solve an inequality constrained Gauss-Helmert model.
One that uses standard solvers and one that does not but therefore takes advantage of the special
structure of the Gauss-Helmert model.
1.3 Scientific Context and Objectives of this Thesis
As implied by the title, this work is located at the transition area between mathematical op-
timization and adjustment theory. Both fields are concerned with the estimation of unknown
parameters in such a way that an objective function is minimized or maximized. As a consequence
the topics overlap to a large extent and sometimes differ merely in the terminology used.
As there exist many different definitions and as the assignment of a certain concept to one of the
two fields might sometimes be arbitrary, we state in the following how the terms will be used within
this thesis.
Adjustment theory. In adjustment theory (cf. Chap. 2) not only the parameter estimate but
also its accuracy is of interest. Thus, we have a functional as well as a stochastic model for
a specific problem as it is common in geodesy. This also includes testing theory and the
propagation of variances.
Furthermore, the term adjustment theory is often connected with three classic geodetic
models: the Gauss-Markov model, the adjustment with condition equations and the Gauss-
Helmert model. All of these models can be combined with an estimator that defines what
characterizes an optimal estimate. By far the most widely used estimator is the L norm
2
estimator which leads to the famous least-squares adjustment. Furthermore, it can be shown
to be the Best Linear Unbiased Estimator (BLUE) in the aforementioned models (cf. Koch,
1999, p. 156–158 and p. 214–216, respectively).
Mathematical optimization. In mathematical optimization (cf. Chap. 3) on the other hand,
mostly the estimate itself matters. Thus, we are usually dealing with deterministic quantities
only.
Furthermore, the optimization theory deals with methods to reformulate a certain problem
in a more convenient form and provides many different algorithms to solve it. In addition,
inequality constrained estimates are usually assigned to this field. While in adjustment theory
the focus lies on the problem and a way to model it mathematically, in optimization the focus
is more on general algorithmic developments leading to powerful methods for many tasks.
Somewordsmightbeinordertodistinguishthetopicofthisthesisfromrelatedworkinotherfields.
1. We are using frequentist instead of Bayesian inference. Thus, we assume that the unknown
parameters have a fixed but unknown value and are deterministic quantities (cf. Koch, 2007,
p.34).Thus,nostochasticpriorinformationontheparametersisused.However,inSect.4.2.3
we borrow the concept of Highest Posterior Density Regions, which originally stems from
Bayesian statistics but can also be applied in a frequentist framework.
4 1. Introduction
2. Recently,manyworkshavebeenpublisheddealingwiththeerrors-in-variablesmodelandthus
leading to a total least-squares estimate. Some of them even include inequality constraints
(De Moor, 1990, Zhang et al., 2013, Fang, 2014, Zeng et al., 2015). However, the errors-in-
variables model can be seen as a special case of the Gauss-Helmert model (cf. Koch, 2014).
Thus, we decided to treat the most general version of the Gauss-Helmert model in Chap. 6
instead of a special case.
3. Whenever we mention the term “inequalities” we are talking about constraints for the quanti-
tieswhichshouldbeestimated.Thisisabigdifferencetothefieldof “censoring” (cf.Kalbfleisch
and Prentice, 2002, p. 2) that deals with incomplete data. An example for an inequality in a
censored data problem would be an observation whose quantity is only known by a lower
bound as in “The house was sold for at least e500000.” As a consequence, it is not known
if the house was sold for e500000, e600000 or e1000000. Within this thesis we explicitely
exclude censored data.
Purpose of this Work. The purpose of this thesis it to make accessible the powerful theory
of convex optimization—especially the estimation with inequality constraints—for classic geodetic
tasks. Besides the extraction of certain analogies between convex optimization methods and ap-
proaches from adjustment theory, this is attempted by removing the three main obstacles identified
in Sect. 1.1 and thus includes
the extension of existing convex optimization algorithms by a stochastic framework.
•
a new strategy to treat rank-deficient problems with inequalities.
•
a formulation of a Gauss-Helmert model with inequality constraints as a quadratic program
•
in standard form.
The contents of this thesis have been partly published in the following articles.
Roese-Koerner, Devaraju, Sneeuw and Schuh (2012). A stochastic framework for inequal-
ity constrained estimation. Journal of Geodesy, 86(11):1005–1018, 2012.
Roese-Koerner and Schuh (2014). Convex optimization under inequality constraints in rank-
deficient systems. Journal of Geodesy, 88(5):415–426, 2014.
Roese-Koerner, Devaraju, Schuh and Sneeuw (2015). Describing the quality of inequality
constrained estimates. In Kutterer, Seitz, Alkhatib, and Schmidt, editors, Proceedings of the
1st International Workshop on the Quality of Geodetic Observation and Monitoring Systems
(QuGOMS’11), IAG Symp. 140, pages 15–20. Springer, Berlin, Heidelberg, 2015.
Roese-Koerner and Schuh (2015). Effects of different objective functions in inequality con-
strained and rank-deficient least-squares problems. In VIII Hotine-Marussi Symposium on
Mathematical Geodesy, IAG Symp. 142. Springer, Berlin, Heidelberg, 2015 (accepted).
Halsig, Roese-Koerner, Artz, Nothnagel and Schuh (2015). Improved parameter estima-
tionofzenithwetdelayusinganinequalityconstrainedleastsquaresmethod. InIAGScientific
Assembly, Potsdam 2013, IAG Symp. 143. Springer, Berlin, Heidelberg, 2015 (accepted)