Foundations of Optimization

202 Pages·1976·4.018 MB·English

by  M. S. Bazaraa, C. M. Shetty (auth.)

Lecture Notes in Economics and Mathematical Systems
(Vol. 1-15: Lecture Notes in Operations Research and Mathematical Economics, Vol. 16-59: Lecture
Notes in Operations Research and Mathematical Systems)
Vol. 1: H. Bühlmann, H. Loellel, E. Nievergelt, Einführung in die Vol. 30: H. Noltemeier, Sensitivitätsanalyse bei diskreten linearen
Theorie und Praxis der Entscheidung bei Unsicherheit. 2. Auflage, Optimierungsproblemen. VI, 102 Seiten. 1970.
IV, 125 Seiten. 1969. Vol. 31: M. Kühlmeyer, Die nichtzentrale t-Verteilung. 11, 106 Sei
Vol. 2: U. N. Bhat, A Study of the Oueueing Systems M/G/1 and ten. 1970.
GI/M/1. VIII, 78 pages. 1968. Vol. 32: F. Bartholomes und G. Hotz, Homomorphismen und Re
Vol. 3: A Strauss, An Introduction to Optimal Control Theory. duktionen linearer Sprachen. XII, 143 Seiten. 1970. DM 18,-
Out of pnnt Vol. 33: K. Hinderer, Foundations of Non-stationary Dynamlc Pro
Vol. 4: Branch and Bound: Eine Einführung. 2., geänderte Auflage. gramming with Discrete Time Parameter. VI, 160 pages. 1970.
Herausgegeben von F. Weinberg. VII, 174 Seiten. 1973.
Vol. 34: H. Störmer, Semi-Markoll-Prozesse mit endlich vielen
Vol. 5: L. P. Hyvännen, Information Theory for Systems Engineers. Zuständen. Theorie und Anwendungen. VII, 128 Seiten. 1970.
VII, 205 pages. 1968.
Vol. 35: F. Ferschl, Markovketten. VI, 168 Seiten. 1970.
Vol. 6: H. P. Künzi, O. Müller, E. Nievergelt, Einführungskursus in
die dynamische Programmierung. IV, 103 Seiten. 1968. Vol. 36: M. J. P. Magill, On a General Economic Theory of Motion.
VI, 95 pages. 1970.
Vol. 7: W. Popp, Einführung in die Theorie der Lagerhaltung. VI,
173 Seiten. 1968. Vol. 37: H. Müller-Merbach, On Round-Oll Errors in Linear Pro
gramming. V, 48 pages. 1970.
Vol. 8: J. Teghem, J. Loris-Teghem, J. P. Lambotte, Modeles
d'Attente M/Gll et GI/M/1 11 Arrivges et Services en Groupea. 111, Vol. 38: Statistische Methoden I. Herausgegeben von E. Walter.
53 pages. 1969. VIII, 338 Seiten. 1970.
Vol. 9: E. Schultze, Einführung in die mathematischen Grundlagen Vol. 39: Statistische Methoden 11. Herausgegeben von E. Walter.
der Informationstheorie. VI, 118 Seiten. 1969. IV, 157 Seiten. 1970.
Vol. 10: D. Hochstädter, Stochastische Lagerhaltungsmodelle. VI, Vol. 40: H. Drygas, The Coordinate-Free Approach to Gauss
269 Seiten. 1969. Markov Estimation. VIII, 113 pages. 1970.
Vol. 11/12: Mathematical Systems Theory and Economics. Edited Vol. 41: U. Ueing, Zwei Lösungsmefhoden für nichtkonvexe Pro
by H. W. Kuhn and G. P. Szegö. VIII, 111, 486 pages. 1989. grammierungsprobleme. IV, 92 Seiten. 1971.
Vol. 13: Heuristische Planungsmethoden. Herausgegeben von Vol. 42: A V. Balakrishnan, Introduction to Optimization Theory in
F. Weinberg und C. A Zehnder. 11, 93 Seiten. 1969. a Hilbert Space. IV, 153 pages. 1971.
Vol. 14: Computing Methods in Optimization Problems. V, 191 pages. Vol. 43: J. AMorales, Bayesian Fullinformation Structural Analy
1969. sis. VI, 154 pages. 1971.
Vol. 15: Economic Models, Estimation and Risk Programming: Vol. 44:· G. Feichtinger, Stochastische Modelle demographischer
Essays in Honor of Gerhard Tintner. Edited by K. A. Fox, G. V. L. Prozesse. IX, 404 Seiten. 1971.
Narasimham and J. K. Sengupta. VIII, 461 pages. 1969. Vol. 45: 'K. Wendler, Hauptaustauschschritte (Principal Pivoting).
Vol. 16: H. P. Künzi und W. Oettli, Nichtlineare Optimierung: 11, 64 Seiten. 1971.
Neuere Verfahren, Bibliographie. IV, 180 Seiten. 1969.
Vol. 46: -Co Soucher, Le90ns sur la theorie des automates ma
Vol. 17: H. Bauer und K. Neumann, Berechnung optimaler Steue thematiques. VIII, 193 pages. 1971.
rungen, Maximumprinzip und dynamische Optimierung. VIII, 188
Vol. 47: H. A Nour Eldin, Optimierung linearer Regelsysteme
Seiten. 1969.
mit quadratischer Zielfunktion. VIII, 163 Seiten. 1971.
Vol. 18: M. WoilI, Optimale Instandhaltungspolitiken in einfachen
Systemen. V, 143 Seiten. 1970. Vol. 48: M. Constam, FORTRAN für Anfänger. 2. Auflage. VI,
148 Seiten. 1973.
Vol. 19: L. P. Hyvärinen, Mathematical Modeling tor Industrial Pro
Vol. 49: Ch. Schneeweiß, Regelungstechnische stochastische
cesses. VI, 122 pages. 1970.
Optimierungsverfahren. XI, 254 Seiten. 1971.
Vol. 20: G. Uebe, Optimale Fahrpläne. IX, 161 Seiten. 1970.
Vol. 50: Unternehmensforschung Heute - Übersichtsvorträge der
Vol. 21: Th. M. Liebling, Graphentheorie in Planungs-und Touren Züricher Tagung von SVOR und DGU, September 1970. Heraus
problemen am Beispiel des städtischen Straßendienstes. IX, gegeben von M. Beckmann. IV, 133 Seiten. 1971.
118 Seiten. 1970.
Vol. 51: Digitale Simulation. Herausgegeben von K. Bauknecht
Vol. 22: W. Eichhorn, Theone der homogenen Produktionsfunk
und W. Nef. IV, 207 Seiten. 1971.
tion. VIII, 119 Seiten. 1970.
Vol. 52: Invariant Imbedding. Proceedings 1970. Edited by R. E.
Vol. 23: A Ghosal, Some Aspects of Oueueing and Storage
Bellman and E. D. Denman. IV, 148 pages. 1971.
Systems. IV, 93 pages. 1970.
Vol. 24: G. Feichtinger, Lernprozesse in stochastischen Automaten. Vol. 53: J. RosenmOlIer, Kooperative Spiele und Märkte. 111, 152
V, 66 Seiten. 1970. Seiten. 1971.
Vol. 25: R. Henn und O. Opit.z, Konsum-und Produktionstheorie I. Vol. 54: C. C. von Weizsäcker, Steady State Capital Theory. 111,
11, 124 Seiten. 1970. 102 pages. 1971.
Vol. 26: D. Hochstädter und G. Uebe, Ökonometrische Methoden. Vol. 55: P. A V. B. Swamy, Statisticallnference iQ Random Coef
XII, 250 Seiten. 1970. ficient Regression Models. VIII, 209 pages. 1971.
Vol. 27: I. H. Mufti, Computational Methods in Optimal Control Vol. 56: Mohamed A. EI-Hodiri, Constrained Extrema. Introduction
Problems. IV, 45 pages. 1970. to the Differentiable Gase with Economic Applications. 111, 130
Vol. 28: Theoretical Approaches to Non-Numerical Problem Sol pages.1971.
ving. Edited by R. B. Banerji and M. D. Mesarovic. VI, 466 pages. Vol. 57: E. Freund, Zeitvariable Mehrgrößensysteme. VIII,160 Sei
1970. ten.1971.
Vol. 29: S. E. Elmaghraby, Some Network Models in Management Vol. 58: P. B. Hagelschuer, Theorie der linearen Dekomposition.
Science. 111, 176 pages. 1970. VII, 191 Seiten. 1971.
contlnuetlon on pege195
Lectu re Notes
in Economics and
Mathematical Systems
Managing Editors: M. Beckmann and H. P. Künzi
Mathematical Programming
122
M. S. Bazaraa
C. M. Shetty
Foundations of Optimization
Springer-Verlag
Berlin . Heidelberg . NewYork 1976
Editorial Board
H. Albach . A. V. Balakrishnan . M.B eckmann (Managing Editor)
P. Dhrymes . J. Green ' W.H ildenbrand . W.K relle
H. P. Künzi (Managing Editor) . KR. itter' R. Sato . HS. chelbert
P. Schönfeld
Managing Editors
Prof. Dr. M. Beckmann Prof. Or. H. P. Künzi
Brown University Universität Zürich
Providence, RI 02912/USA 8090 Zürich/Schweiz
Authors
M. S. Bazaraa
C. M. Sehtty
Georgia Insttiute of T echnol09Y
School of nIdustrial and Systems Engineering
Allanta, GA 30332/USA
Bauraa, M S 19~J·
Foun:Iati<:nS of optUd.zation.
(Lecture notes in economiC5 an<! IIIIItheJNIt1caJ.
'ystems ; :l22)
Bibl1ography: p.
Inc:lu:;les index.
1. Mathefll4tiee.1 OptiJrdzation. 2. Honllnear
pI'OJrMmirlg. J. Dual1ty theory (Hathulatic.s)
:r. Shetty, C. M., 1929~ joint a\<thor.
U . Title. ur. series.
QAACl2.5.BJ9 519.7'6 76--657.
AMS sub;ect Carssifications(1970): 90C25, 9OC30, 52A20,
54C30, 54-01
ISBN 978-3-540·07680-3 ISBN 978-3-642~48294-6 (eBook)
001 10.1007/978·3-642-48294-6
This work is subject to copyright. AU rights are reserved, whether the whole
or part of the material is concerned, specifically those of translation, re·
printing, re·use of illustrations, broadcasting, reproduction by photocopying
machine or similar means, and storage in dala banks.
Under § 54 of the German Copyright Law where copies are made for other
than private use, a fee is payable 10 the pubHsher, the amounl of Ihe fee to
be determined by agreement with the publisher.
C by Springer' Verlag Berlin . Heidelberg 1976
PREFACE
Current1y there is a vast amount of literature on nonlinear programming in
finite dimensions. The pub1ications deal with convex analysis and severa1 aspects
of optimization. On the conditions of optima1ity they deal mainly with generaliza-
tions of known results to more general problems and also with less restrictive
assumptions. There are also more general results dealing with duality. There are
yet other important publications dealing with algorithmic deve10pment and their
applications. This book is intended for researchers in nonlinear programming, and
deals mainly with convex analysis, optimality conditions and duality in nonlinear
programming. It consolidates the classic results in this area and some of the recent
results.
The book has been divided into two parts. The first part gives a very compre-
hensive background material. Assuming a background of matrix algebra and a senior
level course in Analysis, the first part on convex analysis is self-contained, and
develops some important results needed for subsequent chapters. The second part
deals with optimality conditions and duality. The results are developed using
extensively the properties of cones discussed in the first part. This has facili-
tated derivations of optimality conditions for equality and inequality constrained
problems. Further, minimum-principle type conditions are derived under less
restrictive assumptions. We also discuss constraint qualifications and treat some
of the more general duality theory in nonlinear programming.
Atlanta, Georgia M.S. Bazaraa
December 1975 C.M. Shetty
TABLE OF CONTENTS
Part I: Convex Analysis
CHAPTER 1: LINEAR SUBSPACES AND AFFINE MANIFOLDS • • . 1
1.1 Linear Subspaces and Orthogonal Complements 1
1.2 Linear Independence and Dimensionality 4
1.3 Projection Theorem 7
1.4 Affine Manifolds • 11
CHAPTER 2: CONVEX SETS • • • . . . • 15
2.1 Convex Cones, Convex Sets and Convex Hills 15
2.2 Carath~odory Type Theorems .•..... 19
2.3 Relative Interior and Related Properties of Convex Sets 29
2.4 Support and Separation Theorems 40
CHAPl'ER 3: CONVEX CONES 54
3.1 Cones, Convex Cones and Polar Cones 54
3.2 Polyhedral Cone s • • . • 61
..
3.3 Cones Generated by Sets 66
3.4 Cone of Tangents .... 73
3.5 Cone of Attainable Directions, Cone of Feasible Directions
and Cone of Interior Directions 81
CHAPTER 4: CONVEX FUNCTIONS . . . . • . . • . • 88
4.1 Definitions and Preliminary Results 88
4.2 Continuity and Directional Differentiability of Convex
Functions 93
4.3 Differentiable Convex Functions 98
4.4 Some Examples of Convex Functions .101
4.5 Generalization of Convex Functions .108
Part II: Qptimality Conditions and Duality
CHAPTER 5: STATIONARY POINT OPTIMALITY CONDITIONS WITH DIFF.ERENTIABILITY .114
5.1 Inequality Constrained Problems .114
5.2 Inequality and Equality Constrained Problems .121
5.3 Optimality Criteria of the Minimum Principle Type .127
VI
CHAPTER 6: CONSTRAINT QUALIFICATIONS . • . . • • .133
6.1 Inequality Constrained Problems .133
6.2 Equality and Inequality Constrained Problems .144
6.3 Necessary and Sufficient Qualification . .151
CHAPTER 7: CONVEX PROGRAMMING WITHOUT DIFFERENTIABILITY .155
7.1 Saddle Point Optimality Criteria ... .155
7.2 Stationary Point Optimality Conditions .163
CHAPTER 8: IAGRANGIAN DUALITY . . . . . ...•.. .169
8.1 Definitions and Preliminary Results .169
8.2 The Strong Duality Theorem .172
CHAPTER 9: CONJUGATE DUALITY ..••. .177
9.1 Closure of a Function .177
9.2 Conjugate Functions .180
9.3 Main Duality Theorem .186
9.4 Nonlinear Programming via Conjugate Functions .188
SELECTED REFERENCES . . . . . . . . . . . . • . . . • . . . . . .192
CHAPTER 1
LINEAR SUESPACES AND AFFINE MANIFDLDS
The first three chapters of this book discuss the concept of linear subspaces
and some of its important subsets -- namely, affine manifolds, convex cones and sets.
The notion of convexity plays a dominant role in nonlinear programming and is
explored in depth in these chapters. Chapter 4 deals with convex and convex-like
functions. As we will see later, certain convex sets can be associated with each of
these functions. The important results of these chapters are used later to develop
optimality conditions for nonlinear programs. The chapters also do contain several
other related results which have been used elsewhere in the study of non linear
programs or, in the opinion of the authors, are likely to be useful in advanced work
in this area.
1.1 Linear Subspaces and Orthogonal Complements
Ik
1.1.1 Definition. Let xl,x2, ... ,xk be points in En. Then x AiXi where
i=l
Ai e El, i = 1,2, ... ,k is said to be a linear combination of xl,x2' ... ,xk. Further-
more if Ai > 0 (Ai;;: 0) for each i, then x is said to be a positive (nonnegative)
linear combination of xl,x2, ... ,xk. x is said to be semipositive combination of
xl,x2, ... ,xk if it is a nonzero nonnegative combination of them: If in addition
k
~ Ai = 1 then x is called a convex combination of xl,x2' ... ,xk.
i=l
The above heirarchy will be found useful in discussing linear subspaces, affine
manifolds, convex cones, and convex sets which are discussed in the first three
chapters.
1.1.2 Definition. Let L be a set in E. L is called a subspace (or linear sub
n
space) if xl,x2 e L imply that A1Xl + ~x2 e L for each Al' ~ e El.
Put differently, a set in En is a subspace if for any two points in the set, all
linear combinations of these points belong to the set. Of course, without any addi-
tional generality, we can state Definition 1.1.2 above in terms of any finite number
of points. The reader may verify this statement by a simple induction argument.
It is obvious that the origin i8 always a member cf any nenempty 8ubspace by
2
letting all A'S to be zero. Trivial examples of subspaces in E are the empty set ~,
n
the origin, and En itself. Examples of subspaces in the plane E2 are lines through
the origin.
Given two subspaces 11 and ~ we may be interested in finding the largest
possible subspace 1 contained in both 11 and ~ as well as the smallest possible sub
n
space l' which contains both 11 and 12. It can be immediately checked that 11 12
is a subspace and since i t is the largest set contained in both 11,and 12, i t follows
that 1 = 1111 12. On the other hand if there is a subspace which contains both 11
and 12, then by Definition 1.1.2 it contains their sum 11 + 12. Since the latter is
=
a subspace then l' 11 +~. From this discussion the following is obvious.
1.1.3 1emma. 1et 11 and ~ be two subspaces in En. Then the largest possible sub
n
space contained in both 11 and ~ is 11 12 and the smallest possible subspace con
taining both 11 and 12 is 11 + 12.
U
The reader may be tempted to believe that since 11 12 is the smallest set
containing both 11 and ~ then 11 + ~ in the above remark should be replaced by
11 U 12. However this cannot be done because 11 U ~ is not necessarily a subspace
even if 11 and 12 are. For example, if 11 and ~ are distinct lines through the
origin in E2 then their union is obviously not a subspace.
If we have two subspaces 11 and 12 then their sum 1 is given by 11 + 12. If in
addition we have 11 n 12 = CO} then 1 is called the direct sum of 11 and 12 and is
=
denoted by 1 11 <:!)~. This notion will be useful when we decompose En into the
direct sum of two orthogonal subspaces.
Given an arbitrary set S in En one can generate a subspace which is spanned by
S. This subspace is given by the following definition.
1.1.4 Definition. Let S be an arbitrary set in En• The subspace spanned (or
generated) by Sand denoted by 1(S) is the set of all linear combinations of points
k
I
in S, i.e., 1(S) = [x: x = AiXi, Xi e S, \ e El, k ., 1}. A set S is said to span
i=l
(generate) a subspace 1 if 1 = 1(S).
The reader can easily verify that L(S) is indeed a subspace according to Defini-
tion 1.1.2. Some examples of linear subspaces generated by sets are given below.
3
(i) S = [(x,y): 0 :s; x :s; 1, y = 01
1(S) = [(x,y): x e El, y = o}
(ii) S = [(x,y): 0 :s; x :s; 1, y = x + l}
1(S) = E2
(iii) S = [(1,0), (1,1)1
=
1(S) E2
(iv) By convention if S = ~ then 1(S) = ~.
The reader may note that the set S of the above definition may or may not con
sist of a finite number of points. We will show in Section 1.2 that any linear sub-
space can be generated bya finite number of points, i.e., given a subspace 1 we can
find a finite set S with 1(S) = 1.
Actually given a set S in En, the subspace 1(S) spanned by S has the following
interesting property. 1(S) is the smallest possible linear subspace which contains
S. This fact is obvious since any subspace which contains S will also contain all
linear combinations of S, namely 1(S). Hence the following lemma is immediate.
1.1.5 Lemma. 1(S) is the smallest subspace containing S.
Corollary.
(i) L(8) is the intersection of all subspaces containing S.
=
(ii) S is a subspace if and only if 1(8) 8.
The following lemma that can be easily verified gives the relationship between
the subspaces spanned by two sub sets, and also the relationship between the subspace
spanned by the intersection of two arbitrary sets and the intersection of the sub
spaces spanned by the individual sets.
1. 1. 6 1emma .
(i) If Sl and S2 are arbitrary sets in En with Sl C S2 then
1(Sl) c: 1(S2)·
n n
(11) 1(Sl S2) c: 1(Sl) l(S2)·
It may be noted that in the second part of the above lemma, the reverse inclu
sion does not hold in general. For example let Sl = (x,y): x,y ~ O} and
4
82 = [(x,y): x,y ~ o}. Therefore L(81) = L(82) = E2 whereas 81 n 82 = [(O,O)} and
n
hence L(81 82) = [(O,O)}.
Closely associated with a subspace L is another subspace called the orthogonal
complement of Land denoted by L,I.. The following is adefinition of LJ..
1.1., Definition. Let L be a subspace in En. The orthogonal complement of L,
denoted by LJ., is the set of vectors which are orthogonal (perpendicular) to each
°
point in L, i. e., LJ. = [y: (x,y) = for each x e: L}. By convention if L is the
empty set then LJ. is E •
n
It is obvious that Land L J. have no points in common other than the origin
°
because if we let x e: L n LJ. then = (x,x) = 11 x 112 which implies that x = 0.
1.2. Linear Independence and Dimensionality
1.2.1 Definition. The vectors al,a2, ..• ,ak in En are said to be linearly independ
k
I °
ent if Cl'iai = implies that Cl'l = Cl'2"" = Cl'k = 0. al,a2,··· ,ak are called
i=l
linearly dependent if they are not linearly independent.
From the above definition it is clear that al,a2, •.. ,ak are linearly dependent
if and only if some ai can be represented as a linear combination of the other
F
vectors. Letting Ai = [aj :j i} then the vectors al,a2, ... ,ak are linearly
dependent if and only if for some i we have a. e: L(A.), the linear subspace spanned
1 1
by Ai' The set is linearly independent if and only if for each i we have ai ~ L(Ai).
From the above definition, it is clear that the zero vector itself forms a
linearly dependent set. Hence any set of vectors containing the zero vector is
linearly dependent.
We will now consider the notion of bases of a subspace and the notion of a
dimension of a subspace.
1.2.2 Definition. Let L be a linear subspace and al,a2, ... ,ak be linearly independ
ent vectors in L. If L(al,a2, ... ,ak), the subspac~ spanned by al,a2, ... ,ak, is equal
to L then the set of vectors al,a2, ... ,ak is said to form a basis of L. The
dimension of L, denoted by dim L is then equal to k. By convention we will let
dim ~ = - 1, and dim [O} = 0.