Nonlinear Programming 3. Proceedings of the Special Interest Group on Mathematical Programming Symposium Conducted by the Computer Sciences Department at the University of Wisconsin–Madison, July 11–13, 1977

471 Pages·1978·18.253 MB·English

by  Olvi L. Mangasarian, etc

ACADEMIC PRESS RAPID MANUSCRIPT REPRODUCTION
Proceedings of the Special Interest Group on
Mathematical Programming Symposium
conducted by the Computer Sciences Department
at the University of Wisconsin—Madison
July 11-13, 1977
Nonlinear
Programming 3
Edited by OIvi L. Mangasarian
Robert R. Meyer
Stephen M. Robinson
Computer Sciences Department
University of Wisconsin —Madison
Madison, Wisconsin
Academic Press
New York San Francisco London 1978
A SUBSIDIARY OF HARCOURT BRACE JOVANOVICH, PUBLISHERS
COPYRIGHT © 1978, BY ACADEMIC PRESS, INC.
ALL RIGHTS RESERVED.
NO PART OF THIS PUBLICATION MAY BE REPRODUCED OR
TRANSMITTED IN ANY FORM OR BY ANY MEANS, ELECTRONIC
OR MECHANICAL, INCLUDING PHOTOCOPY, RECORDING, OR ANY
INFORMATION STORAGE AND RETRIEVAL SYSTEM, WITHOUT
PERMISSION IN WRITING FROM THE PUBLISHER.
ACADEMIC PRESS, INC.
Ill Fifth Avenue, New York, New York 10003
United Kingdom Edition published by
ACADEMIC PRESS, INC. (LONDON) LTD.
24/28 Oval Road, London NW1 7DX
Library of Congress Cataloging in Publication Data
Symposium on Nonlinear Programming, 3d, Madison, Wis.,
1977.
Nonlinear programming 3.
Includes index.
1. Nonlinear programming—Congresses.
I. Mangasarian, 01viL.,Date II. Meyer,
Robert R. III. Robinson, Stephen M. IV. Wisconsin.
University—Madison. Computer Sciences Dept.
V. Title.
T57.8.S9 1977 519.7'6 78-15926
ISBN 0-12-468660-3
PRINTED IN THE UNITED STATES OF AMERICA
CONTRIBUTORS
Numbers in parentheses indicate the pages on which the authors' contributions begin.
A. Auslender (429), Université de Clermont, Department de Mathématiques Ap­
pliquées, Boite Postale 45, 63170 Aubière, France
S. M. Chung (197), Computer Center, University of Hong Kong, Hong Kong
Richard W. Cottle (361), Department of Operations Research, Stanford University,
Stanford, California 94305
G. B. Dantzig (283), Department of Operations Research, Stanford University,
Stanford, California 94305
B. Curtis Eaves (391), Department of Operations Research, Stanford University,
Stanford, California 94305
Lidia Filus (407), Computing Center of the Polish Academy of Sciences, Warsaw,
Poland
David M. Gay (245), Massachusetts Institute of Technology, Center for Compu­
tational Research in Economics and Management Sciences, 575 Technology Square,
Cambridge, Massachusetts Q2139
Fred Glover (303), School of Business, University of Colorado, Boulder,
Colorado 80302
Mark S. Goheen (361), Department of Operations Research, Stanford University,
Stanford, California 94305
S. Gonzalez (337), 230 Ryon Building, Rice University, Houston, Texas 77001
Shih-Ping Han (65), Department of Mathematics, University of Illinois, Urbana,
Illinois 61801
Darwin Klingman (303), School of Business, University of Texas, Austin, Texas
78712
Javier Maguregui (461), Apartado 226, Caracas, Venezuela
vii
Vili CONTRIBUTORS
Garth P. McCormick (165), Institute for Management Science and Engineering,
George Washington University, 707 22nd Street, N.W., Washington, D.C. 20037
Claude McMillan (303), School of Business, University of Colorado, Boulder,
Colorado 80309
A. Miele (337), 230 Ryon Building, Rice University, Houston, Texas 77001
S. C. Parikh (283), Department of Operations Research, Stanford University, Stan­
ford, California 94305
M. J. D. Powell (27), Department of Applied Mathematics and Theoretical Physics,
University of Cambridge, Silver Street, Cambridge CB3 9EW, England
Klaus Ritter (225), Mathematisches Institut A, Universität Stuttgart, 7 Stuttgart 80,
Pfaffenwaldring 57, Federal Republic of Germany
J. B. Rosen (97), Department of Computer Science, 114 Lind Hall, University of
Minnesota, Minneapolis, Minnesota 55455
R. T. Rockafellar (1), Department of Mathematics GN-50, University of Washing­
ton, Seattle, Washington 98195
Robert B. Schnabel (245), Department of Computer Science, University of
Colorado, Boulder, Colorado 80309
R. A. Tapia (125), Department of Mathematical Sciences, Rice University,
Houston, Texas 77001
PREFACE
This volume contains 17 of the papers prepared for the Nonlinear Programming
Symposium 3 held at Madison on July 11-13, 1977. The first two Nonlinear Pro­
gramming Symposia were held in Madison in May 1970 and April 1974, and their
proceedings have also been published by Academic Press.
The first eight papers describe some of the most effective methods available
today for solving linearly and nonlinearly constrained optimization problems. The
ninth paper, which is contributed by Gay and Schnabel, gives algorithms for the
solution of nonlinear equations together with computational experience. Dantzig
and Parikh, and Glover, McMillan, and Klingman give some modern applications
of optimization in operations research. Miele and Gonzalez propose a measure­
ment procedure for optimization algorithm efficiency. Cottle and Goheen give
methods for solving large quadratic programs. Eaves and Filus describe algorithms
for solving stationary and fixed point problems. Auslender discusses the minimiza­
tion of certain types of nondifferentiable functions, while Maguregui discusses a
type of Newton method.
Once again it is hoped that this volume will bring to the scientific community
the latest ideas and contributions of some of the leading researchers in the field of
nonlinear programming.
The editors would like to thank the National Science Foundation for funding
this symposium under grant MCS76-24152, the Computer Sciences Department of
the University of Wisconsin—Madison for making its staff available for this
meeting, and the Wisconsin Center for providing its facilities. We would like to
thank Mrs. Gail Hackensmith for her important role as Symposium Secretary and
Mrs. Judy Swenson for her efficient and capable typing of this manuscript.
Nonlinear Programming 3
MONOTONE OPERATORS AND AUGMENTED
LAGRANGIAN METHODS IN NONLINEAR PROGRAMMING
R. T. Rockafellar1
ABSTRACT
The Hestenes-Powell method of multipliers in convex program­
ming is modified to obtain a superior global convergence property
under a stopping criterion that is easier to implement. The
convergence results are obtained from the theory of the proximal
point algorithm for solving OeT(z) when T is a maximal mono­
tone operator. An extension is made to an algorithm for solving
variational inequalities with explicit constraint functions.
Research sponsored by the Air Force Office of Scientific Research,
Air Force Systems Command, USAF, under grant number 77-3204 at
the University of Washington, Seattle.
Copyright © by Academic Press, Inc.
All rights of reproduction in any form reserved.
ISBN 0-12-468660-3
2 R.T. ROCKAFELLAR
1. INTRODUCTION
Let X be a nonempty, closed, convex subset of a Hilbert
(or Euclidean) space H , and let f. : H+R be a differentiable
convex function for i = 0,l,...,m . We shall be concerned with
the problem
(P) minimize ^ (χ) subject to
π
x e X, f .(x) <_ 0 for i = 1,. ..,m .
The ordinary dual of (P) is
m
(D) maximize inf (f (x) + V y.f.(x)}
A
X 0 .^ !
subject to 0 <_ y = (y ,.. . ,y ) e R
It will be assumed in what follows that (P) has at least one
optimal solution characterized by the Kuhn-Tucker conditions.
Then min (P) = max (D) of course, and the pairs (x,y) satis­
fying the Kuhn-Tucker conditions are precisely the ones such that
x is optimal for (P) and y is optimal for (D).
In recent years there has been much interest in computational
methods for (P) (and its nonconvex version) based on the augmented
Lagrangian, which is the expression L(x,y,c) defined for all
x e X, y e R , and parameter values c > 0 by
y.f.(x) +- f.(x)z if y. +cf.(x) >0
m Ji l 2 i Ji l —
L(x,y,c) = f (x) + I
Q 1 2
i=l - —2c Jyi . if J yl. + cfl .(x)— < 0
This is convex in x , concave in y , and continuously differ­
entiable in all arguments. Its saddle points (for arbitrary fixed
c) are the Kuhn-Tucker pairs (x,y) for (P) and (D). If each
f. happens to be continuously twice differentiable, then so is
L in all arguments, except on the hypersurfaces y. +cf.(x) = 0 .
Anyway, the first derivatives of L are everywhere Lipschitz
continuous with one-sided directional derivatives.
For more discussion of the properties of the augmented La­
grangian, see [1], [2]. The most recent survey of the "multiplier
methods" based on the augmented Lagrangian is that of Bertsekas
MONOTONE OPERATORS 3
[3]. Many extensions and modifications of the multiplier method
have been explored since it was originally suggested independently
by Hestenes and Powell in 1968. In essence, all are aimed at
replacing the constrained problem (P) by a sequence of uncon­
strained, or more simply constrained problems that can be solved
efficiently by the very powerful algorithms now known for that
special case. They resemble penalty methods in this respect,
but they generally are better behaved than penalty methods in
their rate of convergence and numerical stability (cf. [3]).
The present article, while concerned only with convex prob­
lems, will treat a new kind of modification which produces some
very favorable properties and also admits a generalization from
convex programming to the solution of variâtional inequalities
with explicit constraints. The following scheme will be called
the proximal multiplier method.
parameters: y>0, 0 < c, / c <_°°
initial guess: (x ,y )
2
^Fj/(x\)A _TΔ/ L(x,k y ,xc ) +u γ— I |xk-i2x | onn R
k+1 k
x « arg min F,(x)
xeX
y. = max {0,y. + c f.(x )} for i = l,...,m .
1 1 K 1
Note that the function F, which must be minimized over X
k
at each iteration is differentiable and convex, in fact strongly
2
convex with modulus μ /c,: for all x, x1 , one has
2
F (x' ) ^ F (x) + (x'-x) · VF (x) + ^- |x'-x|2 .
k k k
k+i °k
The sense in which x is an approximate minimizer depends
on the choice of stopping rule for the minimization step.
The usual multiplier method corresponds to μ = 0 (no auto­
matic strong convexity). The modified method was introduced in
[4] with y = 1 and shown to have two theoretical advantages
besides the strong convexity. The sequence {x } has better
properties, and global convergence can be obtained under a more
4 R.T. ROCKAFELLAR
easily implementable stopping rule in terms of the magnitude
k+1
of VF, (X ) . For the usual multiplier rule, the magnitude
k+1
of F, (x ) - inf F must be monitored if global convergence
(i.e. from any starting point) is to be ensured, although local
convergence (i.e. from a starting point "sufficiently" close to
being optimal) has been established by Polyak and Tretyakov [5]
k+1
and Bertsekas [6], [7], in terms of VF, (x ) in the case where
K
X = H = R and the strong second-order optimality conditions
are satisfied.
Numerical experiments have disclosed, however, that the
proximal multiplier method with μ = 1 moves rather slowly in
initial stages in comparison with the usual multiplier method,
despite its ultimate convergence properties. The reason for this
appears to be that, when c is too low, the quadratic term in
K
F, (x) dominates and does not allow the Lagrangian term to have
k+1
a strong enough effect in the selection of x .On the other
hand, when c is too high, the "penalty" aspects of the aug-
K
mented Lagrangian are too strong, and the prime advantage over
penalty methods gets lost.
The introduction here of the factor y restores flexibility
in allowing the role of the quadratic term to be damped while c,
is still reasonably low. The same type of convergence results
as for y > 0 will be demonstrated for arbitrarily small y > 0 ,
although the algorithm tends to resemble the usual multiplier
method more and more as y \ 0 .
The fact that the multiplier method can be approximated in
this sense by an algorithm possessing global convergence under
k+1
a rule involving VF, (x ) is interesting for applications to
the solution of variational inequalities. As explained below
in §5, these can be handled in the same theoretical framework,
essentially by replacing the gradient mapping Vf by a more
n
general "monotone" mapping. The minimization step equivalent
to finding an "approximate" solution to the equation VF, (x) = 0 ,
becomes a matter of solving a more general (but "nice") equation