Athena Scientific ABOUT THE AUTHOR Belmont, Mass. 02478 U.S.A. Email: info@athenasc.com Dimitri Bcrtse~ stu~ied '."1echanical and Electrical Engineering at the WWW: http://www.athenasc.com National Tcchmca! U111vers1ty of Athens, Greece, and obtained his Ph.D. in system scie11~e. from _the Massa~huse~ts Institute of Technology. He has held faculty pos1t10ns with the Engmeermg-Economic Systems Department Stanford University, and the Electrical Engineering Department of the Uni~ versity of Illinois, Urbana. Since 1979 he has been teaching at the Electrical Engineering and Computer Science Department of the Massachusetts In Cover image: The graph of the two-dimensional cost function of a simple stitute of Technology (M.I.T.), where he i6 currently the McAfee Professor neural network training problem. of Engineering. His teaching and research spans several fields, including determinis tic optimization, dynamic programming and stochastic control, large-scale and distributed computation, and data communication networks. He has authored or coauthored numerous research papers and sixteen books, sev eral of which are currently used as textbooks in MIT classes, including © 2016, 1999, 1995 Dimitri P. Bertseka.s "Dynamic Programming and Optimal Control," "Data Networks," "Intro All rights reserved. No part of this book may be reproduced in any form duction to Probability," "Convex Optimization Theory," "Convex Opti by any electronic or mechanical means (including photocopying, recording, mization Algorithms," as well as the present book. or information storage and retrieval) without permission in writing from Professor Bertsekas was a.warded the INFORMS 1997 Prize for Re the publisher. search Excellence in the Interface Between Operations Research and Corn p~ter Science for his book "Neuro-Dynamic Programming'' (co-authored with John Tsitsiklis), the 2001 AACC John R. Ragazzini Education Award, the 2009 INFORMS Expository Writing Award, the 2014 AACC Richard Bellman Heritage Award, the 2014 Khachiyan Prize for Life-Time Accom pli~hments in Optimization, and the MOS/SIAM 2015 George B. Dantzig PEriz~. In_ 2001, he was elected to the United States National Academy of ngmeenng. Publisher's Cataloging-in-Publication Data Bertsekas, Dimitri P. Nonlinear Programming: Second Edition Includes bibliographical references and index 1. Nonlinear Programming. 2. Mathematical Optimization. I. Title. T57.8.B47 2016 519.703 ISBN-10: 1-886529-05-1, ISBN-13: 978-1-886529-05-2 iii Nonlinear Programming THIRD EDITION Dimitri P. Bertsekas Massachusetts Institute of Technology www. site for book information and orders http:/ /www.athenasc.com th A ena Scientific B 1 ' e mont' Massachusetts cr:EN'fiflC A'ffil~:NA S UTA'fION SERIES :NP coJYlP A.'flON A oPTINilZ ' ])i111it.ri P. [3ertsdmH, Contents .. '\·cl EJiL1011, by . ' prop;ru,utr n,ng,. , · l8 80 pngns ,. l ' NzoolOllh, nJ . I Ot• llll .11 · · JS --. Prognunn11ng, iy 1. Unconstrained Optimization: Basic Methods . p. 1 ·t Dvnannc 2"6 nip;n; 3. A[Sbl3s _1;Nr:c19c78 -i-88Gfi29-42-7' ,> 0I . t' l.i lt l ,v O, II t, I ·t,1 ' T.w o-Volnme Set, 1.1. Optimality Conditiom1 . . . . p. 5 . -.. Progn1111111i11p; n11d \l>S,1~N l-88oG2D-OH-n, l020 pages l.J.J. Vnrif,l,tional Ideas ... .. . p. 5 ,1. D'·yy nDa1irun1ictr i I'. 13c1. ts,,e} m,; , 20 l 2, • D' 'tri p Bertsckw,, ,2 0(bn , l.1.2. Main Optimality Condil:ions . p, 15 v .· nizat,ion rf'}w m·y.' by· 11111 , · 1..2. Gradient Met;hodl:! -- Convergence p. 28 l ri. Convr.x Optll .;., c , -1 2fiG pages l.2. l. Descent Directions and Stepsizc Rulc1:1 p. 28 1Sl3N 078-l-88fli!20-.ll ' . . l Diinitri P. Bert,sekas l l Tty 2nd Echtion, .>.Y , ~ 11 . l.2.2. Convergence Results . . . . . . . . p, 49 G. !11Lroductio11 to Pro)_1i)1 l ' JSJ3N !)78-l-H8GG2!J-23-b, ,J'' pagcH 1.3. Grnclicnt Methods - Rate of Convergence p. 67 Tsils1khs 2008, •- all(\ John N' . " ' . ' l D' nitri r. BNl.sckn;;, An- ] .:3. 1. The Local Analysis Approach . . p. 69 7 Convex Analysiti a11c l Q pt irn1z'· tt1onI, .1.Y 2 001'13 T· SBN l-886G2, ( ) _,i ru: · o, 1.3.2. The Role of the Condition Number p. 70 . gelin. Nudi6, ancl Asuruall E. Oz ' 1.4.2. 1:'l:ust Region Methods p. 103 . . ., . I , Il. Tyrrell Rock- 1.4.3. Variants of Newton's Method p. 105 9. Network Flows and Monotrop1c Optmnzat10n, >.) - - afcllar, 1998, ISBN 1-886529-06-X, 5:34 pages 1.4.4. Lcust Squares and the Gauss-Newton Method p. 107 1.5. Notes and Sources . . . . . . . . . . . . . p. 117 . . . D' · t _. , Bc,rtsimo.s and lO. Introduction to Linear Optnmiation, by mu ,ns ' .JohnN. Tsitsiklis, 1997, ISBN l-886G29-19-l, 608 pages 2. Unconstrained Optimization: Additional Methods p. 119 11. Pamllel and Dist.ribnted Computation: N11mrrical Method~~;: 2.1. Conjngnte Direction MethoclH ..... .. ... . p. 120 6 Dimitri P. Bertsekas and John N. Tsitsiklis, 1997, ISBN 1-8S u · 2.1. l. The Conjugate Gradient Method . . . . . . . . p. 125 Ol-9, 7"18 pages 2.1.2. Convergence Rate of Conjugate Gradient l\tlethod p. 132 12· Ncuro-Dyuamic Programming, by Dimitri P. [3ertsdms and .John 2.2. Qua,si-Ncwton Methods . . . . . . . . . . . p. 138 N. Tsitsiklis, 1096, ISBN l-88GG29-10-8, 512 piiges 2.3. Nonclcrivative Methods p. 148 2.3.l. Coordinate Descent . µ. 149 t3. C?n~tr~ined Optimization and Lagrange Multiplier Methods, by 2.3.2. Direct Search Methods p. 154 Dmntn P. Bcrtsekas, 1996, ISBN l-886529-04-:3, 410 pages 2 .4. Incremental Methods . . . . . . p. 158 14. Spt o8ch astic Opt'r1i 1·, il Co ntrol: 'f he D.1 screte-T.n ne Caso, 11 y D1'1n, itri 22.4.1. Incremental Gradient Methods . . . . . p. 161 : crtsekas and Steven E. Shreve 11)\J6 ISI3N l-88652D-03-5, ,4.2. Incremental Aggregated Gradient Methods p. 172 330 pages ' ' 2,4.3. Incremental Gauss-Newton Methods . . p. 178 2 ') -4.3. Incremental Newton Methods ... p. 185 5 ~- . Dititributed Asynchronous Algorithms p. 194 iv V Contents vii Contents vi 2.5.1 Tot.ally iind Part,i,dly Asyuchronous Algorithms p. 197 4.3.2. Suffkient Conditions and Sensitivity . . . . . p. 383 p. 198 4.3.3. Fritz .John Optimality Conditions . . . . . p. 386 2.5.2. Totally Asynehronous Convcrgenee_. . . . . . . . 2.5 :l. Partially Asyncl1ronow; GradieJ1t-L1kc Al~onLhms p. 203 4.3.4. Constraint Qualifications and Pseudonormality p. 392 p. 204 4.3.5. Abstract Set Constraints and tho Tangent Cone p. 399 2.5.4. Convergmice IlaLc of Asynehro11011s Algonthrm, p. 210 4.3.6. Abstract Set Constraints, Equality, and Inequality 2.6. Discrete-Time Optimal Control Problerrn; • • • • Constraints . . . . . . . . . . . . . . . p. 415 2.6.l. Gradient and Conjugate Gradient Methods for p. 221 4.4. Linear Constraints and Duality . . . . . . . . p. 429 Optimal Control . . . . • • • • · · · · · p. 222 4.4.1. Convex Cost Function and Linear Constraints p. 429 2.6.2. Newton's Method for Optimal Control . . • 4.4.2. Duality Theory: A Simple Form for Linear 2.7. Solving Nonlinear Programming Problems - Some p. 227 Constraints . . p. 432 Practical Guidelines p. 232 4.5. Notes and Sources p. 441 2.8. Notes and Sources p. 235 5. Lagrange Multiplier Algorithms p. 445 3. Optimization Over a Convex Set p. 236 5.1. Barrier and Interior Point Methods . p. 446 3.1. Constrained Optimization Problems p. 236 5.1.1. Path Following Methods for Linear Programming p. 450 3.1.l. Necessary and Sufficient Conditions for Optimality p. 246 5.1.2. Primal-Dual Methods for Linear Programming . p. 458 3.1.2. Existence of Optimal Solutions . . . . . . . p. 257 5.2. Penalty and Augmented Lagrangian Methods . p. 469 3.2. Feasible Directions - Conditional Gradient Method p. 257 5.2.1. The Quadratic Penalty Function Method . p. 471 3.2.1. Descent Directions and Stepsize Rules p. 262 5.2.2. Multiplier Methods - Main Ideas . . . . . p. 479 3.2.2. The Conditional Gradient Method . . . . . . p. 272 5.2.3. Convergence Analysis of Multiplier Methods . p. 488 3.3. Gradient Projection Methods . . . . . . . . . . 5.2.4. Duality and Second Order Multiplier Methods . . . . . p. 492 3.3.1. Feasible Directions and Stepsize Rules Based on p. 272 5.2.5. Nonquadratic Augmented Lagrangians - The Exponential Projection p. 283 Method of Multipliers p. 494 3.3.2. Convergence Analysis . . . . . p. 292 5.3. Exact Penalties - Sequentiai Qu~d;at.ic ·P;o~r~m~i~~ · · · p. 502 3.4. Two-Metric Projection Methods p. 298 5.3.1. Nondifferentiable Exact Penalty Functions p. 503 3.5. Manifold Suboptimization Methods p. 307 5.3.2. Sequential Quadratic Programming .. p. 513 3.6. Proximal Algorithms . . . . . . p. 312 5.3.3. Differentiable Exact Penalty Functions p. 520 3.6.1. Rate of Convergence . . . . . p. 318 5.4. Lagrangian Methods . . . . . . . . . p. 527 3.6.2. Variants of the Proximal Algorithm p. 323 5.4.1. First-Order Lagrangian Methods ... .. . p. 528 3.7. Block Coordinate Descent Methods 3.7.1. Variants of Coordinate Descent p. 327 5.4.2. Newton-Like Methods for Equality Constraints· p. 535 3.8. Network Optimization Algorithms . p. 331 5.4.3. Global Convergence . . . . . . p. 545 p. 338 5.4.4. A Comparison of Various Methods p. 548 3.9. Notes and Sources 5.5. Notes and Sources . . . . . . p. 550 p. 343 4. Lagrange Multiplier Theory 6. Duality and Convex Programming p. 553 4.1. Necessary Conditions for Equality Constraints p. 345 4.1.1. The Penalty Approach . . p. 349 5.1. Duality and Dual Problems . p. 554 4.1.2. The Elimination Approach p. 352 5.1.1. Geometric Multipliers . . p. 556 4.1.3. The Lagrangian Function . p. 356 6-1.2. The Weak Duality Theorem . . . . p. 561 4.2. Sufficient Conditions and Sensitivity Analysis . p. 364 6.1.3. Primal and Dual Optimal Solutions p. 566 4.2.l. The Augmented Lagrangian Approach p. 365 5.1.4. Treatment of Equality Constraints . . . p. 568 4.2.2. The Feasible Direction Approach . p. 369 5.1.5. Separable Problems and their Geometry p. 570 4.2.3. Sensitivity . . . . . . . . . . p. 370 6-1.6. Additional Issues About Duality p. 575 4.3. Inequality Constraints . . p. 376 6·2· Convex Cost - Linear Constraints . p. 582 4.3.1. Karush-Kuhn-Tucker Necessary Conditions p. 378 6·3· Convex Cost - Convex Constraints p. 589 -- ix Contents Contents viii A.6. Convergence Theorems p. 770 p. 598 G.4. Cor~jugate Functions aJHl Fenchel Duality 6.4.1. Conic Prograrnming p. 604 Appendix B: Convex Analysis p. 783 p. 612 6.4.2. Monotropic Programming . . . . p. 617 B.l. Convex Sets and Functions p. 783 6.4.3. Network Optimization . . . . . p. 620 B.2. Hyperplanes . . . . . . . . p. 793 6.4.1. Games and the Minimax Theorem p. 623 B.3. Cones and Polyhedral Convexity p. 796 6.4.5. The Primal F\.mction and Sensitivity Analysis p. 630 B.4. Extreme Points and Linear Programming p. 798 6.5. Discrete Optimization and Duality . . . . . . p. 631 B.5. Differentiability Issues . . . . . . p. 803 6.5.1. Examples of Discrete Optimization Problems p. 639 6.5.2. Branch-and-Bound . . Appendix C: Line Search Methods p. 809 p. 648 6.5.3. Lagrangian Relaxation p. 660 :i C.l. Cubic Interpolation . . . . p. 809 6.6. Notes and Sources C.2. Quadratic Interpolation . . p. 810 p. 663 C.3. The Golden Section Method p. 812 II 7. Dual Methods ,',• ii p. 666 ; 'I 7.1. Dual Derivatives and Subgradients Appendix D: Implementation of Newton's Method p. 815 p. 673 7.2. Dual Ascent Methods for Differentiable Dual Problems p. 673 I~ D.l. Cholesky Factorization p. 815 !: 7.2.l. Coordinate Ascent for Quadratic Programming p. 675 D.2. Application to a Modified Ne;t~n .M.ethod : p. 817 7.2.2. Separable Problems and Primal Strict Convexity . "' I.1' p. 677 7.2.3. Partitioning and Dual Strict Concavity . . p. 682 References p. 821 7.3. Proximal and Augmented Lagrangian Methods 7.3.1. The Method of Multipliers as a Dual . p. 682 Index p. 857 Proximal Algorithm 7.3.2. Entropy Minimization and Exponential p. 686 Method of Multipliers . . . . . p. 687 7.3.3. Incremental Augmented Lagrangian Methods p. 691 7.4. Alternating Direction Methods of Multipliers . . p. 699 7.4.l. ADMiVI Applied to Separable Problems . . . 7.4.2. Connections Between Augmented Lagrangian p. 703 Related Methods . . . . . . . . . . p. 709 7.5. Subgradient-Based Optimization Methods . . . p. 709 7.5.1. Subgradient Methods . . . . . . . . . . p. 714 7.5.2. Approximate and Incremental Subgradient Methods p. 717 7.5.3. Cutting Plane Methods . . . . . . . . p. 724 7.5.4. Ascent and Approximate Ascent Methods . . . . . p. 735 7.6. Decomposition Methods . . . . . . . . . . . . p. 736 7.6.1. Lagrangian Relaxation of the Coupling Constraints . p. 739 7.6.2. Decomposition by Right-Hand Side Allocation p. 742 7.7. Notes and Sources . . . . . . . . . p. 745 Appendix A: Mathematical Background p. 746 A.l. Vectors and Matrices . . . . . . . . . p. 749 A.2. Norms, Sequences, Limits, and Continuity p. 757 A.3. Square Matrices and Eigenvalues . . . . p. 760 A.4. Symmetric and Positive Definite Matrices p. 765 A.5. Derivatives . . . . . . . . . . - Prefa ce to the First Edition I ~ I I Nouliucnr programming is u, llUJ,ttll'nt still importnnL, i:-1 the increa.,wd c111phasis ou l1tr(.!;c-sr:1:tlP pro\Jlmrni, and the nsi;odated algorithm;; l;ha,t take advo.iitnge of problem i;t;ructurn u.9 wnll HH pnrnJld !11.trdw,tt'

Nonlinear Programming

Similar Nonlinear Programming