Optimization Toolbox™ User's Guide - MathWorks

884 Pages·2016·6.6 MB·English

Optimization Toolbox™
User's Guide
R2017a
How to Contact MathWorks
Latest news: www.mathworks.com
Sales and services: www.mathworks.com/sales_and_services
User community: www.mathworks.com/matlabcentral
Technical support: www.mathworks.com/support/contact_us
Phone: [phone]
The MathWorks, Inc.
3 Apple Hill Drive
Natick, MA 01760-2098
Optimization Toolbox™ User's Guide
© COPYRIGHT 1990–2017 by The MathWorks, Inc.
The software described in this document is furnished under a license agreement. The software may be used
or copied only under the terms of the license agreement. No part of this manual may be photocopied or
reproduced in any form without prior written consent from The MathWorks, Inc.
FEDERAL ACQUISITION: This provision applies to all acquisitions of the Program and Documentation
by, for, or through the federal government of the United States. By accepting delivery of the Program
or Documentation, the government hereby agrees that this software or documentation qualifies as
commercial computer software or commercial computer software documentation as such terms are used
or defined in FAR 12.212, DFARS Part 227.72, and DFARS 252.227-7014. Accordingly, the terms and
conditions of this Agreement and only those rights specified in this Agreement, shall pertain to and
govern the use, modification, reproduction, release, performance, display, and disclosure of the Program
and Documentation by the federal government (or other entity acquiring for or through the federal
government) and shall supersede any conflicting contractual terms or conditions. If this License fails
to meet the government's needs or is inconsistent in any respect with federal procurement law, the
government agrees to return the Program and Documentation, unused, to The MathWorks, Inc.
Trademarks
MATLAB and Simulink are registered trademarks of The MathWorks, Inc. See
www.mathworks.com/trademarks for a list of additional trademarks. Other product or brand
names may be trademarks or registered trademarks of their respective holders.
Patents
MathWorks products are protected by one or more U.S. patents. Please see
www.mathworks.com/patents for more information.
Revision History
November 1990 First printing
®
December 1996 Second printing
For MATLAB 5
January 1999 Third printing For Version 2 (Release 11)
September 2000 Fourth printing For Version 2.1 (Release 12)
June 2001 Online only Revised for Version 2.1.1 (Release 12.1)
September 2003 Online only Revised for Version 2.3 (Release 13SP1)
June 2004 Fifth printing Revised for Version 3.0 (Release 14)
October 2004 Online only Revised for Version 3.0.1 (Release 14SP1)
March 2005 Online only Revised for Version 3.0.2 (Release 14SP2)
September 2005 Online only Revised for Version 3.0.3 (Release 14SP3)
March 2006 Online only Revised for Version 3.0.4 (Release 2006a)
September 2006 Sixth printing Revised for Version 3.1 (Release 2006b)
March 2007 Seventh printing Revised for Version 3.1.1 (Release 2007a)
September 2007 Eighth printing Revised for Version 3.1.2 (Release 2007b)
March 2008 Online only Revised for Version 4.0 (Release 2008a)
October 2008 Online only Revised for Version 4.1 (Release 2008b)
March 2009 Online only Revised for Version 4.2 (Release 2009a)
September 2009 Online only Revised for Version 4.3 (Release 2009b)
March 2010 Online only Revised for Version 5.0 (Release 2010a)
September 2010 Online only Revised for Version 5.1 (Release 2010b)
April 2011 Online only Revised for Version 6.0 (Release 2011a)
September 2011 Online only Revised for Version 6.1 (Release 2011b)
March 2012 Online only Revised for Version 6.2 (Release 2012a)
September 2012 Online only Revised for Version 6.2.1 (Release 2012b)
March 2013 Online only Revised for Version 6.3 (Release 2013a)
September 2013 Online only Revised for Version 6.4 (Release 2013b)
March 2014 Online only Revised for Version 7.0 (Release 2014a)
October 2014 Online only Revised for Version 7.1 (Release 2014b)
March 2015 Online only Revised for Version 7.2 (Release 2015a)
September 2015 Online only Revised for Version 7.3 (Release 2015b)
March 2016 Online only Revised for Version 7.4 (Release 2016a)
September 2016 Online only Revised for Version 7.5 (Release 2016b)
March 2017 Online only Revised for Version 7.6 (Release 2017a)
Contents
Acknowledgments

Getting Started
1
Optimization Toolbox Product Description . . . . . . . . . . . . . . 1-2
Key Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Solve a Constrained Nonlinear Problem . . . . . . . . . . . . . . . . 1-3
Problem Formulation: Rosenbrock's Function . . . . . . . . . . . . 1-3
Defining the Problem in Toolbox Syntax . . . . . . . . . . . . . . . . 1-4
Running the Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-5
Interpreting the Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-10
Set Up a Linear Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-12
Convert a Problem to Solver Form . . . . . . . . . . . . . . . . . . . 1-12
Model Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-12
Solution Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-13
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-20
Setting Up an Optimization
2
Optimization Theory Overview . . . . . . . . . . . . . . . . . . . . . . . . 2-3
Optimization Toolbox Solvers . . . . . . . . . . . . . . . . . . . . . . . . . 2-4
Optimization Decision Table . . . . . . . . . . . . . . . . . . . . . . . . . . 2-6
v
Choosing the Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
fmincon Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-8
fsolve Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-9
fminunc Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-10
Least Squares Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 2-11
Linear Programming Algorithms . . . . . . . . . . . . . . . . . . . . . 2-12
Quadratic Programming Algorithms . . . . . . . . . . . . . . . . . . 2-13
Large-Scale vs. Medium-Scale Algorithms . . . . . . . . . . . . . . 2-13
Potential Inaccuracy with Interior-Point Algorithms . . . . . . 2-14
Problems Handled by Optimization Toolbox Functions . . . 2-16
Complex Numbers in Optimization Toolbox Solvers . . . . . . 2-19
Types of Objective Functions . . . . . . . . . . . . . . . . . . . . . . . . . 2-21
Writing Scalar Objective Functions . . . . . . . . . . . . . . . . . . . 2-22
Function Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-22
Anonymous Function Objectives . . . . . . . . . . . . . . . . . . . . . 2-24
Including Gradients and Hessians . . . . . . . . . . . . . . . . . . . . 2-24
Writing Vector and Matrix Objective Functions . . . . . . . . . 2-34
Jacobians of Vector Functions . . . . . . . . . . . . . . . . . . . . . . . 2-34
Jacobians of Matrix Functions . . . . . . . . . . . . . . . . . . . . . . 2-35
Jacobians with Matrix-Valued Independent Variables . . . . . 2-36
Writing Objective Functions for Linear or Quadratic
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-38
Maximizing an Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-39
Matrix Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-40
Types of Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-41
Iterations Can Violate Constraints . . . . . . . . . . . . . . . . . . . . 2-42
Intermediate Iterations can Violate Constraints . . . . . . . . . 2-42
Algorithms That Satisfy Bound Constraints . . . . . . . . . . . . 2-42
Solvers and Algorithms That Can Violate Bound Constraints 2-42
Bound Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-44
vi Contents
Linear Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-45
Linear Inequality Constraints . . . . . . . . . . . . . . . . . . . . . . . 2-45
Linear Equality Constraints . . . . . . . . . . . . . . . . . . . . . . . . 2-46
Nonlinear Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-47
Including Gradients in Constraint Functions . . . . . . . . . . . 2-47
Anonymous Nonlinear Constraint Functions . . . . . . . . . . . . 2-48
Or Instead of And Constraints . . . . . . . . . . . . . . . . . . . . . . . . 2-51
How to Use All Types of Constraints . . . . . . . . . . . . . . . . . . 2-56
Objective and Nonlinear Constraints in the Same Function 2-58
Passing Extra Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-63
Extra Parameters, Fixed Variables, or Data . . . . . . . . . . . . 2-63
Anonymous Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-63
Nested Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-65
Global Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-65
What Are Options? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-67
Options in Common Use: Tuning and Troubleshooting . . . 2-68
Set and Change Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-69
Choose Between optimoptions and optimset . . . . . . . . . . . . 2-70
View Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-74
Tolerances and Stopping Criteria . . . . . . . . . . . . . . . . . . . . . 2-76
Checking Validity of Gradients or Jacobians . . . . . . . . . . . 2-79
Check Gradient or Jacobian in Objective Function . . . . . . . 2-79
How to Check Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . 2-79
Example: Checking Derivatives of Objective and Constraint
Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-80
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-87
vii
Examining Results
3
Current Point and Function Value . . . . . . . . . . . . . . . . . . . . . 3-2
Exit Flags and Exit Messages . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
Exit Flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
Exit Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-4
Enhanced Exit Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
Exit Message Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Iterations and Function Counts . . . . . . . . . . . . . . . . . . . . . . 3-10
First-Order Optimality Measure . . . . . . . . . . . . . . . . . . . . . . 3-11
What Is First-Order Optimality Measure? . . . . . . . . . . . . . . 3-11
Stopping Rules Related to First-Order Optimality . . . . . . . . 3-11
Unconstrained Optimality . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12
Constrained Optimality Theory . . . . . . . . . . . . . . . . . . . . . . 3-12
Constrained Optimality in Solver Form . . . . . . . . . . . . . . . . 3-14
Iterative Display . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-16
Common Headings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17
Function-Specific Headings . . . . . . . . . . . . . . . . . . . . . . . . . 3-17
Output Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-25
Lagrange Multiplier Structures . . . . . . . . . . . . . . . . . . . . . . . 3-26
Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-27
fminunc Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-27
fmincon Hessian . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-28
Plot Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-30
Plot an Optimization During Execution . . . . . . . . . . . . . . . . 3-30
Using a Plot Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-30
Output Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36
What Is an Output Function? . . . . . . . . . . . . . . . . . . . . . . . 3-36
Example: Using Output Functions . . . . . . . . . . . . . . . . . . . 3-36
viii Contents
Steps to Take After Running a Solver
4
Overview of Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
When the Solver Fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3
Too Many Iterations or Function Evaluations . . . . . . . . . . . . 4-3
Converged to an Infeasible Point . . . . . . . . . . . . . . . . . . . . . 4-7
Problem Unbounded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9
fsolve Could Not Solve Equation . . . . . . . . . . . . . . . . . . . . . 4-10
Solver Takes Too Long . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11
Enable Iterative Display . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11
Use Appropriate Tolerances . . . . . . . . . . . . . . . . . . . . . . . . 4-11
Use a Plot Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-12
Enable CheckGradients . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-12
Use Inf Instead of a Large, Arbitrary Bound . . . . . . . . . . . . 4-12
Use an Output Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13
Use a Sparse Solver or a Multiply Function . . . . . . . . . . . . 4-13
Use Parallel Computing . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13
When the Solver Might Have Succeeded . . . . . . . . . . . . . . . 4-14
Final Point Equals Initial Point . . . . . . . . . . . . . . . . . . . . . 4-14
Local Minimum Possible . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-14
When the Solver Succeeds . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-21
What Can Be Wrong If The Solver Succeeds? . . . . . . . . . . . 4-21
1. Change the Initial Point . . . . . . . . . . . . . . . . . . . . . . . . . 4-22
2. Check Nearby Points . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23
3. Check your Objective and Constraint Functions . . . . . . . 4-24
Local vs. Global Optima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26
Why Didn't the Solver Find the Smallest Minimum? . . . . . . 4-26
Searching for a Smaller Minimum . . . . . . . . . . . . . . . . . . . 4-27
Basins of Attraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-27
Optimizing a Simulation or Ordinary Differential
Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-31
What Is Optimizing a Simulation or ODE? . . . . . . . . . . . . . 4-31
Potential Problems and Solutions . . . . . . . . . . . . . . . . . . . . 4-31
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-36
ix
Optimization App
5
Optimization App . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Optimization App Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Specifying Certain Options . . . . . . . . . . . . . . . . . . . . . . . . . . 5-8
Importing and Exporting Your Work . . . . . . . . . . . . . . . . . . 5-11
Nonlinear algorithms and examples
6
Unconstrained Nonlinear Optimization Algorithms . . . . . . . 6-2
Unconstrained Optimization Definition . . . . . . . . . . . . . . . . . 6-2
fminunc trust-region Algorithm . . . . . . . . . . . . . . . . . . . 6-2
fminunc quasi-newton Algorithm . . . . . . . . . . . . . . . . . . . 6-5
fminsearch Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-11
fminunc Unconstrained Minimization . . . . . . . . . . . . . . . . . 6-13
Step 1: Write a file objfun.m. . . . . . . . . . . . . . . . . . . . . . . . 6-13
Step 2: Set options. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-13
Step 3: Invoke fminunc using the options. . . . . . . . . . . . . . . 6-13
Minimization with Gradient and Hessian . . . . . . . . . . . . . . 6-15
Step 1: Write a file brownfgh.m that computes the objective
function, the gradient of the objective, and the sparse
tridiagonal Hessian matrix. . . . . . . . . . . . . . . . . . . . . . . . 6-15
Step 2: Call a nonlinear minimization routine with a starting
point xstart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-15
Minimization with Gradient and Hessian Sparsity Pattern 6-17
Step 1: Write a file brownfg.m that computes the objective
function and the gradient of the objective. . . . . . . . . . . . . 6-17
Step 2: Call a nonlinear minimization routine with a starting
point xstart. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6-18
Constrained Nonlinear Optimization Algorithms . . . . . . . . 6-21
Constrained Optimization Definition . . . . . . . . . . . . . . . . . . 6-21
fmincon Trust Region Reflective Algorithm . . . . . . . . . . . . . 6-21
x Contents