A first course in optimization

313 Pages·2015·1.643 MB·English

by  Charles L Byrne

Mathematics
A First Course in Optimization teaches the basics of continuous optimiza- A
tion and helps readers better understand the mathematics from previous

courses. It gives readers the proper groundwork for future studies in opti- F
mization.
i
r
The book focuses on general problems and the underlying theory. It
s
introduces all the necessary mathematical tools and results. The text covers t

the fundamental problems of constrained and unconstrained optimization C
as well as linear and convex programming. It also presents basic iterative
o
solution algorithms (such as gradient methods and the Newton–Raphson
algorithm and its variants) and more general iterative optimization methods. u
r
Features
s
• Explains how to find exact and approximate solutions to systems of e
linear equations
i
• Shows how to use linear programming techniques, iterative methods, n

and specialized algorithms in various applications
O
• Discusses the importance of speeding up convergence
• Presents the necessary mathematical tools and results to provide the p
proper foundation t
i
• Prepares readers to understand how iterative optimization methods are
m
used in inverse problems
A First Course
i
This text builds the foundation to understand continuous optimization. It z
prepares readers to study advanced topics found in the author’s companion a
book, Iterative Optimization in Inverse Problems, including sequential un- t
i
constrained iterative optimization methods. o in
n
Optimization
B
y
r
n
e Charles L. Byrne
K22492
K22492_cover.indd 1 6/25/14 4:30 PM
A First Course
in
Optimization
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
A First Course
in
Optimization
Charles L. Byrne
University of Massachusetts Lowell
Lowell, USA
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2015 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Version Date: 20140703
International Standard Book Number-13: 978-1-4822-2658-4 (eBook - PDF)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the
validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the
copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to
publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let
us know so we may rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted,
or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, includ-
ing photocopying, microfilming, and recording, or in any information storage or retrieval system, without written
permission from the publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com
(http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers,
MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety
of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment
has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
To my wife Eileen,
with thanks for forty-four wonderful years of
marriage.
TThhiiss ppaaggee iinntteennttiioonnaallllyy lleefftt bbllaannkk
Contents
Preface xvii
Overview xxi
1 Optimization Without Calculus 1
1.1 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . 1
1.2 The Arithmetic Mean-Geometric Mean Inequality . . . . 2
1.3 Applying the AGM Inequality: the Number e . . . . . . . 2
1.4 Extending the AGM Inequality . . . . . . . . . . . . . . . 3
1.5 Optimization Using the AGM Inequality . . . . . . . . . 4
1.5.1 Example 1: Minimize This Sum . . . . . . . . . . 4
1.5.2 Example 2: Maximize This Product . . . . . . . 4
1.5.3 Example 3: A Harder Problem? . . . . . . . . . . 4
1.6 The H¨older and Minkowski Inequalities . . . . . . . . . . 5
1.6.1 H¨older’s Inequality . . . . . . . . . . . . . . . . . 5
1.6.2 Minkowski’s Inequality . . . . . . . . . . . . . . . 6
1.7 Cauchy’s Inequality . . . . . . . . . . . . . . . . . . . . . 6
1.8 Optimizing Using Cauchy’s Inequality . . . . . . . . . . . 8
1.8.1 Example 4: A Constrained Optimization . . . . . 8
1.8.2 Example 5: A Basic Estimation Problem . . . . . 9
1.8.3 Example 6: A Filtering Problem . . . . . . . . . 10
1.9 An Inner Product for Square Matrices . . . . . . . . . . . 11
1.10 Discrete Allocation Problems . . . . . . . . . . . . . . . . 13
1.11 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2 Geometric Programming 19
2.1 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . 19
2.2 An Example of a GP Problem . . . . . . . . . . . . . . . 19
2.3 Posynomials and the GP Problem . . . . . . . . . . . . . 20
2.4 The Dual GP Problem . . . . . . . . . . . . . . . . . . . 21
2.5 Solving the GP Problem . . . . . . . . . . . . . . . . . . 24
2.6 Solving the DGP Problem . . . . . . . . . . . . . . . . . . 24
2.6.1 The MART . . . . . . . . . . . . . . . . . . . . . 25
vii
viii Contents
2.6.2 MART I . . . . . . . . . . . . . . . . . . . . . . . 25
2.6.3 MART II . . . . . . . . . . . . . . . . . . . . . . 26
2.6.4 Using the MART to Solve the DGP Problem . . 26
2.7 Constrained Geometric Programming . . . . . . . . . . . 28
2.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3 Basic Analysis 31
3.1 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . 31
3.2 Minima and Infima . . . . . . . . . . . . . . . . . . . . . 31
3.3 Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.4 Completeness . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.5 Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.6 Limsup and Liminf . . . . . . . . . . . . . . . . . . . . . 36
3.7 Another View . . . . . . . . . . . . . . . . . . . . . . . . 38
3.8 Semi-Continuity . . . . . . . . . . . . . . . . . . . . . . . 39
3.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4 Convex Sets 41
4.1 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . 41
4.2 The Geometry of Real Euclidean Space . . . . . . . . . . 42
4.2.1 Inner Products . . . . . . . . . . . . . . . . . . . 42
4.2.2 Cauchy’s Inequality . . . . . . . . . . . . . . . . 43
4.2.3 Other Norms . . . . . . . . . . . . . . . . . . . . 43
4.3 A Bit of Topology . . . . . . . . . . . . . . . . . . . . . . 43
4.4 Convex Sets in RJ . . . . . . . . . . . . . . . . . . . . . . 45
4.4.1 Basic Definitions . . . . . . . . . . . . . . . . . . 45
4.4.2 Orthogonal Projection onto Convex Sets . . . . . 49
4.5 More on Projections . . . . . . . . . . . . . . . . . . . . . 52
4.6 Linear and Affine Operators on RJ . . . . . . . . . . . . . 53
4.7 The Fundamental Theorems . . . . . . . . . . . . . . . . 54
4.7.1 Basic Definitions . . . . . . . . . . . . . . . . . . 54
4.7.2 The Separation Theorem . . . . . . . . . . . . . 55
4.7.3 The Support Theorem . . . . . . . . . . . . . . . 55
4.8 Block-Matrix Notation . . . . . . . . . . . . . . . . . . . 57
4.9 Theorems of the Alternative . . . . . . . . . . . . . . . . 58
4.10 Another Proof of Farkas’ Lemma . . . . . . . . . . . . . . 62
4.11 Gordan’s Theorem Revisited . . . . . . . . . . . . . . . . 64
4.12 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Contents ix
5 Vector Spaces and Matrices 71
5.1 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . 71
5.2 Vector Spaces . . . . . . . . . . . . . . . . . . . . . . . . 71
5.3 Basic Linear Algebra . . . . . . . . . . . . . . . . . . . . 74
5.3.1 Bases and Dimension . . . . . . . . . . . . . . . . 74
5.3.2 The Rank of a Matrix . . . . . . . . . . . . . . . 75
5.3.3 The “Matrix Inversion Theorem” . . . . . . . . . 77
5.3.4 Systems of Linear Equations . . . . . . . . . . . 77
5.3.5 Real and Complex Systems of Linear Equations . 78
5.4 LU and QR Factorization . . . . . . . . . . . . . . . . . . 80
5.5 The LU Factorization . . . . . . . . . . . . . . . . . . . . 80
5.5.1 A Shortcut . . . . . . . . . . . . . . . . . . . . . 81
5.5.2 A Warning! . . . . . . . . . . . . . . . . . . . . . 82
5.5.3 The QR Factorization and Least Squares . . . . 85
5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
6 Linear Programming 87
6.1 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . 87
6.2 Primal and Dual Problems . . . . . . . . . . . . . . . . . 88
6.2.1 An Example. . . . . . . . . . . . . . . . . . . . . 88
6.2.2 Canonical and Standard Forms . . . . . . . . . . 89
6.2.3 From Canonical to Standard and Back . . . . . . 89
6.3 Converting a Problem to PS Form . . . . . . . . . . . . . 90
6.4 Duality Theorems . . . . . . . . . . . . . . . . . . . . . . 91
6.4.1 Weak Duality . . . . . . . . . . . . . . . . . . . . 91
6.4.2 Primal-Dual Methods . . . . . . . . . . . . . . . 92
6.4.3 Strong Duality . . . . . . . . . . . . . . . . . . . 92
6.5 A Basic Strong Duality Theorem . . . . . . . . . . . . . . 92
6.6 Another Proof . . . . . . . . . . . . . . . . . . . . . . . . 94
6.7 Proof of Gale’s Strong Duality Theorem . . . . . . . . . . 97
6.8 Some Examples . . . . . . . . . . . . . . . . . . . . . . . 99
6.8.1 The Diet Problem . . . . . . . . . . . . . . . . . 99
6.8.2 The Transport Problem . . . . . . . . . . . . . . 99
6.9 The Simplex Method . . . . . . . . . . . . . . . . . . . . 100
6.10 Yet Another Proof . . . . . . . . . . . . . . . . . . . . . . 102
6.11 The Sherman–Morrison–Woodbury Identity . . . . . . . . 102
6.12 An Example of the Simplex Method . . . . . . . . . . . . 103
6.13 Another Example . . . . . . . . . . . . . . . . . . . . . . 106
6.14 Some Possible Difficulties . . . . . . . . . . . . . . . . . . 107
6.14.1 A Third Example . . . . . . . . . . . . . . . . . . 108
6.15 Topics for Projects . . . . . . . . . . . . . . . . . . . . . . 109
6.16 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 109