NOTES ON OPTIMAL CONTROL THEORY with economic models and exercises Andrea Calogero

162 Pages·2017·1.32 MB·Italian

NOTES ON OPTIMAL CONTROL THEORY
with economic models and exercises
Andrea Calogero
Dipartimento di Matematica e Applicazioni – Universit`a di Milano-Bicocca
([email protected])
October 30, 2018
ii
Contents
1 Introduction to Optimal Control 1
1.1 Some examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Statement of problems of Optimal Control . . . . . . . . . . . . . 5
1.2.1 Admissible control and associated trajectory . . . . . . . 5
1.2.2 Optimal Control problems . . . . . . . . . . . . . . . . . . 9
2 The simplest problem of OC 11
2.1 The necessary condition of Pontryagin . . . . . . . . . . . . . . . 11
2.1.1 The proof in a particular situation . . . . . . . . . . . . . 16
2.2 Sufficient conditions . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.3 First generalizations . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.3.1 Initial/final conditions on the trajectory . . . . . . . . . . 22
2.3.2 On minimum problems. . . . . . . . . . . . . . . . . . . . 22
2.4 The case of Calculus of Variation . . . . . . . . . . . . . . . . . . 22
2.5 Examples and applications. . . . . . . . . . . . . . . . . . . . . . 25
2.5.1 The curve of minimal length . . . . . . . . . . . . . . . . 29
2.5.2 A problem of business strategy I . . . . . . . . . . . . . . 29
2.5.3 A two-sector model. . . . . . . . . . . . . . . . . . . . . . 31
2.5.4 A problem of inventory and production I. . . . . . . . . . 34
2.6 Singular and bang-bang controls . . . . . . . . . . . . . . . . . . 36
2.6.1 The building of a mountain road: a singular control . . . 37
2.7 The multiplier as shadow price I: an exercise . . . . . . . . . . . 40
3 General problems of OC 43
3.1 Problems of Bolza, of Mayer and of Lagrange . . . . . . . . . . . 43
3.2 Problems with fixed or free final time. . . . . . . . . . . . . . . . 44
3.2.1 Fixed final time. . . . . . . . . . . . . . . . . . . . . . . . 44
3.2.2 Free final time . . . . . . . . . . . . . . . . . . . . . . . . 46
3.2.3 The proof of the necessary condition . . . . . . . . . . . . 48
3.2.4 The moonlanding problem . . . . . . . . . . . . . . . . . . 50
3.3 The Bolza problem in Calculus of Variations. . . . . . . . . . . . 55
3.3.1 Labor adjustment model of Hamermesh. . . . . . . . . . . 57
3.4 Existence and controllability results. . . . . . . . . . . . . . . . . 58
3.4.1 Linear time optimal problems . . . . . . . . . . . . . . . . 62
3.5 Time optimal problem . . . . . . . . . . . . . . . . . . . . . . . . 64
3.5.1 The classical example of Pontryagin and its boat . . . . . 66
3.5.2 The Dubin car . . . . . . . . . . . . . . . . . . . . . . . . 69
3.6 Infinite horizon problems . . . . . . . . . . . . . . . . . . . . . . 73
iii
3.6.1 The model of Ramsey . . . . . . . . . . . . . . . . . . . . 76
3.7 Current Hamiltonian . . . . . . . . . . . . . . . . . . . . . . . . . 79
3.7.1 A model of optimal consumption with log–utility I . . . . 82
4 Constrained problems of OC 85
4.1 The general case . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2 Pure state constraints . . . . . . . . . . . . . . . . . . . . . . . . 89
4.2.1 Commodity trading . . . . . . . . . . . . . . . . . . . . . 92
4.3 Isoperimetric problems in CoV . . . . . . . . . . . . . . . . . . . 96
4.3.1 Necessary conditions with regular constraints . . . . . . . 97
4.3.2 The multiplier ν as shadow price . . . . . . . . . . . . . . 99
4.3.3 The foundation of Cartagena . . . . . . . . . . . . . . . . 100
4.3.4 The Hotelling model of socially optimal extraction . . . . 101
5 OC with dynamic programming 105
5.1 The value function: necessary conditions . . . . . . . . . . . . . . 105
5.1.1 The final condition on V: an first necessary condition . . 106
5.1.2 Bellman’s Principle of optimality . . . . . . . . . . . . . . 107
5.2 The Bellman-Hamilton-Jacobi equation . . . . . . . . . . . . . . 108
5.2.1 Necessary condition of optimality . . . . . . . . . . . . . . 108
5.2.2 Sufficient condition of optimality . . . . . . . . . . . . . . 111
5.2.3 Affine Quadratic problems . . . . . . . . . . . . . . . . . . 113
5.3 Regularity of V and viscosity solution . . . . . . . . . . . . . . . 115
5.3.1 Viscosity solution . . . . . . . . . . . . . . . . . . . . . . . 119
5.4 More general problems of OC . . . . . . . . . . . . . . . . . . . . 124
5.4.1 On minimum problems. . . . . . . . . . . . . . . . . . . . 125
5.5 Examples and applications. . . . . . . . . . . . . . . . . . . . . . 125
5.5.1 A problem of business strategy II . . . . . . . . . . . . . . 130
5.5.2 A problem of inventory and production II. . . . . . . . . . 133
5.6 The multiplier as shadow price II: the proof . . . . . . . . . . . . 138
5.7 Infinite horizon problems . . . . . . . . . . . . . . . . . . . . . . 140
5.7.1 Particular infinite horizion problems . . . . . . . . . . . . 143
5.7.2 A model of consumption with HARA–utility . . . . . . . 144
5.7.3 Stochastic consumption: the idea of Merton’s model . . . 146
5.7.4 A model of consumption with log–utility II . . . . . . . . 147
5.8 Problems with discounting and salvage value . . . . . . . . . . . 148
5.8.1 A problem of selecting investment . . . . . . . . . . . . . 150
iv
Chapter 1
Introduction to Optimal
Control
1.1 Some examples
Example 1.1.1. Thecurveofminimallengthandtheisoperimetricprob-
lem
Supposeweareinterestedtofindthecurveofminimallengthjoiningtwodistinct
points in the plane. Suppose that the two points are (0,0) and (a,b). Clearly
we can suppose that a = 1. Hence we are looking for a function x : [0,1] → R
such that x(0)=0 and x(1)=b.
The length of such curve is defined by x
(cid:82)1
ds, i.e. as the “sum” of arcs of in-
0
finitesimal length ds; using the picture x+dx
ds
and the Theorem of Pitagora we obtain x
b
(ds)2 =(dt)2+(dx)2
(cid:112)
⇒ ds= 1+x˙2dt,
t
where x˙ = dxd(tt). t t+dt 1
Hence the problem is
 (cid:90) 1
min (cid:112)1+x˙2(t)dt
x 0 (1.1)
x(0)=0
x(1)=b
It is well known that the solution is a line. We will solve this problem in
subsection 2.5.1.
Amorecomplicateproblemistofindtheclosedcurveintheplaneofassigned
length such that the area inside such curve is maximum: we call this problem
the foundation of Cartagena.1 This is the isoperimetric problem. Without loss
1WhenCartagenawasfounded,itwasgrantedforitsconstructionasmuchlandasaman
could circumscribe in one day with his plow: what form should have the groove because it
obtainsthemaximumpossibleland,beinggiventothelengthofthegroovethatcandigaman
inaday? Or,mathematicallyspeaking,whatistheshapewiththemaximumareaamongall
thefigureswiththesameperimeter?
1
2 CHAPTER 1. INTRODUCTION TO OPTIMAL CONTROL
of generality, we consider a curve x : [0,1] → R such that x(0) = x(1) = 0.
(cid:82)1
Clearly the area delimited by the curve and the t axis is given by x(t)dt.
0
Hence the problem is
 (cid:90) 1
mx(xa0x)=00x(t)dt
(1.2)
x(1)=0
(cid:90) 1(cid:112)1+x˙2(t)dt=A>1
0
Notethatthelengthoftheinterval[0,1]isexactly1and,clearly,itisreasonable
to require A>1. We will present the solution in subsection 4.3.3.
Example 1.1.2. A problem of business strategy
Afactoryproducesauniquegoodwitharatex(t),attimet. Ateverymoment,
such production can either be reinvested to expand the productive capacity
or sold. The initial productive capacity is α > 0; such capacity grows as the
reinvestment rate. Taking into account that the selling price is constant, what
fractionu(t)oftheoutputattimetshouldbereinvestedtomaximizetotalsales
over the fixed period [0,T]?
Let us introduce the function u:[0,T]→[0,1]; clearly, if u(t) is the fraction of
the output x(t) that we reinvest, (1−u(t))x(t) is the part of x(t) that we sell
at time t at the fixed price P >0. Hence the problem is
 (cid:90) T
mu∈aCx 0 (1−u(t))x(t)P dt
x˙ =ux (1.3)
Cx(=0){=uα:[,0,T]→[0,1]⊂R, u∈KC}
whereαandT arepositiveandfixed. Wewillpresentthesolutioninsubsection
2.5.2 and in subsection 5.5.1.
Example 1.1.3. The building of a mountain road
The altitude of a mountain is given by a differentiable function y, with y :
[t ,t ] → R. We have to construct a road: let us determinate the shape of
0 1
the road, i.e. the altitude x = x(t) of the road in [t ,t ], such that the slope
0 1
of the road never exceeds α, with α > 0, and such that the total cost of the
construction
(cid:90) t1
(x(t)−y(t))2dt
t0
is minimal. Clearly the problem is
 (cid:90) t1
min (x(t)−y(t))2dt
u∈C t0 (1.4)
x˙ =u
C ={u:[t ,t ]→[−α,α]⊂R, u∈KC}
0 1
where y is an assigned and continuous function. We will present the solution in
subsection 2.6.1 of this problem introduced in chapter IV in [22].
1.1. SOME EXAMPLES 3
Example 1.1.4. “In boat with Pontryagin”.
Suppose we are on a boat that at time t =0 has distance d >0 from the pier
0 0
oftheportandhasvelocityv inthedirectionoftheport. Theboatisequipped
0
with a motor that provides an acceleration or a deceleration. We are looking
for a strategy to arrive to the pier in the shortest time with a “soft docking”,
i.e. with vanishing speed in the final time T.
We denote by x=x(t) the distance from the pier at time t, by x˙ the velocity of
theboatandbyx¨=utheacceleration(x¨>0)ordeceleration(x¨<0).Inorder
to obtain a “soft docking”, we require x(T)=x˙(T)=0, where the final time T
is clearly unknown. We note that our strategy depends only on our choice, at
every time, on u(t). Hence the problem is the following
minT
xx¨u∈(=0C)u=d
0 (1.5)
x˙(0)=v
Cx(=T){=u:x˙0[(0T,∞)=)→0 [−1,1]⊂R}
where d and v are fixed and T is free.
0 0
This is one of the possible ways to introduce a classic example due to Pon-
tryagin; it shows the various and complex situations in the optimal control
problems (see page 23 in [22]). We will solve this problem in subsection 3.5.1.
(cid:52)
Example 1.1.5. A model of optimal consumption.
Consideraninvestorwho,attimet=0,isendowedwithaninitialcapitalx(0)=
x > 0. At any time he and his heirs decide about their rate of consumption
0
c(t)≥0. Thus the capital stock evolves according to
x˙ =rx−c
where r > 0 is a given and fixed rate to return. The investor’s time utility for
consumingatratec(t)isU(c(t)).Theinvestor’sproblemistofindaconsumption
plain so as to maximize his discounted utility
(cid:90) ∞
e−δtU(c(t))dt
0
where δ, with δ ≥r, is a given discount rate, subject to the solvency constraint
that the capital stock x(t) must be positive for all t≥0 and such that vanishes
at ∞. Then the problem is
 (cid:90) ∞
mx˙c∈a=Cxrx0−ec−δtU(c)dt
x(0)=x0 >0 (1.6)
Cxtl→im≥=∞x0{c(t:)[=0,∞0 )→[0,∞)}
with δ ≥r ≥0 fixed constants. We will solve this problem in subsections 3.7.1
and 5.7.4 for a logarithmic utility function, and in subsection 5.7.2 for a HARA
utility function. (cid:52)
4 CHAPTER 1. INTRODUCTION TO OPTIMAL CONTROL
One of the real problems that inspired and motivated the study of optimal
control problems is the next and so called “moonlanding problem”.
Example 1.1.6. The moonlanding problem.
Consider the problem of a spacecraft attempting to make a soft landing on the
moon using a minimum amount of fuel. To define a simplified version of this
problem, let m=m(t)≥0 denote the mass, h=h(t)≥0 and v =v(t) denote
the height and vertical velocity of the spacecraft above the moon, and u=u(t)
denote the thrust of the spacecraft’s engine. Hence in the initial time t = 0,
0
we have initial height and vertical velocity of the spacecraft as h(0) = h > 0
0
and v(0) = v < 0; in the final time T, equal to the first time the spacecraft
0
reaches the moon, we require h(T) = 0 and v(T) = 0. Such final time T is not
fixed. Clearly
h˙ =v.
LetM denotethemassofthespacecraftwithoutfuel,F theinitialamountof
fuel and g the gravitational acceleration of the moon. The equations of motion
of the spacecraft is
mv˙ =u−mg
wherem=M+candc(t)istheamountoffuelattimet.Letαbethemaximum
thrust attainable by the spacecraft’s engine (α > 0 and fixed): the thrust u,
0≤u(t)≤α, of the spacecraft’s engine is the control for the problem and is in
relation with the amount of fuel with
m˙ =c˙=−ku,
with k a positive constant.
h
spacecraft h0 u
v
0 spacecraft
mg
Moon Moon
On the left, the spacecraft at time t=0 and, on the right, the forces that act on it.
The problem is to land using a minimum amount of fuel:
(cid:0) (cid:1)
min m(0)−m(T) =M +F −maxm(T)
1.2. STATEMENT OF PROBLEMS OF OPTIMAL CONTROL 5
Let us summarize the problem
maxm(T)
hmm˙u˙∈=v˙=C=v−uk−u mg
(1.7)
h(0)=h , h(T)=0
0
vmm(((0t0)))=≥=v0M0,, +Fhv((tT))≥=00
C ={u:[0,T]→[0,α]}
where h , M, F, g, −v , k and α are positive and fixed constants; the final
0 0
time T is free. The solution for this problem is very hard; we will present it in
subsection 3.2.4. (cid:52)
1.2 Statement of problems of Optimal Control
1.2.1 Admissible control and associated trajectory
Let us consider a problem where the development of the system is given by a
function
x:[t ,t ]→Rn, with x=(x ,x ,...,x ),
0 1 1 2 n
withn≥1.Ateverytimet,thevaluex(t)describesoursystem. Wecallxstate
variable (or trajectory): thestate variableis at least a continuous function. We
suppose that the system has an initial condition, i.e.
x(t )=α, (1.8)
0
where α=(α ,α ,...,α )∈Rn. In many situation we require that the trajec-
1 2 n
tory satisfies x satisfies a final condition; in order to do that, let us introduce a
set T ⊂[t ,∞)×Rn called target set. In this case the final condition is
0
(t ,x(t ))∈T. (1.9)
1 1
Forexample, thefinalconditionx(t )=β withβ fixedinRn hasT ={(t ,β)}
1 1
astargetset; ifwehaveafixedfinaltimet andnoconditionsonthetrajectory
1
at such final time, then the target set is T ={t }×Rn.
1
Letussupposethatoursystemdependsonsomeparticularchoice(orstrat-
egy), at every time. Essentially we suppose that the strategy of our system is
given by a measurable2 function
u:[t ,t ]→U, with u=(u ,u ,...,u ),
0 1 1 2 k
where U is a fixed closed set in Rk called control set. We call such function
u control variable. However, it is reasonable is some situations and models to
2In many situations that follows we will restrict our attention to the class of piecewise
continuous functions (and replace “measurable” with “KC”); more precisely, we denote by
KC([t0,t1]) the space of piecewise continuous function u on [t0,t1], i.e. u is continuous in
[t0,t1] up to a finite number of points τ such that lim u(t) and lim u(t) exist and are
t→τ+ t→τ−
finite.
6 CHAPTER 1. INTRODUCTION TO OPTIMAL CONTROL
require that the admissible controls are in KC and not only measurable (this is
the point of view in the book [13]).
The fact that u determines the system is represented by the dynamics, i.e.
the relation
x˙(t)=g(t,x(t),u(t)), (1.10)
where g : [t ,t ]×Rn×Rk → Rn. From a mathematical point of view we are
0 1
interesting in solving the Ordinary Differential Equation (ODE) of the form

x˙ =g(t,x,u) in [t ,t ]
 0 1
x(t )=α (1.11)
0
(t ,x(t ))∈T
1 1
where u is an assigned function. In general, without assumption on g and u, it
isnotpossibletoguaranteethatthereexistsauniquesolutionfor(1.10)defined
in all the interval [t ,t ]; moreover, since the function t (cid:55)→ g(t,x,u(t)) can be
0 1
not regular, we have to be precise on the notion of “solution” of such ODE. In
the next pages we will give a more precise definition of solution x for (1.11).
Controllability
Let us give some examples that show the difficulty to associated a trajectory to
a control:
Example 1.2.1. Letusconsider
(cid:26) √
x˙ =2u x in[0,1]
x(0)=0
Provethatthefunctionu(t)=a,withapositiveconstant,isnotanadmissiblecontrolsince
thetwofunctionsx1(t)=0andx2(t)=a2t2 solvethepreviousODE.
Example 1.2.2. Letusconsider
(cid:26)x˙ =ux2 in[0,1]
x(0)=1
Provethatthefunctionu(t)=a,withaconstant,isanadmissiblecontrolifandonlyifa<1.
Provethatthetrajectoryassociatedtosuchcontrolisx(t)= 1 .
1−at
Example 1.2.3. Letusconsider
x˙ =ux in[0,2]
x(0)=1
x(2)=36

|u|≤1
Prove3 thatthesetofadmissiblecontrolisempty.
The problem to investigate the possibility to find admissible control for an
optimal controls problem is called controllability (see section 3.4). In order to
guarantee the solution of (1.11), the following well–known theorem is funda-
mental
Theorem 1.1. Let us consider G = G(t,x) : [t ,t ]×Rn → Rn and let G be
0 1
continuous and Lipschitz continuous with respect to x in an open set D ⊆Rn+1
with (t ,α)∈D ⊂R×Rn. Then, there exists a neighborhood I ⊂R of t such
0 0
that the ODE
(cid:26)
x˙ =G(t,x)
x(t )=α
0
3Notethat0≤x˙ =ux≤3xandx(0)=1imply0≤x(t)≤e3t.