Stochastic control theory. Dynamic programming principle

263 Pages·2015·1.362 MB·English

by  Makiko Nisio

Probability Theory and Stochastic Modelling 72
Makiko Nisio
Stochastic
Control
Theory
Dynamic Programming Principle
Second Edition
Probability Theory and Stochastic Modelling
Volume 72
Editors-in-Chief
SørenAsmussen,Aarhus,Denmark
PeterW.Glynn,Stanford,CA,USA
ThomasKurtz,Madison,WI,USA
YvesLeJan,Paris,France
AdvisoryBoard
JoeGani,Canberra,Australia
MartinHairer,Coventry,UK
PeterJagers,Gothenburg,Sweden
IoannisKaratzas,NewYork,NY,USA
FrankP.Kelly,Cambridge,UK
AndreasE.Kyprianou,Bath,UK
BerntØksendal,Oslo,Norway
GeorgePapanicolaou,Stanford,CA,USA
EtiennePardoux,Marseille,France
EdwinPerkins,Vancouver,Canada
HalilMeteSoner,Zurich,Switzerland
The Stochastic Modelling and Probability Theory series is a merger and con-
tinuation of Springer’s two well established series Stochastic Modelling and
Applied Probability and Probability and Its Applications series. It publishes
research monographsthat make a significant contribution to probability theory or
an applications domain in which advanced probability methods are fundamental.
Booksinthisseriesareexpectedtofollowrigorousmathematicalstandards,while
alsodisplayingtheexpositoryqualitynecessarytomakethemusefulandaccessible
toadvancedstudentsaswellasresearchers.Theseriescoversallaspectsofmodern
probabilitytheoryincluding
(cid:129) Gaussianprocesses
(cid:129) Markovprocesses
(cid:129) RandomFields,pointprocessesandrandomsets
(cid:129) Randommatrices
(cid:129) Statisticalmechanicsandrandommedia
(cid:129) Stochasticanalysis
aswellasapplicationsthatinclude(butarenotrestrictedto):
(cid:129) Branchingprocessesandothermodelsofpopulationgrowth
(cid:129) Communicationsandprocessingnetworks
(cid:129) Computational methods in probability and stochastic processes, including
simulation
(cid:129) Geneticsandotherstochasticmodelsinbiologyandthelifesciences
(cid:129) Informationtheory,signalprocessing,andimagesynthesis
(cid:129) Mathematicaleconomicsandfinance
(cid:129) Statisticalmethods(e.g.empiricalprocesses,MCMC)
(cid:129) Statisticsforstochasticprocesses
(cid:129) Stochasticcontrol
(cid:129) Stochasticmodelsinoperationsresearchandstochasticoptimization
(cid:129) Stochasticmodelsinthephysicalsciences
Moreinformationaboutthisseriesathttp://www.springer.com/series/13205
Makiko Nisio
Stochastic Control Theory
Dynamic Programming Principle
123
MakikoNisio(emeritus)
KobeUniversity
Kobe,Japan
OsakaElectro–CommunicationUniversity
Osaka,Japan
FirsteditionpublishedintheseriesISILectureNotes,No9,byMacMillanIndiaLimitedpublishers,
Delhi,(cid:2)c MakikoNisio,1981
ISSN2199-3130 ISSN2199-3149(electronic)
ISBN978-4-431-55122-5 ISBN978-4-431-55123-2(eBook)
DOI10.1007/978-4-431-55123-2
SpringerTokyoHeidelbergNewYorkDordrechtLondon
LibraryofCongressControlNumber:2014953914
MathematicsSubjectClassification:93E20,60H15
©SpringerJapan2015
Thisworkissubjecttocopyright.AllrightsarereservedbythePublisher,whetherthewholeorpartof
thematerialisconcerned,specificallytherightsoftranslation,reprinting,reuseofillustrations,recitation,
broadcasting,reproductiononmicrofilmsorinanyotherphysicalway,andtransmissionorinformation
storageandretrieval,electronicadaptation,computersoftware,orbysimilarordissimilarmethodology
nowknownorhereafterdeveloped.Exemptedfromthislegalreservationarebriefexcerptsinconnection
with reviews or scholarly analysis or material supplied specifically for the purpose of being entered
and executed on a computer system, for exclusive use by the purchaser of the work. Duplication of
this publication or parts thereof is permitted only under the provisions of the Copyright Law of the
Publisher’slocation,initscurrentversion,andpermissionforusemustalwaysbeobtainedfromSpringer.
PermissionsforusemaybeobtainedthroughRightsLinkattheCopyrightClearanceCenter.Violations
areliabletoprosecutionundertherespectiveCopyrightLaw.
Theuseofgeneraldescriptivenames,registerednames,trademarks,servicemarks,etc.inthispublication
doesnotimply,evenintheabsenceofaspecificstatement,thatsuchnamesareexemptfromtherelevant
protectivelawsandregulationsandthereforefreeforgeneraluse.
While the advice and information in this book are believed to be true and accurate at the date of
publication,neithertheauthorsnortheeditorsnorthepublishercanacceptanylegalresponsibilityfor
anyerrorsoromissionsthatmaybemade.Thepublishermakesnowarranty,expressorimplied,with
respecttothematerialcontainedherein.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Thepurposeofthisbookistoprovideanintroductiontostochasticcontrolstheory,
via the method of dynamic programming. The dynamic programming principle,
originated by R. Bellman in 1950s, is known as the two stage optimization
procedure.Whenwecontrolthebehaviorofastochasticdynamicalsysteminorder
to optimize some payoff or cost function, which depends on the control inputs to
the system, the dynamic programming principle gives a powerful tool to analyze
problems. Exploiting the dependence of the value function (optimal payoff) on
its terminal cost function, we will construct a nonlinear semigroup which allows
onetoformulatethedynamicprogrammingprincipleandwhosegeneratorprovides
theHamilton–Jacobi–Bellmanequation.Herewearemainlyconcernedwithfinite
timehorizonstochasticcontrols.Wealsoapplythesemigroupapproachtocontrol-
stopping problems and stochastic differential games, and provide with examples
fromtheareaoffinancialmarketmodels.
Thisbookisorganizedasfollows.Chapters1–4dealwithcompletelyobservable
finite-dimensional controlled diffusions. Chapters 5 and 6 are concerned with
Hilbert space valued stochastic processes, related to partially observable control
problems.
Chapter1isareviewofstochasticanalysisandstochasticdifferentialequations
withrandomcoefficientsforlateruses.Chapter2dealswithcontrolproblemswith
finite-time horizon. By a time-discretization method we construct a semigroup,
associatedwiththevaluefunction,whosegeneratorprovidestheHamilton–Jacobi–
Bellman equation. When the value function is smooth, it becomes a classical
solutionoftheHamilton–Jacobi–Bellmanequation.However,itsatisfies theequa-
tioninviscositysenseevenifitisnotsmooth.Chapter3isconcernedwithviscosity
solutions of nonlinear parabolic equation, including Hamilton–Jacobi–Bellman
equationsofstochasticcontrolsandalsostochasticoptimalcontrol-stoppingprob-
lems. Chapter 4 presents zero sum, two-player, time-homogeneous, stochastic
differential games and the Isaacs equations. We consider stochastic differential
games by using progressive strategies. Then we construct semigroups associated
with the upper and lower values, by using a semidiscretization method. These
semigroups lead to the formulation of the dynamic programming principle and
v
vi Preface
to the upper and lower Isaacs equations. The link between stochastic control and
differential game is given via the risk sensitive control. Chapter 5 is a review on
stochastic evolutionequationson Hilbertspaces, in particularstochastic parabolic
equations with colored Wiener noises. Basic definitions and results and Itô’s
formula are presented. Chapter 6 is concerned with control problems for Zakai
equations. We again construct semigroups associated with the value functions.
The dynamicprogrammingprincipleand viscosity solutionsof Hamilton–Jacobi–
Bellman equations on Hilbert spaces are treated by using results obtained in the
previouschapters.WeshowtheconnectionbetweencontrolledZakaiequationsand
controlofpartiallyobservablediffusions.
Kobe,Japan MakikoNisio
Acknowledgement
This book was planned as a new edition of Stochastic Control Theory, ISI
Lecture Notes 9 (1981) following F. Delbaen’s recommendation. I would like to
acknowledgehisrecommendationtogetherwithvaluableadviceduringpreparation
ofthemanuscript.
The author is greatly indebted to W. H. Fleming, who read carefully the
manuscript and offered many valuable comments and suggestions, especially for
Chap.4, which led to a much improved version. Many thoughtful helps and
encouragements had been given by experts and friends, particularly F. Asakura,
Y.Fujita,H.Nagai,andT.Urataniwhohelpedtoimprovethebookatvariousstages.
H.Morimotoassistedinwritingthemanuscriptbycarefullyreadingitandmaking
valuablecomments,especiallyonmathematicaleconomics.
vii

Contents
1 StochasticDifferentialEquations .......................................... 1
1.1 ReviewofStochasticProcesses ........................................ 1
1.1.1 RandomVariables .............................................. 1
1.1.2 StochasticProcesses............................................ 5
1.1.3 ItôIntegrals ..................................................... 10
1.1.4 Itô’sFormula.................................................... 13
1.2 StochasticDifferentialEquations ...................................... 15
1.2.1 LipschitzContinuousSDEswithRandomCoefficients ...... 15
1.2.2 GirsanovTransformations...................................... 18
1.2.3 SDEswithDeterministicBorelCoefficients .................. 22
1.2.4 BackwardStochasticDifferentialEquations.................. 24
1.3 AssetPricingProblems................................................. 26
1.3.1 Formulation..................................................... 26
1.3.2 BackwardSDEfortheSellingPrice........................... 27
1.3.3 ParabolicEquationAssociatedwith(1.84).................... 29
2 OptimalControlforDiffusionProcesses ................................. 31
2.1 Introduction ............................................................. 31
2.1.1 Formulations.................................................... 32
2.1.2 ValueFunctions:BasicProperties ............................. 34
2.2 DynamicProgrammingPrinciple(DPP)............................... 40
2.2.1 Discrete-TimeDynamicProgrammingPrinciple ............. 41
2.2.2 ApproximationTheorem....................................... 46
2.2.3 DynamicProgrammingPrinciple.............................. 47
2.2.4 BrownianAdaptedControls.................................... 50
2.2.5 CharacterizationoftheSemigroup(V ;™(cid:2)t)............... 53
(cid:2)t
2.3 VerificationTheoremsandOptimalControls.......................... 58
2.3.1 VerificationTheorems.......................................... 58
2.3.2 ExamplesofOptimalControl.................................. 62
2.4 OptimalInvestmentModels............................................ 68
2.4.1 Formulations.................................................... 68
ix