Table Of ContentMAT H E MAT I C S
I N SCIENCE
AND ENGINEERING
A SERIES OF MONOGRAPHS AND TEXTBOOKS
Edited by Richard Bellman
University of Southern California
1. TRACYY. THOMASC. oncepts from Tensor Analysis and Differential
Geometry. Second Edition. 1965
2. TRACYY. THOMAPSla.s tic Flow and Fracture in Solids. I961
3. RUTHERFORADR IS. The Optimal Design of Chemical Reactors: A Study
in Dynamic Programming. 1961
4. JOSEPH LASALLEa nd SOLOMONL EFSCHETZS. tability by Liapunov’s
Direct Method with Applications. 1961
.
5. GEORGEL EITMAN(Ne d.) Optimization Techniques: With Applications to
Aerospace Systems. 1962
6. RICHARDB ELLMANa nd KENNETHL . COOKE.D ifferential-Difference
Equations. 1963
7. FRANKA . HAIGHTM. athematical Theories of Traffic Flow. 1963
8. F. V. ATKINSOND. iscrete and Continuous Boundary Problems. 1964
9. A. JEFFREY and T. TANIUTNIo.n -Linear Wave Propagation: With Appli-
cations to Physics and Magnetohydrodynamics. 1964
10. JULIUS T. Tou. Optimum Design of Digital Control Systems. 1963
11. HARLEYFL ANDERDS.i fferential Forms: With Applications to the Physical
Sciences. 1963
12. SANFORMD . ROBERTSD. ynamic Programming in Chemical Engineering
and Process Control. 1964
13. SOLOMOLNE FSCHETZS.t ability of Nonlinear Control Systems. 1965
14. DIMITRISN . CHORAFASSy. stems and Simulation. 1965
15. A. A. PERVOZVANSKRIaIn. dom Processes in Nonlinear ControI Systems.
1965
16. MARSHALCL. PEASE1, 11. Methods of Matrix Algebra. 1965
17. V. E. BENES.M athematical Theory of Connecting Networks and Tele-
phone Traffic. 1965
18. WILLIAMF . AMES.N onlinear Partial Differential Equations in Engineering.
1965
19. J. ACZ~LL.e ctures on Functional Equations and Their Applications. 1966
20. R. E. MURPHYA. daptive Processes in Economic Systems. 1965
21. S. E. DREYFUSD. ynamic Programming and the Calculus of Variations.
1965
22. A. A. FEL’DBAUMO.p timal Control Systems. 1965
MATHEMATICS IN SCIENCE A N D ENGINEERING
23. A. HALANAYD. ifferential Equations: Stability, Oscillations, Time Lags.
1966
24. M. NAMIKO ZUZTORELIT. ime-Lag Control Systems. 1966
25. DAVIDS WORDERO. ptimal Adaptive Control Systems. 1966
26. MILTONA SH. Optimal Shutdown Control of Nuclear Reactors. 1966
27. DIMITRISN . CHORAFASC.o ntrol System Functions and Programming Ap-
proaches. (In Two Volumes.) 1966
28. N. P. ERUGINL. inear Systems of Ordinary Differential Equations. 1966
29. SOLOMONM ARCUSA. lgebraic Linguistics; Analytical Models. 1967
30. A. M. LIAPUNOVSt.a bility of Motion. 1966
31. GEORGEL EITMANN(e d.). Topics in Optimization. 1967
32. MASANAAOO KI. Optimization of Stochastic Systems. 1967
33. HAROLDJ. KUSHNERS. tochastic Stability and Control. 1967
34. MINORUU RABEN. onlinear Autonomous Oscillations. 1967
35. F. CALOGEROV.a riable Phase Approach to Potential Scattering. 1967
36. A. KAUFMANGNr.a phs, Dynamic Programming, and Finite Games. 1967
37. A. KAUFMANanNd R. CRUOND. ynamic Programming: Sequential Scien-
tific Management. 1967
38. J. H. AHLBERGE, . N. NILSON,a nd J. L. WALSH.T he Theory of Splines
and Their Applications. 1967
39. Y. SAWARAGYI,. SUNAHARAan,d T. NAKAMIZOS.t atistical Decision
Theory in Adaptive Control Systems. 1967
40. RICHARDBE LLMANIn. troduction to the Mathematical Theory of Control
Processes Volume I 1967 (Volume I1 and I11 in preparation)
In preparation
A. KAUFMANaNnd R. FAUREI.n troduction to Operations Research
E. STANLEYLE E. Quasilinearization and Invariant Imbedding
WILLARDM ILLER,J R. Lie Theory and Special Functions
PAULB . BAILEY,L AWRENCFE. SHAMPINEa,n d PAULE . WALTMANN.o nlinear
Two Point Boundary Value Problems
N. K. JAISWALP.r iority Queues
ROBERTH ERMANDNi.f ferential Geometry and the Calculus of Variations
P. PETROVV. ariational Methods in Optimum Control Theory
Introduction to the
MATHEMATICAL THEORY
OF CONTROL PROCESSES
Richard Bellman
DEPARTMENTS OF MATHEMATICS,
ELECTRICAL ENGINEERING, AND MEDICINE
UNIVERSITY OF SOUTHERN CALIFORNIA
LOS ANGELES, CALIFORNIA
V O L U M E I
Linear Equations and Quadratic Criteria
I967
A C A D E M I C PRESS NewYorkand London
COPYRIGH0T 1967, BY ACADEMIPCR ESS INC.
ALL RIGHTS RESERVED.
NO PART OF THIS BOOK MAY BE REPRODUCED IN ANY FORM,
BY PHOTOSTAT, MICROFILM, OR ANY OTHER MEANS, WITHOUT
WRITTEN PERMISSION FROM THE PUBLISHERS.
ACADEMIC PRESS INC.
111 Fifth Avenue, New York, New York 10003
United Kingdom Edition published by
ACADEMIC PRESS INC. (LONDON) LTD
Berkeley Square House, London W. 1
LIBRAROYF CONGRESS CATALOG CARD NUMBER67: -23153
PRINTED IN THE UNITED STATES OF AMERICA
INTRODUCTION
A new mathematical discipline has emerged from the bustling scien-
tific activity of the last fifteen years, the theory of control processes. Its
mathematical horizons are unlimited and its applications increase in
range and importance with each passing year. Consequently, it is reason-
able to believe that introductory courses in control theory are essential
for training the modern graduate student in pure and applied mathe-
matics, engineering, mathematical physics, economics, biology, opera-
tions research, and related fields.
The extremely rapid growth of the theory, associated intimately with
the continuing trend toward automation, makes it imperative that
courses of this nature rest upon a broad basis. In this first volume, we
wish to cover the fundamentals of the calculus of variations, dynamic
programming, discrete control processes, use of the digital computer,
and functional analysis.
The basic problem in attempting to present this material is to provide
a rigorous, yet elementary, account geared to the abilities of a student
who can be safely presumed to possess only an undergraduate training
in calculus and differential equations, together with a course in matrix
theory. We can achieve our objectives provided that the order of pre-
sentation in the text is carefully scheduled, and that we focus upon the
appropriate problems.
In Chapters 1 and 3, we discuss in a purely expository fashion a
number of topics in the study of systems and stability which furnish the
cultural background for the purely analytic theory. Chapter 2 provides
a review of the properties of second-order linear differential equations
that we employ in Chapters 4 and 5.
vii
viii Introduction
In Chapter 4, we begin the mathematical exposition with the problem
of minimizing the quadratic functional
+
J(u, v) = joT(u2 v2) dt,
where u and v are related by the linear differential equation
-du = au + u, u(0) = c.
dt
To treat this problem rigorously in a simple fashion, we pursue a
roundabout course. First we employ a formal method to obtain the
Euler equation, a fundamental necessary condition. Then we show by
means of a simple direct calculation that this equation has a unique
solution, and that this solution provides the desired minimum value.
The analytic techniques used are carefully selected so as to generalize
to the multidimensional problems treated in Chapter 7.
Chapter 5 contains an introductory account of dynamic programming
with applications to the problems resolved in the previous chapter.
Using the explicit solutions obtained from the calculus of variations, we
can easily demonstrate that the methods of dynamic programming
furnish valid results. Once again we follow the same path: first a formal
derivation of the desired results and then a proof that the results
obtained in this fashion are correct.
In this chapter, we also give a brief account of discrete control pro-
cesses. The technique of dynamic programming is particularly well
suited to this important type of process and there is no difficulty in
applying dynamic programming in a rigorous fashion from the very
beginning.
Having described the dual approaches of the calculus of variations
and dynamic programming in the scalar case, we are ready to tackle the
multidimensional optimization problems. In order to do this, it is
necessary to employ some vector-matrix notation, together with some
elementary properties of matrices. In Chapter 6, we review the applica-
tion of matrix theory to linear differential equations of the form
N
2dx ’= C aijxj, xi(0)= ci, i = 1,2, . . . , N.
dt
j= 1
Introduction iX
We are now ready to study the problem of minimizing the quadratic
functional
joT
J(x, Y)= [(x,X I + (Y,Y )1 df,
where x and y are related by means of the differential equation
dx +
- = Ax y, x(0) = c.
dt
Here x and y are N-dimensional vectors.
A straightforward generalization of the methods of Chapters 4 and
5, aided and abetted by the power of matrix analysis, enables us in
Chapters 7 and 8 to give, respectively, a treatment by means of the
calculus of variations and dynamic programming.
What is new here, and completely masked in the scalar case, is the
difficult problem of obtaining an effective numerical solution of high-
dimensional optimization problems. In each chapter there are short
discussions of some of the questions of numerical analysis involved in
this operation.
The concluding chapter, Chapter 9, contains an account of a direct
attack upon the original variational problems by means of functional
analysis. This is the only point where the book is not self-contained,
since we require the rudiments of Hilbert space theory for the first part
of the chapter, and a bit more for the final part of the chapter.
At the ends of the various sections inside the chapters, there are a
number of exercises that illustrate the text. At the ends of the chapters,
we have placed a number of problems that are either of a more difficult
nature or that relate to topics that we have reluctantly omitted from the
text in order to avoid any undue increase in the mathematical level.
References to original sources and to more detailed results are given with
these exercises. Much of this material can be used for classroom presen-
tation if the instructor feels the class is at the appropriate level.
The second volume in this series will be devoted to a discussion of the
probltms associated with the analytic and computational treatment of
more general optimization problems of deterministic type involving non-
linear differential equations and functionals of more general type.
The third volume will begin the treatment of stochastic control pro-
cesses with emphasis upon Markovian decision processes and linear
equations with quadratic criteria.
Los Angeles, California Richard Bellman
-
WHAT IS CONTROL THEORY?
1.1. Introduction
This is the first of a series of volumes devoted to an exposition of a
new mathematical theory, the theory of control processes. Despite the
fact that no attempt will be made to discuss any of the many possible
applications of the methods and concepts that are presented in the
following pages, it is worthwhile to provide the reader with some idea
of the scientific background of the mathematical theory. There are many
reasons for doing this. In the first place, it is very easy for the student to
become overwhelmed with the analytic formalism unless he has some
clear idea of the general direction that is being pursued; in the second
place, control theory is still a rapidly developing field with the greater
part yet to come. We prefer to have the reader strike out boldly into
the new domains rather than trudge in well-ploughed fields, and we fe&
strongly that the most significant and exciting mathematics will arise in
the study of the most important of contemporary areas requiring control
and decisionmaking.
Finally, there is the matter of intellectual curiosity. A person who
claims the distinction of being well-educated should know the origins
and applications of his field of specialization.
2 1. What Is Control Theory?
1.2. Systems
Control theory centers about the study of systems. Indeed, one might
describe control theory as the care and feeding of systems. Intuitively,
we can consider a system to be a set of interacting components subject
to various inputs and producing various outputs. There is little to be
gained at this stage of development from any formal definition.
We are all familiar with many different types of systems : mechanical
systems such as docks, electrical systems such as radios, electro-
mechanical-chemical systems such as automobiles, industrial systems
such as factories, medical systems such as hospitals, educational systems
such as universities, animate systems such as human beings, information
processing systems such as digital computers, and many others that
could be catalogued. This brief list, however, is sufficient to emphasize
the fact that one of the most profound concepts in current culture is
that of a system.” The ramifications of this product of civilization will
“
occupy the intellectual world for a long time to come. In particular, it
is an unlimited source of fertile mathematical ideas.
1.3. Schematics
It is often a convenient guide to both intuition and exposition to con-
ceive of a system in terms of a block diagram. A first attempt is:
- -
I
Input System J Output
Frequently, this pictogram is much too crude, and we prefer the follow-
ing diagram to indicate the fact that a system usually has a variety of
inputs and outputs.
-
I,
-
12
System
-
1M.
1.3. Schematics 3
Diagrams of this type, in turn, must yield to still more sophisticated
representations. Thus, for example, in chemotherapy, we often use the
following diagram in connection with the study of the time history of a
drug and its by-products injected into the bloodstream. Here, S, repre-
sents the heart pumping blood through the extracellular and intra-
cellular tissues of two different organs, [S,, S,] and [S,, S,].
Figure 1.1
In multistage zpr1odu4ctiNon p-ro c-es[s-es., .or. -chMemic-a l separation processes,
we may employ a schematic such as:
0,
- - - -
...
12 0, t -
This indicates that the inputs, I,, I,, to subsystem S, produce outputs
0,, 0, that are themselves inputs to subsystem S, , and so on until the
final output.
In the study of atmospheric physics, geological exploration, and
cardiology, we frequently think in terms of diagrams such as
bringing clearly to the fore that there are internal subsystems which we
can neither directly examine nor influence. The associated questions of
identification and control of S, are quite challenging.
