Introduction to the Mathematical Theory of Control Processes: Nonlinear Processes

311 Pages·1967·5.385 MB·ii-xix, 1-306\311

by  Richard Bellman (Eds.)

Introduction to the
MATHEMATICAL THEORY
OF CONTROL PROCESSES
Richard Be1 I man
DEPARTMENTS OF MATHEMATICS
AND ELECTRICAL ENGINEERING
UNIVERSITY OF SOUTHERN CALIFORNIA
LOS ANGELES, CALIFORNIA
VOLUME II
NonI i near Processes
1971
ACADEMIC PRESS New York and London
COPYRIGH0T 1 971, BY ACADEMPICR ESSI,N C.
ALL RIGHTS RESERVED.
NO PART OF THIS BOOK MAY BE REPRODUCED IN ANY FORM,
BY PHOTOSTAT, MICROFILM, RETRIEVAL SYSTEM, OR ANY
OTHER MEANS, WITHOUT WRITTEN PERMISSION FROM
THE PUBLISHERS.
ACADEMIC PRESS, INC.
111 Fifth Avenue, New York, New York 10003
United Kingdom Edition published by
ACADEMIC PRESS, INC. (LONDON) LTD.
Berkeley Square House, London WlX 6BA
LIBRAROYF CONGRECSAS TALOCGA RDN UMBER67:- 23153
PRINTED IN THE UNITED STATES OF AMERICA
To Stan Ulam,f riend and mentor
PREFACE
In the first volume of this series, devoted to an exposition of some of the
basic ideas of modern control theory, we considered processes governed by
linear equations and quadratic criteria and discussed analytic and computa-
tional questions associated with both continuous and discrete versions. Thus
typical problems are those of minimizing the quadratic functional
1
T
J(x, r) = C(X, Ax) + (v,4 41 dt
0
where the vectors x and y are connected by the differential equations x’ =
+
Cx Dy, y(0) = c, and its discrete counterpart. In this volume we wish to
broaden the scope of these investigations.
First we shall consider the more general problem of minimizing a func-
tional
jOT
dx,Y )d l
where x and y are related by the equation x’ = h(x, y), x(0) = c and its
companion discrete version, the question of minimizing the expression
N
where
xn+l = Nxn, yn), xo = C.
This latter can be readily treated by means of the theory of dynamic pro-
gramming.
We shall pursue a parallel development. Chapters 2 and 3 are devoted to
analytic and computational aspects of the foregoing using dynamic pro-
gramming. Chapters 4 and 5 are devoted to the same types of questions for
the continuous process using the calculus of variations.
xvii
xviii Preface
The order of presentation is the reverse of the order in Volume I where the
calculus of variations was treated first. There the motivation was that a
simple rigorous account could easily be given and that this approach could
in turn be used to provide a rigorous derivation of the fundamental Riccati
equation of dynamic programming.
Here we use dynamic programming to provide a simple rigorous approach
to general discrete control processes. At the cost of essentially no additional
effort, we can in this way handle constraints and stochastic effects. On the
other hand, the consideration of these important aspects of general continuous
processes requires a non-negligible mathematical training and sophistication.
The reader who has mastered the material in Chapters 2 and 3 is well
prepared to consider a number of significant control processes that arise in
biology, medicine, economics, engineering, and the newly emerging environ-
mental sciences. We have avoided any discussion of applications here,
although it is clear that our mathematical models and our continued emphasis
upon numerical solution are both motivated by the kinds of questions that
arise in numerous scientific investigations.
In Chapters 6, 7, and 8 we consider various types of interconnections,
between different types of control processes, between discrete and continuous
processes, between finite and infinite processes, and between dynamic
programming and the calculus of variations. The application of dynamic
programming to continuous processes introduces the basic nonlinear partial
differential equation
fi- = fin [dc, 4 + vf,l,
0
an object of great interest in its own right. We have, however, not pursued
the study in any depth since this probably involves the theory of partial
differential equations much more than control theory. The reader interested in
derivation of a number of fundamental results in the calculus of variations,
plus Hamilton-Jacobi theory, directly from this equation may consult the
cited book by S. Dreyfus.
In Chapters 9, 10, and 11 we turn to methods specifically aimed at the
numerical solution of control problems : duality, reduction of dimensionality,
and routing. In Chapters 12 and 13 we present a brief introduction to the
study of distributed control processes, specifically processes governed by
partial differential and differential-difference equations. Finally in Chapter 14
we very briefly indicate some of the vast, relatively unexplored fields of control
theory awaiting study.
Throughout our aim has been to provide a relatively simple introduction
to one of the most important new mathematical fields, the theory of control
processes. Numerous references are given to books and research papers
where more intensive and extensive study of dynamic programming, the
Preface xix
calculus of variations, computational techniques, and other methods may be
found. We have tried to preserve the impartial attitude of a guide through a
museum pointing out the many fascinating treasures that it contains without
dwelling overly on individual favorites.
I would like to express my appreciation to Art Lew who diligently read
the manuscript and made many valuable suggestions, as well as to Edward
Angel, David Collins, John Casti, Tom Higgins, and Daniel Tvey, and to
Rebecca Karush who patiently typed a number of revisions.
-
BASIC CONCEPTS OF CONTROL THEORY
1.1. Introduction
The feasible operation and the effective control of large and complex systems
constitute two of the central themes of our tumultuous times. Although con-
siderable time, human and computer resources, and intellectual effort have
already been devoted to their study, one can foresee that their dominating
position in contemporary application and research will be maintained and even
accentuated. As always, a nexus of problems of social, economic and scien-
tific import creates its associated host of novel, intriguing and formidable
mathematical problems. A number of these questions may be profitably
contemplated and treated with the aid of the existing mathematical theory of
control processes. Many, however, require further elaborations and exten-
sions of this theory ; many more appear to require new theoretical develop-
ments.
In this chapter we wish to lay the groundwork for much of the discussion
and analysis that follows and to present some of the conceptual foundations
for both discrete and continuous control processes of deterministic type. Both
of these processes, needless to say, represent drastically simplified versions of
realistic and significant control processes, which we shall discuss in some
detail in Chapter 14. Nonetheless, they serve the most useful purpose of
providing convenient starting points for analytic and computational discus-
sions of didactic nature. They can readily be used to illustrate the advantages
and disadvantages of both the calculus of variations and dynamic program-
1
2 1. Basic Concepts of Control Theory
ming. Furthermore, a careful investigation of their numerous shortcomings
provides us with points of departure for extensive research into the formula-
tion and the treatment of more meaningful control processes.
We shall, in this and other endeavours, occasionally retread ground already
covered in the first volume in order to make the presentation here as self-
contained as possible. We feel that no serious harm is done a certain
b.7
amount of repetition.
1.2. Systems and State Variables
We shall employ the useful word system" in a precise fashion to describe
"
a combination of a state vector, x(t), and a rule for determining the behavior
of this vector over time. We shall be principally concerned in what follows
with finite-dimensional vectors, although in later chapters we shall briefly
consider infinite-dimensional vectors in connection with distributed control
processes.
The two simplest, and therefore most common presumptions for future
behavior, are expressed analytically either in terms of a difference equation,
(1.2.1)
or in terms of a differential equation,
dx
-dt = dx), x(0) = c. (1.2.2)
In the first case, the variable we may consider to represent time is discrete-
valued, t = 0, 1, 2, . . . , in the second case, time is continuous.
In both formulations we make the basic assumption that the future depends
only on the present and not at all on the past. This is more a question of the
choice and dimension of the state vector than any intrinsic property of the
system.
1.3. Discussion
What are the connections between this mathematical definition of the term
system and the complex physical entities of the real world that we blithely
" "
call systems "?There is no easy answer. One preliminary answer, however, is
"
that a physical system can be, and should be, treated mathematically in many
different ways. Hence, to any '' real system corresponds innumerable
"
mathematical abstractions, or systems in the foregoing sense. We can
" "
consider the real system to include the set of all such realizations.
1.4. Control Variables 3
This one-to-many correspondence, this cardinal concept, cannot be over-
emphasized. Too often, the student, who is familiar only with one conven-
tional mathematical realization (a heritage of bygone eras), consciously and
unconsciously identifies a particular set of equations with the real system. He
may then promptly forget about the physical system and focus solely upon
the mathematical system.
This narrowness of vision results in a number of losses. First, there is a loss
of flexibility and versatility. Secondly, fundamental questions are easily over-
looked in favor of mathematical idiosyncrasies. Thirdly, modern systems
contain many basic features not present in classical formulations.
The mathematical realization utilized should depend critically upon both
the type of results desired and the tools available for the production of these
results. Many new problems arise in connection with the choice of an
appropriate mathematical representation.
1.4. Control Variables
Descriptive equations of the type shown in Section 1.2 enable us to predict
the future behavior of the system given a knowledge of its current state. On
the basis of calculations of this nature, and occasionally even simply on the
basis of observation, we can readily conclude that many systems do not per-
form in a completely acceptable fashion. There can be many reasons for our
dissatisfaction: components of the system may begin to malfunction; the in-
puts to the system, as determined by the epironment, may change in quantity
and quality over time; and, finally, there may be significant changes in our
own goals and evaluation of performance. All of these phenomena are par-
ticularly evident in the case of large systems, especially social systems.
A first thought in the search for a system exhibiting better performance
over time is to introduce an entirely new system, or at least to rebuild the old
system. The first alternative is generally not practical. For example, in the
biomedical sphere it is not feasible to think of using an entirely different body
run by the old brain, which is what one would consider to be the real identity.
It is feasible, however, to use organ transplants, artificial kidneys, hearts and
SO forth, to alleviate certain medical conditions. The field of prosthetics and
orthotics is one of the challenging new areas of control theory.
In the economic and social spheres, it is not feasible to contemplate the
use of entirely different systems because we don't know enough to accomplish
this change effectively without the introduction of transient effects, which can
be both unexpected and quite unpleasant. History makes us all gradualists.
If we don't understand very much about the operation of a complex system,
changes must be introduced slowly and carefully.
4 1. Basic Concepts of Control Theory
Let us then turn our attention to a different approach, the concept of the
use of additional and compensating effects which modify or remove undesired
phenomena. We shall avoid in what follows the exceedingly complex engineer-
ing questions connected with precise ways in which these effects are actually
introduced into the system. Clearly, these are basic problems of the utmost
significance as far as the construction of meaningful mathematical models is
concerned. Often, consideration of these questions leads to new and important
mathematical problems. Furthermore, it must be admitted that sufficient
attention has not been paid to these questions in the past. Some discussion
of operational considerations will be found in Chapter 14.
Since, however, we are primarily interested in presenting an introductory
treatment of the many mathematical areas of modern control theory, we will
suppose simply that the net result of the various control activities in and
around the physical system is the appearance of another function in the
defining equations of Section 1.2.
Thus, in the discrete case, the descriptive equation takes the form
xn+l=g(xn,Yn), xO=c, (1.4.1)
while the in continuous case we have
x’ = g(x, y), x(0) = c. (1.4.2)
We call this new function y appearing on the right-hand side the control vector
or control variable.
1.5. Criterion Function
The question immediately arises about the determination of effective ways
of choosing the control variable. In general, there are severe limitations on
the freedom of choice that exists. These limitations, or constraints as they are
usually called, arise both from the nature of the physical system itself and
from the type of devices that are available for exerting control. Recognition
of these realistic bounds on behavior usually introduces severe analytic diffi-
culties. Sometimes these constraints present serious obstacles to a numerical
solution; sometimes they greatly facilitate it. It is often a question of the
method employed. For pedagogical reasons we shall defer any serious con-
sideration of control processes of this nature until the fourth volume in this
series.
Here, we will suppose that a choice of control is dictated solely by questions
of cost. We distinguish two types of cost:
(a) the cost of exerting control,
(1.5.1)
(b) the cost of undesired behavior of the system.