BSTJ 60: 3. March 1981: On Newton-Direction Algorithms and Diffeomorphisms. (Sandberg, I.W.)

4 MB·English

‘On Newton-Direction Algorithms and
Diffeomorphisms*

By |W. SANDBERG
(thane aoaved August 29,1980)

“This paper reports on results that complement those in an earlier
oper by tis triter which yines a constructive proof of the existence
ofan algorithm that foreach rghu-hand side a, produces a sequence
tohich converges globally and supertinearty to soladin x of fis) =
fa whenever ta a C-difvomorphism (Le, is. continuously-diferen
table invertible map with continuously differentiable inverse) of
Hunach space B onto iself and either = K” or [satisfies eertain
‘ther conditions that are often met in applications. ere tne consider
the cave in which fis Lipscite on each bounded eubset of B. We
tive results tshich, while along the lines of those obtained sartar,
funcern a fendamentall different Newton direction ulgorthin which
‘does not appear to have been introduoed previous, and which has
the edeantage that ite implementation does not require the use of
certain search procedures,

1. TRODUCTION

Let fhe function rom U into, whore Dis a Banach space with
norm |-[, and Ue a nonempty apn subbed of B. We say that Ps
diferentiable on a set SC U if fhas a Prechet derivative /'(s) at each
point ¢ of Sf Uf, for example, B~ A” with the usual Wacideen norm,
{hen fi differentiable on U Wit i continously differentiable on 218
the usual sense) By f a C'diffewmorphiven, we mam that [ik &
hhomeoenorphism of Uonto Zand and (F "exist and nee continuous
fon UT and A, respectively. (We emphasize that here continuity refers to
tthe devendence of the derivatives on the points at which They are

ViESiRe sony Walfvenseie us £0 C leech 2 the Youn
cde ie fish Wed Baath fs Ai my'9) POS toate
‘evaluate, no co their boundedness as operators, which in aasured by
‘efincon’) C'-diffeororphisma frequenty urise in applications,

"The purpoce of this paper isco report on resale that complement
thoee in Reb. 1 where a constrictive proof is given of che existence of
tNewlondirection algorichm that, for each a e B, generates a se
fquenos in U' which converges plobslly and superlinesrly to a solution
off x) ~ a whenever fie nC’ diffeomorphism of U onto Band lther
B= R° orf satisfies cartsin other conditions that are frequently cet
in applications, War the cae of an important class of monotone
diffeomorphiaun fin a There space 2, the “ether conditions” reduce
to simply the requirement that f° be uniformly cootinnous on lowed
hounded subsets of HZ A specific example in which #7 is infinite
dimensional is iven in Re 1)

"The algorithm described in Ref 1 typically involves the recursive
determination of postive sealare yw, «++ (which determine the
nceestve steplengths) such that certain ratio Ry (yn) (which depends
fn the Ach fverate x) Ties hetweon prescribed bounds forall = 0,1,
sos, While its proved that the y ean be choson as required, and
that j= Lor all sufcintty lage f the seta determination of the
tn ine specific cme would ordinary requir the use of @one-dimen-

tonal search procedure for » faite (and poesibly large) number of
alae of

In this paper we address the case in which U7 = Band fia Lipschite
on bounded wubsels of Be, 2 such that for each bounded abet S
(FB chere is w constant A sach that [f'%4) ~ '(0)| = Ale ~ 0| forall,
tend vin S). We give results which while along the lines of chose in
[Ret 1, conoorn 0 fundamentally cifferent Newtan-direction algorithm.
tac dows nol anpear to havo been intrduced earlier, and which does
rot requir the uee of eoarch procedures eo olve cupproblems of the
{ype outlined above.

‘Our results are presented in Section T, As a consequence of the
Lipochite hypothesis proof are comparatively simple and we are able
to establish quadratic father than superlinese) convorgonce.(Iecall
hata sequence. x4, -- in B converges quadratically to an element
‘sof Rif the sequence converges lo # and there is a constant ¢ such
that |e" =| el2" — xP forall et)

General relationshipa beiwen diffeomorphisms and computation af
the type describod in Ref. 1 anu in Section Ido not appear io have

soca tia th pes ew nd mr of «
"dar epee io uy fm then Ra 9 te Lie hte
ies cern Tr at ae nn

‘380 THE BELL SYSTEM TECHNICAL JOURNAL, MARCH 108:
been raportd on cation by other writers. On the other hand, es in
Ref. 1 our approach involves the minimization of a functional, and
therefore ina general congo there anv elated Hterntare. (Seo, for
feumple, Ref? and note (p19) Unst the least-quares Newton-
direction methods deseibed there requie, in particular, che existence
of second derivatives of f (our notation) Additional background
‘material can be found in Re.

I. PROCESSES Ns and W,

‘Throughout this section we wie the term Lipechite and converges
‘quadratically in the way indicated in Section I, we denote the wel
These won af @ Lines? map A of into Fy [Al, and we take U
B.

‘With fdiereniableon Bi, hat not neceasarlly a CiReomorphism,
and with 2" and a wy tn elements of B, consider the following
Process in which s denotes [/(a") ~ al whenever x* © His defined,

Process Ny: Chocee p ©
Le

Ife) ~ 9,900 xt = at
LE fts"y 0, determine 4: Mosel thal

18.1) ana A 0. De the fllowrng fr & =

Psyc =~ fs). Thon

Thee y= ar i oan
Wasa |

2 Lees = 2 + yt

1 Set a0 = yh Gh either > tM and [f(y — |= 01 —

(25) “fe, oro ph an [fey"8) ~ al Ans. Mf nether pale of
conditions ie met, place} by 2k in Step 1 andthe eencence preceding
his aontones and return wo Seep 1

(Our main repute the following

‘Theorem i: Suppose that fs @ C™-difoomorphiam of Bondo B. Let
1 ln Lipnchits om bounded subsets of Band let | (P| be bounded
‘on Bounded subsets of B. Thi for each a and each x", Process Ny
fan be carried ont uni x, 44, +» converges quadratically to the
lunigue soltion suf f18) =

Lata and "be given,
‘We first prove ea lems which concern cases in which need not
toe Chdittemnonphiam Lot F-~ {e © B: [feb — al = [/t2")~ al),
and let £ denote (wt a fw)"fa ~ flwlw € L, @ € 10, 1) when
PC) exits on L, (Acsming thar Lia defined, notice that it is
bounded if Zs bound! and [/"(-)"| ix bounded on [This observa-
‘on is used later in the proof of Theorem 1 and in connection with
Lemsias 1 and 2 below) With 9 a positive constant, consider the
following process.

Process No Chante p © [%, 1. Do the following for
efi
1 f') # a, determine 4 © such that

Fa ibe~ @— fla). Thon let
fra) if 209
1 Ww Oop

Seca aha ae

Lemina ts Antune that L is bounded andl that [sand (46) ‘exist
on L with |f'twl"| 2K or w= L and sume constant K. Assume also
that (+ exiata and Lipsoits, with Eipechite constant A, on £.
Then fory = AK

(a) Provess Ns can be curr cut, and for each k auch that se # 0
we have

a o

sores em if a= 2p °

(0) 008k >
fe) If there ts an x © B wach that fle) = a and ah» x a8 b+
then (a) converges quadratically

‘We will use the following proposition.

Proposition 1: Suppose thatthe hypotheses of Lemma 1 are met. If
PE Ly and 7 © [0, Land gu denotes fs") Ta ~ f(T) then, for 7
AKT webave fo" + yyol~ a] =~ pifte") =a] + i6ny|
att

Proofs We have

Ite! + y60) — al = [fie =e +2 phn +8]
in which
5 = Flat yo) ~ Fla) — Fe

542 THE DELL SYSTEM TECHNICAL JOURNAL, MARCH 1981
he
[reat + yo — |= = Use) al + 18h
and

[FG yp — Pi dB

‘Since [5 | = 64K" f(x) ~ a, we have proved the proposition.”
‘Assure now that 2 AK’, and that the hypotheses of Lemma 1 are
‘lat be such that either f= 0, or Process Ne can be mse la generate

aly cvosat? with 2 © for 1,2, --s A Suppose that vs 0. Since

TYEE, gy ean be determinod, Since @ © ¥% 1), when «> 2p" we

Ihave (yat"! <1. Thus, by Une proposin, (1) holés. On the other

hand, obviously nse» when se~ 2 "and thus, by the proposition,

{Gh is met, Thieshows dat 3°" cam be devermined, that iesatises (1)

land (Band that 3°" © F, which proves Part (2.

‘Par (6) ia direct conmmquance of Part (a, because, by Pat (a) it
sn des not approach zero aa f= we must have sy > 299°! fora
Snowbich cose [1-— ya) "T=, 1) and ay [1 — qs" for k=
1 hich i contradiction,

‘Assume now that H cuniains an x ouch that f(r) = wand x4» as
Asin Since x! € Fofor all hand Z is cleaed, x € L. Let denote
(x). Since ison invertible Bounded linear map of into itself there
fe positive constants dl eaueh eat fui [Sal = Bell fore
EB. For each & we hive

[ea] =[/e) aah) + 6)

Dak +e,

in which [4s ==
[Notice that for sre
[yet 0 /s4ls| for hem.
hus for k=,
(ta) ~ al = tet = 29 | “1812
donee other bad,
Mla) — al = Jeet — 29 + 1d = [tet ~ 29] fala —
We have sys = a rf

late — 29] 2 %)x* ah,

M for come Mm

ae NRE GERGANA pacton tat ai spect 9 ie

NEWTONDIRECTION ALGORITHMS 943
Theretore,
[ene SM ai Bila ol, ea,
‘which completes the proof ofthe Iemma."

Lemma 2 Suppose that L is Bounded and that #"(-) and f')"
exist on L with "C1" Bounded on E. Suppose also that f+) exists
find & Lipschitz on F. Than Procese NY, an be sare ott, oe hae
fn On he, and if there iy a x © H auch that f(x) ~ a and $°
eae t+ =, then 27, ~~ converges quadratically

2.1.2 Proof of emma 2

Consider Process Ny. Ry Terme 1, there it a constant 2y that
depends only on f @, and x" such that if\ in Step | and the fist
fontence of Slep S satis = 2g, and iether FU and so #0, oF
=O and Process N; can be used to determine 3" with #4 #0 and sy =
‘then Seep 3 can be enmied out_on the firs pass, Notice that
tmhonever 2° feet equal ta in Step 3 we have ses <u

‘ince for any \ > 0 there i a nonnegative inceger p such that 2°
= Avit follows thor Proccas NV can bo crzied cut, and thet for some
nonnegative integers andr, we haves! = oa" for A> 9, where oe
STL hs) when s! 2 201272)" and os ~ p otherwise. Since
t-< Tork git clear that p> One A> 2, and therefore that sus,
2 Ui ahal for A= AF for some MF Thus, by the proof of Port (el of
Lemania 1, our proof of Tama 2 complete

"Now ltthe hypotheses of Thearem 1 be meth proof of Theorem
3 of Ref. I showa chat Z is bounded, thet /(-) ! exists on B, and that
1G is bounded on £. Since Fis w homeomorphiam of B onto B, 4
“SG as =» o implies that > x as A > where x satis f(a) —
@. By Lemma 2, this vorplees the proof of Theorem 1.

Lat 4:10, =) + [0 9) be eontinacns,stritly inereasing, and sich
that YOO) = 0, Yla) ee an a yal "a >for (0,8) or
some postive constants e andl ee Notice that, for example, Ye) =
‘meet these conditions

‘Tei no) omar cui flo fm ds ngs of
PULP. f Rahs rhe tripe it ware
ou Sanding Pan Ea nade tw SN

neh here te ant ean rune Tho e's eset ch sh
puree etn Pad Soe eh ne a tr
cometary wotne es ee» ee

‘944 THE BELL SYSTEM TECHNICAL JOURNAL, MARCH 1081
Theorea 2 Let {map a reat Hilbert save H, with inner product
toon itself seh that fla) flo), w— 0) =u — a] yllae— of)
for alt, © H, Assume thar f' exists and is Lipschitz on bounuied
fubsets of H. Then fie 0 Chifecmorphism of H onto Hf, and the
‘eonetesion of Theorem 1 hotds

‘Using hens 4, a proof of Theorem 2 ean be obtnined hy tivially
nanifying the proof of Theorem 4 of Rel 1

age=m

"The folowing vomnplete result ia a dicot corollary of Theorem 1 (eae
the pronf of Thearem § of Rel

"Thowrem 3: Let B= R°, and lt f° be Lipachite end eontinunsly
Aifarentiable on bounced subsets of. Then fs @ Clennam
fam of ° onte ise if and only af

Ti) Process N, can Be carried Hut foreach a and each x"

in) For each a, the sequence produced by Process N converses
quadratically toa solution of 8) ~ a, and x dacs not depend on 2"

2.4 Comments

‘As in Rot 1, our primary purpose ix to foc attention on genera
_catioships between diffeomorphiens and computation. Cleany, no
fttempe ie made to optimize the performance o ll aspect of the type
‘Walgoithin described, However, thee are some besically sl-cvidext
Ioifications hae ar somerinna nafs For example, the total nua
Tr of Werations required in w spc cage can sometimes be reduced
significantly by repeatedly, or arciionalls, stopping the algorithm
lle a suber of steps and rseting the initial valve of \ in Process
Ny co a smaller number. (Ili not dificult to give rules of thura
‘rmcering niten to stop the slorilun and by how much to reduce,
bt se have nol tried to prove theorems that hour on thee: matters)
Gr'eamese, ound on the location wf the solation apd eatimaes of K
nl A which ae avilable in some problem, can bose in an bins
fray. Similan for example ~ 2,» yloballs convergent steepest
dlegcent proce (ne Ref. 6) might be used inisaly to obs a beter
fanprorimation tothe solution before she Newton-dgcetinn ulgnithin
fa uted, (In fee, well knosen and oen use strategy is vo comin
steepest descent al ure Newco itrarions in thi wat)

se sae rt eet sepa coe One

raph as gtd =

NEWION-OIRECTION ALGORITHMS 345
REFERENCES

1. 1 Sng “Diteoneethine and Newman icon Ageing" BST 9
Rowe fp
2s A Ba hertz of Fenton, Eager Ci,

‘tac
E10 onion Dj. Rewn, "Siving amin Stns Ran rn

eto Toners TEER Ro Cae and
tad Spel eo Nose Cae

Shift

48, THE BELL SYSTEM TECHNIGAL JOURNAL, MARCH 1981