Extrapolation, interpolation, and smoothing of stationary time series, with engineering applications.

170 Pages·1966·36.36 MB·English

by  Norbert Wiener

EXTRAPOLATlON,
INTERPOLATlON,
SMOOTHING
AND OF
STATIONARY
TIME SERIES
With Engineering Applications
by Norbert Wiener
1IIIII1
THE M.I.I. PRESS
MASSACHUSETTS INSTITUTE OF TECHNOLOGY
CAMBRIDGE, MASSACHUSETTS
•
lilT Pr•••
.~2730057
11111111111111
Fir.,prWlng,AUfutl,1949
&contiprinling,June, 1950
Thirdprinting,Jcmuory, 1957
FOlll'lhprinting,Odober,1960
Fiflhprinl....F....". 1964
FintM.l.T. PreuPoper6GeJlEdition, FebruD.r:Y, 1964
PREFACE
Largely because of the impetus gained during World War II, com
munication and control engineering have reached a very high level
ofdevelopment today. Manyperhaps do notrealize that the present
age isready forasignificantturn inthe developmenttoward fargreater
heights than we have ever anticipated. The point of departure may
wellbe the recasting and unifying of the theories ofcontrol and com
munication in the machine and in the animal on a statistical basis.
The philosophy of this subject is contained in my book entitled
Cybernetics.* The presentmonographrepresentsODe phaseofthe new
theory pertainingto themethods and techniques in the design ofcom
municationsystems; itwasfirstpublishedduringthewarasaclassified
report to Section D National Defense Research Committee, and is
21
nowreleasedforgeneral use. Inorder to supplement the present text
by less complete but simpler engineering methods two notes by
ProfessorNorman Levinson, in which he developssome of the main
ideas in a simpler mathematical form, have been added as Appendixes
BandC. Thismaterial, whichfirst appearedinthe Journal of!IIaihe
matics and Physics, is reprinted by permission.
In themain, the mathematicaldevelopmentshere presentedare new.
However, they are along the lines suggested by A. Kolmogoroff
(Interpolation und Extrapolation von stationaren zufalligen Folgen,
Bulletin de l'academie des sciences de U.R.S.S., Ser. Math. S,pp- 3-14,
1941; cf. also P. A. Kosulajeff, Sur les problemes d'interpolation et
d'extrapoletion des suites stationnaires, Comptes rendus de l'academie
des sciences de U.R.S.S., Vol.30, pp. 13-17, 1941.) An earlier note of
Kolmogoroffappears in the ParisComptes rendusfor 1939.
To theseveralcolleagueswhohave helped mehy theircriticism, and
in particular to President Karl T. Compton, Professor H. 111. James,
Dr. WarrenWeaver, Mr. Julian H. Bigelow, and Professor Norman
Levinson, I wishto expressmy gratitude. Also, I wish to give credit
to Mr. Gordon Raisheck for his meticulous attention to the proof
reading of this book.
Norbert Wiener
CClmbridge,MCI'$C1chu$eth
Maren.1949
"Published by The 11.I.T. Press,Cambridge,Massachusetts
v
CONTENTS
INTRODUCTION
0.1 Tbe PurposeofThIsBook . I
0.2 TimeSerieo . I
0.3 CommunicationEngineering 2
0.4 Techniques of Time Series and Communication Engineering Oee-
trasted 3
0.41 The Eneemble 4
0.42 Correlation . 4
0.43 The Periodognun 6
0.44 OperationalCalculna 7
0.46 TheFourierIntegral; NeedoftheComplexPlane. 8
0.5 TimeSeriesand CommunicationEngineering-TheSynthesis 8
0.51 Prediction. 9
0.52 Filtering. . 9
0.53 PolicyProblems . . . . . 10
0.6 PermissibleOperators: Translation GroupinTime. 11
0.61 Pastand Future. 12
0.62 BubclaeseaofOperators. 12
0.7 Normsand Minimization 13
0.71 The CalculusofVariations. 14
0.8 ErgodicTheory . 15
0.81 Brownian Motion 20
0.9 Summary ofChapters 21
ChapterI RESUME OF FUNDAMENTAL MATHEMATICAL NOTIONS 25
1.00 FourierSeries 25
1.01 Orthogonal Functiona . 31
1.02 TbeFourier Integral 34
1.03 LaguerreFuoctiona . 35
1.04 Moreonthe FourierIntegralj Realizability ofFilters. 36
1.1 Genera1ized Harmonic Analyaie . . 37
1.18 Discrete Arraya and TbeirSpectra 43
1.2 MultipleHarmonicAnalysisand Coherency Matriees . 44
1.3 SmoothingProblems 46
1.4 ErgodicTheory . 46
1.5 BrownianMotion 47
1.6 PoissonDistributions 51
1.7 Harmonic Ana1ysiaintheComplexDomain. 52
ChapterII THE LINEAR PREDICTOR FOR A SINGLE TIME SERIES 56
2.01 Formulation oftheProblemofthe LinearPredictor 56
2.02 The MinimizationProblem. 57
2.03 TbeFactorisationProblem. 60
vii
CONTENI'S
2.04 The PredictorFormula. • . . . . • . . . . . . • M
2.1 ExempleoofPrediction. . . . . . • . . . . . . . Il6
2.2 ALimilinCExempte01Prediction . . . . . . . . 68
2.3 ThePredictionofFuneUonaWhoseDerivativesPOIIIJeI!IIAuto-ooml...
linn Coefliciento . . . . . . . • • . . . • 71
2.4 Spectrum LiD..endNon-abeolutelyConlinuo...Spectra . 74
2.5 Predictionby thelinearCombinotion01Given Operatora 78
2.8 ThelinearPredictorlor 0 DiecreteTime Be..... 78
ChapterIII THE LINEAR FLTER FOR A SINGLE TIME SERIES 81
3.0 FormulationoltbeGeneral FilterProblem 81
3.1 Minimiza.tion ProblemforFilters. . . . . . . 82
3.2 The FactorizationoCthe Spectrum . . . . . . 84
3.3 Predictionend Filtering . . . . . . . . . 88
3.4 TheErrorofPerformanceofaFilter; 1.00&:4161Filters 88
3.5 Filtera endErgodic Theory . . . . . . . . lID
11.8 Computetion01SpecificFilterCberaeterillti.. . . . 91
3.7 LeggingFilten • . . . . . . . . . . . 92
3.g The Determinolinn01Leg endNumber01Mesh..in0 Filter. 94
3.9 DeteelingFilteralor High NoieeLevel . . . . . . . 95
3.91 FilteralorPuIeea. . . . . . . . • . . . . . 95
3.92 FilteraHoving CberaeterilltieslinearlyDependenton Given Cberac-
teristi08 . . . . . . . . . . . . . . . . . . t¥1
3.93 Computetion01Filter: R6lum6 . . . . . . . . . . . 101
Chapterrt THE LINEAR PREDICTOR AND FlTER FOR MUl.nPlE TIME 104
SERIES
4.0 SymbolismendDefinitionslor MultipleTime Be..... 104
4.1 Minimi..tion Problemfor MultipleTimeSeries. 105
4.2 Methnd01UndeterminedCoefficients. lOll
4.3 MultiplePredietion . . . . . . . 109
U Speei&lC 01Predietion. . . . . liD
4.5 ADiscreteC 01Predietion. . . . III
4.6 GeneralTecboique01DiscretePrediction 112
ChapterV MISCBLANEOUS PROBlEMS ENCOMPASSED BY THE 117
TECHNIQUE OF THIS BOOK
5.0 TheProblem01ApproximeteDillerentietioo 117
U An Bumple01ApproximateDillerentielinn. 119
5.2 AMisle&dingExemple01ApproximateDillerentietion. 120
5.3 Interpolalinnend ErtrepolatioD . . . . . . 121
Appendix A TABLE OF THE LAGUERRE FUNCTIONS 124
Appendix 8 THEWIENER RMS(ROOTMEANSQUARE)ERROR CRIT5- 129
RlCNIN FlTER DESIGN ANDPREDlcnoN(by Norman
Levinson)
I. LiDe&rFiltera. . . . . 180
2. MiDimisetioD01RMSEnor 131
CONTENTS
3. DeterminationoftheWeightingFunction 136
4. Realization ofOperator- MathematicalFormulation 139
5. RCFilter. 143
6. PredictionandLagwithandwithoutNoise . 146
Appendix C A HEURISilCEXPOSITION OFWIENER'SMATHEMATICAL 149
THEORY OF PREDICTION AND FILTERING (by Normcn
Levinson)
I. The Auto-correlation Function. 150
2. The IntegralEquation . 152
3. The Modltied Integral Equation 153
4. The FactorizationProblem 155
5. The Functions""10"'2. and ox 157
6. ThePredictionOperator 158
INTRODUCTION
0.1 The Purpose ofThis Book
This book represents an attempt to unite the theory and practice of
two fieldsof work which are of vital importance in the present emer
gency, and which have a complete natural methodological unity, but
whichhave up to the present drawn their inspiration fromtwo entirely
distinct traditions, and which are widelydifferent in their vocabulary
and the training oftheir personnel. These'two fieldsare those oftime
seriesin statisticsand ofcommunication engineering.
0.2 Time Series
Time series are sequences, discrete or continuous, of quantitative
data assigned to specific moments in time and studied with respect to
the statisticsoftheir distributionintime.They maybesimple,inwhich
case they consist of a single numerically given observation at each
moment of the discrete or continuous base sequence; or multiple, in
which case they consist of a number of separate quantities tabulated
accordingtoatimecommontoall.Theclosingpriceofwheatat Chicago,
tabulated by days, is a simple time series. The closing prices of all
grainsconstitutea multiple time series.
The fields of statistical practice in which time series arise divide
themselves roughly into two categories: the statistics of economic,
sociologic, and short-time biological data, on the one hand; and the
statistics of astronomical, meteorological, geophysical, and physical
data, on the other. In the first category our time seriesare relatively
short under anythinglikecomparablebasicconditions.Theseshortruns
forbid the drawing ofconclusionsinvolvingthe variable orvariables at
adistantfuturetimetoanyhighdegreeofprecision.Thewholeemphasis
is onthe drawing ofsomesort ofconclusionwith a reasonableexpecta
tion that it be significant and accurate within a very liberal error. On
the other hand, since the quantities measured are often subject to
human control,questionsofpolicyandoftheeffectofa changeofpolicy
on the statistical characterofthe time seriesassumemuchimportance.
In the second category oftime series,typified by series ofmeteoro
logicaldata,longrunsofaccuratedatatakenundersubstantiallyuniform
external conditionsare the rule rather than the exception.Accordingly
1
2 INTRODUCTION
quite refined methods of using these date for prediction and other
related purposesareworthconsidering.Ontheotherhand,owing to the
length of the runs and the relative intractability of the physical bases
of the phenomena, policy questionsdonotappearso generally as in the
economic case. Of course, as the problem of flood control will show,
they doappear, and the distinctionbetween thetwo typesofstatistical
work isnot perfectly sharp.
0.3 CommunicationEngineering
Let us nowtumfrom the study oftimeseriesto that ofcommunica
tion engineering.Thisis the study of messagesand their transmission,
whetherthesemessagesbesequencesofdotsanddashes,asinthe Morse
codeor the teletypewriter,orsound-wave patterns,as in thetelephone
or phonograph, or patterns representing visual images, 3S in telephoto
service and television. In all communication engineering-if wedonot
count such rude expedients as the pigeon post as communication engi
neering-the message to be transmitted is represented as some sort
of array of measurable quantities distributed in time. In other words,
by coding or the useofthe voice or scanning,themessageto be trans
mitted isdevelopedintoa timeseries.Thistimeseriesisthensubjected
to transmission by an apparatus which carries it through a succession
of stages, at each of which the time series appears by transformation
as a new time series. These operations, although carried out by elec
trical or mechanical or other such means, are in no way essentially
different from theoperations computationally carried out by the time
series statisticianwithslide ruleand computingmachine.
Theproper fieldofcommunicationengineering isfar widerthanthat
generally assigned to it. Communication engineering concerns itself
with the transmission of messages.Fortheexistence of a message,it is
indeed essential that variable information be transmitted. The trans
mission of a single fixed item of information is of no communicative
value. We must have a repertory of possible messages, and over this
repertory a.measuredeterminingtheprobability ofthesemessages.
Amessageneed not be the result of a conscioushuman effort for the
transmission of ideas. For example, the records of current and voltage
kept ontheinstrumentsofanautomaticsubstationareastrulymessages
as a telephone conversation.From this point of view, the recordof the
thickness of a roll ofpaper kept by a condenser working an automatic
stop on a Fourdrinier machine is also a message, and the servo-mecha..
nism stoppingthemachine at a flawbelongsto the field ofcommunica
tion engineering,as indeed doallservo-mechanisms.This fundamental
unity of all fields ofcommunication engineering has been obscured by
COMMUNICATION ENGINEERING 3
the traditional division of engineering into what the Germans call
Starkstromiechnik and SchwachstTomtechnik-the engineering of strong
currentsandtheengineeringofweakcurrents.Therehasbeenatendency
to identify this split with that between power and communications
engineering. As a result of the consequent division of personnel,the
communications problems of the power engineer are often handled by
a technique different from that which the ordinary communications
engineer employs, and useful notions such as that of impedance or
voltage ratio as a function ofthe frequency are often much neglected.
Thisisstillfurtheraccentuatedbythe widedifferenceinthefrequency
range ofinterest to the telephone engineerand to the servo-mechanism
engineer. Ordinary.passive electric circuits have time constants of a
small fraction of a second.'For time'constants of seconds orminutes,
passive circuits requireimpedances of ordersofmagnitude not at all to
berealizedbythe conventionaltechnique ofinductancesandcapacities.
This differencein techniquehas often blinded the communications and
the powerengineersto the essential unity of their problems.
It is, of course, true that the main function of power engineering is
the transmission ofenergy orpowerfrom oneplace to anothertogether
with its generation by appropriate generators and its employment by
appropriate motors or lamps or other such apparatus. So long as this
is not associated with the transmission of a particular pattern, as for
example in processes of automatic control, power engineering remains
a separate entity with its own technique. On the other hand, in that
moment in whichcircuits oflargepowerareused to transmit apattern
orto control the time behavior ofa machine, power engineeringdiffers
fromcommunicationengineeringonly in the energy levels involved and
in the particular apparatus used suitable for such energy levels, but is
not infact aseparate branchof engineeringfromcommunications.
0.4 Techniques of Time Series and Communication Engineering
Contrasted
Let us nowseewhat are the fieldsfrom whichthe present-day.stat
istician and the present-day communication engineer draw their tech
niques. First, let us considerthe statistician. Behind all statisticalwork
liesthe theory ofprobabilities. The events whichactually happen in a
single instance arealways referredtoa collection ofeventswhichmight
have happened; and to different subcollections of such events, weights
or probabilities are assigned, varying from zero or complete improb
ability (rather than certainty of not occurring), to unity or complete
probability (rather than certainty of occurring). The strictly mathe
matical theory corresponding to this theory ofprobability isthe theory
INTRODUCTION
of measure, particularlv in the form given by Lebesgue. A statistical
method, asforexamplea methodofextrapolatinga timeseriesintothe
future, is judged by the probability with which it will yield an answer
correct within certain bounds, or by the mean (taken with respect to
probability) ofsomepositivefunction ornormoftheerror containedin
itsanswer.
0.41 The Ensemble
In otherwords,thestatisticaltheory oftimeseriesdoesnot consider
the individual time series by itself, but a distribution or ensemble of
timeseries. Thusthemathematical operationstowhicha. timeseriesis
subjected are judged, not by their effect in a particular case, but by
theiraverageeffect.Whileonedoesnot ordinarilythink ofcommunica
tion engineering in the same terms, this statistical point of view is
equally valid there. No apparatus for conveying information is useful
unless it isdesigned to operate, not ona particularmessage, but on 8.
set ofmessages,andits effectivenessistobejudgedbythewayinwhich
it performs on the average on messages of this set. HOn the average"
meansthatwehaveawayofestimatingwhichmessagesarefrequentand
whichrareOf, inotherwords,that wehaveameasureorprobability of
possiblemessages.Theapparatus to beusedfor a particular purposeis
that which gives the best result "on the average" in an appropriate
senseoftheword"average."
0.42 Correlation
Another familiar tool of the statistician is the theory of correlation.
If:tI, ..'J x" arethenumbersof onesetandY1,..'J u« the numbers of
anotherset,thecoefficient ofcorrelationbetweenthetwois
(0.421)
• •
I:xl I: yl
1 1
This quantity must liebetween -1 and 1. Mathematically it is to be
interpretedasthecosineoftheanglebetweenthetwovectors (Xto' •" x.)
and (Yto . . " y.). When this quantity is nearly ±1, there is a strong
d~el"Pp.ordirect orreverse lineardependencebetweentheXl'SandYk'S.
On the other hand, if this quantity is nearly 0, a low degreeoflinear
dependence between the XkJS and YkJS is indicated. This correlation
coefficient has been normalized by the denominator which has been