Arithmetic complexity of computations

100 Pages·1987·10.17 MB·English

by  Shmuel Winograd

CBMS-NSF REGIONAL CONFERENCE SERIES
IN APPLIED MATHEMATICS
A series of lectures on topics of current research interest in applied mathematics undei
the direction of the Conference Board of the Mathematical Sciences, supported by the
National Science Foundation and published by SIAM.
GARRETT BIRKHOFF, The Numerical Solution of Elliptic Equations
D. V. LINDLEY, Bayesian Statistics, A Review
R. S. VARGA, Functional Analysis and Approximation Theory in Numerical Analysis
R. R. BAHADUR, Some Limit Theorems in Statistics
PATRICK BILLINGSLEY, Weak Convergence of Measures: Applications in Probability
J. L. LIONS, Some Aspects of the Optimal Control of Distributed Parameter Systems
ROGER PENROSE, Techniques of Differential Topology in Relativity
HERMAN CHERNOFF, Sequential Analysis and Optimal Design
J. DURBIN, Distribution Theory for Tests Based on the Sample Distribution Function
SOL I. RUBINOW, Mathematical Problems in the Biological Sciences
P. D. LAX, Hyperbolic Systems of Conservation Laws and the Mathematical Theory of Shock Waves
I. J. SCHOENBERG, Cardinal Spline Interpolation
IVAN SINGER, The Theory of Best Approximation and Functional Analysis
WERNER C. RHEINBOLDT, Methods of Solving Systems of Nonlinear Equations
HANS F. WEINBERGER, Variational Methods for Eigenvalue Approximation
R. TYRRELL ROCKAFELLAR, Conjugate Duality and Optimization.
SIR JAMES LIGHTHILL, Mathematical Biofluiddynamics
GERARD SALTON, Theory of Indexing
CATHLEEN S. MORAWETZ, Notes on Time Decay and Scattering for Some Hyperbolic Problems
F. HOPPENSTEADT, Mathematical Theories of Populations: Demographics, Genetics and Epidemics
RICHARD ASKEY, Orthogonal Polynomials and Special Functions
L. E. PAYNE, Improperly Posed Problems in Partial Differential Equations
S. ROSEN, Lectures on the Measurement and Evaluation of the Performance of Computing Systems
HERBERT B. KELLER, Numerical Solution of Two Point Boundary Value Problems
J. P. LASALLE, The Stability of Dynamical Systems—Z. ARTSTEIN, Appendix A: Limiting Equations
and Stability of Nonautonomous Ordinary Differential Equations
D. GOTTLIEB and S. A. ORSZAG, Numerical Analysis of Spectral Methods: Theory and Applications
PETER J. HUBER, Robust Statistical Procedures
HERBERT SOLOMON, Geometric Probability
FRED S. ROBERTS, Graph Theory and Its Applications to Problems of Society
JURIS HARTMANIS, Feasible Computations and Provable Complexity Properties
ZOHAR MANNA, Lectures on the Logic of Computer Programming
ELLIS L. JOHNSON, Integer Programming: Facets, Subadditivity, and Duality for Group and Semi-
Group Problems
SHMUEL WINOGRAD, Arithmetic Complexity of Computations
J. F. C. KINGMAN, Mathematics of Genetic Diversity
(continued on inside back cover)
SHMUELWINOGRAD
IBM Thomas J. Watson Research Center
Arithmetic Complexity
of Computations
SOCIETY FOR INDUSTRIAL AND APPLIED MATHEMATICS
PHILADELPHIA, PENNSYLVANIA 1980
All rights reserved. No part of this book may be reproduced, stored, or transmitted in any man-
ner without the written permission of the Publisher. For information, write the Society for In-
dustrial and Applied Mathematics, 1400 Architect's Building, 117 South 17th Street, Philadelphia,
Pennsylvania 19103-5052.
Copyright© 1980 by the Society for Industrial and Applied Mathematics.
Second printing 1986.
Third printing 1990.
Library of Congress Catalog Card Number 79-93154.
ISBN 0-89871-163-0.
Printed for the Society for Industrial and Applied Mathematics by J. W. Arrowsmith, Ltd., Bristol,
England.
Contents
Chapter I
INTRODUCTION 1
Chapter II
THREE EXAMPLES
Ila. Product of integers 3
lib. Discrete Fourier transform 4
He. Matrix multiplication 5
Chapter III
GENERAL BACKGROUND
Ilia. Definitions and basic results 7
Illb. Linear functions 11
IIIc. Quadratic and bilinear forms 13
Chapter IV
PRODUCT OF POLYNOMIALS
IVa. Minimal algorithms 25
IVb. Classification of the algorithms 28
IVc. Heuristic algorithms 32
Chapter V
FIR FILTERS
Va. Filters and polynomials 39
Vb. Filters with decimation 46
Vc. Symmetric filters 49
Chapter VI
PRODUCT OF POLYNOMIALS MODULO A POLYNOMIAL
VIa. An illustrative example 57
VIb. Multiplication modulo irreducible polynomial 60
VIc. Multiplication modulo a general polynomial 62
VId. Multiplication modulo several polynomials 63
Chapter VII
CYCLIC CONVOLUTION AND DISCRETE FOURIER
TRANSFORM
VIla. Cyclic convolution 71
Vllb. DFTOO-n prime 76
VIIc. DFT(pr)-p odd prime 79
Vlld. DFT(2r) 83
VIIe. Multidimension DFT 87
Bibliography 93
This page intentionally left blank
CHAPTER I
Introduction
The two major problem areas which are the concern of Arithmetic Complexity of
Computations are:
1. What is the minimum number of arithmetic operations which are needed to
perform the computation?
2. How can we obtain a better algorithm when improvement is possible?
These two questions are very large in scope, since they pertain to any compu-
tation which is arithmetic in nature. In these lectures we will not attempt to cover
the whole theory, but concentrate on a narrower class of computational prob-
lems—that of computing a system of bilinear forms. That is we aim at a better
understanding of the amount of arithmetic needed to compute the quantities «Ai,
where the A:,'S and y/s denote the inputs to the algorithm, that is the data, and the
a 's are constant independent of the data. As we will see later, there are many
ijk
problems of practical interest which can be viewed as computing a system of
bilinear forms. Yet finding the most efficient algorithm for computing a system of
bilinear forms can be most vexing. In spite of the apparent simplicity of systems of
bilinear forms many of the problems connected with their computations have not
been solved.
In developing these lectures we will try to keep a balance between the
mathematical aspects of the theory and the applicability of the results. We will
emphasize the results which lead to applications in the area of signal processing.
This choice was motivated by several reasons: Many signal processing problems
place a very heavy computational load on even the largest computers which are
available, thus even a modest reduction in their execution time may have practical
significance. Secondly, the results which are applicable to signal processing are
relatively new and consequently do not appear in any of the books on compu-
tational complexity which are available, but are scattered in journal articles. Last,
and not least, the choice of the material which was made gives a good indication of
the flavor of complexity of computation (admittedly at the expense of indicating
its full scope).
In the next section, we will describe three algorithms which were discovered in
the last two decades and which were the motivation for much of the development
in complexity of computations. The following section will provide a general
background of the basic results, and the next three sections will deal with the
l
2 CHAPTER 1
complexity of computing convolution, digital filtering, and the discrete Fourier
transform.
I would like to end this introductory section with another qualification. Even
though the title speaks of arithmetic complexity, that is the consideration of both
the number of additions and the number of multiplications, we will concentrate
our attention on the number of multiplications. The main reason is that the theory
concerning the number of multiplications is much more developed than that of the
number of additions. However, in the discussion of some of the applications we
will have to pay attention to the number of additions, and not just to the number of
multiplications.
CHAPTER II
Three Examples
The systematic investigation of the number of arithmetic operations needed to
perform a computation received its greatest impetus by the discovery, during the
1960's, of three very surprising algorithms—for the multiplication of two integers,
for computing the discrete Fourier transform, and for the product of two matrices.
These three computational problems have been with us for a very long time, and
the fact that more efficient algorithms for their execution were discovered
indicated that a study of the limits of the efficiency of algorithms may yield some
more interesting results. We will devote this section to describing these
algorithms. Their importance is not only historic; the new algorithm for comput-
ing the discrete Fourier transform has had a very profound impact on the way
many computations are being performed.
Ha. Product of integers. In the early 1960's two Russian mathematicians,
Keratsuba and Toom, discovered a new way for multiplying two large integers.
The regular way, we all learned in school, for obtaining the product of two n -digit
numbers called for performing n2 multiplications of single digits and about the
same number of single digit additions. The new method enables us to obtain the
product using many fewer operations.
The specific method which will be described is different from the original one.
The original paper of Karatusba [3] and the follow-up by Toom [4] are more
complicated (and yield sharper results).
Let x and y be two n -digit numbers and assume that n = 2m is an even number.
If b denotes the base then we can write x = x + x i • b m and y = yo + yi • bm, where
0
XQ, xi, yo, yi are m-digit numbers. The product z of x and y can then be written as
Thus the problem of computing the product x - y can be viewed as that of
computing the three quantities XQ, yo, Xoy\ + JCiyo, x\, yi, and performing 2m = n
single digit additions.
The key to the new algorithm is the way x y , x yi + Xiy, and JCiyi are
0 0 0 0
computed. The computation of these quantities is based on the identities:
Thus one has to compute the three products x0y0, x\yi, and (x0 - Xi)(y0 — yi), and
perform two more additions (or subtractions) of 2m(=w)-digit numbers. The
3
4 CHAPTER II
computation of x — x\, and yi — yo necessitates two more subtractions of m-digit
0
numbers. If we take as a unit of addition the addition (or subtraction) of two
single-digit numbers with the possibility of a carry-(or borrow), we obtain that the
product of two n -digit numbers uses 3 multiplications of m-digit numbers and 4n
units of additions. We can, of course, use the same scheme to obtain the product of
Xoyo, *iy i, and (XQ — *i)(yo ~ y i). Thus if n = 2s is a power of 2, and using the initial
condition that for m = 2° = 1 only one single digit multiplication and no units of
additions are needed, we obtain that 3s multiplications of single digits and
8(3s — 2s) units of addition suffice. In the case that n is not a power of 2, we can
"pad" the numbers by adding enough leading zeros. So the formulas are still
valid if we take s — [loga n] for any n.
To summarize, we have shown a method for computing the product of two
n-digit numbers which uses at most 3 • 3log2" =3 . l0^3 single digits multi-
n
plication and 8 • 3 • 3log2" = 24 • nlog23 units of additions. A more refined analysis
can reduce the constants 3 and 24, or their sum (the total number of operations),
but the rate of growth of the number of operations will remain nlog23.
In his paper, Toom describes other methods which further reduce the exponent
of n. We will not discuss them here. The main point we wish to emphasize is that
even such a timeworn computational task as the product of two large integers may
not be as simple and as strightforward as we had thought.
lib. Discrete Fourier transform. The Fourier transform and harmonic analysis
have been recognized for a long time as powerful tools in the study of functions
and linear systems. This importance was not reflected in numerical computation
because of the amount of computations which was needed to calculate the discrete
version of the Fourier transform. In 1965 the situation changed radically. Cooley
and Tukey published a new algorithm for computing the discrete Fourier trans-
form (DFT) which substantially reduced the amount of computations [5]. Their
algorithm known as FFT (fast Fourier transform) is now an indispensible ingre-
dient in all aspects of signal processing.
The discrete Fourier transform of n points is given by
where cu is the nth root of 1 (w = e2m/n). Straightforward computations of each A,
uses n (complex) multiplications and n — 1 (complex) additions. So the straight-
forward computation of all the n A,'s uses n2 (complex) multiplications and n2-n
(complex) additions.
In 1965 Cooley and Tukey observed that whenever n = n\ • n is a composite
2
number a more efficient method of computation exists. Each /' in the range
0 ^ / g n -1 can be written uniquely as / = i\ + i • n i (0 ^ /1 < n i; 0 ^ i < n ), and
2 2 2
each / in the range 0 g/ ^ n — 1 can be written uniquely as / =/i • «2+/2 (p =J\ <
«i; Osy <tt ), and therefore
2 2