Least Absolute Deviations: Theory, Applications and Algorithms

362 Pages·1984·12.07 MB·English

by  Peter Bloomfield

Progress in Probability and Statistics
Vol. 6
Edited by
Peter Huber
Murray Rosenblatt
Birkhauser
Boston· Basel· Stuttgart
Peter Bloomfield
William L. Steiger
Least Absolute
Deviations
Theory, Applications,
and Algorithms
1983 Birkhiiuser
Boston • Basel • Stuttgart
Authors:
Peter Bloomfield
Department of Statistics
North Carolina State University
Raleigh, N.C. 27650
William L. Steiger
Department of Computer Science
Rutgers University
New Brunswick, N.J. 08903
Library of Congress Cataloging in Publication Data
Bloomfield, Peter, 1946-
Least absolute deviations.
(Progress in probability and statistics; vol. 6)
Bibliography: p.
Includes indexes.
1. Least absolute deviations (Statistics) 2. Regres-
sion analysis. 3. Curve fitting. I. Steiger, William L.,
1939- II. Title. III. Series: Progress in
probability and statistics; v. 6.
QA275.B56 1983 519.5 83-25846
ISBN 978-1-4684-8576-9 ISBN 978-1-4684-8574-5 (eBook)
001 10.1007/978-1-4684-8574-5
CIP-Kurztitelaufnahme der Deutschen Bibliothek
Bloomfield, Peter:
Least absolute deviations: theory, applications,
and algorithms / Peter Bloomfield; William L.
Steiger. -Boston; Basel; Stuttgart: Birkhauser,
1983.
(Progress in probability and statistics; Vol. 6)
NE: Steiger, William L.:; GT
All rights reserved. No part of this publication may be reproduced, stored in a
retrieval system, or transmitted, in any form or by any means, electronic,
mechanical, photocopying, recording or otherwise, without prior permission of
the copyright owner.
© Birkhauser Boston, Inc., 1983
Softcover reprint of the hardcover 1s t edition 1983
To our children
Reuben and Nina Steiger
David and Gareth Bloomfield
and our muses
vii
PREFACE
Least squares is probably the best known method for
fitting linear models and by far the most widely used.
Surprisingly, the discrete L 1 analogue, least absolute deviations
(LAD) seems to have been considered first. Possibly the LAD
criterion was forced into the background because of the com
putational difficulties associated with it.
Recently there has been a resurgence of interest in LAD.
It was spurred on by work that has resulted in efficient al
gorithms for obtaining LAD fits. Another stimulus came from
robust statistics. LAD estimates resist undue effects from a
feyv, large errors. Therefore. in addition to being robust, they
also make good starting points for other iterative, robust
procedures.
The LAD criterion has great utility. LAD fits are optimal
for linear regressions where the errors are double exponential.
However they also have excellent properties well outside this
narrow context. In addition they are useful in other linear
situations such as time series and multivariate data analysis.
Finally, LAD fitting embodies a set of ideas that is important in
linear optimization theory and numerical analysis.
viii PREFACE
In this monograph we will present a unified treatment of
the role of LAD techniques in several domains. Some of the
material has appeared in recent journal papers and some of it
is new.
This presentation is organized in the following way.
There are three parts, one for Theory, one for Applicatior.s
and one for Algorithms.
Part I consists of the first three chapters. Chapter 1 is
a short introduction to LAD curve fitting. It begins by tracing
the history of the LAD criterion in fitting linear models. The
main points in this development involve algorithms or ideas on
which algorithms could be constructed. The key section of the
chapter develops - from first principles - some of the
properties of the LAD fit itself, especially those describing
uniqueness and optimality.
Chapter 2 is devoted to linear regression. The behavior
of the LAD estimate is described in a result originally due to
Bassett and Koenker. This theorem gives the limiting error
distribution for LAD, and is shown, in part, to be a con
sequence of an earlier, more general result on R-estimators,
the trick being the identification of LAD as a particular R
estimator. Next, some of the robustness properties are
developed for the LAD regression estimator. Finally a Monte-
ix
Carlo experiment compares the behavior of LAD to least
squares and to some Huber M-estimators on a variety of
regression models.
Chapter 3 deals with linear time series, specifically sta
tionary autoregressions. The main theorem here gives a rate
for the convergence of the LAD estimator to the autoregres
sive parameters. It is surprising that the rate increases as the
process becomes more dispersed, or heavy-tailed. Once again,
Monte-Carlo results comparing LAD to LSQ and Huber's M
estimator are given for several autoregressions. These portray
the behavior described in the main theorem, and convey a
sense of the efficiency of LAD in comparison to the other
estimators. They also provide evidence for a conjecture that
would extend the convergence rate result.
The next two chapters deal with applications and com
prise Part II. Chapter 4 treats additive models for two-way
tables. It describes some properties of Tukey's median polish
technique, and its relationship to the LAD fit for the table.
Recent work of Siegel and Kemperman sheds new light on this
subject. Chapter 5 discusses the interpretation of the LAD
regression as an estimate of the conditional median of y given
x. The requirement that the conditional median be a linear func
tion of x is then weakened to the requirement that it merelv
be a smooth function. This leads to the introduction of cubic
x PREFACE
splines as estimates of the conditional median and, by a minor
modification, of other conditional quantiles.
The final chapters constitute Part III. dealing with al
gorithmic considerations. We discuss some particular LAD
alogrithms and their computational complexities. The equiv
alence of LAD fitting to solving bounded, feasible linear pro
gramming problems is established and the relationship between
LAD algorithms and the simplex method is discussed. We
conclude with some recent work on exact, finite, algorithms
for robust estimation. These emphasize that not only is LAD a
member of larger classes of robust methods, but also that the
flavor of the LAD procedures carries across to algorithms for
other, general. robust methods.
Each chapter concludes with a section of Notes. These
go over points somewhat tangential to the material covered in
the chapter. Sometimes we trace evolution of ideas, or try to
show how items in the existing literature influenced our
presentation. Topics that are original are delineated. In ad
dition some conjectures, possibilities for future work, and in
teresting open questions are mentioned. We thought it
preferable to deal with such issues separately, so as not to in
terrupt the main flow of the presentation.
Literature on LAD techniques appears mainly within two
xi
separate disciplines, statistics and numerical analysis. We have
made a reasonably serious attempt to give a comprehensive
union of two, fairly disjoint sets of sources.
Our convention for numbering of equations and the like
is self-evident. Within sections it is sequential, theorems,
definitions, etc., each with their own sequences. Within chap
ters, i.j refers to item j of Section i. Otherwise i.j.k. refers to
item k in Section j of Chapter i. Finally the symbol • will mark
the end of a proof.
Many debts have accrued since we began working on
this book. The first thanks go to colleagues. Professor Mich
ael Osborne sharpened our understanding of the subject. We
also benefitted from talking with Eugene Seneta, Scott Zeger,
Shuia Gross, David Anderson, and Geoffrey Watson. We are
especially grateful to Professor Watson for his generous help
with support and facilities. We have received support from
Department of Energy grant DE-AC02-81 ER 1 0841 to the
StatistiCS Department at Princeton. We thank the departments
of Computer Science at Rutgers, Statistics at Princeton, and
Statistics at North Carolina State University for the stimulating
work environments they have provided. Don Watrous in the
systems group of the Laboratory for Computer Science
Research at Rutgers was our Scribe guru, par-excellence.
Computations were carried out on the Rutgers University LCSR