Linear Models: Least Squares and Alternatives

438 Pages·1999·2.59 MB·English

by  C. Radhakrishna Rao

Springer Series in Statistics
Advisors:
P. Bickel, P. Diggle, S. Fienberg, K. Krickeberg,
I. Olkin, N. Wermuth, S. Zeger
Springer
New York
Berlin
Heidelberg
Barcelona
Hong Kong
London
Milan
Paris
Singapore
Tokyo
Springer Series in Statistics
Andersen/Borgan/Gill/Keiding: Statistical Models Based on Counting Processes.
Andrews/Herzberg: Data: A Collection of Problems from Many Fields for the Student
and Research Worker.
Anscombe: Computing in Statistical Science through APL.
Berger: Statistical Decision Theory and Bayesian Analysis, 2nd edition.
Bolfarine/Zacks: Prediction Theory for Finite Populations.
Borg/Groenen: Modern Multidimensional Scaling: Theory and Applications
Bremaud: Point Processes and Queues: Martingale Dynamics.
Brockwell/Davis: Time Series: Theory and Methods, 2nd edition.
Daley/Vere-Jones: An Introduction to the Theory of Point Processes.
Dzhaparidze: Parameter Estimation and Hypothesis Testing in Spectral Analysis of
Stationary Time Series.
Efromovich: Nonparametric Curve Estimation: Methods, Theory, and Applications.
Fahrmeir/Tutz: Multivariate Statistical Modelling Based on Generalized Linear
Models.
Farebrother: Fitting Linear Relationships: A History of the Calculus of Observations
1750-1900.
Farrell: Multivariate Calculation.
Federer: Statistical Design and Analysis for Intercropping Experiments, Volume I:
Two Crops.
Federer: Statistical Design and Analysis for Intercropping Experiments, Volume II:
Three or More Crops.
Fienberg/Hoaglin/Kruskal/Tanur (Eds.): A Statistical Model: Frederick Mosteller's
Contributions to Statistics, Science and Public Policy.
Fisher/Sen: The Collected Works of Wassily Hoeffding.
Good: Permutation Tests: A Practical Guide to Resampling Methods for Testing
Hypotheses.
Goodman/Kruskal: Measures of Association for Cross Classifications.
Gourieroux: ARCH Models and Financial Applications.
Grandell: Aspects of Risk Theory.
Haberman: Advanced Statistics, Volume I: Description of Populations.
Hall: The Bootstrap and Edgeworth Expansion.
Hardle: Smoothing Techniques: With Implementation in S.
Hart: Nonparametric Smoothing and Lack-of-Fit Tests.
Hartigan: Bayes Theory.
Hedayat/Sloane/Stufken: Orthogonal Arrays: Theory and Applications.
Heyde: Quasi-Likelihood and its Application: A General Approach to Optimal
Parameter Estimation.
Heyer: Theory of Statistical Experiments.
Huet/Bouvier/Gruet/Jolivet: Statistical Tools for Nonlinear Regression: A Practical
Guide with S-PLUS Examples.
Jolliffe: Principal Component Analysis.
Kolen/Brennan: Test Equating: Methods and Practices.
Kotz/Johnson (Eds.): Breakthroughs in Statistics Volume I.
(continued after index)
C. Radhakrishna Rao
Helge Toutenburg
Linear Models
Least Squares and Alternatives
Second Edition
With Contributions by Andreas Fieger
With 33 Illustrations
Springer
C. Radhakrishna Rao Helge Toutenburg
Department of Statistics Institut fur Statistik
Pennsylvania State University Universitat Miinchen Akademiestrasse 1
University Park, PA 16802 80799 Miinchen
USA Germany
crrl @ psuvm.psu.edu [e-mail]
Library of Congress Cataloging-in-Publication Data
Rao, C. Radhakrishna (Calyampudi Radhakrishna), 1920-
Linear models: least squares and alternatives/C. Radhakrishna
Rao, Helge Toutenburg. — [2nd ed.]
p. cm. — (Springer series in statistics)
Includes bibliographical references and index.
ISBN 0-387-98848-3 (alk. paper)
1. Linear models (Statistics) I. Toutenburg, Helge. II. Title,
m. Series.
QA279.R3615 1999
519.5'36—dc21 99-14735
© 1999, 1995 Springer-Verlag New York, Inc.
All rights reserved. This work may not be translated or copied in whole or in part without the
written permission of the publisher (Springer-Verlag New York, Inc., 175 Fifth Avenue, New York,
NY 10010, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use
in connection with any form of information storage and retrieval, electronic adaptation, computer
software, or by similar or dissimilar methodology now known or hereafter developed is forbidden.
The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the
former are not especially identified, is not to be taken as a sign that such names, as understood by
the Trade Marks and Merchandise Marks Act, may accordingly be used freely by anyone.
ISBN 0-387-98848-3 Springer-Verlag New York Berlin Heidelberg SPIN 10726080
Preface to the First Edition
Thebookisbasedonseveralyearsofexperienceofbothauthorsinteaching
linearmodelsatvariouslevels.Itgivesanup-to-dateaccountofthetheory
and applications of linear models. The book can be used as a text for
courses in statistics at the graduate level and as an accompanying text for
courses in other areas. Some of the highlights in this book are as follows.
A relatively extensive chapter on matrix theory (Appendix A) provides
the necessary tools for proving theorems discussed in the text and offers a
selectionofclassicalandmodernalgebraicresultsthatareusefulinresearch
work in econometrics, engineering, and optimization theory. The matrix
theory of the last ten years has produced a series of fundamental results
aboutthedefinitenessofmatrices,especiallyforthedifferencesofmatrices,
which enable superiority comparisons of two biased estimates to be made
for the first time.
We have attempted to provide a unified theory of inference from linear
models with minimal assumptions. Besides the usual least-squares theory,
alternative methods of estimation and testing based on convex loss func-
tions and general estimating equations are discussed. Special emphasis is
given to sensitivity analysis and model selection.
A special chapter is devoted to the analysis of categorical data based on
logit, loglinear, and logistic regression models.
The material covered, theoretical discussion, and a variety of practical
applications will be useful not only to students but also to researchers and
consultants in statistics.
WewouldliketothankourcolleaguesDr.G.TrenklerandDr.V.K.Sri-
vastava for their valuable advice during the preparation of the book. We
vi Preface to the First Edition
wish to acknowledge our appreciation of the generous help received from
AndreaScho¨pp,AndreasFieger,andChristianKastnerforpreparingafair
copy.Finally,wewouldliketothankDr.MartinGilchristofSpringer-Verlag
for his cooperation in drafting and finalizing the book.
We request that readers bring to our attention any errors they may
find in the book and also give suggestions for adding new material and/or
improving the presentation of the existing material.
University Park, PA C. Radhakrishna Rao
Mu¨nchen, Germany Helge Toutenburg
July 1995
Preface to the Second Edition
The first edition of this book has found wide interest in the readership.
A first reprint appeared in 1997 and a special reprint for the Peoples Re-
public of China appeared in 1998. Based on this, the authors followed
the invitation of John Kimmel of Springer-Verlag to prepare a second edi-
tion, which includes additional material such as simultaneous confidence
intervals for linear functions, neural networks, restricted regression and se-
lectionproblems(Chapter3);mixedeffectmodels,regression-likeequations
in econometrics, simultaneous prediction of actual and average values, si-
multaneousestimationofparametersindifferentlinearmodelsbyempirical
Bayessolutions(Chapter4);themethodoftheKalmanFilter(Chapter6);
and regression diagnostics for removing an observation with animating
graphics (Chapter 7).
Chapter 8, “Analysis of Incomplete Data Sets”, is completely rewrit-
ten, including recent terminology and updated results such as regression
diagnostics to identify Non-MCAR processes.
Chapter 10, “Models for Categorical Response Variables”, also is com-
pletely rewritten to present the theory in a more unified way including
GEE-methods for correlated response.
At the end of the chapters we have given complements and exercises.
We have added a separate chapter (Appendix C) that is devoted to the
software available for the models covered in this book.
We would like to thank our colleagues Dr. V. K. Srivastava (Lucknow,
India) and Dr. Ch. Heumann (Mu¨nchen, Germany) for their valuable ad-
viceduringthepreparationofthesecondedition.WethankNinaLieskefor
her help in preparing a fair copy. We would like to thank John Kimmel of
viii Preface to the Second Edition
Springer-Verlagforhiseffectivecooperation.Finally,wewishtoappreciate
the immense work done by Andreas Fieger (Mu¨nchen, Germany) with re-
spect to the numerical solutions of the examples included, to the technical
managementofthecopy,andespeciallytothereorganizationandupdating
of Chapter 8 (including some of his own research results). Appendix C on
software was written by him, also.
We request that readers bring to our attention any suggestions that
would help to improve the presentation.
University Park, PA C. Radhakrishna Rao
Mu¨nchen, Germany Helge Toutenburg
May 1999
Contents
Preface to the First Edition v
Preface to the Second Edition vii
1 Introduction 1
2 Linear Models 5
2.1 Regression Models in Econometrics . . . . . . . . . . . . 5
2.2 Econometric Models . . . . . . . . . . . . . . . . . . . . . 8
2.3 The Reduced Form . . . . . . . . . . . . . . . . . . . . . 12
2.4 The Multivariate Regression Model . . . . . . . . . . . . 14
2.5 The Classical Multivariate Linear Regression Model . . . 17
2.6 The Generalized Linear Regression Model. . . . . . . . . 18
2.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3 The Linear Regression Model 23
3.1 The Linear Model . . . . . . . . . . . . . . . . . . . . . . 23
3.2 The Principle of Ordinary Least Squares (OLS) . . . . . 24
3.3 Geometric Properties of OLS . . . . . . . . . . . . . . . . 25
3.4 Best Linear Unbiased Estimation . . . . . . . . . . . . . 27
3.4.1 Basic Theorems. . . . . . . . . . . . . . . . . . . 27
3.4.2 Linear Estimators . . . . . . . . . . . . . . . . . 32
3.4.3 Mean Dispersion Error . . . . . . . . . . . . . . . 33
3.5 Estimation (Prediction) of the Error Term (cid:1) and σ2 . . . 34
x Contents
3.6 Classical Regression under Normal Errors . . . . . . . . . 35
3.6.1 The Maximum-Likelihood (ML) Principle . . . . 36
3.6.2 ML Estimation in Classical Normal Regression . 36
3.7 Testing Linear Hypotheses . . . . . . . . . . . . . . . . . 37
3.8 Analysis of Variance and Goodness of Fit . . . . . . . . . 44
3.8.1 Bivariate Regression . . . . . . . . . . . . . . . . 44
3.8.2 Multiple Regression . . . . . . . . . . . . . . . . 49
3.8.3 A Complex Example . . . . . . . . . . . . . . . . 53
3.8.4 Graphical Presentation . . . . . . . . . . . . . . 56
3.9 The Canonical Form. . . . . . . . . . . . . . . . . . . . . 57
3.10 Methods for Dealing with Multicollinearity . . . . . . . . 59
3.10.1 Principal Components Regression. . . . . . . . . 59
3.10.2 Ridge Estimation . . . . . . . . . . . . . . . . . . 60
3.10.3 Shrinkage Estimates . . . . . . . . . . . . . . . . 64
3.10.4 Partial Least Squares . . . . . . . . . . . . . . . 65
3.11 Projection Pursuit Regression . . . . . . . . . . . . . . . 68
3.12 Total Least Squares . . . . . . . . . . . . . . . . . . . . . 70
3.13 Minimax Estimation. . . . . . . . . . . . . . . . . . . . . 72
3.13.1 Inequality Restrictions . . . . . . . . . . . . . . . 72
3.13.2 The Minimax Principle . . . . . . . . . . . . . . 75
3.14 Censored Regression. . . . . . . . . . . . . . . . . . . . . 80
3.14.1 Overview . . . . . . . . . . . . . . . . . . . . . . 80
3.14.2 LAD Estimators and Asymptotic Normality . . . 81
3.14.3 Tests of Linear Hypotheses . . . . . . . . . . . . 82
3.15 Simultaneous Confidence Intervals . . . . . . . . . . . . . 84
3.16 Confidence Interval for the Ratio of Two Linear
Parametric Functions . . . . . . . . . . . . . . . . . . . . 85
3.17 Neural Networks and Nonparametric Regression . . . . . 86
3.18 Logistic Regression and Neural Networks . . . . . . . . . 87
3.19 Restricted Regression . . . . . . . . . . . . . . . . . . . . 88
3.19.1 Problem of Selection . . . . . . . . . . . . . . . . 88
3.19.2 Theory of Restricted Regression . . . . . . . . . 88
3.19.3 Efficiency of Selection . . . . . . . . . . . . . . . 91
3.19.4 Explicit Solution in Special Cases. . . . . . . . . 91
3.20 Complements . . . . . . . . . . . . . . . . . . . . . . . . 93
3.20.1 Linear Models without Moments: Exercise . . . . 93
3.20.2 Nonlinear Improvement of OLSE for
Nonnormal Disturbances. . . . . . . . . . . . . . 93
3.20.3 A Characterization of the Least Squares
Estimator . . . . . . . . . . . . . . . . . . . . . . 94
3.20.4 A Characterization of the Least Squares
Estimator: A Lemma. . . . . . . . . . . . . . . . 94
3.21 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4 The Generalized Linear Regression Model 97