Table Of ContentNonparametric Regression
and
Generalized Linear Models
A ROUGHNESS PENALTY APPROACH
P.J. GREEN
and
B. W. SILVERMAN
School of Mathematics
University of Bristol
UK
CHAPMAN & HALL
London · Glasgow · Weinheim · New York · Tokyo · Melbourne · Madras
Published by Chapman & Hall, 2-6 Boundary Row, London SE1 SHN
Chapman & Hall, 2-6 Boundary Row, London SE1 8HN, UK
Blackie Academic & Professional, Wester Cleddens Road, Bishopbriggs,
Glasgow G64 2NZ, UK
Chapman & Hall GmbH, Pappelallee 3, 69469 Weinheim, Germany
Chapman & Hall USA, 115 Fifth Avenue, New York, NY 10003, USA
Chapman & Hall Japan, ITP-Japan, Kyowa Building, 3F, 2-2-1
Hirakawacho, Chiyoda-ku, Tokyo 102, Japan
Chapman & Hall Australia, 1 02 Dodds Street, South Melbourne,
Victoria 3205, Australia
Chapman & Hall India, R. Seshadri, 32 Second Main Road, CIT East,
Madras 600 035, India
First edition 1 994
Reprinted 1 995
© 1994 P.J. Green and B.W. Silverman
Printed in England by St Edmundsbury Press Ltd, Bury St Edmunds, Suffolk
ISBN 0 412 30040 0
Apart from any fair dealing for the purposes of research or private
study, or criticism or review, as permitted under the UK Copyright
Designs and Patents Act, 1988, this publication may not be
reproduced, stored, or transmitted, in any form or by any means,
without the prior permission in writing of the publishers, or in the case
of reprographic reproduction only in accordance with the terms of the
licences issued by the Copyright Licensing Agency in the UK, or in
accordance with the terms of licences issued by the appropriate
Reproduction Rights Organization outside the UK. Enquiries concerning
reproduction outside the terms stated here should be sent to the
publishers at the London address printed on this page.
The publisher makes no representation, express or implied, with
regard to the accuracy of the information contained in this book and
cannot accept any legal responsibility or liability for any errors or
omissions that may be made.
A catalogue record for this book is available from the British Library
Library of Congress Cataloging-in-Publication Data available
Contents
Preface xi
1 Introduction 1
1.1 Approaches to regression
1.1.1 Linear regression
1.1.2 Polynomial regression 2
1.2 Roughness penalties 2
1.2.1 The aims of curve fitting 2
1.2.2 Quantifying the roughness of a curve 4
1.2.3 Penalized least squares regression 5
1.3 Extensions of the roughness penalty approach 7
1.4 Computing the estimates 9
1.5 Further reading 9
2 Interpolating and smoothing splines 11
2.1 Cubic splines 11
2.1.1 What is a cubic spline? 11
2.1.2 The value-second derivative representation 12
2.2 Interpolating splines 13
2.2.1 Constructing the interpolating natural cubic spline 15
2.2.2 Optimality properties of the natural cubic spline
interpolant 16
2.3 Smoothing splines 17
2.3.1 Restricting the class of functions to be considered 18
2.3.2 Existence and uniqueness of the minimizing
spline curve 18
2.3.3 The Reinsch algorithm 19
2.3.4 Some concluding remarks 21
2.4 Plotting a natural cubic spline 22
vi CONTENTS
2.4.1 Constructing a cubic given values and second
derivatives at the ends of an interval 22
2.4.2 Plotting the entire cubic spline 23
2.5 Some background technical properties 24
2.5.l The key property for g to be a natural cubic spline 24
2.5.2 Expressions for the roughness penalty 24
2.6 Band matrix manipulations 25
2.6.1 The Cholesky decomposition 26
2.6.2 Solving linear equations 26
3 One-dimensional case: further topics 29
3.1 Choosing the smoothing parameter 29
3.2 Cross-validation 30
3.2.l Efficient calculation of the cross-validation score 31
3.2.2 Finding the diagonal elements of the hat matrix 33
3.3 Generalized cross-validation 35
3.3.1 The basic idea 35
3.3.2 Computational aspects 35
3.3.3 Leverage values 36
3.3.4 Degrees of freedom 37
3.4 Estimating the residual variance 38
3.4.1 Local differencing 38
3.4.2 Residual sum of squares about a fitted curve 39
3.4.3 Some comparisons 40
3.5 Weighted smoothing 40
3.5.l Basic properties of the weighted formulation 41
3.5.2 The Reinsch algorithm for weighted smoothing 41
3.5.3 Cross-validation for weighted smoothing 42
3.5.4 Tied design points 43
3.6 The basis functions approach 44
3.6.1 Details of the calculations 46
3.7 The equivalent kernel 47
3.7.l Roughness penalty and kernel methods 47
3.7.2 Approximating the weight function 47
3.8 The philosophical basis of roughness penalties 49
3.8.1 Penalized likelihood 50
3.8.2 The bounded roughness approach 51
3.8.3 The Bayesian approach 51
3.8.4 A finite-dimensional Bayesian formulation 54
3.8.5 Bayesian inference for functionals of the curve 55
3.9 Nonparametric Bayesian calibration 57
3.9.l The monotonicity constraint 58
CONTENTS vii
3.9.2 Accounting for the error in the prediction
observation 59
3.9.3 Considerations of efficiency 59
3.9.4 An application in forensic odontology 60
4 Partial splines 63
4.1 Introduction 63
4.2 The semiparametric formulation 64
4.3 Penalized least squares for semiparametric models 65
4.3.1 Incidence matrices 65
4.3.2 Characterizing the minimum 65
4.3.3 Uniqueness of the solution 66
4.3.4 Finding the solution in practice 68
4.3.5 A direct method 69
4.4 Cross-validation for partial spline models 70
4.5 A marketing example 71
4.5.1 A partial spline approach 72
4.5.2 Comparison with a parametric model 73
4.6 Application to agricultural field trials 75
4.6.1 A discrete roughness penalty approach 75
4.6.2 Two spring barley trials 77
4.7 The relation between weather and electricity sales 79
4.7.1 The observed data and the model assumed 79
4.7.2 Estimating the temperature response 81
4.8 Additive models 83
4.9 An alternative approach to partial spline fitting 85
4.9.1 Speckman's algorithm 85
4.9.2 Application: the marketing data 86
4.9.3 Comparison with the penalized least squares
method 86
5 Generalized linear models 89
5.1 Introduction 89
5.1.1 Unifying regression models 89
5.1.2 Extending the model 90
5.2 Generalized linear models 91
5.2.1 Exponential families 91
5.2.2 Maximum likelihood estimation 93
5.2.3 Fisher scoring 94
5.2.4 Iteratively reweighted least squares 95
5.2.5 Inference in GLMs 95
5.3 A first look at nonparametric GLMs 98
viii CONTENTS
5.3.l Relaxing parametric assumptions 98
5.3.2 Penalizing the log-likelihood 98
5.3.3 Finding the solution by Fisher scoring 99
5.3.4 Application: estimating actuarial death rates 101
5.4 Semiparametric generalized linear models 104
5.4.l Maximum penalized likelihood estimation 105
5.4.2 Finding maximum penalized likelihood estimates
by Fisher scoring 105
5.4.3 Cross-validation for GLMs 107
5.4.4 Inference in semiparametric GLMs 110
5.5 Application: tumour prevalence data 111
5.6 Generalized additive models 112
6 Extending the model 115
6.1 Introduction 115
6.2 The estimation of branching curves 115
6.2.l An experiment on sunflowers 116
6.2.2 The estimation method 116
6.2.3 Some results 118
6.3 Correlated responses and non-diagonal weights 120
6.4 Nonparametric link functions 121
6.4.1 Application to the tumour prevalence data 124
6.5 Composite likelihood function regression models 125
6.6 A varying coefficient model with censoring 126
6.7 Nonparametric quantile regression 129
6.7.1 The LMS method 129
6.7.2 Estimating the curves 130
6.8 Quasi-likelihood 134
7 Thin plate splines 137
7.1 Introduction 137
7.2 Basic definitions and properties 137
7.2.1 Quantifying the smoothness of a surface 138
7.3 Natural cubic splines revisited 139
7.4 Definition of thin plate splines 142
7.5 Interpolation 143
7.5.1 Constructing the interpolant 143
7.5.2 An optimality property 144
7.5.3 An example 144
7.6 Smoothing 147
7.6.1 Constructing the thin plate spline smoother 147
7.6.2 An example 148
CONTENTS ix
7.6.3 Non-singularity of the defining linear system 148
7.7 Finite window thin plate splines 150
7. 7.1 Formulation of the finite window problem 150
7.7.2 An example of finite window interpolation and
smoothing 151
7.7.3 Some mathematical details 153
7.8 Tensor product splines 155
7 .8.1 Constructing tensor products of one-dimensional
families 155
7.8.2 A basis function approach to finite window
roughness penalties 156
7.9 Higher order roughness functionals 159
7.9.l Higher order thin plate splines 160
8 Available software 163
8.1 Routines within the S language 163
8.1.1 The S routine smooth. spline 163
8.1.2 The new S modelling functions 164
8.2 The GCVPACK routines 165
8.2.1 Details of the algorithm 166
References 169
Author index 175
Subject index 177
Preface
In the last fifteen years there has been an upsurge of interest and activity in
the general area of nonparametric smoothing in statistics. Many methods
have been proposed and studied. Some of the most popular of these are
primarily 'data-analytic' in flavour and do not make particular use of
statistical models. The roughness penalty approach, on the other hand,
provides a 'bridge' towards classical and parametric statistics. It allows
smoothing to be incorporated in a natural way not only into regression
but also much more generally, for example into problems approached by
generalized linear modelling. In this monograph we have tried to convey
our personal view of the roughness penalty method, and to show how it
provides a unifying approach to a wide range of smoothing problems.
We hope that the book will be of interest both to those coming to
the area for the first time and to readers more familiar with the field.
While advanced mathematical ideas have been valuable in some of the
theoretical development, the methodological power of roughness penalty
methods can be demonstrated and discussed without the need for highly
technical mathematical concepts, and as far as possible we have aimed
to provide a self-contained treatment that depends only on simple linear
algebra and calculus.
For various ways in which they have helped us, we would like to
thank Tim Cole, John Gavin, Trevor Hastie, Guy Nason, Doug Nychka,
Christine Osborne, Glenn Stone, Rob Tibshirani and Brian Yandell.
Preliminary versions of parts of the book have been used as the basis of
courses given to successive classes of MSc students at the University of
Bath, and to graduate students at Stanford University and at the University
of Sao Paulo; in all cases the feedback and reactions have been most
helpful. The book was mainly written while one ofus (BWS) held a post
at the University of Bath, and he would like to pay grateful tribute to the
intellectual and material environment provided there.
Peter Green and Bernard Silverman
Bristol, September 1993
