Regression Analysis Under A Priori Parameter Restrictions

244 Pages·2012·2.06 MB·English

by  Pavel S. Knopov, Arnold S. Korkhin (auth.)

Springer Optimization and Its Applications
VOLUME54
ManagingEditor
PanosM.Pardalos(UniversityofFlorida)
Editor–CombinatorialOptimization
Ding-ZhuDu(UniversityofTexasatDallas)
AdvisoryBoard
J.Birge(UniversityofChicago)
C.A.Floudas(PrincetonUniversity)
F.Giannessi(UniversityofPisa)
H.D.Sherali(VirginiaPolytechnicandStateUniversity)
T.Terlaky(McMasterUniversity)
Y.Ye(StanfordUniversity)
AimsandScope
Optimization has been expanding in all directions at an astonishing rate
during the last few decades. New algorithmic and theoretical techniques
havebeendeveloped,thediffusionintootherdisciplineshasproceededata
rapidpace,andourknowledgeofallaspectsofthefieldhasgrownevenmore
profound.Atthe sametime, oneofthe moststriking trendsin optimization
is the constantly increasing emphasis on the interdisciplinary nature of the
field.Optimizationhasbeenabasictoolinallareasofappliedmathematics,
engineering,medicine,economics,andothersciences.
The series Springer Optimization and Its Applications publishes under-
graduate and graduate textbooks, monographs and state-of-the-art exposi-
tory work that focus on algorithms for solving optimization problems and
alsostudyapplicationsinvolvingsuchproblems.Someofthetopicscovered
include nonlinear optimization (convex and nonconvex), network flow
problems, stochastic optimization, optimal control, discrete optimization,
multi-objectiveprogramming,descriptionofsoftwarepackages,approxima-
tiontechniquesandheuristicapproaches.
Forfurthervolumes:
http://www.springer.com/series/7393

Pavel S. Knopov • Arnold S. Korkhin
Regression Analysis Under
A Priori Parameter
Restrictions
123
PavelS.Knopov ArnoldS.Korkhin
DepartmentofMathematicalMethods DepartmentofEconomicalCybernetics
ofOperationResearch andInformationTechnology
V.M.GlushkovInstituteofCybernetics NationalMiningUniversity
NationalAcademyofScienceofUkraine 49005Dnepropetrovsk
03187Kiev Ukraine
Ukraine [email protected]
[email protected]
ISSN1931-6828
ISBN978-1-4614-0573-3 e-ISBN978-1-4614-0574-0
DOI10.1007/978-1-4614-0574-0
SpringerNewYorkDordrechtHeidelbergLondon
LibraryofCongressControlNumber:2011935145
©SpringerScience+BusinessMedia,LLC2012
Allrightsreserved.Thisworkmaynotbetranslatedorcopiedinwholeorinpartwithoutthewritten
permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York,
NY10013, USA),except forbrief excerpts inconnection with reviews orscholarly analysis. Usein
connectionwithanyformofinformationstorageandretrieval,electronicadaptation,computersoftware,
orbysimilarordissimilarmethodologynowknownorhereafterdevelopedisforbidden.
Theuseinthispublicationoftradenames,trademarks,servicemarks,andsimilarterms,eveniftheyare
notidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornottheyaresubject
toproprietaryrights.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
Preface
Regressionanalysishasquitealonghistory.Itisconventionaltothinkthatitgoes
back to the works of Gauss on approximation of experimental data. Nowadays,
regression analysis represents a separate scientific branch, which is based on
optimizationtheoryandmathematicalstatistics.Formally,thereexisttwobranches
ofregressionanalysis:theoreticalandapplied.
Up to recent time, developments in regression analysis were based on the
hypothesisthatthedomainofregressionparametershasnorestrictions.Divergence
from that approach came later on when equality constraints were taken into
account, which allowed use of some a priori information about the regression
model.Methodsof constructingthe regressionwith equality constraintswere first
investigatedinRao(1965)andBard(1974).
Usage of inequality constraints in a regression model gives much more pos-
sibilities to utilize available a priori information. Moreover, the representation of
theadmissibledomainofparametersintheformofinequalityconstraintsnaturally
includesthecaseswhenconstraintsaregivenasequalities.
Propertiesoftheregressionwithinequalityconstraintsareinvestigatedinmany
papers,inparticular,inZellner(1971),Liew(1976),NagarajandFuller(1991)and
ThomsonandSchmidt(1982),wheresomeparticularcasesareconsidered.Detailed
qualitativeanalysis of the propertiesof estimates in case of linear regressionwith
linearconstraintsisgiveninthemonograph(Malinvaud1969,Section9.8).
Asymptotic properties of the estimates of regression parameters in regression
withfinitenumberofparametersundersomeknownaprioriinformationarestudied
in Dupacovaand Wets (1986),Knopov(1997a–c),Korkhin(1985),Wang (1996),
etc. We note that the results obtained in Korkhin (1985) and Wang (1996) under
different initial assumptions, almost coincide. There are many results concerning
practical implementation of regression models with inequality constraints, for
example, Liew (1976), Rezk (1996) and McDonald (1999), Thomson (1982),
Thomson and Schmidt (1982). This problem was also studied in Gross (2003,
Subsection3.3.2).
Inthismonograph,wepresentinfulldetailtheresultsonestimationofunknown
parametersin regressionmodelsundera prioriinformation,describedin the form
v
vi Preface
ofinequalityconstraints.Thebookcoverstheproblemofestimationofregression
parametersas well as the problemof accuracyof such estimation. Both problems
are studied is cases of linear and nonlinear regressions. Moreover, we investigate
the applicability of regression with constraints to problems of point and interval
prediction.
Thebookisorganizedasfollows.
InChapter1,weconsidermethodsofcalculationofparameterestimatesinlinear
and nonlinear regression with constraints. In this chapter we describe methods
of solving optimization problems which take into account the specification of
regressionanalysis.
Chapter2isdevotedtoasymptoticpropertiesofregressionparametersestimates
inlinearandnonlinearregression.Bothcasesofequalityandinequalityconstraints
areconsidered.
In Chapter 3, we consider various generalizations of the estimation problem
by the least squaresmethod in nonlinearregressionwith inequality constraintson
parameters.Inparticular,wediscusstheresultsconcerningrobustHuberestimates
andregressorswhicharecontinuousfunctionsoftime.
Chapter 4 is devoted to the problem of accuracy estimation in (linear and
nonlinear)regression,whenparametersareestimatedbymeansoftheleastsquares
method.
InChapter5,wediscuss/considerstatisticalpropertiesofestimatesofparameters
innonlinearregression,whichareobtainedoneachiterationofthesolutiontothe
estimationproblem.HereweusealgorithmsdescribedinChap.1.Obtainedresults
mightbeusefulinpracticalimplementationofregressionanalysis.
Chapter 6 is devoted to problemsof predictionby linear regression with linear
constraints.
Kiev,Ukraine PavelS.Knopov
Dnepropetrovsk,Ukraine ArnoldS.Korkhin
Acknowledgments
Weareverygratefultothescientificeditorofthisbook,ProfessorPanosPardalos,
seniorpublishingeditor,ElizabethLoew,andtotheassociateeditorinmathematics,
Nathan Brothers, for their helpful support and collaboration in preparation of the
manuscript.
We thank our colleagues from V.M. Glushkov Institute of Cybernetics of
National Academy of Science of Ukraine for many helpful discussions on the
problemsandresultsdescribedandpresentedinthisbook.
We thank our colleagues L. Belyavina, L. Vovk, V. Knopova, Yu. Kolesnik,
E.Odinzova,forinvaluablehelpduringthepreparationofourbookforpublication.
vii

Contents
1 EstimationofRegressionModelParameters
withSpecificConstraints.................................................... 1
1.1 Estimationofthe ParametersofaLinearRegression
withInequalityConstraints............................................. 2
1.1.1 MethodofEstimatingtheSolutionto(1.7) ................... 2
1.1.2 AlgorithmofFindingtheSolutionto(1.9).................... 5
1.1.3 SpecialCaseoftheProblem(1.7) ............................. 6
1.2 Estimation of Parametersof Nonlinear Regression
withNonlinearInequalityConstraints................................. 10
1.2.1 StatementoftheProblemandaMethodofItsSolution...... 10
1.2.2 SolutiontotheAuxiliaryProblem............................. 19
1.2.3 CompatibilityofConstraintsintheAuxiliaryProblem....... 19
1.2.4 CalculationoftheConstants(cid:2) andı.......................... 24
1.3 EstimationofMultivariateLinearRegressionParameters
withNonlinearEqualityConstraints................................... 25
2 Asymptotic Properties of Parameters in Nonlinear
RegressionModels........................................................... 29
2.1 ConsistencyofEstimatesinNonlinearRegressionModels........... 29
2.2 Asymptotic Properties of Nonlinear Regression
ParametersEstimatesObtainedbytheLeastSquares
MethodUnderaPrioryInequalityConstraints(ConvexCase)....... 38
2.2.1 Introduction..................................................... 38
2.2.2 AuxiliaryResults ............................................... 40
2.2.3 FundamentalResults ........................................... 52
2.3 Asymptotic Properties of Nonlinear Regression
ParametersEstimates bythe LeastSquaresMethod
UnderaPrioryInequalityConstraints(Non-ConvexCase)........... 57
2.3.1 AssumptionsandAuxiliaryResults ........................... 57
2.3.2 FundamentalResult............................................. 58
ix
x Contents
2.4 Limit Distribution of the Estimate of Regression
ParametersWhichAreSubjecttoEqualityConstraints............... 61
2.5 AsymptoticPropertiesoftheLeastSquaresEstimates
ofParametersofaLinearRegressionwithNon-Stationary
VariablesUnderConvexRestrictionsonParameters.................. 64
2.5.1 Settings.......................................................... 64
2.5.2 ConsistencyofEstimator....................................... 65
2.5.3 LimitDistributionoftheParameterEstimate ................. 67
3 Method of Empirical Means in Nonlinear Regression
andStochasticOptimizationModels ...................................... 73
3.1 Consistency of Estimates Obtainedby the Method
of Empirical Means with IndependentOr Weakly
DependentObservations................................................ 74
3.2 RegressionModelsforLongMemorySystems ....................... 81
3.3 Statistical Methods in Stochastic Optimization
andEstimationProblems ............................................... 85
3.4 EmpiricalMeanEstimatesAsymptoticDistribution.................. 89
3.4.1 AsymptoticDistributionofEmpiricalEstimates
for Models with Independentand Weakly
DependentObservations........................................ 89
3.4.2 AsymptoticDistributionofEstimatesforLong
MemoryStochasticSystems ................................... 99
3.4.3 AsymptoticDistributionoftheLeastSquares
EstimatesforLongMemoryStochasticSystems ............. 101
3.5 LargeDeviationsofEmpiricalMeansin Estimation
andOptimizationProblems............................................. 104
3.5.1 LargeDeviationsoftheEmpiricalMeansMethod
forDependentObservations.................................... 104
3.5.2 Large Deviations of Empiric Estimates
forNon-StationaryObservations............................... 112
3.5.3 LargeDeviationsinNonlinearRegressionProblems......... 118
4 DeterminationofAccuracyofEstimationofRegression
ParametersUnderInequalityConstraints................................ 121
4.1 PreliminaryAnalysisoftheProblem................................... 121
4.2 Accuracy of Estimation of Nonlinear Regression
Parameters:TruncatedEstimates....................................... 123
4.3 DeterminationoftheTruncatedSampleMatrixofm.s.e.
oftheEstimateofParametersinNonlinearRegression............... 137
4.4 AccuracyofParameterEstimationinLinearRegression
withConstraintsandwithoutaTrend.................................. 138
4.4.1 AuxiliaryResults ............................................... 139
4.4.2 MainResults.................................................... 148
4.5 Determinationof AccuracyofEstimation ofLinear
RegressionParametersinRegressionwithTrend ..................... 154