A concise introduction to statistical inference

231 Pages·2014·1.416 MB·English

by  Thijssen J

A Concise
Introduction to
Statistical
Inference

A Concise
Introduction to
Statistical
Inference
Jacco Thijssen
The University of York, UK
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742
© 2014 by Taylor & Francis Group, LLC
CRC Press is an imprint of Taylor & Francis Group, an Informa business
No claim to original U.S. Government works
Printed on acid-free paper
Version Date: 20160610
International Standard Book Number-13: 978-1-4987-5596-2 (Hardback)
This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been
made to publish reliable data and information, but the author and publisher cannot assume responsibility for the valid-
ity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright
holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this
form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may
rectify in any future reprint.
Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or uti-
lized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopy-
ing, microfilming, and recording, or in any information storage or retrieval system, without written permission from the
publishers.
For permission to photocopy or use material electronically from this work, please access www.copyright.com (http://
www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers, MA 01923,
978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For
organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged.
Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for
identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
To my family

Contents
List of Figures xi
Preface xiii
Acknowledgments xix
1 Statistical Inference 1
1.1 What statistical inference is all about . . . . . . . . . . . . . 1
1.2 Why statistical inference is difficult . . . . . . . . . . . . . . 2
1.3 What kind of parameters are we interested in? . . . . . . . . 3
1.3.1 Center of a distribution . . . . . . . . . . . . . . . . . 4
1.3.2 Spread of a distribution . . . . . . . . . . . . . . . . . 5
1.3.3 Association between variables . . . . . . . . . . . . . 6
1.4 Statistics and probability . . . . . . . . . . . . . . . . . . . . 8
1.5 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Theory and Calculus of Probability 11
2.1 Probability models . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Random variables and their distributions . . . . . . . . . . . 15
2.3 Moments of random variables . . . . . . . . . . . . . . . . . 19
2.3.1 Expectation . . . . . . . . . . . . . . . . . . . . . . . 19
2.3.2 Higher order moments. . . . . . . . . . . . . . . . . . 21
2.3.3 Variance . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.4 Multivariate distributions . . . . . . . . . . . . . . . . . . . 23
2.4.1 Association between two random variables . . . . . . 24
2.5 Normal distribution . . . . . . . . . . . . . . . . . . . . . . . 25
2.5.1 Bivariate normal distribution . . . . . . . . . . . . . . 27
2.6 Limit theorems for the sample mean . . . . . . . . . . . . . 28
2.7 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 32
2.8 Exercises and problems . . . . . . . . . . . . . . . . . . . . . 32
3 From Probability to Statistics 39
3.1 A first stab at statistical inference . . . . . . . . . . . . . . . 40
3.2 Sampling distributions . . . . . . . . . . . . . . . . . . . . . 42
3.3 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4 Exercises and problems . . . . . . . . . . . . . . . . . . . . . 44
vii
viii Contents
4 Statistical Inference for the Mean based on a Large Sample 47
4.1 Simple statistical model for the mean . . . . . . . . . . . . . 47
4.2 Confidence intervals . . . . . . . . . . . . . . . . . . . . . . . 49
4.3 Hypothesis tests . . . . . . . . . . . . . . . . . . . . . . . . . 51
4.3.1 The p-value. . . . . . . . . . . . . . . . . . . . . . . . 52
4.3.2 Errors and power . . . . . . . . . . . . . . . . . . . . 54
4.4 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 Exercises and problems . . . . . . . . . . . . . . . . . . . . . 57
5 Statistical Models and Sampling Distributions 61
5.1 Statistical models . . . . . . . . . . . . . . . . . . . . . . . . 61
5.2 Some examples of statistical models . . . . . . . . . . . . . . 64
5.3 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
5.4 Sampling distributions . . . . . . . . . . . . . . . . . . . . . 68
5.5 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 72
5.6 Exercises and problems . . . . . . . . . . . . . . . . . . . . . 72
6 Estimation of Parameters 77
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.2 Maximum likelihood estimators . . . . . . . . . . . . . . . . 77
6.3 Comparing estimators . . . . . . . . . . . . . . . . . . . . . 82
6.3.1 Unbiased estimators . . . . . . . . . . . . . . . . . . . 82
6.3.2 Mean squared error . . . . . . . . . . . . . . . . . . . 86
6.4 Method of moments . . . . . . . . . . . . . . . . . . . . . . . 90
6.5 A useful asymptotic result . . . . . . . . . . . . . . . . . . . 92
6.6 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 93
6.7 Exercises and problems . . . . . . . . . . . . . . . . . . . . . 94
7 Confidence Intervals 99
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.2 Basic idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.3 Confidence intervals for means . . . . . . . . . . . . . . . . . 102
7.3.1 Mean of a normal population with variance known . . 102
7.3.2 Mean of a normal population with variance unknown 103
7.3.3 Meanofanunknowndistributionbasedonalargesam-
ple . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.4 Confidence interval for any parameter based on a large sample 105
7.5 Differences between populations based on large samples . . 106
7.5.1 Difference between population means . . . . . . . . . 106
7.5.2 Difference between proportions . . . . . . . . . . . . . 107
7.6 Confidence intervals for the variance of a normal sample . . 108
7.7 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 110
7.8 Exercises and problems . . . . . . . . . . . . . . . . . . . . . 110
Contents ix
8 Hypothesis Testing 117
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
8.2 Hypotheses, decisions, and errors . . . . . . . . . . . . . . . 117
8.3 What test statistic should we use? . . . . . . . . . . . . . . 122
8.4 Examples of commonly used tests . . . . . . . . . . . . . . . 126
8.4.1 Test of a normal mean with known variance . . . . . 127
8.4.2 Test of a normal mean with unknown variance . . . . 128
8.4.3 Test of a mean based on a large sample . . . . . . . . 130
8.4.4 Test of a proportion based on a large sample . . . . . 130
8.4.5 Test of the difference between two means based on a
large sample . . . . . . . . . . . . . . . . . . . . . . . 131
8.4.6 Testofthedifferencebetweentwoproportionsinalarge
sample . . . . . . . . . . . . . . . . . . . . . . . . . . 132
8.5 p-value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
8.6 Statistical significance . . . . . . . . . . . . . . . . . . . . . 134
8.7 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 136
8.8 Exercises, problems, and discussion . . . . . . . . . . . . . . 137
9 Linear Regression 145
9.1 Basic ideas . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
9.2 Estimation: Least squares method . . . . . . . . . . . . . . . 148
9.3 Decomposition of errors . . . . . . . . . . . . . . . . . . . . 150
9.4 Inferences based on the OLS estimator . . . . . . . . . . . . 151
9.5 Linear regression,causation, and correlation . . . . . . . . . 153
9.6 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 154
9.7 Exercises, problems, and discussion . . . . . . . . . . . . . . 155
10 Bayesian Inference 161
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
10.2 Basic concepts . . . . . . . . . . . . . . . . . . . . . . . . . . 162
10.3 A simple statistical model from a Bayesian perspective . . . 164
10.4 Highest density regions . . . . . . . . . . . . . . . . . . . . . 166
10.5 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . 167
10.6 Evidence and the likelihood principle . . . . . . . . . . . . . 170
10.7 Decision theory . . . . . . . . . . . . . . . . . . . . . . . . . 173
10.8 An economic decision problem with statistical information . 177
10.9 Linear regression . . . . . . . . . . . . . . . . . . . . . . . . 179
10.10Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . 180
10.11Exercises and problems . . . . . . . . . . . . . . . . . . . . . 180
Appendices 183
A Commonly used discrete distributions 185
B Commonly used continuous distributions 187