Probability and Statistical Inference, Second Edition

655 Pages·2008·15.383 MB·English

by  Robert Bartoszy?ski, Magdalena Niewiadomska?Bugaj(auth.)

PROBABILITY AND
STATISTICAL INFERENCE
THE WILEY BICENTENNIAL-KNOWLEDGE FOR GENERATIONS
G
ach generation has its unique needs and aspirations. When Charles Wiley first
opened his small printing shop in lower Manhattan in 1807, it was a gencration
of boundless potential searching for an identity. And we were there. helping to
define a new American literary tradition. Over half a century later, in the midst
of thc Second Industrial Revolution, it was a generation focused on building the
future. Once again, we were there, supplying the critical scientific, technical, and
engineering knowledge that helped frame the world. Throughout the 20th
Century, and into the new millennium, nations began to reach out beyond their
own borders and a new inter~iationalc ommunity was born. Wiley was there,
expanding its operations around the world to enable a global exchange of ideas,
opinions, and know-how.
For 200 years, Wiley has been an integral part of each generation's journey,
enabling the flow of inforniation and understanding necessary to meet their needs
and fulfill their aspirations. Today, bold new technologies are changing the way
we live and learn. Wiley will be there, providing you the must-have knowledge
you need to imagine new worlds, new possibilities, and new opportunities.
Generations come and go, but you can always count on Wiley to provide you the
knowledge you need, when and where you need it!
*
n
WILLIAM J. PESCE PETERB OOTH WILEY
PRESIDENT AND CHIEF EXECMVE OFFICER CHAIRMAN OF THE BOARD
PROBABILITY AND
STATISTICAL INFERENCE
Second Edition
Robert Bartoszynski
Magdalena Niewiadomska-Bugaj
BICENTENNIAL
BICENTTNNIAL
WILEY-INTERSCIENCE
A John Wiley & Sons, Inc., Publication
Copyright 0 2008 by John Wiley & Sons, Inc. All rights reserved.
Published by John Wiley & Sons, Inc., Hoboken, New Jersey
Published simultaneously in Canada.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form
or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as
permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior
written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to
the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, [phone], fax
[phone], or on the web at www.copyright.com. Requests to the Publisher for permission should
be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 I River Street, Hoboken, NJ
07030, (201) 748-601 I, fax [phone], or online at http:l/www.wiley.com/go/permission.
Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in
preparing this book, they make no representations or warranties with respect to the accuracy or
completeness of the contents of this book and specifically disclaim any implied warranties of
merchantability or fitness for a particular purpose. No warranty may be created or extended by sales
representatives or written sales materials. The advice and strategies contained herein may not be
suitable for your situation. You should consult with a professional where appropriate. Neither the
publisher nor author shall be liable for any loss of profit or any other commercial damages, including
but not limited to special, incidental, consequential, or other damages.
For general information on our other products and services or for technical support, please contact our
Customer Care Department within the United States at [phone], outside the United States at
[phone] or fax [phone].
Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may
not be available in electronic format. For information about Wiley products, visit our web site at
www.wiley.com.
Wiley Bicentennial Logo: Richard J. Pacific0
Library of Congress Cataloging-in-Publication Data is available.
ISBN 978-0-47 1-69693-3
Printed in the United States of America.
10 9 8 7 6 5 4 3 2 1
CONTENTS
Preface X
1 Experiments, Sample Spaces, and Events 1
1.1 Introduction 1
1.2 Sample Space 2
1.3 Algebra of Events 9
1.4 Infinite Operations on Events 16
25
Probability
2.1 Introduction 25
2.2 Probability as a Frequency 25
2.3 Axioms of Probability 26
2.4 Consequences of the Axioms 31
2.5 Classical Probability 36
2.6 Necessity of the Axioms* 37
42
2.7 Subjective Probability*
V
vi CONTENTS
3 Counting 47
3.1 Introduction 47
3.2 Product Sets, Orderings, and Permutations 47
3.3 Binomial Coefficients 55
3.4 Extension of Newton’s Formula 68
3.5 hlultinomial Coefficients 69
4 Conditional Probability: Independence 73
4.1 Introduction 73
4.2 Conditional Probability 74
4.3 Partitions; Total Probability Formula 80
4.4 Bayes’ Formula 87
4.5 Independence 92
4.6 Exchangeability; Conditional Independence 99
5 Markov Chains* 103
5.1 Introduction and Basic Definitions 103
5.2 Definition of a Markov Chain 106
5.3 n-Step Transition Probabilities 111
5.4 The Ergodic Theorem 114
5.5 Absorption Probabilities 122
6 Random Variables: Univariate Case 125
6.1 Introduction 125
6.2 Distributions of Random Variables 126
6.3 Discrete and Continuous Random Variables 139
6.4 Functions of Random Variables 150
6.5 Survival and Hazard Functions 157
7 Random Variables: Multivariate Case 161
7.1 Bivariate Distributions 161
7.2 Marginal Distributions; Independence 168
7.3 Conditional Distributions 180
7.4 Bivariate Transformations 187
7.5 Multidimensional Distributions 196
8 Expectation 203
8.1 Introduction 203
8.2 Expected Value 204
CONTENTS vii
8.3 Expectation as an Integral* 212
8.4 Properties of Expectation 220
8.5 Moments 228
8.6 Variance 236
8.7 Conditional Expectation 248
8.8 Inequalities 252
257
9 Selected Families of Distributions
9.1 Bernoulli Trials and Related Distributions 257
9.2 Hypergeometric Distribution 2 70
9.3 Poisson Distribution and Poisson Process 2 76
9.4 Exponential, Gamma and Related Distributions 290
9.5 Normal Distribution 296
9.6 Beta Distribution 306
10 Random Samples 311
10.1 Statistics and their Distributions 311
10.2 Distributions Related to Normal 313
10.3 Order Statistics 319
10.4 Generating Random Samples 325
10.5 Convergence 330
10.6 Central Limit Theorem 342
11 Introduction to Statistical Inference 351
11.1 Overview 351
11.2 Descriptive Statistics 353
11.3 Basic Model 358
11.4 Bayesian Statistics 360
11.5 Sampling 360
11.6 Measurement Scales 367
12 Estimation 373
12.1 Introduction 373
12.2 Consistency 378
12.3 Loss, Risk, and Admissibility 381
12.4 Efficiency 386
12.5 Methods of Obtaining Estimators 394
12.6 Sufficiency 424
12.7 Interval Estimation 440
viii CONTENTS
13 Testing Statistical Hypotheses 455
13.1 Introduction 455
13.2 Intuitive Background 460
13.3 Most Powerful Tests 469
13.4 Uniformly Most Powerful Tests 48 1
13.5 Unbiased Tests 487
13.6 Generalized Likelihood Ratio Tests 491
13.7 Conditional Tests 498
13.8 Tests and Confidence Intervals 501
13.9 Review of Tests for Normal Distributions 502
13.10 Monte Carlo, Bootstrap, and Permutation Tests 512
14 Linear Models 517
14.1 Introduction 517
14.2 Regression of the First and Second Kind 519
14.3 Distributional Assumptions 525
14.4 Linear Regression in the Normal Case 528
14.5 Testing Linearity 535
14.6 Prediction 538
14.7 Inverse Regression 540
14.8 BLUE 542
14.9 Regression Toward the Mean 545
14.10 Analysis of Variance 546
14.11 One-way Layout 547
14.12 Two-way Layout 550
14.13 ANOVA Models with Interaction 553
14.14 Further Extensions 557
15 Rank Methods 559
15.1 Introduction 559
15.2 Glivenko-Cantelli Theorem 560
15.3 Kolmogorov-Smirnov Tests 564
15.4 One-Sample Rank Tests 571
15.5 Two-Sample Rank Tests 578
15.6 Kruskal-Wallis Test 582
16 Analysis of Categorical Data 585
16.1 Introduction 585
16.2 Chi-square Tests 587
CONTENTS ix
16.3 Homogeneity and Independence 593
16.4 Consistency and Power 599
16.5 2x2 Contingency Tables 604
16.6 r x c Contingency Tables 612
Statistical Tables 617
Bibliography 629
Answers to Odd-Numbered Problems 634
Index 642
Preface
The first edition ofthis book was published in 1996. Since then, powerful computers
have come into wide use, and it became clear that our text should be revised and
material on computer-intensive methods of statistical inference should be added. To
my delight, Steve Quigley, Executive Editor of John Wiley and Sons, agreed with
the idea, and work on the second edition began.
Unfortunately, Robert Bartoszynski passed away in 1998, so I was left to carry
out this revision by myself. I revised the content by creating a new chapter on
random samples, adding sections on Monte Carlo methods, bootstrap estimators
and tests, and permutation tests. More problems were added, and existing ones were
reorganized. Hopefully nothing was lost of the “spirit” of the book which Robert
liked so much and of which he was very proud.
This book is intended for seniors or first-year graduate students in statistics, math-
ematics, natural sciences, engineering, and any other major where an intensive ex-
posure to statistics is necessary. The prerequisite is a calculus sequence that includes
multivariate calculus. We provide the material for a two-semester course that starts
with the necessary background in probability theory, followed by the theory of statis-
tics.
What distinguishes our book from other texts is the way the material is presented
and the aspects that are stressed. To put it succinctly, understanding “why” is pri-
oritized over the skill of “how to.” Today, in an era of undreamed-of computational
facilities, a reflection in an attempt to understand is not a luxury but a necessity.
Probability theory and statistics are presented as self-contained conceptual struc-
tures. Their value as a means of description and inference about real-life situations
X