3864 words - 16 pages

Bootstrapping Regression Models

Appendix to An R and S-PLUS Companion to Applied Regression

John Fox

January 2002

1

Basic Ideas

Bootstrapping is a general approach to statistical inference based on building a sampling distribution for

a statistic by resampling from the data at hand. The term ‘bootstrapping,’ due to Efron (1979), is an

allusion to the expression ‘pulling oneself up by one’s bootstraps’ – in this case, using the sample data as

a population from which repeated samples are drawn. At ﬁrst blush, the approach seems circular, but has

been shown to be sound.

Two S libraries for bootstrapping are associated with extensive treatments of the subject: Efron and

...view middle of the document...

A traditional approach to statistical inference is

to make assumptions about the structure of the population (e.g., an assumption of normality), and, along

with the stipulation of random sampling, to use these assumptions to derive the sampling distribution of T ,

on which classical inference is based. In certain instances, the exact distribution of T may be intractable,

and so we instead derive its asymptotic distribution. This familiar approach has two potentially important

deﬁciencies:

1. If the assumptions about the population are wrong, then the corresponding sampling distribution of

the statistic may be seriously inaccurate. On the other hand, if asymptotic results are relied upon,

these may not hold to the required level of accuracy in a relatively small sample.

2. The approach requires suﬃcient mathematical prowess to derive the sampling distribution of the statistic of interest. In some cases, such a derivation may be prohibitively diﬃcult.

In contrast, the nonparametric bootstrap allows us to estimate the sampling distribution of a statistic

empirically without making assumptions about the form of the population, and without deriving the sampling

distribution explicitly. The essential idea of the nonparametric bootstrap is as follows: We proceed to draw a

sample of size n from among the elements of S, sampling with replacement. Call the resulting bootstrap sample

∗

∗

∗

S∗ = {X11 , X12 , ..., X1n }. It is necessary to sample with replacement, because we would otherwise simply

1

reproduce the original sample S. In eﬀect, we are treating the sample S as an estimate of the population

1 Alternatively,

P could be an inﬁnite population, speciﬁed, for example, by a probability distribution function.

1

P; that is, each element Xi of S is selected for the bootstrap sample with probability 1/n, mimicking the

original selection of the sample S from the population P. We repeat this procedure a large number of times,

∗

∗

∗

R, selecting many bootstrap samples; the bth such bootstrap sample is denoted S∗ = {Xb1 , Xb2 , ..., Xbn }.

b

The key bootstrap analogy is therefore as follows:

The population is to the sample

as

the sample is to the bootstrap samples.

∗

Next, we compute the statistic T for each of the bootstrap samples; that is Tb = t(S∗ ). Then the

b

∗

distribution of Tb around the original estimate T is analogous to the sampling distribution of the estimator

T around the population parameter θ. For example, the average of the bootstrapped statistics,

∗

T = E ∗ (T ∗ ) =

R

b=1

R

∗

Tb

∗

estimates the expectation of the bootstrapped statistics; then B ∗ = T − T is an estimate of the bias of T ,

that is, T − θ. Similarly, the estimated bootstrap variance of T ∗ ,

V ∗ (T ∗ ) =

R

∗

b=1 (Tb

∗

− T )2

R−1

estimates the sampling variance of T .

The random selection of bootstrap samples is not an essential aspect of the nonparametric bootstrap:

At least in principle, we...

Beat writer's block and start your paper with the best examples