Bootstrap vs Permutation Tests: Two Resampling Ideas That Answer Different Questions

Bootstrap and permutation procedures are both resampling methods, but they are built for different inferential tasks. The bootstrap typically uses repeated sampling with replacement to approximate the sampling distribution of a statistic and support estimation, standard errors, and confidence intervals. A permutation test instead rearranges observations or labels in a way that represents the null hypothesis, producing a null distribution against which the observed statistic is compared (Devore & Berk, 2012; Moore et al., 2014, 2021).

The computational resemblance between the methods should not obscure this difference. Bootstrap and permutation procedures are not interchangeable.

What Is Resampling?

Classical statistical inference depends on sampling distributions: distributions describing how statistics would vary across repeated samples generated under specified conditions. Resampling methods use the observed data and repeated computer-generated rearrangements or samples to approximate distributions needed for inference (Moore et al., 2014, 2021).

So “resampling” describes a broad computational strategy. It does not identify the inferential question being answered.

The Bootstrap Method: Approximate the Sampling Distribution

Suppose a researcher observes a sample of n cases and calculates a statistic such as a mean, median, correlation, or difference between group means. The researcher has only one realized sample but wants to understand how the statistic would vary across repeated samples from the population.

The bootstrap method in statistics approximates that repeated-sampling behavior by treating the observed sample as a stand-in for the population and repeatedly drawing new samples from it (Lovric, 2011; Moore et al., 2021).

What Is a Bootstrap Sample?

For the ordinary nonparametric bootstrap:

1. Start with the observed sample of n cases.
2. Randomly draw n cases with replacement from that sample.
3. Calculate the statistic of interest from the resulting bootstrap sample.
4. Repeat the procedure many times (Moore et al., 2014, 2021).

Sampling with replacement matters. After an observation is selected, it remains available for selection again. A bootstrap sample can therefore contain the same original observation several times while omitting another observation entirely (Devore & Berk, 2012; Moore et al., 2021).

The bootstrap sample normally has the same size as the original sample (Moore et al., 2014, 2021).

What Is a Bootstrap Distribution?

Calculate the statistic separately for every bootstrap sample. The resulting collection of bootstrap statistics forms the bootstrap distribution (Moore et al., 2014, 2021).

Its purpose is to approximate the sampling distribution of the statistic. Its standard deviation provides a bootstrap estimate of the statistic's standard error (Moore et al., 2014, 2021).

Bootstrap Estimation and Confidence Intervals

A major reason to bootstrap is estimation. Once a bootstrap distribution has been generated, researchers can use its spread and shape to assess uncertainty around the statistic and construct a bootstrap confidence interval (Devore & Berk, 2012; Moore et al., 2014, 2021).

For example, the standard deviation of the bootstrap distribution can estimate the standard error. Bootstrap procedures can also construct intervals from the distribution itself. Moore et al. discuss bootstrap t, percentile, and bias-corrected and accelerated (BCa) intervals, emphasizing that the shape and bias of the bootstrap distribution matter when selecting an interval procedure (Moore et al., 2014, 2021).

A 95% percentile interval, for example, uses the 2.5th and 97.5th percentiles of the bootstrap distribution. Moore et al. caution that simple bootstrap procedures can be inaccurate when bias, skewness, or small samples create problems; their treatment recommends examining the bootstrap distribution rather than assuming that resampling automatically solves every inferential difficulty (Moore et al., 2014, 2021).

Permutation Tests: Build a Distribution Under the Null

A permutation test in statistics begins from a hypothesis-testing question rather than an estimation question.

The general testing logic is familiar: calculate a statistic from the observed data and ask how extreme that statistic would be if the null hypothesis were true. The distinctive feature of a permutation test is how the null distribution is obtained (Lovric, 2011; Moore et al., 2021).

For a two-group comparison in which the null hypothesis permits group membership to be rearranged, a permutation procedure can:

1. Calculate the observed group difference or another suitable test statistic.
2. Rearrange the observations among groups in a way permitted under the null hypothesis.
3. Recalculate the statistic.
4. Repeat across all permissible arrangements, or a large random sample of them.
5. Compare the observed statistic with the resulting permutation distribution (Devore & Berk, 2012; Lovric, 2011; Moore et al., 2021).

The proportion of null-consistent permutations producing a statistic at least as extreme as the observed statistic provides the basis for the permutation p-value (Devore & Berk, 2012; Lovric, 2011).

The Permutation Distribution Is a Null Distribution

The conceptual center of permutation testing is the null distribution.

Bootstrap resampling ordinarily preserves the empirical structure of the sample so that the resulting distribution can approximate the sampling behavior of an estimator. A permutation test deliberately resamples according to what should be possible if the null hypothesis holds (Moore et al., 2021).

For a simple two-group problem, that may mean pooling the observations and repeatedly reallocating them to groups without replacement. For paired observations, the permissible permutations must respect the paired design. For testing no relationship between two variables, one variable can be reassigned relative to the other when that rearrangement represents the null hypothesis (Moore et al., 2021).

Randomization, Exchangeability, and What May Be Permuted

Permutation logic depends on the legitimacy of the rearrangements being used.

Exchangeability provides a useful general concept: observations are exchangeable when their joint distribution is invariant to permutation, so their ordering can be changed without changing that joint distribution (Lovric, 2011). In permutation inference, the relevant question is whether the particular observations, outcomes, or labels being rearranged can legitimately be treated as interchangeable under the null hypothesis or under the study's randomization mechanism.

For a standard independent two-sample permutation test, Devore and Berk state the null condition in terms of the two population distributions being identical, so that group labels are arbitrary under the null. They caution that a permutation procedure is not assumption-free: differences in distributional shape or spread can matter when a test is being interpreted specifically as a test of equal means (Devore & Berk, 2012).

Randomized experiments provide another basis for permutation logic. When treatments were assigned randomly, the resampling procedure can reproduce permissible treatment reallocations under the randomization scheme. The International Encyclopedia of Statistical Science notes that randomization-based permutation inference can rely on random assignment of treatments to experimental units rather than requiring random sampling from populations (Lovric, 2011).

One Synthetic Dataset, Two Different Analyses

The observed means are 10 for Group A and 6 for Group B, giving an observed difference:

x̄_A − x̄_B = 4.

The data do not determine whether the researcher should “use bootstrap or permutation.” The research question does.

Bootstrap analysis: How uncertain is the estimated difference?

Suppose the question is:

What is the estimated difference between the two population means, and how uncertain is that estimate?

A two-sample bootstrap would resample within each group separately and with replacement. One bootstrap iteration might produce something conceptually like:

• Group A*: 8, 10, 10, 12, 12
• Group B*: 4, 4, 6, 7, 8

The researcher calculates the difference between the two bootstrap means. Another bootstrap iteration produces another pair of samples and another difference. Repeating this process many times generates a bootstrap distribution of the estimated mean difference (Devore & Berk, 2012; Moore et al., 2021).

That distribution is used to quantify the sampling uncertainty around the observed estimate of 4 and, with an appropriate bootstrap interval procedure, construct a confidence interval.

Notice what has not happened: the researcher has not imposed a null hypothesis that the groups are identical. Group membership remains part of the observed data structure, and each group is resampled separately.

Permutation analysis: How unusual is a difference of 4 under the null?

Now suppose the question is:

If group membership made no difference under the specified null hypothesis, how unusual would an observed difference in means of 4 be?

For a simple two-group permutation test where the null makes the labels legitimately interchangeable, combine the ten outcome values:

4, 5, 6, 7, 8, 8, 9, 10, 11, 12.

Then repeatedly assign five observations to Group A and the remaining five to Group B without replacement, calculating the difference in means each time. The resulting permutation distribution represents differences that could arise under the null-consistent rearrangement (Devore & Berk, 2012; Moore et al., 2021).

The observed difference of 4 is then located in this null distribution. The permutation p-value depends on how many permissible rearrangements produce a statistic at least as extreme as the observed one in the direction or directions specified by the alternative hypothesis (Lovric, 2011; Moore et al., 2021).

That is why bootstrap and permutation tests should not be treated as interchangeable versions of the same procedure.

Bootstrap or Permutation? A Decision Table

The distinctions in this table follow the different resampling constructions described by Devore and Berk (2012), Lovric (2011), and Moore et al. (2014, 2021).

Common Mistakes

A Practical Workflow

1. Define the statistical question

   Ask whether the main objective is estimation or testing a specified null hypothesis.

   If the question is “How precisely have we estimated this parameter?”, a bootstrap approach may be appropriate. If the question is “How extreme is this statistic under a defensible null-based rearrangement?”, consider a permutation test (Devore & Berk, 2012; Moore et al., 2021).

2. Identify the statistic

   Specify the quantity being estimated or tested: a mean, difference in means, correlation, median, or another statistic. Both bootstrap and permutation methods can accommodate many statistics, but the statistic must correspond to the substantive research question (Lovric, 2011; Moore et al., 2021).

3. Match resampling to the design

   Do not ignore pairing, group structure, or random assignment. Bootstrap and permutation schemes must reflect the relevant structure of the data and study design (Moore et al., 2021).

4. For a bootstrap, examine the bootstrap distribution

   Inspect its center, spread, shape, and evidence of bias. The appropriate confidence-interval procedure can depend on these features, and small samples can limit bootstrap accuracy (Moore et al., 2014, 2021).

5. For a permutation test, define the null before shuffling

   State what the null hypothesis implies about the observations or labels. Then construct only those permutations that are legitimate under that null and the research design (Devore & Berk, 2012; Lovric, 2011; Moore et al., 2021).

6. Interpret the result according to the question asked

   A bootstrap confidence interval describes uncertainty in an estimated parameter under the bootstrap procedure. A permutation p-value describes how extreme the observed statistic is relative to the null-consistent permutation distribution.

   Do not translate one automatically into the interpretation of the other.

Limitations and Cautions

Bootstrap accuracy depends on whether the observed sample and chosen resampling mechanism adequately approximate the relevant sampling process. Small samples, bias, skewness, dependence, or other data structures may require modifications or may limit the quality of ordinary bootstrap inference (Lovric, 2011; Moore et al., 2014, 2021).

Permutation inference depends on whether the proposed rearrangements are legitimate under the null hypothesis or randomization mechanism. In a simple two-sample setting, interpreting a permutation procedure narrowly as a test of equal means can require stronger distributional equality conditions than the phrase “nonparametric test” might suggest (Devore & Berk, 2012).

Neither method repairs weaknesses in the underlying research design. Resampling changes how a reference distribution is approximated; it does not by itself justify causal conclusions, correct selection bias, or eliminate confounding (Moore et al., 2014, 2021).

Conclusion

The clearest way to understand bootstrap vs permutation test methods is to ask what distribution each procedure is trying to create.

The bootstrap method in statistics repeatedly samples with replacement to approximate the sampling distribution of an estimator. That makes bootstrap methods especially useful for standard errors, estimation, and bootstrap confidence intervals (Lovric, 2011; Moore et al., 2014, 2021).

A permutation test in statistics instead rearranges data in a way justified by the null hypothesis and study design. Its permutation distribution represents what the test statistic could look like under that null, allowing the observed statistic to be evaluated through a p-value (Devore & Berk, 2012; Lovric, 2011; Moore et al., 2021).

Both belong to the broader family of resampling methods. But sharing a computational family does not mean sharing an inferential purpose.

Choose between them by starting with the statistical question—not by starting with the fact that both involve resampling.

References

Devore, J. L., & Berk, K. N. (2012). Modern mathematical statistics with applications (2nd ed.). Springer.

Lovric, M. (Ed.). (2011). International encyclopedia of statistical science. Springer. https://doi.org/10.1007/978-3-642-04898-2

Moore, D. S., McCabe, G. P., & Craig, B. A. (2014). Introduction to the practice of statistics (8th ed.). W. H. Freeman.

Moore, D. S., McCabe, G. P., & Craig, B. A. (2021). Introduction to the practice of statistics (10th ed.). Macmillan Learning.