Bayesian vs Frequentist Statistics in Medical Research: What Researchers Need to Understand
Understand how Bayesian and frequentist statistics differ in medical research, including confidence and credible intervals, priors, posterior inference, prediction, hierarchical models, model checking, and clinical trials. The Resource focuses on matching the inferential framework to the actual scientific or decision problem.
The difference between Bayesian vs frequentist statistics is often reduced to two slogans:
Bayesian methods
“Bayesian methods use priors.”
Frequentist methods
“Frequentist methods use P values.”
Both statements point to real features, but neither is an adequate guide to statistical reasoning in medical research.
The more useful distinction concerns what probability statements are being made, what inferential question is being answered, and how uncertainty is represented. Frequentist inference evaluates procedures through sampling behavior under repeated hypothetical observations or experiments. Bayesian inference uses probability distributions to represent uncertainty about unknown quantities and updates those distributions using observed data. Both depend on probability models, and both require researchers to think carefully about study design, model assumptions, estimation, and interpretation (Gelman et al., 2014; Piantadosi, 2005).
For medical researchers, the practical question is not “Which statistical philosophy wins?”
It is: What inferential or decision problem do we have, and what does the chosen analysis actually allow us to conclude?
The central distinction is the inferential question
Suppose a clinical study is estimating a treatment effect, denoted by θ.
Frequentist framing
A frequentist analysis treats θ as an unknown fixed quantity and evaluates statistical procedures through the distribution of possible data or estimates under repeated sampling.
Confidence intervals and hypothesis tests are built from this sampling framework (Altman, 1991; Piantadosi, 2005).
Bayesian framing
A Bayesian analysis places a probability distribution on the unknown parameter and updates that distribution after observing the data.
The resulting posterior distribution becomes the basis for inference about θ, prediction, and—when utilities or losses are supplied—formal decisions (Gelman et al., 2014).
This difference matters because apparently similar numerical summaries can answer different questions.
A frequentist confidence interval is constructed from a procedure with specified sampling properties. A Bayesian posterior interval summarizes probability under the posterior distribution. These should not be treated as interchangeable simply because both are often reported as intervals around an estimated effect (Altman, 1991; Gelman et al., 2014).
Frequentist inference: data viewed through repeated sampling
The conventional statistical framework used throughout much medical research is based on sampling distributions.
Altman develops estimation by considering how a statistic would vary across repeated samples from the relevant population. Standard errors quantify this sampling variability, and confidence intervals use that sampling distribution to express uncertainty around an estimate (Altman, 1991).
Piantadosi similarly characterizes the frequentist perspective in clinical trials as appealing to hypothetical repetition of experiments and to operating characteristics such as Type I error control. He also emphasizes that frequentist methods remain predominant in the clinical-trial literature largely for historical and practical reasons rather than because alternative perspectives are inherently invalid (Piantadosi, 2005).
Hypothesis testing
In a conventional hypothesis test, the researcher specifies a null hypothesis and evaluates how the observed data compare with what the probability model predicts under that hypothesis.
The resulting significance calculation is therefore tied to the behavior of possible data under the null model. It is not automatically a probability statement about whether the null hypothesis itself is true.
Interpretation caution: Medical researchers should not reduce frequentist inference to “P < .05” decisions. Altman's broader emphasis is on estimation and uncertainty: an effect estimate accompanied by a confidence interval provides information about magnitude and precision that a binary significance classification does not provide (Altman, 1991).
Confidence intervals
A confidence interval is rooted in the sampling distribution of the estimator. Altman explains the interval through repeated samples: a 95% confidence procedure is constructed so that the resulting intervals include the true population value on approximately 95% of repeated applications under the relevant assumptions (Altman, 1991).
The important practical questions are therefore:
- What effect was estimated?
- How precise is that estimate?
- Which clinically relevant values remain compatible with the uncertainty represented by the interval?
A frequentist confidence interval should not simply be relabeled as a posterior probability statement.
Bayesian inference: probability distributions for unknown quantities
Bayesian analysis begins with a probability model for the unknown parameters as well as a model for the observed data.
At its core are three components:
Prior + likelihood → posterior
More formally, the posterior distribution is proportional to the prior distribution multiplied by the likelihood:
p(θ | y) ∝ p(θ)p(y | θ).
The likelihood describes how the observed data are modeled conditional on the parameter. The prior describes uncertainty about the parameter before incorporating the current observations. Bayes' rule combines the two into the posterior distribution, which represents updated uncertainty conditional on the model and observed data (Gelman et al., 2014).
Gelman et al. show this structure throughout Bayesian modeling; for example, conditional regression inference takes the form:
p(θ | X, y) ∝ p(θ)p(y | X, θ).
(Gelman et al., 2014).
The prior is not a decorative addition
The statement that “Bayesian methods use priors” is true but incomplete.
A prior is part of the probability model. Depending on the problem, it can encode substantive prior information or provide regularization. Its influence depends on both the prior and the information in the likelihood.
Gelman et al. caution against assuming that there is always an obvious universal “noninformative” prior. A distribution that appears flat under one parameterization need not be flat under another, and inappropriate default choices can create problems, particularly in more complex models (Gelman et al., 2014).
Prior choice creates a modeling responsibility rather than a license to insert arbitrary beliefs.
When data are highly informative relative to the prior, posterior inference can be relatively insensitive to a range of reasonable priors. When information is limited or the model has many parameters, prior specification can become substantially more consequential. Gelman et al. specifically note that priors and hierarchical modeling become more important with smaller samples or larger numbers of parameters (Gelman et al., 2014).
What does the posterior distribution provide?
The posterior distribution is not merely a Bayesian substitute for a point estimate.
It describes uncertainty about unknown quantities conditional on the specified model, prior, and observed data. Researchers can summarize it using posterior means, medians, quantiles, intervals, probabilities, or simulations, depending on the scientific question (Gelman et al., 2014).
This makes questions such as the following natural within the Bayesian framework:
- What posterior probability lies above a scientifically important treatment-effect threshold?
- What range contains a specified proportion of posterior probability?
- What outcomes should we predict for future observations?
These are different from asking how a test statistic would behave across repeated hypothetical samples.
Bayesian credible intervals and frequentist confidence intervals are not the same object
This is one of the most important distinctions for researchers.
| Question | Frequentist confidence interval | Bayesian posterior/credible interval |
|---|---|---|
| What generates the uncertainty statement? | Sampling distribution of an estimator/procedure | Posterior distribution conditional on data and model |
| Is the parameter represented probabilistically? | Conventionally treated as fixed | Represented by a posterior probability distribution |
| What does the interval summarize? | A procedure with specified repeated-sampling coverage | A specified amount of posterior probability |
| Does prior specification enter? | Not as a Bayesian prior | Yes |
| Can the observed interval be interpreted simply as a posterior probability statement? | No | Probability statements refer directly to the posterior distribution |
(Altman, 1991; Gelman et al., 2014).
Gelman et al. routinely summarize posterior distributions with posterior intervals—for example, reporting central intervals from simulations of a hierarchical model (Gelman et al., 2014).
The key interpretive discipline is to name the interval correctly and explain the probability framework that generated it.
Bayesian analysis is also predictive
Bayesian inference naturally extends from unknown parameters to unobserved or future data.
Rather than stopping at p(θ | y), researchers can construct a posterior predictive distribution by averaging predictions over uncertainty in the parameters.
This is important because many medical questions are ultimately predictive:
- What might happen to a future patient?
- What outcomes would the fitted model generate?
- If the trial continued, what future evidence might emerge?
Posterior predictive simulation is also central to Bayesian model checking. Gelman et al. compare observed data with replicated data generated under the fitted model to investigate whether the model can reproduce features of the observations that matter scientifically or diagnostically (Gelman et al., 2014).
Piantadosi identifies a related distinction in trial monitoring. Frequentist monitoring procedures can use current evidence or frequentist predictions under continued sampling, whereas corresponding Bayesian approaches can use the posterior distribution or predictive distribution (Piantadosi, 2005).
Hierarchical modeling is a major Bayesian strength—but it is still modeling
Medical data frequently have multilevel structure: patients within centers, outcomes across related subgroups, repeated estimates from related studies, or treatment effects that may differ across settings.
Bayesian hierarchical models provide a framework in which group-specific parameters can themselves be modeled as arising from a population distribution. Information is then shared across related groups rather than estimating every group entirely independently.
This can produce partial pooling: noisy group-specific estimates are informed by the overall hierarchical structure while uncertainty about the group-level distribution is propagated through the posterior analysis (Gelman et al., 2014).
Hierarchical modeling is particularly important when the amount of information differs substantially between groups. But the advantages depend on the hierarchical model being scientifically and statistically defensible. Distributional assumptions at the observation and group levels still require scrutiny.
Gelman et al.'s treatment of hierarchical modeling therefore connects directly to model checking and sensitivity analysis rather than treating hierarchical Bayes as automatically reliable.
Bayesian models still need to be checked
A common misunderstanding is that obtaining a posterior distribution completes the analysis.
It does not.
Gelman et al. devote substantial attention to posterior predictive model checking. The general strategy is to simulate replicated data under the fitted model and compare relevant features of those simulations with the actual observations (Gelman et al., 2014).
The objective is not simply to assign a pass/fail label to a model. Researchers should ask whether the model reproduces aspects of the data that matter for the intended scientific inference.
Even a model that is not contradicted by a particular check is not thereby proven correct. Gelman et al. explicitly note that alternative reasonable models can fit adequately yet yield different posterior conclusions. Sensitivity analysis is therefore useful for examining how posterior inference changes under alternative priors or model assumptions (Gelman et al., 2014).
Posterior precision cannot compensate for a poorly chosen likelihood, an inappropriate hierarchical structure, or an inadequately examined model.
Where decision analysis enters
Statistical inference and clinical decision making are related, but they are not identical.
A posterior distribution tells researchers what uncertainty remains under a Bayesian model. It does not, by itself, determine which action should be taken.
Formal Bayesian decision analysis adds consequences to the probability model by defining utilities or losses for possible actions and outcomes. Decisions can then be evaluated using posterior uncertainty rather than treating inference itself as the final action rule (Gelman et al., 2014).
This distinction is especially relevant in medical research because clinical decisions can depend on more than whether an effect exists. Benefits, harms, competing outcomes, and consequences of incorrect decisions can all matter.
Piantadosi makes a parallel point in trial monitoring: frequentist procedures require error-control structure, Bayesian procedures require prior beliefs, and decision methods require quantitative loss functions. None of these removes the need for scientific and clinical judgment (Piantadosi, 2005).
Bayesian vs frequentist statistics: compare the actual research question
The most useful comparison is not “subjective versus objective” or “priors versus P values.” It is to ask what the investigator needs to learn.
| Actual research question | Frequentist framing | Bayesian framing | Main caution |
|---|---|---|---|
| How large is the treatment effect, and how uncertain is it? | Estimate the effect and construct a confidence interval using its sampling distribution. | Summarize the posterior distribution with an estimate and posterior interval. | Do not give the two intervals identical probability interpretations. |
| How incompatible are the data with a specified null model? | Use an appropriate hypothesis test and its sampling distribution. | Examine relevant posterior quantities under the specified Bayesian model. | A frequentist P value is not a posterior probability of the null. |
| What is the probability that the effect exceeds a clinically meaningful value? | Conventional confidence procedures do not directly turn this into posterior probability about the parameter. | Calculate the relevant probability from the posterior distribution. | The answer depends on the likelihood, prior, and model. |
| What should we expect for future observations? | Construct predictions using the fitted sampling model and its uncertainty. | Use the posterior predictive distribution. | Prediction requires an adequate model in either framework. |
| How should related centers or subgroups share information? | Multilevel or random-effects methods can model hierarchical structure. | Hierarchical Bayesian models estimate group and population-level distributions jointly. | Pooling assumptions and between-group structure require checking. |
| Does the model reproduce important features of the observed data? | Use residuals, goodness-of-fit procedures, diagnostics, and related checks appropriate to the model. | Use posterior predictive checks and other model-evaluation tools. | Neither framework makes model assumptions disappear. |
| Should a clinical trial stop early? | Sequential/group-sequential, alpha-spending, repeated-CI, or related monitoring procedures may be used. | Posterior or predictive distributions can inform monitoring rules. | Statistical stopping guidelines are only part of the clinical monitoring decision. |
| Which action should be taken given uncertain outcomes and unequal consequences? | Decision rules can be designed around operating characteristics and prespecified criteria. | Posterior uncertainty can be combined with explicit utilities/losses. | Statistical evidence alone does not specify the value of consequences. |
(Gelman et al., 2014; Piantadosi, 2005).
What Bayesian clinical trials change—and what they do not
Bayesian clinical trials are sometimes described as if Bayesian inference removes conventional trial-design concerns.
Piantadosi's discussion argues against such a simplistic conclusion.
He notes that frequentist, Bayesian, and likelihood approaches share important foundations. All require clearly formulated scientific questions, appropriate experimental design, careful data production, suitable probability models, and responsible interpretation. In his view, these common methodological requirements can be more consequential to trial quality than disagreements between statistical schools (Piantadosi, 2005).
The paradigms can nevertheless differ materially in some settings, particularly interim monitoring.
Frequentist monitoring emphasizes operating characteristics such as Type I error under repeated sampling. Bayesian monitoring can instead use posterior or predictive distributions. Decision-based approaches add explicit loss functions. Piantadosi argues that any of these may be useful in particular circumstances rather than declaring one universally appropriate (Piantadosi, 2005).
Different inferential structures can answer different questions or encode different design priorities, but none eliminates the need for a properly designed trial.
Where the approaches can give similar answers
Bayesian and frequentist analyses are not necessarily numerically opposed.
Gelman et al. note that under a standard noninformative prior in the normal linear regression model, Bayesian estimates and standard errors can coincide with classical results. The Bayesian framework can still add posterior simulation for prediction and model checking even when conventional parameter estimates agree closely (Gelman et al., 2014).
This is important because “Bayesian vs frequentist” should not be interpreted as implying that every dataset must produce dramatically different estimates under the two approaches.
Differences become especially important when the inferential targets differ, prior information is consequential, hierarchical structures matter, prediction is central, or monitoring and decision rules are formulated differently.
Advantages and limitations of frequentist inference
Advantages
Frequentist methods provide a well-developed framework for evaluating procedures through repeated-sampling properties. This is especially natural for questions involving error rates, confidence procedures, randomized experiments, and prespecified hypothesis-testing rules.
They are also deeply embedded in medical-research practice and clinical-trial methodology. Piantadosi notes their predominance in the clinical-trial literature, while explicitly distinguishing prevalence from philosophical superiority (Piantadosi, 2005).
Confidence intervals provide a particularly valuable estimation framework because they place effect magnitude and precision at the center of interpretation rather than reducing the study to a significance label (Altman, 1991).
Limitations and cautions
Frequentist summaries are frequently misinterpreted.
A P value should not be converted into a probability that a scientific hypothesis is true, and a confidence interval should not automatically be interpreted as though it were a Bayesian posterior interval.
Frequentist inference also does not rescue an inappropriate probability model, biased design, poor data quality, or an estimand that does not match the scientific question. Piantadosi emphasizes that careful design, data production, estimation, and objective reporting remain more fundamental than allegiance to an inferential school (Piantadosi, 2005).
Advantages and limitations of Bayesian inference
Advantages
Bayesian inference provides a coherent posterior distribution for unknown quantities conditional on the specified model and data. This makes posterior probabilities, posterior intervals, prediction, hierarchical modeling, and decision analysis natural extensions of the same framework (Gelman et al., 2014).
Hierarchical models can share information across related parameters, while posterior predictive simulation provides a practical framework for prediction and model checking.
Prior distributions can also regularize noisy estimates. Gelman et al. discuss how informative priors can stabilize estimation when many predictors or weak information would otherwise produce unstable estimates (Gelman et al., 2014).
Limitations and cautions
Bayesian answers are conditional on the chosen probability model and prior.
Prior choice can matter substantially when data contain limited information, and supposedly “noninformative” priors are not automatically neutral. Different parameterizations and complex hierarchical structures can make default prior choices problematic (Gelman et al., 2014).
Bayesian computation also does not remove model uncertainty. A precisely computed posterior under a poorly fitting model is still inference under that poorly fitting model. Posterior predictive checking and sensitivity analysis are therefore integral rather than optional additions (Gelman et al., 2014).
Common mistakes researchers should avoid
Mistake 1: “Bayesian means subjective; frequentist means objective.”
This is too crude to guide an analysis.
Bayesian models require explicit prior distributions, but all analyses require modeling and design choices. Piantadosi emphasizes substantial agreement across inferential schools concerning experimental design, probability models, careful data production, estimation, and reporting (Piantadosi, 2005).
Mistake 2: “A Bayesian credible interval is just another name for a confidence interval.”
No.
The intervals arise from different inferential constructions and support different probability statements (Altman, 1991; Gelman et al., 2014).
Mistake 3: “A P value is the frequentist probability that the null is true.”
No.
The frequentist test evaluates possible data relative to a specified null model; it does not automatically assign posterior probability to the hypothesis.
Mistake 4: “A Bayesian analysis is valid once the posterior has converged computationally.”
Computational success is not the same as model adequacy.
Model assumptions, predictive checks, sensitivity to prior specification, and substantive plausibility still matter (Gelman et al., 2014).
Mistake 5: “Bayesian trials do not need conventional design discipline.”
They do.
Clear questions, appropriate design, unbiased data production, suitable probability models, and careful interpretation remain essential regardless of inferential framework (Piantadosi, 2005).
Mistake 6: “One paradigm is always more scientific.”
The mandatory sources do not support that universal conclusion.
Piantadosi explicitly presents frequentist, Bayesian, likelihood, and decision approaches as potentially useful in different circumstances and emphasizes methodological concerns shared across schools (Piantadosi, 2005).
A practical framework for choosing the inferential approach
Before asking whether an analysis should be Bayesian or frequentist, define the problem in this order:
Research question → study design → data structure → target quantity → required uncertainty statement → prediction or decision goal → model → assumptions and diagnostics → interpretation.
Then ask:
- Is the main target estimation, testing, prediction, or decision making?
- Do you need probability statements directly about unknown parameters under a specified model?
- Is credible prior information available, and can it be represented transparently?
- Does the problem contain hierarchical structure where partial pooling is scientifically appropriate?
- Are repeated-sampling error properties central to the study design or regulatory decision rule?
- Will the study involve interim monitoring or prediction of future trial outcomes?
- What model checks and sensitivity analyses will be used?
- What interpretation will actually be communicated to clinicians and readers?
The final choice should follow the scientific and inferential problem rather than an allegiance to a statistical label.
Bottom line
The essential difference in Bayesian vs frequentist statistics is not that one uses priors while the other uses P values.
Frequentist inference evaluates statistical procedures through sampling behavior and supports tools such as confidence intervals, hypothesis tests, and controlled error rates. Bayesian inference combines a prior distribution with the likelihood to obtain a posterior distribution and naturally supports posterior probabilities, posterior intervals, predictive inference, hierarchical modeling, model checking, and decision analysis (Altman, 1991; Gelman et al., 2014; Piantadosi, 2005).
Neither framework makes poor study design disappear. Neither makes model assumptions irrelevant. Neither converts statistical evidence automatically into clinical importance.
For medical researchers, the most productive question is therefore not:
“Am I Bayesian or frequentist?”
It is:
“What exactly am I trying to learn or decide, what probability model supports that question, and what interpretation does my chosen inferential framework actually justify?”
Frequently Asked Questions
What is the main difference between Bayesian and frequentist statistics?
Frequentist inference evaluates estimators and tests through their behavior across hypothetical repeated samples, while Bayesian inference represents uncertainty about unknown quantities through probability distributions and updates prior uncertainty with observed data to obtain a posterior distribution (Gelman et al., 2014; Piantadosi, 2005).
What is a Bayesian credible interval?
A Bayesian posterior or credible interval is a region containing a specified amount of posterior probability for the parameter under the chosen model and prior. It should not be treated as merely another name for a frequentist confidence interval (Gelman et al., 2014).
What is a frequentist confidence interval?
A frequentist confidence interval is generated by a procedure designed to have specified coverage under repeated sampling. Altman explains confidence intervals through the sampling distribution of an estimator and emphasizes their value for displaying estimation uncertainty and precision (Altman, 1991).
Does Bayesian analysis always require subjective prior beliefs?
Bayesian analysis requires a prior distribution, but that does not imply that every prior must be a strongly held subjective belief. Priors can encode substantive information or provide weak information or regularization. Gelman et al. caution, however, that supposedly noninformative priors are not universally neutral and require careful consideration (Gelman et al., 2014).
Can Bayesian and frequentist analyses give similar results?
Yes. Gelman et al. show that in standard normal linear regression with a conventional noninformative prior, Bayesian estimates and standard errors can coincide with classical results. Differences in interpretation and additional Bayesian possibilities such as posterior prediction can remain even when numerical estimates are similar (Gelman et al., 2014).
Are Bayesian methods better for clinical trials?
The sources do not support a universal claim that Bayesian methods are better. Piantadosi notes that Bayesian approaches may be particularly useful for some clinical-trial problems, including monitoring, but also emphasizes shared requirements such as clear questions, sound experimental design, careful data production, probability modeling, estimation, and objective reporting (Piantadosi, 2005).
What is the role of Bayesian methods in interim monitoring?
Bayesian trial-monitoring approaches can use posterior or predictive distributions when evaluating accumulating evidence. Frequentist monitoring includes sequential and group-sequential procedures, alpha-spending approaches, repeated confidence intervals, and related methods. Piantadosi treats these as alternative structures whose usefulness depends on the problem rather than ranking one universally above the others (Piantadosi, 2005).
Do Bayesian methods eliminate the need for model checking?
No. Gelman et al. emphasize posterior predictive checking and sensitivity analysis. A posterior distribution is conditional on the model; researchers still need to investigate whether the model can reproduce relevant features of the observed data and whether conclusions change under reasonable alternative assumptions (Gelman et al., 2014).
References
Altman, D. G. (1991). Practical statistics for medical research. Chapman & Hall.
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2014). Bayesian data analysis (3rd ed.). CRC Press.
Piantadosi, S. (2005). Clinical trials: A methodologic perspective (2nd ed.). John Wiley & Sons.
Need help with a similar research question?
Share a short, non-confidential summary of your study and the decision you need to make.