Publication Bias in Meta-Analysis: What Funnel Plot Asymmetry Can—and Cannot—Tell You

What Publication Bias Means in Meta-Analysis

Publication bias arises when the studies available for synthesis differ systematically from the studies that should have been available. If the missing studies were essentially a random subset of all relevant studies, their absence would reduce information and precision but would not systematically shift the effect estimate. The problem arises when availability is related to study results (Borenstein et al., 2021).

A common mechanism involves statistical significance. Borenstein et al. describe evidence that statistically significant studies are more likely to enter the published literature. For a given sample size, larger observed effects are more likely to reach statistical significance. Selective availability can therefore leave the published literature disproportionately populated by studies with larger observed effects (Borenstein et al., 2021).

Altman describes the same problem in clinical-trial synthesis. Medical journals may be more likely to publish trials with significant treatment effects, while investigators themselves may make less effort to publish nonsignificant results. Consequently, an overview based only on published trials can magnify publication bias (Altman, 1991).

Why Selective Availability Matters for the Pooled Effect

Suppose the scientific target is the effect represented by all relevant studies satisfying the review criteria. The meta-analysis instead observes only the subset that can be located.

If study availability is unrelated to the estimated effect, the main consequence of missing studies is loss of information. But if studies with smaller, null, or nonsignificant effects are disproportionately unavailable, the observed set can overrepresent larger effects. The meta-analysis can then provide a mathematically correct weighted synthesis of the observed studies while estimating the wrong picture of the underlying evidence base (Borenstein et al., 2021).

This is why publication bias is a validity problem, not simply a precision problem.

A narrow confidence interval around a pooled effect does not address whether the studies entering that calculation were selectively available.

What a Funnel Plot Actually Shows

A funnel plot displays the relationship between a study's estimated effect and a measure related to its precision.

In the formulation described by Borenstein et al., the effect size is plotted on the horizontal axis and its standard error on the vertical axis. Because the standard-error scale is reversed, larger, more precise studies appear toward the top and smaller, less precise studies toward the bottom (Borenstein et al., 2021).

That geometry is central to funnel plot interpretation.

Large studies generally have smaller standard errors, so their estimates occupy the narrower upper part of the plot. Smaller studies have greater sampling variability and occupy the wider lower portion.

Under the publication-bias mechanism considered by Borenstein et al., large studies may be publishable regardless of whether their effects fall somewhat above or below the mean because they have greater ability to achieve statistical significance. Among smaller studies, however, those with effects on one side of the distribution may be less likely to reach significance and therefore less likely to appear in the available literature. The result can be an asymmetric lower portion of the funnel (Borenstein et al., 2021).

Funnel Plot Asymmetry Is Better Described First as a Small-Study Effect

One of the most important interpretive distinctions is between publication bias and a small-study effect.

Borenstein et al. use small-study effect for a pattern in which effect sizes differ systematically according to study size—for example, when smaller studies tend to report larger effects. The terminology is deliberately neutral about mechanism (Borenstein et al., 2021).

A small-study effect could arise because smaller studies with unremarkable findings were selectively unavailable.

But it could also arise because the true effect actually differs between the types of studies that tend to be small and the types that tend to be large. In that situation, study size is associated with other characteristics that modify the effect. Borenstein et al. therefore describe a genuine small-study effect as one possible expression of heterogeneity rather than automatically as evidence of selection bias (Borenstein et al., 2021).

Why an Asymmetric Funnel Plot Does Not Prove Publication Bias

The visual pattern alone does not identify the mechanism that generated it.

An asymmetric funnel plot may be compatible with selective publication, but Borenstein et al. emphasize that a small-study effect can also occur when the effect really is different in smaller studies. Both mechanisms can even operate simultaneously (Borenstein et al., 2021).

Sampling error also matters. The analyst must distinguish a systematic relationship between study size and effect from random variation before trying to explain its cause.

Context can then strengthen or weaken a publication-bias interpretation. For example, Borenstein et al. note that when most studies are statistically significant and smaller studies show larger effects, publication bias becomes a plausible explanation. If relatively few studies are statistically significant, selective inclusion based on statistical significance is a less compelling explanation for the pattern (Borenstein et al., 2021).

How the studies were assembled also matters. If a synthesis draws retrospectively from the published literature, selective publication is plausible. In a genuinely prospective meta-analysis in which the relevant studies were identified in advance and all are included, a small-study effect cannot be attributed to publication bias through missing trials (Borenstein et al., 2021).

A contour-enhanced funnel plot may provide additional context by distinguishing regions associated with statistical significance from regions associated simply with smaller effects. It still does not turn the funnel plot into a definitive causal diagnostic (Borenstein et al., 2021).

Why a Symmetric Funnel Plot Does Not Prove Publication Bias Is Absent

The reverse error is equally important.

A symmetric-looking plot tells you that the observed studies do not show an obvious asymmetric size–effect pattern of the type being inspected. It does not establish that every relevant study was identified or that no selective availability occurred.

Publication-bias procedures depend on particular patterns in the observed data. Borenstein et al. explicitly caution that the model underlying commonly used procedures is simplified: it typically assumes that statistically significant studies are more likely to enter the analysis than nonsignificant studies, whereas actual publication processes may be more complicated and may vary across fields, journals, and stages of a literature (Borenstein et al., 2021).

The absence of the expected asymmetry is therefore not logically equivalent to evidence that the selection process was unbiased.

What Common Funnel-Plot Findings Can and Cannot Support

Alternative Causes of Funnel Plot Asymmetry

When asymmetry is present, the next step is explanation rather than automatic labeling.

Borenstein et al. emphasize two broad possibilities after random sampling error is considered:

The second possibility makes asymmetry part of the broader heterogeneity problem. Smaller and larger studies may represent different populations, interventions, methods, settings, or other conditions. Study size itself need not be causal; it may mark other study characteristics associated with the effect (Borenstein et al., 2021).

Publication-Bias Tests Have Limited Diagnostic Meaning

Statistical procedures can test for relationships between study size or precision and effect size, but that does not solve the identification problem.

Borenstein et al. discuss procedures including rank-correlation and regression-based approaches, while noting that tests intended to identify whether bias exists have limited utility for the more important substantive question: how much might the potential bias change the conclusion?

• A statistically detectable size–effect relationship does not uniquely identify publication bias.
• A nonsignificant test does not prove that publication bias is absent.
• Even when a procedure detects a pattern compatible with bias, it does not automatically tell the researcher the magnitude of distortion in the target effect.

There are also circumstances in which the usual procedures have little useful information. Borenstein et al. note, for example, that when study sample sizes occupy only a narrow range, procedures based on relationships between size and effect may be ineffective (Borenstein et al., 2021).

The statistical test should therefore be treated as one component of an interpretive assessment, not as a binary publication-bias detector.

Treat Adjustment as Sensitivity Analysis, Not Reconstruction of Truth

Methods such as trim and fill attempt to estimate studies that might be missing under a particular asymmetry model and examine how the pooled result would change after adding imputed studies (Borenstein et al., 2021).

This can be useful—but only if the inferential question is framed correctly.

Borenstein et al. explicitly state that because a small-study effect cannot be uniquely separated from publication bias, adjusted estimates should be interpreted as sensitivity analyses. The adjusted estimate should never be asserted to be the uniquely correct or true effect (Borenstein et al., 2021).

A useful sensitivity interpretation is therefore qualitative as well as numerical:

• the substantive conclusion appears largely unchanged under the adjustment;
• the effect estimate changes, but the main substantive conclusion remains similar; or
• the conclusion is sensitive enough that the missing-evidence problem could materially alter interpretation.

This keeps the analysis focused on robustness rather than pretending to reconstruct data that were never observed.

Why Statistical Adjustment Cannot Recover an Unknown Unpublished Evidence Base

No publication-bias adjustment has direct access to the studies that were never located.

An adjustment instead posits something about the mechanism that generated the observed pattern and then asks what the synthesis would look like under that model. Trim and fill, for example, identifies asymmetry and algorithmically imputes studies that could restore a particular pattern; those imputed studies are not observations recovered from the actual unpublished literature (Borenstein et al., 2021).

The difficulty is deeper than choosing the right algorithm.

If asymmetry can arise from either selective publication or genuine differences in effects between small and large studies, the observed data do not uniquely reveal which missing studies actually exist, what their effect estimates would have been, or even whether the missing-study mechanism assumed by the adjustment is correct. Borenstein et al. consequently reject interpretation of adjusted estimates as the “correct” effect (Borenstein et al., 2021).

Altman's discussion provides the complementary practical point: obtaining information about unpublished trials is inherently difficult, even though doing so is preferable to relying solely on the published literature (Altman, 1991).

Five Statements a Funnel Plot Cannot Justify

A Practical Funnel Plot Interpretation Framework

Use the funnel plot as part of a sequence rather than as a verdict.

1. Define the evidence target. Ask what studies should have been represented in the synthesis under the review's eligibility criteria.

2. Examine study availability. Consider whether the synthesis depends mainly on retrospectively located published studies or whether study identification provides stronger assurance that the relevant evidence base was captured.

3. Inspect the size–effect relationship. Look at effect estimates in relation to standard errors or another supported measure of precision. Describe an apparent relationship neutrally as a small-study effect before assigning a mechanism.

4. Consider random variation and heterogeneity. Ask whether smaller studies may differ systematically from larger studies in populations, interventions, methods, or other characteristics capable of producing real effect differences.

5. Consider selective publication as one explanation. Statistical significance patterns and the way studies were identified can make this mechanism more or less plausible (Borenstein et al., 2021).

6. Use formal procedures cautiously. Statistical tests or funnel-plot-based adjustments do not uniquely identify the missing-data mechanism.

7. Focus sensitivity analysis on the substantive conclusion. Ask whether plausible adjustment would leave the interpretation essentially unchanged, alter its magnitude without changing the key conclusion, or call the conclusion into question (Borenstein et al., 2021).

8. Report uncertainty about the evidence-selection process explicitly. Do not replace that uncertainty with an apparently definitive “bias corrected” estimate.

Special Caution for Diagnostic-Study Synthesis

Diagnostic-test meta-analysis adds another layer of interpretation.

Pepe discusses meta-analysis of diagnostic studies and emphasizes that diagnostic accuracy itself depends on how the test is evaluated and on characteristics of the underlying studies. Her broader diagnostic framework shows that study design, patient spectrum, verification procedures, reference-standard quality, and related features can influence estimated test accuracy (Pepe, 2003).

For diagnostic-study synthesis, an association between study size and an accuracy measure therefore needs especially careful interpretation. Differences among small and large diagnostic studies may reflect not only selective availability but also differences in the populations studied or in the processes used to establish disease status and evaluate the index test.

For example, an unrepresentative spectrum of cases and controls, selective verification, or an imperfect reference standard can distort estimated diagnostic performance (Pepe, 2003). These validity differences should be investigated rather than automatically relabeled as publication bias.

Reporting Publication Bias Without Overclaiming

A defensible report should separate three questions.

This produces wording such as:

“The funnel plot showed evidence of asymmetry consistent with a small-study effect. Selective publication is one possible explanation, but the observed pattern does not uniquely identify publication bias. Sensitivity analyses were therefore used to assess how strongly the substantive conclusion depended on assumptions about missing studies.”

That wording communicates what was observed, what remains uncertain, and what the sensitivity analysis was actually designed to answer.

Publication Bias Interpretation Checklist

Before concluding that a meta-analysis has—or does not have—publication bias, ask:

• What body of studies was the review intended to represent?
• Could relevant studies have remained unavailable?
• Is missingness plausibly related to study findings?
• Does the funnel plot show a relationship between precision and effect?
• Am I calling that relationship a small-study effect before claiming a cause?
• Could genuine heterogeneity explain why smaller and larger studies differ?
• Are study characteristics correlated with study size?
• Is the range of study sizes wide enough for size-based procedures to be informative?
• Does the pattern of statistical significance make selective publication plausible?
• How were the studies located?
• Am I treating publication-bias tests as diagnostic evidence rather than proof?
• Am I treating adjusted estimates as sensitivity analyses rather than corrected truth?
• Would the substantive conclusion change under plausible missing-study assumptions?
• Have I kept publication bias separate from other forms of study-level bias?

Bottom Line

The most important lesson in publication bias meta analysis is that funnel plots show patterns in observed studies; they do not reveal the missing evidence directly.

Publication bias can arise when the available studies are systematically different from the complete relevant evidence base. Selective publication of statistically significant or larger effects is one mechanism, and pooling only published evidence can carry that bias into the meta-analysis (Altman, 1991; Borenstein et al., 2021).

But funnel plot asymmetry is not synonymous with publication bias. A relationship between study size and effect should first be interpreted as a small-study effect because genuine effect heterogeneity can produce the same pattern (Borenstein et al., 2021).

Likewise, symmetry does not prove that selective publication is absent.

Publication-bias detection and adjustment methods are therefore best used to investigate robustness:

That is a question the analysis can help address.

“What is the exact true effect after restoring the unknown unpublished studies?”

The observed evidence cannot guarantee an answer to that question.

FAQs

What is publication bias in meta-analysis?

Publication bias occurs when the studies included in a meta-analysis differ systematically from the full set of relevant studies that should have been included. One important mechanism is preferential availability of studies with statistically significant or larger effects, which can bias the pooled estimate (Altman, 1991; Borenstein et al., 2021).

What does funnel plot asymmetry mean?

Funnel plot asymmetry indicates a relationship between study precision or size and observed effect that warrants investigation. It can be consistent with publication bias, but it can also arise because true effects differ between smaller and larger studies. The pattern should therefore initially be described as a small-study effect rather than proof of publication bias (Borenstein et al., 2021).

Does a symmetric funnel plot mean there is no publication bias?

No. Symmetry means that the observed studies do not show a clear asymmetric size–effect pattern. It does not prove that every relevant study was available or that selection occurred according to no other mechanism.

What are small study effects?

A small-study effect is a pattern in which observed effects differ systematically with study size, commonly with larger effects appearing in smaller studies. Publication bias can cause such a pattern, but genuine differences in effects associated with characteristics of smaller studies can also produce it (Borenstein et al., 2021).

Can trim and fill correct publication bias?

Trim and fill can estimate how a pooled effect would change after imputing studies under a particular model of funnel-plot asymmetry. Borenstein et al. recommend treating the result as a sensitivity analysis rather than as the uniquely corrected or true effect (Borenstein et al., 2021).

Why can't publication-bias methods recover the true effect?

The analyst does not observe the missing studies or know their actual results. Moreover, the observed small-study pattern can have mechanisms other than publication bias. Consequently, adjusted estimates depend on assumptions about an unidentified selection process and cannot be guaranteed to reconstruct the true unpublished evidence base (Borenstein et al., 2021).

How should publication bias detection be reported?

Report the observed size–effect pattern, plausible explanations, limitations of the detection procedure, and the effect of sensitivity analyses on the substantive conclusion. Avoid stating that asymmetry proves publication bias, that symmetry proves its absence, or that an adjusted estimate is the true effect.

Does funnel plot interpretation require extra caution in diagnostic-test meta-analysis?

Yes. Diagnostic accuracy can vary with patient spectrum, verification, reference-standard quality, and other design features. Differences between smaller and larger diagnostic studies may therefore reflect study-design or population differences as well as selective availability (Pepe, 2003).

References

Altman, D. G. (1991). Practical statistics for medical research. Chapman & Hall.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2021). Introduction to meta-analysis (2nd ed.). Wiley.

Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford University Press.