Outlier, Leverage Point, or Influential Case? A Regression Investigation

The research problem

Suppose a researcher is studying whether a standardized predictor (X₁) is associated with a continuous outcome (Y), controlling for a second standardized predictor (X₂).

The fitted model is:

Y_i = β₀ + β₁X_1i + β₂X_2i + ε_i

The synthetic dataset contains 80 observations: 77 ordinary observations and three deliberately unusual cases.

The three deliberately unusual cases

First distinction: an outlying outcome is not the same as leverage

A residual is the difference between an observed response and the value predicted by the regression model. Residual plots therefore provide evidence about observations whose outcomes are poorly represented by the fitted relationship. Large residuals may identify cases requiring investigation, but they do not by themselves tell us why the observation is unusual (Moore et al., 2021).

In the synthetic data, U_residual has predictors near the center of the data:

X₁

0.05

X₂

-0.10

Observed Y

18.50

Fitted Y

9.99

Residual

8.51

Externally studentized residual

4.84

Leverage

0.013

Its defining feature is therefore not an unusual predictor configuration. Its outcome is simply much higher than the model predicts.

This is an important form of residual evidence. But it does not automatically make the case highly influential.

Moore et al. (2021) explicitly distinguish regression outliers from influential observations: an observation may have an unusual response without substantially changing the fitted regression line, while an influential observation may not reveal itself through an unusually large residual because the fitted line can be pulled toward it.

Second distinction: leverage comes from the predictors

Leverage concerns where a case lies in predictor space. An observation with an unusual combination of predictor values has more potential to affect a fitted regression than an otherwise comparable observation near the center of the predictor distribution.

The synthetic U_leverage case has:

X₁

-4.80

X₂

4.20

Leverage

0.263

Its predictor configuration is far from the ordinary observations. Yet its observed outcome is reasonably consistent with the fitted relationship:

Observed Y

7.63

Fitted Y

9.10

Residual

-1.47

Studentized residual

-0.85

Cook’s distance

0.087

This case therefore demonstrates the distinction between having leverage and being strongly influential. The case has substantial capacity to affect the regression because of its predictor position, but its response does not pull the model dramatically away from the pattern in the remaining observations.

Moore et al. (2021) similarly note that influential observations are often unusual in the explanatory-variable direction but need not have large residuals.

Third distinction: influence is about changing the fitted analysis

An observation is influential when the fitted analysis changes meaningfully because of that observation. The core idea is comparative: what happens to the estimated relationship when the case is included versus omitted?

Cook’s distance was developed specifically to assess the influence of individual cases on estimated coefficients in linear regression. Its logic compares the fitted analysis with and without an observation rather than treating unusualness alone as sufficient evidence for removal (Lovric, 2011).

The synthetic U_influence case has:

X₁

4.80

X₂

-3.80

Observed Y

5.00

Fitted Y

10.98

Residual

-5.98

Studentized residual

-3.69

Leverage

0.246

Cook’s distance

1.27

Unlike U_leverage, this case combines an unusual predictor configuration with a response that conflicts with the relationship suggested by much of the remaining data.

Its effect becomes clear when the regression is re-estimated.

The sensitivity analysis

The deletion therefore changes both the estimated magnitude of the X₁ coefficient and the inferential conclusion.

That is evidence that the observation is influential.

Why “remove it so the model becomes significant” is the wrong rule

Deleting observations because their removal improves statistical significance reverses the logic of model checking. The desired inferential result becomes the criterion for deciding what counts as acceptable data.

Tabachnick and Fidell (2013) warn specifically against identifying cases after the main analysis and then changing or eliminating them merely to make the solution look better; they characterize such retrospective model improvement as appropriate, at most, for exploratory work rather than as a defensible confirmatory procedure.

The same principle follows from the distinction between influence and error. An observation can materially change a regression precisely because it contains information about a part of the population that is not well represented by the remaining sample. Deleting it because that information weakens a desired result would produce a cleaner-looking analysis at the cost of changing the evidence being analyzed.

The proper sequence

1. Determine whether the observation is legitimate.
2. Determine how it differs from the rest of the data.
3. Determine how strongly the fitted result depends on it.
4. Report that dependence transparently.

Is the unusual observation actually a data error?

Before interpreting an unusual case statistically, investigate its provenance.

Tabachnick and Fidell (2013) distinguish several reasons that extreme observations may appear, including incorrect data entry, miscoded missing values, cases that do not belong to the intended population, and genuinely extreme observations that do belong to that population.

That distinction changes the decision.

Newton and Rudestam (1999) similarly distinguish cases that do not belong to the relevant population from legitimate extreme cases. For valid extreme observations, they recommend examining how conclusions behave with the observations both included and excluded rather than treating deletion as automatic.

What the three cases tell us

The synthetic diagnostics reveal three different patterns.

Cook’s distance is useful here because it formalizes the case-deletion idea for linear regression and summarizes how strongly a case affects the estimated regression coefficients (Lovric, 2011).

Why residuals and Cook’s distance answer different questions

Residual diagnostics and influence diagnostics should not be collapsed into a single “outlier test.”

Cook’s distance provides one way to operationalize the third question in linear regression by comparing full-data coefficient estimates with estimates obtained after deleting individual observations (Lovric, 2011).

These diagnostics are related, but they are not interchangeable.

Do not convert diagnostics into automatic deletion rules

Tabachnick and Fidell (2013) emphasize that the interpretation of unusual cases depends on why the case is unusual and whether it belongs to the population under study. They also stress documenting transformations, altered values, or deletions and their rationale.

A rigid rule such as “delete every case above a certain Cook’s distance” ignores the substantive question of whether the observation is valid and what its removal does to the population represented by the analysis.

A defensible workflow for an influential case

1. Verify the raw observation, coding and measurement process. If the values are incorrect, correct the error and document the correction.
2. Determine whether the case belongs to the intended population. A case that entered the dataset despite being outside that population presents a design or eligibility issue rather than simply an “outlier problem” (Tabachnick & Fidell, 2013).
3. If the observation is valid and in-scope, retain it in the primary analysis unless there is a defensible methodological reason to do otherwise.
4. Conduct a sensitivity analysis. Show how conclusions change when the influential observation is omitted. Newton and Rudestam (1999) explicitly describe analysis with and without outliers as a useful way to evaluate whether conclusions are dependent on those cases.
5. Interpret the sensitivity rather than selecting a preferred result. The purpose is to reveal how dependent the inference is on a particular observation.

How this case should be reported

A transparent results section could say:

The primary regression retained all 80 valid observations. Diagnostic examination identified one case whose predictor configuration and residual were both unusual and whose inclusion materially affected the fitted coefficients. With all cases included, the estimated coefficient for X₁ was 0.390 (p = .057). A prespecified sensitivity analysis omitting that observation produced a coefficient of 0.641 (p = .002). Because the record was treated as a valid member of the target population in this synthetic example, it was retained in the primary model. The difference between analyses indicates that the estimated association is sensitive to this observation and should therefore be interpreted cautiously.

That report preserves both pieces of evidence: what the full dataset says and how fragile that conclusion is.

It does not conceal the sensitivity, and it does not use significance as a data-cleaning criterion.

Decision guide

Conclusion

An outlier is not automatically a leverage point. A leverage point is not automatically influential. And an influential observation is not automatically erroneous.

Residuals provide evidence about how poorly a case’s outcome is represented by the fitted model. Leverage describes unusualness in predictor space. Influence concerns how much the fitted analysis changes because of a case. Cook’s distance is one case-deletion-based diagnostic for influence in linear regression (Lovric, 2011; Moore et al., 2021).

The researcher’s question should therefore never be, “Which observations can I remove so the model becomes significant?”

That question leads to data verification, model diagnostics and transparent sensitivity analysis rather than significance-driven deletion.

References

Lovric, M. (Ed.). (2011). International encyclopedia of statistical science. Springer. https://doi.org/10.1007/978-3-642-04898-2

Moore, D. S., McCabe, G. P., & Craig, B. A. (2021). Introduction to the practice of statistics (10th ed.). Macmillan Learning.

Newton, R. R., & Rudestam, K. E. (1999). Your statistical consultant: Answers to your data analysis questions. SAGE Publications.

Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Pearson.