The Regression Was Significant—But Was It Trustworthy? A Linear Regression Diagnostics Case Study

StatsAlly synthetic case study · Education research · Mock data

A regression coefficient table can look reassuring: a highly significant overall F test, a respectable R², and a focal predictor with p < .001. But those results answer questions conditional on the fitted model being an adequate representation of the data. Regression diagnostics ask the question that the coefficient table cannot answer by itself: Is this model trustworthy enough to interpret?

This case study uses a synthetic sample of 260 university students enrolled in an introductory quantitative-methods course. The initial multiple linear regression appears convincing. Diagnostics then reveal three important complications: the relationship between study time and assessment performance is curved rather than adequately represented by one straight-line slope; residual variance changes across the fitted range; and one observation combines an unusual residual with high leverage and substantial influence.

The research question comes before the regression

The hypothetical study asks:

Among students taking a quantitative-methods course, how are weekly independent study time, prior academic achievement, academic self-efficacy, study-strategy quality, and average sleep duration associated with end-of-term assessment performance?

The outcome is an end-of-term quantitative-methods assessment scored from 0 to 140 points. The predictors are:

• weekly independent study time, in hours;
• prior academic achievement, on a 0–100 scale;
• academic self-efficacy, on a 1–7 scale;
• study-strategy quality, on a 1–7 scale; and
• average nightly sleep, in hours.

Model specification also needs substantive justification. A multiple regression model should not be assembled simply by entering whatever variables happen to be available; relevant predictors, controls, functional form, and omitted variables all affect what its coefficients mean. Meier et al. (2014) emphasize that correct specification requires substantive knowledge and defensible reasons for including explanatory variables, while Newton and Rudestam (1999) note that specification errors can seriously distort a statistical model.

For this case study, the first model was deliberately conventional:

The key restriction is easy to overlook: this equation treats each continuous predictor as having a linear additive relationship with expected assessment performance.

That restriction must be checked rather than assumed.

First look: the regression appears convincing

The initial ordinary least-squares model was statistically significant.

Thus, the fitted predictors collectively accounted for about 53.8% of the observed variation in the synthetic assessment scores.

Initial coefficient table

At this stage, a tempting conclusion would be:

“The regression model was highly significant. Holding the other predictors constant, each additional weekly study hour was associated with an estimated 0.62-point increase in assessment performance.”

Mathematically, that is what the fitted coefficient says. But it assumes that a constant 0.62-point increment adequately describes the study-time relationship throughout the observed range.

Regression inference should not be interpreted without examining whether the model conditions are at least approximately compatible with the data. Moore et al. (2021) explicitly emphasize examining residual distributions and plotting residuals against explanatory variables before multiple-regression inference; residual plots can expose curvature, changing variance, unusual observations, and other evidence that a fitted line is inadequate (Moore et al., 2014, 2021).

So the analysis continues.

Diagnostic 1: Does the model answer the research question in the right form?

The research question asks about the association between study time and performance. It does not require that this association be a straight line.

That distinction matters. Applying a linear model when the population relationship is nonlinear makes the chosen statistical model inappropriate for the relationship being studied (Newton & Rudestam, 1999). Meier et al. (2014) similarly identify linearity as an assumption of linear regression and discuss polynomial terms when curved relationships need to be represented.

The initial model therefore answers a narrower question than it first appears to:

What is the estimated single linear slope for study time after adjustment for the other predictors?

Before interpreting that slope as a useful summary, we need evidence that one slope is adequate.

Diagnostic 2: Linearity—the residuals reveal curvature

Residuals are the differences between observed and model-predicted outcomes. When a linear model captures the systematic structure of the relationship, residual plots should not retain an obvious systematic pattern. A curved residual pattern indicates that a straight line has left systematic structure unexplained (Moore et al., 2014, 2021).

In the synthetic data, the residual pattern across study hours was curved. Students at low and moderately high study levels tended to fall systematically on different sides of the fitted linear prediction.

That changes the interpretation of the original coefficient.

This is a model-form problem, not something that a small p value can repair.

Diagnostic 3: Residual behavior—random noise or remaining structure?

Residual analysis is useful precisely because a regression coefficient table compresses the data into a few estimates. Residual plots return attention to what the fitted equation has failed to explain. Moore et al. (2014, 2021) recommend examining both overall residual patterns and striking individual observations.

The initial residuals showed two different issues:

These are substantively different problems. Curvature concerns the modeled mean relationship. Changing residual spread concerns the model's representation of variability around that mean.

A model can therefore have a strong R² and highly significant coefficients while still describing important features of the data inadequately.

Diagnostic 4: Constant variance—the uncertainty was not constant

Linear-regression inference conventionally assumes constant error variance. In a homoscedastic pattern, the spread of residuals remains broadly similar across the predictor or fitted-value range. A systematic increase or decrease in residual spread indicates heteroskedasticity (Meier et al., 2014; Newton & Rudestam, 1999).

The synthetic residual plot showed noticeably greater variability in portions of the fitted range. A Breusch–Pagan diagnostic was also consistent with nonconstant variance (p < .001).

That result is important because heteroskedasticity affects the uncertainty attached to coefficient estimates. Meier et al. (2014) note that violation of constant error variance can make conventional slope standard errors and associated t statistics inaccurate. They also describe approaches such as weighted least squares, robust standard errors, and transformations, while emphasizing that the appropriate response should depend on the source of the heteroskedasticity.

Diagnostic 5: Normality—relevant to inference, but not the whole diagnosis

The multiple linear regression model used for conventional inference assumes errors with a Normal distribution in addition to independence and constant variance (Moore et al., 2021). Normality is therefore principally an inferential condition here; it is not a requirement that every predictor itself have a Normal distribution.

The synthetic residuals showed heavier tails than an ideal Normal pattern. A Shapiro–Wilk test also detected departure from Normality (W = .977, p < .001).

That test result should not be converted into a mechanical “regression invalid” decision. Graphical examination remains important, and diagnostic evidence needs to be considered together rather than as independent pass/fail gates. Tabachnick and Fidell (2013) discuss assessing normality through distributions and residuals while also cautioning that formal assessments of multivariate assumptions can be sensitive.

More importantly in this case, the tail departure was partly connected to identifiable unusual observations. That leads to a more informative question than simply asking whether a Normality test was significant:

Which observations are producing the unusual residual behavior, and do they matter to the fitted model?

Diagnostic 6: Multicollinearity—some overlap, but not the central problem

Multiple regression estimates the adjusted contribution of a predictor while the other included predictors are held constant. When predictors contain strongly overlapping information, separating their individual contributions becomes difficult. Multicollinearity can therefore destabilize individual coefficient estimates even when overall model prediction remains useful (Lovric, 2011; Meier et al., 2014; Moore et al., 2014, 2021; Newton & Rudestam, 1999).

Academic self-efficacy and study-strategy quality were moderately correlated in the synthetic data (r ≈ .686). Their variance inflation factors were approximately 1.93 and 1.94. The remaining predictor VIFs were close to 1.

These diagnostics indicate some shared information between self-efficacy and strategy, which helps explain why their individual coefficients should be interpreted conditionally. But compared with the curvature, changing residual variance, and unusual cases, collinearity was not the dominant diagnostic concern in this model.

This illustrates why regression diagnostics should not be reduced to finding one number above or below a universal cutoff. The practical question is what the predictor dependence is doing to the estimates being interpreted.

Diagnostic 7: Unusual residuals—case 18 stands out

An observation can be unusual because its observed outcome is far from what the fitted model predicts. Such observations appear as large residuals.

The largest externally studentized residual in the synthetic model belonged to case 18, at approximately -4.66. Another observation had a studentized residual of approximately -3.83.

An unusual residual does not automatically justify deleting an observation. Data errors should be corrected when verified, but a genuine observation may instead reveal inadequacy in the model or genuine heterogeneity in the population. Tabachnick and Fidell (2013) emphasize that outlier handling requires judgment and that transformations, changes, or deletions should be reported with their rationale.

The next question is therefore not “Can we remove case 18?”

It is:

Does case 18 have enough leverage to materially affect the fitted regression?

Diagnostic 8: Leverage—case 18 is unusual in predictor space too

Leverage concerns how unusual an observation's predictor configuration is relative to the rest of the sample. An observation can have high leverage without having a large residual, and a large residual can occur without high leverage. These are different properties (Moore et al., 2014, 2021; Tabachnick & Fidell, 2013).

Case 18 had leverage of approximately 0.101, compared with a sample median of about 0.020.

Its predictor profile was therefore considerably more unusual than that of a typical case in this synthetic sample. In particular, it reported 35 weekly study hours, placing it at the extreme end of the observed study-time distribution.

High leverage by itself does not establish harmful influence. An extreme predictor configuration that follows the same relationship as the rest of the data can fit comfortably within the model.

But case 18 also had a very large residual.

That combination deserves closer examination.

Diagnostic 9: Influence—does one observation change the fitted answer?

Influence concerns how much a fitted regression changes when an observation is removed. Tabachnick and Fidell (2013) describe influence as reflecting the combination of leverage and discrepancy and discuss Cook's distance, DFFITS, and DBETAS as influence diagnostics.

Case 18 had the largest Cook's distance in the sample.

The important point is not to turn Cook's distance into an automatic deletion rule. The diagnostic identifies an observation deserving investigation and sensitivity analysis. Influence measures themselves are imperfect, and outlier diagnostics should be interpreted cautiously rather than mechanically (Tabachnick & Fidell, 2013).

Case 18 was therefore retained in the primary diagnostic investigation, while the repaired model was also examined with the observation omitted as a sensitivity check.

Diagnostic 10: Overall model fit—R² does not diagnose model adequacy

The initial R² = .538 sounds strong because the fitted model accounts for a substantial portion of observed outcome variation.

But R² measures fit in a particular sense: it summarizes how much variation in the response is accounted for by the fitted regression relative to total variation. It does not demonstrate that the functional form is correct, that residual variance is constant, that individual observations are harmless, or that the model supports causal conclusions (Newton & Rudestam, 1999).

Similarly, the highly significant overall F test evaluates whether the predictor coefficients are collectively zero under the specified regression model. It does not test every assumption or establish that the selected model is the right representation of the data (Moore et al., 2021).

Repairing the model: represent the curvature explicitly

The study-time residual pattern suggested that the single straight-line effect was inadequate. A quadratic study-time term was therefore added.

To make the polynomial representation easier to work with, study hours were centered before calculating the squared term. The repaired mean model became:

Because unequal residual variance remained relevant, heteroskedasticity-robust standard errors were used for the inferential sensitivity analysis, an approach among those discussed for nonconstant error variance by Meier et al. (2014).

Repaired-model results

Fit improved, but the more important improvement was interpretive: the model now represented the observed curvature rather than forcing it into a single slope.

The study-hours coefficient now means something different

Once a quadratic term is included, the association between study hours and expected performance is not constant.

In the synthetic fitted model, the estimated instantaneous study-time slope was approximately:

They show why the original B = 0.623 was incomplete. The first regression compressed a changing relationship into one average linear slope. The repaired model instead suggests that the positive adjusted association becomes progressively weaker across the observed study-time range and eventually turns negative at the high end.

This does not establish that excessive studying causes lower performance. The data remain observational, and high study time could reflect unmeasured characteristics such as course difficulty, preparation deficits, stress, or other factors.

What happened to the other coefficients?

Diagnostics also changed the apparent story for academic self-efficacy and study-strategy quality.

The point is not that the second set of p values is automatically “correct” because it is more elaborate. It is that coefficient estimates and tests are conditional on the model in which they are estimated. Moore et al. (2021) explicitly note that multiple-regression coefficient inference changes when explanatory variables are added to or removed from the model.

When predictors overlap and model form changes, researchers should interpret coefficients as conditional estimates rather than fixed properties of variables.

Sensitivity analysis for the influential observation

The quadratic model was refitted without case 18 to determine whether the central conclusion depended on that observation.

The quadratic study-time coefficient changed from approximately -0.045 to -0.036, while remaining clearly negative in the synthetic sensitivity analysis. The model R² changed from approximately .594 to .606.

That is a more defensible use of influence diagnostics than simply deleting the case because it looked inconvenient. The observation is investigated, its impact is quantified, and the sensitivity of the conclusion is reported.

Before and after diagnostics

What should the researcher report?

A defensible report would not say simply:

“Multiple regression showed that study hours significantly predicted assessment performance, p < .001.”

That statement hides the most consequential diagnostic finding.

A more informative interpretation is:

In the synthetic sample, the initial multiple linear regression was statistically significant and explained approximately 54% of the observed variation in assessment scores. However, diagnostic examination showed systematic curvature in the study-time relationship, nonconstant residual variance, and an influential high-leverage observation. A revised model incorporating a quadratic study-time term provided a better representation of the observed relationship and increased R² to approximately .59. The fitted association between study time and assessment performance was positive at lower and moderate study levels but weakened as study time increased, indicating that a single linear study-hours coefficient was not an adequate summary. Sensitivity analysis excluding the most influential observation did not remove the evidence of curvature. Because the synthetic scenario is observational, these adjusted associations should not be interpreted causally.

That conclusion reflects the question, design, model, diagnostics, interpretation, and limitations, rather than treating the smallest p value as the final answer.

A practical regression-diagnostics workflow

For applied research, a useful sequence is:

1. Return to the research question. Decide whether the intended conclusion concerns association, prediction, explanation, or causation and whether the outcome and predictors fit the proposed model.
2. Inspect the data and relationships. Look for implausible values, restricted ranges, unusual observations, predictor overlap, and nonlinear patterns before relying on regression output.
3. Fit the prespecified model. Record coefficients, uncertainty intervals, overall fit, and the overall model test.
4. Examine residual patterns. Look for curvature or other systematic structure that indicates the mean model is incomplete.
5. Assess variance behavior. Determine whether residual spread is reasonably stable or changes systematically.
6. Consider residual Normality when it matters for inference. Use graphical evidence together with other diagnostics rather than reducing the decision to a single Normality test.
7. Examine predictor dependence. Determine whether multicollinearity makes individual adjusted coefficients difficult to estimate or interpret.
8. Investigate unusual residuals and leverage separately. They answer different questions.
9. Assess influence. Determine whether particular observations materially alter the fitted results; investigate and use sensitivity analyses rather than deleting observations mechanically.
10. Reconsider model specification. If diagnostics expose curvature, omitted structure, or another defensible modeling problem, revise the model for a substantive reason and document the change.
11. Reinterpret every coefficient after revision. Coefficient meaning and statistical significance are conditional on the model being fitted.
12. Report remaining limitations. A better-fitting regression does not convert an observational design into an experiment or eliminate unmeasured confounding.

This workflow is consistent with the broader principle that regression assumptions and diagnostics are part of statistical analysis rather than optional checks performed after the desired coefficient has already been interpreted (Meier et al., 2014; Moore et al., 2014, 2021; Newton & Rudestam, 1999; Tabachnick & Fidell, 2013).

The main lesson

The initial regression was not useless. Its overall F test was significant, its R² was substantial, and its study-hours coefficient had a very small p value.

It was still not an adequate final model.

The diagnostics showed that the apparent simplicity of the coefficient table concealed a curved relationship, unequal residual variance, unusual observations, and an influential high-leverage case. Once the model was reconsidered, the substantive interpretation changed from “each additional study hour is associated with a constant 0.62-point increase” to “the adjusted association changes across the observed study-time range.”

That distinction is the purpose of regression diagnostics.

References

Lovric, M. (Ed.). (2011). International encyclopedia of statistical science. Springer. https://doi.org/10.1007/978-3-642-04898-2

Meier, K. J., Brudney, J. L., & Bohte, J. (2014). Applied statistics for public and nonprofit administration (9th ed.). Cengage Learning.

Moore, D. S., McCabe, G. P., & Craig, B. A. (2014). Introduction to the practice of statistics (8th ed.). W. H. Freeman.

Moore, D. S., McCabe, G. P., & Craig, B. A. (2021). Introduction to the practice of statistics (10th ed.). Macmillan Learning.

Newton, R. R., & Rudestam, K. E. (1999). Your statistical consultant: Answers to your data analysis questions. SAGE Publications.

Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Pearson.