Variable Selection and Overfitting: Why Stepwise Regression Can Produce Unstable Models

The first decision: are you explaining or predicting?

Before choosing predictors, define what the regression is intended to accomplish.

These objectives overlap, but they do not impose identical criteria for model construction or evaluation.

For prediction, generalization to independent observations is fundamental. Hastie et al. define generalization performance in terms of prediction on independent test data and place model assessment and selection within the bias–variance/model-complexity problem (Hastie et al., 2009).

For clinical prediction, Steyerberg likewise treats predictor selection, shrinkage, internal validation, and predictive performance as parts of a broader development strategy. A predictor can be useful for prediction without having the scientific interpretation that would be required of an explanatory or causal variable.

Why univariable screening is not a neutral first step

A common predictor-selection strategy is:

1. fit one model for each candidate predictor;
2. retain predictors whose univariable p-values cross a chosen threshold;
3. put those predictors into a multivariable model; and
4. interpret the retained variables as the important or “independent” predictors.

This procedure allows outcome information to determine which predictors are eligible for the final multivariable analysis. The reported model therefore reflects more than the coefficient estimation visible in its final output.

The same principle applies to automated forward selection and backward elimination. The algorithm repeatedly examines the observed data, compares candidate models, and makes decisions about which predictors survive. The final model is conditional on that search.

Harrell's treatment of model selection emphasizes why the variance calculation for a reduced model cannot simply behave as though the selected model had been prespecified; variation arising from the way an approximate model was selected may itself require bootstrap assessment (Harrell, 2015).

Why stepwise regression can be unstable

Suppose two predictors contain overlapping information. In one sample, random variation may make predictor A appear slightly more useful than predictor B. A stepwise algorithm retains A.

With a modest perturbation of the observations, B may instead appear preferable.

The resulting equations can look decisively different even though the underlying information in the predictors is similar.

This is one reason a clean final stepwise table can give a misleading impression of certainty. The table shows the model that won the search in this dataset. It does not show how often another plausible sample from the same population would have produced a different predictor set.

Harrell's discussion of ordinary stepwise procedures emphasizes that conventional post-selection calculations can effectively pretend that the reduced model was specified in advance, when it was actually chosen using the observed outcome (Harrell, 2015).

Steyerberg similarly identifies uncertainty in model selection as a source of overfitting and recommends limiting stepwise or related procedures where possible (Steyerberg, 2019).

Selection creates a second layer of uncertainty

Ordinary regression already contains sampling uncertainty in its coefficient estimates.

Data-driven variable selection adds another question:

Would we have selected the same model if the study had produced a slightly different sample?

A final model summary generally does not answer that question.

That is why reporting only coefficients, confidence intervals, and p-values from the selected model can underrepresent the uncertainty involved in reaching that model.

Model complexity and regression overfitting

Model flexibility is useful because a model must be sufficiently rich to represent reproducible relationships. But flexibility also creates opportunities to fit chance characteristics of the development sample.

Hastie et al. frame model selection through the interaction among bias, variance, and model complexity. Generalization error—not simply fit to the training observations—is the relevant target for predictive modeling.

As complexity increases, training fit can continue to improve even when the additional adaptation does not improve prediction of new observations. A model that adapts excessively to sample-specific variation has high effective variance and may generalize poorly.

This is the basic mechanism of regression overfitting.

That distinction is crucial.

Predictive optimism: why the development model can look better than it is

When a prediction model is evaluated on the same observations used to develop it, the result is apparent performance.

Because the development data influenced the fitted model, apparent performance can be optimistic about performance in new observations. The more the development strategy adapts to the sample—including through predictor selection—the more important it becomes to evaluate that complete process rather than only the final equation.

Steyerberg explicitly identifies model-selection uncertainty as an important source of overfitting (Steyerberg, 2019).

Overfitting can also produce predictions that are too extreme: low predicted probabilities can be too low and high predicted probabilities too high. Internal validation and shrinkage are therefore linked problems in prediction-model development.

Why predictor selection must happen inside validation

This is one of the most important rules in model validation.

Suppose a researcher:

1. uses the full dataset to select predictors;
2. fixes the resulting predictor set;
3. performs cross-validation only on that already selected equation.

The validation procedure has not reproduced the full model-building process. Information from observations later treated as validation observations has already influenced predictor selection.

Hastie et al. explicitly distinguish the wrong and right ways to perform cross-validation and place this issue within model assessment and selection (Hastie et al., 2009).

That includes predictor selection. Depending on the analysis, it can also include tuning or other outcome-driven modeling decisions.

Otherwise, validation evaluates a simpler procedure than the one actually used to produce the model and can therefore underestimate the optimism introduced during development.

Think of validation as validating an algorithm, not merely an equation

If the development rule is:

“Start with these candidate predictors, run backward elimination, retain variables according to this criterion, estimate the resulting coefficients, and produce predictions,”

then that entire rule is the object requiring validation.

The final equation is merely one realization of that algorithm in the observed sample.

Bootstrap assessment of instability and optimism

Resampling provides a practical way to ask how sensitive the modeling strategy is to perturbations of the data.

In bootstrap internal validation, repeated bootstrap samples are drawn and the model-development process is repeated. Performance in the bootstrap sample can then be compared with performance when that fitted model is applied back to the original observations, providing an estimate of optimism.

For data-driven predictor selection, the important point is to repeat the selection procedure in the bootstrap samples, rather than simply refitting one fixed set of variables.

Bootstrap resampling can also reveal model instability. Researchers can examine whether the same predictors repeatedly survive the selection process, whether model size changes substantially, and whether coefficients or predictions vary materially across resamples.

These assessments expose information that the single final model conceals.

Harrell's framework places resampling and validation directly within model development and simplification, while Steyerberg emphasizes that uncertainty from model selection contributes to overfitting and must be taken seriously (Harrell, 2015; Steyerberg, 2019).

Shrinkage: reduce the consequences of overfitting

Variable deletion is not the only way to control model complexity.

Shrinkage deliberately pulls estimated predictor effects toward zero. The idea is to accept some bias in coefficient estimates in exchange for reducing variance and improving performance on new observations.

This bias–variance trade-off is central to statistical learning. Hastie et al. discuss selection and shrinkage methods together and show how continuous shrinkage approaches differ from discrete subset selection (Hastie et al., 2009).

Shrinkage is particularly relevant when the problem is not that certain predictors are known to be irrelevant, but that coefficient estimates are too variable for the amount of information available.

Ridge regression: shrink rather than delete

Ridge regression penalizes coefficient magnitude and shrinks coefficients continuously toward zero rather than selecting a subset through hard inclusion/exclusion decisions.

Hastie et al. contrast ridge regression with subset-selection approaches and describe ridge as shrinking coefficients smoothly. Their comparison notes that ridge, lasso, and subset methods behave differently as the amount of regularization changes (Hastie et al., 2009).

This distinction can matter when predictors are correlated or when many predictors contain modest amounts of predictive information. Rather than forcing the analysis to declare one variable “in” and another “out,” ridge can distribute predictive information across coefficients while controlling their magnitude.

The resulting model is usually not sparse: coefficients are reduced but generally not set exactly to zero.

Lasso: shrinkage plus variable selection

The lasso also penalizes regression coefficients, but its penalty can drive some coefficients exactly to zero. It therefore combines shrinkage with a form of predictor selection.

Hastie et al. describe the lasso as lying between ridge regression and best-subset regression in its shrinkage behavior, combining characteristics of both approaches (Hastie et al., 2009).

Steyerberg identifies the lasso and related methods as promising when both prediction and parsimony are objectives (Steyerberg, 2019).

Lasso vs stepwise: the practical distinction

The key contrast is not simply “modern versus old.”

Ridge provides another option when shrinkage is desirable but a sparse predictor set is not the primary objective.

What to do instead of blindly running stepwise selection

1. Define the modeling goal first

Decide whether the primary objective is explanation, association estimation, or prediction.

For prediction, judge the model primarily by performance in new observations. For explanatory modeling, predictor inclusion must follow the scientific question and intended interpretation rather than simply whichever variables survive an automated prediction-oriented search.

2. Define candidate predictors before looking at their final p-values

Use the research question and available subject-matter information to define a defensible candidate predictor set.

Do not automatically treat univariable statistical significance as an admission criterion for multivariable modeling.

3. Control model complexity deliberately

Ask whether the available information can support the number of parameters, nonlinear terms, interactions, and selection decisions being considered.

A complex candidate model followed by aggressive searching does not become low-risk merely because the final equation happens to be short.

4. Prefer shrinkage when the problem is excessive estimation variance

When prediction is the objective, consider whether shrinkage or penalization is more appropriate than repeatedly testing predictors for inclusion.

Ridge regression provides continuous shrinkage. Lasso combines shrinkage with sparse selection. Their suitability depends on the prediction problem and should be judged through appropriate validation rather than assumed in advance (Hastie et al., 2009; Steyerberg, 2019).

5. Treat every data-driven modeling decision as part of development

If you perform predictor screening, backward elimination, forward selection, penalty tuning, or another adaptive procedure, document it as part of the model-building algorithm.

Do not describe the final model as though its specification had been fixed beforehand.

6. Validate the complete modeling strategy

For prediction, use resampling to estimate how the development process generalizes.

When variable selection is used, repeat that selection inside each relevant resample. When a regularization parameter is tuned, the tuning procedure must likewise be contained within the validation framework used to assess performance.

This principle follows directly from the distinction between genuine out-of-sample assessment and allowing information from nominal validation observations to influence model construction (Hastie et al., 2009).

7. Examine stability, not only the winning model

Ask how often predictors are retained across resamples and whether coefficients, model size, or predictions change materially.

A predictor appearing in the one final model is weaker evidence of reproducibility than a predictor whose role remains reasonably stable when the data are perturbed.

8. Report optimism-corrected predictive performance

Do not present development-sample performance as though it were expected performance in new observations.

Use appropriate internal validation to quantify optimism. For clinical prediction models, bootstrap approaches are particularly useful because the development process can be repeated while making efficient use of the available development data (Harrell, 2015; Steyerberg, 2019).

A practical predictor-selection decision framework

Before selecting variables, work through the following questions.

Decision sequence: target → candidate model → supportable complexity → need for selection → selection uncertainty → optimism assessment → tuning inside validation.

Common stepwise regression problems

“Only significant univariable predictors should enter the multivariable model.”

That rule lets marginal outcome associations determine eligibility for a model intended to estimate conditional relationships or predictions. Univariable screening is itself a data-driven selection procedure and should not be treated as neutral.

“Backward elimination solved overfitting because the final model has only five predictors.”

Not necessarily. Final model size does not reveal how much searching was required to obtain it. A five-predictor model selected after extensive outcome-driven searching is not equivalent to five predictors specified in advance.

“The final coefficients have standard errors, so selection uncertainty is already included.”

Ordinary selected-model output generally conditions on the selected specification rather than fully representing the uncertainty generated by searching across alternative models. Harrell specifically contrasts ordinary step-down calculations that behave as though the reduced model were prespecified with approaches that recognize selection-related variation (Harrell, 2015).

“Cross-validation fixes the problem even if selection was performed first.”

Not if the full dataset influenced the predictor-selection step before cross-validation. Model construction must be confined to the training portion within the validation procedure. Hastie et al.'s treatment of the wrong and right ways to perform cross-validation addresses precisely this general leakage problem (Hastie et al., 2009).

“Lasso eliminates overfitting.”

Lasso controls complexity through penalization and can provide sparse models, but it remains a data-dependent modeling procedure. Penalty selection and predictive performance still require appropriate validation.

“Ridge is useless because it does not select variables.”

That conclusion assumes that parsimony is the only objective. Ridge is a shrinkage method rather than a sparse-selection method. When prediction and variance control matter more than producing the shortest equation, retaining predictors with shrunken coefficients can be entirely consistent with the modeling goal (Hastie et al., 2009).

Reporting checklist

Before reporting a model developed with predictor selection, check that the manuscript states:

• the primary objective—prediction, explanation, or association estimation;
• the complete candidate predictor set;
• whether predictors were screened using univariable analyses;
• the exact forward, backward, subset, or penalized procedure used;
• the selection or tuning criterion;
• whether any predictors were forced into the model;
• whether nonlinear terms and interactions were candidates for selection;
• how missing data were handled before and during model development;
• how model stability was assessed;
• whether the complete development procedure was repeated during internal validation;
• the distinction between apparent and optimism-corrected predictive performance; and
• the limitations of interpreting coefficients after data-driven model searching.

Bottom line

The most important stepwise regression problems arise from treating variable selection as though it created no additional uncertainty.

Forward selection, backward elimination, and univariable p-value screening all use observed data to determine the final model. The resulting specification can be sample-dependent, and the search itself can contribute to regression overfitting. Harrell warns against treating ordinary selected models as though they had been prespecified, while Steyerberg identifies model-selection uncertainty as an important source of overfitting (Harrell, 2015; Steyerberg, 2019).

For prediction, the remedy is not simply to replace stepwise regression with another automatic algorithm. Start from the prediction goal, control complexity deliberately, consider shrinkage or penalization where appropriate, and validate the entire development procedure. Ridge and lasso offer principled forms of regularization, but they too must be tuned and validated as complete modeling strategies (Hastie et al., 2009).

FAQs

What are the main problems with stepwise regression?

Stepwise regression uses the observed data to determine which predictors appear in the final model. This introduces model-selection uncertainty that is not adequately represented by treating the resulting equation as though it had been specified beforehand. Selection can also contribute to overfitting and instability, particularly when the available information is limited relative to the candidate model complexity (Harrell, 2015; Steyerberg, 2019).

Why can stepwise regression select different variables in different samples?

The inclusion and exclusion decisions depend on estimated relationships in the particular sample. When several predictors contain overlapping information or estimated effects lie near a selection boundary, modest changes in the observations can change which variables are retained. This sample dependence is one manifestation of model-selection uncertainty.

Should I screen predictors with univariable p-values before regression?

Univariable screening should not be treated as a default or neutral way to construct a multivariable model. It uses outcome information to determine predictor eligibility and ignores the distinction between marginal and multivariable roles. Predictor strategy should instead follow the modeling objective and scientific context.

Does backward elimination prevent overfitting?

Not automatically. Reducing the number of predictors can reduce the complexity of the final equation, but the process used to choose that equation remains data-dependent. The selection procedure itself should be incorporated into validation when predictive performance is being estimated.

What is the difference between lasso and stepwise regression?

Stepwise procedures generally make discrete inclusion/exclusion decisions according to a search rule. Lasso controls complexity by penalizing coefficient magnitude and can shrink some coefficients exactly to zero. Thus lasso combines regularization with sparse predictor selection, whereas ordinary stepwise regression does not inherently provide continuous coefficient shrinkage (Hastie et al., 2009).

What is the difference between ridge regression and lasso?

Both are regularization methods. Ridge regression continuously shrinks coefficients toward zero but generally retains them, whereas lasso can shrink some coefficients exactly to zero and therefore perform variable selection. Hastie et al. describe lasso as having properties intermediate between ridge and best-subset approaches (Hastie et al., 2009).

Why must variable selection be repeated inside cross-validation?

If predictor selection is performed before cross-validation using all observations, information from nominal validation observations has already influenced model construction. The validation estimate therefore does not represent the performance of the complete development procedure on genuinely unseen observations. Selection should occur using only the training data available within each resampling iteration (Hastie et al., 2009).

How can bootstrap resampling help with variable selection?

Bootstrap resampling can repeatedly reproduce the development procedure on perturbed versions of the dataset. This permits assessment of optimism and can also reveal whether predictor inclusion, coefficients, or predictions are stable across resamples. For prediction models, the relevant selection steps should be replayed during validation rather than validating only the already selected final equation (Harrell, 2015; Steyerberg, 2019).

Is stepwise regression always unacceptable?

The sources support a more contextual conclusion. Ordinary stepwise procedures have important limitations and should not be treated as a default strategy or as though their selected models were prespecified. Harrell also discusses settings in which backward selection can be competitive, such as when a large fraction of candidate predictors are genuinely irrelevant, while emphasizing alternative model-approximation and shrinkage strategies (Harrell, 2015).

The appropriate question is therefore not whether a method is universally forbidden, but whether its assumptions, instability, selection uncertainty, and validation have been handled appropriately for the modeling goal.

References

Harrell, F. E., Jr. (2015). Regression modeling strategies: With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer. https://doi.org/10.1007/978-3-319-19425-7

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.

Steyerberg, E. W. (2019). Clinical prediction models: A practical approach to development, validation, and updating (2nd ed.). Springer. https://doi.org/10.1007/978-3-030-16399-0