Reliability is not validity: a measurement decision guide

A consistent score can still fail to measure the intended construct or support the comparison a study needs.

Ask what inference the score must support

Reliability concerns consistency under a stated model. Validity concerns the evidence and reasoning supporting a particular interpretation and use.

Inspect more than one coefficient

Item content, response process, dimensionality, group behavior, missingness, and consequences can expose problems that a high internal-consistency value cannot.

Match evidence to the claim

A scale used for group comparison may need invariance evidence; a screening decision may require calibration and classification evidence. There is no universal validity checkbox.

Failure modes to check

• Choosing a method because it is familiar rather than because it matches the estimand and data structure.
• Treating assumptions as a generic pass/fail checklist instead of evidence about a specific model.
• Changing the workflow after seeing results without recording the decision or its effect on uncertainty.
• Reporting software output without tracing the claim to final code, diagnostics, and limitations.

Continue the decision path

Explore the related statistical consulting service, or discuss the decision in your own study.

Related decision guides

• EFA vs CFA: choose from the research decision, not the software menu
• Choosing a regression family from the outcome and data structure