Steps in validating an instrument
For an instrument to have high external validity, it must be applicable to a diverse group of people and a wide array of natural environments. When the study is complete, researchers may still invalidate their data by making a conclusion error. A Type I error is concluding there is no relationship between experimental variables when, in fact, there is.
Conversely, a Type II error is claiming a relationship exists when the correlation is merely the result of flawed data.
There are numerous statistical tests and measures to assess the validity of quantitative instruments, which generally involves pilot testing.
The remainder of this discussion focuses on external validity and content validity. Establishing eternal validity for an instrument, then, follows directly from sampling.
Instruments fall into two broad categories, researcher-completed and subject-completed, distinguished by those instruments that researchers administer versus those that are completed by participants.
Some qualitative researchers reject the concept of validity due to the constructivist viewpoint that reality is unique to the individual, and cannot be generalized.
These researchers argue for a different standard for judging research quality.
Consider the SAT, used as a predictor of success in college.
It is a reliable test (high scores relate to high GPA), though only a moderately valid indicator of success (due to the lack of structured environment – class attendance, parent-regulated study, and sleeping habits – each holistically related to success).