Psychometrics
arnold on Sep 17, 2008
Reliability is the desired consistency (or reproducibility) of test scores. In practical terms reliability is the degree to which individual’ deviation scores, or z-scores, remain relatively consistent over repeated administration of the same test or alternate test forms (Crocker and Algina, 1986, p. 105).
Test developers must demonstrate that the scores obtained are reliable otherwise the results are not useful because they may be inconsistent. This is done with empirical studies which can demonstrate the reliability of scores obtained from their tests.
What makes test scores unreliable?
There are two types of errors, random and systematic. Systematic measurement errors are those which consistently affect and individuals score because of some particular characteristic of the person or the test that has nothing to do with the construct being meaured (Crocker and Algina, 1986, p. 105). This causes test scores to be inaccurate. Random errors of measurement affect and individual’s score because of purely chance happenings. For example, guessing, distractions and fluctuations in the individuals state (Crocker and Algina, 1986, p. 106). This reduces both the consistency and usefulness of the test scores.
The Reliability Coefficient can be defined as the correlation between scores on parallel test forms. According to classical true score theory, two tests are defined as parallel when:
1) Each examinee has the same true score on both forms of the test, and
2) The error variances for the two forms are equal (Crocker and Algina, 1986, p. 115).
What does the reliability coefficient tell you?
Let s say that its value is .81
1) 81% of the observed score variance is attributable to true score variance for the examinee group.
2) If the standard deviation of the observed score was 4 points, you could predict that the standard deviation of the unobservable true score distribution would be 3.6 points.
3) 65% of the observed score variance on the second test could be predicted by the variance of the observed scores on the first test. (By squaring .81)
4) The correlation between observed scores and true scores is .9 (by taking the standard deviation)
Reference: Crocker and Algina (1986), “Introduction to Classical & Modern Test Theory,” CBS College Publishing.
