[sidebar to What Pinker Got Wrong]
Every reader of Pinker’s is familiar with media reports of Gallup polls. Reported results always give a “margin of error” for national surveys, typically “+/- 3%.” The same concept applies to tests of mental ability. The College Board, for instance, speaks of test results being accurate within a range; historically, to within +/- 30 points.
The underlying statistical model is sometimes expressed as follows:
Obtained score = true score + random error.
Thus, obtaining a score of 500 on the SAT might correspond to a true score at any point in the range of between 470 and 530. Your true abilities can’t be estimated any more accurately; the resolution of the test is limited to +/- 30. Because error is random, your obtained score may be biased upward or downward.
So far, so good. Now consider a score of 790. Error cannot be +/- 30 here; there’s no such thing as a true score of 810 or 820, because these scores don’t exist in the universe of this test. Hence, for any score above 770, error has to be more positive than negative. Translation: scores of 780, 790, and 800 are more and more likely to be over-estimates of the student’s true level of ability.
* See Nunnally, Psychometric Theory, 2nd edition, Figure 6-3, p. 223, for a discussion
A college that required near perfect test scores would often admit lucky students whose obtained test scores were upwardly biased. Many students at that Ivy League college would have a spuriously inflated sense of self-worth.
Hmm…