[sidebar to What Pinker Got Wrong]
Suppose we were serious about using test-based measures of academic merit to determine eligibility for top colleges. We’d need to construct a new test. Here’s an example, which I’ll call SECT, for super elite comprehensive testing.
To be eligible for SECT, a student would have to score at the 95th percentile on the SAT, the ACT, or other equivalent. This would produce about 100,000 eligible test takers, more or less. The point of SECT is to raise the ceiling on test performance: to effectively discriminate between the 99th percentile, the 99.9th percentile, and the 99.99th percentile, in a way that is not possible for the SAT, the ACT, or any other test that has been normed on the total population.
SECT might involve 12 -15 hours of testing over 2-3 days. We could start with the Verbal and Math sections of the Graduate Record Examination; add portions of the LSAT to test for verbal analytic reasoning; add portions of the GMAT to test for quantitative analytic reasoning; add the relevant portions of the MCAT to test for scientific understanding; and maybe draw on portions of the relevant GRE subject tests and AP tests to tap knowledge of history.
The goal here would be a test so damned hard that a student just nudging over the 99th percentile on the SAT would be expected to score only at the 50th percentile on the SECT. We’d norm it to four standard deviations instead of three, with a range of 100 to 900 instead of 200 to 800. If need be, we can select louche passages from Kant or Hegel for the verbal section, and solicit physics questions from Stephen Hawking for the science section. If well constructed, then in some years, none of the 100,000 test takers will score a 900; in others, only one or two will receive a perfect score.
Students scoring at the 90th percentile and above on the SECT, maybe 10,000 or so, would automatically have an application generated for up to four Ivy League schools of their choice, with the near certainty of acceptance to at least one. I set the yield at 10,000, on the expectation that about one third of qualifying students, for inexplicable reasons, might choose to attend CalTech or the University of Chicago, leaving just enough top scorers to fill the 7000 seats available at the Ivy League.
By constructing a test on, and normed for, only the upper tail of the distribution of academic merit, we can obtain reliable scores through most of that tail, and successfully discriminate the 99th percentile student from the 99.9th, and those students in turn from the 99.99th. We still won’t be able to reliably discriminate the 99.99th percentile students in the totalk population from the 99.95th percentile students—the laws of random error in test scores continue to apply—but we don’t need to, because the numbers are now small enough we can accept all such students.
Whether SECT would solve the problem of determining those students most worthy of attending a top-ranked and prestigious university is an empirical question. Because of the small number of test-takers, in conjunction with today’s Big Data capabilities, we could do a comprehensive outcomes assessment over decades. Does SECT predict not only which students will complete Law School, but which ones go on to become judges or law professors?* Do SECT scores predict not only who will attend Ph.D. school, but who will complete the degree, and publish, and get tenure?
* old Law School joke: The alumni office reports the results of a survey as follows. “We found that our A students get appointments as law professors. Our B students get appointed as judges. And our C students … get rich.”
These really are empirical questions. If success in life and the world were instead to follow a threshold model, then obtained outcomes might NOT covary with SECT scores. Under a threshold model, every high scorer on the SAT—everyone scoring at the 95th percentile and above—would have good prospects in life, better prospects than those scoring around the 50th percentile on the SAT. But it would make no difference whether you scored a 300 or a 700 on the SECT, because after you come in above threshold on the broad population measure, life success is a function of drive, grit, emotional intelligence and sundry other skills not measured by SECT; or conversely, of the slings and arrows of an outrageous fortune.
SECT would remain a matter of being able to sit still for hours, reading, writing and figuring. Human abilities comprise a larger set of activities. I wouldn’t accept any SECT score, no matter how high, as the sole qualification needed to be my airline pilot. Or as the sole admissions criterion for the Air Force Academy program to train fighter pilots.