The Emperor's New Clothes

I am intimately familiar with the SAT and its traveling companions—dread and disgust. Until a few months ago, I spent almost every evening for a year at the homes of high-school sophomores, juniors, and seniors, selling insights into the most despised test in the land. Sure, some of my students had anxiety about how their test performance would affect their college choices, which would in turn affect—well, something else, maybe. But most of them were just beaten down by the barrage of importance everyone has attached to the test, and looked forward to that brief window of pain-free days—post-test, pre-results—when they didn�t have the weight of the SAT slung around their necks like an albatross of analogies.

So it was with great relief that I heard that the test, formerly known as the SAT I, was undergoing major changes. Richard Atkinson, who was the president of the University of California system through 2003, had threatened to stop using the test as a factor in UC admissions decisions unless the College Board did something to make its test more relevant. In a last-ditch attempt to keep the money flowing from the hundreds of thousands of students who would have gladly dropped the test, the non-profit College Board (whose president, Gaston Caperton, once stated that dropping the old SAT would be akin to dropping grades from high schools) retooled its prize creation into Frankenstein�s monster, sewing on questions from another test and cutting out some old parts that Atkinson didn't like.

University of California Richard Atkinson

Atkinson had several legitimate criticisms of the old test (including that it had an inherent bias against minorities), so when the California board of Regents evaluated the new one and decided to give it a three-year trial, I figured it must be better—or at least different.

The new SAT, which has its first clinical trial with high-school guinea pigs this coming Saturday, is neither. Somehow, it is both the same and slightly worse.

Here�s why:

Coachability

BusinessWeek cited figures from Eduventures, an education market-research firm, putting test-prep revenue at $960 million this year, up from $702 million in 2003. Business will continue to boom for the coaching business as anxious parents send their charges to tutors in an effort to get a leg up on the new test. Sadly, they�re doing the right thing.

Students can get a better score on the SAT through tutoring. I say this even though the College Board has statistics that show that coached students are only slightly more likely to get large point gains than non-coached students, and that one-third of coached students see no change at all in their scores or even lose points. The reason? There are a lot of bad teachers and books out there chasing those $960 million. Some even advocate shortcuts that ultimately hurt the students� scores, such as guessing if you don't know the answer right away.

In independent studies by International Communications Research, a market-research and polling firm, students who received tutoring from the Princeton Review—a national tutoring service—were shown to gain, on average, 120 to 140 points on their SAT after three months of classroom tutoring. More high-end tutoring services, such as the full-service one-on-one tutoring operation I worked for, help increase their students� scores by an average of 200 points. If it looks to you like the more students pay, the more they benefit, you�re on the right track.

Atkinson told Gelf that the Board�s new test will be less coachable than its previous incarnation. �A lot of the old SAT I was based on trickery,� he said. �So the coaching was based on trickery. It will all disappear [with the changes].�

But the new writing section (read: a barely modified old writing SAT II) is perhaps the most coachable of any part of any SAT, including all of the SAT II subject tests. (Another ICR survey found that students who received Princeton Review tutoring on their Writing SAT II raised their scores an average of 137 points. Comparable preparation on other SAT II subject tests yielded an increase of around 80 points). A colleague of mine in New York claimed that if given six months' time, he could help any student get an 800 on the writing test, regardless of the student's previous score—even 500 or lower. Some of my students also gained about 300 points on the writing test.

Curriculum Change

The College Board has been placed in an awkward position by the new test. It needs to continue to appease Atkinson and others who demanded changes. But it also needs to convince its other client colleges and universities that the same standards they have used for years will continue to be applicable. So while the Board states the SAT has been completely overhauled, it simultaneously claims that the sections of the old and new tests will result in the same scores for students. �It�s an exact correlation,� Caren Scoropanos, the Board�s Associate Director of Public Affairs, told Gelf. �The math and verbal scores match up.�

It would be an impressive feat to simultaneously implement a radical change and maintain the status quo—the testing equivalent of a bloodless coup. If only it were so. Instead, the Board has made relatively few concessions, while its former foes are pre-emptively claiming victory.

Atkinson thinks the new test will become a force for positive curriculum change. �It will cause a dramatic change,� he said. �It will transform what happens in high schools.� According to Atkinson, teachers can now focus on teaching important skills such as writing, rather than the bastion of arcane trickery that was the SAT I. �The point is to send a message to kids and schools: Don�t mess around with silly stuff like analogies,� he said.

In reality, though, the few changes the Board has made—such as adding higher-level math questions, more technical jargon, and an essay—will hurt students from poor families and underfunded schools. In an address at the annual meeting of the American Educational Research Association in San Diego last year, Atkinson inadvertently supported this point. In explaining how the changes to the SAT impacted his granddaughter, he said, �Her high school quickly adjusted to the proposed changes, and now has students writing a 25-minute essay once a week in preparation for the new test.�

It�s great that she attends a school with the resources to do this, but only high schools with enough teachers and funding can be flexible enough to incorporate the demands of the new SAT into their curricula. The others can play catch-up, as usual. Atkinson admits that some schools will take longer to adjust, but insists that the well documented socio-economic gap in old SAT scores will shrink with the new test. �I really believe it�s going to be a fairer test,� he said. �Over time it will become even more fair.�

Enhanced Biases?

Atkinson objected to chunks of the old SAT that he felt were out of touch with what K-12 education should be about. In his 2001 speech at a meeting of the American Council on Education that eventually led to SAT reform, Atkinson said that his concern about schools' overemphasis on test-taking skills was cemented when he visited a private school that was spending class time teaching 12-year olds to learn complicated verbal analogies. At the same time, schools with fewer resources simply gave up, thinking the test evaluated only innate intelligence, which can't be coached.

Atkinson told Gelf that the changes to the test were part of a conscious effort to move the SAT away from any associations with innate-intelligence testing, and the connotation to some that studying can't help. �Underrepresented minorities will reduce the size of the gap,� he said. �It will be clear to them what they have to accomplish. They�ve got to know it�s not an I.Q. test.�

So analogies were banished from the verbal section—those 15 slots have been replaced by more reading-comprehension questions. Other stuff in the verbal section remains virtually untouched. Also gone are the analogies� geeky brothers—the quantitative comparisons of the math section, better known to math-phobes as column A/B. (Is A bigger than B, or B bigger than A, or are they same, or do we not have enough information?) In place of those questions, students can now find more technical questions—add Algebra II to the list of subjects they need to study for the test.

Ironically, the two sections that were eliminated were the same two in which minorities consistently overperformed, according to research done by Jay Rosner, the Executive Director of the Princeton Review Foundation, a non-profit branch of the testing company that, among other things, serves as an advocate for low-income and minority students. By analyzing scores from hundreds of thousands of students, Rosner has shown that both the quantitative comparisons and the analogies sections were relatively advantageous to black students, meaning that the score gap between blacks and whites was significantly lower in those sections.

The College Board, though, has its own studies that show that the difference in scores between whites and minorities does not change in the different subsections. �It�s nonsense to think it would be biased in that way,� Scoropanos said of Rosner�s claims.

But Rosner said that, if the Board chose to do so, it could choose questions that significantly reduce the scoring gap between white and minority students throughout the entire test. He has picked apart four tests to find viable questions with the smallest gap in racial performance—creating one test in which the scoring gap between whites and blacks is reduced by 40%. Though he can�t spot any obvious difference between the questions that produce smaller gaps and those that don�t, Rosner has shown that such questions exist. �What you can conclude is that somehow the question-selection process generates these results,� he said. �SAT questions capture something about race.�

Strategically manipulating the test for the express purpose of reducing the race gap does not sit well with me. But it may be the best idea yet about how to deal with the newly revived institution that is the SAT, and that shows just how imperfect the test remains.

Email this entry to:

Your email address:

Message (optional):

Email this entry to:

Your email address:

Message (optional):