Research on the Efficacy and Validity of Test Prep

The National Association for College Admissions Counseling recently released some research regarding SAT test prep. The results are spun in many different ways by different publications. Many agree that the most comprehensive write-up about the research findings can be found here

One colleague, Shuan McElroy (counselor at the Shanghai American School), summoned up the results very well in a post on the NACAC e-list.

Technically a 30 points increase is NOT statiscally significant, even if it feels that way. You see, any standardized test has a Standard Error of difference:

Standard Error of the Difference (SED): The SED is a tool for assessing how much two test scores must differ before they indicate ability differences. To be confident that two scores indicate a true difference in ability, the scores must differ by at least the SED times 1.5. For example, SAT verbal and math scores must differ by 60 points (40 ׳ 1.5) in order to indicate true differences of ability.

30 points versus 60 points. Hmmmm. Yet an astounding "20% to 40% of officials at 130 colleges that consider the SAT in admissions said a 20-point math increase or a 10-point reading increase would "significantly improve a student's chances of admissions" if all other factors in a student's application were the same." Given that the author's study concludes:

* All students should practice for tests, but potential benefits must be balanced against the costs, including time and money.

* Students should be counseled to use less costly forms of test prep, such as books or the Internet, and be "wary of coaching rip-offs."

* If time and money are no object, commercial or private coaching may be worth the cost but "this will primarily be true for students with above-average admission test scores in the first place."