New Research on Admissions Testing and Test Preparation
This fall the online journal International Research in Education published the final component of my PhD research in an article titled Preparing for High-Stakes Admissions Tests: A Moderation Mediation Analysis. This peer-reviewed study examined the factors for success in high stakes admissions testing. Collaborating with Dr. Karen Zabrucky from Georgia State University, Dr. DeWayne Moore and Janelle Cheung from Clemson University, we investigated the relationships between various aspects of test preparation, socioeconomic and demographic factors, to better understand what makes test prep effective, and in what contexts.
Our research team crafted a theoretical model of particular factors which could influence testing outcomes. These factors included the amount of homework completed; the number of tutoring sessions, practice tests, and official tests; the distribution of sessions over time, and the timing of the preparation. We conducted statistical analyses on these factors and determined the relationships between these variables and the increase on SAT scores.
We had multiple hypotheses to test: Does it matter if sessions are spread out or massed together? Do individual tutoring hours lead to greater increases than group hours? How important are practice tests? Do students continue to attain score increases by taking additional official SATs? Does it matter if students complete their assigned SAT homework? Our statistical analysis confirmed the value of practice tests, which conferred a 6.9 point increase on final SAT score, the value of completing SAT homework, and the importance of taking the SAT multiple times. Our analysis of individual tutoring hours revealed that 1-1 tutoring hours yielded 60% higher score returns over group tutoring hours. The model demonstrated the importance of spreading tutoring sessions out over time and the benefits of starting SAT prep earlier in the junior year.
We added another layer to our model, exploring whether particular demographic factors might influence outcomes. We were curious whether homework mattered more or less for different types of students. Do males or females get a bigger bump from completing homework? Do students from higher or lower income levels benefit differently from homework completion? How about students from public versus private schools? We explored these effects, called moderating or conditional effects, throughout the model.
We also explored relationships within the model. We found that students who start prepping earlier in the year tend to have greater score increases, but do these gains come from having more distributed sessions, more prep time, more practice and official tests? We were looking for mediating (i.e. indirect) effects within the model. And the big kahuna, we explored moderated mediation which examines how mediation only shows up for certain kinds of students. We ran our model again using a statistical method called path analysis to better understand these new relationships.
Our analysis found that score gains which appear to derive from early start times must be partially attributed to the greater distribution of sessions, increase in contact hours, greater homework completion and more practice and official SAT tests. So it’s not the early start time that matters, it’s what students do with the extra time that matters: more preparation, more practice, more official tests.
One finding that surprised our team was that the positive relationship between homework completion and SAT score increase was much more powerful for public school students: our private school students seemed to derive less of a benefit from completing their SAT homework than did their public school counterparts. The net benefit of completing 100% of assigned homework would be 42 points higher for a student in public school, a practically meaningful increase in terms of college admissions and scholarships. We found no conditional effects for gender, indicating that gender was not a moderating factor for any relations depicted in the model.
When we examined the effects of public school/private school attendance and lower versus higher socioeconomic status, we found several conditional effects in the model. Private school students benefit more from starting early through their increase in private and group preparation. Each additional month of SAT preparation would yield .53 hours of individual tutoring and .73 hours of group tutoring for a student in private school, compared to .16 hours of individual and .34 hours of group tutoring for a public school student. In a similar fashion, socioeconomic status (SES) moderated the relationship between start time and both individual and group tutoring hours. Starting SAT preparation a month earlier led to an increase in .76 individual and .82 group hours for a high SES student and an increase of only .24 individual hours and .55 group hours for a low SES student, indicating the increased tendency of more affluent students to procure more expensive and individualized forms of SAT preparation.
Our team was happy to contribute to the academic literature exploring the efficacy of SAT preparation and the various factors which contribute to success on the SAT. In the end, cramming isn’t great, and practice effects are very real and powerful. When you invest more time in this process, you tend to have a more meaningful score increase. Starting early is helpful, but only in so much as it leads to more time preparing, more distributed sessions and more practice and official tests. Gender seems to have nothing to do with the efficacy of SAT preparation. One of my favorite findings from this research was that an hour of test preparation and an additional SAT practice test seemed to have the same effects on a student from private or public school or a student from a more affluent family or a less affluent family. The effectiveness of test preparation seems to be blind to gender and class. If we can deliver the preparation, we can help all kinds of students, from all walks of life.