Data Driven Nonsense--Few Credible Studies of Charter Schools

Teachers in the trenches have long suspected that the “data-driven” reforms being shoved down their throats by Gates, Broad, Walton, Obama, Duncan, Rhee, et al, had little credible data to back them up and little hope of improving educational outcomes for their students. An article in the latest issue of Science (reposted on EurekAlert) lends credibility to these suspicions.

According to UC San Diego educational economist JuIian Betts and Richard Atkinson, president emeritus of the University of California and former director of the National Science Foundation, most studies of charter schools "use unsophisticated methods that tell us little about causal effects."  There are very few randomized studies. Most rely on comparisons between charter schools and nearby traditional public schools in which subjects are self-selected. Bretts called such studies "naïve and essentially meaningless."

In a meta-analysis of the available literature on charter schools, Betts and Emily Tang found that 75% of studies had to be discarded because they failed to account for differences in the backgrounds and academic histories of traditional public-school students and those who chose to go to a charter. Children and families who choose to go to charter schools and who go to the meetings, fill out the extra paper work and jump through the necessary hoops may be significantly different from those who stay in traditional schools in terms of academic background, motivation, parental support, prerequisite skills, perseverance and numerous other factors that affect academic achievement.

Betts and Atkinson argue that lottery-based studies can provide meaningful data as students are randomized: those who lose the lottery ostensibly do so by the luck of the draw, and not by poverty or lack of parental pressure. The losers continue in the neighborhood school and can be compared with peers who won the lottery and transferred to the charter school.

The problem with this analytical approach is that only 2% of charter schools have been examined in this way and the majority of charter schools do not hold lotteries, thus precluding them from this methodology. It is also likely, as Betts pointed out, that the schools that do hold lotteries are the most popular and best quality, suggesting that their data is not at all representative of charter schools in general.