How charter school lotteries work like randomized controlled trials

graphic randomized controlled trial charter school selection bias lottery test scores

A reader’s comment about my recent post, “It’s not because of skimming”, shows possible confusion about what we can learn from studies comparing students who entered the lottery and lost versus those who entered the lottery and won.

I made a graphic to illustrate. Let’s assume the kids who apply for charter schools (on the blue side) are a self-selected group with supportive, motivated families and other characteristics for success. On average, the kids on the blue side are likely to do better than the kids on the purple side, no matter what school they go to.

The interesting question is, “What difference does the school make?” The graphic shows two blue arrows: one for kids who entered the lottery and lost, and one for kids who entered the lottery and WON a place at a charter school.

According to the studies cited in the op-ed, the kids who won a place at a charter school got “about four extra years worth of education by the time they complete high school.” That sounds like a very effective intervention.

Medical trials work in a similar way. Research volunteers are randomly divided into two groups: an experimental group and a control group. The experimental group gets the drug (or, charter school) and the control group gets the placebo (or, no charter school). The results are compared to see if the drug is effective. Even if the volunteers are more healthy or less healthy than the population as a whole, that’s irrelevant because they are randomly assigned to two groups.

So that’s how charter school lotteries work like a randomized controlled trial. Hope that clears things up.

sachartermoms

Parent-activist and education blogger in San Antonio, Texas, USA. Helping parents make informed school choices and explore cultural activities.

One Comment

  1. Thanks for the graphic. All you have to do is change the third arrow to better test scores and you will have an accurate description of the results of my study of test scores BEFORE students enter a charter. Specifically, my point is that kids entering charter schools generally have higher achievement than those mot entering charter schools. This–in conjunction with the loss of lower performing kids (some argue that they are pushed out and qualitative research supports such a conclusion) produces a peer effect in charter schools that cannot be replicated by public schools. Thus, some degree of the greater achievement in SOME charter schools (the majority of charters in Texas perform worse than average) is due to peer effects.

    In sum: kids not entering charters–low scores. Kids applying to enter–higher scores. Kids entering charters–higher scores. Yep–kids entering charters HAVE HIGHER SCORES. My study does not examine achievement, but I do suggest that peer effects play some role in higher achievement. More sophisticated analyses need to be done to parse out what percentage of the gain is due to peer effects.

    Oh–by the way–the majority of KIPP elementary schools in Texas have negative student growth on TAKS.

Leave a Reply

Your email address will not be published. Required fields are marked *