Average 2009 NAEP Score By State Teacher Contract Laws
States with binding teacher contracts
4th grade: Math 240.0 Reading 220.7
8th grade: Math 282.1 Reading 263.7
States without binding teacher contracts
4th grade: Math 237.7 Reading 217.5
8th grade: Math 281.2 Reading 259.5
Reprinted from PERFORMANCE-ENHANCING TEACHER CONTRACTS?
Some possible explanations: contracts, tenure, collective bargaining all contribute toward retaining more experienced teachers. Job security and due process rights encourage more open and honest discourse during meetings and collaboration which help foster more effective reform and school improvement.
The results of the Shankar study should, of course, be taken with a grain of salt. There were lots of variables there weren't controlled or at least disaggregated from their data. Most of the non-unionized states were from the South, with higher degrees of poverty and in many (but not all) cases lower levels of per pupil spending.
Nevertheless, if unions were the only problem, as many critics assert in their reductionist claims, then non-unionized states should have higher test scores, and they clearly do not.