Photograph Credit; GuidoVrola; iStock
By Jordan Weissmann | Originally Published at Slate. June 12 2014 |

The amazing, completely made-up statistic a judge used to strike down teacher tenure in California

This week Los Angeles Superior Court Judge Rolf Treu handed the education reform movement a stunning legal victory, when he struck down California’s teacher tenure laws for discriminating against poor and minority students. The statutes made it so onerous to fire bad teachers, he wrote, that they all but guaranteed needy kids would be stuck in classrooms with incompetent instructors—rendering the laws unconstitutional.

As evidence, Treu cited a statistic that sounded damning: According to a state witness, between 1 and 3 percent of California’s teachers could be considered “grossly ineffective.” Here was the passage:

There is also no dispute that there are a significant number of grossly ineffective teachers currently active in California classrooms. Dr. Berliner, an expert called by State Defendants, testified that 1 to 3% of teachers in California are grossly ineffective. Given that that the evidence showed roughly 275,000 active teachers in this state, the extrapolated number of grossly ineffective teachers ranges from 2,750 to 8,250. Considering the effect of grossly ineffective teachers on students … it therefore cannot be gainsaid that the number of grossly ineffective teachers has a direct, real, appreciable, and negative impact on a significant number of California students, now and well into the future for as long as said teachers hold their positions.

This seemed like a fairly important piece of the decision—if you’re going to argue in court that a state law is dooming children to second-rate educations, you ought to be able to quantify the problem. Politically, it also seemed liked a pretty awful indictment of the state government if officials knew for certain that so many useless teachers were lounging around California’s classrooms. But where did this number come from?

Nowhere, it turns out. It’s made up. Or a “guesstimate,” as David Berliner, the expert witness Treu quoted, explained to me when I called him on Wednesday. It’s not based on any specific data, or any rigorous research about California schools in particular. “I pulled that out of the air,” says Berliner, an emeritus professor of education at Arizona State University. “There’s no data on that. That’s just a ballpark estimate, based on my visiting lots and lots of classrooms.” He also never used the words “grossly ineffective.”

The expert cited in the ruling doesn’t even necessarily believe that low test scores qualify somebody as a bad teacher.

The phrase appears to have been Treu’s shorthand to describe teachers whose students consistently perform poorly on standardized tests. But Berliner is a well-known critic of using student test scores—or “value-added models,” in the parlance of education experts—to measure teaching skills. In part, that’s because research suggests that teachers don’t really control much of how their pupils perform on exams; according to the American Statistical Association, they influence anywhere between 1 percent and 14 percent of the variation in students’ scores. As result, teachers often don’t deliver the same results year after year.

Still, if you look at enough data, there are always a few teachers who consistently underperform on test results. “There’s an occasional teacher who shows up really good a few years in a row,” Berliner said. “There are a few who show up really bad.” And that’s where the now-infamous statistic comes in. During a deposition, Berliner told me, the plaintiffs’ lawyers asked how many teachers deliver low test scores year after year. He didn’t have a hard number, so he said 1 percent to 3 percent, which he thought sounded suitably small. That led to the following exchange with a plaintiffs’ lawyer during cross-examination. (Berliner emailed me part of the transcript.)

Lawyer: Dr. Berliner, over four years value-added models should be able to identify the very good teachers, right?

Berliner: They should.

Lawyer: And over four years value-added models should be able to identify the very bad teachers, right?

Berliner: They should.

Lawyer: That is because there is a small percentage of teachers who consistently have strong negative effects on student outcomes no matter what classroom and school compositions they deal with, right?

Berliner: That appears to be the case.

Lawyer: And it would be reasonable to estimate that 1 to 3 percent of teachers fall in that category, right?

Berliner: Correct.

Berliner seems to have let something important get lost in translation on the stand. Because, as he said to me, he doesn’t necessarily believe that low test scores qualify somebody as a bad teacher. They might do other things well in the classroom that don’t show up on an exam, like teach social skills, or inspire their students to love reading or math. And while he has observed teachers he didn’t particularly like, or thought could use more training, he’s never encountered one he would consider “grossly ineffective.”

“In hundreds of classrooms, I have never seen a ‘grossly ineffective’ teacher,” he told me. “I don’t know anybody who knows what that means.”

I asked Stuart Biegel, a law professor and education expert at UCLA, whether he thought that the odd origins of the 1–3 percent figure might undermine Treu’s decision on appeal. Biegel, who represented the winning plaintiffs in one of the key cases Treu cited, said it might. But he thought that the decision’s “poor legal reasoning” and “shaky policy analysis” would be bigger problems. “If 97 to 99 percent of California teachers are effective, you don’t take away basic, hard-won rights from everybody. You focus on strengthening the process for addressing the teachers who are not effective, through strong professional development programs, and, if necessary, a procedure that makes it easier to let go of ineffective teachers,” he wrote to me in an email.

To me, the tale of this particular statistic is a good example of why the education reform battle doesn’t really belong in the courts, at least not yet. We haven’t come to an accord on what makes a “good” or “bad” teacher, or how many of either are out there—even if one judge is convinced we have.

Jordan Weissmann is Slate‘s senior business and economics correspondent.