Project Description
Debate Intensifies Over Value-Added Research
Reviewer responds to research team’s claim of a connection
between teacher value-added scores and student lifetime earnings
URL for this press release: http://tinyurl.com/n9wnvry
BOULDER, CO (June 24, 2014) – Although policymakers may grab onto easy answers, questions about teacher effectiveness—how we measure it and what we can conclude about a teacher’s long-term impact—are being heatedly debated among scholars.
Today, the National Education Policy Center published a clear and detailed response to some of the most influential research claims about teacher effectiveness.
Those claims were made by researchers Raj Chetty, John Friedman, and Jonah Rockoff, who assert a connection between teachers’ “value-added” scores and what their students will earn over their lifetimes. Those assertions made their way into President Obama’s State of the Union message two years ago; they also surfaced in this month’s ruling by a California judge, in the Vergara case, who found that certain due process protections for teachers violated that state’s constitution.
That research, however, cannot bear the weight of critical scrutiny, according to an expert review by Moshe Adler published by the NEPC in April.
Adler’s review – published by NEPC as part of its Think Twice think tank review project – examined two working papers presenting research by Chetty and his colleagues and published by the National Bureau of Economic Research. Adler is an economist affiliated with both Columbia’s Urban Planning Department as well as the Harry Van Arsdale Jr. Center for Labor Studies at Empire State College, SUNY, and the author the 2010 book, Economics for the Rest of Us: Debunking the Science That Makes Life Dismal.
Chetty, Friedman, and Rockoff wrote a response—which is posted on the NEPC website—taking issue with the criticisms. Accordingly, we asked Adler to continue this important debate, replying to those responses. Adler has now presented an item-by-item explanation of why those responses are inadequate to address the study’s weaknesses.
The influence of the Chetty team’s research includes this statement in President Obama’ 2012 State of the Union address: “We know a good teacher can increase the lifetime income of a classroom by over $250,000.” Similarly, the judge in California’s Vergara litigation cited Chetty’s testimony and the team’s research as evidence that “a single year in a classroom with a grossly ineffective teacher costs students $1.4 million in lifetime earnings per classroom.”
The research claim rests on complex statistical analyses that attempt to attribute a student’s test score growth over a year to one or more of that student’s teachers—and then attempts to link those scores to subsequent student earnings.
According to Adler, however, the judge’s and the president’s conclusions step far beyond what the research can validly demonstrate. Adler explains why the evidence that Chetty and his colleagues cite cannot adequately support either claim: that a teacher’s value-added scores reflect that teacher’s quality, or that the scores predict future student earnings.
“Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy,” Professor Adler concluded in his April 10 review. That review raised at least nine different concerns, including the improper use of prior research and the failure to report important results when those results contradicted the authors’ conclusions.
The points Adler makes should give pause to policymakers who have assumed that the evidence is sufficiently solid to be relied upon to frame education reform strategies.
Although the concerns Adler discusses are sometimes technical, they are very clearly explained. Policymakers and researchers considering using the study from Chetty and his colleagues are strongly encouraged to read this robust exchange, which brings forth and clarifies serious and important issues.
Find Moshe Adler’s review; the Chetty, Friedman, and Rockoff reply; and Adler’s response to their reply, on the NEPC website at: http://nepc.colorado.edu/thinktank/review-measuring-impact-of-teachers.
Find Measuring the Impacts of Teachers, by Raj Chetty, John N. Friedman, and Jonah E. Rockoff, on the web at: http://www.nber.org/papers/w19423 and http://www.nber.org/papers/w19424.
The Think Twice think tank review project (http://thinktankreview.org) of the National Education Policy Center (NEPC) provides the public, policymakers, and the press with timely, academically sound reviews of selected publications. NEPC is housed at the University of Colorado Boulder School of Education. The Think Twice think tank review project is made possible in part by support provided by the Great Lakes Center for Education Research and Practice.
The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information on the NEPC, please visit http://nepc.colorado.edu/.
The initial review is also found on the GLC website at http://www.greatlakescenter.org/.azs
Abstract; Review of Measuring the Impacts of Teachers. By Moshe Adler
A response from the authors is below. Additional information on the study can be found at: http://obs.rc.fas.harvard.edu/chetty/value_added.html
A further response from Moshe Adler follows below.
