Project Description

Debate Intensifies Over Value-Added Research

Debate Intensifies Over Value-Added Research
Reviewer responds to research team’s claim of a connection 
between teacher value-added scores and student lifetime earnings

Originally Published at National Education Policy Center. June 24, 2014
Contact: 
William J. Mathis, (802) 383-0058, wmathis@sover.net
 Moshe Adler, (917) 453-4921, ma820@columbia.edu

URL for this press release: http://tinyurl.com/n9wnvry

BOULDER, CO (June 24, 2014) – Although policymakers may grab onto easy answers, questions about teacher effectiveness—how we measure it and what we can conclude about a teacher’s long-term impact—are being heatedly debated among scholars.

Today, the National Education Policy Center published a clear and detailed response to some of the most influential research claims about teacher effectiveness.

Those claims were made by researchers Raj Chetty, John Friedman, and Jonah Rockoff, who assert a connection between teachers’ “value-added” scores and what their students will earn over their lifetimes. Those assertions made their way into President Obama’s State of the Union message two years ago; they also surfaced in this month’s ruling by a California judge, in the Vergara case, who found that certain due process protections for teachers violated that state’s constitution.

That research, however, cannot bear the weight of critical scrutiny, according to an expert review by Moshe Adler published by the NEPC in April.

Adler’s review – published by NEPC as part of its Think Twice think tank review project – examined two working papers presenting research by Chetty and his colleagues and published by the National Bureau of Economic Research. Adler is an economist affiliated with both Columbia’s Urban Planning Department as well as the Harry Van Arsdale Jr. Center for Labor Studies at Empire State College, SUNY, and the author the 2010 book, Economics for the Rest of Us: Debunking the Science That Makes Life Dismal.

Chetty, Friedman, and Rockoff wrote a response—which is posted on the NEPC website—taking issue with the criticisms. Accordingly, we asked Adler to continue this important debate, replying to those responses. Adler has now presented an item-by-item explanation of why those responses are inadequate to address the study’s weaknesses.

The influence of the Chetty team’s research includes this statement in President Obama’ 2012 State of the Union address: “We know a good teacher can increase the lifetime income of a classroom by over $250,000.” Similarly, the judge in California’s Vergara litigation cited Chetty’s testimony and the team’s research as evidence that “a single year in a classroom with a grossly ineffective teacher costs students $1.4 million in lifetime earnings per classroom.”

The research claim rests on complex statistical analyses that attempt to attribute a student’s test score growth over a year to one or more of that student’s teachers—and then attempts to link those scores to subsequent student earnings.

According to Adler, however, the judge’s and the president’s conclusions step far beyond what the research can validly demonstrate. Adler explains why the evidence that Chetty and his colleagues cite cannot adequately support either claim: that a teacher’s value-added scores reflect that teacher’s quality, or that the scores predict future student earnings.

“Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy,” Professor Adler concluded in his April 10 review. That review raised at least nine different concerns, including the improper use of prior research and the failure to report important results when those results contradicted the authors’ conclusions.

The points Adler makes should give pause to policymakers who have assumed that the evidence is sufficiently solid to be relied upon to frame education reform strategies.

Although the concerns Adler discusses are sometimes technical, they are very clearly explained. Policymakers and researchers considering using the study from Chetty and his colleagues are strongly encouraged to read this robust exchange, which brings forth and clarifies serious and important issues.

Find Moshe Adler’s review; the Chetty, Friedman, and Rockoff reply; and Adler’s response to their reply, on the NEPC website at:
http://nepc.colorado.edu/thinktank/review-measuring-impact-of-teachers.

Find Measuring the Impacts of Teachers, by Raj Chetty, John N. Friedman, and Jonah E. Rockoff, on the web at:
http://www.nber.org/papers/w19423 and
http://www.nber.org/papers/w19424.

The Think Twice think tank review project (http://thinktankreview.org) of the National Education Policy Center (NEPC) provides the public, policymakers, and the press with timely, academically sound reviews of selected publications. NEPC is housed at the University of Colorado Boulder School of Education. The Think Twice think tank review project is made possible in part by support provided by the Great Lakes Center for Education Research and Practice.

The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information on the NEPC, please visit http://nepc.colorado.edu/.

The initial review is also found on the GLC website at http://www.greatlakescenter.org/.azs

Abstract; Review of Measuring the Impacts of Teachers. By Moshe Adler

Originally Published at National Education Policy Center. April 10, 2014 | Press Release | Media Citations
Can the quality of teachers be measured the way that a person’s weight or height is measured? Some economists have tried, but the “value-added” they have attempted to measure has proven elusive. The results have not been consistent over tests or over time. Nevertheless, a two-part report by Raj Chetty and his colleagues claims that higher value-added scores for teachers lead to greater economic success for their students later in life. This review of the methods of Chetty et al. focuses on their most important result: that teacher value-added affects income in adulthood. Five key problems with the research emerge. First, their own results show that the calculation of teacher value-added is unreliable. Second, their own research also generated a result that contradicts their main claim—but the report pushed that inconvenient result aside. Third, the trumpeted result is based on an erroneous calculation. Fourth, the report incorrectly assumes that the (miscalculated) result holds across students’ lifetimes despite the authors’ own research indicating otherwise. Fifth, the report cites studies as support for the authors’ methodology, even though they don’t provide that support. Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy.

A response from the authors is below. Additional information on the study can be found at:
http://obs.rc.fas.harvard.edu/chetty/value_added.html

A further response from Moshe Adler follows below.

Adler Response by National Education Policy Center