The Dumb Math Error Heard ‘Round the World

Rarely has any one study had such an immediate impact on global policy. Usually, folks wait until a study is replicated and put through robust follow-up before any one takes research to heart. Reinhart & Rogoff (2010) basically played into the narrative of the plutocracy and what the ‘very serious people’ around the world wanted. So, its significant findings were taken very seriously before all those secondary tests of robustness and such were undertaken. Well, now we find out the wunderkind study that justified a lot of unnecessary austerity has some serious math mistakes. I’m still reading through all the criticisms but, as Krugman puts it “Holy Coding Error, Batman”!  Let’s just call this some serious MATH FAIL!

The intellectual edifice of austerity economics rests largely on two academic papers that were seized on by policy makers, without ever having been properly vetted, because they said what the Very Serious People wanted to hear. One was Alesina/Ardagna on the macroeconomic effects of austerity, which immediately became exhibit A for those who wanted to believe in expansionary austerity. Unfortunately, even aside from the paper’s failure to distinguish between episodes in which monetary policy was available and those in which it wasn’t, it turned out that their approach to measuring austerity was all wrong; when the IMF used a measure that tracked actual policy, it turned out that contractionary policy was contractionary.

The other paper, which has had immense influence — largely because in the VSP world it is taken to have established a definitive result — was Reinhart/Rogoff on the negative effects of debt on growth. Very quickly, everyone “knew” that terrible things happen when debt passes 90 percent of GDP.

Some of us never bought it, arguing that the observed correlation between debt and growth probably reflected reverse causation. But even I never dreamed that a large part of the alleged result might reflect nothing more profound than bad arithmetic.

The best explanation of the problem that I’ve seen comes from Mike Konczal at RortyBomb.

In 2010, economists Carmen Reinhart and Kenneth Rogoff released a paper, “Growth in a Time of Debt.” Their “main result is that…median growth rates for countries with public debt over 90 percent of GDP are roughly one percent lower than otherwise; average (mean) growth rates are several percent lower.” Countries with debt-to-GDP ratios above 90 percent have a slightly negative average growth rate, in fact.

This has been one of the most cited stats in the public debate during the Great Recession. Paul Ryan’s Path to Prosperity budget states their study “found conclusive empirical evidence that [debt] exceeding 90 percent of the economy has a significant negative effect on economic growth.” The Washington Post editorial board takes it as an economic consensus view, stating that “debt-to-GDP could keep rising — and stick dangerously near the 90 percent mark that economists regard as a threat to sustainable economic growth.”

Is it conclusive? One response has been to argue that the causation is backwards, or that slower growth leads to higher debt-to-GDP ratios. Josh Bivens and John Irons made this case at the Economic Policy Institute. But this assumes that the data is correct. From the beginning there have been complaints that Reinhart and Rogoff weren’t releasing the data for their results (e.g. Dean Baker). I knew of several people trying to replicate the results who were bumping into walls left and right – it couldn’t be done.

In a new paper, “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff,” Thomas Herndon, Michael Ash, and Robert Pollin of the University of Massachusetts, Amherst successfully replicate the results. After trying to replicate the Reinhart-Rogoff results and failing, they reached out to Reinhart and Rogoff and they were willing to share their data spreadhseet. This allowed Herndon et al. to see how how Reinhart and Rogoff’s data was constructed.

They find that three main issues stand out. First, Reinhart and Rogoff selectively exclude years of high debt and average growth. Second, they use a debatable method to weight the countries. Third, there also appears to be a coding error that excludes high-debt and average-growth countries. All three bias in favor of their result, and without them you don’t get their controversial result.

Whenever you model a system, you have to make some assumptions going in.  These assumptions–coupled with the coding error–basically show the statistician’s slight of hand.  You can prove just about everything and anything with numbers if you manipulate the data enough. The UMAss-Amherst professor et al has really pulled the curtain away from the big green talking head this time.  Here’s the abstract.

Herndon, Ash and Pollin replicate Reinhart and Rogoff and find that coding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. They find that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not -0:1 percent as published in Reinhart and Rogoff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower.

The authors also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff’s claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.

Shorter Abstract:  They REALLY screwed the pooch.

So, there’s now this bigger problem out there which is the very serious people that are crashing economies based on a set of very biased assumptions and a very serious coding error.  To put that in a more politically correct way: How Much Unemployment Was Caused by Reinhart and Rogoff’s Arithmetic Mistake? Followed by, will the very serious people correct their very serious policy errors now?

This is a big deal because politicians around the world have used this finding from R&R to justify austerity measures that have slowed growth and raised unemployment. In the United States many politicians have pointed to R&R’s work as justification for deficit reduction even though the economy is far below full employment by any reasonable measure. In Europe, R&R’s work and its derivatives have been used to justify austerity policies that have pushed the unemployment rate over 10 percent for the euro zone as a whole and above 20 percent in Greece and Spain. In other words, this is a mistake that has had enormous consequences.

In fairness, there has been other research that makes similar claims, including more recent work by Reinhardt and Rogoff. But it was the initial R&R papers that created the framework for most of the subsequent policy debate. And HAP has shown that the key finding that debt slows growth was driven overwhelmingly by the exclusion of 4 years of data from New Zealand.

If facts mattered in economic policy debates, this should be the cause for a major reassessment of the deficit reduction policies being pursued in the United States and elsewhere. It should also cause reporters to be a bit slower to accept such sweeping claims at face value.

I spent most of Monday’s Morning Reads showing the current US economic data that shows that the deficit and debt of the US are melting like the wicked witch under that bucket of water.  I worry any more deficit reduction will throw our economy into recession and pave the way for Republican take over of the Senate.  However, the big green talking heads have been ignoring the data and just about everything else that a legion of economists have said citing this one–now clearly known to be flawed–study.  As Krugman mentioned in his blogs, their results was counter-intuitive and controversial among economists from day one of publication.  Policymakers through out Europe and the US gratuitously ignored all that because the questions did not fit their plan to push the mistakes of banks on a lot of hapless citizenry.

I have to give Matthew Ygleisas his due for this insight.  The damage is done, continues to be done, and has folks planning to do more and I believe they will just ignore this very serious error.

This is literally the most influential article cited in public and policy debates about the importance of debt stabilization, so naturally this is going to change everything.

Or, rather, it will change nothing. As I’ve said many times, citations of the Reinhart/Rogoff result in a policy context obviously appealing to a fallacious form of causal inference. There is an overwhelming theoretical argument that slow real growth will lead to a high debt:GDP ratio and thus whether or not you can construct a dataset showing a correlation between the two tells us absolutely nothing about whether high debt loads lead to small growth. The correct causal inference doesn’t rule out causation in the direction Reinhart and Rogoff believe in, but the kind of empirical study they’ve conducted couldn’t possibly establish it. To give an example from another domain, you might genuinely wonder if short kids are more likely to end up malnourished because they’re not good at fighting for food or something. A study where you conclude that short stature and malnourishment are correlated would give us zero information about this hypothesis, since everyone already knows that malnourishment leads to stunted growth. There might be causation in the other direction as well, but a correlation study woudn’t tell you.

The fact that Reinhart/Rogoff was widely cited despite its huge obvious theoretical problems leads me to confidently predict that the existence of equally huge, albeit more subtle, empirical problems won’t change anything either. As of 2007 there was a widespread belief among elites in the United States and Europe that reductions in retirement benefits were desirable, and subsequent events regarding economic crisis and debt have simply been subsumed into that longstanding view.

The very serious policymakers were looking for any justification for their austerity pogrom.  This is mainly because German taxpayers and pols don’t want to be on the hook for what German and US bankers did around the Eurozone. It is also because Republican law makers and their plutocratic overlords–like the Dr. Strangelove of Wall Street Pete Peterson–don’t want any funds floating around anywhere that could possibly find residency in their fee-churning ponzi schemes of investment funds.

It is not unusual, unfortunately, for some academics to neatly choose assumptions to drive results towards their hypothesis.  That is why peer review is extremely important.  Nearly every major study done using empirical data should be easily replicated.  It is usual for the authors to share their database and R&R obliged on this matter.  But, this emphasizes why major studies with major findings that don’t fit snugly with the current body of theory should undergo robust challenge.  Many economists had challenged the findings back in 2010 and the fact that some felt compelled to repeat their research indicated a healthy level of skepticism which is the hallmark of good research and researchers.

What is most disturbing about this is that agendas that drive the interests of a few start to reflect these theories-in-process.  R&R 2010 fit  the gross ambitions of people that were less concerned about truth than philosophy and ability to drive policy that basically is at odds with everything we’ve known about fiscal policy.  So, this takes us back to Matt’s question. There is incredible discussion on this in nearly all economics and finance blogs and circles.  Will these findings engender the same discussion and any course correction of the very serious people that used this very serious paper to do some very serious damage around the world.

I know it’s too much to hear those wonderful words ” We were wrong” on top of some course corrections.  But, hey it’s not too late for our President to give up the debt and deficit hysteria or is it?