Brookings has come out with a report suggesting a dramatic decrease in the number of people living in poverty (using the $1.25/day mark as a measure of poverty) since 2004.  The report suggests that where 1.3 billion people met this description in 2004, today less than 900 million are dealing with similar circumstances.  In short, we are on target to achieve the first Millennium Development Goal (MDG) of cutting the global rate of poverty to half of the 1990 rate – indeed, the report suggests that:

the MDG1a target has already been met—approximately three years ago. Furthermore, by 2015, we will not only have halved the global poverty rate, as per MDG1a, but will have halved it again. (p.4)

This is remarkable news.  Brookings notes that the rate of poverty reduction varies dramatically by region, with East and South Asia cutting rates by about 50% between 2005 and 2010, while sub-Saharan Africa’s rate fell just under 8% in that same period.  Further, just two countries can account for the majority of this drop: India and China.  So there are still big challenges out there to be addressed, but things are looking up.

Or are they?

A glance at the methodology employed by this study leads me to think that the error bars on this study are rather huge.  Indeed, the authors are fully aware of the data and analytic challenges related to any effort to estimate poverty levels.  As the authors note, in development

we find it remarkably difficult to measure whether it is happening, and if so how fast. This is especially the case when it comes to producing global poverty data, as the challenges of national poverty data collection are multiplied several times over and then further compounded by the tricky—and unsatisfactory—business of converting national results into internationally comparable terms.

In short, the authors know that the project on which they have embarked is likely to generate estimates with significant potential errors – “error bars” as it were, around their data points, in which reality might actually exist.  Oddly, the report makes no effort to present these error bars.  Instead, it makes rather bold claims about reductions in the level of poverty largely without caveat.  I am not convinced these claims are warranted.

First, there are major data issues here.  Their 2005-2010 measures are predicated on recent household survey data.  Here is the problem with household survey data in sub-Saharan Africa: a lot of it is junk.  I’ve tried to deal with such data in Ghana, a country that has a relatively robust infrastructure for this sort of work, and found their survey data to be a mess.  I suspect that in some regions (Latin America, parts of Asia) the data is actually quite good, on the whole.  But in a lot of places (most of SSA and Southeast Asia) the data is likely very problematic.  And even where the data infrastructure is pretty good, the survey methodologies are often found wanting.  I was part of a team that tried to get a handle on livelihoods near a forest reserve area in Southern Malawi – to do so, we sampled 300 households across four villages (75 households/village) quarterly for a year, to capture things like seasonality in our dataset.  2400 structured interviews had to be undertaken to do this, and those interviews were supplemented by semi-structured interviews with subsamples of the group to explore issues like household power and gender relations to give context to that larger dataset.  This was enormously labor-intensive . . . and necessary to really understand what was going on in those villages.  Most household surveys are not done in this manner, and thus are subject to seasonal bias, or the presentation of data as comparable across the country when, in fact, it has very locally-specific meanings rooted in local social context. I do not expect that all national household surveys will be as rigorous or labor-intensive as ours was, but one should acknowledge the limitations of the data.  No discussion of this in the paper, but that can put a pretty wide margin of error on your findings.

I won’t even wade into the issues with population data that they gloss over in this study – I spend a good bit of time in chapter 9 of Delivering Development talking about census issues and the problems of compounding data error in estimations of economic growth.  Let’s just say that there are significant uncertainties around census data that compound any other errors in the data – again, growing error bars.

Second, there is a moment in the analysis that I found stunning – their projections to 2015 predicated on a surprising assumption – that distribution of wealth will stay the same.  Well, given that economic growth is, by and large, predicated on unevenness within regions, countries and between countries, there is basically no chance that the distribution of wealth will remain the same in any place that is growing.  Generally speaking, the GINI coefficient goes up as growth goes up . . . and a lot of places they are talking about are meant to experience fairly robust rates of growth now and in the near future.  More error.

What does this mean?  Well, to me it means that the data they presented like this:

Really has a wide margin of error, even for past observed data (but compounded going forward) that should look be presented like this (with margins of error in red, and not to scale.  I did not calculate them, as this is just illustrative):

OK, so perhaps there should have been some error bars in there.  So what?  Well, this is a policy brief, with policy recommendations that might actually be followed by someone . . . and this brief is arguing that we are on top of the whole poverty reduction thing, which is sure to become an argument for looking for ways to trim development budgets.

Even if the budgetary ax does not fall because of this brief, there is a risk of reprioritization that may not yet be appropriate.  In the recommendation

aid donors must adapt to the evolving poverty landscape and update their policies and programming to reflect current needs and priorities

the brief implicitly argues that agencies should be reevaluating their programming based on the findings in the brief – toward a focus on Africa and fragile states, and away (apparently) from much of Asia and those parts of Latin America, the Caribbean, and the Pacific where we currently work.  However, this is a recommendation based on much thinner evidence than it seems.

The worst part is that I think this presentation of the data undermines one of their excellent policy points:

One final policy recommendation revealed by this analysis is the need to improve the quantity, quality and timeliness of poverty data, at both the national and the global level. For both developing country governments and aid agencies working to fight poverty, it is impossible to efficiently allocate resources toward this goal using poverty data that is incomplete, unreliable or out of date.

At the country level, there has already been a significant uptake in the use of household surveys and an improvement in their quality. Yet in remarkably few countries is there a standardized, recurrent—and therefore consistent—approach to household survey data collection and analysis.  A renewed, long-term commitment to build capacity in domestic statistical agencies would be a valuable use of aid agencies’ resources.

I agree completely, and have argued for this need, but by presenting the data as so clear and robust, they have essentially undermined this argument.  Any policy maker looking at this will wonder why s/he should give more funding to something that already works . . .

Folks, policy makers will never learn to deal with uncertainty until they are faced with it . . . if we keep copping out and “firming up” mushy results into single bold trendlines, they will expect certain outcomes from uncertain data indefinitely.