On his blog Shanta Devarajan, the World Bank Chief Economist for Africa, has a post discussing the debate about the performance and results of the Millennium Villages Project (MVP).  The debate, which takes shape principally in papers by Matt Clemens and Gabriel Demombynes of Center for Global Development and Paul Pronyk, John McArthur, Prabhjot Singh, and Jeffrey Sachs of the Millennium Villages Project, questions how the MVP is capturing the impacts of its interventions in the Millennium Villages.  As Devarajan notes, the paper by Clemens and Demombynes rightly notes that the MVP’s claims about its performance are not really that clearly framed in evidence, which makes it hard to tell how much of the changes in the villages can be attributed to their work, and how much is change driven by other factors.  Clemens and Demombynes are NOT arguing that the MVP has had no impact, but that there are ways to rigorously evaluate that impact – and when impact is rigorously evaluated, it turns out that the impact of MVP interventions is not quite as large as the project would like to claim.

This is not all that shocking, really – it happens all the time, and it is NOT evidence of malfeasance on the part of the MVP.  It just has to do with a simple debate about how to rigorously capture results of development projects.  But this simple debate will, I think, have long-term ramifications for the MVP.  As Devarajan points out:

In short, Clemens and Demombynes have undertaken the first evaluation of the MVP.  They have shown that the MVP has delivered sizeable improvements on some important development indicators in many of the villages, albeit with effects that are smaller than those described in the Harvests of Development paper.  Of course, neither study answers the question of whether these gains are sustainable, or whether they could have been obtained at lower cost.  These should be the subject of the next evaluation.

I do not, however, think that this debate is quite as minor as Devarajan makes it sound – and he is clearly trying to downplay the conflict here.  Put simply, the last last two sentences in the quote above are, I think, what has the MVP concerned – because the real question about MVP impacts is not in the here and now, but in the future.  While I have been highly critical of the MVP in the past, I am not at all surprised to hear that their interventions have had some measurable impact on life in these villages.  The project arrived in these villages with piles of money, equipment and technical expertise, and went to work.  Hell, they could have simply dumped the money (the MVP is estimated to cost about $150 per person per year) into the villages and you would have seen significant movement in many target areas of the MVP.  I don’t think that anyone doubts that the project has had a measurable impact on life in all of the Millennium Villages.

Instead, the whole point here is to figure out if what has been done is sustainable – that is the measure of performance here.  Anyone can move the needle in a community temporarily – hell, the history of aid (and development) is littered with such projects.  The hard part is moving the needle in a permanent way, or doing so in a manner that creates the processes by which lasting change can occur.  As I have argued elsewhere (and much earlier that in this debate), and as appears to be playing out on the ground now, the MVP was never conceptually framed in a way that would bring about such lasting changes.  Clemens and Demombynes’ work is important because it provides an external critique of the MVP’s claims about its own performance – and it is terrifying to at least some in the MVP, as external evaluations are going to empirically demonstrate that the MVP is not, and never was, a sustainable model for rural development.

While I would not suggest that Clemens and Demombynes’ approach to evaluation is perfect (indeed, they make no such claim), I think it is important because it is trying to move past assumptions to evidence.  This is a central call of my book – the MVP is exhibit A of a project founded on deeply problematic assumptions about how development and globalization work, and framed and implemented in a manner where data collection and evaluation cannot really question those assumptions . . . thus missing what is actually happening (or not happening) on the ground.  This might also explain the somewhat non-responsive response to Clemens and Demombynes in the Pronyk et al article – the MVP team is having difficulty dealing with suggestions that their assumptions about how things work are not supported by evidence from their own project, and instead of addressing those assumptions, are trying to undermine the critique at all costs.  This is not a productive way forward, this is dogma.  Development is many things, but if it is to be successful by any definition, it cannot be dogmatic.