Category: research

Challenging development dogma

On his blog Shanta Devarajan, the World Bank Chief Economist for Africa, has a post discussing the debate about the performance and results of the Millennium Villages Project (MVP).  The debate, which takes shape principally in papers by Matt Clemens and Gabriel Demombynes of Center for Global Development and Paul Pronyk, John McArthur, Prabhjot Singh, and Jeffrey Sachs of the Millennium Villages Project, questions how the MVP is capturing the impacts of its interventions in the Millennium Villages.  As Devarajan notes, the paper by Clemens and Demombynes rightly notes that the MVP’s claims about its performance are not really that clearly framed in evidence, which makes it hard to tell how much of the changes in the villages can be attributed to their work, and how much is change driven by other factors.  Clemens and Demombynes are NOT arguing that the MVP has had no impact, but that there are ways to rigorously evaluate that impact – and when impact is rigorously evaluated, it turns out that the impact of MVP interventions is not quite as large as the project would like to claim.
This is not all that shocking, really – it happens all the time, and it is NOT evidence of malfeasance on the part of the MVP.  It just has to do with a simple debate about how to rigorously capture results of development projects.  But this simple debate will, I think, have long-term ramifications for the MVP.  As Devarajan points out:

In short, Clemens and Demombynes have undertaken the first evaluation of the MVP.  They have shown that the MVP has delivered sizeable improvements on some important development indicators in many of the villages, albeit with effects that are smaller than those described in the Harvests of Development paper.  Of course, neither study answers the question of whether these gains are sustainable, or whether they could have been obtained at lower cost.  These should be the subject of the next evaluation.

I do not, however, think that this debate is quite as minor as Devarajan makes it sound – and he is clearly trying to downplay the conflict here.  Put simply, the last last two sentences in the quote above are, I think, what has the MVP concerned – because the real question about MVP impacts is not in the here and now, but in the future.  While I have been highly critical of the MVP in the past, I am not at all surprised to hear that their interventions have had some measurable impact on life in these villages.  The project arrived in these villages with piles of money, equipment and technical expertise, and went to work.  Hell, they could have simply dumped the money (the MVP is estimated to cost about $150 per person per year) into the villages and you would have seen significant movement in many target areas of the MVP.  I don’t think that anyone doubts that the project has had a measurable impact on life in all of the Millennium Villages.
Instead, the whole point here is to figure out if what has been done is sustainable – that is the measure of performance here.  Anyone can move the needle in a community temporarily – hell, the history of aid (and development) is littered with such projects.  The hard part is moving the needle in a permanent way, or doing so in a manner that creates the processes by which lasting change can occur.  As I have argued elsewhere (and much earlier that in this debate), and as appears to be playing out on the ground now, the MVP was never conceptually framed in a way that would bring about such lasting changes.  Clemens and Demombynes’ work is important because it provides an external critique of the MVP’s claims about its own performance – and it is terrifying to at least some in the MVP, as external evaluations are going to empirically demonstrate that the MVP is not, and never was, a sustainable model for rural development.
While I would not suggest that Clemens and Demombynes’ approach to evaluation is perfect (indeed, they make no such claim), I think it is important because it is trying to move past assumptions to evidence.  This is a central call of my book – the MVP is exhibit A of a project founded on deeply problematic assumptions about how development and globalization work, and framed and implemented in a manner where data collection and evaluation cannot really question those assumptions . . . thus missing what is actually happening (or not happening) on the ground.  This might also explain the somewhat non-responsive response to Clemens and Demombynes in the Pronyk et al article – the MVP team is having difficulty dealing with suggestions that their assumptions about how things work are not supported by evidence from their own project, and instead of addressing those assumptions, are trying to undermine the critique at all costs.  This is not a productive way forward, this is dogma.  Development is many things, but if it is to be successful by any definition, it cannot be dogmatic.

Posted on by Ed 1

For everyone who doesn't understand social research . . .

OK, two posts for today, because I can’t help myself. Yeah, I am a social scientist. Which means that people either think I run control experiments on various populations (an idea that freaks me out)*, or they think that I have no method to my research at all – I just sort of run around, talk to a few people until I get bored or run out of money, and then come back and write it up.
Of course, both views are crap.  Good social science is founded on rigorous fieldwork and data whose validity can be verified.  How one collects that data, and verifies that validity, varies – it depends on what you are studying.  For whatever reason, though, people have a hard time understanding this.  Quick story: a former chair of my department, during a debate about field methods, actually once asked me if it was really possible to teach someone to do interviews and participant observation.  My response: “I didn’t pop out of the womb able to do this, you know.”  End of discussion, thankfully.
But now I have found someone who has written this up nicely – Wronging Rights (absolutely hilarious, and totally awful, all at the same time – just go read for a bit and then feel bad about yourself for laughing.  Everyone does) has a great post on the subject that links to a series of even better posts at Texas in Africa that covers it (see the Wronging Rights post link to connect to the relevant Texas in Africa posts).
Social scientists, get to reading.  Journalists, read this and understand why you are not social scientists.  Especially you, Thomas Friedman.  And the rest of you . . . never, ever ask me if you can teach someone to do social science . . .
*controlled experiment: what, am I supposed to pick two identical villages (no such thing), and then start to work with one village while studiously ignoring the other village no matter what happens to that community (i.e. drought, food insecurity, disease, what have you) because I need to preserve the integrity of my control group?  There are other ways to establish the validity of one’s results . . .

Posted on by Ed 0

Page proofs . . .

are killing me.  But, the book is here, and I am cleaning it up.  I hate page proofs.  Deeply.  This is the sort of detail work I loathe – combing back through 90,000 words looking for misspellings and erroneous punctuation.  It is taking days, because you can only focus that hard for so long.  And at the same time, I am cleaning up the index.
Oh, and that is on top of the article that was due back in today – I worked with two of my Ph.D. students, Mary Thompson and Manali Baruah, to produce a paper that examines how REDD+ functions as a form of unacknowledged environmental governance (defining legitimate terms and actors within debates over how to implement terrestrial carbon sequestration projects in forest areas).  We’ll see how it does in this round of peer review.
And then there is the talk I am supposed to be giving at UNC – Chapel Hill on Friday.  I’ll be discussing how we think about livelihoods in development, how current framings might have carried us as far as they are going to, and what a new framing might look like.  Yeah, it is coming together, but not as quickly as I’d hoped.
But, without further ado, the first few hundred words of Delivering Development:

Posted on by Ed 0

Explaining myself

So, today I was challenged by an old friend, and a very well-known senior scholar in my field, about working for USAID.  He did so on two of the largest listservs in my field – admittedly, because I had just posted an offhand follow-up to some AID job postings to the list inviting people to apply.  Ben is great guy, and one of the founders of what might be thought of as hazards research – he’s also got his own political positions (which are evident below).  I like him a lot – he pushes me all the time, which I find very, very productive (and that is his intent).  I think his challenge, and my reply, help articulate why more people ought to be straddling the academic and practice worlds in development.
First, Ben:

Dear Ed,

I am sure all of us involved in Africa specialty group as well as the CAPE discussion list would benefit by hearing more detail about why you feel that the land tenure team at USAID has “an outstanding reputation” and why you believe “USAID is dead serious about its goal of becoming an intellectual leader in development…”.  Furthermore, if you are correct about the agency’s dead seriousness, what are the constraints and obstacles that have to be overcome?

From my point of view, until USAID is removed from its current position within the Department of State and made an independent agency like DFID in the UK or GTZ in Germany, everything done in the development field by anyone, alas, even you dear comrade, falls under the shadow of US geopolitical special interest.  There is also a case one could make that, in particular, all research on issues of resource access, land tenure falling into this category, needs to be free of ALL national and international development assistance agencies because of their usual commitment to what UNDP calls “alignment with host country interests.”  So, for example, to follow up on the World Bank’s recent report on land grabbing, it is doubtful if any development assistance partner (USAID, DFID, GTZ, UNDP, FAO, etc.) would criticize the corrupt practices in many countries leading to land grabbing.

Those seem to me to be macro and meso challenges to your optimism and jolly invitation to join you.  Finally, at the micro scale, it would seem, a fortiori, that those of us who work in the mode of participatory action research, something as you well know from your excellent past work demands a great deal of trust, can ask our friends and informants in various parts of rural Africa to put aside generations of mistrust of the great powers that ravaged their continent with surrogate conflicts during the Cold War and which continue to prop up corrupt regimes with development assistance.

Your scholarly credentials and intelligence are so obvious to those who know you, I am sure you must have good reasons for your sojourn at USAID and for your widely disseminated invitation to others to join you there.  Please share them with us.

All the best,

BEN

Dr. Ben Wisner

Aon Benfield UCL Hazard Research Centre, University College London, UK

Environmental Studies Program, Oberlin College, Oberlin, Ohio, USA

And my reply:

Hi Ben (and all):

One of the things I love most about Ben is his ability to pin me down – whether arguing about the modeling community or agreeing about the tragedy that was the Spanish anarchists in Catalunya during the Spanish Civil War.  There’s no such thing as an offhand invitation!  So I am happy to elaborate, at least as much as I can in a generally-circulated email – and please note, I am speaking for myself here.  No official agency messages coming from my mouth . . .

First, I am at USAID out of a serious desire to bridge the absurd and growing gulf between the academic and practitioner communities in development – we all know that the practitioner community is not reading the academic lit (and indeed they are not, though the reasons for this are complex, and include the fact that the agencies do not have subscriptions to the journals because they have difficulty justifying the expense [yes, this is absurd]), but the academic community does not spend a heck of a lot of time reading the practitioner stuff either – except mostly to throw (intellectual) stones without actually understanding the institutional context of the various documents they are critiquing.  Let’s be honest, the number of development geographers out there that have actually worked in a development agency (not just consulting, but actually in the organization) is tiny, which means that most of us (including me, at least until about 6 weeks ago) are critiquing something we understand very poorly, at best.  The result: two parallel literatures, and very little productive interplay.  So I am learning about how to translate between these communities to facilitate greater communication and cooperation.  It seems there is tremendous mistrust on both sides of this divide, for good reason and for not so good reasons.  I suppose I am trying to parse through those reasons as well.

That said, you certainly can call the “authenticity” of my experience into question.  I occupy a unique space here at USAID.  I am a fellow, which gives me freedom to move around beyond my obvious job description and to challenge things that I see as problematic.  Further, I am on leave from South Carolina – I did not surrender my position or my tenure.  So, I have a lot of security – I don’t worry about speaking up in a meeting (or responding to an email) in a manner that might have repercussions for my career – the worst that can happen to me is to be sacked before my fellowship is up and sent back to my tenured position.  So I cannot say that I fully understand the pressures that some of my colleagues must feel on a day-to-day basis.  Then again, I know my positionality and, trained as a poststructuralist, I’ve long thought that authenticity was sort of crap, anyway . . .

At another level, I fear that I (and perhaps many on this list share this feeling) was at risk of becoming the new extractive industry.  Speaking for myself, I found that I was going to various places in the world, doing serious fieldwork, writing it up and trying to push the literature forward . . . only to watch that work gain no traction at all in the policy and practice world. The same mistakes just kept happening.  So, all that my research really did was get me promotions and pay raises.  Going to a place in the Global South, gathering a resource (in this case knowledge and information), and then redistributing that resource in the Global North to my financial benefit?  Sounds like extraction/expropriation to me . . .  I found that untenable, and I am actively looking for ways to make my research “do something”.  Yes, this is fraught and intellectually dangerous territory.  But I found the alternative unacceptable.

Second, your critique of USAID’s position vis a vis US foreign policy is to the point – we are absolutely constrained by State’s vision for the world, and this does limit us somewhat.  That said, there is a lot of critical awareness of these limitations at USAID (much, much more than I’d expected), and significant efforts to push back and shift the views that are seen as problematic.  For example, there is excellent work on environment-conflict connections coming out of my bureau that aggressively challenges the absurd “water wars” mentality that seems to drive some corners of our foreign policy, referencing really good academic work on the nuanced, difficult connections between environmental change/resources and conflict.  Hell, they have Homer-Dixon thoroughly beaten.

At the same time, I don’t want to fall into the position of arguing from one end of a continuum (“thoroughly compromised”?), with academic research implicitly at the “free and untainted” other end.  There are a hell of a lot of unacknowledged politics in academic research (though I know from our conversations you are quite aware) – for example, NSF sets priorities all the time which are shaped by Congressional funding and partnerships with various agencies in the executive branch (CNH sound familiar, there CAPE-ers?), and we all run off to apply for these funds as if they were apolitical – a terribly naive position.  Put another way, one could argue that it is much easier to be critically aware of one’s position, role and influences when they are clearly articulated in a memo.

To answer fully, and with illustrations, your concerns for issues like land grabbing and “alignment with host country interests” is impossible in a public forum.  First, I don’t speak for the Agency.  Second, examples would invoke countries, and that is a bad idea when you can be seen (incorrectly, in my case) as speaking for US foreign policy.  Third, there are really good people here in the Agency who are actively working to address the very things you are worried about in a lot of different ways . . . many quite subtle.  It is not fair for me to place them in the spotlight without their consent.  You will have to trust me on this – which is hardly evidence in and of itself, but you do know me well.

The desire to become an intellectual leader in development seems sincere.  Sure, broad public statements may or may not have much meaning.  However, I have been struck by the pride folks have in the mission of this organization – and the rage they feel over the ways in which the Agency was downsized and stripped of many of its best thinkers over its recent history.  This is not merely a front office “feel good” thing – I see this as a feeling that permeates the agency, from the administrator down to the line offices and the field missions.  To that end, they are staffing up – and they seem sincere about bringing highly qualified people in to develop cutting-edge programming.  How this will play out if those highly qualified people start pushing back against existing programs and policy, I have no idea.  But this agency is not a monolith, and a lot of people I interact with are very open to criticism and respond very quickly and positively to it – again, to an extent I have found surprising.

And it is from this that I issued the invitation – either to apply for these jobs, or to consider taking an AAAS Science and Technology Policy Fellowship, or for you more senior types (am I more senior?  Nah) a Jefferson Science Fellowship, and serve a sabbatical or leave year at AID.  They want good people.  Most of the folks I work with want to be challenged constructively.  And if those of us who have the training, experience and critical faculties don’t apply for these opportunities, we cede the field to a bunch of people with MAs in Political Science who for their research likely ran massive regressions on the relationship between conflict and natural resources without bothering to contextualize either the type of conflict or the natural resource in question (yes, I have actually seen this very project proposal – structured because they could not get a large enough N to regress if they parsed by natural resource.  Mercifully, the researcher in question is not here at AID).

Finally, your point on addressing generations of mistrust is an excellent one, and one that I have no good answer for.  USAID is particularly challenged in this regard because it implements so little of its own programming – basically, most of the agency’s programs are contracted out, and AID staff are generally limited to monitoring the contractors and their products.  This creates a major problem for the agency – I think there is a real gap between what people in the agency know about what is really happening in the world, even at the level of the field missions, and actual events in the world.  This was a central point of my book <<plug alert>> (Delivering Development – forthcoming from Palgrave MacMillan in February, available for preorder at all major booksellers now!) and it seems to be borne out by my experiences thus far.  But given budgetary constraints, likely to be tightened starting roughly a week from today, USAID will never be allowed to staff up to levels necessary to implement its own programs, and therefore get that handle on what really happens in the world.  I will be interacting with country missions quite a bit over the next several months, and I suspect I might pick up some insights along the way . . . or at least I hope so.

I’ve spent way too much time on this response, and anyone still reading at this point probably wants the last 10 minutes of their lives back.  Ben, I am genuinely thankful for you and the challenges you pose – you make me a better thinker and person.  And one with less sleep, dammit.

Best,

Ed

Posted on by Ed 1

Are we really that bad?

So, the Center for Global Development, a non-partisan think tank focused on reducing poverty and making globalization work for the poor (a paraphrase of their mission statement, which can be found here), has issued a report that more or less says that USAID’s quality and effectiveness of aid is very low when compared to other agencies.
Well, I’m not all that freaked out by this assessment, principally because it fails to ask important questions relevant to understanding development needs and development outcomes.  In fact, the entire report is rigged – not intentionally, mind you, but I suspect out of a basic ignorance of the difference between the agencies being evaluated, and an odd (mis)understanding of what development is.
For me, the most telling point in the report came right away, on pages 3 and 4:

Given these difficulties in relating aid to development impact on the ground, the scholarly literature on aid effectiveness has failed to convince or impress those who might otherwise spend more because aid works (as in Sachs 2005) or less because aid doesn’t work often enough (Easterly 2003).

Why did this set me off?  Well, in my book I argue that the “poles” of Sachs and Easterly in the development literature are not poles at all – they operate from the same assumptions about how development and globalization work, and I just spent 90,000 words worth of a book laying out those assumptions and why they are often wrong.  In short, this whole report is operating from within the development echo chamber from which this blog takes its name.  But then they really set me off:

In donor countries especially, faced with daunting fiscal and debt problems, there is new and healthy emphasis on value for money and on maximizing the impact of their aid spending.

Folks, yesterday I posted about how the desire to get “value for our money” in development was putting all the wrong pressures on agencies . . . not because value is bad, but because it puts huge pressures on the development agencies to avoid risk (and associated costs), which in turn chokes off innovation in their programs and policies.  And here we have a report, evaluating the quality of aid (their words) in terms of its cost-effectiveness.  One of their four pillar analyses is the ability of agencies to maximize aid efficiency.  This is nuts.
Again, its not that there should be no oversight of the funds or their uses, or that there should be no accountability for those uses.  But to demand efficiency is to largely rule out high risk efforts which could have huge returns but carry a significant risk of failure.  Put another way, if this metric was applied to the Chilean mine rescue, then it would score low for efficiency because they tried three methods at once and two failed.  Of course, that overlooks the fact that they GOT THE MINERS OUT ALIVE.  Same thing for development – give me an “inefficient” agency that can make transformative leaps forward in our understandings of how development works and how to improve the situation of the global poor over the “efficient” agency that never programs anything of risk, and never makes those big leaps.
Now, let’s look at the indicators – because they tell the same story.  One of the indicators under efficiency is “Share of allocation to well-governed countries.”  Think about the pressure that places on an agency that has to think about where to set up its programming.  What about all of the poor, suffering people in poorly-governed countries?  Is USAID not supposed to send massive relief to Haiti after an earthquake because its government is not all we might hope?  This indicator either misses the whole point of development as a holistic, collaborative process of social transformation, or it is a thinly-veiled excuse to start triaging countries now.
They should know better – Andrew Natsios is one of their fellows, and he has explained how these sorts of evaluation pressures choke an agency to death.  Amusingly, they cite this work in here . . . almost completely at random on page 31, for a point that has no real bearing on that section of the text.  I wonder what he thinks of this report . . .
In the end, USAID comes out 126th of 130 agencies evaluated for “maximizing efficiency.”  Thank heavens.  It probably means that we still have some space to experiment and fail left.  Note that of the top 20% of donors, the highest scores went to the World Bank and UN Agencies, arguably the groups that do the least direct programming on the ground – in other words, the “inefficiencies” of their work are captured elsewhere, when the policies and programs they set up for others to run begin to come apart.  The same could be said of the Millennium Challenge Corporation here in the US, which also scored high.  In other words, they are rewarding the agencies that don’t actually do all that much on the ground for their efficiency, while the agencies that actually have to deal with the uncertainties of real life get dinged for it.
And the Germans ended up ranking high, but hey, nothing goes together like Germans and efficiency.  That one’s for you, Daniel Esser.
What a mess of a report . . . and what a mess this will cause in the press, in Congress, etc.  For no good reason.

Posted on by Ed 0

Required reading . . .

I’ve worked in the field of development studies for more than a decade now, mostly from the academic side.  In academia, we are very good at looking at the nuances of language and practice to try and detect why people do the things that they do.  As a result, in development studies we spend a lot of time thinking about discourses of development – the ways that we think about, speak about and act in the world – and how those shape the ways in which we “do development”.  Mostly, academics do this to explain why it is that development agencies and practitioners keep doing the same things over and over, hoping for a different result (which, you might remember, is how Einstein defined insanity).  There are some wonderful studies based in this approach that everyone should be reading, including Ferguson’s The Anti-Politics Machine, Scott’s Seeing Like A State, and Mitchell’s Rule of Experts (links in the sidebar to the right).  All help us to better understand why development doesn’t seem to work as well as we hope.  I suppose my forthcoming book (link also to the right) falls into this category as well, though I do not wade explicitly into social theory there (if you know the theory, you will see it in there – if you don’t, no worries, the argument is still perfectly intelligible).
What we academic types are not so good at is understanding the reality of life and work in a development organization.  Many of us have never worked in one, or did so a long time ago as a relatively low-ranking person.  However, when you rise in rank in an agency, you start to see the various organizational and political impediments to good work . . . and these impediments are at least as important for explaining development’s many failures as the (often-flawed) discursive framings of the world these agencies employ to understand the world.
With that in mind, I now strongly recommend you read The Clash of the Counter-bureaucracy and Development by former USAID Administrator Andrew Natsios.  Now, I don’t agree with a lot of the things that Natsios says about development in general – indeed, I think some of his logic with regard to economic growth as the core of development is very flawed – but I cannot argue at all with his gloves-off reading of how accountability measures, like monitoring and evaluation, are slowly choking USAID to death.  And it is gloves off – the man names names.  I was not AID under his leadership, but my colleagues all agree that he was a great administrator to work for, even if they did not agree with him all the time.  The man knows development . . . which is more than I can say about some previous administrators here.
By the way, even if you don’t work in development, you should read this – it is a wider lesson about how the best intentions related to accountability can go all wrong.  Those of you working for larger organizations will likely recognize parts of this storyline from where you sit.  And it is a pretty entertaining read, if for no other reason then to watch Natsios just lay it out there on a few people.  Must be nice to be retired . . .

Posted on by Ed 4

We come to our senses. Canada goes nuts.

via Climate Science Watch

“Documents obtained through freedom of information law show how Canada’s Harper government is controlling federal scientists’ ability to communicate with journalists on scientific issues. The requirement for ministerial-level pre-approval for media contacts applies broadly, not only to politically contentious issues like climate change and oil sands. “It’s Orwellian,” says Andrew Weaver, a climatologist at University of Victoria, quoted in the Montreal Gazette. The public has a right to know what federal scientists are discovering and learning. On this issue, the Ottawa Citizen suggests the Harper government is engaged in “a creeping and worrisome authoritarianism.”

Oh lord, we unmuzzle our federal scientists here in the US, and the Canadians (the Canadians!?!) go the other way.  Well, if nothing else this demonstrates that science is always political – though not always this obviously.  People forget/don’t realize that governments determine how pots of research money are to be spent, which tends to shape what research gets done.  This decision is political.  So the administration in charge of those research dollars can have a huge impact on science and knowledge . . . without resorting to such crude measures as muzzling their scientists.
I think Rick’s response is on point: this is research being paid for with taxpayer dollars – therefore the taxpayer has a right to hear what the scientists found out with those dollars, even if the government in charge thinks those findings are inconvenient politically.  Any time a government starts to muzzle science that is paid for with taxpayer dollars, the citizenry needs to push back . . .

Posted on by Ed 0