Bill Gates, in his annual letter, makes a compelling argument for the need to better measure the effectiveness of aid. There is a nice, 1 minute summary video here. This is becoming a louder and louder message in development and aid, having been pushed now by folks ranging from Raj Shah, the Administrator of USAID, to most everyone at the Center for Global Development. There are interesting debates going on about how to shift from a focus on outputs (we bought this much stuff for this many dollars) to a focus on impacts (the stuff we bought did the following good things in the world). Most of these discussions are technical, focused on indicators and methods. What is not discussed is the massively failure-averse institutional culture of development donors, and how this culture is driving most of these debates. As a result, I think that Gates squanders his bully pulpit by arguing that we should be working harder on evaluation. We all know that better evaluation would improve aid and development. Suggesting that this is even a serious debate in development requires a nearly-nonexistent straw man that somehow thinks learning from our programs and projects is bad.
Like most everyone else in the field, I agree with the premise that better measurement (thought very broadly, to include methods and data across the quantitative to qualitative spectrum) can create a learning environment from which we might make better decisions about aid and development. But none of this matters if all of the institutional pressures run against hearing bad news. Right now, donors simply cannot tolerate bad news, even in the name of learning. Certainly, there are lots of people within the donor agencies that are working hard on finding ways to better evaluate and learn from existing and past programs, but these folks are going to be limited in their impact as long as agencies such as USAID answer to legislators that seem ready to declare any misstep a waste of taxpayer money, and therefore a reason to cut the aid budget…so how can they talk about failure?
So, a modest proposal for Bill Gates. Bill (may I call you Bill?), please round up a bunch of venture capitalists. Not the nice socially-responsible ones (who could be dismissed as bleeding-heart lefties or something of the sort), the real red-in-tooth-and-claw types. Bring them over to DC, and parade out these enormously wealthy, successful (by economic standards, at least) people, and have them explain to Congress how they make their money. Have them explain how they got rich failing on eight investments out of ten, because the last two investments more than paid for the cost of the eight failures. Have them explain how failure is a key part of learning, of success, and how sometimes failure isn’t the fault of the investor or donor – sometimes it is just bad luck. Finally, see if anyone is interested in taking a back-of-the-envelope shot at calculating how much impact is lost due to risk-averse programming at USAID (or any other donor, really). You can shame Congress, who might feel comfortable beating up on bureaucrats, but not so much on economically successful businesspeople. You could start to bring about the culture change needed to make serious evaluation a reality. The problem is not that people don’t understand the need for serious evaluation – I honestly don’t know anyone making that argument. The problem is creating a space in which that can happen. This is what you should be doing with your annual letter, and with the clout that your foundation carries.
Failing that (or perhaps alongside that), lead by demonstration – create an environment in your foundation in which failure becomes a tag attached to anything from which we do not learn, instead of a tag attached to a project that does not meet preconceived targets or outcomes. Forget charter cities (no, really, forget them), become the “charter donor” that shows what can be done when this culture is instituted.
The evaluation agenda is getting stale, running aground on the rocky shores of institutional incentives. We need someone to pull it off the rocks. Now.
I am a big fan of the idea of admitting failure and trying to learn from it. I like ambitious projects with potentially huge payoffs, but a lot of risk of failure – they’re just much more interesting than going at things incrementally. Besides, if you are going to fail, why not fail spectacularly? As I tell my grad students, if you are going to ride it all the way to the ground, you might as well dig a big hole when you get there. At least people will notice the hole, and try to figure out what the hell you were up to . . . of course, I am an academic (with tenure), so I have a pretty big cushion to land on these days.
All that said, I wonder about the utility of these admitting failure efforts that I see coming from groups like Engineers without Borders. I had the good fortune to catch up with Tom Murphy (or, as the twitterati know him, @viewfromthecave) the other day while he was here in DC, and we started talking about learning from failure. In the course of our conversation, we came around to two key problems. First, really admitting failure requires reframing the public image of development as an inherently do-no-harm effort, where just doing something is better than nothing. Second, given this first problem, when we really start talking about what failure means, even in the most constructive of settings, we will call the entire development enterprise into question. How do we avoid throwing the baby out with the bathwater?
We have long allowed ourselves and our donor constituencies to believe that development work should never have bad outcomes – there is a pervasive belief (under challenge right now, at least by some) that, at worst, a failed project will not change anything – that is what development failure means. Of course, this is simply untrue – development efforts can make things much, much worse for people if they are poorly framed, designed, and implemented – a point I try to make in Delivering Development. This has a lot to do with the very imagery of a helpless and oppressed global poor the aid world relies upon to raise funds. When people see someone in a situation that difficult, they assume things could not get worse. There is no discussion of what is working in the lives of the poor, and therefore the public has little sense that there are fragile things in peoples’ lives and livelihoods that should be protected as we bring new programs and projects to ground. As a result, development takes on the image of a low-risk enterprise in which social protection and “do no harm” safeguards are superfluous, as the worst we could do is leave people as they were.
Up against that worldview, admitting failure seems just fine – “hey, we didn’t really move the needle with that project, but we’ll figure out what we did wrong and try again” sounds much better than “we are incredibly sorry for utterly devastating the physical basis of your livelihoods and forcing many of you to abandon your farms because we ignored your existing land management practices.” Unfortunately, admitting failure means a lot of the latter, and I am not at all convinced that anyone has the stomach to really wade into that.
This issue has to be combined with a concern for the scale of failure. It is all well and good to admit failure, even ugly failures, at the project level – stuff happens. A failed project can usually be traced to concrete causes that can then be addressed and remedied. But how can a bilateral aid agency, or even a multilateral agency, do the same for its programs? It is one thing for such huge organizations to talk about the failure of individual projects, and learn from them, but how can we talk about learning from entire programs that don’t live up to expectations without attracting serious challenges to the aid budget that end up wrecking even successful programs, or preventing the scale-up of things that we know work? Put another way, how can we create an environment where learning from our activities is truly possible, and balance that environment with the political reality of aid agencies and NGOs that answer to (different) constituencies that expect only good things to happen?
This framing of global poverty, and the persistent need to justify aid budgets, puts everyone involved with development on a terrible tightrope – at least for those of us interested in evidence-based programming and policy. Just saying that admitting failure is good does not begin to get us to a world in which we can see that as more than a slogan. We will have to unwind decades of public relations and fundraising practice, and back out of some very long-standing and pervasive views of global poverty, before we have any real hope of bringing real learning to the fore of development practice.
Or, we could just give everyone tenure . . .