Open The Echo Chamber

While behavioral economics continues to open old questions in development to new scrutiny, I am still having a lot of problems with the very unreflexive approach BE takes toward its own work (see earlier takes on this here and here).  Take, for example, Esther Duflo’s recent lectures discussing mistakes the poor make.  To discuss the mistakes the poor make, we must first understand what the goals of the poor are.  However, I simply don’t see the behavioral economists doing this.  There is still a lurking, underlying presumption that in making livelihoods decisions people are trying to maximize income and or the material quality of their lives.  This, however, is fundamentally incorrect.  In Delivering Development and a number of related publications (for example, here, here, and here) I have laid out how, in the context of livelihoods, material considerations are always bound up in social considerations.  If you only evaluate these actions as aimed at material goals, you’ve only got a part of the picture – and not the most important part, in most cases.  Instead, what you are left with are a bunch of decisions and outcomes that appear illogical, that can be cast as mistakes.  Only most of the time, they are not mistakes – they are conscious choices.

Let me offer an example from Delivering Development and some of my other work – the constraint of women’s farming by their husbands.  I have really compelling qualitative evidence from two villages in Ghana’s Central Region that demonstrates that men are constraining their wives’ farm production to the detriment of the overall household income.  The chart below shows a plot of the size of a given farm versus its market orientation for the households operating under what I call a “diversified” strategy – where the husband farms for market sale, and the wife for subsistence (a pretty common model in sub-Saharan Africa).  As you move up the Y axis, the farm gets more oriented toward market sale (1 on that scale is “eat everything”, 3 is sell and eat equally, and 5 is sell everything).  Unsurprisingly, since men’s role requires them to produce for market, the size of their farm has little impact on their orientation.  But look at the women’s farms – just a tenth of a hectare produces a marked shift in orientation from subsistence to market production…because women own that surplus beyond subsistence, and sell it.  They take the proceeds of these sales, buy small goods, and engage in petty trading, eventually multiplying that small surplus into significant gains in income, nearly equaling their husbands.  What is not to like?

Well, from the perspective of those in these villages, here is something: among the Akan, being a “good man” means being in control of the household and out-earning your wife.  If you don’t, your fitness as a man gets called into question, which can cost you access to land.  For wives, this is bad because they get their land through their husbands.  So as a result, being in a household where the woman out-earns her husband is not a viable livelihoods outcome (as far as members of these households are concerned).  Even if a man wanted to let his wife earn more money, he would do so at peril of his access to land. So he is not going to do that.  What he is going to do is shrink his wife’s farm the next season to ensure she does not out-earn him (and I have three years of data where this is exactly what happens to wives who earn too much).  There is a “mistake” here – some of these men underestimated their wives’ production, which is pretty easy to do under rain-fed agriculture in a changing climate.  That they are this accurate with regard to land allocation is rather remarkable, really.  But the decision to constrain women’s production is not a mistake, per se: it is a choice.

We can agree or disagree with the premises of these choices, and their outcomes, but labeling them as mistakes creates a false sense of simplicity in addressing problematic outcomes – because people only require “correction” to get to the outcomes we all want and need.  This, in turn, rests on/reproduces a sense of superiority on the part of the researcher – because s/he knows what is best (see a previous post on this point here).  That attitude, applied to the case above, would not result in a productive project design aimed at addressing income or other challenges in these villages.

Yes, people do things against material interest…but there is always a logic behind a decision, and that logic is often deeply entrenched.  We would be better off talking about decisions poor people make (for better or worse), and dedicating our time to understanding why they make these decisions before we start deciding who is mistaken, and what to do about it.

I’ve just burned 15,000 words in Third World Quarterly laying out my argument for how to think about livelihoods as more than material outcomes – and how to make that vision implementable, at least via fieldwork that runs in length from days to months.  I am happy to send a copy of the preprint to anyone who is interested –and I will post a version to my website shortly.

Update: 11/22: So, after seeing Tom Murphy’s Storify of the twitter exchange, it is now clear that Sachs was on fire – the man was engaged in several conversations at once along the lines below…and he seems to have been responding to all of them pretty coherently, and in real time. I admit to being impressed (No, seriously, click on the Storify link there and just scroll. It is boggling). So recognize that what you see below is what I saw in my feed (his other conversations were with people I don’t follow, so I didn’t realize they were ongoing). Still, glad to get geography’s foot back in the door…

So, quite by surprise, I found myself on the end of an extended twitter exchange with Jeff Sachs.  I’ve hassled him via twitter before, and never had a response. So, I was a bit taken aback to see my feed light up about 30 seconds after I tweeted with @JeffDSachs at the front end! To give Sachs credit, he stayed quite engaged and did seem to be taking on some of my points. Granted, 140 characters is hardly enough to really convey the issues at hand, but I did the best I could to represent contemporary human geography. Y’all be the judge – this is the feed, slightly rejiggered to clarify that at times Sachs and I were crossing each other’s messages – he was clearly responding to a previous message sometimes when he tweeted back after one of my tweets. Also, Samuel Danthine was also on the conversation, and I kept him in the timeline as it seems he and I were coming from the same place:

I’ve long hated the term “poverty traps,” development shorthand for conditions in which poverty becomes self-reinforcing and therefore inescapable without some sort of external intervention.  They made no analytic sense (nobody ever defined poverty clearly across this literature, for example), and generally the idea of the poverty trap was hitched to a revival of “big push” development efforts that had failed in the 1950s and 1960s.  Further, it was always clear to me that the very idea of a poverty trap cast those living in difficult circumstances as helpless without the intervention of benevolent outsiders.  This did not align at all with my experiences on the ground in rural sub-Saharan Africa.

This is not to suggest that there is no such thing as structural inequality in the world – the running head start enjoyed by the Global North in terms of economic development has created significant barriers to the economic development of those residing in the Global South.  These barriers, perhaps most critically the absurd and damaging regime of subsidies that massively distorts global agricultural markets, must be addressed, and soon.  Such barriers generally result in perverse outcomes that impact even those in the Global North (anyone who thinks the American food system makes any sense at all really needs to read more.  Start with Fast Food Nation, move to The Omnivore’s Dilemma, and work out from there. And don’t get me going on the potential climate impacts of structural inequality).

But this enduring focus on structural problems in the global economy has had the effect of reducing those in the Global South to a bunch of helpless children in need of salvation by the best and most noble of those in the Global North, who were to bring justice, opportunity, and a better future to all.  If this isn’t the 21^st Century version of the White Man’s Burden, then I don’t know what is.  Bill Easterly makes a very similar point very eloquently, and at much greater length, here.

I am a social scientist*, and I believe that the weight of evidence eventually wins arguments.  And today it occurred to me that in this case, this long line of arguing that those who insisted on talking about poverty traps were a) generally misrepresenting the world and b) inappropriately infantilizing those living in the Global South now has that weight of evidence behind it.  Andy Sumner’s work on the New Bottom Billion basically blows up the idea of the poverty trap – he demonstrates that since the 1990s, a lot of people that were thought to be living in poverty traps have improved their incomes such that many have moved out of poverty (at least if one defines poverty on the basis of income).  People who were thought to be trapped by structural inequality have been defying expectations and improving their circumstances without clear correlations to aid or development efforts, let alone the “big push” arguments of Sachs and others.  In short, it looks like we don’t really understand what people are doing at the margins of the Global South, and that the global poor are a lot more capable than development seems to think.  Poor people attached to the anchor of structural inequality are dragging it to improved incomes and well-being in thousands of small, innovative ways that are adding up to a massive aggregate change in the geography and structure of global poverty.

In short, the Global South never needed the most enlightened of the Global North to clear the path and push them up the ladder of development (if you want to get all Rostow about it).  Instead, what is clearly needed is a new, substantial effort to better understand what is happening out on Globalization’s Shoreline, and to work with the global poor to examine these efforts, identify innovative, locally-appropriate, and locally-owned means of transforming their quality of life, and find means of bringing those ideas to (appropriate) scale.  Anything else is just hubris at best, and subtle class/race bigotry at worst.

The data is speaking. Anyone ready to listen?

*Well, I am a qualitative social scientist which means my work is more generative and humanities/arts flavored than is typical in the sciences, which generally value the reporting of observations in the framework of already-established biophysical processes.

The Guardian recently ran a piece titled “Food scarcity: the timebomb setting nation against nation.” It was retweeted a lot across my social network, enough that I feel the need to respond to it.  So, here it goes.  The article is yet another example of the remarkably durable narrative of production crisis that dominates discussions of food security today.  The article operates from the assumption that we are running out of food, and then selectively interprets quotes from Lester Brown and Oxfam to support this attention-grabbing story.   The problem here is that Brown/Oxfam make much more nuanced claims than suggested by the headline, which perpetuates the neo-Malthusian agenda of scarcity that dominates modern food security.  In short, I find the very title and tone of the article to be terribly irresponsible – in attempting to bring attention to the very serious issue of global hunger, this article sets back intelligent conversation about the causes of the problem, and therefore its solutions.

It takes little but careful reading to see that the Guardian piece doesn’t actually have the evidence to say that food scarcity is a geopolitical timebomb.  Brown never says we have an absolute scarcity of food in the world, just increased levels of pressure on the food system.  The issue of increased pressure is not, as the article suggests, about production, per se: it is about a complex global political economy that intersects in complicated ways with the remnants of colonialism, failed development, and environmental change in particular ways in specific places.  Sure, US grain production is down 15%…but that isn’t a big deal against the GLOBAL 40% rate of waste in the food system.  We can cover the current US shortfall (indeed, more or less any conceivable global shortfall) with ease just by cleaning up some low-hanging fruit in the global supply chain, such as improving the transportation networks from farm to market in the Global South.

The only actual argument for scarcity in the article is buried down the page, in Evan Fraser’s claim:

“For six of the last 11 years the world has consumed more food than it has grown. We do not have any buffer and are running down reserves. Our stocks are very low and if we have a dry winter and a poor rice harvest we could see a major food crisis across the board.”

It seems to me that Fraser is misreading his evidence. It is possible that the world has consumed more food than has been available on agricultural markets…but this is NOT THE SAME THING as the amount of food grown.  In each of the last 11 years, humanity grew much more food than it consumed. It’s just that each year we then wasted about 40% of that production as it either rotted on the way to market (a common problem in the Global South) or we threw it away uneaten (a problem in the wealthy countries).

So if there is no global food production crisis, why are we seeing land grabbing that will set “nation against nation”?  After all, if there is plenty of agricultural production globally, land-grabbing for food supplies is nonsensical behavior. Prices are where they are because the global food system has significant problems that could be addressed relatively easily and at relatively low cost (when compared to the challenge of completely reengineering an agricultural ecosystem). Anyone analyzing things in a serious way should see this, and recognize that food prices are a bubble that could be popped by a serious infrastructural development push.  And, as it happens, they have.  If you read the article carefully, you realize that there is no evidence in this article that land grabs are for food as much as they are for biofuels. Oxfam’s report is more to the point – the planting of biofuels has to be taken seriously, as that does take arable land out of local production, which can stress local food systems.  But if anyone thought there was a serious global food shortage, they would not buy arable land for biofuels – they would buy it for food itself, as after a certain point food prices become inelastic. The very fact the land grabs are heavily for biofuels tells us all we need to know about the idea of a global food shortage.

Rising food prices in today’s world just signal a stress point on today’s (astonishingly inefficient) food system. Leveraged correctly, these pressures could bring about dramatic changes in global food markets, as saving even half of the food that rots on the way to market in the Global South would more than offset all but the most extreme local food deficits.  This is an opportunity to make changes in the food system that are immediate and relatively cost-efficient.  For all of the noble intents here, ginning up cries of false scarcity in the name of focusing attention on global hunger drags the policy conversation away from real, achievable solutions.

Man, has there ever been a less enticing blog post title?  But it pays to be direct – so there it is.  I have funding for a Ph.D. student, starting in January, to help me on my USAID-funded work on climate services for development.  So, without further ado, the ad:

Graduate Student Opportunity for January 2013

University of South Carolina, Department of Geography

Ed Carr is seeking a Ph.D. student to support ongoing work on climate services for development in sub-Saharan Africa and develop an independent research program in this broad area of inquiry.  The funding for this position is attached to USAID’s Climate Change Resilient Development (CCRD) program, and the candidate will have specific responsibilities supporting the the development of field methods and the analysis of preliminary data, as well as conducting extensive fieldwork in one or more Malian communities in May-July 2013 as part of the project “An Assessment of Mali Meteorological Service’s Agrometeorological Program.”

Qualifications:

• Candidates will have to be admitted to the geography graduate program at the University of South Carolina
• Candidates should be from a country in which USAID operates. Preference will be given to candidates from West Africa, then other parts of sub-Saharan Africa, as this is the current target region for the project.
• Candidates should have experience in one or more of the following: climate change adaptation, rural/community development, rural agriculture, climate science
  • The bulk of initial project work will focus on community-level information needs, and therefore preference will be given to those candidates with experience conducting qualitative research in rural settings.
• Candidates should hold a Masters degree in Geography, Anthropology, Planning or another closely related field
• Excellent written and spoken English.  French language ability is preferred.

The duration of funding is January-July 2013, with likely continuation through July 2014.  The candidate will receive tuition, a living stipend, and salary/research support for work to be conducted in May-July 2013.  Candidates who meet departmental expectations of progress and excellence will be eligible for additional semesters of support to complete their degrees.

Please note the very short lead time for this opportunity – viable candidates will likely have to have a visa in hand if they are to start in January 2013.  Candidates who cannot make this deadline, or who are not selected in this round, should stay tuned – I am hoping to open up a few more slots in the fall.

Prospective candidates are encouraged to contact Ed Carr at carr@sc.edu.  Applications are due on 1 November, 2012 via the instructions on the departmental web page: http://artsandsciences.sc.edu/geog/academics/admissions.html

Friend/colleague/journalist Keith Kloor has a very interesting piece about the problematic character of some research that recently purported to cast doubt on the health safety of GMOs.  The piece is an excellent effort to push back against crap science and crap evidence in the GMO debate, and stands as an interesting example of a general need to critically evaluate “scientific” claims about any number of issues from the harmful character of vaccines to those would deny anthropogenic climate change.  However, the piece, in making a strenuous argument for evidence, overreaches in its conclusions and too-quickly dismisses the ecological issues associated with GMOs – exactly the opposite outcome Keith was seeking.

In general, I agree with Keith that the screaming about the health impacts of GMOs has greatly outstripped the evidence, but I take issue with the idea that GMOs have been largely proven safe all around.  There remain significant questions about ecological impacts that have not attracted a solid scientific consensus (i.e the impact/cost/tradeoffs of gene flow between GMOs and the surrounding ecology, pest resistance, etc. –.  The evidence base in this area is pretty small, and ecological systems are very diverse and complex, so the levels of uncertainty here are fairly epic (Pamela Ronald, whose article Keith references approvingly, even notes that the ecological impacts of GMOs are still an open issue).  Honestly, Keith is too dismissive of this challenge:

Some of these folks are worried about new genes being introduced into plant and animal species. But humans have been selectively breeding plants and animals pretty much since we moved out of caves, manipulating their genes all the while. The process was just slower before biotechnology came along.

Modern genetic engineering does not have all that much to do with selective breeding.  We are doing things in GMOs that could not occur in nature, which is rather different then applying what amounts to a modified evolutionary force (human selection of seeds to plant the next season or animals to breed) in the selection of traits within certain crops.

Another big concern that has been widely reported is the “rapid growth of tenacious super weeds” that now defy Monsanto’s trademark Roundup herbicide. That has led farmers to spray their fields with an increasing amount of the chemical weed-killer. Additionally, some research suggests that other pests are evolving a resistance to GMO crops. But these problems are not unique to genetic engineering. The history of agriculture is one of a never-ending battle between humans and pests.

Again, this is a bit off to one side.  Yes, agriculture has always been about the radical simplification and management of ecosystems in a complex world, and therefore is indeed an endless battle for control between farmer and surrounding ecology.  What this elides, though, is the fact that some of these GMOs have made things harder for us in the long run.  It is hard to sell this as a good, or even neutral, outcome.

I raise this issue because of its implications for conversations about food security. People (some, but not all) working in food security tend to be a bit dismissive of the ecological concerns surrounding GMOs, loosely equating them with the human consumption concerns that have largely been disproven at this point.  The response to insistent questioning about ecological impacts is the argument that, on balance, GMOs have done more good than harm and/or that GMOs are necessary for global food security going forward.  The “more harm than good” argument seems to me based on evidence that is hardly complete, and often references a glowing, startlingly unproblematic vision of the Green Revolution.  The “we need GMOs” is a crisis narrative of food production that drives current and future claims about their value.  At the moment, much of the food security world is arguing that there is a dire food production crisis either coming or just arrived (embodied in the 2008 and 2011 food price spikes), and therefore we cannot wait for the burden of proof surrounding any issue associated with GMOs – we must act now!  This crisis narrative effectively reduces anyone who raises objections to an all-out push for more production to some sort of monster who would rather let innocents starve than engage with the messiness of the real world. Basically, precaution is conflated with timidity, making those who show concern over the ecological impacts of GMOs into the food security equivalents of Neville Chamberlain.

This would perhaps be true…if, in fact there were a global production crisis and we needed GMOs to feed everyone. This is a much harder thing to sell, however, when one realizes that the world produces roughly twice as much food as needed to feed everyone adequately each year.  There isn’t a global production crisis (though there are local production crises that arise from complex causes and need to be addressed in locally-specific ways). I’ve said it before, and I will say it again: globally, there is plenty of food. Now, and likely for quite some time. Period.

In the twitter conversations that emerged around Keith’s article, Robert Wilson, a mathematical ecologist, argued for “an adjusted precautionary principle where we consider the risks of inaction, as well as action.”  This strikes me as both reasonable and necessary when discussing food security policy and programs these days.  If we did this, though, I think we would find that the risks of inaction are, in fact, substantially smaller than the risks of action – this is not to say there are necessarily huge risks to action, just that when you are producing twice the calories needed already, the risk from doing nothing is, at least in the short term, pretty small.  There is time to test more crops, more widely, under more conditions, before we arrive at any production crunch.  There is no need to rush.

In short, we need to be sure that our efforts to push back against crap science (which is really what Keith was doing in his article) don’t overreach and inadvertently empower narratives and arguments that are not supported by the evidence – exactly the opposite outcome Keith sought.

As I mentioned a few posts ago, I am working through James Scott’s The Art of Not Being Governed (my endorsement is in the linked post).  In the course of my reading, I have been thinking about what Scott calls the State Accessible Product, which he sets in contrast to the Gross Domestic Product.  To Scott’s thinking, the states/kingdoms he is discussing in Southeast Asia were motivated not to maximize the value of all goods and services in their realms, as such maximization might include the production of goods that could not be transported/taxed/otherwise used to enrich the state.  Instead, it was in the state’s interest to maximize the production of things it could see, count and move – in other words, to push the growth of a State Accessible Product.  Two things resonate for me about this idea:

1) It scales. Just as states pushed for the production of SAP, so too the households I I discuss in Delivering Development tend to divide up livelihoods roles and activities in a manner that maximizes not individual well-being, but activities that only make sense when bundled with the activities of other members of the household – a sort of Household Accessible Product.  In an uncertain economy and environment, it makes no sense to focus one’s entire agricultural production on market sale, or to focus entirely on subsistence reproduction of the household.  Yet this is just what we see men (playing the former role) and women (playing the latter) doing in some of the households I examined in Ghana.  They do this for a lot of complex reasons, but certainly there is something to the idea that these roles force the members of the household into the production of a HAP that certainly does not maximize all possible production and income, but does a lot to reproduce social roles and social stability.

2) It explains why my argument that a lot of farmers on globalization’s shoreline strategically deglobalize was both surprising and, at least to some people, threatening: the opting in and out of global markets is exactly the sort of thing states fear, as it means that the production of these farmers goes in and out of legibility from year to year – making it hard to extract value from that production. In short, there is a GDP that is not coterminus with an SAP along most of globalization’s shoreline – and that non SAP production is critical to the well-being of those engaged in those activities.

It strikes me that a key question here is whether or not our focus on governance in development has led us to inadvertently emphasize activities, projects and programs that render greater and greater percentages of GDP as SAP – certainly, without access to the financial resources produced by the control of a SAP, states are in a weak position.  But if many of the activities that actually keep people alive on a day-to-day basis are non-SAP activities, what are we to do?  Are we to wipe out/make legible these activities so the state can profit from them?  If we do, are we going to enhance the vulnerability of the populations whose livelihoods we alter?  Is the enhancement of vulnerability an appropriate trade-off for the creation of a state-legible economy?  Can addressing vulnerability and building a strong state be made to rhyme at all?

Ben Leo at ONE.org (formerly of CGD) put forth an intriguing proposal recently on Huffington Post Impact: It’s Time to Ask the World’s Poor What They Really Want.  In short, Ben is trying to argue that the current top-down definition of development goals, no matter how well-intentioned, is unlikely to reflect the views of the people these development goals are meant to benefit.

Hear, hear.  I made a similar point in Delivering Development. Actually, that sort of was one of the main points of the book.  See also my articles here and here.

But I am concerned that Leo is representing this effort a little too idealistically.  Just because we decide to ask people what they want doesn’t mean that we will really find out what they want.  Getting to this sort of information has everything to do with asking the right questions in the right way – there is no silver bullet for participation that will ensure that everyone’s voices will be heard.  To that end, what worries me here is that Ben does not explain exactly how ONE plans to develop the standardized survey they will put out there, or how exactly they will administer this survey.  So, here are a few preliminary questions for Ben and the ONE team:

1)   Does a standardized survey make sense? Given the very different challenges that people face around the world, and the highly variable capacity of people to deal with those challenges, it seems to me that going standardized is going to result in one of two outcomes: either you ask focused questions that only partially capture the challenges facing most people, or you ask really general questions that basically capture the suite of challenges we see globally, but do so in a manner that is so vague as to be unactionable.  How will ONE thread this needle?

2)   Who is designing the survey? To my point above, what questions are asked determine who will answer, and therefore determines what you will learn.  While the information gleaned from this sort of survey is likely to be very interesting, it is not the same thing as an open participatory process – full participation includes defining the questions, not just the answers.  Indeed, I would suggest that ONE needs to ditch the term participatory here, as in the end I fear it will be misleading.

3)   How will you administer the survey? Going out with enumerators takes a lot of time and money, and is subject to “investigator bias” – that is, the simple problem that some enumerators will do their job in a different manner than others, thus getting you different kinds/qualities of answers to the same questions.  On the other hand, if you are reliant on mobile technology, how will you incentivize those rural populations with mobile handsets to participate?  If you can’t do this, you will end up with a highly unrepresentative sample, making the results far less useful.

This is not to dismiss the effort Ben is spearheading – indeed, it is fantastic to see a visible organization make this argument and take concrete steps to actually get the voices of the global poor into the agenda-setting exercises.  However, this is not a participatory process – it is, instead, an information-driven process (which is good) that is largely shaped by the folks at ONE in the name of the global poor.  If ONE wants this to be more than information-driven, it needs to think about how it is going to let a representative sample of the global poor define the questions as well as the answers.  That is no easy task.

In all sincerity, I am happy to talk this through with anyone who is interested – I do think it is a good idea in principle, but execution is everything if you want it to be more than a publicity stunt…

Alright, last post I laid out an institutional problem with M&E in development – the conflict of interest between achieving results to protect one’s budget and staff, and the need to learn why things do/do not work to improve our effectiveness.  This post takes on a problem in the second part of that equation – assuming we all agree that we need to know why things do/do not work, how do we go about doing it?

As long-time readers of this blog (a small, but dedicated, fanbase) know, I have some issues with over-focusing on quantitative data and approaches for M&E.  I’ve made this clear in various reactions to the RCT craze (see here, here, here and here). Because I framed my reactions in terms of RCTs, I think some folks think I have an “RCT issue.”  In fact, I have a wider concern – the emerging aggressive push for quantifiable data above all else as new, more rigorous implementation policies come into effect.  The RCT is a manifestation of this push, but really is a reflection of a current fad in the wider field.  My concern is that the quantification of results, while valuable in certain ways, cannot get us to causation – it gets us to really, really rigorously established correlations between intervention and effect in a particular place and time (thoughtful users of RCTs know this).  This alone is not generalizable – we need to know how and why that result occurred in that place, to understand the underlying processes that might make that result replicable (or not) in the future, or under different conditions.

As of right now, the M&E world is not doing a very good job of identifying how and why things happen.  What tends to happen after rigorous correlation is established is what a number of economists call “story time”, where explanation (as opposed to analysis) suddenly goes completely non-rigorous, with researchers “supposing” that the measured result was caused by social/political/cultural factor X or Y, without any follow on research to figure out if in fact X or Y even makes sense in that context, let alone whether or not X or Y actually was causal.  This is where I fear various institutional pushes for rigorous evaluation might fall down.  Simply put, you can measure impact quantitatively – no doubt about it.  But you will not be able to rigorously say why that impact occurred unless someone gets in there and gets seriously qualitative and experiential, working with the community/household/what have you to understand the processes by which the measured outcome occurred.  Without understanding these processes, we won’t have learned what makes these projects and programs scalable (or what prevents them from being scaled) – all we will know is that it worked/did not work in a particular place at a particular time.

So, we don’t need to get rid of quantitative evaluation.  We just need to build a strong complementary set of qualitative tools to help interpret that quantitative data.  So the next question to you, my readers: how are we going to build in the space, time, and funding for this sort of complementary work? I find most development institutions to be very skeptical as soon as you say the words qualitative…mostly because it sounds “too much like research” and not enough like implementation. Any ideas on how to overcome this perception gap?

(One interesting opportunity exists in climate change – a lot of pilot projects are currently piloting new M&E approaches, as evaluating impacts of climate change programming requires very long-term horizons.  In at least one M&E effort I know of, there is talk of running both quantitative and qualitative project evaluations to see what each method can and cannot answer, and how they might fit together.  Such a demonstration might catalyze further efforts…but this outcome is years away)

One of the things I have had the privilege to witness over the past two years is the movement of a large donor toward a very serious monitoring and evaluation effort aimed at its own programs.  While I know some in the development community, especially in academia, are skeptical of any new initiative that claims to want to do a better job of understanding the impact of programs, and learning from existing programs, what I saw in practice leads me to believe that this is a completely sincere effort with a lot of philosophical buy-in.

That said, there are significant barriers coming for monitoring and evaluation in development.  I’m not sure that those making evaluation policy fully grasp these barriers, and as a result I don’t see evidence that they are being effectively addressed by anyone.  Until they are, this sincere effort is likely to underperform, if not run aground.

In this post, I want to point out a huge institutional/structural problem for M&E: the conflict of interest that is created on the implementation side of things.  On one hand, donors are telling people that we need to learn about what works, and that monitoring and evaluation is not meant to be punitive, but part of a learning process to help all of us do our jobs better.  On the other hand, at most donors the budgets are under pressure, and the message from the top is that development must focus on “what works.”  Think about what this means to a mission director or a chief of party.  On one hand, they are told that M&E is about learning, and failure is now to be expected and can be tolerated as long as we learn why the failure occurred and can remedy the problem and prevent that problem in the future in other places.  On the other, they are told that budgets will focus on what works.  So if they set up rigorous M&E, they are likely to identify programs that are underperforming (and perhaps learn why)…but there is no guarantee that this learning won’t result in that program being cut, with a commensurate loss of staff and budget.  I have yet to see anyone meaningfully address this conflict of interest, and until someone figures out how to do so, there will be significant and creative resistance to the implementation of rigorous M&E.

Any ideas, folks?  Surely some of you have seen this at work…

Simply put, the donors are going to have to decide what is more important – learning what works, and improving on development’s 60+ year track record of spotty results with often limited correlation to programs and projects, or maintaining the appearance of efficiency and efficacy by cutting anything that does not seem to work, and likely throwing out a lot of babies with the bathwater. I know which one I would choose.  It remains unclear where the donors’ choices will fall.  In a politically challenging environment, the pressure to go with the latter approach is high, and the protection of a learning agenda that will really change how development works will require substantial political courage.  That courage exists…but whether or not it comes to the fore is a different question.