Search Results for “rct”.


Well, the response to part one was great – really good comments, and a few great response posts.  I appreciate the efforts of some of my economist colleagues/friends to clarify the terminology and purpose behind RCTs.  All of this has been very productive for me – and hopefully for others engaged in this conversation.

First, a caveat: On the blog I tend to write quickly and with minimal editing – so I get a bit fast and loose at times – well, faster and looser than I intend.  So, to this end, I did not mean to suggest that nobody was doing rigorous work in development research – in fact, the rest of my post clearly set out to refute that idea, at least in the qualitative sphere.  But I see how Marc Bellemare might have read me that way.  What I should have said was that there has always been work, both in research and implementation, where rigorous data collection and analysis were lacking.  In fact, there is quite a lot of this work.  I think we can all agree this is true . . . and I should have been clearer.

I have also learned that what qualitative social scientists/social theorists mean by theory, and what economists mean by theory, seems to be two different things.  Lee defined theory as “formal mathematical modeling” in a comment on part 1 of this series of posts, which is emphatically not what a social theorist might mean.  When I say theory, I am talking about a conjectural framing of a social totality such that complex causality can at least be contained, if not fully explained.  This framing should have reference to some sort of empirical evidence, and therefore should be testable and refinable over time – perhaps through various sorts of ethnographic work, perhaps through formal mathematical modeling of the propositions at hand (I do a bit of both, actually).  In other words, what I mean by theory (and what I focus on in my work) is the establishment of a causal architecture for observed social outcomes.  I am all about the “why it worked” part of research, and far less about the “if it worked” questions – perhaps mostly because I have researched unintended “development interventions” (i.e. unplanned road construction, the establishment of a forest reserve that alters livelihoods resource access, etc.) that did not have a clear goal, a clear “it worked!” moment to identify.  All I have been looking at are outcomes of particular events, and trying to establish the causes of those outcomes.  Obviously, this can be translated to an RCT environment because we could control for the intervention and expected outcome, and then use my approaches to get at the “why did it work/not work” issues.

It has been very interesting to see the economists weigh in on what RCTs really do – they establish, as Marc puts it, “whether something works, not in how it works.”  (See also Grant’s great comment on the first post).  I don’t think that I would get a lot of argument from people if I noted that without causal mechanisms, we can’t be sure why “what worked” actually worked, and whether the causes of “what worked” are in any way generalizable or transportable.  We might have some idea, but I would have low confidence in any research that ended at this point.  This, of course, is why Marc, Lee, Ruth, Grant and any number of other folks see a need for collaboration between quant and qual – so that we can get the right people, with the right tools, looking at different aspects of a development intervention to rigorously establish the existence of an impact, and the establish an equally rigorous understanding of the causal processes by which that impact came to pass.  Nothing terribly new here, I think.  Except, of course, for my continued claim that the qualitative work I do see associated with RCT work is mostly awful, tending toward bad journalism (see my discussion of bad journalism and bad qualitative work in the first post).

But this discussion misses a much larger point about epistemology – what I intended to write in this second part of the series all along.  I do not see the dichotomy between measuring “if something works” and establishing “why something worked” as analytically valid.  Simply put, without some (at least hypothetical) framing of causality, we cannot rigorously frame research questions around either question.  How can you know if something worked, if you are not sure how it was supposed to work in the first place?  Qualitative research provides the interpretive framework for the data collected via RCT4D efforts – a necessary framework if we want RCT4D work to be rigorous.  By separating qualitative work from the quant oriented RCT work, we are assuming that somehow we can pull data collection apart from the framing of the research question.  We cannot – nobody is completely inductive, which means we all work from some sort of framing of causality.  The danger is when we don’t acknowledge this simple point – under most RCT4D work, those framings are implicit and completely uninterrogated by the practitioners.  Even where they come to the fore (Duflo’s 3 I s), they are not interrogated – they are assumed as framings for the rest of the analysis.

If we don’t have causal mechanisms, we cannot rigorously frame research questions to see if something is working – we are, as Marc says, “like the drunk looking for his car keys under the street lamp when he knows he lost them elsewhere, because the only place he can actually see is under the street lamp.”  Only I would argue we are the drunk looking for his keys under a streetlamp, but he has no idea if they are there or not.

In short, I’m not beating up on RCT4D, nor am I advocating for more conversation – no, I am arguing that we need integration, teams with quant and qual skills that frame the research questions together, that develop tests together, that interpret the data together.  This is the only way we will come to really understand the impact of our interventions, and how to more productively frame future efforts.  Of course, I can say this because I already work in a mixed-methods world where my projects integrate the skills of GIScientists, land use modelers, climate modelers, biogeographers and qualitative social scientists – in short, I have a degree of comfort with this sort of collaboration.  So, who wants to start putting together some seriously collaborative, integrated evaluations?

Those following this blog (or my twitter feed) know that I have some issues with RCT4D work.  I’m actually working on a serious treatment of the issues I see in this work (i.e. journal article), but I am not above crowdsourcing some of my ideas to see how people respond.  Also, as many of my readers know, I have a propensity for really long posts.  I’m going to try to avoid that here by breaking this topic into two parts.  So, this is part 1 of 2.

To me, RCT4D work is interesting because of its emphasis on rigorous data collection – certainly, this has long been a problem in development research, and I have no doubt that the data they are gathering is valid.  However, part of the reason I feel confident in this data is because, as I raised in an earlier post,  it is replicating findings from the qualitative literature . . . findings that are, in many cases, long-established with rigorously-gathered, verifiable data.  More on that in part 2 of this series.

One of the things that worries me about the RCT4D movement is the (at least implicit, often overt) suggestion that other forms of development data collection lack rigor and validity.  However, in the qualitative realm we spend a lot of time thinking about rigor and validity, and how we might achieve both – and there are tools we use to this end, ranging from discursive analysis to cross-checking interviews with focus groups and other forms of data.  Certainly, these are different means of establishing rigor and validity, but they are still there.

Without rigor and validity, qualitative research falls into bad journalism.  As I see it, good journalism captures a story or an important issue, and illustrates that issue through examples.  These examples are not meant to rigorously explain the issue at hand, but to clarify it or ground it for the reader.  When journalists attempt to move to explanation via these same few examples (as far too often columnists like Kristof and Friedman do), they start making unsubstantiated claims that generally fall apart under scrutiny.  People mistake this sort of work for qualitative social science all the time, but it is not.  Certainly there is some really bad social science out there that slips from illustration to explanation in just the manner I have described, but this is hardly the majority of the work found in the literature.  Instead, rigorous qualitative social science recognizes the need to gather valid data, and therefore requires conducting dozens, if not hundreds, of interviews to establish understandings of the events and processes at hand.

This understanding of qualitative research stands in stark contrast to what is in evidence in the RCT4D movement.  For all of the effort devoted to data collection under these efforts, there is stunningly little time and energy devoted to explanation of the patterns seen in the data.  In short, RCT4D often reverts to bad journalism when it comes time for explanation.  Patterns gleaned from meticulously gathered data are explained in an offhand manner.  For example, in her (otherwise quite well-done) presentation to USAID yesterday, Esther Duflo suggested that some problematic development outcomes could be explained by a combination of “the three I s”: ideology, ignorance and inertia.  This is a boggling oversimplification of why people do what they do – ideology is basically nondiagnostic (you need to define and interrogate it before you can do anything about it), and ignorance and inertia are (probably unintentionally) deeply patronizing assumptions about people living in the Global South that have been disproven time and again (my own work in Ghana has demonstrated that people operate with really fine-grained information about incomes and gender roles, and know exactly what they are doing when they act in a manner that limits their household incomes – see here, here and here).  Development has claimed to be overcoming ignorance and inertia since . . . well, since we called it colonialism.  Sorry, but that’s the truth.

Worse, this offhand approach to explanation is often “validated” through reference to a single qualitative case that may or may not be representative of the situation at hand – this is horribly ironic for an approach that is trying to move development research past the anecdotal.  This is not merely external observation – I have heard from people working inside J-PAL projects that the overall program puts little effort into serious qualitative work, and has little understanding of what rigor and validity might mean in the context of qualitative methods or explanation.  In short, the bulk of explanation for these interesting patterns of behavior that emerges from these studies resorts to uninterrogated assumptions about human behavior that do not hold up to empirical reality.  What RCT4D has identified are patterns, not explanations – explanation requires a contextual understanding of the social.

Coming soon: Part 2 – Qualitative research and the interpretation of empirical data

Just a quick thought, given the interest in my last two posts. I have (obviously) been driven a bit crazy by the RCT/behavioral economics folks who suddenly seem to be coming around to either qualitative methods to explain their results . . . which are generally results that qualitative researchers have known about for a long time.  In other words, it seems to me that there is a real danger here that a sort of waving at qualitative methods (i.e. the “bad journalism” approach, where you just do a bunch of interviews without considering who to interview, how to interview them, why interviews vs. focus groups vs. whatever, etc.) might become yet another way to prolong the hegemony of economics over development thinking.

I’m worried about this because while I think economic approaches and theory have purchase on explanation to varying degrees depending on the subject at hand and the scale of analysis, in the end any effort to explain the emergence of and means of addressing a development issue or challenge at a scale that might have meaningful impact that relies on the economic alone will not result in a particularly complete explanation for observed events, nor will it help us understand likely outcome pathways in future similar situations.  Put another way, only sometimes can economics get us to a “good enough” solution that enables really productive development work.

What if I told you that I had really good, concrete empirical data from a really tiny dataset (two villages in Ghana, but the entire population of those two villages, so no sampling issues) that clearly demonstrated that any effort to explain livelihoods decision-making in these villages cannot be productively explained by economic approaches – whether crude (i.e. assumptions about maximizing behavior) or complex (game theoretic approaches)?  Instead, the data makes it remarkably clear that the economic, while a component of decision-making, is just one component of a project of household governance – it is a clearly external (etic, for you anthro types out there) heuristic that improperly parses the social processes that lead to livelihoods decisions. In short, I can show where the explanatory power of “the economic” stops, and where meaningful explanation requires a re-embedding of the economic in larger social processes that cannot be reduced to the economic (and, at the same time, which demonstrates that the economic cannot be reduced to any of these other processes).

Basically, I’m starting to walk you through my retheorization of livelihoods as what Foucault called governmentality . . . but I could also work this up as a means of discrediting the RCT and behavioral economics turn toward the qualitative by arguing that these efforts are tails wagging the dog . . .

Thoughts?

Raj Shah has announced his departure from USAID. Honestly, this surprises nobody at the Agency, or anyone in the development world who’s been paying attention. If anything, folks are surprised he is still around – it is well-known (or at least well-gossiped) that he was looking for the door, and at any number of opportunities, at least since the spring of 2012. There are plenty of reviews of Shah’s tenure posted around the web, and I will not rehash them. While I have plenty of opinions of the various initiatives that Shah oversaw/claims credit for (and these are not always the same, by the way), gauging what did and did not work under a particular administrator is usually a question for history, and it will take a bit of space and time before anyone should feel comfortable offering a full review of this administrator’s work.

I will say that I hope much of what Shah pushed for under USAID Forward, especially the rebuilding of the technical capacity of USAID staff, the emphasis on local procurement, and the strengthening of evaluation, becomes entrenched at the agency. Technical capacity is critical – not because USAID is ever going to implement its own work. That would require staffing the Agency at something like three or four times current levels, and nobody is ever going to approve that. Instead, it is critical for better monitoring and evaluating the work of the Agency’s implementing partners. In my time at USAID, I saw implementer work and reports that ran the gamut from “truly outstanding” to “dumpster fire”. The problem is that there are many cases where work that falls on the dumpster fire end of the spectrum is accepted because Agency staff lack the technical expertise to recognize the hot mess they’ve been handed. This is going to be less of a problem going forward, as long as the Agency continues to staff up on the technical side.

Local procurement is huge for both the humanitarian assistance and development missions of USAID. For example, there is plenty of evidence supporting the cost/time effectiveness of procuring emergency food aid in or near regions of food crisis. Further, mandates that push more USAID funding to local organizations and implementers will create incentives to truly build local capacity to manage these funds and design/implement projects, as it will be difficult for prime contractors to meet target indicators and other goals without high-capacity local partners.

A strong evaluation policy will be huge for the Agency…if it ever really comes to pass. While I have seen real signs of Agency staff struggling with how to meaningfully evaluate the impact of their programs, the overall state of evaluation at the Agency remains in flux. The Evaluation Policy was never really implementable, for example because it seems nobody actually considered who would do the evaluations. USAID staff generally lack the time and/or expertise to conduct these evaluations, and the usual implementing partners suffer from a material conflict of interest – very often, they would have to evaluate programs and projects implemented by their competitors…even projects where they had lost the bid to a competitor. Further, the organizations I have seen/interacted with that focus on evaluation remain preoccupied with quantitative approaches to evaluation that, while perhaps drawing on Shah’s interest in the now-fading RCT craze in development, really cannot identify or measure the sorts of causal processes that connect development interventions and outcomes. Finally, despite the nice words to the contrary, the culture at USAID remains intolerant of project failure, and the leadership of the Agency never mounted the strong defense of this culture change to the White House or Congress needed to create the space for a new understanding of evaluation, nor did it ever really convey a message of culture change that the staff of USAID found convincing across the board. There are some groups/offices at USAID (for example, in the ever-growing Global Development Lab) where this culture is fully in bloom, but these are small offices with small budgets. Most everyone else remains mired in very old thinking on evaluation.

At least from an incrementalist perspective, entrenching and building on these aspects of USAID Forward would be a major accomplishment for Shah’s successor. Whoever comes next will not simply run out the clock of the Obama Administration – there are two years left. I therefore expect the administration to appoint an administrator (rather than promote a career USAID staff caretaker with no political mandate) to the position. In a perfect world, this would be a person who understands development as a discipline, but also has the government and implementing experience to understand how development thought intersects with development practice in the real world. Someone with a real understanding of development and humanitarian assistance as a body of thought and practice with a long history that can be learned from and built upon would be able to parse the critical parts of USAID Forward from the fluff, could prevent the design and implementation of projects that merely repeat the efforts (and often failures) of decades ago, and could perhaps reverse the disturbing trend at USAID to view development challenges as technical challenges akin to those informed by X-Prizes – a trend that has shoved the social aspects of development to the back seat at the Agency. At the same time, someone with implementing and government experience would understand what is possible within the current structure, thus understanding where incremental victories might push the Agency in important and productive directions that move toward the achievement of more ideal, long-term goals

There are very, very few people out there who meet these criteria. Steve Radelet does, and he served as the Chief Economist at USAID while I was there, but I have no idea if he is interested or, more importantly, if anyone is interested in him. Much the pity if not. More likely, the administration is going to go with the relatively new Deputy Administrator Alfonso Lenhardt. Looking at his background, he’s already been vetted by the Senate for his current position, has foreign service experience, time in various implementer-oriented positions, and he is well-positioned to avoid a long confirmation process as a former lobbyist and from his time as House Sergeant-at-Arms, which likely give him deep networks on both sides of the aisle. In his background, I see no evidence of a long engagement with development as a discipline, and I wonder how reform-minded a former Senior Vice President for Government Relations at an implementer can be. I do not know Deputy Administrator Lenhardt at all, and so I cannot speak to where he might fall on any or all of the issues above. According to Devex, he says his goal is to “improve management processes and institutionalize the reforms and initiatives that Shah’s administration has put in place.” I have no objection to either of these goals – they are both important. But what this means in practice, should Lenhardt be promoted, is an open question that will have great impact on the future direction of the Agency.

Five and half years ago, at the end of the spring semester of 2009, I sat down and over the course of 30 days drafted my book Delivering Development. The book was, for me, many things: an effort to impose a sort of narrative on the work I’d been doing for 12 years in Ghana and other parts of Africa; an effort to escape the increasingly claustrophobic confines of academic writing and debates; and an effort to exorcise the growing frustration and isolation I felt as an academic working on international development in a changing climate, but without a meaningful network into any development donors. Most importantly, however, it was a 90,000 word scream at the field that could be summarized in three sentences:

  1. Most of the time, we have no idea what the global poor are doing or why they are doing it.
  2. Because of this, most of our projects are designed for what we think is going on, which rarely aligns with reality
  3. This is why so many development projects fail, and if we keep doing this, the consequences will get dire

The book had a generous reception, received very fair (if sometimes a bit harsh) reviews, and actually sold a decent number of copies (at least by the standards of the modern publishing industry, which was in full collapse by the time the book appeared in January 2011). Maybe most gratifying, I heard from a lot of people who read the book and who heard the message, or for whom the book articulated concerns they had felt in their jobs.

This is not to say the book is without flaws. For example, the second half of the book, the part addressing the implications of being wrong about the global poor, was weaker than the first – and this is very clear to me now, as the former employee of a development donor. Were I writing the book now, I would do practically nothing to the first half, but I would revise several parts of the second half (and the very dated scenarios chapter really needs revision at this point, anyway). But, five and a half years after I drafted it, I can still say one thing clearly.

I WAS RIGHT.

Well, I was right about point #1 above, anyway. The newest World Development Report from the World Bank has empirically demonstrated what was so clear to me and many others, and what I think I did a very nice job of illustrating in Delivering Development: most people engaged in the modern development industry have very little understanding of the lives and thought processes of the global poor, the very people that industry is meant to serve. Chapter 10 is perfectly titled: “The biases of development professionals.” All credit to the authors of the report for finally turning the analytic lens on development itself, as it would have been all too easy to simply talk about the global poor through the lens of perception and bias. And when the report turns to development professionals’ perceptions…for the love of God. Just look at the findings on page 188. No, wait, let me show you some here:

Screen Shot 2014-12-21 at 10.05.06 PM

 

For those who are chart-challenged, let me walk you through this. In three settings, the survey asked development professionals what percentage of their beneficiaries thought “what happens in the future depends on me.” For the bottom third, the professionals assumed very few people would say this. Except that a huge number of very poor people said this, in all settings. In short, the development professionals were totally wrong about what these people thought, which means they don’t understand their mindsets, motivations, etc. Holy crap, folks. This isn’t a near miss. This is I-have-no-idea-what-I-am-talking-about stuff here. These are the error bars on the initial ideas that lead to projects and programs at development donors.

WDR’s frames these findings in pretty stark terms (page 180):

Perhaps the most pressing concern is whether development professionals understand the circumstances in which the beneficiaries of their policies actually live and the beliefs and attitudes that shape their lives.

And their proposed solution is equally pointed (page 190):

For project and program design, development professionals should “eat their own dog food”: that is, they should try to experience firsthand the programs and projects they design.

Yes. Or failing that, they should really start either reading the work of people who can provide that experience for them, or start funding the people who can generate the data that allows for this experience (metaphorically).

On one hand, I am thrilled to see this point in mainstream development conversation. On the other…I said this five years ago, and not that many people cared. Now the World Bank says it…or maybe more to the point, the World Bank says it in terms of behavioral economics, and everyone gets excited. Well, my feelings on this are pretty clear:

  1. Just putting this in terms of behavioral economics is actually putting the argument out there in the least threatening manner possible, as it is still an argument from economics that preserves that disciplinary perspective’s position of superiority in development
  2. The things that behavioral economics have been “discovering” about the global poor that anthropology, geography, sociology, and social history have been saying for decades. Further, their analyses generally lack explanatory rigor or anything resembling external validity – see my posts here, here, and here.

Also, the WDR never makes a case for why we should care that we are probably misunderstanding/ misrepresenting the global poor. As a result, this just reads as an extended “oopsie!” piece that needs not be seriously addressed as long as we look a little sheepish – then we can get back to work. But getting this stuff wrong is really, really important – this was the central point of the second half of Delivering Development (a point that Duncan Green unfortunately missed in his review). We can design projects that not only fail to make things better, we can actually make things much worse: we can kill people by accident. We can gum up the global environment, which is not going to only hurt some distant, abstract global poor person – it will hit those in the richest countries, too. We can screw up the global economy, another entity that knows few borders and over which nobody has complete control. This is not “oopsie!” This is a disaster that requires serious attention and redress.

So, good first step World Bank, but not far enough. Delivering Development still goes a lot further than you are willing to now. Delivering Development goes much further than behavioral development economics has gone, or really can go. Time to catch up to the real nature of this problem, and the real challenges it presents. Time to catch up to things I was writing five years ago, before it’s too late.

First up on my week up update posts is a re-introduction to my reworked livelihoods approach. As some of you might remember, the formal academic publication laying out the theoretical basis for this approach came out in early 2013. This approach presented in the article is the conceptual foundation for much of the work we are doing in my lab. This pub is now up on my home page, via the link above or through a link on the publications page.

The premise behind this approach, and why I developed it in the first place, is simple. Most livelihoods approaches implicitly assume that the primary motivation for livelihoods decisions is the maximization of some sort of material return on that activity. Unfortunately, in almost all cases this is a massive oversimplification of livelihoods decision-making processes, and in many cases is fundamentally incorrect. Think about the number of livelihoods studies where there are many decisions or behaviors that seem illogical when held up to the logic of material maximization (which would be any good livelihoods study, really). We spend a lot of time trying to explain these decisions away (idiosyncrasy, incomplete information, etc.). But this makes no sense – if you are living on $1.25 a day, and you are illogical or otherwise making decisions against interest, you are likely dead. So there must be a logic behind these decisions, one that we must engage if we are to understand why people do what they do, and if we are to design and implement development interventions that are relevant to the needs of the global poor. My livelihoods approach provides a means of engaging with and explaining these behaviors built on explicit, testable framings of decision-making, locally-appropriate divisions of the population into relevant groupings (i.e. gender, age, class), and the consideration of factors from the local to the global scale.

The article is a straight-ahead academic piece – to be frank, the first half of the article is not that accessible to those without backgrounds in social theory and livelihoods studies. However, the second half of the article is a case study that lays out what the approach allows the user to see and explain, which should be of interest to most everyone who works with livelihoods approaches.

For those who would like a short primer on the approach and what it means in relatively plain English, I’ve put up a “top-line messages” document on the preprints page of my website.

Coming soon is an implementation piece that guides the user through the actual use of the approach. I field-tested the approach in Kaffrine, Senegal with one of my graduate students from May-July 2013. I am about to put the approach to work in a project with the Red Cross in the Zambezi Basin in Zambia next month. In short, this is not just a theoretical pipe dream – it is a real approach that works. In fact, the reason we are working with Red Cross is because Pablo Suarez of Boston University and the Red Cross Climate Centre read the academic piece and immediately grasped what it could do, and then reached out to me to bring me into one of their projects. The implementation piece is already fully drafted, but I am circulating it to a few people in the field to get feedback before I submit it for review or post it to the preprints page. I am hoping to have this up by the end of January.  Once that is out the door, I will look into building a toolkit for those who might be interested in using the approach.

I’m really excited by this approach, and the things that are emerging from it in different places (Mali, Zambia, and Senegal, at the moment). I would love feedback on the concept or its use – I’m not a defensive or possessive person when it comes to ideas, as I think debate and critique tend to make things stronger. The reason I am developing a new livelihoods approach is because the ones we have simply don’t explain the things we need to know, and the other tools of development research that dominate the field at the moment (i.e. RCTs) cannot address the complex, integrative questions that drive outcomes at the community level. So consider all of this a first draft, one that you can help bring to final polished form!

I have a confession. For a long time now I have found myself befuddled by those who claim to have identified the causes behind observed outcomes in social research via the quantitative analysis of (relatively) large datasets (see posts here, here, and here).  For a while, I thought I was seeing the all-to-common confusion of correlation and causation…except that a lot of smart, talented people seemed to be confusing correlation with causation.  This struck me as unlikely.

Then, the other day in seminar (I was covering for a colleague in our department’s “Contemporary Approaches to Geography” graduate seminar, discussing the long history of environmental determinism within and beyond the discipline), I found myself in a similar discussion related to explanation…and I think I figured out what has been going on.  The remote sensing and GIS students in the course, all of whom are extraordinarily well-trained in quantitative methods, got to thinking about how to determine if, in fact, the environment was “causing” a particular behavior*. In the course of this discussion, I realized that what they meant by “cause” was simple (I will now oversimplify): when you can rule out/control for the influence of all other possible factors, you can say that factor X caused event Y to happen.  Indeed, this does establish a causal link.  So, I finally get what everyone was saying when they said that, via well-constructed regressions, etc., one can establish causality.

So it turns out I was wrong…sort of. You see, I wasn’t really worried about causality…I was worried about explanation. My point was that the information you would get from a quantitative exercise designed to establish causal relationships isn’t enough to support rigorous project and program design. Just because you know that the construction of a borehole in a village caused girl-child school attendance to increase in that village doesn’t mean you know HOW the borehole caused this change in school attendance to happen.  If you cannot rigorously explain this relationship, you don’t understand the mechanism by which the borehole caused the change in attendance, and therefore you don’t really understand the relationship. In the “more pure” biophysical sciences**, this isn’t that much of a problem because there are known rules that particles, molecules, compounds, and energy obey, and therefore under controlled conditions one can often infer from the set of possible actors and actions defined by these rules what the causal mechanism is.

But when we study people it is never that simple.  The very act of observing people’s behaviors causes shifts in that behavior, making observation at best a partial account of events. Interview data are limited by the willingness of the interviewee to talk, and the appropriateness of the questions being asked – many times I’ve had to return to an interviewee to ask a question that became evident later, and said “why didn’t you tell me this before?”  (to which they answer, quite rightly, with something to the effect of “you didn’t ask”).  The causes of observed human behavior are staggeringly complex when we get down to the real scales at which decisions are made – the community, household/family, and individual. Decisions may vary by time of the year, or time of day, and by the combination of gender, age, ethnicity, religion, and any other social markers that the group/individual chooses to mobilize at that time.  In short, just because we see borehole construction cause increases in girl-child school attendance over and over in several places, or even the same place, doesn’t mean that the explanatory mechanism between the borehole and attendance is the same at all times.

Understanding that X caused Y is lovely, but in development it is only a small fraction of the battle.  Without understanding how access to a new borehole resulted in increased girl-child school attendance, we cannot scale up borehole construction in the context of education programming and expect to see the same results.  Further, if we do such a scale-up, and don’t get the same results, we won’t have any idea why.  So there is causality (X caused Y to happen) and there are causal mechanisms (X caused Y to happen via Z – where Z is likely a complex, locally/temporally specific alignment of factors).

Unfortunately, when I look at much quantitative development research, especially in development economics, I see a lot of causality, but very little work on causal mechanisms that get us to explanation.  There is a lot of story time, “that pivot from the quantitative finding to the speculative explanation.”  In short, we might be programming development and aid dollars based upon evidence, but much of the time that evidence only gets us part of the way to what we really need to know to really inform program and project design.

This problem is avoidable –it does not represent the limits of our ability to understand the world. There is one obvious way to get at those mechanisms – serious, qualitative fieldwork.  We need to be building research and policy teams where ethnographers and other qualitative social scientists learn to respect the methods and findings of their quantitative brethren such that they can target qualitative methods at illuminating the mechanisms driving robust causal relationships. At the same time, the quantitative researchers on these teams will have to accept that they have only partially explained what we need to know when they have established causality through their methods, and that qualitative research can carry their findings into the realm of implementation.

The bad news for everyone…for this to happen, you are going to have to pick your heads up out of your (sub)disciplinary foxholes and start reading across disciplines in your area of interest.  Everyone talks a good game about this, but when you read what keeps getting published, it is clear that cross-reading is not happening.  Seriously, the number of times I have seen people in one field touting their “new discoveries” about human behavior that are already common conversation in other disciplines is embarrassing…or at least it should be to the authors. But right now there is no shame in this sort of thing, because most folks (including peer reviewers) don’t read outside their disciplines, and therefore have no idea how absurd these claims of discovery really are. As a result, development studies gives away its natural interdisciplinary advantage and returns to the problematic structure of academic knowledge and incentives, which not only enable, but indeed promote narrowly disciplinary reading and writing.

Development donors, I need a favor. I need you to put a little research money on the table to learn about whatever it is you want to learn about. But when you do, I want you to demand it be published in a multidisciplinary development-focused journal.  In fact, please start doing this for all of your research-related money. People will still pursue your money, as the shrinking pool of research dollars is driving academia into your arms. Administrators like grant and contract money, and so many academics are now being rewarded for bringing in grants and contracts from non-traditional sources (this is your carrot). Because you hold the carrot, you can draw people in and then use “the stick” inherent in the terms of the grant/contract to demand cross-disciplinary publishing that might start to leverage change in academia. You all hold the purse, so you can call the tune…

 

 

 

*Spoiler alert: you can’t.  Well, you probably can if 1) you pin the behavior you want to explain down to something extraordinarily narrow, 2) can limit the environmental effect in question to a single independent biophysical process (good luck with that), and 3) limit your effort to a few people in a single place. But at that point, the whole reason for understanding the environmental determinant of that behavior starts to go out the window, as it would clearly not be generalizable beyond the study. Trust me, geography has been beating its head against this particular wall for a century or more, and we’ve buried the idea.  Learn from our mistakes.

 

**by “more pure” I am thinking about those branches of physics, chemistry, and biology in which lab conditions can control for many factors. As soon as you get into field sciences, or starting asking bigger questions, complexity sets in and things like causality get muddied in the manner I discuss below…just ask an ecologist.

While behavioral economics continues to open old questions in development to new scrutiny, I am still having a lot of problems with the very unreflexive approach BE takes toward its own work (see earlier takes on this here and here).  Take, for example, Esther Duflo’s recent lectures discussing mistakes the poor make.  To discuss the mistakes the poor make, we must first understand what the goals of the poor are.  However, I simply don’t see the behavioral economists doing this.  There is still a lurking, underlying presumption that in making livelihoods decisions people are trying to maximize income and or the material quality of their lives.  This, however, is fundamentally incorrect.  In Delivering Development and a number of related publications (for example, here, here, and here) I have laid out how, in the context of livelihoods, material considerations are always bound up in social considerations.  If you only evaluate these actions as aimed at material goals, you’ve only got a part of the picture – and not the most important part, in most cases.  Instead, what you are left with are a bunch of decisions and outcomes that appear illogical, that can be cast as mistakes.  Only most of the time, they are not mistakes – they are conscious choices.

Let me offer an example from Delivering Development and some of my other work – the constraint of women’s farming by their husbands.  I have really compelling qualitative evidence from two villages in Ghana’s Central Region that demonstrates that men are constraining their wives’ farm production to the detriment of the overall household income.  The chart below shows a plot of the size of a given farm versus its market orientation for the households operating under what I call a “diversified” strategy – where the husband farms for market sale, and the wife for subsistence (a pretty common model in sub-Saharan Africa).  As you move up the Y axis, the farm gets more oriented toward market sale (1 on that scale is “eat everything”, 3 is sell and eat equally, and 5 is sell everything).  Unsurprisingly, since men’s role requires them to produce for market, the size of their farm has little impact on their orientation.  But look at the women’s farms – just a tenth of a hectare produces a marked shift in orientation from subsistence to market production…because women own that surplus beyond subsistence, and sell it.  They take the proceeds of these sales, buy small goods, and engage in petty trading, eventually multiplying that small surplus into significant gains in income, nearly equaling their husbands.  What is not to like?

Well, from the perspective of those in these villages, here is something: among the Akan, being a “good man” means being in control of the household and out-earning your wife.  If you don’t, your fitness as a man gets called into question, which can cost you access to land.  For wives, this is bad because they get their land through their husbands.  So as a result, being in a household where the woman out-earns her husband is not a viable livelihoods outcome (as far as members of these households are concerned).  Even if a man wanted to let his wife earn more money, he would do so at peril of his access to land. So he is not going to do that.  What he is going to do is shrink his wife’s farm the next season to ensure she does not out-earn him (and I have three years of data where this is exactly what happens to wives who earn too much).  There is a “mistake” here – some of these men underestimated their wives’ production, which is pretty easy to do under rain-fed agriculture in a changing climate.  That they are this accurate with regard to land allocation is rather remarkable, really.  But the decision to constrain women’s production is not a mistake, per se: it is a choice.

We can agree or disagree with the premises of these choices, and their outcomes, but labeling them as mistakes creates a false sense of simplicity in addressing problematic outcomes – because people only require “correction” to get to the outcomes we all want and need.  This, in turn, rests on/reproduces a sense of superiority on the part of the researcher – because s/he knows what is best (see a previous post on this point here).  That attitude, applied to the case above, would not result in a productive project design aimed at addressing income or other challenges in these villages.

Yes, people do things against material interest…but there is always a logic behind a decision, and that logic is often deeply entrenched.  We would be better off talking about decisions poor people make (for better or worse), and dedicating our time to understanding why they make these decisions before we start deciding who is mistaken, and what to do about it.

I’ve just burned 15,000 words in Third World Quarterly laying out my argument for how to think about livelihoods as more than material outcomes – and how to make that vision implementable, at least via fieldwork that runs in length from days to months.  I am happy to send a copy of the preprint to anyone who is interested –and I will post a version to my website shortly.

Alright, last post I laid out an institutional problem with M&E in development – the conflict of interest between achieving results to protect one’s budget and staff, and the need to learn why things do/do not work to improve our effectiveness.  This post takes on a problem in the second part of that equation – assuming we all agree that we need to know why things do/do not work, how do we go about doing it?

As long-time readers of this blog (a small, but dedicated, fanbase) know, I have some issues with over-focusing on quantitative data and approaches for M&E.  I’ve made this clear in various reactions to the RCT craze (see herehere, here and here). Because I framed my reactions in terms of RCTs, I think some folks think I have an “RCT issue.”  In fact, I have a wider concern – the emerging aggressive push for quantifiable data above all else as new, more rigorous implementation policies come into effect.  The RCT is a manifestation of this push, but really is a reflection of a current fad in the wider field.  My concern is that the quantification of results, while valuable in certain ways, cannot get us to causation – it gets us to really, really rigorously established correlations between intervention and effect in a particular place and time (thoughtful users of RCTs know this).  This alone is not generalizable – we need to know how and why that result occurred in that place, to understand the underlying processes that might make that result replicable (or not) in the future, or under different conditions.

As of right now, the M&E world is not doing a very good job of identifying how and why things happen.  What tends to happen after rigorous correlation is established is what a number of economists call “story time”, where explanation (as opposed to analysis) suddenly goes completely non-rigorous, with researchers “supposing” that the measured result was caused by social/political/cultural factor X or Y, without any follow on research to figure out if in fact X or Y even makes sense in that context, let alone whether or not X or Y actually was causal.  This is where I fear various institutional pushes for rigorous evaluation might fall down.  Simply put, you can measure impact quantitatively – no doubt about it.  But you will not be able to rigorously say why that impact occurred unless someone gets in there and gets seriously qualitative and experiential, working with the community/household/what have you to understand the processes by which the measured outcome occurred.  Without understanding these processes, we won’t have learned what makes these projects and programs scalable (or what prevents them from being scaled) – all we will know is that it worked/did not work in a particular place at a particular time.

So, we don’t need to get rid of quantitative evaluation.  We just need to build a strong complementary set of qualitative tools to help interpret that quantitative data.  So the next question to you, my readers: how are we going to build in the space, time, and funding for this sort of complementary work? I find most development institutions to be very skeptical as soon as you say the words qualitative…mostly because it sounds “too much like research” and not enough like implementation. Any ideas on how to overcome this perception gap?

(One interesting opportunity exists in climate change – a lot of pilot projects are currently piloting new M&E approaches, as evaluating impacts of climate change programming requires very long-term horizons.  In at least one M&E effort I know of, there is talk of running both quantitative and qualitative project evaluations to see what each method can and cannot answer, and how they might fit together.  Such a demonstration might catalyze further efforts…but this outcome is years away)

So, how do we fix the way we think about development to address the challenges of global environmental change?  Well, there are myriad answers, but in this post I propose two – we have to find ways of evaluating the impact of our current projects such that those lessons are applicable to other projects that are implemented in different places and at various points in the future . . . and we have to better evaluate just where things will be in the future as we think about the desired outcomes of development interventions.

To achieve the first of these two is relatively easy, at least conceptually: we need to fully link up the RCT4D crowd with the qualitative research/social theory crowd.  We need teams of people that can bring the randomista obsession with sampling frames and serious statistical tools – in other words, a deep appreciation for rigor in data collection – and connect it to the qualitative social theoretical emphasis on understanding causality by interrogating underlying social process – in other words, a deep appreciation for rigor in data interpretation.  Such teams work to cover the weaknesses of their members, and could bring us new and very exciting insights into development interventions and social process.

Of course, everyone says we need mixed methodologies in development (and a lot of other fields of inquiry), but we rarely see projects that take this on in a serious way.  In part, this is because very few people are trained in mixed methods – they are either very good at qualitative methods and interpretation, or very good at sampling and quantitative data analysis.  Typically, when a team gets together with these different skills, one set of skills or the other predominates (in policy circles, quant wins every time).  To see truly mixed methodologies, this cannot happen – as soon as one trumps the other, the value of the mixing declines precipitously.

For example, you need qualitative researchers to frame the initial RCT – an RCT framed around implicit, unacknowledged assumptions about society is unlikely to “work” – or to capture the various ways in which an intervention works.  At the same time, the randomista skill of setting up a sampling frame and obtaining meaningful large-scale data sets requires attention to how one frames the question, and where the RCT is to be run . . . which impose important constraints on the otherwise unfettered framings of social process coming from the qualitative side, framings that might not really be testable in a manner that can be widely understood by the policy community.  Then you need to loop back to the qualitative folks to interpret the results of the initial RCT – to move past whether or not something worked to the consideration of the various ways in which it did and did not work, and a careful consideration of WHY it worked.  Finally, these interpretations can be framed and tested by the qualitative members of the team, starting an iterative interpretive process that blends qualitative and quantitative analysis and interpretation to rigorously deepen our understanding of how development works (or does not work).

The process I have just described will require teams of grownups with enough self-confidence to accept criticism and to revise their ideas and interpretations in the face of evidence of varying sorts.  As soon as one side of this mixed method team starts denigrating the other, or the concerns of one side start trumping those of the other, the value of this mixing drops off – qualitative team members become fig leaves for “story time” analyses, or quantitative researchers become fig leaves for weak sampling strategies or overreaching interpretations of the data.  This can be done, but it will require team leaders with special skill sets – with experience in both worlds, and respect for both types of research.  There are not many of these around, but they are around.

Where are these people now?  Well, interestingly the answer to this question leads me to the second answer for how development might better answer the challenges of global environmental change: development needs to better link itself with the global environmental change community.  Despite titles that might suggest otherwise (UNEP’s Fourth Global Environment Outlook was titled Environment for Development), there is relatively little interplay between these communities right now.  Sure, development folks say the right things about sustainability and climate change these days, but they are rarely engaging the community that has been addressing these and many other challenges for decades.  At the same time, the global environmental change community has a weak connection to development, making their claims about the future human impacts of things like climate change often wildly inaccurate, as they assume current conditions will persist into the future (or they assume equally unrealistic improvements in future human conditions).

Development needs to hang out with the scenario builders of the global environmental change community to better understand the world we are trying to influence twenty years hence – the spot to which we are delivering the pass, to take up a metaphor from an earlier post on this topic.  We need to get with the biophysical scientists who can tell us about the challenges and opportunities the expect to see two or more decades hence.  And we need to find the various teams that are already integrating biophysical scientists and social scientists to address these challenges – the leaders already have to speak quant and qual, science and humanities, to succeed at their current jobs.  The members of these teams have already started to learn to respect their colleagues’ skills, and to better explain what they know to colleagues who may not come at the world with the same framings, data or interpretations.  They are not perfect, by any stretch (I voice some of my concerns in Delivering Development), but they are great models to go on.

Meanwhile, several of my colleagues and I are working on training a new generation of interdisciplinary scholars with this skill set.  All of my current Ph.D. students have taken courses in qualitative methods, and have conducted qualitative fieldwork . . . but they also have taken courses on statistics and biogeographic modeling.  They will not be statisticians or modelers, but now they know what those tools can and cannot do – and therefore how they can engage with them.  The first of this crew are finishing their degrees soon . . . the future is now.  And that gives me reason to be realistically optimistic about things . . .