So, how do we fix the way we think about development to address the challenges of global environmental change? Well, there are myriad answers, but in this post I propose two – we have to find ways of evaluating the impact of our current projects such that those lessons are applicable to other projects that are implemented in different places and at various points in the future . . . and we have to better evaluate just where things will be in the future as we think about the desired outcomes of development interventions.
To achieve the first of these two is relatively easy, at least conceptually: we need to fully link up the RCT4D crowd with the qualitative research/social theory crowd. We need teams of people that can bring the randomista obsession with sampling frames and serious statistical tools – in other words, a deep appreciation for rigor in data collection – and connect it to the qualitative social theoretical emphasis on understanding causality by interrogating underlying social process – in other words, a deep appreciation for rigor in data interpretation. Such teams work to cover the weaknesses of their members, and could bring us new and very exciting insights into development interventions and social process.
Of course, everyone says we need mixed methodologies in development (and a lot of other fields of inquiry), but we rarely see projects that take this on in a serious way. In part, this is because very few people are trained in mixed methods – they are either very good at qualitative methods and interpretation, or very good at sampling and quantitative data analysis. Typically, when a team gets together with these different skills, one set of skills or the other predominates (in policy circles, quant wins every time). To see truly mixed methodologies, this cannot happen – as soon as one trumps the other, the value of the mixing declines precipitously.
For example, you need qualitative researchers to frame the initial RCT – an RCT framed around implicit, unacknowledged assumptions about society is unlikely to “work” – or to capture the various ways in which an intervention works. At the same time, the randomista skill of setting up a sampling frame and obtaining meaningful large-scale data sets requires attention to how one frames the question, and where the RCT is to be run . . . which impose important constraints on the otherwise unfettered framings of social process coming from the qualitative side, framings that might not really be testable in a manner that can be widely understood by the policy community. Then you need to loop back to the qualitative folks to interpret the results of the initial RCT – to move past whether or not something worked to the consideration of the various ways in which it did and did not work, and a careful consideration of WHY it worked. Finally, these interpretations can be framed and tested by the qualitative members of the team, starting an iterative interpretive process that blends qualitative and quantitative analysis and interpretation to rigorously deepen our understanding of how development works (or does not work).
The process I have just described will require teams of grownups with enough self-confidence to accept criticism and to revise their ideas and interpretations in the face of evidence of varying sorts. As soon as one side of this mixed method team starts denigrating the other, or the concerns of one side start trumping those of the other, the value of this mixing drops off – qualitative team members become fig leaves for “story time” analyses, or quantitative researchers become fig leaves for weak sampling strategies or overreaching interpretations of the data. This can be done, but it will require team leaders with special skill sets – with experience in both worlds, and respect for both types of research. There are not many of these around, but they are around.
Where are these people now? Well, interestingly the answer to this question leads me to the second answer for how development might better answer the challenges of global environmental change: development needs to better link itself with the global environmental change community. Despite titles that might suggest otherwise (UNEP’s Fourth Global Environment Outlook was titled Environment for Development), there is relatively little interplay between these communities right now. Sure, development folks say the right things about sustainability and climate change these days, but they are rarely engaging the community that has been addressing these and many other challenges for decades. At the same time, the global environmental change community has a weak connection to development, making their claims about the future human impacts of things like climate change often wildly inaccurate, as they assume current conditions will persist into the future (or they assume equally unrealistic improvements in future human conditions).
Development needs to hang out with the scenario builders of the global environmental change community to better understand the world we are trying to influence twenty years hence – the spot to which we are delivering the pass, to take up a metaphor from an earlier post on this topic. We need to get with the biophysical scientists who can tell us about the challenges and opportunities the expect to see two or more decades hence. And we need to find the various teams that are already integrating biophysical scientists and social scientists to address these challenges – the leaders already have to speak quant and qual, science and humanities, to succeed at their current jobs. The members of these teams have already started to learn to respect their colleagues’ skills, and to better explain what they know to colleagues who may not come at the world with the same framings, data or interpretations. They are not perfect, by any stretch (I voice some of my concerns in Delivering Development), but they are great models to go on.
Meanwhile, several of my colleagues and I are working on training a new generation of interdisciplinary scholars with this skill set. All of my current Ph.D. students have taken courses in qualitative methods, and have conducted qualitative fieldwork . . . but they also have taken courses on statistics and biogeographic modeling. They will not be statisticians or modelers, but now they know what those tools can and cannot do – and therefore how they can engage with them. The first of this crew are finishing their degrees soon . . . the future is now. And that gives me reason to be realistically optimistic about things . . .
Tag: qualitative methods
Why should the aid/relief/development community care about global environmental change (Pt. 3)?
OK, ok, you say: I get it, global environmental change matters to development/aid/relief. But aside from thinking about project-specific intersections between the environment and development/aid/relief, what sort of overarching challenges does global environmental change pose to the development community? Simply put, I think that the inevitability of various forms of environmental change (a level of climate change cannot be stopped now, certain fisheries are probably beyond recovery, etc.) over the next 50 or so years forces the field of development to start thinking very differently about the design and evaluation of policies, programs, and projects . . . and this, in turn, calls into question the value of things like randomized control trials for development.
In aid/development we tend to be oriented to relatively short funding windows in which we are supposed to accomplish particular tasks (which we measure through output indicators, like the number of judges trained) that, ideally, change the world in some constructive manner (outcome indicators, like a better-functioning judicial system). Outputs are easier to deliver and measure than outcomes, and they tend to operate on much shorter timescales – which makes them perfect for end-of-project reporting even though they often bear little on the achievement of the desired outcomes that motivated the project in the first place (does training X judges actually result in a better functioning judicial system? What if the judges were not the problem?). While there is a serious push in the development community to move past outputs to outcomes (which I generally see as a very positive trend), I do not see a serious conversation about the different timescales on which these two sorts of indicators operate. Outputs are very short-term. Outcomes can take generations. Obviously this presents significant practical challenges to those who do development work, and must justify their expenditures on an annual basis.
This has tremendous implications, I think, for development practice in the here and now – especially in development research. For example, I think this pressure to move to outcomes but deliver them on the same timescale as outputs has contributed to the popularity of the randomized control trials for development (RCT4D) movement. RCT4D work gathers data in a very rigorous manner, and subjects it to interesting forms of quantitative analysis to determine the impact of a particular intervention on a particular population. As my colleague Marc Bellemare says, RCTs establish “whether something works, not how it works.”
The vast majority of RCT4D studies are conducted across a few months to years, directly after the project is implemented. Thus, the results seem to move past outputs to impacts without forcing everyone to wait a very long time to see how things played out. This, to me, is both a strength and a weakness of the approach . . . though I never hear anyone talking about it as a weakness. The RCT4D approach seems to suggest that the evaluation of project outcomes can be effectively done almost immediately, without need for long-term follow-up. This sense implicitly rests on the forms of interpretation and explanation that undergird the RCT4D approach – basically, what I see as an appallingly thin approach to the interpretation of otherwise interesting and rigorously gathered data. My sense of this interpretation is best captured by Andrew Gelman’s (quoting Fung) use of the term “story time”, which he defines as a “pivot from the quantitative finding to the speculative explanation.” It seems that many practitioners of RCT4D seem to think that story time is unavoidable . . . which to me reflects a deep ignorance of the concerns for rigor and validity that have existed in the qualitative research community for decades. Feel free to check the methods section of any of my empirically-based articles (i.e. here and here): they address who I interviewed, why I interviewed them, how I developed interview questions, and how I knew that my sample size had grown large enough to feel confident that it was representative of the various phenomena I was trying to understand. Toward the end of my most recent work in Ghana, I even ran focus groups where I offered my interpretations of what was going on back to various sets of community members, and worked with them to strengthen what I had right and correct what I had wrong. As a result, I have what I believe is a rigorous, highly nuanced understanding of the social causes of the livelihoods decisions and outcomes that I can measure in various ways, qualitative and quantitative, but I do not have a “story time” moment in there.
The point here is that “story time”, as a form of explanation, rests on uncritical assumptions about the motivations for human behavior that can make particular decisions or behaviors appear intelligible but leave the door open for significant misinterpretations of events on the ground. Further, the very framing of what “works” in the RCT4D approach is externally defined by the person doing the evaluation/designing the project, and is rarely revised in the face of field realities . . . principally because when a particular intervention does not achieve some externally-defined outcome, it is deemed “not to have worked.” That really tends to shut down continued exploration of alternative outcomes that “worked” in perhaps unpredictable ways for unexpected beneficiaries. In short, the RCT4D approach tends to reinforce the idea that development is really about delivering apolitical, technical interventions to people to address particular material needs.
The challenge global environmental change poses to the RCT4D randomista crowd is that of the “through ball” metaphor I raised in my previous post. Simply put, identifying “what works” without rigorously establishing why it worked is broadly useful if you make two pretty gigantic assumptions: First, you have to assume that the causal factors that led to something “working” are aspects of universal biophysical and social processes that are translatable across contexts. If this is not true, an RCT only gives you what works for a particular group of people in a particular place . . . which is not really that much more useful than just going and reading good qualitative ethnographies. If RCTs are nothing more than highly quantified case studies, they suffer from the same problem as ethnography – they are hard to aggregate into anything meaningful at a broader scale. And yes, there are really rigorous qualitative ethnographies out there . . .
Second, you have to assume that the current context of the trial is going to hold pretty much constant going forward. Except, of course, global environmental change more or less chucks that idea for the entire planet. In part, this is because global environmental change portends large, inevitable biophysical changes in the world. Just because something works for improving rain-fed agricultural outputs today does not mean that the same intervention will work when the enabling environmental conditions, such as rainfall and temperature, change over the next few decades. More importantly, though, these biophysical changes will play out in particular social contexts to create particular impacts on populations, who will in turn develop efforts to address those impacts. Simply put, when we introduce a new crop today and it is taken up and boosts yields, we know that it “worked” by the usual standards of agricultural development and extension. But the take-up of new crops is not a function of agricultural ecology – there are many things that will grow in many places, but various social factors ranging from the historical (what crops were introduced via colonialism) to gender (who grows what crops and why) are what lead to particular farm compositions. For example, while tree crops (oil palm, coconut, various citrus, acacia for charcoal) are common on farms around the villages in which I have worked in Ghana, almost none of these trees are found on women’s farms. The reasons for this are complex, and link land tenure, gender roles, and household power relations into livelihoods strategies that balance material needs with social imperatives (for extended discussions, see here and here, or read my book).
Unless we know why that crop was taken up, we cannot understand if the conditions of success now will exist in the future . . . we cannot tell if what we are doing will have a durable impact. Thus, under the most reliable current scenario for climate change in my Ghanaian research context, we might expect the gradual decline in annual precipitation, and the loss of the minor rainy season, to make tree crops (which tend to be quite resilient in the face of fluctuating precipitation) more and more attractive. However, tree crops challenge the local communal land tenure system by taking land out of clan-level recirculation, and allowing women to plant them would further challenge land tenure by granting them direct control over access to land (which they currently lack). Altering the land tenure system would, without question, set off a cascade of unpredictable social changes that would be seen in everything from gender roles to the composition of farms. There is no way to be sure that any development intervention that is appropriate to the current context will be even functional in that future context. Yet any intervention we put into place today should be helping to catalyze long-term changes . . .
Simply put: Global environmental change makes clear the limitations of our current thinking on aid/development (of which RCT4D is merely symptomatic). Just like RCTs, our general framing of development does not move us any closer to understanding the long-term impact of our interventions. Further, the results of RCTs are not generalizable past the local context (which most good randomistas already know), limiting their ability to help us transform how we do development. In a world of global environmental change, our current approaches to development just replicate our existing challenges: they don’t really tell us if what we are doing will be of any lasting benefit, or even teach us general lessons about how to deliver short-term benefits in a rigorous manner.
Next up: The Final Chapter – Fixing It
Stories, development and adaptation
Mike Hulme has an article in the July issue of Nature Climate Change titled “Meet the humanities,”[paywalled] in which he argues that “An introduction needs to be made between the rich cultural knowledge of social studies and the natural sciences.” Overall, I like this article – Hulme understands the social science side of things, not least through his own research and his work as editor of Global Environmental Change, one of the most influential journals on the human dimensions of global change*. Critically, he lays out how, even under current efforts to include a wider range of disciplines in major climate assessments, the conversation has been dominated for so long by the biophysical sciences and economics that it is difficult for other voices to break in:
policy discussions have become “improving climate predictions” and “creating new economic policy instruments”; not “learning from the myths of indigenous cultures” or “re-thinking the value of consumption.”
Hulme is not arguing that we are wrong to be trying to improve climate predictions or develop new economic policy instruments – instead, he is subtly asking if these are the right tools for the job of addressing climate change and its impacts. My entire research agenda is one of unearthing a greater understanding of why people do what they do to make a living, how they decide what to do when their circumstances change, and what the outcomes of those decisions are for their long-term well being. Like Hulme, I am persistently surprised at the relative dearth of work on this subject – especially because the longer I work on issues of adaptation and livelihoods, the more impressed I am with the capacity of communities to adjust to new circumstances, and the less impressed I am with anyone’s ability to predictably (and productively) intervene in these adjustments.
This point gets me to my motivation for this post. Hulme could not cover everything in his short commentary, but I felt it important to identify where a qualitative social science perspective can make an immediate impact on how we think about adaptation (which really is about how we think about development, I think). I remain amazed that so many working in development fail to grasp that there is no such things as a completely apolitical, purely technical intervention. For example, in development we all too often assume that a well is just a well – that it is a technical intervention that delivers water to people. However, a well is highly political – it reshapes some people’s lives, alters labor regimes, could empower women (or be used as an excuse to extract more of their labor on farms, etc.) – all of this is contextual, and has everything to do with social relations and social power. So, we can introduce the technology of a well . . . but the idea and meaning of a well cannot be introduced in the same manner – these are produced locally, through local lenses. It is this basic failure of understanding that lies at the heart of so many failed development projects that passed technical review and various compliance reviews: they were envisioned as neutral and technical, and were probably very well designed in those arenas. However, these project designers gave little concern to the contextual, local social processes that would shape the use and outcomes of the intervention, and the result was lots of “surprise” outcomes.
When we start to approach these issues from a qualitative social scientific standpoint, or even a humanities standpoint (Hulme conflates these in his piece, I have no idea why. They are not the same), the inherent politics of development become inescapable. This was the point behind my article “The place of stories in development: creating spaces for participation through narrative analysis.” In that article, I introduce the story I used to open Delivering Development to illustrate how our lived experience of development often plays out in ways best understood as narratives, “efforts to present information as a sequence of connected events with some sort of structural coherence, transforming ‘the real into an object of desire through a formal coherence and a moral order that the real.” These narratives emerge in the stories we are told and that we overhear in the course of our fieldwork, but rarely make it into our articles or reports (though they do show up on a few fantastic aid blogs, like Shotgun Shack and Tales from the Hood). They become local color to personal stories, not sources of information that reveal the politics of our development efforts (though read the two aforementioned blogs for serious counterpoints).
In my article, I demonstrated how using the concept of narrative, drawn from the humanities, has allowed me to identify moments in which I am placed into a plot, a story of development and experience not of my making:
In this narrative [“the white man is so clever,” a phrase I heard a lot during fieldwork], I was cast as the expert, one who had knowledge and resources that could improve their lives if only I would share it with them. [The community] cast themselves in the role of recipients of this knowledge, but not participants in its formation. This narrative has been noted time and again in development studies (and post-colonial studies), and in the era of participation we are all trained to subvert it when we see it emerge in the work of development agencies, governments, and NGOs. However, we are less trained to look for its construction by those living in the Global South. In short, we are not trained to look for the ways in which others emplot us.
The idea of narrative is useful not only for identifying when weird neocolonial moments crop up, but also for destabilizing those narratives – what I call co-authoring. For example, when I returned to the site of my dissertation fieldwork a few years later, I found that my new position as a (very junior) professor created a new set of problems:
This new identity greatly hindered my first efforts at fieldwork after taking this job, as several farmers openly expected me to tell them what to plant and how to plant it. I was able to decentre this narrative when, after one farmer suggested that I should be telling him what to plant instead of asking him about his practices, I asked him ‘Do I look like a farmer?’ He paused, admitted that I did not, and then started laughing. This intervention did not completely deconstruct his narrative of white/developed and black/developing, or my emplotment in that narrative. I was still an expert, just not about farming. This created a space for him to speak freely to me about agriculture in the community, while still maintaining a belief in me as the expert.
Certainly, this is not a perfect outcome. But this is a lot better than the relationship that would have developed without an awareness of this emerging narrative, and my efforts to co-author that narrative. Long and short, the humanities have a lot to offer both studies of climate change impacts and development – if we can bring ourselves to start taking things like stories seriously as sources of data. As Hulme notes, this is not going to be an easy thing to do – there is a lot of inertia in both development and climate change studies. But changes are coming, and I for one plan to leverage them to improve our understandings of what is happening in the world as a result of our development efforts, climate change, global markets, and any number of other factors that impact life along globalization’s shoreline – and to help co-author different, and hopefully better, outcomes than what has come before.
*full disclosure: I’ve published in Global Environmental Change, and Hulme was one of the editors in charge of my article.
Wait, economists get to just guess and call it explanation?
An interesting review of Paul Collier’s The Bottom Billion and Wars, Guns and Votes by Yale Anthropologist Mike McGovern has gotten a little bit of attention recently in development circles, speaking as it does to ongoing debates about the role of statistical analysis, what counts as explanation, and where qualitative research fits into all of this. I will take up McGovern’s good (but incomplete, in my opinion) review in another post. Here, I needed to respond to a blog entry about this review.
On the Descriptive Statistics, Causal Inference and Social Science blog, Andrew Gelman discusses McGovern’s review. While there is a lot going on in this post, one issue caught my attention in particular. In his review, McGovern argues that “Much of the intellectual heavy lifting in these books is in fact done at the level of implication or commonsense guessing,” what Gelman (quoting Fung) calls “story time”, the “pivot from the quantitative finding to the speculative explanation.” However, despite the seemingly dismissive term for this sort of explanation, in his blog post Gelman argues “story time can’t be avoided.” His point:
On one hand, there are real questions to be answered and real decisions to be made in development economics (and elsewhere), and researchers and policymakers can’t simply sit still and say they can’t do anything because the data aren’t fully persuasive. (Remember the first principle of decision analysis: Not making a decision is itself a decision.)
From the other direction, once you have an interesting quantitative finding,of course you want to understand it, and it makes sense to use all your storytelling skills here. The challenge is to go back and forth between the storytelling and the data. You find some interesting result (perhaps an observational data summary, perhaps an analysis of an experiment or natural experiment), this motivates a story, which in turn suggests some new hypotheses to be studied.
Now, on one hand I take his point – research is iterative, and answering one set of questions (or one set of new data) often raises new questions which can be interrogated. But Gelman seems to presume that explanation only comes from more statistical analysis, without considering what I saw as McGovern’s subtle point: qualitative social scientists look at explanation, and do not revert to story time to do so (good luck getting published if you do). We spend a hell of a lot of time fleshing out the causal processes behind our observations, including establishing rigor and validity for our data and conclusions, before we present stories. This is not to say that our explanations are immediately complete or perfect, nor is it to suggest that our explanations do not raise new questions to pursue. However, there is no excuse for the sort of “story time” analysis that McGovern is pointing out in Collier’s work – indeed, I would suggest that is why the practice is given a clearly derisive title. That is just guessing, vaguely informed by data, often without even thinking through alternative explanations for the patterns at hand (let alone presenting those alternatives).
I agree with Gelman’s point, late in the post – this is not a failing of statistics, really. It is a failure to use them intelligently, or to use appropriate frameworks to interpret statistical findings. It would be nice, however, if we could have a discussion between quant and qual on how to avoid these outcomes before they happen . . . because story time is most certainly avoidable.
The Qualitative Research Challenge to RCT4D: Part 2
Well, the response to part one was great – really good comments, and a few great response posts. I appreciate the efforts of some of my economist colleagues/friends to clarify the terminology and purpose behind RCTs. All of this has been very productive for me – and hopefully for others engaged in this conversation.
First, a caveat: On the blog I tend to write quickly and with minimal editing – so I get a bit fast and loose at times – well, faster and looser than I intend. So, to this end, I did not mean to suggest that nobody was doing rigorous work in development research – in fact, the rest of my post clearly set out to refute that idea, at least in the qualitative sphere. But I see how Marc Bellemare might have read me that way. What I should have said was that there has always been work, both in research and implementation, where rigorous data collection and analysis were lacking. In fact, there is quite a lot of this work. I think we can all agree this is true . . . and I should have been clearer.
I have also learned that what qualitative social scientists/social theorists mean by theory, and what economists mean by theory, seems to be two different things. Lee defined theory as “formal mathematical modeling” in a comment on part 1 of this series of posts, which is emphatically not what a social theorist might mean. When I say theory, I am talking about a conjectural framing of a social totality such that complex causality can at least be contained, if not fully explained. This framing should have reference to some sort of empirical evidence, and therefore should be testable and refinable over time – perhaps through various sorts of ethnographic work, perhaps through formal mathematical modeling of the propositions at hand (I do a bit of both, actually). In other words, what I mean by theory (and what I focus on in my work) is the establishment of a causal architecture for observed social outcomes. I am all about the “why it worked” part of research, and far less about the “if it worked” questions – perhaps mostly because I have researched unintended “development interventions” (i.e. unplanned road construction, the establishment of a forest reserve that alters livelihoods resource access, etc.) that did not have a clear goal, a clear “it worked!” moment to identify. All I have been looking at are outcomes of particular events, and trying to establish the causes of those outcomes. Obviously, this can be translated to an RCT environment because we could control for the intervention and expected outcome, and then use my approaches to get at the “why did it work/not work” issues.
It has been very interesting to see the economists weigh in on what RCTs really do – they establish, as Marc puts it, “whether something works, not in how it works.” (See also Grant’s great comment on the first post). I don’t think that I would get a lot of argument from people if I noted that without causal mechanisms, we can’t be sure why “what worked” actually worked, and whether the causes of “what worked” are in any way generalizable or transportable. We might have some idea, but I would have low confidence in any research that ended at this point. This, of course, is why Marc, Lee, Ruth, Grant and any number of other folks see a need for collaboration between quant and qual – so that we can get the right people, with the right tools, looking at different aspects of a development intervention to rigorously establish the existence of an impact, and the establish an equally rigorous understanding of the causal processes by which that impact came to pass. Nothing terribly new here, I think. Except, of course, for my continued claim that the qualitative work I do see associated with RCT work is mostly awful, tending toward bad journalism (see my discussion of bad journalism and bad qualitative work in the first post).
But this discussion misses a much larger point about epistemology – what I intended to write in this second part of the series all along. I do not see the dichotomy between measuring “if something works” and establishing “why something worked” as analytically valid. Simply put, without some (at least hypothetical) framing of causality, we cannot rigorously frame research questions around either question. How can you know if something worked, if you are not sure how it was supposed to work in the first place? Qualitative research provides the interpretive framework for the data collected via RCT4D efforts – a necessary framework if we want RCT4D work to be rigorous. By separating qualitative work from the quant oriented RCT work, we are assuming that somehow we can pull data collection apart from the framing of the research question. We cannot – nobody is completely inductive, which means we all work from some sort of framing of causality. The danger is when we don’t acknowledge this simple point – under most RCT4D work, those framings are implicit and completely uninterrogated by the practitioners. Even where they come to the fore (Duflo’s 3 I s), they are not interrogated – they are assumed as framings for the rest of the analysis.
If we don’t have causal mechanisms, we cannot rigorously frame research questions to see if something is working – we are, as Marc says, “like the drunk looking for his car keys under the street lamp when he knows he lost them elsewhere, because the only place he can actually see is under the street lamp.” Only I would argue we are the drunk looking for his keys under a streetlamp, but he has no idea if they are there or not.
In short, I’m not beating up on RCT4D, nor am I advocating for more conversation – no, I am arguing that we need integration, teams with quant and qual skills that frame the research questions together, that develop tests together, that interpret the data together. This is the only way we will come to really understand the impact of our interventions, and how to more productively frame future efforts. Of course, I can say this because I already work in a mixed-methods world where my projects integrate the skills of GIScientists, land use modelers, climate modelers, biogeographers and qualitative social scientists – in short, I have a degree of comfort with this sort of collaboration. So, who wants to start putting together some seriously collaborative, integrated evaluations?
The Qualitative Research Challenge to RCT4D: Part 1
Those following this blog (or my twitter feed) know that I have some issues with RCT4D work. I’m actually working on a serious treatment of the issues I see in this work (i.e. journal article), but I am not above crowdsourcing some of my ideas to see how people respond. Also, as many of my readers know, I have a propensity for really long posts. I’m going to try to avoid that here by breaking this topic into two parts. So, this is part 1 of 2.
To me, RCT4D work is interesting because of its emphasis on rigorous data collection – certainly, this has long been a problem in development research, and I have no doubt that the data they are gathering is valid. However, part of the reason I feel confident in this data is because, as I raised in an earlier post, it is replicating findings from the qualitative literature . . . findings that are, in many cases, long-established with rigorously-gathered, verifiable data. More on that in part 2 of this series.
One of the things that worries me about the RCT4D movement is the (at least implicit, often overt) suggestion that other forms of development data collection lack rigor and validity. However, in the qualitative realm we spend a lot of time thinking about rigor and validity, and how we might achieve both – and there are tools we use to this end, ranging from discursive analysis to cross-checking interviews with focus groups and other forms of data. Certainly, these are different means of establishing rigor and validity, but they are still there.
Without rigor and validity, qualitative research falls into bad journalism. As I see it, good journalism captures a story or an important issue, and illustrates that issue through examples. These examples are not meant to rigorously explain the issue at hand, but to clarify it or ground it for the reader. When journalists attempt to move to explanation via these same few examples (as far too often columnists like Kristof and Friedman do), they start making unsubstantiated claims that generally fall apart under scrutiny. People mistake this sort of work for qualitative social science all the time, but it is not. Certainly there is some really bad social science out there that slips from illustration to explanation in just the manner I have described, but this is hardly the majority of the work found in the literature. Instead, rigorous qualitative social science recognizes the need to gather valid data, and therefore requires conducting dozens, if not hundreds, of interviews to establish understandings of the events and processes at hand.
This understanding of qualitative research stands in stark contrast to what is in evidence in the RCT4D movement. For all of the effort devoted to data collection under these efforts, there is stunningly little time and energy devoted to explanation of the patterns seen in the data. In short, RCT4D often reverts to bad journalism when it comes time for explanation. Patterns gleaned from meticulously gathered data are explained in an offhand manner. For example, in her (otherwise quite well-done) presentation to USAID yesterday, Esther Duflo suggested that some problematic development outcomes could be explained by a combination of “the three I s”: ideology, ignorance and inertia. This is a boggling oversimplification of why people do what they do – ideology is basically nondiagnostic (you need to define and interrogate it before you can do anything about it), and ignorance and inertia are (probably unintentionally) deeply patronizing assumptions about people living in the Global South that have been disproven time and again (my own work in Ghana has demonstrated that people operate with really fine-grained information about incomes and gender roles, and know exactly what they are doing when they act in a manner that limits their household incomes – see here, here and here). Development has claimed to be overcoming ignorance and inertia since . . . well, since we called it colonialism. Sorry, but that’s the truth.
Worse, this offhand approach to explanation is often “validated” through reference to a single qualitative case that may or may not be representative of the situation at hand – this is horribly ironic for an approach that is trying to move development research past the anecdotal. This is not merely external observation – I have heard from people working inside J-PAL projects that the overall program puts little effort into serious qualitative work, and has little understanding of what rigor and validity might mean in the context of qualitative methods or explanation. In short, the bulk of explanation for these interesting patterns of behavior that emerges from these studies resorts to uninterrogated assumptions about human behavior that do not hold up to empirical reality. What RCT4D has identified are patterns, not explanations – explanation requires a contextual understanding of the social.
Coming soon: Part 2 – Qualitative research and the interpretation of empirical data
Qualitative research was (already) here . . .
You know, qualitative social scientists of various stripes have long complained of their marginalization in development. Examples abound of anthropologists, geographers, and sociologists complaining about the influence of the quantitatively-driven economists (and to a lesser extent, some political scientists) over development theory and policy. While I am not much for whining, these complaints are often on the mark – quantitative data (of the sort employed by economists, and currently all the rage in political science) tends to carry the day over qualitative data, and the nuanced lessons of ethnographic research are dismissed as unimplementable, ideosyncratic/place-specific, without general value, etc. This is not to say that I have an issue with quantitative data – I believe we should employ the right tool for the job at hand. Sadly, most people only have either qualitative or quantitative skills, making the selection of appropriate tools pretty difficult . . .
But what is interesting, of late, is what appears to be a turn toward the lessons of the qualitative social sciences in development . . . only without actually referencing or reading those qualitative literatures. Indeed, the former quantitative masters of the development universe are now starting to figure out and explore . . . the very things that the qualitative community has known for decades. What is really frustrating and galling is that these “new” studies are being lauded as groundbreaking and getting great play in the development world, despite the fact they are reinventing the qualitative wheel, and without much of the nuance of the current qualitative literature and its several decades of nuance.
What brings me to today’s post is the new piece on hunger in Foreign Policy by Abhijit Banerjee and Esther Duflo. On one hand, this is great news – good to see development rising to the fore in an outlet like Foreign Policy. I also largely agree with their conclusions – that the poverty trap/governance debate in development is oversimplified, that food security outcomes are not explicable through a single theory, etc. On the other hand, from the perspective of a qualitative researcher looking at development, there is nothing new in this article. Indeed, the implicit premise of the article is galling: When they argue that to address poverty, “In practical terms, that meant we’d have to start understanding how the poor really live their lives,” the implication is that nobody has been doing this. But what of the tens of thousands of anthropologists, geographers and sociologists (as well as representatives of other cool, hybridized fields like new cultural historians and ethnoarchaeologists). Hell, what of the Peace Corps?
Whether intentional or not, this article wipes the qualitative research slate clean, allowing the authors to present their work in a methodological and intellectual vacuum. This is the first of my problems with this article – not so much with its findings, but with its appearance of method. While I am sure that there is more to their research than presented in the article, the way their piece is structured, the case studies look like evidence/data for a new framing of food security. They are not – they are illustrations of the larger conceptual points that Banerjee and Duflo are making. I am sure that Banerjee and Duflo know this, but the reader does not – instead, most readers will think this represents some sort of qualitative research, or a mixed method approach that takes “hard numbers” and mixes it in with the loose suppositions that Banerjee and Duflo offer by way of explanation for the “surprising” outcomes they present. But loose supposition is not qualitative research – at best, it is journalism. Bad journalism. My work, and the work of many, many colleagues, is based on rigorous methods of observation and analysis that produce validatable data on social phenomena. The work that led to Delivering Development and many of my refereed publications took nearly two years of on-the-ground observation and interviewing, including follow-ups, focus groups and even the use of archaeology and remotely-sensed data on land use to cross-check and validate both my data and my analyses.
The result of all that work was a deep humility in the face of the challenges that those living in places like Coastal Ghana or Southern Malawi manage on a day-to-day basis . . . and deep humility when addressing the idea of explanation. This is an experience I share with countless colleagues who have spent a lot of time on the ground in communities, ministries and aid organizations, a coming to grips with the fact that massively generalizable solutions simply don’t exist in the way we want them to, and that singular interventions will never address the challenges facing those living in the Global South.
So, I find it frustrating when Banerjee and Duflo present this observation as in any way unique:
What we’ve found is that the story of hunger, and of poverty more broadly, is far more complex than any one statistic or grand theory; it is a world where those without enough to eat may save up to buy a TV instead, where more money doesn’t necessarily translate into more food, and where making rice cheaper can sometimes even lead people to buy less rice.
For anyone working in food security – that is, anyone who has been reading the literature coming out of anthropology, geography, sociology, and even some areas of ag econ, this is not a revelation – this is standard knowledge. A few years ago I spent a lot of time and ink on an article in Food Policy that tried to loosely frame a schematic of local decision-making that leads to food security outcomes – an effort to systematize an approach to the highly complex sets of processes and decisions that produce hunger in particular places because there is really no way to get a single, generalized statistic or finding that will explain hunger outcomes everywhere.
In other words: We know. So what do you have to tell us?
The answer, unfortunately, is not very much . . . because in the end they don’t really dive into the social processes that lead to the sorts of decisions that they see as interesting or counterintuitive. This is where the heat is in development research – there are a few of us working down at this level, trying to come up with new framings of social process that move us past a reliance solely on the blunt tool of economistic rationality (which can help explain some behaviors and decisions) toward a more nuanced framing of how those rationalities are constructed by, and mobilize, much larger social processes like gender identification. The theories in which we are dealing are very complex, but they do work (at least I think my work with governmentality is working – but the reviewers at Development and Change might not agree).
And maybe, just maybe, there is an opening to get this sort of work out into the mainstream, to get it applied – we’re going to try to do this at work, pulling together resources and interests across two Bureaus and three offices to see if a reframing of livelihoods around Foucault’s idea of governmentality can, in fact, get us better resolution on livelihoods and food security outcomes than current livelihoods models (which mostly assume that decisionmaking is driven by an effort to maximize material returns on investment and effort). Perhaps I rest too much faith on the idea of evidence, but if we can implement this idea and demonstrate that it works better, perhaps we will have a lever with which to push oversimplified economistic assumptions out of the way, while still doing justice to the complexity of social process and explanation in development.
Interesting but flawed . . .
The Yale Project on Climate Change Communication recently put out a report on Americans’ Knowledge of Climate Change. The findings are pretty interesting, but at times really problematic. This project has a history of putting out cool products that address the complexity of communication and opinion surrounding climate change, such as their Six Americas project.
This graphic, from that report, shows that dividing the country (or indeed any group of people) into global warming alarmists and global warming sceptics is a gross oversimplification of public feeling and perception. The poles of alarmed and dismissive are less than 25% of the population. Disengaged, doubtful and dismissive are only 34% of the population. Alarmed and concerned are 41%. Note that neither category is a majority (though alarmed and concerned is a plurality). Anthropogenic global climate change is NOT dead in public opinion at all.
Well, how did we get to this spectrum of opinions? The new report suggests that while we spend a lot of time talking politics, the larger issue might be education and outreach. There are some really interesting findings in here – for example:
Majorities of American adults correctly understand that weather often changes from year to year (83%) and that “climate” means the average weather conditions in a region (74%). Majorities, however, incorrectly believe that the climate often changes from year to year or that “weather” means the average climate conditions in a region, suggesting that many people continue to confuse weather and climate.
Yep. And I blame the media, who seem to constantly conflate these two on all ends of the political spectrum. A heavy snowfall does not discredit climate change (or even warming), but a heat wave is not a signal of warming unto itself, either.
A majority of Americans (73%) correctly understands that current conditions are not colder than ever before in Earth’s history, but a majority (55%) incorrectly believes the opposite – that the Earth’s climate is now warmer than it has ever been before (this is false – global temperatures have been warmer than current conditions many times in the past).
Wait, who ever said it was the coldest it has ever been? I get what they are trying to do, but that is just an odd thing to throw in. And the fact a majority thinks we are at our warmest point ever speaks to a deeply distressing lack of understanding of our history – things have been warmer in the past, and we know from the geologic records associated with those times what sorts of sea level rise, etc. we can expect. We are not in terra incognita entirely right now – we have records of sudden changes in the state of the global climate as it warmed beyond where we are today. The past is prelude . . .
There is a lot of this sort of thing in the report. All of it is interesting. But it needs to be read with a careful, critical eye. I am worried about some of the questions in this study – or at least their phrasing and the interpretation of the results. For example:
Thirty-nine percent (39%) say that most scientists think global warming is happening, while 38 percent say there is a lot of disagreement among scientists whether or not global warming is happening
At first, this simply seems to be an illustration of the wide divide in the public on the understanding of the nature of the scientific consensus around climate change. But this question is too broad to really capture what is going on here. Answers probably varied greatly depending on the respondent’s level of knowledge (highly variable, as the report noted) – for example, a well-informed person inclined to think that the human causes of global climate change are overstated could take the real and significant (but very narrow) debates about the exact workings of various greenhouse gases, or how to best model the climate, and argue that this represents significant disagreements about whether or not anthropogenic global warming is happening (which is a serious mischaracterization), while someone who is more environmentally inclined but has less understanding of the field might simply assume there is no debate in the science at all, which is not true. To get to 38% thinking that scientists are debating whether climate change is happening or not suggests that something like this happened on this question. There is more or less no scientific debate, and very minimal public debate, over whether or not the climate is changing – the instrument record is pretty clear. The question is how fast, and by what exact mechanisms. Nearly all skeptics agree that some change is taking place – they just tend to doubt that humans are the cause. If only 7% of the study’s respondents thought that climate change was not happening at all, why would they think that scientists had a greater level of debate?
I really dislike the following questions/data:
Respondents were given the current temperature of the Earth’s surface (approximately 58ºFahrenheit) as a reference point. They were then asked what they thought the average temperature was during the last ice age. The correct answer is between 46º and 51º. The median public response, however, was 32º – the freezing point of water – while many other people responded 0º.
Americans, however, did much better estimating the Earth’s surface temperature 150 years ago (before the Industrial Revolution). The correct answer is approximately 56º to 57º Fahrenheit. The median public response was 54º.
When asked what temperature they thought it would be by the year 2020 if no additional actions are taken to reduce global warming, the median response was 60º, slightly higher than the scientific estimate of 58.4º Fahrenheit.
Realistically, this is a bunch of wild guesses. We Americans are not so good at simply saying “I don’t know”. Hell, I would not have nailed these, and I work in this area. The question requires too much precision to have any reasonable expectation of meaningful data.
Finally, a few moments of oversimplification in the data analysis that bother me – even though I like the idea of the report, and I generally agree with the premise that climate change is anthropogenic:
Majorities of Americans, however, incorrectly believe that the hole in the ozone layer, toxic wastes, aerosol spray cans, volcanic eruptions, the sun, and acid rain contribute to global warming.
Again, the analysis assumes a uniform, low level of understanding of climate change across the sample. However, a well-informed person would know that the sun is, in fact, technically a contributor to climate change – it is a small forcing on our climate, dwarfed by that of greenhouse gases, to be sure, but still a forcing. Had I been asked this question, I would have gotten it “wrong” by their analysis . . . but their analysis is predicated on an incorrect assumption about the drivers of climate change. I could make the same argument for toxic wastes, as depending on what they are and how they are stored, they may well change land cover or decompose and release greenhouse gases, thus impacting climate change. The analysis here is too simplistic.
I’m a bit surprised that this sort of a report would be full of problematically phrased questions and even more problematic interpretations of the data (i.e. predicated on misunderstandings of the science). This is amateur hour stuff that any of my grad students could pick up on and address in their work long before they got to publication . . . too bad, as the effort and some of the information is really interesting. It would have been nice to have a consistently interesting, rigorous report.