Entries tagged with “explanation”.
Did you find what you wanted?
Mon 4 Feb 2013
I have a confession. For a long time now I have found myself befuddled by those who claim to have identified the causes behind observed outcomes in social research via the quantitative analysis of (relatively) large datasets (see posts here, here, and here). For a while, I thought I was seeing the all-to-common confusion of correlation and causation…except that a lot of smart, talented people seemed to be confusing correlation with causation. This struck me as unlikely.
Then, the other day in seminar (I was covering for a colleague in our department’s “Contemporary Approaches to Geography” graduate seminar, discussing the long history of environmental determinism within and beyond the discipline), I found myself in a similar discussion related to explanation…and I think I figured out what has been going on. The remote sensing and GIS students in the course, all of whom are extraordinarily well-trained in quantitative methods, got to thinking about how to determine if, in fact, the environment was “causing” a particular behavior*. In the course of this discussion, I realized that what they meant by “cause” was simple (I will now oversimplify): when you can rule out/control for the influence of all other possible factors, you can say that factor X caused event Y to happen. Indeed, this does establish a causal link. So, I finally get what everyone was saying when they said that, via well-constructed regressions, etc., one can establish causality.
So it turns out I was wrong…sort of. You see, I wasn’t really worried about causality…I was worried about explanation. My point was that the information you would get from a quantitative exercise designed to establish causal relationships isn’t enough to support rigorous project and program design. Just because you know that the construction of a borehole in a village caused girl-child school attendance to increase in that village doesn’t mean you know HOW the borehole caused this change in school attendance to happen. If you cannot rigorously explain this relationship, you don’t understand the mechanism by which the borehole caused the change in attendance, and therefore you don’t really understand the relationship. In the “more pure” biophysical sciences**, this isn’t that much of a problem because there are known rules that particles, molecules, compounds, and energy obey, and therefore under controlled conditions one can often infer from the set of possible actors and actions defined by these rules what the causal mechanism is.
But when we study people it is never that simple. The very act of observing people’s behaviors causes shifts in that behavior, making observation at best a partial account of events. Interview data are limited by the willingness of the interviewee to talk, and the appropriateness of the questions being asked – many times I’ve had to return to an interviewee to ask a question that became evident later, and said “why didn’t you tell me this before?” (to which they answer, quite rightly, with something to the effect of “you didn’t ask”). The causes of observed human behavior are staggeringly complex when we get down to the real scales at which decisions are made – the community, household/family, and individual. Decisions may vary by time of the year, or time of day, and by the combination of gender, age, ethnicity, religion, and any other social markers that the group/individual chooses to mobilize at that time. In short, just because we see borehole construction cause increases in girl-child school attendance over and over in several places, or even the same place, doesn’t mean that the explanatory mechanism between the borehole and attendance is the same at all times.
Understanding that X caused Y is lovely, but in development it is only a small fraction of the battle. Without understanding how access to a new borehole resulted in increased girl-child school attendance, we cannot scale up borehole construction in the context of education programming and expect to see the same results. Further, if we do such a scale-up, and don’t get the same results, we won’t have any idea why. So there is causality (X caused Y to happen) and there are causal mechanisms (X caused Y to happen via Z – where Z is likely a complex, locally/temporally specific alignment of factors).
Unfortunately, when I look at much quantitative development research, especially in development economics, I see a lot of causality, but very little work on causal mechanisms that get us to explanation. There is a lot of story time, “that pivot from the quantitative finding to the speculative explanation.” In short, we might be programming development and aid dollars based upon evidence, but much of the time that evidence only gets us part of the way to what we really need to know to really inform program and project design.
This problem is avoidable –it does not represent the limits of our ability to understand the world. There is one obvious way to get at those mechanisms – serious, qualitative fieldwork. We need to be building research and policy teams where ethnographers and other qualitative social scientists learn to respect the methods and findings of their quantitative brethren such that they can target qualitative methods at illuminating the mechanisms driving robust causal relationships. At the same time, the quantitative researchers on these teams will have to accept that they have only partially explained what we need to know when they have established causality through their methods, and that qualitative research can carry their findings into the realm of implementation.
The bad news for everyone…for this to happen, you are going to have to pick your heads up out of your (sub)disciplinary foxholes and start reading across disciplines in your area of interest. Everyone talks a good game about this, but when you read what keeps getting published, it is clear that cross-reading is not happening. Seriously, the number of times I have seen people in one field touting their “new discoveries” about human behavior that are already common conversation in other disciplines is embarrassing…or at least it should be to the authors. But right now there is no shame in this sort of thing, because most folks (including peer reviewers) don’t read outside their disciplines, and therefore have no idea how absurd these claims of discovery really are. As a result, development studies gives away its natural interdisciplinary advantage and returns to the problematic structure of academic knowledge and incentives, which not only enable, but indeed promote narrowly disciplinary reading and writing.
Development donors, I need a favor. I need you to put a little research money on the table to learn about whatever it is you want to learn about. But when you do, I want you to demand it be published in a multidisciplinary development-focused journal. In fact, please start doing this for all of your research-related money. People will still pursue your money, as the shrinking pool of research dollars is driving academia into your arms. Administrators like grant and contract money, and so many academics are now being rewarded for bringing in grants and contracts from non-traditional sources (this is your carrot). Because you hold the carrot, you can draw people in and then use “the stick” inherent in the terms of the grant/contract to demand cross-disciplinary publishing that might start to leverage change in academia. You all hold the purse, so you can call the tune…
*Spoiler alert: you can’t. Well, you probably can if 1) you pin the behavior you want to explain down to something extraordinarily narrow, 2) can limit the environmental effect in question to a single independent biophysical process (good luck with that), and 3) limit your effort to a few people in a single place. But at that point, the whole reason for understanding the environmental determinant of that behavior starts to go out the window, as it would clearly not be generalizable beyond the study. Trust me, geography has been beating its head against this particular wall for a century or more, and we’ve buried the idea. Learn from our mistakes.
**by “more pure” I am thinking about those branches of physics, chemistry, and biology in which lab conditions can control for many factors. As soon as you get into field sciences, or starting asking bigger questions, complexity sets in and things like causality get muddied in the manner I discuss below…just ask an ecologist.
Mon 17 Sep 2012
There has mixed response to my posts on disaster awareness among college students (well, the Horn of Africa drought among my current students) – see posts here and here. Some see something hopeful and interesting in the idea that the students want more complex explanations for the problems they see. Others are significantly more negative, suggesting that people such as my students are just symptomatic of a larger societal, if not species-level, lack of empathy for distant others. I fall on the optimistic side of things, perhaps because I am a geographer. Let me explain…
Geography, as a discipline, spends a hell of a lot of time thinking about how places are created and maintained. Places are not locations (folks get this mixed up all the time) – places are our experiences of particular locations – at least this is how I choose to think about it. And when you think of it that way, it becomes impossible to see life in a particular place as independent from events in other places. The experience of living in Columbia, South Carolina is shaped by the weather, the cost of living, the infrastructure, the schools (I am a parent), etc. But each of these is in turn shaped by other factors that transcend Columbia. The cost of living and state of the infrastructure are intimately tied to the history of the state of South Carolina within the United States (where the South has historically been the underdeveloped agrarian other of the industrialized Northeast), but are also tied to the global economy. South Carolina is now the last stopping point for large-scale manufacturing before it heads out of the US to find the most favorable conditions of production possible. The overseas shift of the textile industry wrought devastation on the state’s economy…and relatively few in the state seem able to come to grips with the fact they were ground up in the jaws of a new global economy that has already spit them out. Even the weather is being reshaped by global factors that drive climate change, as a new regime of reduced rainfall seems to be settling in. At what point do you stop calling a prolonged rainfall deficit a drought and start calling it the new normal? Turns out about three decades. We are about 20 years into a significant decline in precipitation, so we are getting there. Thus, the policy decisions (regarding industrial policy and emissions policy) of actors in China and India drive shifts in the economy and environment of the State of South Carolina. We are thoroughly tied up in larger global forces here. To understand South Carolina today, we have to understand the larger world today – there is simply no way around this.
As soon as this lesson settles in (and it can take a while), it becomes obvious that these forces flow both ways – that is, as Columbia, SC is constituted by global forces, so too what we do here in Columbia contributes to global forces that play out in other places. Thus, when we vote for federal lawmakers who keep absurd ethanol subsidies in place no matter what the price/maize production conditions, we create a driver of food price increases that can radiate around the world. And while we in Columbia feel those increases, when the price of a loaf of bread goes up by a dollar, most of us are inconvenienced and annoyed. For someone who was already living on less than $2/day, this same price increase blows up their capacity to feed themselves.
All of this then goes back to my earlier point about what the students wanted – complex explanations. The kids already get it, folks – they already understand an interconnected world (to some extent), and they mistrust oversimplified explanations. When you feed them simple explanations, you often have to root out the interconnections that connect us to events in other parts of the world – the very things that students would grab on to. In short, by oversimplifying things, we are making it harder for people to feel connected to the places in which things like famine happen.
The lesson: find yourself a geographer, work with them to tell the damn story in all its complex glory, and get out of the way. The kids are waiting…
Thu 4 Aug 2011
Marc Bellemare’s blog pointed me to an interesting paper by Pascaline Dupas and Jonathan Robinson titled “Why Don’t the Poor Save More? Evidence from Health Savings Experiments.” It is an interesting paper, taking a page from the RCT4D literature to test some different tools for savings in four Kenyan villages. I’m not going to wade into the details of the paper or its findings here (they find some tools to be more effective than others at promoting savings for health expenditures), because they are not what really caught me about this paper. Instead, what struck me was the absence of a serious consideration of “the social” in the framing of the questions asked and the results. Dupas and Robinson expected three features to impact health savings: adequate storage facilities/technology, the ability to earmark funds, and the level of social commitment of the participant. The social context of savings (or, more accurately, barriers to savings) are treated in what I must say is a terribly dismissive way [emphases are mine]:
a secure storage technology can enable individuals to avoid carrying loose cash on their person and thus allow people to keep some physical distance between themselves and their money. This may make it easier to resist temptations, to borrow the terminology in Banerjee and Mullainathan (2010), or unplanned expenditures, as many of our respondents call them. While these unplanned expenditures include luxury items such as treats, another important category among such unplanned expenditures are transfers to others.
A storage technology can increase the mental costs associated with unplanned expenditures, thereby reducing such expenditures. Indeed, if people use the storage technology to save towards a specic goal, such as a health goal in our study, people may consider the money saved as unavailable for purposes other than the specic goal – this is what Thaler (1990) coined mental accounting. By enabling such mental accounting, a designated storage place may give people the strength to resist frivolous expenditures as well as pressure to share with others, including their spouse.
I have seen many cases of unplanned expenditures to others in my fieldwork. Indeed, my village-based field crews in Ghana used to ask for payment on as infrequent a basis as possible to avoid exactly these sorts of expenditures. They would plan for large needed purchases, work until they had earned enough for that purchase, then take payment and immediately make the purchase, making their income illiquid before family members could call upon them and ask for loans or handouts.
However, the phrasing of Dupas and Robinson strikes the anthropologist/ geographer in me as dismissive. These expenses are seen as “frivolous”, things that should be “resisted”. The authors never consider the social context of these expenditures – why people agree to make them in the first place. There seems to be an implicit assumption here that people don’t know how to manage their money without the introduction of new tools, and that is not at all what I have seen (albeit in contexts other than Kenya). Instead, I saw these expenditures as part of a much larger web of social relations that implicates everything from social status to gender roles – in this context, the choice to give out money instead of saving it made much more sense.
In short, it seems to me that Dupas and Robinson are treating these savings technologies as apolitical, purely technical interventions. However, introducing new forms of savings also intervenes in social relations at scales ranging from the household to the extended family to the community. Thus, the uptake of these forms of savings will be greatly effected by contextual factors that seem to have been ignored here. Further, the durability of the behavioral changes documented in this study might be much better predicted and understood – from my perspective, the declining use of these technologies over the 33 month scope of the project was completely predictable (the decline, that is, not the size of the decline). Just because a new technology enables savings that might result in a greater standard of living for the individual or household does not mean that the technology will be seen as desirable – instead, that standard of living must also work within existing social roles and relations if these new behaviors are to endure. Therefore, we cannot really explain the declining use of these technologies over time . . . yet development is, to me, about catalyzing enduring change. While this study shows that the introduction of these technologies has at least a short term transformative effect on savings behavior, I’m not convinced this study does much to advance our understanding of how to catalyze changes that will endure.
Wed 13 Jul 2011
An interesting review of Paul Collier’s The Bottom Billion and Wars, Guns and Votes by Yale Anthropologist Mike McGovern has gotten a little bit of attention recently in development circles, speaking as it does to ongoing debates about the role of statistical analysis, what counts as explanation, and where qualitative research fits into all of this. I will take up McGovern’s good (but incomplete, in my opinion) review in another post. Here, I needed to respond to a blog entry about this review.
On the Descriptive Statistics, Causal Inference and Social Science blog, Andrew Gelman discusses McGovern’s review. While there is a lot going on in this post, one issue caught my attention in particular. In his review, McGovern argues that “Much of the intellectual heavy lifting in these books is in fact done at the level of implication or commonsense guessing,” what Gelman (quoting Fung) calls “story time”, the “pivot from the quantitative finding to the speculative explanation.” However, despite the seemingly dismissive term for this sort of explanation, in his blog post Gelman argues ”story time can’t be avoided.” His point:
On one hand, there are real questions to be answered and real decisions to be made in development economics (and elsewhere), and researchers and policymakers can’t simply sit still and say they can’t do anything because the data aren’t fully persuasive. (Remember the first principle of decision analysis: Not making a decision is itself a decision.)
From the other direction, once you have an interesting quantitative finding,of course you want to understand it, and it makes sense to use all your storytelling skills here. The challenge is to go back and forth between the storytelling and the data. You find some interesting result (perhaps an observational data summary, perhaps an analysis of an experiment or natural experiment), this motivates a story, which in turn suggests some new hypotheses to be studied.
Now, on one hand I take his point – research is iterative, and answering one set of questions (or one set of new data) often raises new questions which can be interrogated. But Gelman seems to presume that explanation only comes from more statistical analysis, without considering what I saw as McGovern’s subtle point: qualitative social scientists look at explanation, and do not revert to story time to do so (good luck getting published if you do). We spend a hell of a lot of time fleshing out the causal processes behind our observations, including establishing rigor and validity for our data and conclusions, before we present stories. This is not to say that our explanations are immediately complete or perfect, nor is it to suggest that our explanations do not raise new questions to pursue. However, there is no excuse for the sort of “story time” analysis that McGovern is pointing out in Collier’s work – indeed, I would suggest that is why the practice is given a clearly derisive title. That is just guessing, vaguely informed by data, often without even thinking through alternative explanations for the patterns at hand (let alone presenting those alternatives).
I agree with Gelman’s point, late in the post – this is not a failing of statistics, really. It is a failure to use them intelligently, or to use appropriate frameworks to interpret statistical findings. It would be nice, however, if we could have a discussion between quant and qual on how to avoid these outcomes before they happen . . . because story time is most certainly avoidable.
Fri 20 May 2011
I was at a talk today where folks from Michigan State were presenting research and policy recommendations to guide the Feed the Future initiative. I greatly appreciate this sort of presentation – it is good to get real research in the building, and to see USAID staff that have so little time turn out in large numbers to engage. Once again, folks, its not that people in the agencies aren’t interested or don’t care, its a question of time and access.
In the course of one of the presentations, however, I saw a moment of “explanation” for observed behavior that nicely captures a larger issue that has been eating at me as the randomized control trials for development (RCT4D) movement gains speed . . . there isn’t a lot of explanation there. There is really interesting data, rigorously collected, but explanation is another thing entirely.
In the course of the presentation, the presenter put up a slide that showed a wide dispersion of prices around the average price received by farmers for their maize crops around a single market area (near where I happen to do work in Malawi). Nothing too shocking there, as this happens in Malawi, and indeed in many places. However, from a policy and programming perspective, it’s important to know that the average price is NOT the same thing as what a given household is taking home. But then the presenter explained this dispersion by noting (in passing) that some farmers were more price-savvy than others.
1) there is no evidence at all to support this claim, either in his data or in the data I have from an independent research project nearby
2) this offhand explanation has serious policy ramifications.
This explanation is a gross oversimplification of what is actually going on here – in Mulanje (near the Luchenza market area analyzed in the presentation), price information is very well communicated in villages. Thus, while some farmers might indeed be more savvy than others, the prices they are able to get are communicated throughout the village, thus distributing that information. So the dispersion of prices is the product of other factors. Certainly desperation selling is probably part of the issue (another offhand explanation offered later in the presentation). However, what we really need, if we want a rigorous understanding of the causes of this dispersion and how to address it, is a serious effort to grasp the social component of agriculture in this area – how gender roles, for example, shape household power dynamics, farm roles, and the prices people will sell at (this is a social consideration that exceeds explanation via markets), or how social networks connect particular farmers to particular purchasers in a manner that facilitates or inhibits price maximization at market. These considerations are both causal of the phenomena that the presenter described, and the points of leverage on which policy might act to actually change outcomes. If farmers aren’t “price savvy”, this suggests the need for a very different sort of intervention than what would be needed to address gendered patterns of agricultural strategy tied to long-standing gender roles and expectations.
This is a microcosm of what I am seeing in the RCT4D world right now – really rigorous data collection, followed by really thin interpretations of the data. It is not enough to just point out interesting patterns, and then start throwing explanations out there – we must turn from rigorous quantitative identification of significant patterns of behavior to the qualitative exploration of the causes of those patterns and their endurance over time. I’ve been wrestling with these issues in Ghana for more than a decade now, an effort that has most recently led me to a complete reconceptualization of livelihoods (shifting from understanding livelihoods as a means of addressing material conditions to a means of governing behaviors through particular ways of addressing material conditions – the article is in review at Development and Change). However, the empirical tests of this approach (with admittedly tiny-n size samples in Ghana, and very preliminary looks at the Malawi data) suggest that I have a better explanatory resolution for explained behaviors than possible through existing livelihoods approaches (which would end up dismissing a lot of choices as illogical or the products of incomplete information) – and therefore I have a better foundation for policy recommendations than available without this careful consideration of the social.
See, for example, this article I wrote on how we approach gender in development (also a good overview of the current state of gender and development, if I do say so myself). I empirically demonstrate that a serious consideration of how gender is constructed in particular places has large material outcomes on whose experiences we can understand, and therefore the sorts of interventions we might program to address particular challenges. We need more rigorous wrestling with “the social” if we are going to learn anything meaningful from our data. Period.
In summary, explanation is hard. Harder, in many ways, than rigorous data collection. Until we start spending at least as much effort on the explanation side as we do on the collection side, we will not really change much of anything in development.