Entries tagged with “quantitative methods”.
Did you find what you wanted?
Sun 24 Jul 2016
Unsolicited publishing advice/reviewing rant to follow. Brace yourselves.
When writing an article based on the quantitative analysis of a phenomena, whatever it may be and however novel your analysis, you are not absolved from reading/understanding the conceptual literature (however qualitative) addressing that phenomena. Sure, you might be using a larger dataset than ever used before. Certainly, the previous literature might have been case-study based, and therefore difficult to generalize. But that doesn’t give you a pass to just ignore that existing literature.
- That literature establishes the meanings of the concepts you are measuring/testing
- That literature captures the current state of knowledge on those concepts
- Often, that literature (if qualitative, especially if ethnographic) can get at explanations for the phenomena that cannot be had through qualitative methods alone
If you ignore this literature:
- You’ll just ask questions that have already been answered. Everybody hates that, especially time-constrained reviewers who already know the answers to your questions because they actually have read/contributed to the literature you ignored.
- You’ll likely end up with results that don’t make sense, and with no means of explaining or even addressing them. Editors and reviewers hate that, too.
- Your results, even if they appear to be statistically significant, will be crap. I don’t care how sophisticated your quantitative analysis is, or how innovative your tools might be, you are shoving crap into a very innovative, sophisticated tool, which means that all you’ll get out the other end is crap. Reviewers hate crap. Editors hate crap. And your crap is probably not actionable (and really shouldn’t be), so nobody outside academia will like your crap.
Please don’t generate more crap. There is plenty around.
Finally, a note on professionalism and your career: Citing around people who have worked on the phenomena you are investigating because you are trying to capture a particular field of knowledge is awful intellectual practice that, beyond needlessly slowing the pace of innovation in the field in question, will never work…because editors will send the people you are not citing the article for review. And they will wreck you.
Mon 4 Feb 2013
I have a confession. For a long time now I have found myself befuddled by those who claim to have identified the causes behind observed outcomes in social research via the quantitative analysis of (relatively) large datasets (see posts here, here, and here). For a while, I thought I was seeing the all-to-common confusion of correlation and causation…except that a lot of smart, talented people seemed to be confusing correlation with causation. This struck me as unlikely.
Then, the other day in seminar (I was covering for a colleague in our department’s “Contemporary Approaches to Geography” graduate seminar, discussing the long history of environmental determinism within and beyond the discipline), I found myself in a similar discussion related to explanation…and I think I figured out what has been going on. The remote sensing and GIS students in the course, all of whom are extraordinarily well-trained in quantitative methods, got to thinking about how to determine if, in fact, the environment was “causing” a particular behavior*. In the course of this discussion, I realized that what they meant by “cause” was simple (I will now oversimplify): when you can rule out/control for the influence of all other possible factors, you can say that factor X caused event Y to happen. Indeed, this does establish a causal link. So, I finally get what everyone was saying when they said that, via well-constructed regressions, etc., one can establish causality.
So it turns out I was wrong…sort of. You see, I wasn’t really worried about causality…I was worried about explanation. My point was that the information you would get from a quantitative exercise designed to establish causal relationships isn’t enough to support rigorous project and program design. Just because you know that the construction of a borehole in a village caused girl-child school attendance to increase in that village doesn’t mean you know HOW the borehole caused this change in school attendance to happen. If you cannot rigorously explain this relationship, you don’t understand the mechanism by which the borehole caused the change in attendance, and therefore you don’t really understand the relationship. In the “more pure” biophysical sciences**, this isn’t that much of a problem because there are known rules that particles, molecules, compounds, and energy obey, and therefore under controlled conditions one can often infer from the set of possible actors and actions defined by these rules what the causal mechanism is.
But when we study people it is never that simple. The very act of observing people’s behaviors causes shifts in that behavior, making observation at best a partial account of events. Interview data are limited by the willingness of the interviewee to talk, and the appropriateness of the questions being asked – many times I’ve had to return to an interviewee to ask a question that became evident later, and said “why didn’t you tell me this before?” (to which they answer, quite rightly, with something to the effect of “you didn’t ask”). The causes of observed human behavior are staggeringly complex when we get down to the real scales at which decisions are made – the community, household/family, and individual. Decisions may vary by time of the year, or time of day, and by the combination of gender, age, ethnicity, religion, and any other social markers that the group/individual chooses to mobilize at that time. In short, just because we see borehole construction cause increases in girl-child school attendance over and over in several places, or even the same place, doesn’t mean that the explanatory mechanism between the borehole and attendance is the same at all times.
Understanding that X caused Y is lovely, but in development it is only a small fraction of the battle. Without understanding how access to a new borehole resulted in increased girl-child school attendance, we cannot scale up borehole construction in the context of education programming and expect to see the same results. Further, if we do such a scale-up, and don’t get the same results, we won’t have any idea why. So there is causality (X caused Y to happen) and there are causal mechanisms (X caused Y to happen via Z – where Z is likely a complex, locally/temporally specific alignment of factors).
Unfortunately, when I look at much quantitative development research, especially in development economics, I see a lot of causality, but very little work on causal mechanisms that get us to explanation. There is a lot of story time, “that pivot from the quantitative finding to the speculative explanation.” In short, we might be programming development and aid dollars based upon evidence, but much of the time that evidence only gets us part of the way to what we really need to know to really inform program and project design.
This problem is avoidable –it does not represent the limits of our ability to understand the world. There is one obvious way to get at those mechanisms – serious, qualitative fieldwork. We need to be building research and policy teams where ethnographers and other qualitative social scientists learn to respect the methods and findings of their quantitative brethren such that they can target qualitative methods at illuminating the mechanisms driving robust causal relationships. At the same time, the quantitative researchers on these teams will have to accept that they have only partially explained what we need to know when they have established causality through their methods, and that qualitative research can carry their findings into the realm of implementation.
The bad news for everyone…for this to happen, you are going to have to pick your heads up out of your (sub)disciplinary foxholes and start reading across disciplines in your area of interest. Everyone talks a good game about this, but when you read what keeps getting published, it is clear that cross-reading is not happening. Seriously, the number of times I have seen people in one field touting their “new discoveries” about human behavior that are already common conversation in other disciplines is embarrassing…or at least it should be to the authors. But right now there is no shame in this sort of thing, because most folks (including peer reviewers) don’t read outside their disciplines, and therefore have no idea how absurd these claims of discovery really are. As a result, development studies gives away its natural interdisciplinary advantage and returns to the problematic structure of academic knowledge and incentives, which not only enable, but indeed promote narrowly disciplinary reading and writing.
Development donors, I need a favor. I need you to put a little research money on the table to learn about whatever it is you want to learn about. But when you do, I want you to demand it be published in a multidisciplinary development-focused journal. In fact, please start doing this for all of your research-related money. People will still pursue your money, as the shrinking pool of research dollars is driving academia into your arms. Administrators like grant and contract money, and so many academics are now being rewarded for bringing in grants and contracts from non-traditional sources (this is your carrot). Because you hold the carrot, you can draw people in and then use “the stick” inherent in the terms of the grant/contract to demand cross-disciplinary publishing that might start to leverage change in academia. You all hold the purse, so you can call the tune…
*Spoiler alert: you can’t. Well, you probably can if 1) you pin the behavior you want to explain down to something extraordinarily narrow, 2) can limit the environmental effect in question to a single independent biophysical process (good luck with that), and 3) limit your effort to a few people in a single place. But at that point, the whole reason for understanding the environmental determinant of that behavior starts to go out the window, as it would clearly not be generalizable beyond the study. Trust me, geography has been beating its head against this particular wall for a century or more, and we’ve buried the idea. Learn from our mistakes.
**by “more pure” I am thinking about those branches of physics, chemistry, and biology in which lab conditions can control for many factors. As soon as you get into field sciences, or starting asking bigger questions, complexity sets in and things like causality get muddied in the manner I discuss below…just ask an ecologist.
This paper is currently available for review at The Winnower.
Wed 15 Aug 2012
Alright, last post I laid out an institutional problem with M&E in development – the conflict of interest between achieving results to protect one’s budget and staff, and the need to learn why things do/do not work to improve our effectiveness. This post takes on a problem in the second part of that equation – assuming we all agree that we need to know why things do/do not work, how do we go about doing it?
As long-time readers of this blog (a small, but dedicated, fanbase) know, I have some issues with over-focusing on quantitative data and approaches for M&E. I’ve made this clear in various reactions to the RCT craze (see here, here, here and here). Because I framed my reactions in terms of RCTs, I think some folks think I have an “RCT issue.” In fact, I have a wider concern – the emerging aggressive push for quantifiable data above all else as new, more rigorous implementation policies come into effect. The RCT is a manifestation of this push, but really is a reflection of a current fad in the wider field. My concern is that the quantification of results, while valuable in certain ways, cannot get us to causation – it gets us to really, really rigorously established correlations between intervention and effect in a particular place and time (thoughtful users of RCTs know this). This alone is not generalizable – we need to know how and why that result occurred in that place, to understand the underlying processes that might make that result replicable (or not) in the future, or under different conditions.
As of right now, the M&E world is not doing a very good job of identifying how and why things happen. What tends to happen after rigorous correlation is established is what a number of economists call “story time”, where explanation (as opposed to analysis) suddenly goes completely non-rigorous, with researchers “supposing” that the measured result was caused by social/political/cultural factor X or Y, without any follow on research to figure out if in fact X or Y even makes sense in that context, let alone whether or not X or Y actually was causal. This is where I fear various institutional pushes for rigorous evaluation might fall down. Simply put, you can measure impact quantitatively – no doubt about it. But you will not be able to rigorously say why that impact occurred unless someone gets in there and gets seriously qualitative and experiential, working with the community/household/what have you to understand the processes by which the measured outcome occurred. Without understanding these processes, we won’t have learned what makes these projects and programs scalable (or what prevents them from being scaled) – all we will know is that it worked/did not work in a particular place at a particular time.
So, we don’t need to get rid of quantitative evaluation. We just need to build a strong complementary set of qualitative tools to help interpret that quantitative data. So the next question to you, my readers: how are we going to build in the space, time, and funding for this sort of complementary work? I find most development institutions to be very skeptical as soon as you say the words qualitative…mostly because it sounds “too much like research” and not enough like implementation. Any ideas on how to overcome this perception gap?
(One interesting opportunity exists in climate change – a lot of pilot projects are currently piloting new M&E approaches, as evaluating impacts of climate change programming requires very long-term horizons. In at least one M&E effort I know of, there is talk of running both quantitative and qualitative project evaluations to see what each method can and cannot answer, and how they might fit together. Such a demonstration might catalyze further efforts…but this outcome is years away)
Tue 26 Apr 2011
You know, qualitative social scientists of various stripes have long complained of their marginalization in development. Examples abound of anthropologists, geographers, and sociologists complaining about the influence of the quantitatively-driven economists (and to a lesser extent, some political scientists) over development theory and policy. While I am not much for whining, these complaints are often on the mark – quantitative data (of the sort employed by economists, and currently all the rage in political science) tends to carry the day over qualitative data, and the nuanced lessons of ethnographic research are dismissed as unimplementable, ideosyncratic/place-specific, without general value, etc. This is not to say that I have an issue with quantitative data – I believe we should employ the right tool for the job at hand. Sadly, most people only have either qualitative or quantitative skills, making the selection of appropriate tools pretty difficult . . .
But what is interesting, of late, is what appears to be a turn toward the lessons of the qualitative social sciences in development . . . only without actually referencing or reading those qualitative literatures. Indeed, the former quantitative masters of the development universe are now starting to figure out and explore . . . the very things that the qualitative community has known for decades. What is really frustrating and galling is that these “new” studies are being lauded as groundbreaking and getting great play in the development world, despite the fact they are reinventing the qualitative wheel, and without much of the nuance of the current qualitative literature and its several decades of nuance.
What brings me to today’s post is the new piece on hunger in Foreign Policy by Abhijit Banerjee and Esther Duflo. On one hand, this is great news – good to see development rising to the fore in an outlet like Foreign Policy. I also largely agree with their conclusions – that the poverty trap/governance debate in development is oversimplified, that food security outcomes are not explicable through a single theory, etc. On the other hand, from the perspective of a qualitative researcher looking at development, there is nothing new in this article. Indeed, the implicit premise of the article is galling: When they argue that to address poverty, “In practical terms, that meant we’d have to start understanding how the poor really live their lives,” the implication is that nobody has been doing this. But what of the tens of thousands of anthropologists, geographers and sociologists (as well as representatives of other cool, hybridized fields like new cultural historians and ethnoarchaeologists). Hell, what of the Peace Corps?
Whether intentional or not, this article wipes the qualitative research slate clean, allowing the authors to present their work in a methodological and intellectual vacuum. This is the first of my problems with this article – not so much with its findings, but with its appearance of method. While I am sure that there is more to their research than presented in the article, the way their piece is structured, the case studies look like evidence/data for a new framing of food security. They are not – they are illustrations of the larger conceptual points that Banerjee and Duflo are making. I am sure that Banerjee and Duflo know this, but the reader does not – instead, most readers will think this represents some sort of qualitative research, or a mixed method approach that takes “hard numbers” and mixes it in with the loose suppositions that Banerjee and Duflo offer by way of explanation for the “surprising” outcomes they present. But loose supposition is not qualitative research – at best, it is journalism. Bad journalism. My work, and the work of many, many colleagues, is based on rigorous methods of observation and analysis that produce validatable data on social phenomena. The work that led to Delivering Development and many of my refereed publications took nearly two years of on-the-ground observation and interviewing, including follow-ups, focus groups and even the use of archaeology and remotely-sensed data on land use to cross-check and validate both my data and my analyses.
The result of all that work was a deep humility in the face of the challenges that those living in places like Coastal Ghana or Southern Malawi manage on a day-to-day basis . . . and deep humility when addressing the idea of explanation. This is an experience I share with countless colleagues who have spent a lot of time on the ground in communities, ministries and aid organizations, a coming to grips with the fact that massively generalizable solutions simply don’t exist in the way we want them to, and that singular interventions will never address the challenges facing those living in the Global South.
So, I find it frustrating when Banerjee and Duflo present this observation as in any way unique:
What we’ve found is that the story of hunger, and of poverty more broadly, is far more complex than any one statistic or grand theory; it is a world where those without enough to eat may save up to buy a TV instead, where more money doesn’t necessarily translate into more food, and where making rice cheaper can sometimes even lead people to buy less rice.
For anyone working in food security – that is, anyone who has been reading the literature coming out of anthropology, geography, sociology, and even some areas of ag econ, this is not a revelation – this is standard knowledge. A few years ago I spent a lot of time and ink on an article in Food Policy that tried to loosely frame a schematic of local decision-making that leads to food security outcomes – an effort to systematize an approach to the highly complex sets of processes and decisions that produce hunger in particular places because there is really no way to get a single, generalized statistic or finding that will explain hunger outcomes everywhere.
In other words: We know. So what do you have to tell us?
The answer, unfortunately, is not very much . . . because in the end they don’t really dive into the social processes that lead to the sorts of decisions that they see as interesting or counterintuitive. This is where the heat is in development research – there are a few of us working down at this level, trying to come up with new framings of social process that move us past a reliance solely on the blunt tool of economistic rationality (which can help explain some behaviors and decisions) toward a more nuanced framing of how those rationalities are constructed by, and mobilize, much larger social processes like gender identification. The theories in which we are dealing are very complex, but they do work (at least I think my work with governmentality is working – but the reviewers at Development and Change might not agree).
And maybe, just maybe, there is an opening to get this sort of work out into the mainstream, to get it applied – we’re going to try to do this at work, pulling together resources and interests across two Bureaus and three offices to see if a reframing of livelihoods around Foucault’s idea of governmentality can, in fact, get us better resolution on livelihoods and food security outcomes than current livelihoods models (which mostly assume that decisionmaking is driven by an effort to maximize material returns on investment and effort). Perhaps I rest too much faith on the idea of evidence, but if we can implement this idea and demonstrate that it works better, perhaps we will have a lever with which to push oversimplified economistic assumptions out of the way, while still doing justice to the complexity of social process and explanation in development.