Those following this blog (or my twitter feed) know that I have some issues with RCT4D work. I’m actually working on a serious treatment of the issues I see in this work (i.e. journal article), but I am not above crowdsourcing some of my ideas to see how people respond. Also, as many of my readers know, I have a propensity for really long posts. I’m going to try to avoid that here by breaking this topic into two parts. So, this is part 1 of 2.
To me, RCT4D work is interesting because of its emphasis on rigorous data collection – certainly, this has long been a problem in development research, and I have no doubt that the data they are gathering is valid. However, part of the reason I feel confident in this data is because, as I raised in an earlier post, it is replicating findings from the qualitative literature . . . findings that are, in many cases, long-established with rigorously-gathered, verifiable data. More on that in part 2 of this series.
One of the things that worries me about the RCT4D movement is the (at least implicit, often overt) suggestion that other forms of development data collection lack rigor and validity. However, in the qualitative realm we spend a lot of time thinking about rigor and validity, and how we might achieve both – and there are tools we use to this end, ranging from discursive analysis to cross-checking interviews with focus groups and other forms of data. Certainly, these are different means of establishing rigor and validity, but they are still there.
Without rigor and validity, qualitative research falls into bad journalism. As I see it, good journalism captures a story or an important issue, and illustrates that issue through examples. These examples are not meant to rigorously explain the issue at hand, but to clarify it or ground it for the reader. When journalists attempt to move to explanation via these same few examples (as far too often columnists like Kristof and Friedman do), they start making unsubstantiated claims that generally fall apart under scrutiny. People mistake this sort of work for qualitative social science all the time, but it is not. Certainly there is some really bad social science out there that slips from illustration to explanation in just the manner I have described, but this is hardly the majority of the work found in the literature. Instead, rigorous qualitative social science recognizes the need to gather valid data, and therefore requires conducting dozens, if not hundreds, of interviews to establish understandings of the events and processes at hand.
This understanding of qualitative research stands in stark contrast to what is in evidence in the RCT4D movement. For all of the effort devoted to data collection under these efforts, there is stunningly little time and energy devoted to explanation of the patterns seen in the data. In short, RCT4D often reverts to bad journalism when it comes time for explanation. Patterns gleaned from meticulously gathered data are explained in an offhand manner. For example, in her (otherwise quite well-done) presentation to USAID yesterday, Esther Duflo suggested that some problematic development outcomes could be explained by a combination of “the three I s”: ideology, ignorance and inertia. This is a boggling oversimplification of why people do what they do – ideology is basically nondiagnostic (you need to define and interrogate it before you can do anything about it), and ignorance and inertia are (probably unintentionally) deeply patronizing assumptions about people living in the Global South that have been disproven time and again (my own work in Ghana has demonstrated that people operate with really fine-grained information about incomes and gender roles, and know exactly what they are doing when they act in a manner that limits their household incomes – see here, here and here). Development has claimed to be overcoming ignorance and inertia since . . . well, since we called it colonialism. Sorry, but that’s the truth.
Worse, this offhand approach to explanation is often “validated” through reference to a single qualitative case that may or may not be representative of the situation at hand – this is horribly ironic for an approach that is trying to move development research past the anecdotal. This is not merely external observation – I have heard from people working inside J-PAL projects that the overall program puts little effort into serious qualitative work, and has little understanding of what rigor and validity might mean in the context of qualitative methods or explanation. In short, the bulk of explanation for these interesting patterns of behavior that emerges from these studies resorts to uninterrogated assumptions about human behavior that do not hold up to empirical reality. What RCT4D has identified are patterns, not explanations – explanation requires a contextual understanding of the social.
Coming soon: Part 2 – Qualitative research and the interpretation of empirical data
I think you are talking about RCTs done badly. I don’t think any researchers deny the importance of theory.
Having said this I do think there is still considerable value in pure data collection and rigorously establishing causal empirical relationships. And there is often a division of labor, with some researchers working on theory and others on evidence and the two literatures complementing each other, even if the individuals involved aren’t directly collaborating or co-publishing.
We’ve had theory for hundreds of years, I don’t think we are even close to the point of having too much quantitative data.
In terms of evaluation – the leading donors are probably IDB and MCC, of whose 30% and 50% of their portfolios have some kind of rigorous evaluation attached to them.
Lee:
You are probably right . . . in some ways. Obviously Duflo and Banerjee were responding to editorial pressure in the writing of their book. They know things are more complex than they present. But that said, the RCT work I have seen does not build a rigorous interpretive frame for the data (this will be the second part of the series of posts) around social theory. This is where the need for really rigorous qualitative work comes in to RCT work. I think you are right about the division of labor (I was going to propose something like this) but you can’t really do that in an uncoordinated way – i.e. drawing on the literature – because 1) most of these interpretive frames don’t exist at a scale or a place-specificity that allows for the rigorous interpretation of the RCT results, which means your qualitative/causal stuff will not necessarily be valid for the data at hand and 2) the throw-it-over-the-wall model limits the iterative learning that can go on for both quant and qual sides of this equation as the research moves forward. So even good RCT work has serious challenges in this regard.
I am not at all convinced that the RCT crowd really understands qualitative work, what we think causality is, and how we think about/build theory. Then again, I would submit that much of the qualitative community would have trouble with causality and theory building, so perhaps this is our fault.
Incidentally, we have indeed had theory for a long time – but when it comes to the social, we are always retheorizing to better capture observed outcomes. Thus my work reframing and retheorizing livelihoods strategies – turns out our framing of them as mostly about material outcomes is wrong, so I had to come up with a means of explaining why people do what they do – that required data (in my case qualitative). So yes, we need a lot more empirical work – and we need it to inform theory, and for the interpretation of the data gathered through empirical work to be informed by theory.
Thanks for the interesting post. I find it fascinating that these same discussions come up over and over in the different worlds I inhabit (which include development where RCTs are newish, but also health where RCTs aren’t even questionned, and I also ‘lead’ on qualitative methods at the UK’s National Audit Office which employs 600 accountants). I’m neither a qualitative researcher, nor a quantiative one, I span both – problems span both, and we need solutions which do too.
What I’m learning is that the world needs specialists, who fight for rigour in their fields (or bits of fields), but it also needs people who specialise in crossing boundaries, whether disciplinary, or the policy/audit/research divides, or the qual quant dichotomy. I guess I’m saying that we need to keep challenging across all work for rigour and transparency, and encouraging those who specialise in any one area to at least acknowledge the value of the other.
But, let’s face it, we all speak different languages, and there is a shortage of people who can translate. Not sure I’m saying anything new, apart from to encourage different ‘sides’ in the debate to respect, listen and learn, and to be patient… not everyone is a good linguist, and not all linguists can specialise either.
Thanks for this comment. I could not agree more with your call for more people who can span different fields/methods. I’ve occupied just such a space in my department at South Carolina, where I work between qualitative social science (my home, as it were), biophysical science and the GISciences. I actually find working across these fields to be enormously stimulating and useful . . . but as Lee pointed out in the comments here, this is hard work and there are few incentives to get people to do it. I am trying to play the role of translator where I can . . . and it seems to me that you are doing so as well on a day-to-day basis. As long as we are working to put the specialists together in a constructive way (which I hope to do in part 2 of this post series), we are doing good in the world . . .
Yeah I think a lot of this is the challenge of cross-discipline communication. Personally I trained in economics, and my main issue with qualitative research is that, not being trained in the methods, I have no clue how to evaluate what is good research and what is bad. I think this probably applies to a lot of people. Cross-disciplinary work is an exciting buzzword but in reality it means that individuals need to be conversant in the methods of multiple fields, which lets face it, is really hard, even if anyone did have the incentive to do so, which they don’t.
(As a footnote, when I talk about theory I’m talking about formal mathematical modelling, of which there is quite a bit in papers with field experiments).
Agreed – communication and trust are key here. And thanks for the clarification on theory – yep, we mean different things when we each say theory, which just proves your point about becoming conversant in multiple fields . . .
Great post and a few comments –
1. You put your finger on a common ‘weakness’ of RCT work: that it fails to uncover the mechanisms which drive the results seen. Often, good RCT work – and there is much of it – tests various mechanisms. Look to Macartan Humphreys, Betsy Levi Palluck, and Don Green for just a few. More importantly though, a lot of the time RCTs don’t get at mechanisms, largely because it isn’t the intention. Rather, the intention is about measurement. Much of actual science is about measurement and validly attempting to estimate quantities of interest. The social sciences to rarely focus on this, which is of extreme importance in development where its all about how to efficiently and effectively allocate resources. RCTs often don’t understand why teachers might make a difference, but estimate the impact; knowing why teachers are important and what their impact is provide two fundamentally different insights. The former illuminates the mechanism and facilitates mechanism replication in development, the latter illuminates impact which facilitates efficient resource allocation. As such, both forms of RCTs are important.
2. The lack of qualitative work is a fair charge. But, economists aren’t in the business of qualitative work nor will they ever be. They have done an excellent job of advocating for a certain method to evaluation. I’ve yet to see qualitative folk try to do the same. Any thoughts on why and does the broader phenomenon point to your frustration rather than RCTs themselves?
Grant:
Thanks for this – you have my point exactly: I’m worried about causal mechanisms. In some ways, the work Duflo et al are doing doesn’t always call for this, as they operate on a small scale where they can do quick experiments to see “what works” without a conceptual framing. The problems come when they try to explain why “what worked” actually worked – in other words, when they start making generalizable claims.
Your distinction in the use of RCTs is great – I’d not thought of this at all, but of course you are quite right. And both uses are valid . . . but I think sometimes the impact studies presume to know why the impact occurred, thus presuming a mechanism. I will most certainly look up the folks you have referenced, as I would like to see good RCT work – I have no intention of tarring the whole area of inquiry with an inappropriate brush. I’d prefer my engagement be constructive.
Ah, your question of why the qualitative folks haven’t come up with a method . . . it’s a great question, and one that I feel uncomfortable answering for “my people”, as it were. In my opinion, I think that qualitative research has an uncomfortable relationship with the ideas of rigor and validity – perhaps out of ignorance for how this might be established, and perhaps in part because our “hard science” colleagues often accuse us of having no data (really, this happened to me in a faculty meeting once) or using data for which there is no means of establishing validity. This, of course, is crap, but I think a combination of these factors often drives this conversation underground. Then, of course, there are those who will argue that efforts to evaluate and systematize data do violence to the lived experiences of our subjects, etc., and might reduce these experiences such that we lose key information. I don’t buy that personally, but some do. Finally, I think causality in the qualitative social sciences is highly contingent, place and temporally specific – so it makes it hard to come up with a general method. I am working on building an approach to livelihoods analysis right now, where the approach might be generalizable but the results will still reflect local specificity. It is really hard. We’ll see if it gets through peer review, and if it does, whether anyone runs with it.
In the end, perhaps you have put your finger on a bit of what bothers me – the larger phenomenon of “failure to communicate” and, to be completely honest, how when such failures occur the economic approaches seem to trump qualitative findings, at least in the popular consciousness (though this often occurs in practice as well). This will require education (on both sides of this divide), and I hope to offer some constructive ideas in part 2 of these posts and in conversations going forward. In the end, I just want us to do our work better . . .
Maybe there’s also a challenge here too to be humble. I have to admit to my economist colleagues that I’m not trained in economics, and to my accountant colleagues that I’m not an accountant (they actually formally class me as ‘unqualified’ in that particular office). I’ve also use quant methods, I know some of the language, I passed all my stats exams (and already I begin to feel defensive – not very good at humility). But it would never have occurred to me to define theory in terms of mathematical modelling (thank you Lee – I’ve learnt something today), so I definitely don’t know everything. To speak to RCTers, I have to admit that although I share an office with people who run trials, there are aspects of them I don’t really get. To qual researchers, I have to confess that my experience is limited.
My approach when I work in a new country, is to assume I know nothing, earnestly ask how to say things in their language and try to stay humble. If only I could be better at doing this in my work where to be challenged professionally always hurts more.
Ed, I look forward to your paper on a generalisable framework for qual methods. Do you know about the work done in this area by systematic reviewers who critically appraisal qualitative work (thus trying to establish standards of rigour)? If not, you might find some of the work of the EPPI-Centre relevant, and also Prof Angela Harden here in London.
This is my mantra – maybe it will help:
(when challenged in my work)…
“I am not a fool. So maybe there’s something more I can learn about this. Or maybe they are fools and don’t realise that I already do know about this. I’d love to tell them they are fools, but it won’t help. Instead I’ll try to ask them to help me understand (and if that’s too hard, keep quiet).
I find “qualitative” to be a very hard word to use well because it covers far too many different ways of doing and thinking about research. I am dong quantitative and qualitative work. My qualitative work is based on country case studies where I don’t have enough data to do a quantitative analysis, so instead I essentially ask “If I am right, then what things would I expect to see? What would I expect to not see?” I can answer these kinds of questions with interviews or crude more/less comparisons or textual analysis… Qualitative work also lets me ask questions about process, “If things worked in the way that I expect, then what should I expect to see and in what sequence?” This kind of approach is my favourite, as I can try to test a causal relationship with a regression and then explore the mechanism with qualitative work. The two work well together. For example, if I can’t find my expected mechanism on the qualitative side, then it is more likely that my regression is spurious.
It is hard to talk about this without getting into my research, but my point is that this kind of qualitative work fits well alongside quant work and is already fairly common. However, a lot of people who do “qualitative” work don’t do this.
For me, the real divide is interpretive vs causal work. No matter my method, I am trying to understand causation. A lot of qualitative people aren’t.
Hi Ed,
I read your post in Google Reader, and wanted to make a comment. Once here in your blog, I found that my concern was already worked out in the comments section. I was wondering about the word/concept of “theory” in your post, and its interlinks with RCT4D and beyond. Paying a quick look to the articles you linked (still waiting for the book, but it is near now), I have no doubt that you are also (very) interested in discussing theory / theories. This made me wonder why you didn´t mentioned it once in the post. I´m sure you enjoyed the comments (and you had the possibility also of expanding your thoughts with no guilt feelings of being too long in the post:)