Wed 15 Aug 2012
Making sure we can learn from monitoring and evaluation…
Posted by Ed under Academia, development, Development Institutions, policy, research
[6] Comments
Alright, last post I laid out an institutional problem with M&E in development – the conflict of interest between achieving results to protect one’s budget and staff, and the need to learn why things do/do not work to improve our effectiveness. This post takes on a problem in the second part of that equation – assuming we all agree that we need to know why things do/do not work, how do we go about doing it?
As long-time readers of this blog (a small, but dedicated, fanbase) know, I have some issues with over-focusing on quantitative data and approaches for M&E. I’ve made this clear in various reactions to the RCT craze (see here, here, here and here). Because I framed my reactions in terms of RCTs, I think some folks think I have an “RCT issue.” In fact, I have a wider concern – the emerging aggressive push for quantifiable data above all else as new, more rigorous implementation policies come into effect. The RCT is a manifestation of this push, but really is a reflection of a current fad in the wider field. My concern is that the quantification of results, while valuable in certain ways, cannot get us to causation – it gets us to really, really rigorously established correlations between intervention and effect in a particular place and time (thoughtful users of RCTs know this). This alone is not generalizable – we need to know how and why that result occurred in that place, to understand the underlying processes that might make that result replicable (or not) in the future, or under different conditions.
As of right now, the M&E world is not doing a very good job of identifying how and why things happen. What tends to happen after rigorous correlation is established is what a number of economists call “story time”, where explanation (as opposed to analysis) suddenly goes completely non-rigorous, with researchers “supposing” that the measured result was caused by social/political/cultural factor X or Y, without any follow on research to figure out if in fact X or Y even makes sense in that context, let alone whether or not X or Y actually was causal. This is where I fear various institutional pushes for rigorous evaluation might fall down. Simply put, you can measure impact quantitatively – no doubt about it. But you will not be able to rigorously say why that impact occurred unless someone gets in there and gets seriously qualitative and experiential, working with the community/household/what have you to understand the processes by which the measured outcome occurred. Without understanding these processes, we won’t have learned what makes these projects and programs scalable (or what prevents them from being scaled) – all we will know is that it worked/did not work in a particular place at a particular time.
So, we don’t need to get rid of quantitative evaluation. We just need to build a strong complementary set of qualitative tools to help interpret that quantitative data. So the next question to you, my readers: how are we going to build in the space, time, and funding for this sort of complementary work? I find most development institutions to be very skeptical as soon as you say the words qualitative…mostly because it sounds “too much like research” and not enough like implementation. Any ideas on how to overcome this perception gap?
(One interesting opportunity exists in climate change – a lot of pilot projects are currently piloting new M&E approaches, as evaluating impacts of climate change programming requires very long-term horizons. In at least one M&E effort I know of, there is talk of running both quantitative and qualitative project evaluations to see what each method can and cannot answer, and how they might fit together. Such a demonstration might catalyze further efforts…but this outcome is years away)
We combine quantitative and qualitative monitoring in what we work on. (Alas confidentiality requires I keep quiet about the details.) But I would go so far as to say that it is the only sensible way to do monitoring in projects working in the environment-development nexus since impact is necessarily some years away.
See http://bottomupthinking.wordpress.com/2012/06/06/social-entrepreneurs-and-me/ for some recent thoughts along similar lines to yours. Radical, experimental project design and development, unfortunately, is not particularly compatible with rigorous experimental design of monitoring programmes.
Ah, but then there is a place for academia – we need to take on the mantle of radical experimental project design, as this is our comparative advantage. This is a great message for academia!
Definitely a place for academia but I wouldn’t necessarily say in project design. Advisers to project design and in developing innovative monitoring techniques yes. But imho academics do not make great project managers. And the person who is going to be managing a project should take the lead on designing it, otherwise you get great designs ruined by uncomprehending implementation.
http://bottomupthinking.wordpress.com/2011/02/07/good-strategies-need-good-implementation/
Ah, but I think you underestimate us…we design and manage projects with rigorous M&E all the time. We just call it research! Sure, some academics aren’t good in teams, or with deadlines, but that is hardly everyone. I do think you are dead on about what happens when design and implementation are held separate…which is basically everything done by donors. I have described the difficulty from the donor side as follows: if hitting a baseball is one of the hardest things to do in all of sport, then when a donor designs a project and then tries to implement through a contractor or other implementer, it is like trying to tell a blindfolded person when to swing at a baseball. You’re just not going to get a hit very often…
I’m not sure you mean to highlight the relative stengths of quantitative vs. qualitative approaches to M&E, but rather empirical vs. theoretical approaches to M&E.
Empiricism tells you whether an intervention did or did not “work,” and *maybe* gets you part of the way to saying why. Theory tells you *why* something *might* or might not work, and generates testable hypotheses (or predictions) for the empirical types to go out and test.
So the “why” in “why did Intervention X work in Place Y” might be answerable by Theory A, but the predictions implied by Theory A could still be quantitative — a quantitative solution to a quantitiative problem.
This isn’t to say that qualitative M&E doesn’t matter — theory has to come from *somewhere* after all — but theory still generates quantitatively ,empirically testable predictions. So it seems to me the point you want to make is that *theory* still matters (a lot), not just empirical results.
(I also think the empirical revolution in the last 10-15 years was a reaction to an overly-theoretical, and overly-quantitative approach to development economics — but now even the empirical approach to M&E is viewed as very quantitative, apparently)
This is an interesting point, but I think the empiricism/theory divide here is problematic. What I want is something in-between. Empiricism is fine if we want to observe what happened. Theory is fine for thinking about why it might have happened…but we need something to stand between them and link the two. In short, it is not good enough to have empirical evidence and a theory of maybe why something happened – there is no need to limit ourselves to this, as we can use qualitative methods to establish why something happened. Hell, there isn’t a need for a priori theories of change if we use qualitative methods (though I think they are a good idea, as they are probably unavoidable – the choice is not between no theory and theory, but between acknowledged theories and unacknowledged theories) – we just need to spend time explaining why something happened. I have yet to see a quantitative method that could serve this purpose.
I do think theory matters a lot…but I like my theories to be derived from extensive work explaining empirical results – this has been my method throughout my career, and it has led me to some really interesting places.
And I do strongly agree with you that empirical has been conflated with quantitative in recent years – of course these are not necessarily the same, nor are they exclusive, but in practice nearly all “empirical” M&E is quantitative in nature. I’ve made arguments within USAID that the evaluation efforts of the Agency need to avoid this conflation, or they won’t work out…