Microfinance is one area of development intervention that has experienced increased use of randomised control trials (RCTs) in the last few years, now seen by many as the ‘gold standard’ methodology for assessing impact. The gold standard approach to telling us what the findings impact studies collectively amount to is the systematic review. Since great claims for poverty reduction have been made for the impact of microcredit programmes in the past, the demand for evidence in this field is very high. Two recent DFID funded systematic reviews in this field have between them been downloaded more than 15,000 times, whereas DFID funded systematic reviews on other topics have had a few hundred downloads at most.
These reviews (including one to which my colleague James Copestake contributed – Duvendack et al, 2011) created a bit of a furore in the sector by setting the standard of rigour so high that relatively few studies were included and even those that were came in for robust methodological criticism. The conclusion from this and the other reviews is that the case for microcredit having impacts on raising incomes is rather weak. Scientifically put: the null hypothesis of “no change” cannot be rejected. The evidence for micro-savings is seen as emerging and possibly positive and similarly for payments systems.
For the moment I will not go into issues around RCTs as a methodology or the questions of whether these studies are actually telling us what we really need to know about impact. For now let’s just ask how this broad finding is to be used? At a meeting on Impact Assessment for Financial Inclusion held by CGAP and DFID in January, which James Copestake and I participated in, Stefan Dercon (DFID’s Chief Economist) argued that at present policy decisions tend to be made on the basis of a compelling case based in theory which uses good examples to make its point. He argued that the ‘hard core’ argument that policy decisions should only be based on evidence is really unachievable due to (i) frequent impurities in the ‘gold standard’; (ii) the difficulties of generalising their results and (iii) the fact that the evidence base will always be incomplete. So, quoting Keynes “when the facts change, I change my mind”, he suggested moving towards an “error correction” model which particularly involves changing course when no evidence of impact is produced after a number of attempts at finding it.
This therefore suggests that we are getting to the point in the microcredit impact debate where, once a few more RCTs have demonstrated rather limited evidence for impact on poor people’s incomes, then its promotion should be scaled down.
So will this change policy? Well interestingly the policy on this has already changed and the microfinance sector has moved on due to (i) broader non-RCT evidence that the benefits of financial inclusion for the poor are indirect via growth and jobs rather than credit for self-employment; (ii) the high cost of building specialist MFIs relative to working across the financial sector as a whole (see Johnson, 2009). The new focus on financial inclusion sees financial services as an infrastructural necessity for growth and the development of markets more broadly, and a means for preventing widening inequality. And assessing multi-stranded sector-level interventions presents different methodological challenges. So it seems that the policy horse had bolted long before the “randomista” evidence gate started to shut!
So is this likely to be a recurring phenomenon? That is, does the time it takes to generate the evidence mean that it is quite likely that policy will have moved on before the results are in – for a range of other reasons? Well possibly yes, although the famous IFPRI study of the Mexican conditional cash transfer programme, Opportunidades, does suggest RCTs can be influence momentum for replication elsewhere.
In microfinance at the moment the RCT surge, along with the rise of behavioural economics, is leading to a number of RCTs being carried out on savings behaviour. So these may lead rather than lag the focus on savings product development and we may see a different pattern in this case. However, this example is a cautionary one. It suggests that evidence from RCTs is not going to be the only factor that moves policy and the “error correction” model of policy making requires broadening to understand where evidence fits relative to other factors in the policy decision-making mix (see for example the upcoming conference on the Politics of Evidence). I cannot help but wonder then whether the policy of evidence-based policy making will itself have moved on by the time we have sufficient evidence from which to decide how it works!