The Devil’s in the Data

To know whether one is doing the right thing is a devilishly hard task. No matter how pure your intention or ingenious your intervention, it is never a simple matter to prove beyond doubt that you are having the impact you desire. The line between cause and effect is often far more blurred than one would wish, there is always the impact of external factors to consider (be they positive, negative or merely ‘noise’). The (unintentional) bias of those giving or receiving the intervention can play havoc with the outcomes, the list of confounding factors can literally be endless.

Enter the powerfully simplistic beauty of the randomised controlled trial. The genius of this scientific approach is grounded in the mathematics of large numbers. If you take two groups sufficiently large to represent the natural variation of any population you care to consider, and randomly assign one group to receive the intervention whilst the second are similarly monitored but without the intervention, then the slings and arrows of outrageous fortune can be allowed to cancel each other out and you can focus all your energies on figuring out whether what you are doing is making its mark or not.

The fundamental importance of this form of scientific enquiry is such that nothing in the field of modern medicine can hope for widespread adoption without proving itself in this way. The level of sophistication within these trials has grown to compensate for any number of additional discrepancies generated by the frailties of human nature. Members of the control group are typically given a placebo (a sugar pill or saline injection) to ensure that the patients own perceptions of treatment are equal. Equally, health care professionals are also often ‘blinded’ from whether they are giving out active drug or sugar pills, lest their own concious or unconscious communication to the patient change and thus make an impact on the outcome.

This emergence of ‘evidence based medicine’ has allowed us to move from the world of trying things because they look like they might work (think leeches at the turn of the century) to the acceptance and rejection of many modern medicines based solely on hard clinical data. Transitioning from the pharmaceutical industry, I was shocked by how infrequently this level of rigour was applied to the many interventions that are designed and funded in the development world. A beacon of light in this darkened space is Dr Esther Duflo and her Poverty Action Lab at Massachusetts Institute of Technology. You can watch her giving an overview of their work here.

Her team aim to bring all the rigour of scientific enquiry to social engineering in the hope that it might advise and inform and shape future policy. Gratifyingly, her work is already placing several well researched cats amongst the development pigeons. For example her team has shown that micro-finance in many cases does little to foster enterprise in the way its backers claim, at least in the short run. At a more pragmatic level she has shown that ensuring teachers show up to teach is a better strategy than paying students to learn and that the best time to encourage farmers to invest in fertiliser is just after they have harvested this years crop (and funds are abundant).

There are many who argue that it is impossible to apply this form of rigorous scientific enquiry to the whole spectrum of development and social engineering and are particularly aghast at the idea that you might ‘withhold’ a certain support from a needy community just to prove a point. Yet this to me is wooly brained development thinking at its very worst. The scale of the problems faced, the scarcity of resources and the quantity of work required to implement real change all speak to me of a profound need to know that every single intervention is proven beyond doubt to be worthwhile and of differentially greater impact than anything else we could be spending our money and time on.

So here’s to Dr Duflo and her team, may they defeat the sceptics and move us to a world of ‘evidence based development’ where the aid equivalents of leeches are consigned to the history books and we let rational evidence cut through the fluffy thinking and lead us to focussing our efforts only on those things that make a real and lasting difference.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: