Experiments, Natural Experiments and Learning about Development Policy - I

A while back I blogged about the Jameel Poverty Action Lab, a non-profit organization started and run by economists carrying out randomized field experiments all over the developing world. The purpose of these experiments is to build an evidence base to inform policy-making, and randomization as a tool towards this end has become quite popular of late. Proponents of randomization, now called "randomistas", argue that, as with medical clinical trials, field experiments are the "gold standard" in development policy evaluation.

But is this really so? In two recent pieces, Angus Deaton and Martin Ravallion argue that the answer is "no." One of their main arguments centers around the idea of heterogeneity in treatment effects, which basically refers to how policies do not have the same impacts for everyone. Consider an example where we are thinking about implementing some large policy and want to learn whether it might be effective. To do so, we consult data from a recent experiment in which some individuals in the sample have been randomized to receive "treatment." We then compare the treatment and control group outcomes.

Randomization of individuals to treatment gives us confidence that the results of the experiments are not biased. However, the concern is whether one can learn something useful about the policy from this experiment. In most field experiments, individuals in the treatment group are either enrolled in a program or incentivized to participate in some way. In most cases, not everyone complies, and some groups of individuals tend to be more likely to comply than others.

The important thing to note is that the program effects that are recovered from the experiment is most reflective of the returns to the group of compliers. This is referred to as a "local average treatment effect", or LATE. Here is where the problem comes in: the LATE that an experiment recovers may not always be policy relevant and, unlike the issue of limited external validity (experimental results in one setting may not apply to others), it is not clear that replications will help get around this problem. To reiterate, the benefits of the program that infer from an experiment may or may not be informative about this program on a larger scale.

Ultimately, this is problem of experiments being "atheoretical." That is, simply looking at experimental averages is not enough: we have to understand who in the treatment group actually responds to the randomization and takes up treatment and whether this group is of interest to the broader policy picture. Building this understanding brings us back to economic theory: we need a model. In this sense, the argument goes, proponents of randomization who argue that field experiments are "easy" by obviating the need for models or (strong) assumptions are badly mistaken.

I find this argument compelling. Indeed, there is a parallel literature in the "natural experiments" world that makes similar points. Ultimately, policy design and resource allocation decisions require a great deal of information, only some of which we can get from randomized experiments. Experiments that incorporate theory and heterogeneity, Deaton argues, will be good step towards making the method more useful towards policy decisions. In the next post, I will list a few examples of experimental and quasi-experimental studies that take an approach more grounded in theory.