Be Careful With Natural Experiments

Those of you who visit this space regularly know a thing or two about my obsession with causal effects. Answering many questions in health economics requires a strategy to understand the causal effect of one variable on another, and recovering such effects demands clever strategies or tools that go beyond simple multivariate models of some X on some Y. The cleanest way to get the causal effect of X on Y is to randomize X. This strategy has been used widely in laboratory and clinical medicine, and is now being exploited in a growing number of influential papers in economics and political science.

In many cases, however, it may not possible to randomize X or the question of interest involves some program or event that occurred in the past. In these situations, researchers looks for other sources of variation in X that are effectively random - the natural experiment. A good (and now famous) example from health economics involves the impact of early life events on health and socioeconomic position later in life. A great deal of early work in epidemiology found links between the disease environment faced by an individual at birth and this individuals health later in life. This link could be causal: fetal health influences organogenesis and development that goes on to influence adult health. At the same time, alternate explanations are possible: children born to poor parents become poor themselves, which affects their health. To complete the story, poor parents may tend to reside in poor, diseased areas.

To get around this issue, Douglas Almond, in an influential 2006 paper, utilized the influenza pandemic of 1918, which struck without warning, over a short period of time, and had large, notable effects. Being exposed to influenza in utero can be thought of as a random shock (a natural experiment), and Almond took advantage of this property to derive the causal effects of health in utero on outcomes later in life.

However, while the "influenza strategy" is as close to a slam dunk as you could possibly get in observational research, other things that may seem like natural experiments a priori may not be as definitively good. In fact, such variation may even lead researchers astray.

A new NBER working paper explores this issue in detail. Kasey Buckles and Daniel Hungerman consider the case of season of birth, which has been shown to be associated with a variety of health and socioeconomic outcomes later in life. These associations have been attributed to fetal exposure to different weather conditions or differential exposure to arbitrary age-cutoffs (in sports or in schooling). At first glance, season of birth appears to be a great source of exogenous variation for a slew of different causal questions: after all, individuals don't have any control over when they are born and it seems like something that would be left to chance. However, Buckles and Hungerman convincingly argue that this is not the case:

In this paper we consider a new explanation: that children born at different times in the year are conceived by women with different socioeconomic characteristics. We document large seasonal changes in the characteristics of women giving birth throughout the year in the United States. Children born in the winter are disproportionally born to women who are more likely to be teenagers and less likely to be married or have a high school degree. We show that controls for family background characteristics can explain up to half of the relationship between season of birth and adult outcomes. We then discuss the implications of this result for using season of birth as an instrumental variable; our findings suggest that, though popular, season-of-birth instruments may produce inconsistent estimates. Finally, we find that some of the seasonality in maternal characteristics is due to summer weather differentially affecting fertility patterns across socioeconomic groups.

This is a neat paper and serves as a good warning to those interested in finding natural variation to identify causal effects. Another excellent paper on the same subject, by Mark Rosenzweig and Ken Wolpin, goes through a variety of other potentially fallacious natural experiment examples and is a must read for anyone doing empirical work.