Small facts can cause big changes to conclusions

Last week I attended the NBER’s Economics of IT and Digitization conference at its Summer Institute. There were lots of interesting papers but I wanted to focus on one, in particular, that demonstrated how important seemingly small facts were for policy conclusions. This paper was by Jason Chan and Anindya Ghose and it explored the impact of Craigslist on sexually transmitted diseases (STDs).

The paper was quite clever in utilising CDC data on reported cases of AIDS and Syphilis and matching that with the differential introduction of Craigslist across US localities. Craigslist, it turns out, has a healthy number of personal ads and from the examples shown to us during the paper presentation, can be quite sexually explicit in their offerings and intent. The authors research question was whether Craigslist, by potentially facilitating more sexual activity, had also increased the prevalence of STDs.

The results were pretty striking. There appeared to be a pretty tight connection between new cases of AIDS and Syphilis and the spread of Craigslist. If Craigslist had caused this, it would suggest some undesirable consequences. Indeed, that is what the paper concluded — for instance, the new 6000+ AIDS cases each year were costing the US $94 million in additional health costs. Now, even if that is the case, that does not necessarily make a case for government intervention but it is worrying and surely would be worrying to Craigslist given its stated social mission.

I have to say that 99 percent of the 120 or so economists there found that causal link pretty convincing. That was until a question from the audience flipped everything. The problem was that the time lag between the introduction of Craigslist and the onset of AIDS was way too short. For the paper, it was one or two years. But in reality, even after you contract HIV, it is usually the case that AIDS does not follow for perhaps another decade. That medical fact alone caused everything to flip. Without that mechanism, other theories emerged. The dominant one was that this was a ‘reporting issue.’ Specifically, the people meeting on Craigslist were sensitive and knowledgable about STDs and may well have asked for proof that people were HIV or AIDS free. That may have led to more testing and more reporting on AIDS — particularly at an early stage — and that may be what this result was picking up.

We don’t know if this theory is correct but what is interesting is that, if true, it completely changes how one might view Craigslist. Rather than an instrument for poor health outcomes it may be an instrument for safer sexual activity. Of course, the Syphilis result may go either way. The point is that sometimes the set of theories that are in people’s mind as they assess empirical results really matters for the conclusions drawn. In this case, a small fact gave rise to a new theory and a major reinterpretation for everyone present.

There were lots of suggestions given to the authors as to how to deal with this and explore it further. But given that their initial finding had been widely reported a few months ago, it seems to me that it is worthwhile to post about the alternate, although equally unproven, theory here. Eventually as the paper finds its way through full peer review a consensus opinion may be reached.

[Update: one of the authors of the paper has pointed me to an additional result regarding HIV where the magnitudes appear to be stronger than those for AIDS although the strength of statistical significance falls. Of course, care still needs to be taken to work out if this is a ‘real’ or ‘reporting’ effect but it is of relevance here.]