Don’t Let it Bring you Down–Neil Young

I came across an article on how a lot of the positives of public health randomized control trials don’t necessarily translate to economics on Dan Gardner’s twitter (see left). Hard to not read it with that kind of endorsement. I have to say, I don’t think I share the enthusiasm.

I have written a lot about RCTs and am clearly a supporter of using them when possible though my context is epidemiology. I’m not an economist, but I’ve come across many development oriented RCTs that have clearly created readily applicable, useful knowledge.

The main criticism of RCTs from this blog post is something called “external validity” and, simply put, is this: just because an RCT works in one place doesn’t mean it will work in another. I can test, say, an educational intervention using an RCT in Inukjuak, Quebec but, if I find an effect, that doesn’t mean that intervention will have the same effect in Windhoek, Namibia. The criticism is entirely valid if you’re doing one of two things:

1) Conducting an RCT on a very specific portion of the population like left-handed nine-year-old boys who are hockey fans and live on a specific street. Clearly, assuming you even have a big enough sample size to being with, any result won’t apply to everyone (or even most people).

2) Looking for a one-size-fits-all solution to economic development. If you find an economic intervention works in one place, there’s no guarantee it’ll work in a different cultural context even, maybe, within the same economic context.

I don’t see how this makes RCTs entirely useless however even for economics. Clearly we have a range of possibilities for which RCTs can be used within populations. The proposed solution confuses me even more:

What does work in areas outside of public health? How is it possible to design, test, and implement effective solutions in environments where complexity and volatility are dominant? The general principle applies: Success requires adaptability as well as structure, flexibility as well as structure—a societal capacity to scale successful efforts combined with an ingrained practice of entrepreneurial exploration. As the uniquely insightful Mancur Olson wrote in his classic Power and Prosperity (pp. 188-189):

Because uncertainties are so pervasive and unfathomable, the most dynamic and prosperous societies are those that try many, many things. They are societies with countless thousands of entrepreneurs who have relatively good access to credit and venture capital.

What works in development, according to Olsen, is entrepreneurial exploration. Why? Because we don’t know what works.