What Social Science Does—and Doesn’t—Know
Our scientific ignorance of the human condition remains profound.
Unlike physics or biology, the social sciences have not demonstrated the capacity to produce a substantial body of useful, nonobvious, and reliable predictive rules about what they study—that is, human social behavior, including the impact of proposed government programs ...Another way of putting the problem is that we have no reliable way to measure counterfactuals—that is, to know what would have happened had we not executed some policy—because so many other factors influence the outcome. This seemingly narrow problem is central to our continuing inability to transform social sciences into actual sciences. Unlike physics or biology, the social sciences have not demonstrated the capacity to produce a substantial body of useful, nonobvious, and reliable predictive rules about what they study—that is, human social behavior, including the impact of proposed government programs.
The missing ingredient is controlled experimentation, which is what allows science positively to settle certain kinds of debates. How do we know that our physical theories concerning the wing are true? In the end, not because of equations on blackboards or compelling speeches by famous physicists but because airplanes stay up. Social scientists may make claims as fascinating and counterintuitive as the proposition that a heavy piece of machinery can fly, but these claims are frequently untested by experiment ...
That all sounds plausible, but I've been a social science stats geek since 1972, when the high school debate topic that year was education, so I'm aware that Manzi's implications are misleading.
First, while experiments are great, correlation studies of naturally occurring data can be extremely useful. Second, a huge number of experiments have been done in the social sciences.
Third, the social sciences have come up with a vast amount of knowledge that is useful, reliable, and nonobvious, at least to our elites.
For example, a few years, Mayor Bloomberg and NYC schools supremo Joel Klein decided to fix the ramshackle admissions process to the gifted schools by imposing a standardized test on all applicants. Blogger Half Sigma immediately predicted that the percentage of Asians and whites admitted would rise at the expense of blacks and Hispanics, which would cause a sizable unexpected political problem for Bloomberg and Klein. All that has come to pass.
This inevitable outcome should have been obvious to Bloomberg and Klein from a century of social science data accumulation, but it clearly was not obvious to them.
First, few programs can be shown to work in properly randomized and replicated trials. …
Second, within this universe of programs that are far more likely to fail than succeed, programs that try to change people are even more likely to fail than those that try to change incentives.
And third, there is no magic. Those rare programs that do work usually lead to improvements that are quite modest, compared with the size of the problems they are meant to address or the dreams of advocates.