Here’s a fellow claiming that he accurately predicted 20 of the 21 replication attempts in Camerer, Nosek et al’s new study:
Jonatan P
Genomics researcher
Aug 27Guess along about which social science studies will replicate
In the recent article Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015, the authors re-do 21 social science studies published in Nature and Science.
In this post I will read the summaries of those studies and guess which ones will replicate, before seeing the results.
My general heuristic is the following:
- If the study implies that social scientists are modern day wizards, that can easily manipulate people’s choices through priming or similar, it won’t replicate
I’ve been writing about priming for years.
- If the study results involves wishful thinking, such as that you can get great advantages from easy, nice sounding actions, it won’t replicate.
- If none of the above are the case, and especially if the result is in agreement with common sense, it will replicate
I will trim down to the nine this guy predicted will fail to replicate. You try to guess which one did replicate.
Participants that evaluate a resume while using a heavier clipboard will rate the resume as better overall compared to the participants that evaluate the resume while using a lighter clipboard.
My guess: Social scientists are modern day wizards, won’t replicate. …
6. Priming analytic thinking via images of ’The Thinker’ increases religious disbelief compared to viewing control images of a visually similar artwork
My guess: Easy wizardry changing deeply held convictions, definitely won’t replicate. …
11. Reading literary fiction improves affective Theory of Mind
My guess: Wishful thinking, definitely won’t replicate. People already have a lifetime of experience with theory of mind, reading a little extra will not make a measurable difference.
My assumption is that the development of the novel in the 18th Century enabled Europeans as a culture to get more easily into the heads of other people. The novel was just better suited than previous literary genres for helping you imagine like what it would feel like to be a castaway on a desert island or whatever.
But can you really test this theory with an experiment in which subjects read modest-length passages and then take a test?
13. Hand washing will significantly reduce the need to justify one’s choice by increasing the perceived difference between alternatives.
My guess: Psychological magical thinking, won’t replicate.
14. Repeatedly imagining eating a food subsequently reduces the actual consumption of that food
My guess: Easy-ish solution to hard problem, won’t replicate.
17. In a high-pressure in-lab math test, those writing for 10 minutes about their deepest thoughts and feelings regarding the upcoming test improve more on that test compared to simply sitting quietly
My guess: Won’t replicate
18. Priming people to trust their intuition increases cooperation in a public goods game compared to priming people to reflect.
My guess: Social scientists are wizards, won’t replicate
19. Low-wealth subjects, that are given fewer chances to win in repeated ’Wheel of Fortune’ type word puzzle games, perform worse in a subsequent attention task (Dots-Mixed task) than do high-wealth individuals
My guess: Not immediately obvious to me, but it looks like an interaction effect, that was possibly only investigated after the main effect wasn’t significant. I then looked up the p-value in the paper, and it was 0.046, so I will now say with certainty that it will not replicate.
20. Computer terms are more accessible than general words after answering a block of hard trivia questions; measured as longer color-naming reaction times in a Modified Stroop Task after priming with computer terms compared to priming with non-computer terms
My guess: Don’t quite understand, but there is a lot about priming, so it won’t replicate.
To find out which of the nine did replicate, go to the original source.
