Picking up from Andrew Gelman's recent post on why so much of what we hear about the replication crisis seems to center on social psychology, My take is that we should talk more about our old friend hype and the way it distorts discussions and corrupts important processes.
When I started to write out a reply in the comment section, it struck me that there was a pretty good analogy here with movie pitches (and obviously, if I come up with a pretty good analogy, I'm going to save it for a blog post). Researchers in today's environment have a strong incentive to generate buzz around their work.
In a sense, you can think of a paper's abstract as a pitch directed at reporters. Though the scale is very different, the basic dynamic is similar to that of a producer approaching a studio. Both Disney and the New York Times are looking for story ideas that will bring in large audiences, both have limited time and attention, and both are going to make the initial decision to proceed based on a quick summary of the story.
That intermediate step is very important. It means that the primary focus of both producer and researcher needs to be on the pitch more than on the quality or appeal of the larger story. Something that is easily pitched has an enormous advantage.
It is almost certainly not a coincidence that, when you see a big, underperforming movie come out, it's generally easy to imagine how the initial presentation went. "Look at those numbers for Night at the Museum. big-budget action comedy with lots of special effects and dinosaurs. We'll do the same thing with that 70s kids show, Land of the Lost . Old TV shows are big. Will Ferrell is big. We'll just put them all together." .
If you were a researcher trying to get a reporter to write up your study, what areas would "pitch well"? Three that come to mind are broad statements about human nature, dietary findings, and most of all "news you can use"/self-help. You'll notice that studies that fall into one or more of these categories tend to do very well in the hype market. As a consequence, there's also more of an incentive to push these through despite low-quality,
.
Of course, along with the hype comes increased scrutiny and a greater chance that someone will try to replicate the findings.
While I don't want to push the analogy too far or over into size one aspect of a complicated process, this might be a useful way of thinking about at least part of the problem.
One thing that continues to stun me is that the hype was so thick in the air that people didn't even recognize it.
ReplyDeleteConsider the notorious concluding sentence from the original "power pose" paper:
"That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications."
Let's set aside data problems and statistical problems with the paper, and of course the authors can't be blamed, in 2010, for not knowing that the experimental results would not show up in replications, years later.
My issue is more basic that that. My issue here is that there was nothing in the experiment about anyone "becoming more powerful," let alone "instantly" becoming more powerful, and nothing at all about any real-world implications, "actionable" or otherwise.
In short, the concluding sentence of the abstract of the paper was pure, unadulterated hype---and nobody caught it. Not the authors, not the journal editors, not the NPR reporters and fawning journalists over the years. Hype has so permeated scientific presentations that nobody noticed it, right in front of their face. Even now, that article appears in the journal with no correction notice.
Hype has become the standard mode of discourse in some areas of science, to the extent that even well-meaning authors, peer-reviewers, etc., just can't see it.
The rise of the hype-based economy has hugely troubling implications. The value of research, businesses, technology are all reduced to how much buzz you can generate (or buy). This threatens our economy, our public policy decisions and the quality of our scientific research.
DeleteA future that pins its hopes on hyperloops and bitcoins will not be bright.