The "replication crisis" is worth revisiting
On Saturday, The Globe and Mail ran an opinion piece by Susan Pinker, a psychologist, about a replication crisis in science. The past year and a half has brought hundreds of thousands of studies about COVID-19, along with an unusual level of enthrallment in science by the media and the public.
As Pinker points out, this wave of research highlights the importance of the principle that scientific advances are not based on single studies, but rather on the rigourous replication of findings. After sending Pinker's article to a colleague, he forwarded me a pre-COVID article on the same topic, which like Pinker's, shares interesting examples from psychology.
This replication principle of course holds true in mental health and addictions research. However, studies that startle us or findings that support our worldview are very tempting. I'm sure I've fallen for more than a few in my life.
In the end, it's the convergence of evidence from a number of settings that validates a finding. As well, the ways that the evidence does not converge on a finding can describe its limitations — for instance, the individuals who don't respond as theorized to a mental health treatment. These limitations can point the direction towards important future research.
As consumers of the science literature we need to be careful not to fall hard for that first study.
Kind of shocking to read that the more interesting and novel-sounding a conclusion, the more likely it is to be cited by other scientists and the media, despite being less likely to be replicated by other scientists!
Perhaps this crisis can also be seen as an opportunity to hold researchers and those who write about research more accountable for our conclusions?
Yes, Emma, I was really troubled by it too. I feel as a knowledge broker I have to be careful not to fall into this trap too.
Do you mind clarifying something? I'm not sure what you mean by "accountable for our conclusions".
Maybe I should rephrase -- instead of "accountable for our conclusions", I mean we should be careful about what we conclude and mindful about the potential consequences of conclusions that are flimsy.
Yes, I agree.
Interesting threat, [@mention:336116034136246360] and [@mention:337383271028434279]!