This issue is important enough to warrant a second posting. In what couldn't be better timing (with respect to my post yesterday), Alex Tabarrok at the Marginal Revolution blogs this morning about "Why Most Published Research Findings are False."
This short post references an illuminating earlier thread. The latter is a great read with some really excellent links. The discussion cites a 2005 PLoS article (of the same name) by John Ioannidis, which goes through the nitty gritty of why so many falsehoods exist. Here is how The New Scientist distills his findings:
Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.
John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.
"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.
My analysis goes through study design related reasons for why these studies are bad and only briefly (and indirectly) touches on things like publication bias and selective reporting. In that sense, Ioannidis' paper is a more complete discussion of the issues raised in my earlier post. Furthermore, the Ioannidis critique discusses bodies of research as a whole rather than individual studies, and his discussion has important implications of what we can and cannot learn from meta-analysis. I highly recommend this paper both for its own merits as well as a complement to my rant yesterday. Enjoy!
5 comments:
yes, when i start reading a paper i assume that the 'conclusion' is incorrect and hopefully the authors will surprise me. usually the data is sound but there are methodological and technical limitations that prevent their generalization beyond the subject population. BUT, often these limitations are actually listed and cited in the paper. The problem occurs when the lay-press casually glances at the title of the paper, or possibly the abstract, and extrapolates that paper X says result Y irrefutably. Most studies have limitations, but on the nightly news you will never hear that. You will hear the correlation between random disease X and random behavior Y without any caveats. That drives the layperson (who should have more statistical knowledge, i agree) to assume that if the story is on the news (usually from JAMA), it must be a scientific fact. [I am likely just as guilty as everyone else - if you don't make broad generalizations about your data, you are unlikely to publish in a high profile journal, unfortunately.]
Neil,
Thanks for your comment. I completely agree with you that:
a) Important statistical points are lost in translation between the press and the public, and
b) The relies on the press to make certain judgments and distill information for them.
I also agree that these papers do cite some methodological and technical limitations. However, authors do not always cite ALL of them. For example, the critiques I made on randomness and data mining are hardly ever acknowledged in papers. The causation-correlation point aside, my suspicion is that these sorts of phenomena can explain the vast majority of results in bad "armchair epidemiology" studies.
Furthermore, few if any studies acknowledge that the body of evidence (i.e., other studies that they cite) may be biased as well due to publication bias, etc.
In the last sentence of your comment you allude to one reason for this: probability of publishing in a high profile journal. I think incentives generated by the publication and tenure processes encourage researchers to be less vigilant (or even blissfully ignorant) of statistical procedures and the data generating process.
Ultimately, though, I believe that a lot of this research is not done properly, and, press and public aside, some of the burden here lies in the research community. Also, my comment is limited to armchair epidemiology...scientific medical research is probably done much more responsibly.
I'd have to agree with your points overall. There is a lot of 'incomplete' research that is biased or lacking in statistical validity in the literature. Much of this burden rests on the research community and appropriate peer review and assessment of good quality research.
Unfortunately, the amount of back-room dealings and quid pro quo's in the research world is quite alarming. A lot of good research is ignored and bad research published based upon name recognition, 'helping' a friend's research in return for a lenient critique on a subsequent grant, etc. (and yes, grad school has evidently jaded me even further:)
but all that aside, the mistakes of current researchers drive future research. at any point in time, 50% of scientific studies will be later proven incorrect. I cannot speak to epidemiological studies directly - but I assume a paper that associates shoes with cancer will drive 10 studies to disprove that ridiculous finding.
a better, more equitable and transparent system of peer review would help decrease the number of bad studies published. and having good statistical training is extremely important to be able to recognize and evaluate good research. if you don't correct for multiple comparisons following your 'hypothesis generation' studies (e.g. data mining) - your results mean very little.
Most research findings are false? That's so 2005, Atheen.
James,
Unfortunately its still a problem in 2007...and unfortunately, many researchers don't get randomness and Bonferroni corrections.
When is the Jeff Sachs thing at Columbia you were talking about?
Atheen
Post a Comment