Most scientific papers are probably wrong

[Mave]

Found this interesting, especially now that I'm taking a closer look at statistics....

*******************************************
Most scientific papers are probably wrong - 30 Aug 05, Kurt Kleiner @ NewScientist.com

http://www.newscientist.com/article.ns?id=dn7915

Most published scientific research papers are wrong, according to a new analysis. Assuming that the new paper is itself correct, problems with experimental and statistical methods mean that there is less than a 50% chance that the results of any randomly chosen scientific paper are true.

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.

In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.
Massaged conclusions

Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.

Odds get even worse for studies that are too small, studies that find small effects (for example, a drug that works for only 10% of patients), or studies where the protocol and endpoints are poorly defined, allowing researchers to massage their conclusions after the fact.

Surprisingly, Ioannidis says another predictor of false findings is if a field is "hot", with many teams feeling pressure to beat the others to statistically significant findings.

But Solomon Snyder, senior editor at the Proceedings of the National Academy of Sciences, and a neuroscientist at Johns Hopkins Medical School in Baltimore, US, says most working scientists understand the limitations of published research.

"When I read the literature, I'm not reading it to find proof like a textbook. I'm reading to get ideas. So even if something is wrong with the paper, if they have the kernel of a novel idea, that's something to think about," he says.

Journal reference: Public Library of Science Medicine (DOI: 10.1371/journal.pmed.0020124)

[Slater]

*nod* Yeah, that makes sense. Often scientists don't check enough to make sure that their findings are very accurate, just because repeating experements and calculations is very tedious.

[Technomancer]

It should be noted that this applies only to certain types of studies, mostly those dealing with problems relating to multivariate statistics, where large sample sizes are not available and where experimental control is not possible. Certainly, some investigators will come to the wrong conclusions, or make mistakes, or fail to see the underlying cause of a phenomenon (and thus misinterpret it). That is why science is an experimental endeavour and why individual results must also be subject to rigorous examination by other experts: because humans are not perfect.

This does not mean that we can reject a finding out of hand merely because we have heard some statistic, and because such a rejection is convenient for us in some matter. Each paper must be evaluated on its own merits, and its impact will ultimately either be felt over time or be unimportant. With many researchers working from different ideas, and with different approaches though, a consensus may ultimately develop (which is why the currently prevailing theories are the currently prevailing theories).

From my own experience, a lot of potentially wrong ideas do get published, even if no one currently knows that they are wrong. Other potentially good ideas might be wrong, or impractical but get published because they are sufficiently interesting or because they illuminate some aspect of an as yet unsolved problem. Ultimately, we need to exchange ideas to separate the good from the bad in what is ultimately a fairly Darwinian process.

This is all more or less common knowledge to those involved in actual research. Unfortunately, the popular media tends to forget the tentative nature of such studies when reporting the latest scare. They're in the business of selling newspapers after all and not necessarily putting in the sort of diligence and caveats that such studies really call for.

[starstoryteller]

In these papers people are coming up with ideas. If the aproch doesn't work one way some one can use it anther way.

[uc pseudonym]

I find it interesting that someone decided to say this formally. Not interesting in a critical sense, merely something I had not expected.

Also, because no one has said it yet: "Recent scientific papers indicate that most scientific papers are probably wrong..."

[Technomancer]

I find it interesting that someone decided to say this formally. Not interesting in a critical sense, merely something I had not expected.

I think I like this one better. It's not that far off the mark, so the reviewers decided to publish it as a joke. It's pretty funny, but people in machine learning will probably get the joke more than others. The authors also did a pretty a good sketch on the closing night of the conference too.

Also, because no one has said it yet: "Recent scientific papers indicate that most scientific papers are probably wrong..."

Well, it just seemed too easy.

[Link Antilles]

"98 percent of Scientific papers are probably wrong, if you write the 2 percent that are probably right, copy and paste this into your sig..."

I'm really sorry, just couldn't resist.


Anyways, interesting article.... I'm surprised someone came forward saying this.

[Eriana]

Unfortunately most people believe scientists because for reasons starting since the beginning of science, most of the people who do chemistry and experiments are very respected people. So even if they are wrong (which is most of the time) they won't be scolded because people have a tendancy to follow the leader in those sorts of situations. Because scientists are usually viewed as geniuses most people refuse to believe any of the theories they make up are wrong. However, science has proven that more scientists were wrong instead of right.
Sorry, I hope that made sense, I sound really weird and confusing probably. Sorry everybody! ^^;;;
@.@;;;;

[Scribs]

I like link antilles comment very much...

[Eriana]

Never heard of them.

[GhostontheNet]

[quote="Eriana"]Unfortunately most people believe scientists because for reasons starting since the beginning of science, most of the people who do chemistry and experiments are very respected people. So even if they are wrong (which is most of the time) they won't be scolded because people have a tendancy to follow the leader in those sorts of situations. Because scientists are usually viewed as geniuses most people refuse to believe any of the theories they make up are wrong. However, science has proven that more scientists were wrong instead of right.
Sorry, I hope that made sense, I sound really weird and confusing probably. Sorry everybody! ^^] It would be fair to say a scientist is only as good as his data, methods, and sample size in experimentation, but as Technomancer said, "This does not mean that we can reject a finding out of hand merely because we have heard some statistic, and because such a rejection is convenient for us in some matter. Each paper must be evaluated on its own merits, and its impact will ultimately either be felt over time or be unimportant. With many researchers working from different ideas, and with different approaches though, a consensus may ultimately develop (which is why the currently prevailing theories are the currently prevailing theories)."

[Mave]

What I've learnt to do with most scientific papers is 1) to review the experiment design and statistical tool FIRST and then 2) hit the findings and conclusions. Problem is most ppl jump straight into #2 first, without questioning the validity of #1 (I'm guilty of it). I currently have the opinion that statistics is an extremely powerful data manipulation tool and while it's a blessing to us, it can also easily be abused in the wrong hands.

Let the reader beware.

[uc pseudonym]

I think I like this one better. It's not that far off the mark, so the reviewers decided to publish it as a joke. It's pretty funny, but people in machine learning will probably get the joke more than others. The authors also did a pretty a good sketch on the closing night of the conference too.

I'm fairly certain that I didn't understand a great deal of it, but it was still an amusing read; it quickly became clear just how far the authors' tongues were in their cheeks. My favorite part was probably, "The problem with an unwritten rule is that you don't know where to go to erase it."

[Technomancer]

Most of what they're getting at is in pretty well summarized in the abstract, so the maths aren't really that important. I'll have my own chance at reviewing papers now too: my supervisor has just asked me to give my impressions of a number of papers submitted to a conference (like I don't have enough crap to do) and to offer my own reviews. A lot of papers on ICA aren't really very practical IMO.