Bug in fMRI software calls 15 years of research into question

[jakeXT]

A bug in the software used by researchers to interpret fMRI data could invalidate fifteen years worth of neuroscientific research, a paper claims.


Three of the most popular pieces of software for fMRI – SPM, FSL and AFNI – were all found to have false positive rates of up to 70 per cent. These findings could invalidate "up to 40,000 papers", researchers claim.

fMRI measures blood flow inside the brain and, by proxy, brain activity. It assumes cerebral blood flow is coupled or correlated with neural activity, and has been used to explore how the human brain responds to robots, how memory and imagination interact, how the brain looks when someone has an idea and more.

"Though fMRI is 25 years old, surprisingly its most common statistical methods have not been validated using real data," said Anders Eklund.

http://www.wired.co.uk/article/fmri-bug-brain-scans-results

https://www.foreignaffairs.com/articles/world/2016-08-25/what-science-cant-explain

[Salviati]

1. Some people may point to this as a failure of science and the scientific method, far from it...

This is successful science at work. A mistake was discovered, past conclusions are brought into question, and given better methods everything we thought we knew now has to be re-examined. Science, like any human endevour, is not perfect, but it is a method that ideally self correcting.

[TheBlackAdder]

4. +1

[Warpy]

2. Not surprised. The PET scan was always more precise

since it measured things like oxygen demand and glucose uptake. Unfortunately, only snapshots could be taken. The fMRI was used to map changing blood flow to various areas to see real time changes. However, it didn't actually measure the activity within the brain, itself.

It looks like people are going to have to slog through those 40,000 papers again, revising the ones that were affected by the bug.

[Jim__]

3. There has been a correction to the paper cited by Wired.

This is the paper cited by Wired

Here is a correction to that paper:

The authors note that on page 7900, in the Significance Statement, lines 9–11, “These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results” should instead appear as “These results question the validity of a number of fMRI studies and may have a large impact on the interpretation of weakly significant neuroimaging results.”

Additionally, the authors note that on page 7904, left column, fifth full paragraph, lines 1–3, “It is not feasible to redo 40,000 fMRI studies, and lamentable archiving and data-sharing practices mean most could not be reanalyzed either” should instead appear as “Due to lamentable archiving and data-sharing practices, it is unlikely that problematic analyses can be redone.”

These errors do not affect the conclusions of the article. The online version has been corrected.

Among other changes, the number 40,000 was removed, and the words weakly significant were added.

[BumRushDaShow]

5. NYT had an analysis piece on this today

[font size="3"]Do You Believe in God, or Is That a Software Glitch?[/font]

By KATE MURPHY AUG. 27, 2016



http://www.nytimes.com/2016/08/28/opinion/sunday/do-you-believe-in-god-or-is-that-a-software-glitch.html?ref=opinion

When I had scanned the opinion section, I saw this somewhat whimsical headline but didn't bother to look... until I saw this OP!

From the article -

Nevertheless, the fM.R.I. community seems determined to be an exemplar. The next issue of the journal NeuroImage: Clinical will lead with an editorial announcing that it will no longer publish research that has not been corrected for multiple comparisons, and there is a push for other journals to do the same, as well as to require authors to make publicly available their data sets and analyses. Data-sharing platforms such as OpenfMRI and Neurovault have already been established to make fM.R.I. data and statistical methods more widely accessible. In fact, it was data sharing that revealed the fM.R.I. software glitch.

So it looks like it's a "hitch up the pants and move on" thing that will probably happen now.

[Helen Borg]

6. And of course, it is not true.

http://www.wired.com/2016/07/dont-quick-flush-15-years-brain-scan-studies/

[Ford_Prefect]

7. The commentary from Foreign Affairs is just that: Opinion not analysis.

If you want to disagree with my Oncologist please let me know. Personally I prefer the chance they may be accurate enough to help cure my wife's cancer.

The key comment in the article was: "These results question the validity of a number of fMRI studies and may have a large impact on the interpretation of weakly significant neuroimaging results." That's the same as suggesting that a political poll falls very close to the margin of error.

It suggests that you take those results with the appropriate grains of salt. Hardly earth shattering news.