Abstract
A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
209,00 € per year
only 17,42 € per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Baker, C.I., Hutchison, T.L. & Kanwisher, N. Does the fusiform face area contain subregions highly selective for nonfaces? Nat. Neurosci. 10, 3–4 (2007).
Simmons, W.K. et al. Imaging the context-sensitivity of ventral temporal category representations using high-resolution fMRI. Soc. Neurosci. Abstr. 262.12 (2006).
Baker, C.I., Simmons, W.K., Bellgowan, P.S. & Kriegeskorte, N. Circular inference in neuroscience: the dangers of double dipping. Soc. Neurosci. Abstr. 104.11 (2007).
Vul, E. & Kanwisher, N. Begging the question: the non-independence error in fMRI data analysis. in Foundations and Philosophy for Neuroimaging (eds. Hanson, S. & Bunzl, M.) (in the press).
Vul, E., Harris, C., Winkielman, P. & Pashler, H. Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition. Perspect. Psychol. Sci. (in the press).
Benjamini, Y. Comment: microarrays, empirical bayes and the two-groups model. Statist. Sci. 23, 23–28 (2008).
Worsley, K.J., Evans, A.C., Marrett, S. & Neelin, P. A three-dimensional statistical analysis for CBF activation studies in human brain. J. Cereb. Blood Flow Metab. 12, 900–918 (1992).
Friston, K.J., Jezzard, P. & Turner, R. Analysis of functional MRI time-series. Hum. Brain Mapp. 1, 153–171 (1994).
Worsley, K.J. & Friston, K.J. Analysis of fMRI time-series revisited—again. Neuroimage 2, 173–181 (1995).
Nichols, T. & Hayasaka, S. Controlling the family-wise error rate in functional neuroimaging: a comparative review. Stat. Methods Med. Res. 12, 419–446 (2003).
Genovese, C.R., Lazar, N.A. & Nichols, T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage 15, 870–878 (2002).
Friston, K.J., Rotshtein, P., Geng, J.J., Sterzer, P. & Henson, R.N. A critique of functional localisers. Neuroimage 30, 1077–1087 (2006).
Saxe, R., Brett, M. & Kanwisher, N. Divide and conquer: a defense of functional localizers. Neuroimage 30, 1088–1096 (2006).
Haxby, J.V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001).
Cox, D.D. & Savoy, R.L. Functional magnetic resonance imaging (fMRI) ‘brain reading’: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 261–270 (2003).
Mitchell, T.M., Hutchinson, R., Niculescu, R.S., Pereira, F. & Wang, X. Learning to decode cognitive states from brain images. Mach. Learn. 57, 145–175 (2004).
Kriegeskorte, N., Goebel, R. & Bandettini, P. Information-based functional brain mapping. Proc. Natl. Acad. Sci. USA 103, 3863–3868 (2006).
Norman, K.A., Polyn, S.M., Detre, G.J. & Haxby, J.V. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10, 424–430 (2006).
Strother, S.C. et al. The quantitative evaluation of functional neuroimaging experiments: the NPAIRS data analysis framework. Neuroimage 15, 747–771 (2002).
Strother, S. et al. Optimizing the fMRI data-processing pipeline using prediction and reproducibility performance metrics. I. A preliminary group analysis. Neuroimage 23, S196–S207 (2004).
Mitchell, T.M. Machine Learning (McGraw Hill, Portland, Oregon, 1997).
Duda, R.O., Hart, P.E. & Stork, D.G. Pattern Classification. 2nd edn. (Wiley, New York, 2001).
Bishop, C.M. Pattern Recognition and Machine Learning (Springer, New York, 2006).
Kant, I. Kritik der reinen Vernunft. 2nd edn. http://www.gutenberg.org/dirs/etext04/8ikc210.txt (1781).
Acknowledgements
We would like to thank P.A. Bandettini, R.W. Cox, J.V. Haxby, D.J. Kravitz, A. Martin, R.A. Poldrack, R.D. Raizada, Z.S. Saad, J.T. Serences and E. Vul for helpful discussions. This work was supported by the Intramural Research Program of the US National Institute of Mental Health.
Author information
Authors and Affiliations
Contributions
N.K., W.K.S., P.S.F.B. and C.I.B. conceived the project and discussed all of the issues. W.K.S. contributed the data, simulation and analysis for Example 1. N.K. designed and performed all other simulations and analyses. P.S.F.B. contributed to data analysis. N.K. wrote the paper and the Supplementary Discussion and made all figures. W.K.S. and P.S.F.B. commented on drafts. C.I.B. edited the paper and guided the project.
Corresponding authors
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–4 and Supplementary Discussion (PDF 1251 kb)
Rights and permissions
About this article
Cite this article
Kriegeskorte, N., Simmons, W., Bellgowan, P. et al. Circular analysis in systems neuroscience: the dangers of double dipping. Nat Neurosci 12, 535–540 (2009). https://doi.org/10.1038/nn.2303
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2303
This article is cited by
-
Neural inhibition as implemented by an actor-critic model involves the human dorsal striatum and ventral tegmental area
Scientific Reports (2024)
-
Cortical glutamate, Glx, and total N-acetylaspartate: potential biomarkers of repetitive transcranial magnetic stimulation treatment response and outcomes in major depression
Translational Psychiatry (2024)
-
The neural origin for asymmetric coding of surface color in the primate visual cortex
Nature Communications (2024)
-
Blood and MRI biomarkers of mild traumatic brain injury in non-concussed collegiate football players
Scientific Reports (2024)
-
Neural representations of situations and mental states are composed of sums of representations of the actions they afford
Nature Communications (2024)
