Abstract
U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (n.d.): NIMH Strategic Plan for Research. Retrieved from https://www.nimh.nih.gov/sites/default/files/documents/about/strategic-planning-reports/NIMH%20Strategic%20Plan%20for%20Research_2022_0.pdf
U.K. Research and Innovation (n.d.): U.K. Research and Innovation. Retrieved from https://www.ukri.org/what-we-offer/browse-our-areas-of-investment-and-support/
Keywords
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Biological Psychiatry: Cognitive Neuroscience and NeuroimagingReferences
- Psychiatry as a clinical neuroscience discipline.JAMA. 2005; 294: 2221-2224
U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health (n.d.): NIMH Strategic Plan for Research. Retrieved from https://www.nimh.nih.gov/sites/default/files/documents/about/strategic-planning-reports/NIMH%20Strategic%20Plan%20for%20Research_2022_0.pdf
U.K. Research and Innovation (n.d.): U.K. Research and Innovation. Retrieved from https://www.ukri.org/what-we-offer/browse-our-areas-of-investment-and-support/
- Quantifying Deviations of Brain Structure and Function in Major Depressive Disorder Across Neuroimaging Modalities.JAMA Psychiatry. 2022; https://doi.org/10.1001/jamapsychiatry.2022.1780
- Robust is not necessarily reliable: From within-subjects fMRI contrasts to between-subjects comparisons.NeuroImage. 2018; 173: 146-152
- Task Reliability Considerations in Computational Psychiatry.Biol Psychiatry Cogn Neurosci Neuroimaging. 2020; 5: 837-839
- Future directions for cognitive neuroscience in psychiatry: recommendations for biomarker design based on recent test re-test reliability work.Curr Opin Behav Sci. 2022; 44101102
- Computational Psychiatry Needs Time and Context.Annu Rev Psychol. 2022; 73: 243-270
Haines N, Kvam PD, Irving LH, Smith C, Beauchaine TP, Pitt MA, et al. (2020): Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox. PsyArXiv. https://doi.org/10.31234/osf.io/xr7y3
- Variability in the analysis of a single neuroimaging dataset by many teams [no. 7810].Nature. 2020; 582: 84-88
- An empirical comparison of SPM preprocessing parameters to the analysis of fMRI data.NeuroImage. 2002; 17: 19-28
Poline J-B, Strother SC, Dehaene-Lambertz G, Egan GF, Lancaster JL (2006): Motivation and synthesis of the FIAC experiment: Reproducibility of fMRI results across expert analyses. 27. https://doi.org/10.1002/hbm.20268
- Model Specification and the Reliability of fMRI Results: Implications for Longitudinal Neuroimaging Studies in Psychiatry.PLOS ONE. 2014; 9e105169
- Single subject fMRI test–retest reliability metrics and confounding factors.NeuroImage. 2013; 69: 231-243
Korucuoglu O, Harms MP, Astafiev SV, Golosheykin S, Kennedy JT, Barch DM, Anokhin AP (2021): Test-Retest Reliability of Neural Correlates of Response Inhibition and Error Monitoring: An fMRI Study of a Stop-Signal Task. Front Neurosci 15. Retrieved December 11, 2022, from https://www.frontiersin.org/articles/10.3389/fnins.2021.624911
- What Is the Test-Retest Reliability of Common Task-Functional MRI Measures? New Empirical Evidence and a Meta-Analysis.Psychol Sci. 2020; 31: 792-806
- Influences on the Test–Retest Reliability of Functional Connectivity MRI and its Relationship with Behavioral Utility.Cereb Cortex. 2017; 27: 5415-5429
- Reliability of subsequent memory effects in children and adults: The good, the bad, and the hopeful.Dev Cogn Neurosci. 2021; 52101037
- A guide to the measurement and interpretation of fMRI test-retest reliability.Curr Opin Behav Sci. 2021; 40: 27-32
- The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences.Behav Res Methods. 2018; 50: 1166-1186
- Large-scale analysis of test–retest reliabilities of self-regulation measures.Proc Natl Acad Sci. 2019; 116: 5472-5477
- Temporal Stability of Implicit and Explicit Measures: A Longitudinal Analysis.Pers Soc Psychol Bull. 2017; 43: 300-312
Klein C (2020, June 3): Confidence Intervals on Implicit Association Test Scores Are Really Rather Large. PsyArXiv. https://doi.org/10.31234/osf.io/5djkh
- Individual Variability and Test-Retest Reliability Revealed by Ten Repeated Resting-State Brain Scans over One Month.PLOS ONE. 2015; 10e0144963
- A decade of test-retest reliability of functional connectivity: A systematic review and meta-analysis.NeuroImage. 2019; 203116157
- The longitudinal stability of fMRI activation during reward processing in adolescents and young adults.NeuroImage. 2021; 232117872
- Why Are Self-Report and Behavioral Measures Weakly Correlated?.Trends Cogn Sci. 2020; 24: 267-269
- The Implicit Association Test: A Method in Search of a Construct.Perspect Psychol Sci. 2021; 16: 396-414
- Why self-report measures of self-control and inhibition tasks do not substantially correlate.Collabra Psychol. 2020; 6https://doi.org/10.1525/collabra.276
- Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.Perspect Psychol Sci. 2014; 9: 641-651
- Reproducible brain-wide association studies require thousands of individuals [no. 7902].Nature. 2022; 603: 654-660
Kragel PA, Han X, Kraynak TE, Gianaros PJ, Wager TD (2021): Functional MRI Can Be Highly Reliable, but It Depends on What You Measure: A Commentary on Elliott et al. (2020). Psychol Sci 32: 622–626.
- Improving the reliability of computational analyses: Model-based planning and its relationship with compulsivity.Biol Psychiatry Cogn Neurosci Neuroimaging. 2020; 5: 601-609
- Trial and error: A hierarchical modeling approach to test-retest reliability.NeuroImage. 2021; 245118647
- A psychometrics of individual differences in experimental tasks.Psychon Bull Rev. 2019; 26: 452-467
Lord FM, Novick MR, Birnbaum A (1968): Statistical Theories of Mental Test Scores. Oxford, England: Addison-Wesley.
Kelley TL (1947): Fundamentals of Statistics. Oxford, England: Harvard U. Press, pp xvi, 755.
Kelley TL (1927): Interpretation of Educational Measurements. Yonkers-on-Hudson, N.Y.: World Book Company.
- Stein’s Estimation Rule and Its Competitors--An Empirical Bayes Approach.J Am Stat Assoc. 1973; 68: 117-130
Stein C (1956): Inadmissibility of the Usual Estimator for the Mean of a Multivariate Normal Distribution. STANFORD UNIVERSITY STANFORD United States. Retrieved August 23, 2022, from https://apps.dtic.mil/sti/citations/AD1028390
James W, Stein C (1961): Estimation with Quadratic Loss. Proc Fourth Berkeley Symp Math Stat Probab Vol 1 Contrib Theory Stat 4.1: 361–380.
- Stein’s Paradox in Statistics.Sci Am. 1977; 236: 119-127
- Forming inferences about some intraclass correlation coefficients.Psychol Methods. 1996; 1: 30-46
- Intraclass correlations: Uses in assessing rater reliability.Psychol Bull. 1979; 86: 420-428
- Choosing the best index for the average score intraclass correlation coefficient.Behav Res Methods. 2016; 48: 994-1003
- Have Multilevel Models Been Structural Equation Models All Along?.Multivar Behav Res. 2003; 38: 529-569
- Equivalence and Differences Between Structural Equation Modeling and State-Space Modeling Techniques.Struct Equ Model Multidiscip J. 2010; 17: 303-332
- Structural equation modeling with interchangeable dyads.Psychol Methods. 2006; 11: 127-141
- Bayesian Measures of Explained Variance and Pooling in Multilevel (Hierarchical) Models.Technometrics. 2006; 48: 241-251
Williams DR, Martin SR, DeBolt M, Oakes L, Rast P (2020, November 20): A Fine-Tooth Comb for Measurement Reliability: Predicting True Score and Error Variance in Hierarchical Models. PsyArXiv. https://doi.org/10.31234/osf.io/2ux7t
- A Bayesian framework for simultaneously modeling neural and behavioral data.NeuroImage. 2013; 72: 193-206
- A model-based fMRI analysis with hierarchical Bayesian parameter estimation.J Neurosci Psychol Econ. 2011; 4: 95-110
- How hierarchical models improve point estimates of model parameters at the individual level.J Math Psychol. 2016; 73: 37-58
Valton V, Wise T, Robinson OJ (2020): Recommendations for Bayesian hierarchical model specifications for case-control studies in mental health. https://doi.org/10.48550/arXiv.2011.01725
- An introduction to Bayesian hierarchical models with an application in the theory of signal detection.Psychon Bull Rev. 2005; 12: 573-604
- An application of multinomial processing tree models and Bayesian methods to understanding memory impairment.J Math Psychol. 2020; 95https://doi.org/10.1016/j.jmp.2020.102328
- Advances in the computational understanding of mental illness.Neuropsychopharmacology. 2021; 46: 3-19
- Reinforcement Learning in Patients With Mood and Anxiety Disorders vs Control Individuals: A Systematic Review and Meta-analysis.JAMA Psychiatry. 2022; 79: 313-322
Eckstein M, Wilbrecht L, Collins A (2021, May 4): What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience. PsyArXiv. https://doi.org/10.31234/osf.io/e7kwx
Lockwood PL, Klein-Flügge MC (2021): Computational modelling of social cognition and behaviour—a reinforcement learning primer. Soc Cogn Affect Neurosci 16: 761–771.
Zhang L, Lengersdorff L, Mikus N, Gläscher J, Lamm C (2020): Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices. Soc Cogn Affect Neurosci 15: 695–707.
- A tutorial on joint models of neural and behavioral measures of cognition.J Math Psychol. 2018; 84: 20-48
SPM12 Software - Statistical Parametric Mapping (n.d.): Retrieved August 23, 2022, from https://www.fil.ion.ucl.ac.uk/spm/software/spm12/
- Approaches to Analysis in Model-based Cognitive Neuroscience.J Math Psychol. 2017; 76: 65-79
- Is Model Fitting Necessary for Model-Based fMRI?.PLOS Comput Biol. 2015; 11e1004237
- Assessing inter-individual differences with task-related functional neuroimaging [no. 9].Nat Hum Behav. 2019; 3: 897-905
- The Outcome-Representation Learning Model: A Novel Reinforcement Learning Model of the Iowa Gambling Task.Cogn Sci. 2018; 42: 2534-2561
Månsson KNT, Waschke L, Manzouri A, Furmark T, Fischer H, Garrett DD (2022): Moment-to-Moment Brain Signal Variability Reliably Predicts Psychiatric Treatment Outcome. Biol Psychiatry 91: 658–666.
Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB (2015): Bayesian Data Analysis, 3rd ed. New York: Chapman and Hall/CRC. https://doi.org/10.1201/b16018
Farrell S, Lewandowsky S (2018): Computational Modeling of Cognition and Behavior. Cambridge University Press.
Bürkner P-C (2017): brms: An R Package for Bayesian Multilevel Models Using Stan. J Stat Softw 80: 1–28.
- Stan: A Probabilistic Programming Language.J Stat Softw. 2017; 76: 1-32
McElreath R (2022, August 23): Statistical Rethinking (2022 Edition) [R]. Retrieved August 23, 2022, from https://github.com/rmcelreath/stat_rethinking_2022
Zhang L (2022, August 17): BayesCog [R]. Retrieved August 23, 2022, from https://github.com/lei-zhang/BayesCog_Wien
Article info
Publication history
Publication stage
In Press Journal Pre-ProofFootnotes
1 A measure itself is defined as any quantity being used to make inference, ranging from summed scores on self-report scales to parameters extracted from some model of BOLD response during an fMRI task.
2 For example, if indicates the true mean response time for person , then the mean squared error between all pooled James-Stein estimates and true values will always be equal or lesser than the mean squared error between person-level sample means and true values . Mathematically, , where is the number of people.
