Abstract
Background
Computational models show great promise in mapping latent decision-making processes
onto dissociable neural substrates and clinical phenotypes. One prominent example
in reinforcement learning is model-based planning, which specifically relates to transdiagnostic
compulsivity. However, the reliability of computational model-derived measures such
as model-based planning is unclear. Establishing reliability is necessary to ensure
that such models measure stable, traitlike processes, as assumed in computational
psychiatry. Although analysis approaches affect validity of reinforcement learning
models and reliability of other task-based measures, their effect on reliability of
reinforcement learning models of empirical data has not been systematically studied.
Methods
We first assessed within- and across-session reliability and effects of analysis approaches
(model estimation, parameterization, and data cleaning) of measures of model-based
planning in patients with compulsive disorders (n = 38). The analysis approaches affecting test-retest reliability were tested in 3
large generalization samples (healthy participants: n = 541 and 111; people with a range of compulsivity: n = 1413).
Results
Analysis approaches greatly influenced reliability: reliability of model-based planning
measures ranged from 0 (no concordance) to above 0.9 (acceptable for clinical applications).
The largest influence on reliability was whether model-estimation approaches were
robust and accounted for the hierarchical structure of estimated parameters. Improvements
in reliability generalized to other datasets and greatly reduced the sample size needed
to find a relationship between model-based planning and compulsivity in an independent
dataset.
Conclusions
These results indicate that computational psychiatry measures such as model-based
planning can reliably measure latent decision-making processes, but when doing so
must assess the ability of methods to estimate complex models from limited data.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Biological Psychiatry: Cognitive Neuroscience and NeuroimagingAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Computational psychiatry.Trends Cogn Sci. 2012; 16: 72-80
- Characterizing a psychiatric symptom dimension related to deficits in goal-directed control.Elife. 2016; 5: 1-24
- Does cognitive-behavioral therapy affect goal-directed planning in obsessive-compulsive disorder?.Psychiatry Res. 2019; 273: 94-99
- Comparison of the association between goal-directed planning and self-reported compulsivity vs obsessive-compulsive disorder diagnosis.JAMA Psychiatry. 2019; 77: 1-10
- Model-based influences on humans’ choices and striatal prediction errors.Neuron. 2011; 69: 1204-1215
- Large-scale analysis of test–retest reliabilities of self-regulation measures.Proc Natl Acad Sci U S A. 2019; 116: 5472-5477
- Empirical recommendations for improving the stability of the dot-probe task in clinical research.Psychol Assess. 2015; 27: 365-376
- Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias.J Abnorm Psychol. 2016; 125: 840-851
- Computational modeling applied to the dot-probe task yields improved reliability and mechanistic insights.Biol Psychiatry. 2019; 85: 606-612
- Valuation in major depression is intact and stable in a non-learning environment.Sci Rep. 2017; 7: 1-9
- Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling.PLoS Comput Biol. 2019; 15e1006803
- Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood.PLoS Comput Biol. 2018; 14e1006679
- Dynamic response-by-response models of matching behavior in rhesus monkeys.J Exp Anal Behav. 2005; 84: 555-579
- A model-based fMRI analysis with hierarchical Bayesian parameter estimation.J Neurosci Psychol Econ. 2011; 4: 95-110
- The importance of falsification in computational cognitive modeling.Trends Cogn Sci. 2017; 21: 425-433
- Bayesian parameter estimation in the expectancy valence model of the Iowa gambling task.J Math Psychol. 2010; 54: 14-27
- The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making: Empirical priors.Psychon Bull Rev. 2018; 25: 2047-2068
- Dopamine selectively remediates “model-based” reward learning: A computational approach.Brain. 2016; 139: 355-364
- When habits are dangerous: Alcohol expectancies and habitual decision making predict relapse in alcohol dependence.Biol Psychiatry. 2017; 82: 847-856
- Working-memory capacity protects model-based learning from stress.Proc Natl Acad Sci U S A. 2013; 110: 20941-20946
- Model-based control in dimensional psychiatry.Biol Psychiatry. 2017; 82: 391-400
- Incentives boost model-based control across a range of severity on several psychiatric constructs.Biol Psychiatry. 2019; 85: 425-433
- From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learning.Psychol Sci. 2016; 27: 848-858
- Variability in dopamine genes dissociates model-based and model-free reinforcement learning.J Neurosci. 2016; 36: 1211-1222
- Model-based learning and individual differences in depression: The moderating role of stress.Behav Res Ther. 2018; 111: 19-26
- Reduced model-based decision-making in schizophrenia.J Abnorm Psychol. 2016; 125: 777-787
- States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning.Neuron. 2010; 66: 585-595
- Anterior prefrontal cortex contributes to action selection through tracking of recent reward trends.J Neurosci. 2012; 32: 8434-8442
- Trial-by-trial data analysis using computational models.in: Delgado M.R. Phelps E.A. Robbins T.W. Decision Making, Affect, and Learning: Attention and Performance XXIII. Oxford University Press, New York, NY2011: 3-38
- Model-based learning protects against forming habits.Cogn Affect Behav Neurosci. 2015; 15: 523-536
- Hypotheses about the relationship of cognition with psychopathology should be tested by embedding them into empirical priors.Front Psychol. 2018; 9: 2504
- Using Bayesian regression to test hypotheses about relationships between parameters and covariates in cognitive models.Behav Res Methods. 2018; 50: 12458-12469
- Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice.Psychon Bull Rev. 2015; 22: 391-407
- Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.Psychol Assess. 1994; 6: 284-290
- Statistical Power Analysis for the Behavioral Sciences.2nd ed. Lawrence Erlbaum Associates, Mahwah, NJ1988
- A roadmap for the development of applied computational psychiatry.Biol Psychiatry Cogn Neurosci Neuroimaging. 2016; 1: 386-392
- Reinforcement Learning: An Introduction.MIT Press, Cambridge, MA1998
- A framework for mesencephalic predictive Hebbian learning.J Neurosci. 1996; 76: 1936-1947
- A neural substrate of prediction and reward.Science. 1997; 275: 1593-1599
- The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences.Behav Res. 2018; 50: 1166-1186
- Empirical priors for reinforcement learning models.J Math Psychol. 2016; 71: 1-6
- Go and no-go learning in reward and punishment: Interactions between affect and effect.Neuroimage. 2012; 62: 154-166
- VBA: A probabilistic treatment of nonlinear models for neurobiological and behavioural data.PLoS Comput Biol. 2014; 10e1003441
- The successor representation in human reinforcement learning.Nat Hum Behav. 2017; 1: 680-692
- Prioritized memory access explains planning and hippocampal replay.Nat Neurosci. 2018; 21: 1609-1617
- Biases in estimating the balance between model-free and model-based learning systems due to model misspecification.J Math Psychol. 2019; 91: 88-102
- The effect of single-dose methylphenidate on the rate of error-driven learning in healthy males: A randomized controlled trial.Psychopharmacology (Berl). 2017; 234: 3353-3360
- Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED).Elife. 2018; 7e35718
- Measurement error and the replication crisis.Science. 2017; 355: 584-585
Article info
Publication history
Published online: January 13, 2020
Accepted:
December 30,
2019
Received in revised form:
December 30,
2019
Received:
November 20,
2019
Identification
Copyright
© 2020 Society of Biological Psychiatry. Published by Elsevier Inc. All rights reserved.
