Advertisement

Improving the Reliability of Computational Analyses: Model-Based Planning and Its Relationship With Compulsivity

Published:January 13, 2020DOI:https://doi.org/10.1016/j.bpsc.2019.12.019

      Abstract

      Background

      Computational models show great promise in mapping latent decision-making processes onto dissociable neural substrates and clinical phenotypes. One prominent example in reinforcement learning is model-based planning, which specifically relates to transdiagnostic compulsivity. However, the reliability of computational model-derived measures such as model-based planning is unclear. Establishing reliability is necessary to ensure that such models measure stable, traitlike processes, as assumed in computational psychiatry. Although analysis approaches affect validity of reinforcement learning models and reliability of other task-based measures, their effect on reliability of reinforcement learning models of empirical data has not been systematically studied.

      Methods

      We first assessed within- and across-session reliability and effects of analysis approaches (model estimation, parameterization, and data cleaning) of measures of model-based planning in patients with compulsive disorders (n = 38). The analysis approaches affecting test-retest reliability were tested in 3 large generalization samples (healthy participants: n = 541 and 111; people with a range of compulsivity: n = 1413).

      Results

      Analysis approaches greatly influenced reliability: reliability of model-based planning measures ranged from 0 (no concordance) to above 0.9 (acceptable for clinical applications). The largest influence on reliability was whether model-estimation approaches were robust and accounted for the hierarchical structure of estimated parameters. Improvements in reliability generalized to other datasets and greatly reduced the sample size needed to find a relationship between model-based planning and compulsivity in an independent dataset.

      Conclusions

      These results indicate that computational psychiatry measures such as model-based planning can reliably measure latent decision-making processes, but when doing so must assess the ability of methods to estimate complex models from limited data.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      References

        • Montague P.R.
        • Dolan R.J.
        • Friston K.J.
        • Dayan P.
        Computational psychiatry.
        Trends Cogn Sci. 2012; 16: 72-80
        • Gillan C.M.
        • Kosinski M.
        • Whelan R.
        • Phelps E.A.
        • Daw N.D.
        Characterizing a psychiatric symptom dimension related to deficits in goal-directed control.
        Elife. 2016; 5: 1-24
        • Wheaton M.G.
        • Gillan C.M.
        • Simpson H.B.
        Does cognitive-behavioral therapy affect goal-directed planning in obsessive-compulsive disorder?.
        Psychiatry Res. 2019; 273: 94-99
        • Gillan C.M.
        • Kalanthroff E.
        • Evans M.
        • Weingarden H.M.
        • Jacoby R.J.
        • Gershkovich M.
        • et al.
        Comparison of the association between goal-directed planning and self-reported compulsivity vs obsessive-compulsive disorder diagnosis.
        JAMA Psychiatry. 2019; 77: 1-10
        • Daw N.D.
        • Gershman S.J.
        • Seymour B.
        • Dayan P.
        • Dolan R.J.
        Model-based influences on humans’ choices and striatal prediction errors.
        Neuron. 2011; 69: 1204-1215
        • Enkavi A.Z.
        • Eisenberg I.W.
        • Bissett P.G.
        • Mazza G.L.
        • MacKinnon D.P.
        • Marsch L.A.
        • et al.
        Large-scale analysis of test–retest reliabilities of self-regulation measures.
        Proc Natl Acad Sci U S A. 2019; 116: 5472-5477
        • Price R.B.
        • Kuckertz J.M.
        • Siegle G.J.
        • Ladouceur C.D.
        • Silk J.S.
        • Ryan N.D.
        • et al.
        Empirical recommendations for improving the stability of the dot-probe task in clinical research.
        Psychol Assess. 2015; 27: 365-376
        • Rodebaugh T.L.
        • Scullin R.B.
        • Langer J.K.
        • Dixon D.J.
        • Huppert J.D.
        • Bernstein A.
        • et al.
        Unreliability as a threat to understanding psychopathology: The cautionary tale of attentional bias.
        J Abnorm Psychol. 2016; 125: 840-851
        • Price R.B.
        • Brown V.M.
        • Siegle G.J.
        Computational modeling applied to the dot-probe task yields improved reliability and mechanistic insights.
        Biol Psychiatry. 2019; 85: 606-612
        • Chung D.
        • Kadlec K.
        • Aimone J.A.
        • McCurry K.
        • King-Casas B.
        • Chiu P.H.
        Valuation in major depression is intact and stable in a non-learning environment.
        Sci Rep. 2017; 7: 1-9
        • Shahar N.
        • Hauser T.U.
        • Moutoussis M.
        • Moran R.
        • Keramati M.
        • Dolan R.J.
        • et al.
        Improving the reliability of model-based decision-making estimates in the two-stage decision task with reaction-times and drift-diffusion modeling.
        PLoS Comput Biol. 2019; 15e1006803
        • Moutoussis M.
        • Bullmore E.T.
        • Goodyer I.M.
        • Fonagy P.
        • Jones P.B.
        • Dolan R.J.
        • et al.
        Change, stability, and instability in the Pavlovian guidance of behaviour from adolescence to young adulthood.
        PLoS Comput Biol. 2018; 14e1006679
        • Lau B.
        • Glimcher P.W.
        Dynamic response-by-response models of matching behavior in rhesus monkeys.
        J Exp Anal Behav. 2005; 84: 555-579
        • Ahn W.-Y.
        • Krawitz A.
        • Kim W.
        • Busemeyer J.R.
        • Brown J.W.
        A model-based fMRI analysis with hierarchical Bayesian parameter estimation.
        J Neurosci Psychol Econ. 2011; 4: 95-110
        • Palminteri S.
        • Wyart V.
        • Koechlin E.
        The importance of falsification in computational cognitive modeling.
        Trends Cogn Sci. 2017; 21: 425-433
        • Wetzels R.
        • Vandekerckhove J.
        • Tuerlinckx F.
        • Wagenmakers E.-J.
        Bayesian parameter estimation in the expectancy valence model of the Iowa gambling task.
        J Math Psychol. 2010; 54: 14-27
        • Spektor M.S.
        • Kellen D.
        The relative merit of empirical priors in non-identifiable and sloppy models: Applications to models of learning and decision-making: Empirical priors.
        Psychon Bull Rev. 2018; 25: 2047-2068
        • Sharp M.E.
        • Foerde K.
        • Daw N.D.
        • Shohamy D.
        Dopamine selectively remediates “model-based” reward learning: A computational approach.
        Brain. 2016; 139: 355-364
        • Sebold M.
        • Nebe S.
        • Garbusow M.
        • Guggenmos M.
        • Schad D.J.
        • Beck A.
        • et al.
        When habits are dangerous: Alcohol expectancies and habitual decision making predict relapse in alcohol dependence.
        Biol Psychiatry. 2017; 82: 847-856
        • Otto A.R.
        • Raio C.M.
        • Chiang A.
        • Phelps E.A.
        • Daw N.D.
        Working-memory capacity protects model-based learning from stress.
        Proc Natl Acad Sci U S A. 2013; 110: 20941-20946
        • Voon V.
        • Reiter A.
        • Sebold M.
        • Groman S.
        Model-based control in dimensional psychiatry.
        Biol Psychiatry. 2017; 82: 391-400
        • Patzelt E.H.
        • Kool W.
        • Millner A.J.
        • Gershman S.J.
        Incentives boost model-based control across a range of severity on several psychiatric constructs.
        Biol Psychiatry. 2019; 85: 425-433
        • Decker J.H.
        • Otto A.R.
        • Daw N.D.
        • Hartley C.A.
        From creatures of habit to goal-directed learners: Tracking the developmental emergence of model-based reinforcement learning.
        Psychol Sci. 2016; 27: 848-858
        • Doll B.B.
        • Bath K.G.
        • Daw N.D.
        • Frank M.J.
        Variability in dopamine genes dissociates model-based and model-free reinforcement learning.
        J Neurosci. 2016; 36: 1211-1222
        • Heller A.S.
        • Ezie C.E.C.
        • Otto A.R.
        • Timpano K.R.
        Model-based learning and individual differences in depression: The moderating role of stress.
        Behav Res Ther. 2018; 111: 19-26
        • Culbreth A.J.
        • Westbrook A.
        • Daw N.D.
        • Botvinick M.M.
        • Barch D.M.
        Reduced model-based decision-making in schizophrenia.
        J Abnorm Psychol. 2016; 125: 777-787
        • Gläscher J.
        • Daw N.D.
        • Dayan P.
        • O’Doherty J.P.
        States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning.
        Neuron. 2010; 66: 585-595
        • Kovach C.K.
        • Daw N.D.
        • Rudrauf D.
        • Tranel D.
        • O’Doherty J.P.
        • Adolphs R.
        Anterior prefrontal cortex contributes to action selection through tracking of recent reward trends.
        J Neurosci. 2012; 32: 8434-8442
        • Daw N.D.
        Trial-by-trial data analysis using computational models.
        in: Delgado M.R. Phelps E.A. Robbins T.W. Decision Making, Affect, and Learning: Attention and Performance XXIII. Oxford University Press, New York, NY2011: 3-38
        • Gillan C.M.
        • Otto A.R.
        • Phelps E.A.
        • Daw N.D.
        Model-based learning protects against forming habits.
        Cogn Affect Behav Neurosci. 2015; 15: 523-536
        • Moutoussis M.
        • Hopkins A.K.
        • Dolan R.J.
        Hypotheses about the relationship of cognition with psychopathology should be tested by embedding them into empirical priors.
        Front Psychol. 2018; 9: 2504
        • Boehm U.
        • Steingroever H.
        • Wagenmakers E.-J.
        Using Bayesian regression to test hypotheses about relationships between parameters and covariates in cognitive models.
        Behav Res Methods. 2018; 50: 12458-12469
        • Scheibehenne B.
        • Pachur T.
        Using Bayesian hierarchical parameter estimation to assess the generalizability of cognitive models of choice.
        Psychon Bull Rev. 2015; 22: 391-407
        • Cicchetti D.
        Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology.
        Psychol Assess. 1994; 6: 284-290
        • Cohen J.
        Statistical Power Analysis for the Behavioral Sciences.
        2nd ed. Lawrence Erlbaum Associates, Mahwah, NJ1988
        • Paulus M.P.
        • Huys Q.J.M.
        • Maia T.V.
        A roadmap for the development of applied computational psychiatry.
        Biol Psychiatry Cogn Neurosci Neuroimaging. 2016; 1: 386-392
        • Sutton R.S.
        • Barto A.G.
        Reinforcement Learning: An Introduction.
        MIT Press, Cambridge, MA1998
        • Montague P.R.
        • Dayan P.
        • Sejnowski T.J.
        A framework for mesencephalic predictive Hebbian learning.
        J Neurosci. 1996; 76: 1936-1947
        • Schultz W.
        • Dayan P.
        • Montague P.R.
        A neural substrate of prediction and reward.
        Science. 1997; 275: 1593-1599
        • Hedge C.
        • Powell G.
        • Sumner P.
        The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences.
        Behav Res. 2018; 50: 1166-1186
        • Gershman S.J.
        Empirical priors for reinforcement learning models.
        J Math Psychol. 2016; 71: 1-6
        • Guitart-Masip M.
        • Huys Q.J.M.
        • Fuentemilla L.
        • Dayan P.
        • Duzel E.
        • Dolan R.J.
        Go and no-go learning in reward and punishment: Interactions between affect and effect.
        Neuroimage. 2012; 62: 154-166
        • Daunizeau J.
        • Adam V.
        • Rigoux L.
        VBA: A probabilistic treatment of nonlinear models for neurobiological and behavioural data.
        PLoS Comput Biol. 2014; 10e1003441
        • Momennejad I.
        • Russek E.M.
        • Cheong J.H.
        • Botvinick M.M.
        • Daw N.D.
        • Gershman S.J.
        The successor representation in human reinforcement learning.
        Nat Hum Behav. 2017; 1: 680-692
        • Mattar M.G.
        • Daw N.D.
        Prioritized memory access explains planning and hippocampal replay.
        Nat Neurosci. 2018; 21: 1609-1617
        • Toyama A.
        • Katahira K.
        • Ohira H.
        Biases in estimating the balance between model-free and model-based learning systems due to model misspecification.
        J Math Psychol. 2019; 91: 88-102
        • Howlett J.R.
        • Huang H.
        • Hysek C.M.
        • Paulus M.P.
        The effect of single-dose methylphenidate on the rate of error-driven learning in healthy males: A randomized controlled trial.
        Psychopharmacology (Berl). 2017; 234: 3353-3360
        • Brandmaier A.M.
        • Wenger E.
        • Bodammer N.C.
        • Kühn S.
        • Raz N.
        • Lindenberger U.
        Assessing reliability in neuroimaging research through intra-class effect decomposition (ICED).
        Elife. 2018; 7e35718
        • Loken E.
        • Gelman A.
        Measurement error and the replication crisis.
        Science. 2017; 355: 584-585