Task Reliability Considerations in Computational Psychiatry

  • Craig Hedge
    Affiliations
    Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff, United Kingdom
    Search for articles by this author
  • Aline Bompas
    Affiliations
    Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff, United Kingdom
    Search for articles by this author
  • Petroc Sumner
    Correspondence
    Address correspondence to Petroc Sumner, Ph.D., or Craig Hedge, Ph.D., School of Psychology, Cardiff University, Tower Building, 70 Park Place, Cardiff, CF10 3AT, United Kingdom.
    Affiliations
    School of Psychology, Cardiff University, Cardiff, United Kingdom
    Search for articles by this author
      Measuring psychological abilities or traits is trickier than it seems from the published literature (

      Flake JK, Fried EI (in press): Measurement schmeasurement: Questionable measurement practices and how to avoid them. Adv Methods Pract Psychol Sci.

      ). We try to study abstract psychological constructs like “inhibition” or “impulsivity,” but we can only measure these indirectly through behaviors such as reaction times or self-report ratings. When we apply these measures in clinical and individual differences research, our goal is typically to understand why a patient group appears to be, for example, more “impulsive” than healthy control subjects or is to use measures of “inhibition” to predict a clinical outcome. There are many potential pitfalls and wrong turns in the path toward achieving this goal. In our recent work, we have shown that our intuitions about what makes a good cognitive task can sometimes lead us astray (
      • Hedge C.
      • Powell G.
      • Sumner P.
      The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences.
      ,
      • Hedge C.
      • Powell G.
      • Bompas A.
      • Vivian-Griffiths S.
      • Sumner P.
      Low and variable correlation between reaction time costs and accuracy costs explained by accumulation models: Meta-analysis and simulations.
      ). Here, we discuss how these issues intersect with the goals of computational psychiatry.
      To read this article in full you will need to make a payment

      References

      1. Flake JK, Fried EI (in press): Measurement schmeasurement: Questionable measurement practices and how to avoid them. Adv Methods Pract Psychol Sci.

        • Hedge C.
        • Powell G.
        • Sumner P.
        The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences.
        Behav Res Methods. 2018; 50: 1166-1186
        • Hedge C.
        • Powell G.
        • Bompas A.
        • Vivian-Griffiths S.
        • Sumner P.
        Low and variable correlation between reaction time costs and accuracy costs explained by accumulation models: Meta-analysis and simulations.
        Psychol Bull. 2018; 144: 1200-1227
        • Stroop J.R.
        Studies of interference in serial verbal reactions.
        J Exp Psychol. 1935; 18: 643-662
        • Ebersole C.R.
        • Atherton O.E.
        • Belanger A.L.
        • Skulborstad H.M.
        • Allen J.M.
        • Banks J.B.
        • et al.
        Many labs 3: Evaluating participant pool quality across the academic semester via replication.
        J Exp Soc Psychol. 2016; 67: 68-82
        • Nunnally J.C.
        Psychometric Theory.
        2nd ed. McGraw-Hill, New York1978
        • Price R.B.
        • Brown V.
        • Siegle G.J.
        Computational modeling applied to the dot-probe task yields improved reliability and mechanistic insights.
        Biol Psychiatry. 2019; 85: 606-612
        • Huys Q.J.
        • Maia T.V.
        • Frank M.J.
        Computational psychiatry as a bridge from neuroscience to clinical applications.
        Nat Neurosci. 2016; 19: 404-413
        • Paulus M.P.
        • Huys Q.J.
        • Maia T.V.
        A roadmap for the development of applied computational psychiatry.
        Biol Psychiatry Cogn Neurosci Neuroimaging. 2016; 1: 386-392
        • Brown V.M.
        • Chen J.
        • Gillan C.M.
        • Price R.B.
        Improving the reliability of computational analyses: Model-based planning and its relationship with compulsivity.
        Biol Psychiatry Cogn Neurosci Neuroimaging. 2020; 5: 601-609