Making Individual Prognoses in Psychiatry Using Neuroimaging and Machine Learning

  • Ronald J. Janssen
    Address correspondence to Ronald J. Janssen, Ph.D., Department of Psychiatry, A01.161, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, the Netherlands.
    Department of Psychiatry, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
    Search for articles by this author
  • Janaina Mourão-Miranda
    Centre for Medical Image Computing, Department of Computer Science, University College London, London, United Kingdom

    Max Planck University College London Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
    Search for articles by this author
  • Hugo G. Schnack
    Department of Psychiatry, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
    Search for articles by this author
Published:April 20, 2018DOI:


      Psychiatric prognosis is a difficult problem. Making a prognosis requires looking far into the future, as opposed to making a diagnosis, which is concerned with the current state. During the follow-up period, many factors will influence the course of the disease. Combined with the usually scarcer longitudinal data and the variability in the definition of outcomes/transition, this makes prognostic predictions a challenging endeavor. Employing neuroimaging data in this endeavor introduces the additional hurdle of high dimensionality. Machine learning techniques are especially suited to tackle this challenging problem. This review starts with a brief introduction to machine learning in the context of its application to clinical neuroimaging data. We highlight a few issues that are especially relevant for prediction of outcome and transition using neuroimaging. We then review the literature that discusses the application of machine learning for this purpose. Critical examination of the studies and their results with respect to the relevant issues revealed the following: 1) there is growing evidence for the prognostic capability of machine learning–based models using neuroimaging; and 2) reported accuracies may be too optimistic owing to small sample sizes and the lack of independent test samples. Finally, we discuss options to improve the reliability of (prognostic) prediction models. These include new methodologies and multimodal modeling. Paramount, however, is our conclusion that future work will need to provide properly (cross-)validated accuracy estimates of models trained on sufficiently large datasets. Nevertheless, with the technological advances enabling acquisition of large databases of patients and healthy subjects, machine learning represents a powerful tool in the search for psychiatric biomarkers.


      To read this article in full you will need to make a payment


        • Dazzan P.
        • Arango C.
        • Fleischacker W.
        • Galderisi S.
        • Glenthoj B.
        • Leucht S.
        • et al.
        Magnetic resonance imaging and the prediction of outcome in first-episode schizophrenia: A review of current evidence and directions for future research.
        Schizophr Bull. 2015; 41: 574-583
        • Gifford G.
        • Crossley N.
        • Fusar-Poli P.
        • Schnack H.G.
        • Kahn R.S.
        • Koutsouleris N.
        • et al.
        Using neuroimaging to help predict the onset of psychosis.
        Neuroimage. 2017; 145: 209-217
        • Arbabshirani M.R.
        • Plis S.
        • Sui J.
        • Calhoun V.D.
        Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls.
        Neuroimage. 2017; 145: 137-165
        • Wolfers T.
        • Buitelaar J.K.
        • Beckmann C.F.
        • Franke B.
        • Marquand A.F.
        From estimating activation locality to predicting disorder: A review of pattern recognition for neuroimaging-based psychiatric diagnostics.
        Neurosci Biobehav Rev. 2015; 57: 328-349
        • Breiman L.
        Statistical modeling: The two cultures (with comments and a rejoinder by the author).
        Stat Sci. 2001; 16: 199-231
        • Arlot S.
        • Celisse A.
        A survey of cross-validation procedures for model selection.
        Stat Surv. 2010; 4: 40-79
        • Varma S.
        • Simon R.
        Bias in error estimation when using cross-validation for model selection.
        BMC Bioinformatics. 2006; 7: 91
        • Varoquaux G.
        Cross-validation failure: Small sample sizes lead to large error bars [published online ahead of print Jun 24].
        Neuroimage. 2017;
        • Kohavi R.
        A study of cross-validation and bootstrap for accuracy estimation and model selection.
        in: IJCAI-95: Proceedings of the 14th International Joint Conference on Artificial Intelligence. Morgan Kaufmann, Montreal, Canada1995: 1137-1143
        • Stelzer J.
        • Chen Y.
        • Turner R.
        Statistical inference and multiple testing correction in classification-based multi-voxel pattern analysis (MVPA): Random permutations and cluster size control.
        Neuroimage. 2013; 65: 69-82
        • Simmons J.P.
        • Nelson L.D.
        • Simonsohn U.
        False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.
        Psychol Sci. 2011; 22: 1359-1366
        • Costafreda S.G.
        • Chu C.
        • Ashburner J.
        • Fu C.H.Y.
        Prognostic and diagnostic potential of the structural neuroanatomy of depression.
        PLoS One. 2009; 4: e6353
        • Khodayari-Rostamabad A.
        • Reilly J.P.
        • Hasey G.
        • de Bruin H.
        • MacCrimmon D.
        Using pre-treatment EEG data to predict response to SSRI treatment for MDD.
        Conf Proc IEEE Eng Med Biol Soc. 2010; 2010: 6103-6106
        • Khodayari-Rostamabad A.
        • Reilly J.P.
        • Hasey G.M.
        • de Bruin H.
        • MacCrimmon D.
        Using pre-treatment electroencephalography data to predict response to transcranial magnetic stimulation therapy for major depression.
        Conf Proc IEEE Eng Med Biol Soc. 2011; 2011: 6418-6421
        • Gong Q.
        • Wu Q.
        • Scarpazza C.
        • Lui S.
        • Jia Z.
        • Marquand A.
        • et al.
        Prognostic prediction of therapeutic response in depression using high-field MR imaging.
        Neuroimage. 2011; 55: 1497-1503
        • Korgaonkar M.S.
        • Williams L.M.
        • Song Y.J.
        • Usherwood T.
        • Grieve S.M.
        Diffusion tensor imaging predictors of treatment outcomes in major depressive disorder.
        Br J Psychiatry. 2014; 205: 321-328
        • Miller J.M.
        • Schneck N.
        • Siegle G.J.
        • Chen Y.
        • Ogden R.T.
        • Kikuchi T.
        • et al.
        fMRI response to negative words and SSRI treatment outcome in major depressive disorder: A preliminary study.
        Psychiatry Res Neuroimaging. 2013; 214: 296-305
        • Erguzel T.T.
        • Ozekes S.
        • Gultekin S.
        • Tarhan N.
        • Hizli Sayar G.
        • Bayram A.
        Neural network based response prediction of rTMS in major depressive disorder using QEEG cordance.
        Psychiatry Investig. 2015; 12: 61
        • Patel M.J.
        • Andreescu C.
        • Price J.C.
        • Edelman K.L.
        • Reynolds C.F.
        • Aizenstein H.J.
        Machine learning approaches for integrating clinical and imaging features in late-life depression classification and response prediction: Prediction models for late-life depression.
        Int J Geriatr Psychiatry. 2015; 30: 1056-1067
        • Schmaal L.
        • Marquand A.F.
        • Rhebergen D.
        • van Tol M.-J.
        • Ruhé H.G.
        • van der Wee N.J.A.
        • et al.
        Predicting the naturalistic course of major depressive disorder using clinical and multimodal neuroimaging information: A multivariate pattern recognition study.
        Biol Psychiatry. 2015; 78: 278-286
        • Williams L.M.
        • Korgaonkar M.S.
        • Song Y.C.
        • Paton R.
        • Eagles S.
        • Goldstein-Piekarski A.
        • et al.
        Amygdala reactivity to emotional faces in the prediction of general and medication-specific responses to antidepressant treatment in the randomized iSPOT-D trial.
        Neuropsychopharmacology. 2015; 40: 2398-2408
        • van Waarde J.A.
        • Scholte H.S.
        • van Oudheusden L.J.B.
        • Verwey B.
        • Denys D.
        • van Wingen G.A.
        A functional MRI marker may predict the outcome of electroconvulsive therapy in severe and treatment-resistant depression.
        Mol Psychiatry. 2015; 20: 609-614
        • Goldstein-Piekarski A.N.
        • Korgaonkar M.S.
        • Green E.
        • Suppes T.
        • Schatzberg A.F.
        • Hastie T.
        • et al.
        Human amygdala engagement moderated by early life stress exposure is a biobehavioral target for predicting recovery on antidepressants.
        Proc Natl Acad Sci U S A. 2016; 113: 11955-11960
        • Redlich R.
        • Opel N.
        • Grotegerd D.
        • Dohm K.
        • Zaremba D.
        • Bürger C.
        • et al.
        Prediction of individual response to electroconvulsive therapy via machine learning on structural magnetic resonance imaging data.
        JAMA Psychiatry. 2016; 73: 557
        • Mumtaz W.
        • Xia L.
        • Mohd Yasin M.A.
        • Azhar Ali S.S.
        • Malik A.S.
        A wavelet-based technique to predict treatment outcome for Major Depressive Disorder.
        PLoS One. 2017; 12: e0171409
        • Al-Kaysi A.M.
        • Al-Ani A.
        • Loo C.K.
        • Breakspear M.
        • Boonstra T.W.
        Predicting brain stimulation treatment outcomes of depressed patients through the classification of EEG oscillations.
        Conf Proc IEEE Eng Med Biol Soc. 2016; 2016: 5266-5269
        • Drysdale A.T.
        • Grosenick L.
        • Downar J.
        • Dunlop K.
        • Mansouri F.
        • Meng Y.
        • et al.
        Resting-state connectivity biomarkers define neurophysiological subtypes of depression.
        Nat Med. 2016; 23: 28-38
        • Crane N.A.
        • Jenkins L.M.
        • Bhaumik R.
        • Dion C.
        • Gowins J.R.
        • Mickey B.J.
        • et al.
        Multidimensional prediction of treatment response to antidepressants with cognitive control and functional MRI.
        Brain. 2017; 140: 472-486
        • Koutsouleris N.
        • Meisenzahl E.M.
        • Davatzikos C.
        • Bottlender R.
        • Frodl T.
        • Scheuerecker J.
        • et al.
        Use of neuroanatomical pattern classification to identify subjects in at-risk mental states of psychosis and predict disease transition.
        Arch Gen Psychiatry. 2009; 66: 700
        • Khodayari-Rostamabad A.
        • Hasey G.M.
        • MacCrimmon D.J.
        • Reilly J.P.
        • Bruin H de
        A pilot study to determine whether machine learning methodologies using pre-treatment electroencephalography can predict the symptomatic response to clozapine therapy.
        Clin Neurophysiol. 2010; 121: 1998-2006
        • Koutsouleris N.
        • Borgwardt S.
        • Meisenzahl E.M.
        • Bottlender R.
        • Möller H.-J.
        • Riecher-Rössler A.
        Disease prediction in the at-risk mental state for psychosis using neuroanatomical biomarkers: Results from the FePsy study.
        Schizophr Bull. 2012; 38: 1234-1246
        • Mourão-Miranda J.
        • Reinders A.A.T.S.
        • Rocha-Rego V.
        • Lappin J.
        • Rondina J.
        • Morgan C.
        • et al.
        Individualized prediction of illness course at the first psychotic episode: A support vector machine MRI study.
        Psychol Med. 2012; 42: 1037-1047
        • Koutsouleris N.
        • Riecher-Rössler A.
        • Meisenzahl E.M.
        • Smieskova R.
        • Studerus E.
        • Kambeitz-Ilankovic L.
        • et al.
        Detecting the psychosis prodrome across high-risk populations using neuroanatomical biomarkers.
        Schizophr Bull. 2015; 41: 471-482
        • Kambeitz-Ilankovic L.
        • Meisenzahl E.M.
        • Cabral C.
        • von Saldern S.
        • Kambeitz J.
        • Falkai P.
        • et al.
        Prediction of outcome in the psychosis prodrome using neuroanatomical pattern classification.
        Schizophr Res. 2016; 173: 159-165
        • Ramyead A.
        • Studerus E.
        • Kometer M.
        • Uttinger M.
        • Gschwandtner U.
        • Fuhr P.
        • Riecher-Rössler A.
        Prediction of psychosis using neural oscillations and machine learning in neuroleptic-naïve at-risk patients.
        World J Biol Psychiatry. 2016; 17: 285-295
        • Nieuwenhuis M.
        • Schnack H.G.
        • van Haren N.E.
        • Lappin J.
        • Morgan C.
        • Reinders A.A.
        • et al.
        Multi-center MRI prediction models: Predicting sex and illness course in first episode psychosis patients.
        Neuroimage. 2017; 145: 246-253
        • de Wit S.
        • Ziermans T.B.
        • Nieuwenhuis M.
        • Schothorst P.F.
        • van Engeland H.
        • Kahn R.S.
        • et al.
        Individual prediction of long-term outcome in adolescents at ultra-high risk for psychosis: Applying machine learning techniques to brain imaging data.
        Hum Brain Mapp. 2017; 38: 704-714
        • Koutsouleris N.
        • Wobrock T.
        • Guse B.
        • Langguth B.
        • Landgrebe M.
        • Eichhammer P.
        • et al.
        Predicting response to repetitive transcranial magnetic stimulation in patients with schizophrenia using structural magnetic resonance imaging: A multisite machine learning analysis.
        Schizophr Bull. 2017; ([published online ahead of print Aug 31])
        • Fleck D.E.
        • Ernest N.
        • Adler C.M.
        • Cohen K.
        • Eliassen J.C.
        • Norris M.
        • et al.
        Prediction of lithium response in first-episode mania using the LITHium Intelligent Agent (LITHIA): Pilot data and proof-of-concept.
        Bipolar Disord. 2017; 19: 259-272
        • Ahmadlou M.
        • Rostami R.
        • Sadeghi V.
        Which attention-deficit/hyperactivity disorder children will be improved through neurofeedback therapy? A graph theoretical approach to neocortex neuronal network of ADHD.
        Neurosci Lett. 2012; 516: 156-160
        • Kim J.-W.
        • Sharma V.
        • Ryan N.D.
        Predicting methylphenidate response in ADHD using machine learning approaches.
        Int J Neuropsychopharmacol. 2015; 18: pyv052
        • Ishii-Takahashi A.
        • Takizawa R.
        • Nishimura Y.
        • Kawakubo Y.
        • Hamada K.
        • Okuhata S.
        • et al.
        Neuroimaging-aided prediction of the effect of methylphenidate in children with attention-deficit hyperactivity disorder: A randomized controlled trial.
        Neuropsychopharmacology. 2015; 40: 2676-2685
        • Plitt M.
        • Barnes K.A.
        • Wallace G.L.
        • Kenworthy L.
        • Martin A.
        Resting-state functional connectivity predicts longitudinal change in autistic traits and adaptive functioning in autism.
        Proc Natl Acad Sci U S A. 2015; 112: E6699-E6706
        • Yang D.
        • Pelphrey K.A.
        • Sukhodolsky D.G.
        • Crowley M.J.
        • Dayan E.
        • Dvornek N.C.
        • et al.
        Brain responses to biological motion predict treatment outcome in young children with autism.
        Transl Psychiatry. 2016; 6: e948
        • Schnack H.G.
        • Kahn R.S.
        Detecting neuroimaging biomarkers for psychiatric disorders: Sample size matters.
        Front Psychiatry. 2016; 7: 50
        • Dluhoš P.
        • Schwarz D.
        • Cahn W.
        • van Haren N.
        • Kahn R.
        • Španiel F.
        • et al.
        Multi-center machine learning in imaging psychiatry: A meta-model approach.
        Neuroimage. 2017; 155: 10-24
        • Thompson P.M.
        • Stein J.L.
        • Medland S.E.
        • Hibar D.P.
        • Vasquez A.A.
        • Renteria M.E.
        • et al.
        The ENIGMA Consortium: Large-scale collaborative analyses of neuroimaging and genetic data.
        Brain Imaging Behav. 2014; 8: 153-182
        • Sudlow C.
        • Gallacher J.
        • Allen N.
        • Beral V.
        • Burton P.
        • Danesh J.
        • et al.
        UK Biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age.
        PLoS Med. 2015; 12: e1001779
      1. First M, Botteron K, Castellanos FX, Dickstein DP, Drevets WC, Kim KL, et al. (2012): Consensus Report of the APA Work Group on Neuroimaging Markers of Psychiatric Disorders. Available at: Accessed January 25, 2018.

      2. PSYSCAN. Translating neuroimaging findings from research into clinical practice. Available at: Accessed November 28, 2017.

      3. PRONIA - FP7 Research Project. Available at: Accessed November 28, 2017.

        • Mendelson A.F.
        • Zuluaga M.A.
        • Lorenzi M.
        • Hutton B.F.
        • Ourselin S.
        Selection bias in the reported performances of AD classification pipelines.
        Neuroimage Clin. 2017; 14: 400-416
        • Schnack H.G.
        Improving individual predictions: Machine learning approaches for detecting and attacking heterogeneity in schizophrenia (and other psychiatric diseases).
        Schizophr Res. 2017; ([published online ahead of print Oct 24])
      4. Donini M, Monteiro JM, Pontil M, Shawe-Taylor J, Mourao-Miranda J (2016): A multimodal multiple kernel learning approach to Alzheimer’s disease detection. Presented at the 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), September 13–16, 2016, Vietri sul Mare, Italy.

        • Filippone M.
        • Marquand A.F.
        • Blain C.R.V.
        • Williams S.C.R.
        • Mourão-Miranda J.
        • Girolami M.
        Probabilistic prediction of neurological disorders with a statistical assessment of neuroimaging data modalities.
        Ann Appl Stat. 2012; 6: 1883-1905
        • Tong T.
        • Gray K.
        • Gao Q.
        • Chen L.
        • Rueckert D.
        Multi-modal classification of Alzheimer’s disease using nonlinear graph fusion.
        Pattern Recognit. 2017; 63: 171-181
        • Meng X.
        • Jiang R.
        • Lin D.
        • Bustillo J.
        • Jones T.
        • Chen J.
        • et al.
        Predicting individualized clinical measures by a generalized prediction framework and multimodal fusion of MRI data.
        Neuroimage. 2017; 145: 218-229
        • Zhang D.
        • Shen D.
        Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer’s disease.
        Neuroimage. 2012; 59: 895-907
        • Zhang D.
        • Wang Y.
        • Zhou L.
        • Yuan H.
        • Shen D.
        Multimodal classification of Alzheimer’s disease and mild cognitive impairment.
        Neuroimage. 2011; 55: 856-867
        • Wan J.
        • Zhang Z.
        • Yan J.
        • Li T.
        • Rao B.D.
        • Fang S.
        • et al.
        Sparse Bayesian multi-task learning for predicting cognitive outcomes from neuroimaging measures in Alzheimer’s disease.
        Conf Proc IEEE Comput Vis Pattern Recognit. 2012; 2012: 940-947
        • Rahim M.
        • Thirion B.
        • Bzdok D.
        • Buvat I.
        • Varoquaux G.
        Joint prediction of multiple scores captures better individual traits from brain images.
        Neuroimage. 2017; 158: 145-154
        • Young J.
        • Modat M.
        • Cardoso M.J.
        • Mendelson A.
        • Cash D.
        • Ourselin S.
        Accurate multimodal probabilistic prediction of conversion to Alzheimer’s disease in patients with mild cognitive impairment.
        Neuroimage Clin. 2013; 2: 735-745
        • Filipovych R.
        • Resnick S.M.
        • Davatzikos C.
        Multi-kernel classification for integration of clinical and imaging data: Application to prediction of cognitive decline in older adults.
        in: Suzuki K. Wang F. Shen D. Yan P. MLMI 2011: Machine Learning in Medical Imaging. 7009. Springer, Berlin2011: 26-34
        • Hinrichs C.
        • Singh V.
        • Xu G.
        • Johnson S.C.
        Predictive markers for AD in a multi-modality framework: An analysis of MCI progression in the ADNI population.
        Neuroimage. 2011; 55: 574-589
        • Owen M.J.
        New approaches to psychiatric diagnostic classification.
        Neuron. 2014; 84: 564-571
        • Barch D.M.
        Biotypes: Promise and pitfalls.
        Biol Psychiatry. 2017; 82: 2-3
        • Mourão-Miranda J.
        • Hardoon D.R.
        • Hahn T.
        • Marquand A.F.
        • Williams S.C.R.
        • Shawe-Taylor J.
        • Brammer M.
        Patient classification as an outlier detection problem: An application of the One-Class Support Vector Machine.
        Neuroimage. 2011; 58: 793-804
        • Evgeniou T.
        • Micchelli C.A.
        • Pontil M.
        Learning multiple tasks with kernel methods.
        J Mach Learn Res. 2005; 6: 615-637