If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Alberta Machine Intelligence Institute, Department of Computing Science, University of Alberta, Edmonton, Alberta, CanadaCanadian VIGOUR Centre, University of Alberta, Edmonton, Alberta, Canada
OCD Clinic, Department of Psychiatry, National Institute of Mental Health and Neuro Sciences, Bangalore, IndiaTranslational Psychiatry Laboratory, Neurobiology Research Centre, National Institute of Mental Health and Neuro Sciences, Bangalore, India
OCD Clinic, Department of Psychiatry, National Institute of Mental Health and Neuro Sciences, Bangalore, IndiaTranslational Psychiatry Laboratory, Neurobiology Research Centre, National Institute of Mental Health and Neuro Sciences, Bangalore, India
Alberta Machine Intelligence Institute, Department of Computing Science, University of Alberta, Edmonton, Alberta, CanadaDepartment of Psychiatry, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada
OCD Clinic, Department of Psychiatry, National Institute of Mental Health and Neuro Sciences, Bangalore, IndiaTranslational Psychiatry Laboratory, Neurobiology Research Centre, National Institute of Mental Health and Neuro Sciences, Bangalore, India
OCD Clinic, Department of Psychiatry, National Institute of Mental Health and Neuro Sciences, Bangalore, IndiaTranslational Psychiatry Laboratory, Neurobiology Research Centre, National Institute of Mental Health and Neuro Sciences, Bangalore, India
Machine learning applications using neuroimaging provide a multidimensional, data-driven approach that captures the level of complexity necessary for objectively aiding diagnosis and prognosis in psychiatry. However, models learned from small training samples often have limited generalizability, which continues to be a problem with automated diagnosis of mental illnesses such as obsessive-compulsive disorder (OCD). Earlier studies have shown that features incorporating prior neurobiological knowledge of brain function and combining brain parcellations from various sources can potentially improve the overall prediction. However, it is unknown whether such knowledge-driven methods can provide a performance that is comparable to state-of-the-art approaches based on neural networks.
Methods
In this study, we apply a transparent and explainable multiparcellation ensemble learning framework EMPaSchiz (Ensemble algorithm with Multiple Parcellations for Schizophrenia prediction) to the task of predicting OCD, based on a resting-state functional magnetic resonance imaging dataset of 350 subjects. Furthermore, we apply transfer learning using the features found effective for schizophrenia to OCD to leverage the commonality in brain alterations across these psychiatric diagnoses.
Results
We show that our knowledge-based approach leads to a prediction performance of 80.3% accuracy for OCD diagnosis that is better than domain-agnostic and automated feature design using neural networks. Furthermore, we show that a selection of reduced feature sets can be transferred from schizophrenia to the OCD prediction model without significant loss in prediction performance.
Conclusions
This study presents a machine learning framework for OCD prediction with neurobiology-aided feature design using resting-state functional magnetic resonance imaging that is generalizable and reasonably interpretable.
In the past few years, there has been an increased adaptation of machine learning methods in psychiatric applications, often to produce models that predict the diagnosis of novel subjects (
). Many studies have demonstrated that functional magnetic resonance imaging (fMRI) of the brain has sufficient information to generate models that can discriminate healthy control (HC) subjects from patients with various psychiatric illnesses, such as autism (
Shenas SK, Halici U, Cicek M (2013): Detection of obsessive compulsive disorder using resting-state functional connectivity data. Presented at the 6th International Conference on Biomedical Engineering and Informatics, December 16–18, 2013, Hangzhou, China.
Multivariate classification of drug-naive obsessive-compulsive disorder patients and healthy controls by applying an SVM to resting-state functional MRI data.
) (see list of OCD studies in Table S2 in Supplement 1). However, most of these studies were conducted on small datasets that used fewer than 100 subjects for both learning and cross-validation of the learned model (
Shenas SK, Halici U, Cicek M (2013): Detection of obsessive compulsive disorder using resting-state functional connectivity data. Presented at the 6th International Conference on Biomedical Engineering and Informatics, December 16–18, 2013, Hangzhou, China.
). Limited samples can severely limit the generalizability of such models, and in fact, many have noted that small sample sizes can lead to overestimated measures of performance (
Multivariate classification of drug-naive obsessive-compulsive disorder patients and healthy controls by applying an SVM to resting-state functional MRI data.
While it is essential to have sufficient information to learn a generalizable model, pragmatically it is challenging to conduct studies that sample thousands of psychiatric patients. Therefore, it is critical to extract as much relevant information as possible from the limited data at hand to build a generalizable model and, in particular, to reduce the chance of overfitting. Our earlier research explored a way to design features—each a combination of one of several regional connectivity-based measures and one of the various parcellation maps (each incorporating a unique source of prior neurobiological knowledge)—and then learn a classifier based on these features. We found that this framework, called EMPaSchiz (Ensemble algorithm with Multiple Parcellations for Schizophrenia prediction, read as “Emphasis”), could effectively discriminate patients with drug-naïve SCZ from HC subjects (
). However, this approach has not been explored to distinguish patients with OCD from HC subjects; none of the earlier machine learning studies in OCD (
Shenas SK, Halici U, Cicek M (2013): Detection of obsessive compulsive disorder using resting-state functional connectivity data. Presented at the 6th International Conference on Biomedical Engineering and Informatics, December 16–18, 2013, Hangzhou, China.
Multivariate classification of drug-naive obsessive-compulsive disorder patients and healthy controls by applying an SVM to resting-state functional MRI data.
) have combined the regional and connectivity features, nor have any used an ensemble approach to learn from multiple parcellations jointly. Moreover, earlier studies have not explored whether such knowledge-driven methods can provide performance comparable to the popular machine learning approach of applying neural networks (NNs), which are generally known to provide state-of-the-art results (
Today, psychiatric disorders are classified based on a consensus about clusters of symptoms that patients experience currently or in the past. Classificatory systems, such as the DSM, have provided clinicians worldwide with a standard framework to identify, treat, and manage these conditions (
). However, while most scholars are convinced that pathology underlying symptoms of psychiatric illnesses is attributable to features of neural pathways and their interfacing axes, most clinical decisions in psychiatric practice are based neither on the etiologic mechanisms (
). Decades of research toward the understanding of neuropathology underlying these illnesses have shown only weak and unclear correspondence between brain measures and clinical categorization (
). This observation can be potentially leveraged when learning models by transferring knowledge gained with a dataset of one condition to help diagnose another condition, where features relevant for predicting one psychiatric disorder can be used to predict another; this is considered a type of transfer learning (
). In this study, we chose to transfer information obtained by learning models for SCZ diagnosis to OCD diagnosis. The two motivating factors for our choice were 1) SCZ and OCD were shown to have shared resting-state features (
Altered resting-state brain activity in schizophrenia and obsessive-compulsive disorder compared with non-psychiatric controls: Commonalities and distinctions across disorders.
). Hence, it is valuable to build models that can provide interpretability in terms of providing 1) auxiliary arguments in favor of the model’s correctness, 2) transparency with the usage of features explainable by domain experts such as psychiatrists, and 3) reproducibility to ensure that the model can be trusted in high-stakes situations such as medical diagnosis.
In this study, we applied the EMPaSchiz learning approach to an OCD versus control dataset to produce a model that can diagnose OCD and show empirically that this use of EMPaSchiz demonstrates some of the points mentioned above. We believe that these factors are relevant to the future of neuroimage-based machine learning methods for psychiatric diagnosis and medical diagnosis in general.
This research explores whether 1) this EMPaSchiz approach can produce models that can predict OCD accurately in an interpretable way; 2) customized feature design, based on prior neurobiological knowledge (parcellations), can produce models that are comparable to knowledge-agnostic automated methods (neural nets); and 3) one can transfer information obtained by learning models for SCZ diagnosis to OCD diagnosis by using the high-level features selected from the model learned for predicting SCZ for the task of predicting OCD.
Methods and Materials
Dataset
Our study sample contained 188 patients attending the OCD Clinic of the National Institute of Mental Health & Neuro Sciences, India, who fulfilled DSM-IV criteria for OCD. The diagnosis of OCD was established using the Mini-International Neuropsychiatric Interview Plus (
The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10.
), which was confirmed by another psychiatrist through an independent clinical interview (YCJR or JCN). We used the Yale-Brown Obsessive Compulsive Scale (
HC subjects were recruited from among consenting healthy volunteers from the same locale to match for age and sex. We used 200 age- and sex-matched HC subjects, who were screened to rule out any psychiatric diagnosis using the Mini-International Neuropsychiatric Interview. For both patients and control subjects, we recruited only right-handed subjects to avoid the potential confounds of differential handedness. None of the study subjects had contraindications to MRI or medical illnesses that could significantly influence central nervous system function or structure, such as seizure disorder, cerebral palsy, or history suggestive of delayed developmental milestones. There was no lifetime history suggestive of DSM-IV psychoactive substance dependence or of head injury associated with loss of consciousness longer than 10 minutes. No subjects had abnormal movements as assessed by the Abnormal Involuntary Movements Scale (
). Pregnant or postpartum women were excluded. Table S3 in Supplement 1 provides details of the demographic and clinical profiles of subjects who qualified to be included in the study.
In addition to patients with OCD and HC subjects, we also used a dataset of patients with drug-naïve SCZ for transfer learning analyses in this study. This cohort has been analyzed in our previous study (
), and basic demographic information of these subjects is provided in Table S4 in Supplement 1.
Methodological details related to image acquisition and preprocessing are provided in Section F in Supplement 1. We excluded images for 8 patients and 7 control subjects from the study based on excessive head movement (translational >2.0 mm and/or rotational >2°) (
) to avoid class differences in head motion. In addition, 5 patients and 18 control subjects were excluded due to incomplete imaging or clinical data. This yielded a total of 350 subjects: 175 control subjects and 175 patients.
We obtained written informed consent after providing a complete description of the study to all subjects. The National Institute of Mental Health & Neuro Sciences ethics committee reviewed and approved the original research protocol. The Research Ethics Board at the University of Alberta, Edmonton, approved the secondary analysis of deidentified, preprocessed data. All methods were performed in accordance with the relevant guidelines and regulations.
EMPaSchiz
We used EMPaSchiz to learn a model to predict the diagnosis of OCD. A detailed description of the EMPaSchiz system is provided in the original paper (
). Briefly, EMPaSchiz extracts six resting-state brain fMRI features, including three regional-based features (amplitude of low-frequency fluctuations [ALFF], fractional ALFFs [fALFFs], regional homogeneity [ReHo]) and three connectivity-based features (using functional connectivity [FC] matrices between each pair of regions [see parcellations below] using one of three statistical measures: FC-Pearson correlation, FC-partial correlation, or FC-precision). ALFF was calculated as total power within the frequency range between 0.01 and 0.08 Hz to estimate the strength of low-frequency oscillations (
). fALFF was calculated as the power within the low-frequency range (0.01–0.08 Hz) divided by the total power in the entire detectable frequency range (
). To obtain neurobiologically relevant feature compressions, EMPaSchiz projected each feature extraction onto 14 different parcellation schemes, each based on a specific predefined brain atlas or set of regions of interest. These schemes varied widely in principle: 1) predefined ontology of brain structures such as postmortem cytoarchitecture (
Multi-subject dictionary learning to segment an atlas of brain spontaneous activity.
in: Székely G. Hahn H.K. Information Processing in Medical Imaging: 22nd International Conference, IPMI 2011, Kloster Irsee, Germany, July 3-8, 2011. Proceedings. Springer,
Berlin, Germany2011: 562-573
Multi-subject dictionary learning to segment an atlas of brain spontaneous activity.
in: Székely G. Hahn H.K. Information Processing in Medical Imaging: 22nd International Conference, IPMI 2011, Kloster Irsee, Germany, July 3-8, 2011. Proceedings. Springer,
Berlin, Germany2011: 562-573
). EMPaSchiz first learns 84 (14 parcellation schemes × [3 + 3] feature types) single-source models (SSMs), each applying L2-regularized logistic regression to learn a classifier, for one (feature type, parcellation) description of the data. It then applies L2-regularized logistic regression to the 84 prediction probabilities produced by each of these learned models over the training set to learn a final ensemble system. At performance time, given a new instance, the learned system will first produce the 84 descriptions of that new instance, run those 84 SSMs to produce 84 responses, and feed those values into the final learned function to output the final OCD-versus-HC label.
Neural Networks
To examine the importance of incorporating prior neuroanatomical and neurofunctional knowledge in the form of feature design and brain parcellations, we compared the performance of EMPaSchiz with NNs, which are one of the most effective approaches today. Recent projects (
Zafar R, Malik AS, Shuaibu AN, Javvad ur Rehman M, Dass SC (2017): Classification of fMRI data using support vector machine and convolutional neural network. Presented at the IEEE International Conference on Signal and Image Processing Applications (ICSIPA), September 12–14, 2017, Kuching, Malaysia.
) using NN models with fMRI data for various psychiatric and neurologic tasks have used several consecutive convolutions and max-pooling layers (where the specific network architecture is not fixed but can depend on the specific task) to extract high-level features. Most of these studies (
) show that applying 3D convolution on fMRI data (here, entire brains rather than slices) is more effective than using two-dimensional convolution to produce accurate predictors. We performed this analysis with three levels, adding more domain knowledge in each step (Figure 1).
Figure 1The architecture of obsessive-compulsive disorder classification using neural network (NN)-1, NN-2, and NN-3. (A) NN-1: two convolution layers with four parameters depicted in order—kernel size, padding, input channel numbers, and number of filters. This is followed by a max-pooling layer with two parameters, kernel size and stride. Then, the linearized output of the max-pooling layer before the layers of long short-term memory (LSTM) has two parameters, number of layers and number of units in each layer. (B) NN-2: model parameters are depicted in a similar format as NN-1. After max pooling, there are two fully connected layers with one parameter, number of units. (C) NN-3: four fully connected layers with one parameter, number of units. For all methods, the final output layer has two nodes, obsessive-compulsive disorder vs. healthy control subjects (for each test subject, the learned system will return the argmax of the values computed here). 4D, four-dimensional; ALFF, amplitude of low-frequency fluctuations; fALFF, fractional ALFF; ReHo, regional homogeneity; Relu, rectified linear unit.
NN-1: NN Using Preprocessed fMRI Images Without Manual Feature Design
Here, we input four-dimensional fMRI images that are completely preprocessed and let NN-1 automatically learn relevant features from the data. The learned NN-1 model takes a fixed input size of 143 × 61 × 73 × 61 for each subject, where 143 is the time dimension and the other three dimensions represent the 3D brain volume. This model consists of two 3D convolution layers with one max-pooling layer, three layers of long short-term memory, and two fully connected layers.
NN-2: NN Using Designed Features but Without Parcellation-Based Aggregation
Here, we extracted the three different feature types (ALFF, fALFF, and ReHo features) using the EMPaSchiz framework but did not aggregate the features based on the brain parcellations. Instead, NN-2 uses a convolutional NN to predict the OCD label for each of the three feature types individually. The learned NN-2 model takes a fixed input size of 61 × 73 × 61. It consists of two convolution layers with one max-pooling layer and two fully connected layers.
NN-3: NN Using Designed Features and Brain Parcellations
Here, we used the feature extractions and parcellation schemes similar to those of EMPaSchiz. Instead of learning with the simplistic and explainable logistic regression framework, NN-3 instead uses a more complex NN method for the final step. The critical difference is that while NNs can update the weights in lower layers using backpropagation, EMPaSchiz cannot. For each subject, the learned NN-3 model takes a concatenated vector of 84 feature sets derived from the EMPaSchiz framework and consists of four fully connected layers of dimensions of sizes: 1000, 100, 40, and 2.
Figure 1 shows the architectures of these four NN models. For each NN model, we used rectified linear unit as an activation function for each layer and cross-entropy as a loss function. To avoid overfitting, 50% of layers were dropped out during the training time. We used a maximum of 1000 epochs to train our models with early-stopping criteria for 100 epochs, i.e., we calculated the validation error after each training epoch, and if the error was found to be not decreasing for a span of 100 epochs, then the training state was reverted by 100 epochs. Models were implemented in PyTorch (v.1.0.1) (
Siu K, Stuart DM, Mahmoud M, Moshovos A (2018): Memory requirements for convolutional neural network hardware accelerators. Presented at the IEEE International Symposium on Workload Characterization (IISWC), September 30–October 2, 2018, Raleigh, North Carolina.
As mentioned above, we earlier used EMPaSchiz to distinguish patients with SCZ from HC subjects; this used a relatively small set of SSMs. Here, we ask whether these SSMs (that were sufficient for SCZ diagnosis) are adequate for diagnosing another psychiatric disorder, OCD. In particular, we limited the SSMs of EMPaSchiz while learning to predict OCD to only those selected in the learned model of SCZ prediction. That earlier model (
) was based on 81 patients with SCZ and 93 HC subjects. Our OCD dataset was also collected from the same site, which initially included 175 patients with OCD and 175 HC subjects. To avoid any bias raising due to overlapping subjects in the HC group, we specifically excluded the control subjects who were included in the SCZ study, leaving 88 HC subjects. Running the EMPaSchiz learner on 81 patients with SCZ and 93 HC subjects [results for SCZ prediction model are available elsewhere (
)] used L1-regularization techniques to select only 10 of the 84 SSMs; we then ran that learner using only those selected 10 SSMs to learn an OCD prediction model, based on a dataset of 175 patients with OCD and 88 control subjects, disjoint from the HC subjects used for producing the SCZ model (SCZ_to_OCD transfer model). We also present the results of the transfer model that includes all HC subjects (SCZ_to_OCD_CommonHC); the sample distributions of these analyses are provided in Table S8 in Supplement 1.
Model Evaluation
For evaluation of learned models, we performed fivefold balanced cross-validation in five shuffled iterations (80% training set, 20% test set, for a total of 25 train-test splits). The models’ generalization performance on the held-out fold is estimated using accuracy, sensitivity, specificity, and precision. We report the mean and standard errors for these metrics and elements of confusion matrices over all 25 train-test splits for each variant.
Results
OCD Prediction
The EMPaSchiz algorithm was able to predict OCD with 80.3% accuracy using the 5 times fivefold cross-validation. The model was 82.7% sensitive, 79.2% precise, and 77.8% specific. Table 1 shows the results for the performance of the EMPaSchiz algorithm and also for submodels that stack SSMs only from specific feature extractions. EMPaSchiz’s performance was significantly better than any of those subset-stacked models (compared with the best subset-stacked model, stacked-FC-precision, at 77.9%, t test, p = .018). Figure 2 shows a comparative profile of accuracies for various SSM predictors, parcellation-wise stacked models, and EMPaSchiz.
Table 1EMPaSchiz Results for OCD Diagnosis Prediction
Figure 2Comparison of 5 × 5-fold cross-validation prediction accuracies for single-source models and EMPaSchiz (error bar corresponds to the standard error of the mean). ALFF, amplitude of low-frequency fluctuations; fALFF, fractional ALFF; FC_corr, functional connectivity with Pearson correlation; FC_part, FC with partial correlation; FC_prec, FC precision; ReHo, regional homogeneity.
Patients with OCD in our sample ranged widely in their symptom severity, which was measured using the Yale-Brown Obsessive Compulsive Scale for obsessions (integer values from 0 to 20) and compulsions (integer values from 0 to 20). For each, we used the first and last quartile of these scales to categorize the least versus the most severely symptomatic patients. We then used EMPaSchiz in a leave-one-out cross-validation setup to predict the high-symptomatic patients against low-symptomatic ones (majority class baseline accuracy was close to 50%). We used leave-one-out cross-validation (rather than fivefold) to deal with a low number of available subjects for this analysis. Prediction accuracy was 58.6% for obsessions and 64.1% for compulsions of OCD psychopathology. In view of subpar performance, we compared this performance to the model that focused on features derived only from the brain regions that are consistently implicated in OCD by meta-analysis of functional and structural neuroimaging studies (
), which included areas from the cortico-striato-thalamo-cortical (CSTC) circuit, namely, the orbitofrontal cortex, anterior cingulate cortex, prefrontal cortex, and ventral striatum (specific regions are listed in Table S6 and Figure S1 in Supplement 1). Prediction accuracy with CSTC was 58.6% for obsessions and 52.6% for compulsions. Details are provided in Tables S5 and S7 in Supplement 1.
Ante Hoc Model Interpretability
To delineate essential pathological alterations in OCD, we estimated the reliability of a feature’s importance for diagnostic prediction by sorting features by their respective mean logistic regression weight divided by the standard error for each feature in a particular learned SSM generated during 25 folds of cross-validation. Figure 3 (respectively, Figure 4) highlights some of the topmost (>98th or 99th percentile) reliable features using representative atlases for regional resting-state measures (respective connectivity). However, because our ensemble model is composed of 84 SSMs, these depictions should be considered representative in nature and cannot be claimed as the only important features for OCD prediction.
Figure 3Key pathological alterations in obsessive-compulsive disorder suggested by topmost reliable features—elevated (red/orange) and suppressed (blue/purple) changes in regional activity. The panels show the top 98th percentile of top regional features overlaid on glass brains. (A) Lower amplitude of low-frequency fluctuations (ALFF), fractional ALFF (fALFF) in left (L) anterior cingulate, L rectus, and right (R) paracentral lobule (destrieux). (B) Higher fALFF in bilateral inferior parietal lobule and lower fALFF in L middle frontal gyrus, R insula, and L paracentral lobule (basc_multiscale_197). (C) Higher regional homogeneity (ReHo) in L middle frontal gyrus and lower ReHo in bilateral putamen and L middle temporal gyrus (basc_multiscale_122).
Figure 4 shows representative alterations in patients with OCD compared with HC subjects that are suggested by the topmost reliable features. Network edges show elevated (red) and suppressed (blue) changes in FC. Panels show the top 99th percentile of top FC features using the Dosenbach and MSDL atlases. Within the Dosenbach atlas, we found decreased FC between the following pairs of regions: right (R) anterior prefrontal cortex and left (L) angular gyrus, R occipital gyrus and L posterior occipital gyrus, and R frontal gyrus and both R intraparietal sulcus and R superior parietal lobule. Increased FC was found between the following pairs of regions: interhemispheric occipital gyri, R dorsal anterior cingulate cortex and R parietal, and L basal ganglia and L posterior parietal cortex. These differences suggest aberrations in frontoparietal, cingulo-opercular, occipital, and sensorimotor networks. Furthermore, with the MSDL atlas, we found decreased FC between the following regions: L superior frontal sulcus and L as well as medial default mode network, and R temporoparietal junction and R parietal cortex. Increased FC was found between L inferior parietal sulcus and bilateral lateral occipital cortex as well as L visual cortex, R anterior insula and motor cortex, and L insula and L auditory cortex. These alterations might imply aberrations in several distributed networks, such as default mode, language, attention, visual, auditory, motor, and salience networks.
Comparison of EMPaSchiz to Neural Networks Methods
We compared the performance of the EMPaSchiz model to NN techniques that do not use their feature types and/or feature compression (parcellations) methods. Results (refer to Figure 5) show that the EMPaSchiz model outperforms these NN-1 methods (paired t test, p < .001), NN-2 methods for each feature type (ALFF: t test, p = .011, fALFF: paired t test, p < .001, ReHo: paired t test, p = .005), and NN-3 methods (paired t test, p < .001). Table 2 presents the 5 × 5-fold cross-validation prediction performance of the different NN models.
Figure 5Comparison of performance across EMPaSchiz and neural network (NN) methods. ALFF, amplitude of low-frequency fluctuations; fALFF, fractional ALFF; Reho, regional homogeneity.
Transfer Learning: Feature Selection Based on the SCZ Model
To deal with the SCZ task, EMPaSchiz’s L1-regularization selected only 10 of 84 SSMs; our SCZ_to_OCD transfer model used only these SSMs when dealing with the OCD task. This included only two feature types, FC-Pearson correlation and FC-precision. Parcellation types included aal, dosenbach, harvard_cort_25, MSDL, power, basc_multiscale_122, basc_multiscale_197, basc_multiscale_325, and basc_multiscale_444. More information is provided in Table S9 in Supplement 1.
When we restricted our OCD learning model to ensemble only these 10 SSMs, the accuracy of the OCD model was 93.1% (Table 3) (note, however, that we expected higher baseline performance ∼66.5% due to class imbalance: OCD = 175, HC = 88). Interestingly, we found that this SCZ_to_OCD model provided a performance that is comparable to the original EMPaSchiz that is retrained on this dataset (accuracy: 91.8%) within statistical significance margin (two-sided t test, p = .15) in OCD prediction. Table S10 in Supplement 1 provides results for the SCZ_to_OCD_CommonHC transfer model. In addition, for the sake of completion, the results for transfer learning models from OCD to SCZ are also provided (Tables S11–S13 in Supplement 1).
Table 3EMPaSchiz Transfer Learning Results for OCD Diagnosis Prediction With Feature Selection Based on SCZ Model
Models
Accuracy, %
Precision, %
Sensitivity, %
Specificity, %
True Positive
True Negative
False Positive
False Negative
SCZ_to_OCD
93.1 (0.6)
94.9 (0.6)
94.7 (0.5)
89.8 (1.2)
165.8 (0.7)
79.0 (0.8)
9.0 (0.8)
9.2 (0.7)
Original EMPaSchiz
91.8 (0.6)
90.1 (0.7)
98.7 (0.5)
77.9 (1.8)
172.8 (0.4)
68.6 (0.8)
19.4 (0.8)
2.2 (0.4)
Data are presented as mean (standard error). OCD, n = 175; HC, n = 88; baseline accuracy: 66.53.
EMPaSchiz, Ensemble algorithm with Multiple Parcellations for Schizophrenia prediction; HC, healthy control; OCD, obsessive-compulsive disorder; SCZ, schizophrenia.
From the study observations, we conclude the following:
1.
Our EMPaSchiz algorithm can predict OCD with 80.3% accuracy, which outperforms base models that use any individual feature type or parcellation scheme.
2.
Feature design based on prior neurobiological knowledge (parcellations) leads to better performance than agnostic and automated feature design (neural nets).
3.
Selection of single-source feature sets can be transferred from SCZ to OCD prediction models without significant loss in prediction performance.
4.
EMPaSchiz provides a generalizable yet reasonably interpretable linear model that uses human expert–understandable features and model structure.
Because many psychiatric disorders usually manifest with a myriad of overlapping symptoms, reliable clinical diagnosis is a challenging task (
). Here, we demonstrate a computational framework that builds on prior neurobiological information that is derived from numerous neuroanatomical and neurophysiological studies and provides a diagnostic performance that matches trained psychiatrists about 8 of 10 times. In this cohort, patients were seen by two experienced psychiatrists (YCJR and JCN) who assessed the patients independently and concurred on the diagnosis. Earlier studies have shown that diagnostic tools such as the DSM/ICD are not always reliable, and psychiatrists do not always agree on the diagnosis, with reported joint rater agreement as low as 0.2 to 0.4 (intraclass kappa) in some circumstances (
The ICD-11 developmental field study of reliability of diagnoses of high-burden mental disorders: Results among adult patients in mental health settings of 13 countries.
). Given this, one might argue that a machine with a predictive accuracy of 80% is close enough for human-like performance. Hence, the technology is worth exploring further with even bigger datasets and more sophisticated algorithms. However, this is not an apples-to-apples comparison because clinician decisions are based on a consensus about clusters of clinical symptomatology, while machine learning models are based on fMRI images, which clinicians are not able to discriminate visually. However, it is desirable to establish a clinical diagnosis on objective laboratory measurements (such as an fMRI image) because it strengthens the reproducibility of disease entities (
). Unfortunately, this is difficult here, given the earlier evidence that such brain-derived measurements only weakly correspond with clinical classifications of psychiatric conditions (
With a cross-validated prediction performance of 80.3%, EMPaSchiz ranks highest among all earlier OCD prediction studies, either using fMRI or other neuroimaging modalities, with a sample size greater than 100 subjects (
Multivariate classification of drug-naive obsessive-compulsive disorder patients and healthy controls by applying an SVM to resting-state functional MRI data.
). The earlier fMRI-based OCD prediction studies have neither combined regional and connectivity features nor used an ensemble approach such as ours to learn from multiple parcellations jointly. Most used a single feature type [such as ALFF, fALFF, ReHo, or FC-Pearson correlation (
Shenas SK, Halici U, Cicek M (2013): Detection of obsessive compulsive disorder using resting-state functional connectivity data. Presented at the 6th International Conference on Biomedical Engineering and Informatics, December 16–18, 2013, Hangzhou, China.
Multivariate classification of drug-naive obsessive-compulsive disorder patients and healthy controls by applying an SVM to resting-state functional MRI data.
Shenas SK, Halici U, Cicek M (2013): Detection of obsessive compulsive disorder using resting-state functional connectivity data. Presented at the 6th International Conference on Biomedical Engineering and Informatics, December 16–18, 2013, Hangzhou, China.
) used structural MRI or diffusion tensor imaging to focus on whole-brain anatomical alterations and white matter abnormalities, respectively, with gray matter volume, white matter volume, fractional anisotropy, and medial diffusivity as their key features. Commonly used dimensionality reduction methods were PCA, Kernel PCA and MRMR [Minimum Redundancy Maximum Relevance (
)], while classification algorithms were support vector machine, logistic regression, and linear discriminant analysis (see Table S2 for details specific to each study).
While high performance on diagnostic prediction is encouraging, note that discriminating patients with OCD from healthy individuals is perhaps an easy task for trained psychiatrists, especially with identifying subtypes based on symptom dimensions or severity. The patients in this study did not have comorbidities with psychosis, bipolar disorder, and other obsessive-compulsive–related disorders such as body dysmorphic disorder, hair pulling disorder, or excoriation disorder. Only a small proportion of the sample had depression and anxiety comorbidity (current depression, n = 21 [13.8%]; current anxiety, n = 8 [5.3%]). It may be noted that the proportion of patients with OCD with comorbidities is small, and hence it may not have affected the specificity of the findings. However, our experiments with learning models for identifying severe cases were only mildly successful. While it is expected that focusing the model on CSTC brain regions [which are most commonly implicated in OCD literature (
)] would potentially increase the prediction of symptom severity, our experiments with CSTC regions showed a performance that is inferior to EMPaSchiz’s. We discuss the possible reasons in Section G in Supplement 1. It is possible that the OCD phenotype that the algorithm learned to differentiate from the healthy brain may be independent of OCD severity. The phenotype identified may represent traits and may not be state dependent. Alternatively, these results could also suggest that fMRI data might not have enough information to perform more complicated psychiatric decisions, at least within the scope of our training size and algorithms.
To ease the bedside translation of these computational models, it is desirable that they demonstrate trustworthiness. Clinicians are more likely to use models that use features that they know, based on familiar anatomical brain regions and well-known properties, such as FC and regional homogeneity. They also prefer models that use easy-to-interpret algorithms, e.g., a linear combination of features. EMPaSchiz’s basic algorithm is a priori explainable because it uses logistic regression classifiers that are inherently interpretable because they involve linear combinations of features (
Poulin B, Eisner R, Szafron D, Lu P, Greiner R, Wishart DS, et al. (2006): Visual explanation of evidence with additive classifiers. Presented at the 21st National Conference on Artificial Intelligence, July 16–20, 2006, Boston, Massachusetts.
). Furthermore, the model’s decision can be viewed as probabilistic. The learned weights suggest how much each feature contributes toward its final decision, thus helping clinicians and researchers understand the factors leading to the decisions.
Our study faces the general limitation of most machine learning studies in psychiatry: the ground truth itself might be ill defined in terms of validity of current psychiatric classifications as unitary disease constructs. While our approach does correctly distinguish patients with OCD from HC subjects, we do not know whether it can differentially distinguish OCD from other psychiatric disorders. Results from this study do not actually prove the specificity of diagnostic prediction, which would be very useful for clinicians in aiding their diagnosis; this could be a potential future direction. In addition, our data were collected using a single MRI scanner and from individuals of fairly homogeneous ethnicity. We do not know whether it will perform similarly in multicentric data collected across mixed population groups and comorbidities. Moreover, we acknowledge that our processing pipeline is only one of the many possible sensible pipelines (
). Similar to any other machine learning study, the reported claims about the performance of our model only apply to fMRI images that are processed as described. Nonetheless, the potential effects of different choices of denoising pipelines, such as global signal regression, filtering, and scrubbing, on the prediction performance need systematic evaluation (
). Given the wide parameter space of these processing options, we believe that such a systematic analysis merits a future separate research study.
Machine learned models produce more accurate predictions as the size of the training dataset increases; therefore, it is critical to have a sufficient sample size (
). Our study used 350 subjects (175 patients with OCD and 175 HC subjects), which is the largest OCD dataset that has been used in machine learning until now [the previously largest sample size is 204 (102 patients with OCD and 100 HC subjects), which reported a prediction performance of 76.6% (
)]. Our study provides two other novel observations. First, it shows that a simple linear model with neurobiology-informed features outperforms complex NN models, even though those models can automatically design new features that can potentially exploit nonlinear interactions. Even though EMPaSchiz was trained on the same ∼252,000-dimensional data as NN-3, we anticipated that EMPaSchiz would work better for two main reasons: 1) EMPaSchiz exploits prior neurologic information because it pools those extracted feature values into brain regions or networks with coherent structure or function based on known parcellation schemes, while NN-3 (which considers all possible combinations of features) does not use this information, and 2) because NN-3 is fully connected, it must fit roughly 1000 times more weights than EMPaSchiz (∼253 million vs. ∼253,000), which for this dataset appears to lead to overfitting. Hence, the EMPaSchiz model is both more accurate and more interpretable. Second, we demonstrate cross-diagnostic transfer learning in psychiatry applications. We show that this can considerably reduce the complexity of data processing in terms of feature engineering by pretraining the model with subjects affected by a different psychiatric condition—SCZ in our case—and still retain the earlier performance accuracy. Our results suggest that this technique can be used to tackle the issue of limited sample sizes in fMRI-based machine learning research. We hypothesize that such transfer learning works in this context because the underlying brain abnormalities are shared across multiple psychiatric disorders (
). Future studies could attempt to replicate these findings between other disease groups, e.g., between OCD and other compulsive disorders such as alcohol addiction or eating disorders, as well as SCZ and bipolar disorder.
Acknowledgments and Disclosures
This study is supported by the IBM Alberta Centre for Advanced Studies (to SVK), Alberta Machine Intelligence Institute and Alberta Innovates Graduate Student Scholarship grants (to AKP), the Government of India (Grant Nos. DST SR/S0/HS/0016/2011 and BT/PR13334/Med/30/259/2009 [to YCJR], DST INSPIRE faculty Grant No. IFA12-LSBM-26 [to JCN], Grant No. DBT BT/06/IYBA/2012 [to JCN], and Grant No. 500236/Z/11/Z [to GV]), Alberta Machine Intelligence Institute and NSERC grants (to RG), and DBT Wellcome Trust India Alliance (Grant No. IA/CRC/19/1/610005 [to GV], Intermediate Fellowship Grant No. IA/CPHI/16/1/502662 [to JCN], and Early Career Fellowship Grant No. IA/CPHE/18/1/503956 [to VS]).
GV, JCN, YCJR, and VS collected the clinical and neuroimaging data. Clinical symptom ratings were done by YCJR and JCN. Data were cleaned and processed by RA, SVK, and AKP. SVK and AKP designed, and AKP implemented, the machine learning models, with supervision by SVK and RG. SVK and AKP managed the literature search, prepared figures, and wrote the first draft of the manuscript along with RG. AJG, SMD, and other authors revised and optimized further versions of the manuscript. All authors have contributed to and approved the final manuscript.
The datasets generated and/or analyzed during this study, as well as relevant computer codes that were used to process the data and to generate the results, are available from corresponding authors on a reasonable request.
The authors report no biomedical financial interests or potential conflicts of interest.
Shenas SK, Halici U, Cicek M (2013): Detection of obsessive compulsive disorder using resting-state functional connectivity data. Presented at the 6th International Conference on Biomedical Engineering and Informatics, December 16–18, 2013, Hangzhou, China.
Multivariate classification of drug-naive obsessive-compulsive disorder patients and healthy controls by applying an SVM to resting-state functional MRI data.
Altered resting-state brain activity in schizophrenia and obsessive-compulsive disorder compared with non-psychiatric controls: Commonalities and distinctions across disorders.
The Mini-International Neuropsychiatric Interview (M.I.N.I.): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10.
Multi-subject dictionary learning to segment an atlas of brain spontaneous activity.
in: Székely G. Hahn H.K. Information Processing in Medical Imaging: 22nd International Conference, IPMI 2011, Kloster Irsee, Germany, July 3-8, 2011. Proceedings. Springer,
Berlin, Germany2011: 562-573
Zafar R, Malik AS, Shuaibu AN, Javvad ur Rehman M, Dass SC (2017): Classification of fMRI data using support vector machine and convolutional neural network. Presented at the IEEE International Conference on Signal and Image Processing Applications (ICSIPA), September 12–14, 2017, Kuching, Malaysia.
Siu K, Stuart DM, Mahmoud M, Moshovos A (2018): Memory requirements for convolutional neural network hardware accelerators. Presented at the IEEE International Symposium on Workload Characterization (IISWC), September 30–October 2, 2018, Raleigh, North Carolina.
The ICD-11 developmental field study of reliability of diagnoses of high-burden mental disorders: Results among adult patients in mental health settings of 13 countries.
Poulin B, Eisner R, Szafron D, Lu P, Greiner R, Wishart DS, et al. (2006): Visual explanation of evidence with additive classifiers. Presented at the 21st National Conference on Artificial Intelligence, July 16–20, 2006, Boston, Massachusetts.
00345-1&id=gr1.jpg){kind=link}
00345-1&id=gr2.jpg){kind=link}
00345-1&id=gr3.jpg){kind=link}
00345-1&id=gr4.jpg){kind=link}
00345-1&id=gr5.jpg){kind=link}