Abstract
Background
Individuals with autism spectrum disorder (ASD) exhibit frequent behavioral deficits
in facial emotion recognition (FER). It remains unknown whether these deficits arise
because facial emotion information is not encoded in their neural signal or because
it is encodes but fails to translate to FER behavior (deployment). This distinction
has functional implications, including constraining when differences in social information
processing occur in ASD, and guiding interventions (i.e., developing prosthetic FER
vs. reinforcing existing skills).
Methods
We utilized a discriminative and contemporary machine learning approach—deep convolutional
neural networks—to classify facial emotions viewed by individuals with and without
ASD (N = 88) from concurrently recorded electroencephalography signals.
Results
The convolutional neural network classified facial emotions with high accuracy for
both ASD and non-ASD groups, even though individuals with ASD performed more poorly
on the concurrent FER task. In fact, convolutional neural network accuracy was greater
in the ASD group and was not related to behavioral performance. This pattern of results
replicated across three independent participant samples. Moreover, feature importance
analyses suggested that a late temporal window of neural activity (1000–1500 ms) may
be uniquely important in facial emotion classification for individuals with ASD.
Conclusions
Our results reveal for the first time that facial emotion information is encoded in
the neural signal of individuals with (and without) ASD. Thus, observed difficulties
in behavioral FER associated with ASD likely arise from difficulties in decoding or
deployment of facial emotion information within the neural signal. Interventions should
focus on capitalizing on this intact encoding rather than promoting compensation or
FER prostheses.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Biological Psychiatry: Cognitive Neuroscience and NeuroimagingAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies.Neuropsychol Rev. 2010; 20: 290-322
- Are emotion recognition abilities related to everyday social functioning in ASD? A meta-analysis.Res Autism Spec Disord. 2016; 32: 24-42
- Emotion in the Human Face: Guidelines for Research and an Integration of Findings. Volume 11 of the Pergamon General Psychology Series.Elsevier, Amsterdam, The Netherlands2013
- Impairments in facial affect recognition associated with autism spectrum disorders: A meta-analysis.Dev Psychopathol. 2014; 26: 933-945
- Recognition of emotions in autism: A formal meta-analysis.J Autism Dev Disord. 2013; 43: 1517-1526
- Neural bases of atypical emotional face processing in autism: A meta-analysis of fMRI studies.World J Biol Psychiatry. 2015; 16: 291-300
- Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography.Neurosci Biobehav Rev. 2017; 80: 488-515
- Atypicality of the N170 event-related potential in autism spectrum disorder: A meta-analysis.Biol Psychiatry Cogn Neurosci Neuroimaging. 2018; 3: 657-666
- Deep learning for healthcare applications based on physiological signals: A review.Comput Methods Programs Biomed. 2018; 161: 1-13
- Deep learning.Nature. 2015; 521: 436-444
- Classification of Alzheimer’s disease structural MRI data by deep learning convolutional neural networks.ArXiv. 2016;
- Building better biomarkers: Brain models in translational neuroimaging.Nat Neurosci. 2017; 20: 365-377
- Diagnosis of autism through EEG processed by advanced computational algorithms: A pilot study.Comput Methods Programs Biomed. 2017; 142: 73-79
- ASD-DiagNet: A hybrid learning approach for detection of autism spectrum disorder using fMRI data.Front Neuroinform. 2019; 13: 70
- Auditory repetition suppression alterations in relation to cognitive functioning in fragile X syndrome: A combined EEG and machine learning approach.J Neurodev Disord. 2018; 10: 4
- Machine learning for real-time single-trial EEG-analysis: From brain-computer interfacing to mental state monitoring.J Neurosci Methods. 2008; 167: 82-90
- Deep learning with convolutional neural networks for EEG decoding and visualization.Hum Brain Mapp. 2017; 38: 5391-5420
- Applying deep learning to single-trial EEG data provides evidence for complementary theories on action control.Commun Biol. 2020; 3: 112
- Scaling human-object interaction recognition through zero-shot learning: Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).1. 2018: 1568-1576
- Multivariate and propensity score matching software with automated balance optimization: The matching package for R.J Stat Softw. 2011; 42: 1-52
- Mapping the development of facial expression recognition.Dev Sci. 2015; 18: 926-939
- The perception of facial emotion in typical and atypical development.in: LoBue V. Pérez-Edgar K. Buss K. Handbook of Emotional Development. Springer, Cham2019: 105-138
- Autism Diagnostic Observation Schedule (ADOS-2).Western Psychological Services, Los Angeles, CA2012
- Kaufman Brief Intelligence Test—Second edition (KBIT-2).American Guidance Service, Inc, Circle Pines, MN2004
- Psychometric characteristics of the DANVA-2 in high-functioning children with ASD.J Autism Dev Disord. 2019; 49: 4147-4158
- Manual for the Receptive Tests of the Diagnostic Analysis of Nonverbal Accuracy 2 (DANVA2).Department of Psychology, Emory University, Atlanta, GA2008
- Guidelines and best practices for electrophysiological data collection, analysis and reporting in autism.J Autism Dev Disord. 2015; 45: 425-443
- EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.J Neurosci Methods. 2004; 134: 9-21
- The PREP pipeline: Standardized preprocessing for large-scale EEG analysis.Front Neuroinform. 2015; 9: 16
- ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features.Psychophysiology. 2011; 48: 229-240
- Independent component analysis: Algorithms and applications.Neural Netw. 2000; 13: 411-430
- Enhanced error decoding from error-related potentials using convolutional neural networks.Annu Int Conf IEEE Eng Med Biol Soc. 2018; 2018: 360-363
- The (un)reliability of saliency methods.in: Samek W. Montavon G. Vedaldi A. Hansen L.K. Müller K.R. Explainable AI: Interpreting, Explaining, and Visualizing Deep Learning. Springer, Cham, Germany2019: 267-280
- Adam: A method for stochastic optimization.arXiv. 2014;
- Learning how to explain neural networks: PatternNet and PatternAttribution.arXiv. 2017;
- How to iNNvestigate neural network’ s predictions!.(Available at:)https://openreview.net/pdf?id=rJgxCt6Vn7(Accessed December 1, 2018)Date: 2018
- Methods for interpreting and understanding deep neural networks.Digit Signal Process. 2018; 73: 1-15
- The development of emotion recognition in individuals with autism.Child Dev. 2009; 80: 1434-1447
- Autism and the extraction of emotion from briefly presented facial expressions: Stumbling at the first step of empathy.Emotion. 2008; 8: 803-809
- Orienting to social stimuli differentiates social cognitive impairment in autism and schizophrenia.Neuropsychologia. 2007; 45: 2580-2588
- Impaired causal awareness and associated cortical-basal ganglia structural changes in youth psychiatric disorders.Neuroimage Clin. 2016; 12: 285-292
- Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: A randomized clinical trial.JAMA Pediatr. 2019; 173: 446-454
- Developing connections for affective regulation: Age-related changes in emotional brain connectivity.J Exp Child Psychol. 2011; 108: 607-620
- Using computerized games to teach face recognition skills to children with autism spectrum disorder: The Let’s Face It! program.J Child Psychol Psychiatry. 2010; 51: 944-952
- Self-regulation of brain oscillations as a treatment for aberrant brain connections in children with autism.Med Hypotheses. 2012; 79: 790-798
- A fine-grained analysis of facial expression processing in high-functioning adults with autism.Neuropsychologia. 2007; 45: 685-695
- Emotion recognition of static and dynamic faces in autism spectrum disorder.Cogn Emot. 2014; 28: 1110-1118
- Deficits in facial, body movement and vocal emotional processing in autism spectrum disorders.Psychol Med. 2010; 40: 1919-1929
- Another advanced test of theory of mind: Evidence from very high functioning adults with autism or Asperger syndrome.J Child Psychol Psychiatry. 1997; 38: 813-822
- Scan path differences and similarities during emotion perception in those with and without autism spectrum disorders.J Autism Dev Disord. 2008; 38: 1371-1381
- Autism, Asperger syndrome and brain mechanisms for the attribution of mental states to animated shapes.Brain. 2002; 125: 1839-1849
- Adaptive deconvolutional networks for mid and high level feature learning.in: Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 2018–2025. 2011
Article Info
Publication History
Published online: April 12, 2021
Accepted:
March 31,
2021
Received in revised form:
March 31,
2021
Received:
March 3,
2021
Identification
Copyright
Published by Elsevier Inc on behalf of Society of Biological Psychiatry.
ScienceDirect
Access this article on ScienceDirectLinked Article
- Novel Insights Into Facial Emotion Encoding in Autism Spectrum Disorder Through Deep LearningBiological Psychiatry: Cognitive Neuroscience and NeuroimagingVol. 7Issue 7
- PreviewTo what extent is social information processing in autism spectrum disorder (ASD) similar to typically developing individuals, and how is it different? More specifically, is facial emotion information, which is central to social interactions, encoded differently in ASD? In the current issue of Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, Mayor Torres et al. (1) used a deep convolutional neural network (CNN) to classify neural signals acquired during facial emotion processing in adolescents with ASD and typical control subjects.
- Full-Text
- Preview
