Facial Emotions Are Accurately Encoded in the Neural Signal of Those With Autism Spectrum Disorder: A Deep Learning Approach

Published:April 12, 2021DOI:



      Individuals with autism spectrum disorder (ASD) exhibit frequent behavioral deficits in facial emotion recognition (FER). It remains unknown whether these deficits arise because facial emotion information is not encoded in their neural signal or because it is encodes but fails to translate to FER behavior (deployment). This distinction has functional implications, including constraining when differences in social information processing occur in ASD, and guiding interventions (i.e., developing prosthetic FER vs. reinforcing existing skills).


      We utilized a discriminative and contemporary machine learning approach—deep convolutional neural networks—to classify facial emotions viewed by individuals with and without ASD (N = 88) from concurrently recorded electroencephalography signals.


      The convolutional neural network classified facial emotions with high accuracy for both ASD and non-ASD groups, even though individuals with ASD performed more poorly on the concurrent FER task. In fact, convolutional neural network accuracy was greater in the ASD group and was not related to behavioral performance. This pattern of results replicated across three independent participant samples. Moreover, feature importance analyses suggested that a late temporal window of neural activity (1000–1500 ms) may be uniquely important in facial emotion classification for individuals with ASD.


      Our results reveal for the first time that facial emotion information is encoded in the neural signal of individuals with (and without) ASD. Thus, observed difficulties in behavioral FER associated with ASD likely arise from difficulties in decoding or deployment of facial emotion information within the neural signal. Interventions should focus on capitalizing on this intact encoding rather than promoting compensation or FER prostheses.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


        • Harms M.B.
        • Martin A.
        • Wallace G.L.
        Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies.
        Neuropsychol Rev. 2010; 20: 290-322
        • Trevisan D.A.
        • Birmingham E.
        Are emotion recognition abilities related to everyday social functioning in ASD? A meta-analysis.
        Res Autism Spec Disord. 2016; 32: 24-42
        • Ekman P.
        • Friesen W.V.
        • Ellsworth P.
        Emotion in the Human Face: Guidelines for Research and an Integration of Findings. Volume 11 of the Pergamon General Psychology Series.
        Elsevier, Amsterdam, The Netherlands2013
        • Lozier L.M.
        • Vanmeter J.W.
        • Marsh A.A.
        Impairments in facial affect recognition associated with autism spectrum disorders: A meta-analysis.
        Dev Psychopathol. 2014; 26: 933-945
        • Uljarevic M.
        • Hamilton A.
        Recognition of emotions in autism: A formal meta-analysis.
        J Autism Dev Disord. 2013; 43: 1517-1526
        • Aoki Y.
        • Cortese S.
        • Tansella M.
        Neural bases of atypical emotional face processing in autism: A meta-analysis of fMRI studies.
        World J Biol Psychiatry. 2015; 16: 291-300
        • Black M.H.
        • Chen N.T.M.
        • Iyer K.K.
        • Lipp O.V.
        • Bölte S.
        • Falkmer M.
        • et al.
        Mechanisms of facial emotion recognition in autism spectrum disorders: Insights from eye tracking and electroencephalography.
        Neurosci Biobehav Rev. 2017; 80: 488-515
        • Kang E.
        • Keifer C.M.
        • Levy E.J.
        • Foss-Feig J.H.
        • McPartland J.C.
        • Lerner M.D.
        Atypicality of the N170 event-related potential in autism spectrum disorder: A meta-analysis.
        Biol Psychiatry Cogn Neurosci Neuroimaging. 2018; 3: 657-666
        • Faust O.
        • Hagiwara Y.
        • Hong T.J.
        • Lih O.S.
        • Acharya U.R.
        Deep learning for healthcare applications based on physiological signals: A review.
        Comput Methods Programs Biomed. 2018; 161: 1-13
        • LeCun Y.
        • Bengio Y.
        • Hinton G.
        Deep learning.
        Nature. 2015; 521: 436-444
        • Sarraf S.
        • Tofighi G.
        Classification of Alzheimer’s disease structural MRI data by deep learning convolutional neural networks.
        ArXiv. 2016;
        • Woo C.W.
        • Chang L.J.
        • Lindquist M.A.
        • Wager T.D.
        Building better biomarkers: Brain models in translational neuroimaging.
        Nat Neurosci. 2017; 20: 365-377
        • Grossi E.
        • Olivieri C.
        • Buscema M.
        Diagnosis of autism through EEG processed by advanced computational algorithms: A pilot study.
        Comput Methods Programs Biomed. 2017; 142: 73-79
        • Eslami T.
        • Mirjalili V.
        • Fong A.
        • Laird A.R.
        • Saeed F.
        ASD-DiagNet: A hybrid learning approach for detection of autism spectrum disorder using fMRI data.
        Front Neuroinform. 2019; 13: 70
        • Knoth I.S.
        • Lajnef T.
        • Rigoulot S.
        • Lacourse K.
        • Vannasing P.
        • Michaud J.L.
        • et al.
        Auditory repetition suppression alterations in relation to cognitive functioning in fragile X syndrome: A combined EEG and machine learning approach.
        J Neurodev Disord. 2018; 10: 4
        • Müller K.R.
        • Tangermann M.
        • Dornhege G.
        • Krauledat M.
        • Curio G.
        • Blankertz B.
        Machine learning for real-time single-trial EEG-analysis: From brain-computer interfacing to mental state monitoring.
        J Neurosci Methods. 2008; 167: 82-90
        • Schirrmeister R.T.
        • Springenberg J.T.
        • Fiederer L.D.J.
        • Glasstetter M.
        • Eggensperger K.
        • Tangermann M.
        • et al.
        Deep learning with convolutional neural networks for EEG decoding and visualization.
        Hum Brain Mapp. 2017; 38: 5391-5420
        • Vahid A.
        • Mückschel M.
        • Stober S.
        • Stock A.K.
        • Beste C.
        Applying deep learning to single-trial EEG data provides evidence for complementary theories on action control.
        Commun Biol. 2020; 3: 112
        • Shen L.
        • Yeung S.
        • Hoffman J.
        • Mori G.
        • Fei-Fei L.
        Scaling human-object interaction recognition through zero-shot learning: Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).
        1. 2018: 1568-1576
        • Sekhon J.S.
        Multivariate and propensity score matching software with automated balance optimization: The matching package for R.
        J Stat Softw. 2011; 42: 1-52
        • Rodger H.
        • Vizioli L.
        • Ouyang X.
        • Caldara R.
        Mapping the development of facial expression recognition.
        Dev Sci. 2015; 18: 926-939
        • Bayet L.
        • Nelson C.A.
        The perception of facial emotion in typical and atypical development.
        in: LoBue V. Pérez-Edgar K. Buss K. Handbook of Emotional Development. Springer, Cham2019: 105-138
        • Lord C.
        • Rutter M.
        • DiLavore P.C.
        • Risi S.
        • Gotham K.
        • Bishop S.
        • et al.
        Autism Diagnostic Observation Schedule (ADOS-2).
        Western Psychological Services, Los Angeles, CA2012
        • Kaufman A.S.
        Kaufman Brief Intelligence Test—Second edition (KBIT-2).
        American Guidance Service, Inc, Circle Pines, MN2004
        • Booth A.J.
        • Rodgers J.D.
        • Volker M.A.
        • Lopata C.
        • Thomeer M.L.
        Psychometric characteristics of the DANVA-2 in high-functioning children with ASD.
        J Autism Dev Disord. 2019; 49: 4147-4158
        • Nowicki S.
        • Duke M.P.
        Manual for the Receptive Tests of the Diagnostic Analysis of Nonverbal Accuracy 2 (DANVA2).
        Department of Psychology, Emory University, Atlanta, GA2008
        • Webb S.J.
        • Bernier R.
        • Henderson H.A.
        • Johnson M.H.
        • Jones E.J.
        • Lerner M.D.
        • et al.
        Guidelines and best practices for electrophysiological data collection, analysis and reporting in autism.
        J Autism Dev Disord. 2015; 45: 425-443
        • Delorme A.
        • Makeig S.
        EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.
        J Neurosci Methods. 2004; 134: 9-21
        • Bigdely-Shamlo N.
        • Mullen T.
        • Kothe C.
        • Su K.M.
        • Robbins K.A.
        The PREP pipeline: Standardized preprocessing for large-scale EEG analysis.
        Front Neuroinform. 2015; 9: 16
        • Mognon A.
        • Jovicich J.
        • Bruzzone L.
        • Buiatti M.
        ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features.
        Psychophysiology. 2011; 48: 229-240
        • Hyvärinen A.
        • Oja E.
        Independent component analysis: Algorithms and applications.
        Neural Netw. 2000; 13: 411-430
        • Mayor Torres J.M.
        • Clarkson T.
        • Stepanov E.A.
        • Luhmann C.C.
        • Lerner M.D.
        • Riccardi G.
        Enhanced error decoding from error-related potentials using convolutional neural networks.
        Annu Int Conf IEEE Eng Med Biol Soc. 2018; 2018: 360-363
        • Kindermans P.-J.
        • Hooker S.
        • Adebayo J.
        • Alber M.
        • Schütt K.T.
        • Dähne S.
        • et al.
        The (un)reliability of saliency methods.
        in: Samek W. Montavon G. Vedaldi A. Hansen L.K. Müller K.R. Explainable AI: Interpreting, Explaining, and Visualizing Deep Learning. Springer, Cham, Germany2019: 267-280
        • Kingma D.P.
        • Lei Ba J.
        Adam: A method for stochastic optimization.
        arXiv. 2014;
        • Kindermans P.-J.
        • Schütt K.T.
        • Alber M.
        • Müller K.-R.
        • Erhan D.
        • Kim B.
        • Dähne S.
        Learning how to explain neural networks: PatternNet and PatternAttribution.
        arXiv. 2017;
        • Alber M.
        • Lapuschkin S.
        • Seegerer P.
        • Hägele M.
        • Schütt K.T.
        • Montavon G.
        • et al.
        How to iNNvestigate neural network’ s predictions!.
        (Available at:) (Accessed December 1, 2018)
        • Montavon G.
        • Samek W.
        • Müller K.R.
        Methods for interpreting and understanding deep neural networks.
        Digit Signal Process. 2018; 73: 1-15
        • Rump K.M.
        • Giovannelli J.L.
        • Minshew N.J.
        • Strauss M.S.
        The development of emotion recognition in individuals with autism.
        Child Dev. 2009; 80: 1434-1447
        • Clark T.F.
        • Winkielman P.
        • McIntosh D.N.
        Autism and the extraction of emotion from briefly presented facial expressions: Stumbling at the first step of empathy.
        Emotion. 2008; 8: 803-809
        • Sasson N.
        • Tsuchiya N.
        • Hurley R.
        • Couture S.M.
        • Penn D.L.
        • Adolphs R.
        • Piven J.
        Orienting to social stimuli differentiates social cognitive impairment in autism and schizophrenia.
        Neuropsychologia. 2007; 45: 2580-2588
        • Griffiths K.R.
        • Lagopoulos J.
        • Hermens D.F.
        • Lee R.S.C.
        • Guastella A.J.
        • Hickie I.B.
        • Balleine B.W.
        Impaired causal awareness and associated cortical-basal ganglia structural changes in youth psychiatric disorders.
        Neuroimage Clin. 2016; 12: 285-292
        • Voss C.
        • Schwartz J.
        • Daniels J.
        • Kline A.
        • Haber N.
        • Washington P.
        • et al.
        Effect of wearable digital intervention for improving socialization in children with autism spectrum disorder: A randomized clinical trial.
        JAMA Pediatr. 2019; 173: 446-454
        • Perlman S.B.
        • Pelphrey K.A.
        Developing connections for affective regulation: Age-related changes in emotional brain connectivity.
        J Exp Child Psychol. 2011; 108: 607-620
        • Tanaka J.W.
        • Wolf J.M.
        • Klaiman C.
        • Koenig K.
        • Cockburn J.
        • Herlihy L.
        • et al.
        Using computerized games to teach face recognition skills to children with autism spectrum disorder: The Let’s Face It! program.
        J Child Psychol Psychiatry. 2010; 51: 944-952
        • Pineda J.A.
        • Juavinett A.
        • Datko M.
        Self-regulation of brain oscillations as a treatment for aberrant brain connections in children with autism.
        Med Hypotheses. 2012; 79: 790-798
        • Humphreys K.
        • Minshew N.
        • Leonard G.L.
        • Behrmann M.
        A fine-grained analysis of facial expression processing in high-functioning adults with autism.
        Neuropsychologia. 2007; 45: 685-695
        • Enticott P.G.
        • Kennedy H.A.
        • Johnston P.J.
        • Rinehart N.J.
        • Tonge B.J.
        • Taffe J.R.
        • Fitzgerald P.B.
        Emotion recognition of static and dynamic faces in autism spectrum disorder.
        Cogn Emot. 2014; 28: 1110-1118
        • Philip R.C.M.
        • Whalley H.C.
        • Stanfield A.C.
        • Sprengelmeyer R.
        • Santos I.M.
        • Young A.W.
        • et al.
        Deficits in facial, body movement and vocal emotional processing in autism spectrum disorders.
        Psychol Med. 2010; 40: 1919-1929
        • Baron-Cohen S.
        • Jolliffe T.
        • Mortimore C.
        • Robertson M.
        Another advanced test of theory of mind: Evidence from very high functioning adults with autism or Asperger syndrome.
        J Child Psychol Psychiatry. 1997; 38: 813-822
        • Rutherford M.D.
        • Towns A.M.
        Scan path differences and similarities during emotion perception in those with and without autism spectrum disorders.
        J Autism Dev Disord. 2008; 38: 1371-1381
        • Castelli F.
        • Frith C.
        • Happé F.
        • Frith U.
        Autism, Asperger syndrome and brain mechanisms for the attribution of mental states to animated shapes.
        Brain. 2002; 125: 1839-1849
        • Zeiler M.D.
        • Taylor G.W.
        • Fergus R.
        Adaptive deconvolutional networks for mid and high level feature learning.
        in: Proceedings of the 2011 International Conference on Computer Vision, Barcelona, Spain, 2018–2025. 2011

      Linked Article

      • Novel Insights Into Facial Emotion Encoding in Autism Spectrum Disorder Through Deep Learning
        Biological Psychiatry: Cognitive Neuroscience and NeuroimagingVol. 7Issue 7
        • Preview
          To what extent is social information processing in autism spectrum disorder (ASD) similar to typically developing individuals, and how is it different? More specifically, is facial emotion information, which is central to social interactions, encoded differently in ASD? In the current issue of Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, Mayor Torres et al. (1) used a deep convolutional neural network (CNN) to classify neural signals acquired during facial emotion processing in adolescents with ASD and typical control subjects.
        • Full-Text
        • PDF