Advertisement

Reliability and validity of TMS-EEG biomarkers

  • Sara Parmigiani
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Stanford University Medical Center, Stanford, CA, 94305, USA

    Veterans Affairs Palo Alto Healthcare System, and the Sierra Pacific Mental Illness, Research, Education, and Clinical Center (MIRECC), Palo Alto, CA, 94394, USA
    Search for articles by this author
  • Jessica M. Ross
    Affiliations
    Veterans Affairs Palo Alto Healthcare System, and the Sierra Pacific Mental Illness, Research, Education, and Clinical Center (MIRECC), Palo Alto, CA, 94394, USA

    Department of Psychiatry and Behavioral Sciences, Stanford University Medical Center, Stanford, CA, 94305, USA
    Search for articles by this author
  • Christopher Cline
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Stanford University Medical Center, Stanford, CA, 94305, USA

    Veterans Affairs Palo Alto Healthcare System, and the Sierra Pacific Mental Illness, Research, Education, and Clinical Center (MIRECC), Palo Alto, CA, 94394, USA
    Search for articles by this author
  • Christopher Minasi
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Stanford University Medical Center, Stanford, CA, 94305, USA

    Veterans Affairs Palo Alto Healthcare System, and the Sierra Pacific Mental Illness, Research, Education, and Clinical Center (MIRECC), Palo Alto, CA, 94394, USA
    Search for articles by this author
  • Juha Gogulski
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Stanford University Medical Center, Stanford, CA, 94305, USA

    Veterans Affairs Palo Alto Healthcare System, and the Sierra Pacific Mental Illness, Research, Education, and Clinical Center (MIRECC), Palo Alto, CA, 94394, USA

    Department of Clinical Neurophysiology, HUS Diagnostic Center, Clinical Neurosciences, Helsinki University Hospital and University of Helsinki, Helsinki, FI-00029 HUS, Finland
    Search for articles by this author
  • Corey J. Keller
    Correspondence
    Correspondence:Corey Keller, MD, PhD, Stanford University, Department of Psychiatry and Behavioral Sciences, 401 Quarry Road, Stanford, CA 94305-5797 Phone: +1 8025786292
    Affiliations
    Department of Psychiatry and Behavioral Sciences, Stanford University Medical Center, Stanford, CA, 94305, USA

    Veterans Affairs Palo Alto Healthcare System, and the Sierra Pacific Mental Illness, Research, Education, and Clinical Center (MIRECC), Palo Alto, CA, 94394, USA
    Search for articles by this author
Published:December 17, 2022DOI:https://doi.org/10.1016/j.bpsc.2022.12.005

      Abstract

      Noninvasive brain stimulation and neuroimaging have revolutionized human neuroscience, with a multitude of applications including diagnostic subtyping, treatment optimization, and relapse prediction. It is therefore particularly relevant to identify robust and clinically valuable brain biomarkers linking symptoms to their underlying neural mechanisms. Brain biomarkers must be reproducible (i.e., have internal reliability) across similar experiments within a laboratory and be generalizable (i.e., have external reliability) across experimental setups, laboratories, brain regions, and disease states. However, reliability (internal and external) is not alone sufficient; biomarkers also must have validity. Validity describes closeness to a true measure of the underlying neural signal or disease state. We propose that these metrics, reliability and validity, should be evaluated and optimized before any biomarker is used to inform treatment decisions. Here, we discuss these metrics with respect to causal brain connectivity biomarkers from coupling transcranial magnetic stimulation (TMS) with electroencephalography (EEG). We discuss controversies around TMS-EEG stemming from the multiple large off-target components (noise) and relatively weak genuine brain responses (signal), as is unfortunately often the case in noninvasive human neuroscience. We review the current state of TMS-EEG recordings, which consist of a mix of reliable noise and unreliable signal. We describe methods for evaluating TMS-EEG biomarkers, including how to assess internal and external reliability across facilities, cognitive states, brain networks, and disorders, and how to validate these biomarkers using invasive neural recordings or treatment response. We provide recommendations to increase reliability and validity, discuss lessons learned, and suggest future directions for the field.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      References

        • Califf R.M.
        Biomarker definitions and their applications.
        Exp Biol Med Maywood NJ. 2018; 243: 213-221
      1. FDA-NIH Biomarker Working Group (2016): BEST (Biomarkers, EndpointS, and Other Tools) Resource. Silver Spring (MD): Food and Drug Administration (US). Retrieved July 13, 2022, from http://www.ncbi.nlm.nih.gov/books/NBK326791/

        • Lioumis P.
        • Rosanova M.
        The role of neuronavigation in TMS-EEG studies: Current applications and future perspectives.
        J Neurosci Methods. 2022; 380: 109677
        • Rogasch N.C.
        • Biabani M.
        • Mutanen T.P.
        Designing and comparing cleaning pipelines for TMS-EEG data: A theoretical overview and practical example.
        J Neurosci Methods. 2022; 371: 109494
        • Bertazzoli G.
        • Esposito R.
        • Mutanen T.P.
        • Ferrari C.
        • Ilmoniemi R.J.
        • Miniussi C.
        • Bortoletto M.
        The impact of artifact removal approaches on TMS-EEG signal.
        NeuroImage. 2021; 239: 118272
        • Conde V.
        • Tomasevic L.
        • Akopian I.
        • Stanek K.
        • Saturnino G.B.
        • Thielscher A.
        • et al.
        The non-transcranial TMS-evoked potential is an inherent source of ambiguity in TMS-EEG studies.
        NeuroImage. 2019; 185: 300-312
        • Salmond S.S.
        Evaluating the reliability and validity of measurement instruments.
        Orthop Nurs. 2008; 27: 28-30
      2. Kerlinger FN (1966): Foundations of Behavioral Research. Holt, Rinehart and Winston: New York.

        • Rothwell J.C.
        • Thompson P.D.
        • Day B.L.
        • Boyd S.
        • Marsden C.D.
        Stimulation of the human motor cortex through the scalp.
        Exp Physiol. 1991; 76: 159-200
        • Kobayashi M.
        • Pascual-Leone A.
        Transcranial magnetic stimulation in neurology.
        Lancet Neurol. 2003; 2: 145-156
        • Rossini P.M.
        • Burke D.
        • Chen R.
        • Cohen L.G.
        • Daskalakis Z.
        • Di Iorio R.
        • et al.
        Non-invasive electrical and magnetic stimulation of the brain, spinal cord, roots and peripheral nerves: Basic principles and procedures for routine clinical and research application. An updated report from an I.F.C.N. Committee.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2015; 126: 1071-1107
        • Thielscher A.
        • Wichmann F.A.
        Determining the cortical target of transcranial magnetic stimulation.
        NeuroImage. 2009; 47: 1319-1330
        • Parmigiani S.
        • Mikulan E.
        • Russo S.
        • Sarasso S.
        • Zauli F.M.
        • Rubino A.
        • et al.
        Simultaneous stereo-EEG and high-density scalp EEG recordings to study the effects of intracerebral stimulation parameters.
        Brain Stimulat. 2022; 15: 664-675
      3. Cattaneo L (2017): Transcranial Magnetic Stimulation. In: Rogers LJ, Vallortigara G, editors. Lateralized Brain Functions: Methods in Human and Non-Human Species. New York, NY: Springer, pp 369–406.

        • Huang Y.-Z.
        • Edwards M.J.
        • Rounis E.
        • Bhatia K.P.
        • Rothwell J.C.
        Theta burst stimulation of the human motor cortex.
        Neuron. 2005; 45: 201-206
        • Kujirai T.
        • Caramia M.D.
        • Rothwell J.C.
        • Day B.L.
        • Thompson P.D.
        • Ferbert A.
        • et al.
        Corticocortical inhibition in human motor cortex.
        J Physiol. 1993; 471: 501-519
        • Rossi S.
        • Pasqualetti P.
        • Tecchio F.
        • Sabato A.
        • Rossini P.M.
        Modulation of corticospinal output to human hand muscles following deprivation of sensory feedback.
        NeuroImage. 1998; 8: 163-175
        • Proessl F.
        • Beckner M.E.
        • Sinnott A.M.
        • Eagle S.R.
        • LaGoy A.D.
        • Conkright W.R.
        • et al.
        Reliability of corticospinal excitability estimates for the vastus lateralis: Practical considerations for lower limb TMS task selection.
        Brain Res. 2021; 1761: 147395
        • Welch J.F.
        • Argento P.J.
        • Mitchell G.S.
        • Fox E.J.
        Reliability of diaphragmatic motor-evoked potentials induced by transcranial magnetic stimulation.
        J Appl Physiol Bethesda Md. 2020; 1985: 1393-1404
      4. Roth Y, Pell GS, Zangen A (2010): Motor evoked potential latency, motor threshold and electric field measurements as indices of transcranial magnetic stimulation depth. Clin Neurophysiol Off J Int Fed Clin Neurophysiol 121: 255–258; author reply 258–259.

        • Awiszus F.
        On relative frequency estimation of transcranial magnetic stimulation motor threshold.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2012; 123: 2319-2320
      5. Rossini PM, Rossi S, Pasqualetti P, Tecchio F (1999): Corticospinal excitability modulation to hand muscles during movement imagery. Cereb Cortex N Y N 1991 9: 161–167.

      6. Ah Sen CB, Fassett HJ, El-Sayes J, Turco CV, Hameer MM, Nelson AJ (2017): Active and resting motor threshold are efficiently obtained with adaptive threshold hunting. PloS One 12: e0186007.

        • Groppa S.
        • Oliviero A.
        • Eisen A.
        • Quartarone A.
        • Cohen L.G.
        • Mall V.
        • et al.
        A practical guide to diagnostic transcranial magnetic stimulation: report of an IFCN committee.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2012; 123: 858-882
      7. Bigoni C, Cadic-Melchior A, Vassiliadis P, Morishita T, Hummel FC (2022): An automatized method to determine latencies of motor-evoked potentials under physiological and pathophysiological conditions. J Neural Eng 19. https://doi.org/10.1088/1741-2552/ac636c

        • Nagle K.J.
        • Emerson R.G.
        • Adams D.C.
        • Heyer E.J.
        • Roye D.P.
        • Schwab F.J.
        • et al.
        Intraoperative monitoring of motor evoked potentials: a review of 116 cases.
        Neurology. 1996; 47: 999-1004
        • Prabhu G.
        • Voss M.
        • Brochier T.
        • Cattaneo L.
        • Haggard P.
        • Lemon R.
        Excitability of human motor cortex inputs prior to grasp.
        J Physiol. 2007; 581: 189-201
        • Bologna M.
        • Suppa A.
        • Conte A.
        • Latorre A.
        • Rothwell J.C.
        • Berardelli A.
        Are studies of motor cortex plasticity relevant in human patients with Parkinson’s disease?.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2016; 127: 50-59
        • Tolmacheva A.
        • Savolainen S.
        • Kirveskari E.
        • Lioumis P.
        • Kuusela L.
        • Brandstack N.
        • et al.
        Long-Term Paired Associative Stimulation Enhances Motor Output of the Tetraplegic Hand.
        J Neurotrauma. 2017; 34: 2668-2674
        • Casarotto S.
        • Romero Lauro L.J.
        • Bellina V.
        • Casali A.G.
        • Rosanova M.
        • Pigorini A.
        • et al.
        EEG responses to TMS are sensitive to changes in the perturbation parameters and repeatable over time.
        PloS One. 2010; 5: e10281
        • Lioumis P.
        • Kicić D.
        • Savolainen P.
        • Mäkelä J.P.
        • Kähkönen S.
        Reproducibility of TMS-Evoked EEG responses.
        Hum Brain Mapp. 2009; 30: 1387-1396
        • Kerwin L.J.
        • Keller C.J.
        • Wu W.
        • Narayan M.
        • Etkin A.
        Test-retest reliability of transcranial magnetic stimulation EEG evoked potentials.
        Brain Stimulat. 2018; 11: 536-544
        • Ilmoniemi R.J.
        • Virtanen J.
        • Ruohonen J.
        • Karhu J.
        • Aronen H.J.
        • Näätänen R.
        • Katila T.
        Neuronal responses to magnetic stimulation reveal cortical reactivity and connectivity.
        Neuroreport. 1997; 8: 3537-3540
      8. Vv N, D K, S K, Rj I (2003): Modulation of electroencephalographic responses to transcranial magnetic stimulation: evidence for changes in cortical excitability related to movement. Eur J Neurosci 18. https://doi.org/10.1046/j.1460-9568.2003.02858.x

        • Komssi S.
        • Kähkönen S.
        • Ilmoniemi R.J.
        The effect of stimulus intensity on brain responses evoked by transcranial magnetic stimulation.
        Hum Brain Mapp. 2004; 21: 154-164
        • Massimini M.
        • Ferrarelli F.
        • Huber R.
        • Esser S.K.
        • Singh H.
        • Tononi G.
        Breakdown of cortical effective connectivity during sleep.
        Science. 2005; 309: 2228-2232
        • Massimini M.
        • Ferrarelli F.
        • Esser S.K.
        • Riedner B.A.
        • Huber R.
        • Murphy M.
        • et al.
        Triggering sleep slow waves by transcranial magnetic stimulation.
        Proc Natl Acad Sci U S A. 2007; 104: 8496-8501
        • Zazio A.
        • Bortoletto M.
        • Ruzzoli M.
        • Miniussi C.
        • Veniero D.
        Perceptual and Physiological Consequences of Dark Adaptation: A TMS-EEG Study.
        Brain Topogr. 2019; 32: 773-782
        • Bortoletto M.
        • Veniero D.
        • Thut G.
        • Miniussi C.
        The contribution of TMS-EEG coregistration in the exploration of the human cortical connectome.
        Neurosci Biobehav Rev. 2015; 49: 114-124
        • Voineskos D.
        • Blumberger D.M.
        • Zomorrodi R.
        • Rogasch N.C.
        • Farzan F.
        • Foussias G.
        • et al.
        Altered Transcranial Magnetic Stimulation-Electroencephalographic Markers of Inhibition and Excitation in the Dorsolateral Prefrontal Cortex in Major Depressive Disorder.
        Biol Psychiatry. 2019; 85: 477-486
        • Canali P.
        • Sarasso S.
        • Rosanova M.
        • Casarotto S.
        • Sferrazza-Papa G.
        • Gosseries O.
        • et al.
        Shared reduction of oscillatory natural frequencies in bipolar disorder, major depressive disorder and schizophrenia.
        J Affect Disord. 2015; 184: 111-115
        • Rosanova M.
        • Casali A.
        • Bellina V.
        • Resta F.
        • Mariotti M.
        • Massimini M.
        Natural frequencies of human corticothalamic circuits.
        J Neurosci Off J Soc Neurosci. 2009; 29: 7679-7685
        • Varone G.
        • Hussain Z.
        • Sheikh Z.
        • Howard A.
        • Boulila W.
        • Mahmud M.
        • et al.
        Real-Time Artifacts Reduction during TMS-EEG Co-Registration: A Comprehensive Review on Technologies and Procedures.
        Sensors. 2021; 21: 637
        • Sekiguchi H.
        • Takeuchi S.
        • Kadota H.
        • Kohno Y.
        • Nakajima Y.
        TMS-induced artifacts on EEG can be reduced by rearrangement of the electrode’s lead wire before recording.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2011; 122: 984-990
        • Huber R.
        • Esser S.K.
        • Ferrarelli F.
        • Massimini M.
        • Peterson M.J.
        • Tononi G.
        TMS-induced cortical potentiation during wakefulness locally increases slow wave activity during sleep.
        PloS One. 2007; 2: e276
        • Gosseries O.
        • Sarasso S.
        • Casarotto S.
        • Boly M.
        • Schnakers C.
        • Napolitani M.
        • et al.
        On the cerebral origin of EEG responses to TMS: insights from severe cortical lesions.
        Brain Stimulat. 2015; 8: 142-149
        • Mutanen T.
        • Mäki H.
        • Ilmoniemi R.J.
        The effect of stimulus parameters on TMS-EEG muscle artifacts.
        Brain Stimulat. 2013; 6: 371-376
        • Ross J.M.
        • Sarkar M.
        • Keller C.J.
        Experimental suppression of transcranial magnetic stimulation-electroencephalography sensory potentials.
        Hum Brain Mapp. 2022; https://doi.org/10.1002/hbm.25990
        • Ozdemir R.A.
        • Boucher P.
        • Fried P.J.
        • Momi D.
        • Jannati A.
        • Pascual-Leone A.
        • et al.
        Reproducibility of cortical response modulation induced by intermittent and continuous theta-burst stimulation of the human motor cortex.
        Brain Stimulat. 2021; 14: 949-964
        • Belardinelli P.
        • Biabani M.
        • Blumberger D.M.
        • Bortoletto M.
        • Casarotto S.
        • David O.
        • et al.
        Reproducibility in TMS-EEG studies: A call for data sharing, standard procedures and effective experimental control.
        Brain Stimulat. 2019; 12: 787-790
        • Rocchi L.
        • Di Santo A.
        • Brown K.
        • Ibáñez J.
        • Casula E.
        • Rawji V.
        • et al.
        Disentangling EEG responses to TMS due to cortical and peripheral activations.
        Brain Stimulat. 2021; 14: 4-18
        • Biabani M.
        • Fornito A.
        • Mutanen T.P.
        • Morrow J.
        • Rogasch N.C.
        Characterizing and minimizing the contribution of sensory inputs to TMS-evoked potentials.
        Brain Stimulat. 2019; 12: 1537-1552
        • Rogasch N.C.
        • Fitzgerald P.B.
        Assessing cortical network properties using TMS-EEG.
        Hum Brain Mapp. 2013; 34: 1652-1669
        • Tremblay S.
        • Rogasch N.C.
        • Premoli I.
        • Blumberger D.M.
        • Casarotto S.
        • Chen R.
        • et al.
        Clinical utility and prospective of TMS-EEG.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2019; 130: 802-844
        • Paus T.
        • Sipila P.K.
        • Strafella A.P.
        Synchronization of neuronal activity in the human primary motor cortex by transcranial magnetic stimulation: an EEG study.
        J Neurophysiol. 2001; 86: 1983-1990
      9. TMS-EEG co-registration: on TMS-induced artifact - PubMed (n.d.): Retrieved July 9, 2022, from https://pubmed-ncbi-nlm-nih-gov.laneproxy.stanford.edu/19535291/

      10. Recovering TMS-evoked EEG responses masked by muscle artifacts - PubMed (n.d.): Retrieved July 9, 2022, from https://pubmed-ncbi-nlm-nih-gov.laneproxy.stanford.edu/27291496/

      11. Automatic and robust noise suppression in EEG and MEG: The SOUND algorithm - PubMed (n.d.): Retrieved July 9, 2022, from https://pubmed-ncbi-nlm-nih-gov.laneproxy.stanford.edu/29061529/

        • Wu W.
        • Keller C.J.
        • Rogasch N.C.
        • Longwell P.
        • Shpigel E.
        • Rolle C.E.
        • Etkin A.
        ARTIST: A fully automated artifact rejection algorithm for single-pulse TMS-EEG data.
        Hum Brain Mapp. 2018; 39: 1607-1625
        • Rogasch N.C.
        • Sullivan C.
        • Thomson R.H.
        • Rose N.S.
        • Bailey N.W.
        • Fitzgerald P.B.
        • et al.
        Analysing concurrent transcranial magnetic stimulation and electroencephalographic data: A review and introduction to the open-source TESA software.
        NeuroImage. 2017; 147: 934-951
        • Russo S.
        • Sarasso S.
        • Puglisi G.E.
        • Dal Palù D.
        • Pigorini A.
        • Casarotto S.
        • et al.
        TAAC - TMS Adaptable Auditory Control: A universal tool to mask TMS clicks.
        J Neurosci Methods. 2022; 370: 109491
      12. A structured ICA-based process for removing auditory evoked potentials - PubMed (n.d.): Retrieved July 9, 2022, from https://pubmed-ncbi-nlm-nih-gov.laneproxy.stanford.edu/35082350/

        • Novembre G.
        • Pawar V.M.
        • Kilintari M.
        • Bufacchi R.J.
        • Guo Y.
        • Rothwell J.C.
        • Iannetti G.D.
        The effect of salient stimuli on neural oscillations, isometric force, and their coupling.
        NeuroImage. 2019; 198: 221-230
      13. Disentangling EEG responses to TMS due to cortical and peripheral activations - PubMed (n.d.): Retrieved July 9, 2022, from https://pubmed-ncbi-nlm-nih-gov.laneproxy.stanford.edu/33127580/

        • Veniero D.
        • Bortoletto M.
        • Miniussi C.
        TMS-EEG co-registration: on TMS-induced artifact.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2009; 120: 1392-1399
        • Mutanen T.P.
        • Metsomaa J.
        • Liljander S.
        • Ilmoniemi R.J.
        Automatic and robust noise suppression in EEG and MEG: The SOUND algorithm.
        NeuroImage. 2018; 166: 135-151
        • Wu W.
        • Keller C.J.
        • Rogasch N.C.
        • Longwell P.
        • Shpigel E.
        • Rolle C.E.
        • Etkin A.
        ARTIST: A fully automated artifact rejection algorithm for single-pulse TMS-EEG data.
        Hum Brain Mapp. 2018; 39: 1607-1625
        • Ross J.M.
        • Ozdemir R.A.
        • Lian S.J.
        • Fried P.J.
        • Schmitt E.M.
        • Inouye S.K.
        • et al.
        A structured ICA-based process for removing auditory evoked potentials.
        Sci Rep. 2022; 12: 1391
        • Meyberg S.
        • Werkle-Bergner M.
        • Sommer W.
        • Dimigen O.
        Microsaccade-related brain potentials signal the focus of visuospatial attention.
        NeuroImage. 2015; 104: 79-88
        • Yuval-Greenberg S.
        • Tomer O.
        • Keren A.S.
        • Nelken I.
        • Deouell L.Y.
        Transient induced gamma-band response in EEG as a manifestation of miniature saccades.
        Neuron. 2008; 58: 429-441
        • Herring J.D.
        • Thut G.
        • Jensen O.
        • Bergmann T.O.
        Attention Modulates TMS-Locked Alpha Oscillations in the Visual Cortex.
        J Neurosci Off J Soc Neurosci. 2015; 35: 14435-14447
        • Mouraux A.
        • Diukova A.
        • Lee M.C.
        • Wise R.G.
        • Iannetti G.D.
        A multisensory investigation of the functional significance of the “pain matrix.” NeuroImage. 2011; 54: 2237-2249
        • Rogasch N.C.
        • Thomson R.H.
        • Farzan F.
        • Fitzgibbon B.M.
        • Bailey N.W.
        • Hernandez-Pavon J.C.
        • et al.
        Removing artefacts from TMS-EEG recordings using independent component analysis: importance for assessing prefrontal and motor cortex network properties.
        NeuroImage. 2014; 101: 425-439
        • Mutanen T.P.
        • Biabani M.
        • Sarvas J.
        • Ilmoniemi R.J.
        • Rogasch N.C.
        Source-based artifact-rejection techniques available in TESA, an open-source TMS-EEG toolbox.
        Brain Stimulat. 2020; 13: 1349-1351
        • Atluri S.
        • Frehlich M.
        • Mei Y.
        • Garcia Dominguez L.
        • Rogasch N.C.
        • Wong W.
        • et al.
        TMSEEG: A MATLAB-Based Graphical User Interface for Processing Electrophysiological Signals during Transcranial Magnetic Stimulation.
        Front Neural Circuits. 2016; 10: 78
        • Wu W.
        • Keller C.J.
        • Rogasch N.C.
        • Longwell P.
        • Shpigel E.
        • Rolle C.E.
        • Etkin A.
        ARTIST: A fully automated artifact rejection algorithm for single-pulse TMS-EEG data.
        Hum Brain Mapp. 2018; 39: 1607-1625
        • Ilmoniemi R.J.
        • Kicić D.
        Methodology for combined TMS and EEG.
        Brain Topogr. 2010; 22: 233-248
        • Tchumatchenko T.
        • Reichenbach T.
        A cochlear-bone wave can yield a hearing sensation as well as otoacoustic emission.
        Nat Commun. 2014; 5: 4160
        • ter Braack EM, de Vos CC, van Putten MJAM
        Masking the Auditory Evoked Potential in TMS-EEG: A Comparison of Various Methods.
        Brain Topogr. 2015; 28: 520-528
        • Mutanen T.
        • Nieminen J.O.
        • Ilmoniemi R.J.
        TMS-evoked changes in brain-state dynamics quantified by using EEG data.
        Front Hum Neurosci. 2013; 7: 155
        • Mäki H.
        • Ilmoniemi R.J.
        The relationship between peripheral and early cortical activation induced by transcranial magnetic stimulation.
        Neurosci Lett. 2010; 478: 24-28
      14. Fecchio M, Pigorini A, Comanducci A, Sarasso S, Casarotto S, Premoli I, et al. (2017): The spectral features of EEG responses to transcranial magnetic stimulation of the primary motor cortex depend on the amplitude of the motor evoked potentials. PloS One 12: e0184910.

        • Biabani M.
        • Fornito A.
        • Coxon J.P.
        • Fulcher B.D.
        • Rogasch N.C.
        The correspondence between EMG and EEG measures of changes in cortical excitability following transcranial magnetic stimulation.
        J Physiol. 2021; 599: 2907-2932
        • Kähkönen S.
        • Komssi S.
        • Wilenius J.
        • Ilmoniemi R.J.
        Prefrontal transcranial magnetic stimulation produces intensity-dependent EEG responses in humans.
        NeuroImage. 2005; 24: 955-960
        • Kähkönen S.
        • Wilenius J.
        • Komssi S.
        • Ilmoniemi R.J.
        Distinct differences in cortical reactivity of motor and prefrontal cortices to magnetic stimulation.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2004; 115: 583-588
        • Lioumis P.
        • Kicić D.
        • Savolainen P.
        • Mäkelä J.P.
        • Kähkönen S.
        Reproducibility of TMS-Evoked EEG responses.
        Hum Brain Mapp. 2009; 30: 1387-1396
        • Rogasch N.C.
        • Zipser C.
        • Darmani G.
        • Mutanen T.P.
        • Biabani M.
        • Zrenner C.
        • et al.
        The effects of NMDA receptor blockade on TMS-evoked EEG potentials from prefrontal and parietal cortex.
        Sci Rep. 2020; 10: 3168
        • Belardinelli P.
        • König F.
        • Liang C.
        • Premoli I.
        • Desideri D.
        • Müller-Dahlhaus F.
        • et al.
        TMS-EEG signatures of glutamatergic neurotransmission in human cortex.
        Sci Rep. 2021; 11: 8159
        • Sun Y.
        • Farzan F.
        • Mulsant B.H.
        • Rajji T.K.
        • Fitzgerald P.B.
        • Barr M.S.
        • et al.
        Indicators for Remission of Suicidal Ideation Following Magnetic Seizure Therapy in Patients With Treatment-Resistant Depression.
        JAMA Psychiatry. 2016; 73: 337-345
        • Hui J.
        • Zomorrodi R.
        • Lioumis P.
        • Salavati B.
        • Rajji T.K.
        • Chen R.
        • et al.
        Pharmacological mechanisms of interhemispheric signal propagation: a TMS-EEG study.
        Neuropsychopharmacol Off Publ Am Coll Neuropsychopharmacol. 2020; 45: 932-939
        • Casula E.P.
        • Maiella M.
        • Pellicciari M.C.
        • Porrazzini F.
        • D’Acunto A.
        • Rocchi L.
        • Koch G.
        Novel TMS-EEG indexes to investigate interhemispheric dynamics in humans.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2020; 131: 70-77
      15. Casali AG, Gosseries O, Rosanova M, Boly M, Sarasso S, Casali KR, et al. (2013): A theoretically based index of consciousness independent of sensory processing and behavior. Sci Transl Med 5: 198ra105.

        • Powden C.J.
        • Hoch J.M.
        • Hoch M.C.
        Reliability and minimal detectable change of the weight-bearing lunge test: A systematic review.
        Man Ther. 2015; 20: 524-532
        • Mokkink L.B.
        • Terwee C.B.
        • Patrick D.L.
        • Alonso J.
        • Stratford P.W.
        • Knol D.L.
        • et al.
        The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study.
        Qual Life Res Int J Qual Life Asp Treat Care Rehabil. 2010; 19: 539-549
        • Ozdemir R.A.
        • Tadayon E.
        • Boucher P.
        • Momi D.
        • Karakhanyan K.A.
        • Fox M.D.
        • et al.
        Individualized perturbation of the human connectome reveals reproducible biomarkers of network dynamics relevant to cognition.
        Proc Natl Acad Sci U S A. 2020; 117: 8115-8125
      16. Tomasevic L, Takemi M, Siebner HR (2017): Synchronizing the transcranial magnetic pulse with electroencephalographic recordings effectively reduces inter-trial variability of the pulse artefact. PloS One 12: e0185154.

        • Thut G.
        • Ives J.R.
        • Kampmann F.
        • Pastor M.A.
        • Pascual-Leone A.
        A new device and protocol for combining TMS and online recordings of EEG and evoked potentials.
        J Neurosci Methods. 2005; 141: 207-217
      17. White WL (2010): Why I hate the index finger. Hand N Y N 5: 461–465.

        • Rogasch N.C.
        • Thomson R.H.
        • Daskalakis Z.J.
        • Fitzgerald P.B.
        Short-latency artifacts associated with concurrent TMS-EEG.
        Brain Stimulat. 2013; 6: 868-876
        • Julkunen P.
        • Pääkkönen A.
        • Hukkanen T.
        • Könönen M.
        • Tiihonen P.
        • Vanhatalo S.
        • Karhu J.
        Efficient reduction of stimulus artefact in TMS-EEG by epithelial short-circuiting by mini-punctures.
        Clin Neurophysiol Off J Int Fed Clin Neurophysiol. 2008; 119: 475-481
        • Mancuso M.
        • Sveva V.
        • Cruciani A.
        • Brown K.
        • Ibáñez J.
        • Rawji V.
        • et al.
        Transcranial Evoked Potentials Can Be Reliably Recorded with Active Electrodes.
        Brain Sci. 2021; 11: 145
        • Casarotto S.
        • Fecchio M.
        • Rosanova M.
        • Varone G.
        • D’Ambrosio S.
        • Sarasso S.
        • et al.
        The rt-TEP tool: real-time visualization of TMS-Evoked Potentials to maximize cortical activation and minimize artifacts.
        J Neurosci Methods. 2022; 370: 109486
        • Tervo A.E.
        • Nieminen J.O.
        • Lioumis P.
        • Metsomaa J.
        • Souza V.H.
        • Sinisalo H.
        • et al.
        Closed-loop optimization of transcranial magnetic stimulation with electroencephalography feedback.
        Brain Stimulat. 2022; 15: 523-531
        • Gordon P.C.
        • Jovellar D.B.
        • Song Y.
        • Zrenner C.
        • Belardinelli P.
        • Siebner H.R.
        • Ziemann U.
        Recording brain responses to TMS of primary motor cortex by EEG - utility of an optimized sham procedure.
        NeuroImage. 2021; 245: 118708
        • Biabani M.
        • Fornito A.
        • Mutanen T.P.
        • Morrow J.
        • Rogasch N.C.
        Characterizing and minimizing the contribution of sensory inputs to TMS-evoked potentials.
        Brain Stimulat. 2019; 12: 1537-1552
        • Raffin E.
        • Harquel S.
        • Passera B.
        • Chauvin A.
        • Bougerol T.
        • David O.
        Probing regional cortical excitability via input-output properties using transcranial magnetic stimulation and electroencephalography coupling.
        Hum Brain Mapp. 2020; 41: 2741-2761
        • Gordon P.C.
        • Desideri D.
        • Belardinelli P.
        • Zrenner C.
        • Ziemann U.
        Comparison of cortical EEG responses to realistic sham versus real TMS of human motor cortex.
        Brain Stimulat. 2018; 11: 1322-1330
        • Du X.
        • Choa F.-S.
        • Summerfelt A.
        • Rowland L.M.
        • Chiappelli J.
        • Kochunov P.
        • Hong L.E.
        N100 as a generic cortical electrophysiological marker based on decomposition of TMS-evoked potentials across five anatomic locations.
        Exp Brain Res. 2017; 235: 69-81
        • Nikulin V.V.
        • Kicić D.
        • Kähkönen S.
        • Ilmoniemi R.J.
        Modulation of electroencephalographic responses to transcranial magnetic stimulation: evidence for changes in cortical excitability related to movement.
        Eur J Neurosci. 2003; 18: 1206-1212
        • Ferrarelli F.
        • Phillips M.L.
        Examining and Modulating Neural Circuits in Psychiatric Disorders With Transcranial Magnetic Stimulation and Electroencephalography: Present Practices and Future Developments.
        Am J Psychiatry. 2021; 178: 400-413
        • Ferrarelli F.
        • Massimini M.
        • Peterson M.J.
        • Riedner B.A.
        • Lazar M.
        • Murphy M.J.
        • et al.
        Reduced evoked gamma oscillations in the frontal cortex in schizophrenia patients: a TMS/EEG study.
        Am J Psychiatry. 2008; 165: 996-1005
        • Massimini M.
        • Ferrarelli F.
        • Murphy M.
        • Huber R.
        • Riedner B.
        • Casarotto S.
        • Tononi G.
        Cortical reactivity and effective connectivity during REM sleep in humans.
        Cogn Neurosci. 2010; 1: 176-183
        • Ferrarelli F.
        • Massimini M.
        • Sarasso S.
        • Casali A.
        • Riedner B.A.
        • Angelini G.
        • et al.
        Breakdown in cortical effective connectivity during midazolam-induced loss of consciousness.
        Proc Natl Acad Sci U S A. 2010; 107: 2681-2686
        • Bodart O.
        • Fecchio M.
        • Massimini M.
        • Wannez S.
        • Virgillito A.
        • Casarotto S.
        • et al.
        Meditation-induced modulation of brain response to transcranial magnetic stimulation.
        Brain Stimulat. 2018; 11: 1397-1400
      18. Local sleep-like cortical reactivity in the awake brain after focal injury - PubMed (n.d.): Retrieved July 12, 2022, from https://pubmed.ncbi.nlm.nih.gov/33188680/

        • Rosanova M.
        • Gosseries O.
        • Casarotto S.
        • Boly M.
        • Casali A.G.
        • Bruno M.-A.
        • et al.
        Recovery of cortical effective connectivity and recovery of consciousness in vegetative patients.
        Brain J Neurol. 2012; 135: 1308-1320
      19. Wang JB, Bruss JE, Oya H, Uitermarkt BD, Trapp NT, Gander PE, et al. (2022, January 21): Effects of transcranial magnetic stimulation on the human brain recorded with intracranial electrocorticography: First-in-human study. bioRxiv, p 2022.01.18.476811.

        • Keller C.J.
        • Honey C.J.
        • Mégevand P.
        • Entz L.
        • Ulbert I.
        • Mehta A.D.
        Mapping human brain networks with cortico-cortical evoked potentials.
        Philos Trans R Soc Lond B Biol Sci. 2014; 369https://doi.org/10.1098/rstb.2013.0528
        • Keller C.J.
        • Bickel S.
        • Entz L.
        • Ulbert I.
        • Milham M.P.
        • Kelly C.
        • Mehta A.D.
        Intrinsic functional architecture predicts electrically evoked responses in the human brain.
        Proc Natl Acad Sci U S A. 2011; 108: 10308-10313
        • Keller C.J.
        • Honey C.J.
        • Entz L.
        • Bickel S.
        • Groppe D.M.
        • Toth E.
        • et al.
        Corticocortical evoked potentials reveal projectors and integrators in human brain networks.
        J Neurosci Off J Soc Neurosci. 2014; 34: 9152-9163
        • Huang Y.
        • Hajnal B.
        • Entz L.
        • Fabó D.
        • Herrero J.L.
        • Mehta A.D.
        • Keller C.J.
        Intracortical Dynamics Underlying Repetitive Stimulation Predicts Changes in Network Connectivity.
        J Neurosci Off J Soc Neurosci. 2019; 39: 6122-6135
        • Keller C.J.
        • Huang Y.
        • Herrero J.L.
        • Fini M.E.
        • Du V.
        • Lado F.A.
        • et al.
        Induction and Quantification of Excitability Changes in Human Cortical Networks.
        J Neurosci Off J Soc Neurosci. 2018; 38: 5384-5398
      20. Cline CC, Lucas MV, Sun Y, Menezes M, Etkin A (2021): Advanced Artifact Removal for Automated TMS-EEG Data Processing. 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER) 1039–1042.

        • Mutanen T.P.
        • Kukkonen M.
        • Nieminen J.O.
        • Stenroos M.
        • Sarvas J.
        • Ilmoniemi R.J.
        Recovering TMS-evoked EEG responses masked by muscle artifacts.
        NeuroImage. 2016; 139: 157-166
        • Pion-Tonachini L.
        • Kreutz-Delgado K.
        • Makeig S.
        ICLabel: An automated electroencephalographic independent component classifier, dataset, and website.
        NeuroImage. 2019; 198: 181-197
        • Kallioniemi E.
        • Daskalakis Z.J.
        Identifying novel biomarkers with TMS-EEG - Methodological possibilities and challenges.
        J Neurosci Methods. 2022; 377: 109631
        • Cao K.-X.
        • Ma M.-L.
        • Wang C.-Z.
        • Iqbal J.
        • Si J.-J.
        • Xue Y.-X.
        • Yang J.-L.
        TMS-EEG: An emerging tool to study the neurophysiologic biomarkers of psychiatric disorders.
        Neuropharmacology. 2021; 197: 108574