logo

SCIENTIA SINICA Informationis, Volume 49, Issue 9: 1097-1118(2019) https://doi.org/10.1360/N112018-00337

A review of EEG features for emotion recognition

More info
  • ReceivedDec 24, 2018
  • AcceptedMar 19, 2019
  • PublishedSep 6, 2019

Abstract

Emotion recognition is an important research topic in the human-machine interaction field, and it can be applied to medicine, education, psychology, military, and other areas. Electroencephalogram (EEG) signals are mostly used among various indices of emotion recognition. High accuracy of emotion classifiers can be achieved by extracting the most relevant and discriminant features of emotion states. This study surveys EEG features that are extensively used in current emotion recognition studies by introducing EEG features from the following four viewpoints: time domain, frequency domain, time–frequency domain, and space domain. An SLDA algorithm is imported to three public EEG-emotion datasets (SEED, DREAMER, and CAS-THU) to evaluate feature capabilities that distinguish emotion valence. Existing problems and future investigations are also discussed in this paper.


Funded by

国家自然科学基金(U1736220,61725204)


References

[1] Huang X T. Introduction to Psychology. Beijing: Peoples Education Press, 1991. Google Scholar

[2] van den Broek E L. Ubiquitous emotion-aware computing. Pers Ubiquit Comput, 2013, 17: 53-67 CrossRef Google Scholar

[3] Posner J, Russell J A, Peterson B S. The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology.. Develop Psychopathol, 2005, 17 CrossRef PubMed Google Scholar

[4] Lang P J. The emotion probe: Studies of motivation and attention.. Am Psychologist, 1995, 50: 372-385 CrossRef Google Scholar

[5] Zhao G Z, Song J J, Ge Y, et al. Advances in emotion recognition based on physiological big data. J Comput Res Dev, 2016, 53: 80--92. Google Scholar

[6] Alarcao S M, Fonseca M J. Emotions recognition using EEG signals: A survey. IEEE Transactions on Affective Computing, 2017. Google Scholar

[7] Chanel G, Kierkels J J M, Soleymani M. Short-term emotion assessment in a recall paradigm. Int J Human-Comput Studies, 2009, 67: 607-627 CrossRef Google Scholar

[8] Hruby T, Marsalek P. Event-related potentials-the P3 wave. Acta Neurobiol Exp, 2002, 63: 55--63. Google Scholar

[9] Luck S J, Kappenman E S. The Oxford Handbook of Event-Related Potential Components. Oxford: Oxford University Press, 2011. Google Scholar

[10] Lithari C, Frantzidis C A, Papadelis C. Are females more responsive to emotional stimuli? A neurophysiological study across arousal and valence dimensions.. Brain Topogr, 2010, 23: 27-40 CrossRef PubMed Google Scholar

[11] Yazdani A, Lee J S, Ebrahimi T. Implicit emotional tagging of multimedia using EEG signals and brain computer interface. In: Proceedings of the 1st SIGMM Workshop on Social Media, Beijing, 2009. 81--88. Google Scholar

[12] Codispoti M, Ferrari V, Bradley M M. Repetition and event-related potentials: distinguishing early and late processes in affective picture perception.. J Cognitive Neuroscience, 2007, 19: 577-586 CrossRef PubMed Google Scholar

[13] Olofsson J K, Nordin S, Sequeira H. Affective picture processing: an integrative review of ERP findings.. Biol Psychology, 2008, 77: 247-265 CrossRef PubMed Google Scholar

[14] Olofsson J K, Polich J. Affective visual event-related potentials: arousal, repetition, and time-on-task.. Biol Psychology, 2007, 75: 101-108 CrossRef PubMed Google Scholar

[15] Gianotti L R R, Faber P L, Schuler M. First valence, then arousal: the temporal dynamics of brain electric activity evoked by emotional stimuli.. Brain Topogr, 2008, 20: 143-156 CrossRef PubMed Google Scholar

[16] Jiang J F, Zeng Y, Tong L, et al. Single-trial ERP detecting for emotion recognition. In: Proceedings of the 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD), Shanghai, 2016. 105--108. Google Scholar

[17] Smith N K, Cacioppo J T, Larsen J T. May I have your attention, please: Electrocortical responses to positive and negative stimuli. Neuropsychologia, 2003, 41: 171-183 CrossRef Google Scholar

[18] Kim M K, Kim M, Oh E. A review on the computational methods for emotional state estimation from the human EEG.. Comput Math Methods Med, 2013, 2013(4): 1-13 CrossRef PubMed Google Scholar

[19] Bernat E, Bunce S, Shevrin H. Event-related brain potentials differentiate positive and negative mood adjectives during both supraliminal and subliminal visual processing. Int J PsychoPhysiol, 2001, 42: 11-34 CrossRef Google Scholar

[20] Cuthbert B N, Schupp H T, Bradley M M. Brain potentials in affective picture processing: covariation with autonomic arousal and affective report. Biol Psychology, 2000, 52: 95-111 CrossRef Google Scholar

[21] Nieuwenhuis S, Aston-Jones G, Cohen J D. Decision making, the P3, and the locus coeruleus-norepinephrine system.. Psychological Bull, 2005, 131: 510-532 CrossRef PubMed Google Scholar

[22] Jenke R, Peer A, Buss M. Feature Extraction and Selection for Emotion Recognition from EEG. IEEE Trans Affective Comput, 2014, 5: 327-339 CrossRef Google Scholar

[23] Wang X W, Nie D, Lu B L. EEG-based emotion recognition using frequency domain features and support vector machines. In: Proceedings of International Conference on Neural Information Processing, Berlin, 2011. 734--743. Google Scholar

[24] Bastos-Filho T F, Ferreira A, Atencio A C, et al. Evaluation of feature extraction techniques in emotional state recognition. In: Proceedings of the 4th International Conference on Intelligent Human Computer Interaction (IHCI), Kharagpur, 2012. Google Scholar

[25] Picard R W, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Machine Intell, 2001, 23: 1175-1191 CrossRef Google Scholar

[26] Kroupi E, Yazdani A, Ebrahimi T. EEG correlates of different emotional states elicited during watching music videos. In: Proceedings of Affective Computing and Intelligent Interaction, Berlin, 2011. 457--466. Google Scholar

[27] Fan C X, Cao L N. Communication principle. Beijing: National Defense Industrial Press, 2001. Google Scholar

[28] Hjorth B. EEG analysis based on time domain properties. Electroencephalography Clin NeuroPhysiol, 1970, 29: 306-310 CrossRef Google Scholar

[29] Petrantonakis P C, Hadjileontiadis L J. Emotion recognition from EEG using higher order crossings.. IEEE Trans Inform Technol Biomed, 2010, 14: 186-197 CrossRef PubMed Google Scholar

[30] Hausdorff J M, Lertratanakul A, Cudkowicz M E. Dynamic markers of altered gait rhythm in amyotrophic lateral sclerosis.. J Appl Physiol, 2000, 88: 2045-2053 CrossRef PubMed Google Scholar

[31] Ansari Asl K, Chanel G, Pun T. A channel selection method for EEG classification in emotion assessment based on synchronization likelihood. In: Proceedings of the 15th European Signal Processing Conference, Poznan, 2007. 1241--1245. Google Scholar

[32] Khosrowabadi R, bin Abdul Rahman A W. Classification of EEG correlates on emotion using features from Gaussian mixtures of EEG spectrogram. In: Proceeding of the 3rd International Conference on Information and Communication Technology for the Moslem World (ICT4M), Jakarta, 2010. 102--107. Google Scholar

[33] Sourina O, Liu Y S. A Fractal-based algorithm of emotion recognition from EEG using arousal-valence model. In: Proceedings of the International Conference on Bio-inspired Systems and Signal Processing (BIOSIGNALS-2011), Rome, 2011. 209--214. Google Scholar

[34] Liu Y, Sourina O. Real-time fractal-based valence level recognition from EEG. In: Proceedings of Transactions on Computational Science XVIII, Berlin, 2013. 101--120. Google Scholar

[35] Conneau A C, Essid S. Assessment of new spectral features for eeg-based emotion recognition. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, 2014. 4698--4702. Google Scholar

[36] Nie D, Wang X W, Duan R N, et al. A survey on EEG based emotion recognition. Chinese J Biomed Eng, 2012, 31: 595--606. Google Scholar

[37] Zheng J L, Ying Q H, Yang W L. Signal and System. 2nd ed. Beijing: Higher Education Press, 2000. Google Scholar

[38] Sammler D, Grigutsch M, Fritz T. Music and emotion: electrophysiological correlates of the processing of pleasant and unpleasant music.. Psychophysiology, 2007, 44: 293-304 CrossRef PubMed Google Scholar

[39] Davidson R J. What does the prefrontal cortex &. Google Scholar

[40] Yuvaraj R, Murugappan M, Ibrahim N M. Optimal set of EEG features for emotional state classification and trajectory visualization in Parkinson's disease.. Int J PsychoPhysiol, 2014, 94: 482-495 CrossRef PubMed Google Scholar

[41] Aftanas L I, Varlamov A A, Pavlov S V. Affective picture processing: event-related synchronization within individually defined human theta band is modulated by valence dimension.. NeuroSci Lett, 2001, 303: 115-118 CrossRef Google Scholar

[42] Yuvaraj R, Murugappan M, Ibrahim N M. Emotion classification in Parkinson's disease by higher-order spectra and power spectrum features using EEG signals: a comparative study.. J Integr Neurosci, 2014, 13: 89-120 CrossRef PubMed Google Scholar

[43] Keil A, Müller M M, Gruber T. Effects of emotional arousal in the cerebral hemispheres: a study of oscillatory brain activity and event-related potentials. Clin NeuroPhysiol, 2001, 112: 2057-2068 CrossRef Google Scholar

[44] Oude Bos D. EEG-based emotion recognition-the influence of visual and auditory stimuli. Emotion, 2007, 57: 1798--1806. Google Scholar

[45] Balconi M, Lucchiari C. Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis.. Int J PsychoPhysiol, 2008, 67: 41-46 CrossRef PubMed Google Scholar

[46] Wei-Long Zheng , Bao-Liang Lu . Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Trans Auton Mental Dev, 2015, 7: 162-175 CrossRef Google Scholar

[47] Bekkedal M Y V, Rossi Iii J, Panksepp J. Human brain EEG indices of emotions: delineating responses to affective vocalizations by measuring frontal theta event-related synchronization.. NeuroSci BioBehaval Rev, 2011, 35: 1959-1970 CrossRef PubMed Google Scholar

[48] Graimann B, Pfurtscheller G. Quantification and visualization of event-related changes in oscillatory brain activity in the time-frequency domain. Progress in brain research, 2006, 159: 79-97. Google Scholar

[49] Balconi M, Lucchiari C. EEG correlates (event-related desynchronization) of emotional face elaboration: a temporal analysis.. NeuroSci Lett, 2006, 392: 118-123 CrossRef PubMed Google Scholar

[50] Duan R N, Zhu J Y, Lu B L. Differential entropy feature for EEG-based emotion classification. In: Proceedings of the 6th International IEEE/EMBS Conference on Neural Engineering (NER), San Diego, 2013. 81--84. Google Scholar

[51] Shi L C, Jiao Y Y, Lu B L. Differential entropy feature for EEG-based vigilance estimation. In: Proceedings of the 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, 2013. 6627--6630. Google Scholar

[52] Behnam H, Sheikhani A, Mohammadi M R, et al. Analyses of EEG background activity in Autism disorders with fast Fourier transform and short time Fourier measure. In: Proceedings of International Conference on Intelligent and Advanced Systems, Kuala Lumpur, 2007. 1240--1244. Google Scholar

[53] Kiymik M K, Güler I, Dizibüyük A. Comparison of STFT and wavelet transform methods in determining epileptic seizure activity in EEG signals for real-time application.. Comput Biol Med, 2005, 35: 603-616 CrossRef PubMed Google Scholar

[54] Yoon H J, Chung S Y. EEG-based emotion estimation using Bayesian weighted-log-posterior function and perceptron convergence algorithm.. Comput Biol Med, 2013, 43: 2230-2237 CrossRef PubMed Google Scholar

[55] Rozgić V, Vitaladevuni S N, Prasad R. Robust EEG emotion classification using segment level decision fusion. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouver, 2013. 1286--1290. Google Scholar

[56] Akin M. Comparison of wavelet transform and FFT methods in the analysis of EEG signals. J Med Syst, 2002, 26: 241-247 CrossRef Google Scholar

[57] Adeli H, Zhou Z, Dadmehr N. Analysis of EEG records in an epileptic patient using wavelet transform. J NeuroSci Methods, 2003, 123: 69-87 CrossRef Google Scholar

[58] Sun Z, Chang C C. Structural Damage Assessment Based on Wavelet Packet Transform. J Struct Eng, 2002, 128: 1354-1361 CrossRef Google Scholar

[59] Hadjidimitriou S K, Hadjileontiadis L J. Toward an EEG-based recognition of music liking using time-frequency analysis.. IEEE Trans Biomed Eng, 2012, 59: 3498-3510 CrossRef PubMed Google Scholar

[60] Davidson R J, Ekman P, Saron C D. Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology: I.. J Personality Social Psychology, 1990, 58: 330-341 CrossRef Google Scholar

[61] Huang D, Guan C, Ang K K, et al. Asymmetric spatial pattern for EEG-based emotion detection. In: Proceedings of International Joint Conference on Neural Networks (IJCNN), Brisbane, 2012. Google Scholar

[62] Takahashi K. Remarks on emotion recognition from bio-potential signals. In: Proceedings of IEEE International Conference on Industrial Technology (IEEE ICIT'04), Hammamet, 2004. 1148--1153. Google Scholar

[63] Davidson R, Fox N. Asymmetrical Brain Activity Discriminates between Positive and Negative Affective Stimuli in Human Infants. Science, 1982, 218: 1235-1237 CrossRef ADS Google Scholar

[64] Blankertz B, Tomioka R, Lemm S. Optimizing Spatial filters for Robust EEG Single-Trial Analysis. IEEE Signal Process Mag, 2008, 25: 41-56 CrossRef ADS Google Scholar

[65] Koelstra S, Yazdani A, Soleymani M, et al. Single trial classification of EEG and peripheral physiological signals for recognition of emotions induced by music videos. In: Proceedings of International Conference on Brain Informatics, Berlin, 2010. 89--100. Google Scholar

[66] Winkler I, Jäger M, Mihajlovic V, et al. Frontal EEG asymmetry based classification of emotional valence using common spatial patterns. World Acad Sci Eng Technol, 2010, 45: 373--378. Google Scholar

[67] Novi Q, Guan C, Dat T H, et al. Sub-band common spatial pattern (SBCSP) for brain-computer interface. In: Proceedings of the 3rd International IEEE/EMBS Conference on Neural Engineering, Kohala Coast, 2007. 204--207. Google Scholar

[68] Ang K K, Chin Z Y, Zhang H, et al. Filter bank common spatial pattern (FBCSP) in brain-computer interface. In: Proceedings of IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), Hong Kong, 2008. 2390--2397. Google Scholar

[69] Duan R N, Wang X W, Lu B L. EEG-based emotion recognition in listening music by using support vector machine and linear dynamic system. In: Proceedings of International Conference on Neural Information Processing, Berlin, 2012. 468--475. Google Scholar

[70] Sakata O, Shiina T, Saito Y. Multidimensional Directed Information and Its Application. Electron Comm Jpn Pt III, 2002, 85: 45-55 CrossRef Google Scholar

[71] Petrantonakis P C, Hadjileontiadis L J. A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition.. IEEE Trans Inform Technol Biomed, 2011, 15: 737-746 CrossRef PubMed Google Scholar

[72] Clemmensen L, Hastie T, Witten D. Sparse Discriminant Analysis. Technometrics, 2011, 53: 406-413 CrossRef Google Scholar

[73] Katsigiannis S, Ramzan N. DREAMER: A Database for Emotion Recognition Through EEG and ECG Signals From Wireless Low-cost Off-the-Shelf Devices.. IEEE J Biomed Health Inform, 2018, 22: 98-107 CrossRef PubMed Google Scholar

[74] Song T F, Zheng W M, Song P, et al. EEG emotion recognition using dynamical graph convolutional neural networks. IEEE Trans Affect Comput, 2018. Google Scholar

[75] Delorme A, Makeig S. EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis.. J NeuroSci Methods, 2004, 134: 9-21 CrossRef PubMed Google Scholar

[76] Liu Y J, Yu M, Zhao G. Real-Time Movie-Induced Discrete Emotion Recognition from EEG Signals. IEEE Trans Affective Comput, 2018, 9: 550-562 CrossRef Google Scholar

[77] Swami A, Mendel C, Nikias C. Higher-order spectral analysis (HOSA) toolbox. Version, 2000, 2: 3. Google Scholar

[78] Sjöstrand K, Clemmensen L H, Larsen R, et al. SpaSM: a matlab toolbox for sparse statistical modeling. J Stat Soft, 2018, 84: 37. Google Scholar

[79] Zhou Z H. Machine Learning. Beijing: Tsinghua University Press, 2016. Google Scholar

[80] Gross J J, Levenson R W. Emotion elicitation using films. Cognition Emotion, 1995, 9: 87-108 CrossRef Google Scholar

[81] Fredrickson B L. Positive emotions and upward spirals in organizations. Positive Organ Scholarship, 2003, 3: 163--175. Google Scholar

[82] Yu C, Sun K, Zhong M Y, et al. One-dimensional handwriting: inputting letters and words on smart glasses. In: Proceedings of CHI Conference on Human Factors in Computing Systems (CHI'16), San Jose, 2016. 71--82. Google Scholar

  • Figure 1

    (Color online) Valence-arousal dimensional emotion model

  • Figure 2

    (Color online) ERP pattern stimulated by event at 0 ms

  • Figure 3

    (Color online) FFT calculation when $N=8$

  • Figure 4

    (Color online) 30 sampling electrodes of 32-channel NeuroScan Quik-cap

  • Table 1   Feature dimensions on SEED, DREAMER and CAS-THU
    Domain Feature SEED DREAMER CAS-THU
    Time Mean 62 14 14
    Standard deviation 62 14 14
    1-order difference 62 14 14
    Normalized 1-order difference 62 14 14
    2-order difference 62 14 14
    Normalized 2-order difference 62 14 14
    Hjorth-activity 62 14 14
    Hjorth-mobility 62 14 14
    Hjorth-complexity 62 14 14
    Energy 62 14 14
    Power 62 14 14
    HOC 310 70 70
    NSI 62 14 14
    FD 62 14 14
    Time-frequency PSD 310 42 70
    HOS 248 56 56
    DE 310 42 70
    Space DASM 135 21 35
    RASM 135 21 35
    Index 27 7 7
    DCAU 115 6 10
    MDI 27 7 7
    CSP 9
    Total 2423 463 542
  • Table 2   Features whose importance values are of top 10 on 2 or 3 datasets of SEED, DREAMER and CAS-THU when $x=$10, 30 or 50
    $x$ Domain 2 datasets 3 datasets
    10 Time Normalized 1-order difference 1-order difference
    Normalized 2-order difference 2-order difference
    Hjorth-activity Hjorth-complexity
    NSI Hjorth-mobility
    FD
    Space DASM
    RASM
    30 Time Normalized 1-order difference 1-order difference
    Normalized 2-order difference 2-order difference
    Hjorth-mobility Hjorth-complexity
    HOC NSI
    FD
    Space DASM
    50 Time Normalized 1-order difference
    Normalized 2-order difference 1-order difference
    Hjorth-complexity 2-order difference
    Hjorth-mobility NSI
    FD
    Time-frequency DE

Copyright 2019 Science China Press Co., Ltd. 《中国科学》杂志社有限责任公司 版权所有

京ICP备18024590号-1