A Systematic Review on Emotion Recognition System Using Physiological Signals: Data Acquisition and Methodology

Tawsif K., Nor Azlina Ab. Aziz, J. Emerson Raja, J. Hossen, Jesmeen M. Z. H.

Abstract


Emotion recognition systems (ERS) have become a popular research field to contribute to human-machine interaction in different areas. Different kinds of applications on ERS can serve different purposes. Artificial intelligence (AI) and the internet of things (IoT) are the technologies behind such applications. The main objective of this study is to enable researchers and developers to search for the most suitable options to develop an emotional state recognition system. More specifically, this paper presents work on ERS, which is built using physiological signals extracted from biosensors. It also presents details of how the extracted physiological signals are used to identify the user's emotional state. In this review, the sensors are categorized based on their modality: contact-based sensors and contactless sensors. Next, the ERS process is presented together with the reported results for each described technique. Articles from four different research databases were reviewed, of which 147 articles from 2009 to 2021 were referred to that are related to ERS using physiological signals. This paper should be significant for researchers developing systems that integrate human emotion recognition capability. The findings reported here can guide them in choosing suitable methods for their systems.

 

Doi: 10.28991/ESJ-2022-06-05-017

Full Text: PDF


Keywords


Emotion Recognition System; Biosensors; Physiological Sensors; Physiological Signals.

References


Mühlbacher-Karrer, S., Mosa, A. H., Faller, L. M., Ali, M., Hamid, R., Zangl, H., & Kyamakya, K. (2017). A Driver State Detection System - Combining a Capacitive Hand Detection Sensor with Physiological Sensors. IEEE Transactions on Instrumentation and Measurement, 66(4), 624–636. doi:10.1109/TIM.2016.2640458.

Ali, M., Al Machot, F., Mosa, A. H., & Kyamakya, K. (2016). A novel EEG-based emotion recognition approach for e-healthcare applications. Proceedings of the 31st Annual ACM Symposium on Applied Computing. doi:10.1145/2851613.2851916.

Ambach, W., & Gamer, M. (2018). Physiological Measures in the Detection of Deception and Concealed Information. Detecting Concealed Information and Deception: Recent Developments. Academic Press, Massachusetts, United States. doi:10.1016/B978-0-12-812729-2.00001-X.

Kang, S., Kim, D., & Kim, Y. (2019). A visual-physiology multimodal system for detecting outlier behavior of participants in a reality TV show. International Journal of Distributed Sensor Networks, 15(7). doi:10.1177/1550147719864886.

Kumar, A., Garg, N., & Kaur, G. (2019). An emotion recognition based on physiological signals. International Journal of Innovative Technology and Exploring Engineering, 8(9S), 335–341. doi:10.35940/ijitee.I1054.0789S19.

Gravina, R., & Li, Q. (2019). Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion. Information Fusion, 48, 1–10. doi:10.1016/j.inffus.2018.08.001.

Joseph, A., & Geetha, P. (2020). Facial emotion detection using modified Eyemap–Mouthmap algorithm on an enhanced image and classification with Tensorflow. Visual Computer, 36(3), 529–539. doi:10.1007/s00371-019-01628-3.

Ko, B. C. (2018). A brief review of facial emotion recognition based on visual information. Sensors (Switzerland), 18(2). doi:10.3390/s18020401.

Udovičić, G., Derek, J., Russo, M., & Sikora, M. (2017). Wearable Emotion Recognition system based on GSR and PPG signals. Proceedings of the 2nd International Workshop on Multimedia for Personal Health and Health Care. doi:10.1145/3132635.3132641.

Joseph, C., & Lekamge, S. (2019). Machine Learning Approaches for Emotion Classification of Music: A Systematic Literature Review. 2019 International Conference on Advancements in Computing (ICAC). doi:10.1109/ICAC49085.2019.9103378.

Chen, D. W., Miao, R., Yang, W. Q., Liang, Y., Chen, H. H., Huang, L., Deng, C. J., & Han, N. (2019). A feature extraction method based on differential entropy and linear discriminant analysis for emotion recognition. Sensors (Switzerland), 19(7), 1631. doi:10.3390/s19071631.

Bhise, P. R., Kulkarni, S. B., & Aldhaheri, T. A. (2020). Brain Computer Interface based EEG for Emotion Recognition System: A Systematic Review. 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA). doi:10.1109/ICIMIA48430.2020.9074921.

Hasnul, M. A., Aziz, N. A. A., Alelyani, S., Mohana, M., & Aziz, A. A. (2021). Electrocardiogram‐based emotion recognition systems and their applications in healthcare—a review. Sensors, 21(15). doi:10.3390/s21155015.

Ekman, P. (1992). An Argument for Basic Emotions. Cognition and Emotion, 6(3–4), 169–200. doi:10.1080/02699939208411068.

Izard, C. E. (2007). Basic Emotions, Natural Kinds, Emotion Schemas, and a New Paradigm. Perspectives on Psychological Science, 2(3), 260–280. doi:10.1111/j.1745-6916.2007.00044.x.

Plutchik, R. (2001). The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist, 89(4), 344–350. doi:10.1511/2001.4.344.

Acheampong, F. A., Wenyu, C., & Nunoo‐Mensah, H. (2020). Text‐based emotion detection: Advances, challenges, and opportunities. Engineering Reports, 2(7). doi:10.1002/eng2.12189.

Cambria, E., Livingstone, A., Hussain, A. (2012). The Hourglass of Emotions. Cognitive Behavioural Systems. Lecture Notes in Computer Science, vol 7403. Springer, Berlin, Germany. doi:10.1007/978-3-642-34584-5_11.

Lang, P. J. (1995). The Emotion Probe. American Psychologist Association, 50(5), 372–385. doi:10.1037/0003-066X.50.5.372.

Bailon, C., Damas, M., Pomares, H., Sanabria, D., Perakakis, P., Goicoechea, C., & Banos, O. (2018). Intelligent Monitoring of Affective Factors Underlying Sport Performance by Means of Wearable and Mobile Technology. Proceedings, 2(19), 1202. doi:10.3390/proceedings2191202.

Posner, J., Russell, J. A., & Peterson, B. S. (2005). The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and Psychopathology, 17(3), 715–734. doi:10.1017/S0954579405050340.

Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018). A review of emotion recognition using physiological signals. Sensors (Switzerland), 18(7). doi:10.3390/s18072074.

Furman, J. M., & Wuyts, F. L. (2012). Front matter. Aminoff’s Electrodiagnosis in Clinical Neurology (6th Ed.). Saunders, Philadelphia, United States. doi:10.1016/b978-1-4557-0308-1.00041-8.

Khalifa, W. H., Roushdy, M. I., Abdel-badeeh, M. S., & Revett, K. (2015). AIS Inspired Approach for User Identification Based on EEG Signals. Recent Advances in Information Science AIS, 84-89.

Heng, H. M., Lu, M. K., Chou, L. W., Meng, N. H., Huang, H. C., Hamada, M., Tsai, C. H., & Chen, J. C. (2020). Changes in balance, gait and electroencephalography oscillations after robot-assisted gait training: An exploratory study in people with chronic stroke. Brain Sciences, 10(11), 1–13. doi:10.3390/brainsci10110821.

Alyasseri, Z. A. A., Khader, A. T., Al-Betar, M. A., Abasi, A. K., & Makhadmeh, S. N. (2020). EEG Signals Denoising Using Optimal Wavelet Transform Hybridized with Efficient Metaheuristic Methods. IEEE Access, 8(1), 10584–10605. doi:10.1109/ACCESS.2019.2962658.

Imad, A., Malik, N. A., Hamida, B. A., Seng, G. H. H., & Khan, S. (2022). Acoustic Photometry of Biomedical Parameters for Association with Diabetes and Covid-19. Emerging Science Journal, 6, 42-56. doi:10.28991/esj-2022-SPER-04.

Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017). An Emotion Recognition Approach based on Wavelet Transform and Second-Order Difference Plot of ECG. Journal of AI and Data Mining, 5(2), 211–221. doi:10.22044/JADM.2017.887.

Tada, Y., Amano, Y., Sato, T., Saito, S., & Inoue, M. (2015). A smart shirt made with conductive ink and conductive foam for the measurement of electrocardiogram signals with unipolar precordial leads. Fibers, 3(4), 463–477. doi:10.3390/fib3040463.

Khan, G. M. (2015). A new electrode placement method for obtaining 12-lead ECGs. Open Heart, 2(1), e000226. doi:10.1136/openhrt-2014-000226.

Zong, C., & Chetouani, M. (2009). Hilbert-Huang transform based physiological signals analysis for emotion recognition. 2009 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT). doi:10.1109/isspit.2009.5407547.

Ladakis, I., & Chouvarda, I. (2021). Overview of biosignal analysis methods for the assessment of stress. Emerging Science Journal, 5(2), 233-244. doi:10.28991/esj-2021-01267.

Matzke, B., Herpertz, S. C., Berger, C., Fleischer, M., & Domes, G. (2014). Facial reactions during emotion recognition in borderline personality disorder: A facial electromyography study. Psychopathology, 47(2), 101–110. doi:10.1159/000351122.

Murata, A., Saito, H., Schug, J., Ogawa, K., & Kameda, T. (2016). Spontaneous facial mimicry is enhanced by the goal of inferring emotional states: Evidence for moderation of “automatic” mimicry by higher cognitive processes. PLoS ONE, 11(4). doi:10.1371/journal.pone.0153128.

Turabzadeh, S., Meng, H., Swash, R., Pleva, M., & Juhar, J. (2018). Facial Expression Emotion Detection for Real-Time Embedded Systems. Technologies, 6(1), 17. doi:10.3390/technologies6010017.

Huang, Y., Chen, F., Lv, S., & Wang, X. (2019). Facial expression recognition: A survey. Symmetry, 11(10), 1189. doi:10.3390/sym11101189.

Hsieh, P. Y., & Chin, C. L. (2011). The emotion recognition system with Heart Rate Variability and facial image features. 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2011). doi:10.1109/FUZZY.2011.6007734.

Huang, C., Liew, S. S., Lin, G. R., Poulsen, A., Ang, M. J., Chia, B. C., ... & Foo, K. (2019). Discovery of irreversible inhibitors targeting histone methyltransferase, SMYD3. ACS Medicinal Chemistry Letters, 10(6), 978-984.. doi:10.1021/acsmedchemlett.9b00170.

Benezeth, Y., Li, P., Macwan, R., Nakamura, K., Gomez, R., & Yang, F. (2018). Remote heart rate variability for emotional state monitoring. 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). doi:10.1109/bhi.2018.8333392.

Perpetuini, D., Chiarelli, A. M., Maddiona, L., Rinella, S., Bianco, F., Bucciarelli, V., ... & Fallica, G. (2019). Multi-site photoplethysmographic and electrocardiographic system for arterial stiffness and cardiovascular status assessment. Sensors, 19(24), 5570. doi:10.3390/s19245570.

Wang, W., Den Brinker, A. C., Stuijk, S., & De Haan, G. (2017). Algorithmic Principles of Remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479–1491. doi:10.1109/TBME.2016.2609282.

Lord, M. P., & Wright, W. D. (1950). The investigation of eye movements. Reports on Progress in Physics, 13(1), 1–23. doi:10.1088/0034-4885/13/1/301.

R.Aguiñaga, A., Lopez Ramirez, M., Alanis Garza, A., Baltazar, R., & M. Zamudio, V. (2013). Emotion analysis through physiological measurements. Workshop Proceedings of the 9th International Conference on Intelligent Environments J.A. Botía and D. Charitos (Eds.), 97–106. doi:10.3233/978-1-61499-286-8-97.

Picot, A., Charbonnier, S., & Caplier, A. (2011). EOG-based drowsiness detection: Comparison between a fuzzy system and two supervised learning classifiers. IFAC Proceedings Volumes, 44(1), 14283–14288. doi:10.3182/20110828-6-it-1002.00706.

Ramkumar, S., Sathesh Kumar, K., Dhiliphan Rajkumar, T., Ilayaraja, M., & Shankar, K. (2018). A review-classification of electrooculogram based human computer interfaces. Biomedical Research (India), 29(6), 1078–1084. doi:10.4066/biomedicalresearch.29-17-2979.

Siddiqui, U., & Shaikh, A. N. (2013). An overview of electrooculography. International Journal of Advanced Research in Computer and Communication Engineering, 2(11), 4328-4330.

Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273. doi:10.1111/j.1469-8986.1993.tb03352.x.

Wu, G., Liu, G., & Hao, M. (2010). The Analysis of Emotion Recognition from GSR Based on PSO. 2010 International Symposium on Intelligence Information Processing and Trusted Computing. doi:10.1109/iptc.2010.60.

van Dooren, M., de Vries, J. J. G. G. J., & Janssen, J. H. (2012). Emotional sweating across the body: Comparing 16 different skin conductance measurement locations. Physiology and Behavior, 106(2), 298–304. doi:10.1016/j.physbeh.2012.01.020.

Kosonogov, V., De Zorzi, L., Honoré, J., Martínez-Velázquez, E. S., Nandrino, J.-L., Martinez-Selva, J. M., & Sequeira, H. (2017). Facial thermal variations: A new marker of emotional arousal. PlosOne, 12(9), e0183592. doi:10.1371/journal.pone.0183592.

Ayata, D., Yaslan, Y., & Kamaşak, M. (2017). Emotion recognition via galvanic skin response: Comparison of machine learning algorithms and feature extraction methods. IU-Journal of Electrical & Electronics Engineering, 17(1), 3147-3156.

Daiana da Costa, T., de Fatima Fernandes Vara, M., Santos Cristino, C., Zoraski Zanella, T., Nunes Nogueira Neto, G., & Nohama, P. (2019). Breathing Monitoring and Pattern Recognition with Wearable Sensors. Wearable Devices - the Big Wave of Innovation, IntechOpen, London, United Kingdom. doi:10.5772/intechopen.85460.

Kim, J., & André, E. (2008). Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(12), 2067–2083. doi:10.1109/TPAMI.2008.26.

Norali, A. N., Abdullah, A. H., Zakaria, Z., Rahim, N. A., & Nataraj, S. K. (2017). Human breathing assessment using Electromyography signal of respiratory muscles. 2016 6th IEEE International Conference on Control System, Computing and Engineering (ICCSCE). doi:10.1109/ICCSCE.2016.7893596.

Kawde, P., & Verma, G. K. (2017). Deep belief network based affect recognition from physiological signals. 2017 4th IEEE Uttar Pradesh Section International Conference on Electrical, Computer and Electronics (UPCON). doi:10.1109/UPCON.2017.8251115.

Zhang, X., Xu, C., Xue, W., Hu, J., He, Y., & Gao, M. (2018). Emotion recognition based on multichannel physiological signals with comprehensive nonlinear processing. Sensors (Switzerland), 18(11), 1–16. doi:10.3390/s18113886.

Jerritta, S., Murugappan, M., Wan, K., & Yaacob, S. (2014). Emotion recognition from facial EMG signals using higher order statistics and principal component analysis. Journal of the Chinese Institute of Engineers, Transactions of the Chinese Institute of Engineers,Series A, 37(3), 385–394. doi:10.1080/02533839.2013.799946.

Gong, P., Ma, H. T., & Wang, Y. (2016). Emotion recognition based on the multiple physiological signals. 2016 IEEE International Conference on Real-time Computing and Robotics (RCAR). doi:10.1109/RCAR.2016.7784015.

Izard, C. E. (2009). Emotion theory and research: Highlights, unanswered questions, and emerging issues. Annual Review of Psychology, 60(60), 1–25. doi:10.1146/annurev.psych.60.110707.163539.

Suchetha, M., Kumaravel, N., Jagannath, M., & Jaganathan, S. K. (2017). A comparative analysis of EMD based filtering methods for 50 Hz noise cancellation in ECG signal. Informatics in Medicine Unlocked, 8, 54–59. doi:10.1016/j.imu.2017.01.003.

Lahmiri, S., & Boukadoum, M. (2015). Physiological signal denoising with variational mode decomposition and weighted reconstruction after DWT thresholding. 2015 IEEE International Symposium on Circuits and Systems (ISCAS). doi:10.1109/ISCAS.2015.7168756.

Mannan, M. M. N., Kamran, M. A., & Jeong, M. Y. (2018). Identification and removal of physiological artifacts from electroencephalogram signals: A review. IEEE Access, 6, 30630–30652. doi:10.1109/ACCESS.2018.2842082.

Hindarto, H., & Sumarno, S. (2016). Feature Extraction of Electroencephalography Signals Using Fast Fourier Transform. CommIT (Communication and Information Technology) Journal, 10(2), 49. doi:10.21512/commit.v10i2.1548.

Al-Fahoum, A. S., & Al-Fraihat, A. A. (2014). Methods of EEG Signal Features Extraction Using Linear Analysis in Frequency and Time-Frequency Domains. ISRN Neuroscience, 2014, 1–7. doi:10.1155/2014/730218.

Vanny, M., Park, SM., Ko, K. E., Sim, KB. (2013). Analysis of Physiological Signals for Emotion Recognition Based on Support Vector Machine. Robot Intelligence Technology and Applications 2012. Advances in Intelligent Systems and Computing, 208. Springer, Berlin, Germany. doi:10.1007/978-3-642-37374-9_12.

Wei, C., Chen, L. Lan, Song, Z. Zhen, Lou, X. Guang, & Li, D. dong. (2020). EEG-based emotion recognition using simple recurrent units network and ensemble learning. Biomedical Signal Processing and Control, 58, 101756. doi:10.1016/j.bspc.2019.101756.

Bal, U. (2012). Dual tree complex wavelet transform based denoising of optical microscopy images. Biomedical Optics Express, 3(12), 3231. doi:10.1364/boe.3.003231.

Chen, T., Ju, S., Yuan, X., Elhoseny, M., Ren, F., Fan, M., & Chen, Z. (2018). Emotion recognition using empirical mode decomposition and approximation entropy. Computers and Electrical Engineering, 72, 383–392. doi:10.1016/j.compeleceng.2018.09.022.

Jerritta, S., Murugappan, M., Wan, K., & Yaacob, S. (2014). Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform. Expert Systems, 31(2), 110–120. doi:10.1111/exsy.12014.

Liu, T., Luo, Z., Huang, J., & Yan, S. (2018). A comparative study of four kinds of adaptive decomposition algorithms and their applications. Sensors (Switzerland), 18(7), 1–51,. doi:10.3390/s18072120.

Hassan, M. M., Alam, M. G. R., Uddin, M. Z., Huda, S., Almogren, A., & Fortino, G. (2019). Human emotion recognition using deep belief network architecture. Information Fusion, 51, 10–18. doi:10.1016/j.inffus.2018.10.009.

Asghar, M. A., Khan, M. J., Fawad, Amin, Y., Rizwan, M., Rahman, M., … Mirjavadi, S. S. (2019). EEG-Based Multi-Modal Emotion Recognition using Bag of Deep Features: An Optimal Feature Selection Approach. Sensors, 19(23), 5218. doi:10.3390/s19235218.

Huang, C., Gong, W., Fu, W., & Feng, D. (2014). A research of speech emotion recognition based on deep belief network and SVM. Mathematical Problems in Engineering, 2014, 1-7. doi:10.1155/2014/749604.

Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2006). Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems, NeurIPS Proceedings, 153–160.

Dissanayake, T., Rajapaksha, Y., Ragel, R., & Nawinne, I. (2019). An ensemble learning approach for electrocardiogram sensor based human emotion recognition. Sensors (Switzerland), 19(20), 1–24. doi:10.3390/s19204495.

Colomer Granero, A., Fuentes-Hurtado, F., Naranjo Ornedo, V., Guixeres Provinciale, J., Ausín, J. M., & Alcañiz Raya, M. (2016). A comparison of physiological signal analysis techniques and classifiers for automatic emotional evaluation of audiovisual contents. Frontiers in Computational Neuroscience, 10, 1–14. doi:10.3389/fncom.2016.00074.

Deb, S., Rabiul Islam, S. M., Johura, F. T., & Huang, X. (2018). Extraction of Linear and Non-Linear Features of Electrocardiogram Signal and Classification. 2017 2nd International Conference on Electrical & Electronic Engineering (ICEEE). doi:10.1109/CEEE.2017.8412857.

Shukla, J., Barreda-Angeles, M., Oliver, J., Nandi, G. C., & Puig, D. (2021). Feature Extraction and Selection for Emotion Recognition from Electrodermal Activity. IEEE Transactions on Affective Computing, 12(4), 857–869. doi:10.1109/TAFFC.2019.2901673.

Konar, A., & Chakraborty, A. (2015). Emotion recognition: A pattern analysis approach. John Wiley & Sons, Hoboken, United States. doi:10.1002/9781118910566.

Mehmood, R. M., & Lee, H. J. (2015). Exploration of prominent frequency wave in EEG signals from brain sensors network. International Journal of Distributed Sensor Networks, 11(11). doi:10.1155/2015/386057.

Kyriakou, K., Resch, B., Sagl, G., Petutschnig, A., Werner, C., Niederseer, D., ... & Pykett, J. (2019). Detecting moments of stress from measurements of wearable physiological sensors. Sensors, 19(17), 3805. doi:10.3390/s19173805.

Kołodziej, M., Tarnowski, P., Majkowski, A., & Rak, R. J. (2019). Electrodermal activity measurements for detection of emotional arousal. Bulletin of the Polish Academy of Sciences: Technical Sciences, 67(4), 813–826. doi:10.24425/bpasts.2019.130190.

Cho, D., Ham, J., Oh, J., Park, J., Kim, S., Lee, N. K., & Lee, B. (2017). Detection of stress levels from biosignals measured in virtual reality environments using a kernel-based extreme learning machine. Sensors (Switzerland), 17(10). doi:10.3390/s17102435.

Gómez-Lara, J. F., Ordóñez-Bolaños, O. A., Becerra, M. A., Castro-Ospina, A. E., Mejía-Arboleda, C., Duque-Mejía, C., … Peluffo-Ordóñez, D. H. (2019). Feature Extraction Analysis for Emotion Recognition from ICEEMD of Multimodal Physiological Signals. Lecture Notes in Computer Science, 351–362. doi:10.1007/978-3-030-14799-0_30.

Peker, M., Arslan, A., Sen, B., Celebi, F. V., & But, A. (2015). A novel hybrid method for determining the depth of anesthesia level: Combining ReliefF feature selection and random forest algorithm (ReliefF+RF). 2015 International Symposium on Innovations in Intelligent SysTems and Applications (INISTA). doi:10.1109/inista.2015.7276737.

Elsayyad, A., Al-Dhaifallah, M., & Nassef, A. M. (2017). Features selection for arrhythmia diagnosis using Relief-F algorithm and support vector machine. 2017 14th International Multi-Conference on Systems, Signals & Devices (SSD). doi:10.1109/SSD.2017.8166920.

Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017). An accurate emotion recognition system using ECG and GSR signals and matching pursuit method. Biomedical Journal, 40(6), 355–368. doi:10.1016/j.bj.2017.11.001.

Smith, G. (2018). Step away from stepwise. Journal of Big Data, 5(1). doi:10.1186/s40537-018-0143-6.

Xu, L., Redman, C. W. G., Payne, S. J., & Georgieva, A. (2014). Feature selection using genetic algorithms for fetal heart rate analysis. Physiological Measurement, 35(7), 1357–1371. doi:10.1088/0967-3334/35/7/1357.

Romeo, L., Cavallo, A., Pepa, L., Bianchi-Berthouze, N., & Pontil, M. (2022). Multiple Instance Learning for Emotion Recognition Using Physiological Signals. IEEE Transactions on Affective Computing, 13(1), 389–407. doi:10.1109/TAFFC.2019.2954118.

Domínguez-Jiménez, J. A., Campo-Landines, K. C., Martínez-Santos, J. C., Delahoz, E. J., & Contreras-Ortiz, S. H. (2020). A machine learning model for emotion recognition from physiological signals. Biomedical Signal Processing and Control, 55, 101646. doi:10.1016/j.bspc.2019.101646.

Chang, C. C., & Lin, C. J. (2011). LIBSVM: A Library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3), 1–39,. doi:10.1145/1961189.1961199.

Chen, T., Ju, S., Ren, F., Fan, M., & Gu, Y. (2020). EEG emotion recognition model based on the LIBSVM classifier. Measurement, 164(108047). doi:10.1016/j.measurement.2020.108047.

Martinez, M., Schauerte, B., Stiefelhagen, R. (2013). “BAM!” Depth-Based Body Analysis in Critical Care. Computer Analysis of Images and Patterns. CAIP 2013. Lecture Notes in Computer Science, 8047. Springer, Berlin, Germany. doi:10.1007/978-3-642-40261-6_56.

Salari, S., Ansarian, A., & Atrianfar, H. (2018). Robust emotion classification using neural network models. 2018 6th Iranian Joint Congress on Fuzzy and Intelligent Systems (CFIS). doi:10.1109/CFIS.2018.8336626.

Song, T., Zheng, W., Song, P., & Cui, Z. (2020). EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks. IEEE Transactions on Affective Computing, 11(3), 532–541. doi:10.1109/TAFFC.2018.2817622.

Santamaria-Granados, L., Munoz-Organero, M., Ramirez-Gonzalez, G., Abdulhay, E., & Arunkumar, N. (2019). Using Deep Convolutional Neural Network for Emotion Detection on a Physiological Signals Dataset (AMIGOS). IEEE Access, 7(1), 57–67. doi:10.1109/ACCESS.2018.2883213.

Al Machot, F., Elmachot, A., Ali, M., Al Machot, E., & Kyamakya, K. (2019). A deep-learning model for subject-independent human emotion recognition using electrodermal activity sensors. Sensors (Switzerland), 19(7), 1–14. doi:10.3390/s19071659.

Pandey, S. K., & Janghel, R. R. (2019). Recent deep learning techniques, challenges and its applications for medical healthcare system: a review. Neural Processing Letters, 50(2), 1907-1935. doi:10.1007/s11063-018-09976-2.

Chao, H., Zhi, H., Dong, L., & Liu, Y. (2018). Recognition of Emotions Using Multichannel EEG Data and DBN-GC-Based Ensemble Deep Learning Framework. Computational Intelligence and Neuroscience, 2018, 1-11. doi:10.1155/2018/9750904.

Huang, J., Xu, X., & Zhang, T. (2017). Emotion classification using deep neural networks and emotional patches. 2017 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). doi:10.1109/BIBM.2017.8217786.

Nakisa, B., Rastgoo, M. N., Tjondronegoro, D., & Chandran, V. (2018). Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Systems with Applications, 93, 143–155. doi:10.1016/j.eswa.2017.09.062.

Wöllmer, M., Kaiser, M., Eyben, F., Schuller, B., & Rigoll, G. (2013). LSTM-modeling of continuous emotions in an audiovisual affect recognition framework. Image and Vision Computing, 31(2), 153–163. doi:10.1016/j.imavis.2012.03.001.

Li, X., Song, D., Zhang, P., Yu, G., Hou, Y., & Hu, B. (2016). Emotion recognition from multi-channel EEG data through Convolutional Recurrent Neural Network. 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). doi:10.1109/bibm.2016.7822545.

Li, Z., Tian, X., Shu, L., Xu, X., & Hu, B. (2018). Emotion recognition from EEG using RASM and LSTM. Communications in Computer and Information Science, 819, 310–318. doi:10.1007/978-981-10-8530-7_30.

Xing, X., Li, Z., Xu, T., Shu, L., Hu, B., & Xu, X. (2019). SAE+LSTM: A new framework for emotion recognition from multi-channel EEG. Frontiers in Neurorobotics, 13(37), 1–14,. doi:10.3389/fnbot.2019.00037.

Alhagry, S., Fahmy, A. A., & El-Khoribi, R. A. (2017). Emotion recognition based on EEG using LSTM recurrent neural network. International Journal of Advanced Computer Science and Applications, 8(10). doi:10.14569/ijacsa.2017.081046.

Ravanelli, M., & Bengio, Y. (2018). Speaker recognition from raw waveform with sincnet. 2018 IEEE Spoken Language Technology Workshop (SLT). doi:10.1109/SLT.2018.8639585.

Zeng, H., Wu, Z., Zhang, J., Yang, C., Zhang, H., Dai, G., & Kong, W. (2019). EEG emotion classification using an improved sincnet-based deep learning model. Brain Sciences, 9(11). doi:10.3390/brainsci9110326.

Gu, Y., Tan, S. L., Wong, K. J., Ho, M. H. R., & Qu, L. (2009). Using GA-based feature selection for emotion recognition from physiological signals. 2008 International Symposium on Intelligent Signal Processing and Communications Systems. doi:10.1109/ISPACS.2009.4806747.

Maaoui, C., & Pruski, A. (2010). Emotion Recognition through Physiological Signals for Human-Machine Communication. Cutting Edge Robotics 2010, 317–333. doi:10.5772/10312.

Agrafioti, F., Hatzinakos, D., & Anderson, A. K. (2012). ECG pattern analysis for emotion detection. IEEE Transactions on Affective Computing, 3(1), 102–115. doi:10.1109/T-AFFC.2011.28.

Selvaraj, J., Murugappan, M., Wan, K., & Yaacob, S. (2013). Classification of emotional states from electrocardiogram signals: A non-linear approach based on hurst. BioMedical Engineering Online, 12(1), 1–18. doi:10.1186/1475-925X-12-44.

Gruebler, A., & Suzuki, K. (2014). Design of a wearable device for reading positive expressions from facial EMG signals. IEEE Transactions on Affective Computing, 5(3), 227–237. doi:10.1109/TAFFC.2014.2313557.

Gouizi, K., Maaoui, C., & Bereksi Reguig, F. (2014). Negative emotion detection using EMG signal. 2014 International Conference on Control, Decision and Information Technologies (CoDIT). doi:10.1109/CoDIT.2014.6996980.

Jang, E. H., Park, B. J., Park, M. S., Kim, S. H., & Sohn, J. H. (2015). Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. Journal of Physiological Anthropology, 34(1), 1–12,. doi:10.1186/s40101-015-0063-5.

Soleymani, M., Asghari-Esfeden, S., Fu, Y., & Pantic, M. (2016). Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection. IEEE Transactions on Affective Computing, 7(1), 17–28. doi:10.1109/TAFFC.2015.2436926.

Guo, H. W., Huang, Y. S., Lin, C. H., Chien, J. C., Haraikawa, K., & Shieh, J. S. (2016). Heart Rate Variability Signal Features for Emotion Recognition by Using Principal Component Analysis and Support Vectors Machine. 2016 IEEE 16th International Conference on Bioinformatics and Bioengineering (BIBE). doi:10.1109/BIBE.2016.40.

He, C., Yao, Yj., Ye, Xs. (2017). An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors. Wearable Sensors and Robots. Lecture Notes in Electrical Engineering, 399. Springer, Singapore. doi.:10.1007/978-981-10-2404-7_2.

Atkinson, J., & Campos, D. (2016). Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Systems with Applications, 47, 35–41. doi:10.1016/j.eswa.2015.10.049.

Mehmood, R. M., & Lee, H. J. (2016). A novel feature extraction method based on late positive potential for emotion recognition in human brain signal patterns. Computers and Electrical Engineering, 53, 444–457. doi:10.1016/j.compeleceng.2016.04.009.

Park, Y. L. (2017). Soft wearable robotics technologies for body motion sensing. Human Modelling for Bio-Inspired Robotics, 161-184. Academic Press, Massachusetts, United States. doi:10.1016/B978-0-12-803137-7.00009-4.

Greco, A., Valenza, G., Citi, L., & Scilingo, E. P. (2017). Arousal and valence recognition of affective sounds based on electrodermal activity. IEEE Sensors Journal, 17(3), 716–725. doi:10.1109/JSEN.2016.2623677.

Chen, J., Hu, B., Wang, Y., Moore, P., Dai, Y., Feng, L., & Ding, Z. (2017). Subject-independent emotion recognition based on physiological signals: A three-stage decision method. BMC Medical Informatics and Decision Making, 17. doi:10.1186/s12911-017-0562-x.

Yin, Z., Zhao, M., Wang, Y., Yang, J., & Zhang, J. (2017). Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Computer Methods and Programs in Biomedicine, 140, 93–110. doi:10.1016/j.cmpb.2016.12.005.

Liu, X., Wang, Q., Liu, D., Wang, Y., Zhang, Y., Bai, O., & Sun, J. (2018). Human emotion classification based on multiple physiological signals by wearable system. Technology and Health Care, 26, 459–469. doi:10.3233/THC-174747.

Al Zoubi, O., Awad, M., & Kasabov, N. K. (2018). Anytime multipurpose emotion recognition from EEG data using a Liquid State Machine based framework. Artificial intelligence in medicine, 86, 1-8. doi:10.1016/j.artmed.2018.01.001.

Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2018). Long short term memory hyperparameter optimization for a neural network based emotion recognition framework. IEEE Access, 6(1), 49325–49338. doi:10.1109/ACCESS.2018.2868361.

Kaur, B., Singh, D., & Roy, P. P. (2018). EEG Based Emotion Classification Mechanism in BCI. Procedia Computer Science, 132, 752–758. doi:10.1016/j.procs.2018.05.087.

Ayata, D., Yaslan, Y., & Kamasak, M. E. (2018). Emotion Based Music Recommendation System Using Wearable Physiological Sensors. IEEE Transactions on Consumer Electronics, 64(2), 196–203. doi:10.1109/TCE.2018.2844736.

Bagherzadeh, S., Maghooli, K., Farhadi, J., & Zangeneh Soroush, M. (2018). Emotion Recognition from Physiological Signals Using Parallel Stacked Autoencoders. Neurophysiology, 50(6), 428–435. doi:10.1007/s11062-019-09775-y.

Chettupuzhakkaran, P., & Sindhu, N. (2018). Emotion recognition from physiological signals using time-frequency analysis methods. 2018 International Conference on Emerging Trends and Innovations in Engineering and Technological Research (ICETIETR). doi:10.1109/ICETIETR.2018.8529145.

Gupta, V., Chopda, M. D., & Pachori, R. B. (2019). Cross-Subject Emotion Recognition Using Flexible Analytic Wavelet Transform from EEG Signals. IEEE Sensors Journal, 19(6), 2266–2274. doi:10.1109/JSEN.2018.2883497.

Albraikan, A., Tobon, D. P., & El Saddik, A. (2019). Toward User-Independent Emotion Recognition Using Physiological Signals. IEEE Sensors Journal, 19(19), 8402–8412. doi:10.1109/JSEN.2018.2867221.

Xing, B., Zhang, H., Zhang, K., Zhang, L., Wu, X., Shi, X., Yu, S., & Zhang, S. (2019). Exploiting EEG Signals and Audiovisual Feature Fusion for Video Emotion Recognition. IEEE Access, 7(1), 59844–59861. doi:10.1109/ACCESS.2019.2914872.

Tiwari, S., Agarwal, S., Adiyarta, K., & Syafrullah, M. (2019). Classification of physiological signals for emotion recognition using IoT. 2019 6th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI). doi:10.23919/EECSI48112.2019.8977062.

Taran, S., & Bajaj, V. (2019). Emotion recognition from single-channel EEG signals using a two-stage correlation and instantaneous frequency-based filtering method. Computer Methods and Programs in Biomedicine, 173, 157–165. doi:10.1016/j.cmpb.2019.03.015.

Pandey, P., & Seeja, K. R. (2022). Subject independent emotion recognition from EEG using VMD and deep learning. Journal of King Saud University - Computer and Information Sciences, 34(5), 1730–1738. doi:10.1016/j.jksuci.2019.11.003.

Shu, L., Yu, Y., Chen, W., Hua, H., Li, Q., Jin, J., & Xu, X. (2020). Wearable emotion recognition using heart rate data from a smart bracelet. Sensors (Switzerland), 20(3), 1–19. doi:10.3390/s20030718.

Wang, F., Wu, S., Zhang, W., Xu, Z., Zhang, Y., Wu, C., & Coleman, S. (2020). Emotion recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia, 146, 107506. doi:10.1016/j.neuropsychologia.2020.107506.

Salama, E. S., El-Khoribi, R. A., Shoman, M. E., & Wahby Shalaby, M. A. (2021). A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egyptian Informatics Journal, 22(2), 167–176. doi:10.1016/j.eij.2020.07.005.

Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020). Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems. Journal of Medical and Biological Engineering, 40(2), 149–157. doi:10.1007/s40846-019-00505-7.

Fourati, R., Ammar, B., Sanchez-Medina, J., & Alimi, A. M. (2022). Unsupervised Learning in Reservoir Computing for EEG-Based Emotion Recognition. IEEE Transactions on Affective Computing, 13(2), 972–984. doi:10.1109/TAFFC.2020.2982143.

Liu, Y., Ding, Y., Li, C., Cheng, J., Song, R., Wan, F., & Chen, X. (2020). Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Computers in Biology and Medicine, 123(March), 103927. doi:10.1016/j.compbiomed.2020.103927.

Nakisa, B., Rastgoo, M. N., Rakotonirainy, A., Maire, F., & Chandran, V. (2020). Automatic emotion recognition using temporal multimodal deep learning. IEEE Access, 8, 225463-225474. doi:10.1109/ACCESS.2020.3027026.

Yin, Z., Liu, L., Chen, J., Zhao, B., & Wang, Y. (2020). Locally robust EEG feature selection for individual-independent emotion recognition. Expert Systems with Applications, 162, 113768. doi:10.1016/j.eswa.2020.113768.

Li, C., Bao, Z., Li, L., & Zhao, Z. (2020). Exploring temporal representations by leveraging attention-based bidirectional LSTM-RNNs for multi-modal emotion recognition. Information Processing and Management, 57(3), 102185. doi:10.1016/j.ipm.2019.102185.

Sarkar, P., & Etemad, A. (2020). Self-supervised ECG representation learning for emotion recognition. IEEE Transactions on Affective Computing, 1(1). doi:10.1109/TAFFC.2020.3014842.

Chen, Y., Chang, R., & Guo, J. (2021). Emotion Recognition of EEG Signals Based on the Ensemble Learning Method: AdaBoost. Mathematical Problems in Engineering, 2021, 1-12. doi:10.1155/2021/8896062.

Salankar, N., Mishra, P., & Garg, L. (2021). Emotion recognition from EEG signals using empirical mode decomposition and second-order difference plot. Biomedical Signal Processing and Control, 65, 102389. doi:10.1016/j.bspc.2020.102389.

Khateeb, M., Anwar, S. M., & Alnowami, M. (2021). Multi-Domain Feature Fusion for Emotion Classification Using DEAP Dataset. IEEE Access, 9, 12134–12142. doi:10.1109/ACCESS.2021.3051281.

Lang, P., & Bradley, M. M. (2007). The International Affective Picture System (IAPS) in the study of emotion and attention. Handbook of emotion elicitation and assessment, 29, 70-73, Oxford University Press, Oxford, United Kingdom.


Full Text: PDF

DOI: 10.28991/ESJ-2022-06-05-017

Refbacks

  • There are currently no refbacks.


Copyright (c) 2022 Chy Mohammed Tawsif Khan, Dr. Nor Azlina Binti Ab Aziz, Dr. Joseph Emerson Raja, Dr. Md. Jakir Hossen, Jesmeen Mohd Zebaral Hoque