Evaluation of Machine Learning Algorithms for Emotions Recognition using Electrocardiogram

Chy Mohammed Tawsif Khan, Nor Azlina Ab Aziz, J. Emerson Raja, Sophan Wahyudi Bin Nawawi, Pushpa Rani

Abstract


In recent studies, researchers have focused on using various modalities to recognize emotions for different applications. A major challenge is identifying emotions correctly with only electrocardiograms (ECG) as the modality. The main objective is to reduce costs by using single-modality ECG signals to predict human emotional states. This paper presents an emotion recognition approach utilizing the heart rate variability features obtained from ECG with feature selection techniques (exhaustive feature selection (EFS) and Pearson’s correlation) to train the classification models. Seven machine learning (ML) models: multi-layer perceptrons (MLP), Support Vector Machine (SVM), Decision Tree (DT), Gradient Boosting Decision Tree (GBDT), Logistic Regression, Adaboost and Extra Tree classifier are used to classify emotional state. Two public datasets, DREAMER and SWELL are used for evaluation. The results show that no particular ML works best for all data. For DREAMER with EFS, the best models to predict valence, arousal, and dominance are Extra Tree (74.6%), MLP and DT (74.6%), and GBDT and DT (69.8%), respectively. Extra tree with Pearson’s correlation are the best method for the ECG SWELL dataset and provide 100% accuracy. The usage of Extra tree classifier and feature selection technique contributes to the improvement of the model accuracy. Moreover, the Friedman test proved that ET is as good as other classification models for predicting human emotional state and ranks highest.

 

Doi: 10.28991/ESJ-2023-07-01-011

Full Text: PDF


Keywords


Electrocardiogram (ECG); Emotion Recognition System; Exhaustive Feature Selection; Gradient Boosting Decision Tree (GBDT); Machine Learning.

References


Picard, R. W. (1997). Affective Computing Cambridge. MIT Press, Cambridge, Massachusetts, United States. doi:10.1037/e526112012-054.

Izard, C. E. (2009). Emotion theory and research: Highlights, unanswered questions, and emerging issues. Annual Review of Psychology, 60(60), 1–25. doi:10.1146/annurev.psych.60.110707.163539.

D’Mello, S. K., & Kory, J. (2015). A review and meta-analysis of multimodal affect detection systems. ACM Computing Surveys, 47(3). doi:10.1145/2682899.

Turabzadeh, S., Meng, H., Swash, R., Pleva, M., & Juhar, J. (2018). Facial Expression Emotion Detection for Real-Time Embedded Systems. Technologies, 6(1), 17. doi:10.3390/technologies6010017.

Konar, A., & Chakraborty, A. (2015). Emotion recognition: A pattern analysis approach. John Wiley & Sons, Hoboken, United States. doi:10.1002/9781118910566.

Ali, M., Al Machot, F., Mosa, A. H., & Kyamakya, K. (2016). A novel EEG-based emotion recognition approach for e-healthcare applications. Proceedings of the 31st Annual ACM Symposium on Applied Computing. doi:10.1145/2851613.2851916.

Bhise, P. R., Kulkarni, S. B., & Aldhaheri, T. A. (2020). Brain Computer Interface based EEG for Emotion Recognition System: A Systematic Review. 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA). doi:10.1109/icimia48430.2020.9074921.

Kumar, A., Garg, N., & Kaur, G. (2019). An emotion recognition based on physiological signals. International Journal of Innovative Technology and Exploring Engineering, 8(9 Special Issue), 335–341. doi:10.35940/ijitee.I1054.0789S19.

Dissanayake, T., Rajapaksha, Y., Ragel, R., & Nawinne, I. (2019). An ensemble learning approach for electrocardiogram sensor based human emotion recognition. Sensors (Switzerland), 19(20), 4495. doi:10.3390/s19204495.

Ayata, D., Yaslan, Y., & Kamasak, M. E. (2020). Emotion Recognition from Multimodal Physiological Signals for Emotion Aware Healthcare Systems. Journal of Medical and Biological Engineering, 40(2), 149–157. doi:10.1007/s40846-019-00505-7.

Li, Z., Tian, X., Shu, L., Xu, X., Hu, B. (2018). Emotion Recognition from EEG Using RASM and LSTM. Internet Multimedia Computing and Service. ICIMCS 2017, Communications in Computer and Information Science, 819. Springer, Singapore. doi:10.1007/978-981-10-8530-7_30.

Nikolova, D., Petkova, P., Manolova, A., & Georgieva, P. (2018). ECG-based emotion recognition: Overview of methods and applications. ANNA'18; Advances in Neural Networks and Applications 2018, 15-17 September, 2018, St. Konstantin and Elena Resort, Bulgaria.

Katsis, C. D., Katertsidis, N., Ganiatsas, G., & Fotiadis, D. I. (2008). Toward emotion recognition in car-racing drivers: A biosignal processing approach. IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans, 38(3), 502–512. doi:10.1109/TSMCA.2008.918624.

Tivatansakul, S., & Ohkura, M. (2013). Healthcare System Focusing on Emotional Aspects Using Augmented Reality - Implementation of Breathing Control Application in Relaxation Service. 2013 International Conference on Biometrics and Kansei Engineering. doi:10.1109/icbake.2013.43.

Kołakowska, A., Landowska, A., Szwoch, M., Szwoch, W., & Wróbel, M. R. (2014). Emotion Recognition and Its Applications. Human-Computer Systems Interaction: Backgrounds and Applications 3, 51–62. doi:10.1007/978-3-319-08491-6_5.

Koldijk, S., Neerincx, M. A., & Kraaij, W. (2018). Detecting Work Stress in Offices by Combining Unobtrusive Sensors. IEEE Transactions on Affective Computing, 9(2), 227–239. doi:10.1109/TAFFC.2016.2610975.

Zhang, T., & Iyengar, V. S. (2002). Recommender Systems Using Linear Classifiers. Journal of Machine Learning Research, 2(3), 313–334. doi:10.1162/153244302760200641.

Garrett, D., Peterson, D. A., Anderson, C. W., & Thaut, M. H. (2003). Comparison of linear, nonlinear, and feature selection methods for EEG signal classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11(2), 141–144. doi:10.1109/tnsre.2003.814441.

Davidson, R. J., Ekman, P., Saron, C. D., Senulis, J. A., & Friesen, W. V. (1990). Approach-Withdrawal and Cerebral Asymmetry: Emotional Expression and Brain Physiology I. Journal of Personality and Social Psychology, 58(2), 330–341. doi:10.1037/0022-3514.58.2.330.

Yang, C., Lu, J., Wu, Q., & Chen, H. (2021). Research progress of speech emotion recognition based on discrete emotion model. Journal of Physics: Conference Series, 2010(1). doi:10.1088/1742-6596/2010/1/012110.

Zhang, Z., Song, Y., Cui, L., Liu, X., & Zhu, T. (2016). Emotion recognition based on customized smart bracelet with built-in accelerometer. PeerJ, 2016(7), 1–14,. doi:10.7717/peerj.2258.

Jang, E. H., Park, B. J., Park, M. S., Kim, S. H., & Sohn, J. H. (2015). Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. Journal of Physiological Anthropology, 34(1), 1–12. doi:10.1186/s40101-015-0063-5.

Cho, D., Ham, J., Oh, J., Park, J., Kim, S., Lee, N. K., & Lee, B. (2017). Detection of stress levels from bio-signals measured in virtual reality environments using a kernel-based extreme learning machine. Sensors (Switzerland), 17(10). doi:10.3390/s17102435.

Gravina, R., & Li, Q. (2019). Emotion-relevant activity recognition based on smart cushion using multi-sensor fusion. Information Fusion, 48, 1–10. doi:10.1016/j.inffus.2018.08.001.

Katsigiannis, S., & Ramzan, N. (2018). DREAMER: A Database for Emotion Recognition through EEG and ECG Signals from Wireless Low-cost Off-the-Shelf Devices. IEEE Journal of Biomedical and Health Informatics, 22(1), 98–107. doi:10.1109/JBHI.2017.2688239.

Koldijk, S., Sappelli, M., Verberne, S., Neerincx, M. A., & Kraaij, W. (2014). The SWELL Knowledge Work Dataset for Stress and User Modeling Research. Proceedings of the 16th International Conference on Multimodal Interaction, Istanbul, Turkey November 12 - 16. doi:10.1145/2663204.2663257.

Goshvarpour, A., Abbasi, A., & Goshvarpour, A. (2017). An accurate emotion recognition system using ECG and GSR signals and matching pursuit method. Biomedical Journal, 40(6), 355–368. doi:10.1016/j.bj.2017.11.001.

Mehmood, R. M., & Lee, H. J. (2015). Exploration of Prominent Frequency Wave in EEG Signals from Brain Sensors Network. International Journal of Distributed Sensor Networks, 2015(386057). doi:10.1155/2015/386057.

Chen, J., Hu, B., Wang, Y., Moore, P., Dai, Y., Feng, L., & Ding, Z. (2017). Subject-independent emotion recognition based on physiological signals: A three-stage decision method. BMC Medical Informatics and Decision Making, 17. doi:10.1186/s12911-017-0562-x.

Makowski, D., Pham, T., Lau, Z. J., Brammer, J. C., Lespinasse, F., Pham, H., Schölzel, C., & Chen, S. H. A. (2021). NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods, 53(4), 1689–1696. doi:10.3758/s13428-020-01516-y.

Shu, L., Yu, Y., Chen, W., Hua, H., Li, Q., Jin, J., & Xu, X. (2020). Wearable emotion recognition using heart rate data from a smart bracelet. Sensors (Switzerland), 20(3), 1–19. doi:10.3390/s20030718.

Bulagang, A. F., Mountstephens, J., & Teo, J. (2021). Multiclass emotion prediction using heart rate and virtual reality stimuli. Journal of Big Data, 8(1). doi:10.1186/s40537-020-00401-x.

El Attaoui, A., Hazmi, M., Jilbab, A., & Bourouhou, A. (2020). Wearable Wireless Sensors Network for ECG Telemonitoring Using Neural Network for Features Extraction. Wireless Personal Communications, 111(3), 1955–1976. doi:10.1007/s11277-019-06967-x.

Fotiadou, E., Xu, M., van Erp, B., van Sloun, R. J. G., & Vullings, R. (2020). Deep Convolutional Long Short-Term Memory Network for Fetal Heart Rate Extraction. 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). doi:10.1109/embc44109.2020.9175442.

Narendra, P. M., & Fukunaga, K. (1977). A Branch and Bound Algorithm for Feature Subset Selection. IEEE Transactions on Computers, C–26(9), 917–922. doi:10.1109/TC.1977.1674939.

Kumar, M., & Rath, S. K. (2016). Feature Selection and Classification of Microarray Data Using Machine Learning Techniques. Emerging Trends in Applications and Infrastructures for Computational Biology, Bioinformatics, and Systems Biology, 213–242. doi:10.1016/b978-0-12-804203-8.00015-8.

Ye, J., Chow, J.-H., Chen, J., & Zheng, Z. (2009). Stochastic gradient boosted distributed decision trees. Proceeding of the 18th ACM Conference on Information and Knowledge Management - CIKM ’09. doi:10.1145/1645953.1646301.

Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29(5), 1189–1232. doi:10.1214/aos/1013203451.

Geurts, P., Ernst, D., & Wehenkel, L. (2006). Extremely randomized trees. Machine Learning, 63(1), 3–42. doi:10.1007/s10994-006-6226-1.


Full Text: PDF

DOI: 10.28991/ESJ-2023-07-01-011

Refbacks

  • There are currently no refbacks.


Copyright (c) 2022 Chy Mohammed Tawsif khan, Dr. Nor Azlina Binti Ab Aziz, Dr. Joseph Emerson Raja, Dr Sophan Wahyudi Bin Nawawi, Pushpa Rani