1. Home
  2. Vol 7, No 1 (2023)
  3. Wirawan

Continuous Capsule Network Method for Improving Electroencephalogram-Based Emotion Recognition

I Made Agus Wirawan, Retantyo Wardoyo, Danang Lelono, Sri Kusrohmaniah

Abstract


The convolution process in the Capsule Network method can result in a loss of spatial data from the Electroencephalogram signal, despite its ability to characterize spatial information from Electroencephalogram signals. Therefore, this study applied the Continuous Capsule Network method to overcome problems associated with emotion recognition based on Electroencephalogram signals using the optimal architecture of the (1) 1st, 2nd, 3rd, and 4th Continuous Convolution layers with values of 64, 128, 256, and 64, respectively, and (2) kernel sizes of 2×2×4, 2×2×64, and 2×2×128 for the 1st, 2nd, and 3rd Continuous Convolution layers, and 1×1×256 for the 4th. Several methods were also used to support the Continuous Capsule Network process, such as the Differential Entropy and 3D Cube methods for the feature extraction and representation processes. These methods were chosen based on their ability to characterize spatial and low-frequency information from Electroencephalogram signals. By testing the DEAP dataset, these proposed methods achieved accuracies of 91.35, 93.67, and 92.82% for the four categories of emotions, two categories of arousal, and valence, respectively. Furthermore, on the DREAMER dataset, these proposed methods achieved accuracies of 94.23, 96.66, and 96.05% for the four categories of emotions, the two categories of arousal, and valence, respectively. Finally, on the AMIGOS dataset, these proposed methods achieved accuracies of 96.20, 97.96, and 97.32% for the four categories of emotions, the two categories of arousal, and valence, respectively.

 

Doi: 10.28991/ESJ-2023-07-01-09

Full Text: PDF


Keywords


Electroencephalogram; Emotion Recognition; Differential Entropy; Baseline Reduction; 3D Cube; Capsule Network; Continuous Convolution.

References


Chao, H., Dong, L., Liu, Y., & Lu, B. (2019). Emotion recognition from multiband EEG signals using CAPSNET. Sensors (Switzerland), 19(9), 2212. doi:10.3390/s19092212.

Murali Krishna, N., Sekaran, K., Vamsi, A. V. N., Pradeep Ghantasala, G. S., Chandana, P., Kadry, S., Blazauskas, T., & Damasevicius, R. (2019). An Efficient Mixture Model Approach in Brain-Machine Interface Systems for Extracting the Psychological Status of Mentally Impaired Persons Using EEG Signals. IEEE Access, 7, 77905–77914. doi:10.1109/ACCESS.2019.2922047.

Scherer, K. R., Schorr, A., & Johnstone, T. (2001). Appraisal processes in emotion: Theory, methods, research. Oxford University Press, Oxford, United Kingdom.

Kort, B., Reilly, R., & Picard, R. W. (2001). An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion. Proceedings IEEE International Conference on Advanced Learning Technologies. doi:10.1109/icalt.2001.943850.

Gottardo, E., Pimentel, A.R. (2019). Inferring Students’ Emotions Using a Hybrid Approach that Combine Cognitive and Physical Data. Enterprise Information Systems, ICEIS 2018. Lecture Notes in Business Information Processing, 363, Springer, Cham, Switzerland. doi:10.1007/978-3-030-26169-6_14.

Shu, L., Xie, J., Yang, M., Li, Z., Li, Z., Liao, D., Xu, X., & Yang, X. (2018). A review of emotion recognition using physiological signals. Sensors (Switzerland), 18(7). doi:10.3390/s18072074.

Ekman, P., Friesen, W. V., & Simons, R. C. (1985). Is the startle reaction an emotion?. Journal of personality and social psychology, 49(5), 1416, doi:10.1037/0022-3514.49.5.1416.

Li, Y., Huang, J., Zhou, H., & Zhong, N. (2017). Human emotion recognition with electroencephalographic multidimensional features by hybrid deep neural networks. Applied Sciences (Switzerland), 7(10). doi:10.3390/app7101060.

Hu, X., Chen, J., Wang, F., & Zhang, D. (2019). Ten challenges for EEG-based affective computing. Brain Science Advances, 5(1), 1–20. doi:10.1177/2096595819896200.

Song, T., Zheng, W., Song, P., & Cui, Z. (2020). EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks. IEEE Transactions on Affective Computing, 11(3), 532–541. doi:10.1109/TAFFC.2018.2817622.

Kawintiranon, K., Buatong, Y., & Vateekul, P. (2016). Online music emotion prediction on multiple sessions of EEG data using SVM. 13th International Joint Conference on Computer Science and Software Engineering. doi:10.1109/JCSSE.2016.7748921.

Wirawan, I.M.A., Wardoyo, R., & Lelono, D. (2022). The challenges of emotion recognition methods based on electroencephalogram signals: A literature review. International Journal of Electrical and Computer Engineering, 12(2), 1508–1519. doi:10.11591/ijece.v12i2.pp1508-1519.

Bhandari, N. K., & Jain, M. (2020). Emotion recognition and classification using EEG: A review. International Journal of Scientific and Technology Research, 9(2), 1827–1836.

Zhang, J., Yin, Z., Chen, P., & Nichele, S. (2020). Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Information Fusion, 59(January), 103–126. doi:10.1016/j.inffus.2020.01.011.

Pane, E. S., Wibawa, A. D., & Pumomo, M. H. (2018, November). Channel selection of EEG emotion recognition using stepwise discriminant analysis. 2018 International Conference on Computer Engineering, Network and Intelligent Multimedia (CENIM). doi:10.1109/CENIM.2018.8711196.

Chen, D. W., Miao, R., Yang, W. Q., Liang, Y., Chen, H. H., Huang, L., Deng, C. J., & Han, N. (2019). A feature extraction method based on differential entropy and linear discriminant analysis for emotion recognition. Sensors (Switzerland), 19(7), 1631. doi:10.3390/s19071631.

Li, J., Zhang, Z., & He, H. (2018). Hierarchical Convolutional Neural Networks for EEG-Based Emotion Recognition. Cognitive Computation, 10(2), 368–380. doi:10.1007/s12559-017-9533-x.

Yang, Y., Wu, Q., Fu, Y., & Chen, X. (2018). Continuous convolutional neural network with 3D input for EEG-based emotion recognition. International Conference on Neural Information Processing. doi:10.1007/978-3-030-04239-4_39.

Jiang, H., & Jia, J. (2019, November). Research on EEG emotional recognition based on LSTM. International Conference on Bio-Inspired Computing: Theories and Applications. doi:10.1007/978-981-15-3415-7_34.

Wardoyo, R., Wirawan, I. M. A., & Pradipta, I. G. A. (2022). Oversampling Approach Using Radius-SMOTE for Imbalance Electroencephalography Datasets. Emerging Science Journal, 6(2), 382–398. doi:10.28991/ESJ-2022-06-02-013.

Wirawan, I. M. A., Wardoyo, R., Lelono, D., Kusrohmaniah, S., & Asrori, S. (2021). Comparison of Baseline Reduction Methods for Emotion Recognition Based On Electroencephalogram Signals. 2021 6th International Conference on Informatics and Computing (ICIC). doi:10.1109/ICIC54025.2021.9632948.

Sabour, S., Frosst, N., & Hinton, G. E. (2017). Dynamic routing between capsules. Advances in Neural Information Processing Systems 30, NeurIPS Proceedings, 4-9 December, CA, United States.

Liu, Y., Ding, Y., Li, C., Cheng, J., Song, R., Wan, F., & Chen, X. (2020). Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network. Computers in Biology and Medicine, 123(March), 103927. doi:10.1016/j.compbiomed.2020.103927.

Tyng, C. M., Amin, H. U., Saad, M. N. M., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8(AUG). doi:10.3389/fpsyg.2017.01454.

Cheng, J., Chen, M., Li, C., Liu, Y., Song, R., Liu, A., & Chen, X. (2021). Emotion Recognition from Multi-Channel EEG via Deep Forest. IEEE Journal of Biomedical and Health Informatics, 25(2), 453–464. doi:10.1109/JBHI.2020.2995767.

Chen, J., Jiang, D., Zhang, Y., & Zhang, P. (2020). Emotion recognition from spatiotemporal EEG representations with hybrid convolutional recurrent neural networks via wearable multi-channel headset. Computer Communications, 154(6), 58–65. doi:10.1016/j.comcom.2020.02.051.

Liu, N., Fang, Y., Li, L., Hou, L., Yang, F., & Guo, Y. (2018, April). Multiple feature fusion for automatic emotion recognition using EEG signals. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). doi:10.1109/ICASSP.2018.8462518.

Zangeneh Soroush, M., Maghooli, K., Setarehdan, S. K., & Nasrabadi, A. M. (2019). A novel EEG-based approach to classify emotions through phase space dynamics. Signal, Image and Video Processing. doi:10.1007/s11760-019-01455-y.

Zheng, W. L., Zhu, J. Y., & Lu, B. L. (2019). Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, 10(3), 417–429. doi:10.1109/TAFFC.2017.2712143.

Mei, H., & Xu, X. (2017, December). EEG-based emotion classification using convolutional neural network. In 2017 International Conference on Security, Pattern Analysis, and Cybernetics (SPAC). Doi:10.1109/SPAC.2017.8304263.

Zhao, Y., Yang, J., Lin, J., Yu, D., & Cao, X. (2020). A 3D Convolutional Neural Network for Emotion Recognition based on EEG Signals. Proceedings of the International Joint Conference on Neural Networks. doi:10.1109/IJCNN48605.2020.9207420.

Liu, Z. T., Xie, Q., Wu, M., Cao, W. H., Li, D. Y., & Li, S. H. (2019). Electroencephalogram Emotion Recognition Based on Empirical Mode Decomposition and Optimal Feature Selection. IEEE Transactions on Cognitive and Developmental Systems, 11(4), 517–526. doi:10.1109/TCDS.2018.2868121.

He, Y., Ai, Q., & Chen, K. (2017). A MEMD method of human emotion recognition based on valence-Arousal model. Proceedings - 9th International Conference on Intelligent Human-Machine Systems and Cybernetics, IHMSC 2017, 2, 399–402. doi:10.1109/IHMSC.2017.201.

Parui, S., Bajiya, A. K. R., Samanta, D., & Chakravorty, N. (2019). Emotion recognition from EEG signal using XGBoost algorithm. 2019 IEEE 16th India Council International Conference (INDICON). doi:10.1109/INDICON47234.2019.9028978.

Pan, C., Shi, C., Mu, H., Li, J., & Gao, X. (2020). EEG-based emotion recognition using logistic regression with gaussian kernel and laplacian prior and investigation of critical frequency bands. Applied Sciences (Switzerland), 10(5), 1–24. doi:10.3390/app10051619.

Garg, D., & Verma, G. K. (2020). Emotion Recognition in Valence-Arousal Space from Multi-channel EEG data and Wavelet based Deep Learning Framework. Procedia Computer Science, 171, 857–867. doi:10.1016/j.procs.2020.04.093.

Huang, Y. D., Wang, K. Y., Ho, Y. L., He, C. Y., & Fang, W. C. (2019). An edge AI system-on-chip design with customized convolutional-neural-network architecture for real-time EEG-based affective computing system. 2019 IEEE Biomedical Circuits and Systems Conference (BioCAS). doi:10.1109/BIOCAS.2019.8919038.

Koelstra, S., Mühl, C., Soleymani, M., Lee, J. S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012). DEAP: A database for emotion analysis; using physiological signals. IEEE Transactions on Affective Computing, 3(1), 18–31. doi:10.1109/T-AFFC.2011.15.

Katsigiannis, S., & Ramzan, N. (2018). DREAMER: A Database for Emotion Recognition through EEG and ECG Signals from Wireless Low-cost Off-the-Shelf Devices. IEEE Journal of Biomedical and Health Informatics, 22(1), 98–107. doi:10.1109/JBHI.2017.2688239.

Miranda-Correa, J. A., Abadi, M. K., Sebe, N., & Patras, I. (2021). AMIGOS: A Dataset for Affect, Personality and Mood Research on Individuals and Groups. IEEE Transactions on Affective Computing, 12(2), 479–493. doi:10.1109/TAFFC.2018.2884461.

Eldor, T. (2018). Capsule Neural Networks - Part2: What is a Capsule?. Towards Data Science. Available online: https://towardsdatascience.com/capsule-neural-networks-part-2-what-is-a-capsule-846d5418929f (accessed on August 2022).

Wirawan, I. M. A., & Darmawiguna, I. G. M. (2017). New Concept of Learning Outcomes Assessment in Adaptive Mobile Learning. 2nd International Conference on Innovative Research across Disciplines (ICIRAD 2017). doi:10.2991/icirad-17.2017.49.

Al-Nafjan, A., Hosny, M., Al-Ohali, Y., & Al-Wabil, A. (2017). Review and classification of emotion recognition based on EEG brain-computer interface system research: A systematic review. Applied Sciences (Switzerland), 7(12), 1239. doi:10.3390/app7121239.


Full Text: PDF

DOI: 10.28991/ESJ-2023-07-01-09

Refbacks

  • There are currently no refbacks.


Copyright (c) 2022 I Made Agus Wirawan, Retantyo Wardoyo, Danang Lelono, Sri Kusrohmaniah