Development of Bimodal Emotion Recognition System Based on Skin Temperature and Heart Rate Variability Using Hybrid Neural Networks
Orynbassar S. Erol Barkana D. Yershov E. Nurgaliyev M. Saymbetov A. Zholamanov B. Dosymbetova G. Kapparova A. Koshkarbay N. Kuttybay N. Bolatbek A. Kopbay K. Almen D.
2025Institute of Electrical and Electronics Engineers Inc.
IEEE Access
2025#1388215 - 88229 pp.
Most studies indicate that bimodal emotion recognition systems are more objective and accurate. However, many of these systems depend on physiological signals that require direct measurement, which introduces certain limitations. This study aims to develop a new bimodal emotion recognition system based on skin temperature (SKT) and heart rate variability (HRV) using hybrid neural networks. Notably, these physiological signals can be measured remotely, addressing the limitations of direct measurement methods. The integration of these modalities enables the model to effectively utilize both spatial and temporal features for robust emotion classification. The hybrid neural network, combining a convolutional neural network and a gated recurrent unit (CNN+GRU), was trained on experimental SKT and HRV data collected from individuals experiencing basic emotions such as anger, disgust, fear, happiness, sadness, and surprise. The trained model achieved an accuracy of 95.58%, outperforming existing approaches that use only a single data modality. Confusion matrix analysis demonstrated high accuracy in recognizing most basic emotions. The results confirm the effectiveness of the proposed approach in combining physiological and visual signals for improved emotion recognition.
CNN , Emotion recognition , heart rate variability , RNN , thermographic images
Text of the article Перейти на текст статьи
Al-Farabi Kazakh National University, Faculty of Physics and Technology, Almaty, 050040, Kazakhstan
Yeditepe Üniversitesi, Faculty of Engineering, Department of Electrical and Electronics Engineering, İstanbul, 34755, Turkey
Al-Farabi Kazakh National University
Yeditepe Üniversitesi
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026