Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow
Buribayev Z. Aouani M. Zhangabay Z. Yerkos A. Abdirazak Z. Zhassuzak M.
May 2025Multidisciplinary Digital Publishing Institute (MDPI)
Applied Sciences (Switzerland)
2025#15Issue 10
Sign languages are characterized by complex and subtle hand movements, which are challenging for computer vision systems to accurately recognize. This study suggests an innovative deep learning pipeline specifically designed for reliable gesture recognition of Kazakh Sign Language. This approach combines key point detection using the YOLO model, optical flow estimation, and a bidirectional long short-term memory (BiLSTM) network. At the initial stage, a dataset is generated using MediaPipe, which is then used to train the YOLO model in order to accurately identify key hand points. After training, the YOLO model extracts key points and bounding boxes from video recordings of gestures, creating consistent representations of movements. To improve the recognition of dynamic gestures, the optical flow is calculated in an area covering 10% of the area around key points, which allows the dynamics of movements to be captured and provides additional time characteristics. The BiLSTM network is trained on multimodal input that combines data on keypoints, bounding boxes, and optical flow, resulting in improved gesture classification accuracy. The experimental results demonstrate that the proposed approach is superior to traditional methods based solely on key points, especially in recognizing fast and complex gestures. The proposed structure promotes the development of sign language recognition technologies, especially for poorly studied languages such as Kazakh, paving the way to more inclusive and effective communication tools.
BiLSTM , gesture analysis , optical flow , sign language recognition , YOLO
Text of the article Перейти на текст статьи
Institute of Information and Computational Technologies, Almaty, 050010, Kazakhstan
Department of Computer Science, Al-Farabi Kazakh National University, Almaty, 050040, Kazakhstan
Institute of Information and Computational Technologies
Department of Computer Science
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026