Speakingfaces: A large-scale multimodal dataset of voice commands with visual and thermal video streams
Abdrakhmanova M. Kuzdeuov A. Jarju S. Khassanov Y. Lewis M. Varol H.A.
2 May 2021MDPI AG
Sensors
2021#21Issue 10
We present SpeakingFaces as a publicly-available large-scale multimodal dataset developed to support machine learning research in contexts that utilize a combination of thermal, visual, and audio data streams; examples include human–computer interaction, biometric authentication, recognition systems, domain transfer, and speech recognition. SpeakingFaces is comprised of aligned high-resolution thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking approximately 100 imperative phrases. Data were collected from 142 subjects, yielding over 13,000 instances of synchronized data (∼3.8 TB). For technical validation, we demonstrate two baseline examples. The first baseline shows classification by gender, utilizing different combinations of the three data streams in both clean and noisy environments. The second example consists of thermal-to-visual facial image translation, as an instance of domain transfer.
Computer vision , Datasets , Domain transfer , Human–computer interaction , Multimodal learning , Thermal imaging
Text of the article Перейти на текст статьи
Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, Nur-Sultan, 010000, Kazakhstan
School of Engineering and Digital Sciences, Nazarbayev University, Nur-Sultan, 010000, Kazakhstan
Institute of Smart Systems and Artificial Intelligence
School of Engineering and Digital Sciences
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026