Proactive robot task sequencing through real-time hand motion prediction in human–robot collaboration
Abilkassov S. Gentner M. Shintemirov A. Steinbach E. Popa M.
March 2025Elsevier Ltd
Image and Vision Computing
2025#155
Human–robot collaboration (HRC) is essential for improving productivity and safety across various industries. While reactive motion re-planning strategies are useful, there is a growing demand for proactive methods that predict human intentions to enable more efficient collaboration. This study addresses this need by introducing a framework that combines deep learning-based human hand trajectory forecasting with heuristic optimization for robotic task sequencing. The deep learning model advances real-time hand position forecasting using a multi-task learning loss to account for both hand positions and contact delay regression, achieving state-of-the-art performance on the Ego4D Future Hand Prediction benchmark. By integrating hand trajectory predictions into task planning, the framework offers a cohesive solution for HRC. To optimize task sequencing, the framework incorporates a Dynamic Variable Neighborhood Search (DynamicVNS) heuristic algorithm, which allows robots to pre-plan task sequences and avoid potential collisions with human hand positions. DynamicVNS provides significant computational advantages over the generalized VNS method. The framework was validated on a UR10e robot performing a visual inspection task in a HRC scenario, where the robot effectively anticipated and responded to human hand movements in a shared workspace. Experimental results highlight the systems effectiveness and potential to enhance HRC in industrial settings by combining predictive accuracy and task planning efficiency.
Egocentric vision , Human–robot collaboration
Text of the article Перейти на текст статьи
School of Engineering and Digital Sciences, Nazarbayev University, Astana, Kazakhstan
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
BMW AG, Munich, Germany
Department of Electrical Engineering and Automation, Aalto University, Espoo, Finland
Department of Advanced Computing Sciences, Faculty of Science and Engineering, Maastricht University, Maastricht, Netherlands
School of Engineering and Digital Sciences
School of Computation
BMW AG
Department of Electrical Engineering and Automation
Department of Advanced Computing Sciences
10 лет помогаем публиковать статьи Международный издатель
Книга Публикация научной статьи Волощук 2026 Book Publication of a scientific article 2026