Mixed Reality Robot Programming Using Hand Gestures

In this paper, we present a hand gesture-based robot path generation system using mixed reality (MR) for interactive robot programming. A hand-gesture recognition scheme is proposed to recognize specific gestures for path generation and modification. A user can intuitively plan a virtual path by hand gestures in the actual working space and send the planned path to the robot controller to execute through the MR glasses. A human-machine interface is designed to allow a user to assign the required task and send control commands to the robot. To ensure the robot can accurately move along the planned path, a calibration procedure is designed to establish the relationship between the MR system and robot navigation system. The proposed method has been realized and verified using a TM5 cobot and HoloLens II. The experimental results show that the robot navigation error is within 2mm using the developed MR-based robot programming system.