Robust 3D skeleton tracking based on openpose and a probabilistic tracking framework

For human skeleton tracking, conventional methods may directly rely on the Kinect built-in function to extract skeleton joints and realize the tracking function. However, the left-right confusion problem and the self-occlusion problem would make the Kinect joints unstable and incorrect. In this paper, we aim to solve these crucial issues and realize a reliable 3D skeleton tracking system via multiple Kinect cameras. First, the proposed method corrected the Kinect skeleton by referring to the OpenPose-extracted joints. Since OpenPose extracts the joints by analyzing image content, we can differentiate the front side and the back side and thus can correct the left-right confusion. Second, to alleviate the self-occlusion problem, we back-project the 2D OpenPose joints from multiple cameras and form an alternative 3D skeleton. The OpenPose reconstructed joints are treated as robust 3D anchors for multiple skeleton fusion. Last but not least, we introduce the inter-joint constraints into our skeleton tracking framework so that we can trace all joints simultaneously, make sure the skeleton movement consistent, and well maintain the length between neighboring joints. We evaluate our method with challenging actions. A practical system is also built for demonstration. The experimental results show the system can track skeleton stably without error propagation and vibration. The average localization error is also smaller than conventional methods.