HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar

Shih Po Lee, Niraj Prakash Kini, Wen-Hsiao Peng, Ching-Wen Ma, Jenq Neng Hwang

研究成果: Paper同行評審


This paper introduces a novel human pose estimation benchmark, Human Pose with Millimeter Wave Radar (HuPR), that includes synchronized vision and radio signal components. This dataset is created using cross-calibrated mmWave radar sensors and a monocular RGB camera for cross-modality training of radar-based human pose estimation. There are two advantages of using mmWave radar to perform human pose estimation. First, it is robust to dark and low-light conditions. Second, it is not visually perceivable by humans and therefore, can be widely applied to applications with privacy concerns, e.g., surveillance systems in patient rooms. In addition to the benchmark, we propose a cross-modality training framework that leverages the ground-truth 2D keypoints representing human body joints for training, which are systematically generated from the pre-trained 2D pose estimation network based on a monocular camera input image, avoiding laborious manual label annotation efforts. The framework consists of a new radar pre-processing method that better extracts the velocity information from radar data, Cross- and Self-Attention Module (CSAM), to fuse multi-scale radar features, and Pose Refinement Graph Convolutional Networks (PRGCN), to refine the predicted keypoint confidence heatmaps. Our intensive experiments on the HuPR benchmark show that the proposed scheme achieves better human pose estimation performance with only radar data, as compared to traditional pre-processing solutions and previous radio-frequency-based methods. Our proposed scheme further outperforms state-of-the-art pointcloud-based methods.
出版狀態Published - 2023
事件 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) - Waikoloa, United States
持續時間: 3 1月 20237 1月 2023


Conference IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
國家/地區United States


深入研究「HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar」主題。共同形成了獨特的指紋。