Distilling knowledge for occlusion robust monocular 3D face reconstruction

Hitika Tiwari*, Vinod K. Kurmi, Venkatesh K. Subramanian, Yong Sheng Chen

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

1 Scopus citations

Abstract

Recently, there have been significant advancements in the 3D face reconstruction field, largely driven by monocular image-based deep learning methods. However, these methods still face challenges in reliable deployments due to their sensitivity to facial occlusions and inability to maintain identity consistency across different occlusions within the same facial image. To address these issues, we propose two frameworks: Distillation Assisted Mono Image Occlusion Robustification (DAMIOR) and Duplicate Images Assisted Multi Occlusions Robustification (DIAMOR). The DAMIOR framework leverages the knowledge from the Occlusion Frail Trainer (OFT) network to enhance robustness against facial occlusions. Our proposed method overcomes the sensitivity to occlusions and improves reconstruction accuracy. To tackle the issue of identity inconsistency, the DIAMOR framework utilizes the estimates from DAMIOR to mitigate inconsistencies in geometry and texture, collectively known as identity, of the reconstructed 3D faces. We evaluate the performance of DAMIOR on two variations of the CelebA test dataset: empirical occlusions and irrational occlusions. Furthermore, we analyze the performance of the proposed DIAMOR framework using the irrational occlusion-based variant of the CelebA test dataset. Our methods outperform state-of-the-art approaches by a significant margin. For example, DAMIOR reduces the 3D vertex-based shape error by 41.1% and the texture error by 21.8% for empirical occlusions. Besides, for facial data with irrational occlusions, DIAMOR achieves a substantial decrease in shape error by 42.5% and texture error by 30.5%. These results demonstrate the effectiveness of our proposed methods.

Original languageEnglish
Article number104763
JournalImage and Vision Computing
Volume137
DOIs
StatePublished - Sep 2023

Keywords

  • 3D face reconstruction
  • Distillation Assisted Mono Image Occlusion Robustification
  • Duplicate Images Assisted Multi Occlusions Robustification
  • Knowledge distillation
  • Occlusion robustness

Fingerprint

Dive into the research topics of 'Distilling knowledge for occlusion robust monocular 3D face reconstruction'. Together they form a unique fingerprint.

Cite this