CA-FER: Mitigating Spurious Correlation With Counterfactual Attention in Facial Expression Recognition

Pin Jui Huang, Hongxia Xie, Hung Cheng Huang, Hong Han Shuai, Wen-Huang Cheng

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Although facial expression recognition based on deep learning has become a major trend, existing methods have been found to prefer learning spurious statistical correlations and non-robust features during training. This degenerates the model's generalizability in practical situations. One of the research fields mitigating such misperception of correlations as causality is causal reasoning. In this paper, we propose a learnable counterfactual attention mechanism, CA-FER, that uses causal reasoning to simultaneously optimize feature discrimination and diversity to mitigate spurious correlations in expression datasets. To the best of our knowledge, this is the first work to study the spurious correlations in facial expression recognition from a counterfactual attention perspective. Extensive experiments on a synthetic dataset and four public datasets demonstrate that our method outperforms previous methods, which shows the effectiveness and generalizability of our learnable counterfactual attention mechanism for the expression recognition task.

Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalIEEE Transactions on Affective Computing
StateAccepted/In press - 2023


  • Causal learning
  • Computational modeling
  • Correlation
  • Deep learning
  • Face recognition
  • Feature extraction
  • Image color analysis
  • Training
  • facial expression recognition
  • spurious correlation


Dive into the research topics of 'CA-FER: Mitigating Spurious Correlation With Counterfactual Attention in Facial Expression Recognition'. Together they form a unique fingerprint.

Cite this