Facial Image Reconstruction from Functional Magnetic Resonance Imaging via GAN Inversion with Improved Attribute Consistency

Pei Chun Chang, Yan Yu Tien, Chia Lin Chen, Li Fen Chen, Yong Sheng Chen, Hui Ling Chan

研究成果: Conference contribution同行評審

3 引文 斯高帕斯(Scopus)

摘要

Neuroscience studies have revealed that the brain encodes visual content and embeds information in neural activity. Recently, deep learning techniques have facilitated attempts to address visual reconstructions by mapping brain activity to image stimuli using generative adversarial networks (GANs). However, none of these studies have considered the semantic meaning of latent code in image space. Omitting semantic information could potentially limit the performance. In this study, we propose a new framework to reconstruct facial images from functional Magnetic Resonance Imaging (fMRI) data. With this framework, the GAN inversion is first applied to train an image encoder to extract latent codes in image space, which are then bridged to fMRI data using linear transformation. Following the attributes identified from fMRI data using an attribute classifier, the direction in which to manipulate attributes is decided and the attribute manipulator adjusts the latent code to improve the consistency between the seen image and the reconstructed image. Our experimental results suggest that the proposed framework accomplishes two goals: (1) reconstructing clear facial images from fMRI data and (2) maintaining the consistency of semantic characteristics.

原文English
主出版物標題2022 International Joint Conference on Neural Networks, IJCNN 2022 - Proceedings
發行者Institute of Electrical and Electronics Engineers Inc.
ISBN(電子)9781728186719
DOIs
出版狀態Published - 2022
事件2022 International Joint Conference on Neural Networks, IJCNN 2022 - Padua, Italy
持續時間: 18 7月 202223 7月 2022

出版系列

名字Proceedings of the International Joint Conference on Neural Networks
2022-July

Conference

Conference2022 International Joint Conference on Neural Networks, IJCNN 2022
國家/地區Italy
城市Padua
期間18/07/2223/07/22

指紋

深入研究「Facial Image Reconstruction from Functional Magnetic Resonance Imaging via GAN Inversion with Improved Attribute Consistency」主題。共同形成了獨特的指紋。

引用此