MPVF: 4D Medical Image Inpainting by Multi-Pyramid Voxel Flows

Tzu Ti Wei, Chin Kuo, Yu Chee Tseng, Jen Jee Chen

Research output: Contribution to journalArticlepeer-review


Generating a detailed 4D medical image usually accompanies with prolonged examination time and increased radiation exposure risk. Modern deep learning solutions have exploited interpolation mechanisms to generate a complete 4D image with fewer 3D volumes. However, existing solutions focus more on 2D-slice information, thus missing the changes on the <inline-formula><tex-math notation="LaTeX">$z$</tex-math></inline-formula>-axis. This paper tackles the 4D cardiac and lung image interpolation problem by synthesizing 3D volumes directly. Although heart and lung only account for a fraction of chest, they constantly undergo periodical motions of varying magnitudes in contrast to the rest of the chest volume, which is more stationary. This poses big challenges to existing models. In order to handle various magnitudes of motions, we propose a <italic>Multi-Pyramid Voxel Flows (MPVF)</italic> model that takes multiple multi-scale voxel flows into account. This renders our generation network rich information during interpolation, both globally and regionally. Focusing on periodic medical imaging, MPVF takes the maximal and the minimal phases of an organ motion cycle as inputs and can restore a 3D volume at any time point in between. MPVF is featured by a Bilateral Voxel Flow (BVF) module for generating multi-pyramid voxel flows in an unsupervised manner and a Pyramid Fusion (PyFu) module for fusing multiple pyramids of 3D volumes. The model is validated to outperform the state-of-the-art model in several indices with significantly less synthesis time. The code and models are available at <uri></uri>

Original languageEnglish
Pages (from-to)1-11
Number of pages11
JournalIEEE Journal of Biomedical and Health Informatics
StateAccepted/In press - 2023


  • 4D image
  • Bioinformatics
  • Biomedical imaging
  • Heart
  • Imaging
  • Interpolation
  • Lung
  • medical image inpainting
  • medical imaging
  • neural network
  • Three-dimensional displays
  • video interpolation
  • voxel flow estimation


Dive into the research topics of 'MPVF: 4D Medical Image Inpainting by Multi-Pyramid Voxel Flows'. Together they form a unique fingerprint.

Cite this