MUGAN: Thermal Infrared Image Colorization Using Mixed-Skipping UNet and Generative Adversarial Network

Hangying Liao, Qian Jiang, Xin Jin, Ling Liu, Lin Liu, Shin Jye Lee, Wei Zhou

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

Common cameras cannot capture high quality image in the night or extreme weather conditions that without enough light, while thermal infrared(TIR) cameras are not limited in this situation. Hence, TIR imaging technique is widely used in military, surveillance, nighttime traffic and other scenarios. However, TIR images are monochromatic, and the majority of details of such images are lost, which are difficult for human or computer system to analyze. Translating TIR images into visible images is beneficial to subsequent observation or further processing. Though there are some advances to realize the transformation from TIR images to color visible images, edge distortion and semantic confusion remain to be solved. Therefore, we propose a Mixed-Skipping UNet(MS-UNet) based image colorization model joint Generative Adversarial Network, which is denoted by MUGAN. Firstly, the dense skip connections of UNet++ and full-scale skip connections of UNet 3+ are combined to form the MS-UNet, which is regarded as the generator. In addition, we design a feature extraction module in MS-UNet to effectively capture the multi-scale features in source image. Then, a novel attention mechanism module is designed for decoding stage which can help decoder of MS-UNet to focus on the important information. Moreover, we explore the effect of different loss functions in TIR image colorization task, and the loss function with excellent performance is selected to further optimize the training process of the model. Extensive experiments prove the superiority of our method in the task of TIR image colorization. The code is available at <uri>https://github.com/HangyingLiao/MUGAN</uri>.

Original languageEnglish
Pages (from-to)1-16
Number of pages16
JournalIEEE Transactions on Intelligent Vehicles
DOIs
StateAccepted/In press - 2022

Keywords

  • Attention mechanism
  • Cameras
  • deep learing
  • Feature extraction
  • generative adversarial network
  • Gray-scale
  • Image color analysis
  • Image colorization
  • Intelligent vehicles
  • Task analysis
  • Thermal infrared imaging
  • UNet
  • Visualization

Fingerprint

Dive into the research topics of 'MUGAN: Thermal Infrared Image Colorization Using Mixed-Skipping UNet and Generative Adversarial Network'. Together they form a unique fingerprint.

Cite this