Low-Resolution Thermal Sensor-Guided Image Synthesis

Sheng Yang Chiu, Yu Chee Tseng, Jen Jee Chen

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

5 Scopus citations

Abstract

Thermopile array sensors are cost-effective thermal imaging alternatives and are less vulnerable to privacy intrusion, light conditions, and obtrusiveness. While numerous occupant surveillance systems have been developed based on such sensors, low spatial resolutions prohibit them from deriving more sophisticated applications. To help relieve the limitation, we propose to enrich thermopile array sensors with additional non-thermal features and develop, to the best of our knowledge, the first low-resolution thermal-guided image synthesis model capable of producing realistic and attribute-aligned color images. These thermal heatmaps are regarded as semantic maps, but have very low resolutions. We propose an extension of SPADE (Spatially-Adaptive Denormalization), namely SPADE-SR, to incorporate the spatial property of a thermal heatmap into a conditional GAN through iterative Self-Resampling. Compared to SPADE, SPADE-SR yields better results in terms of image quality and reconstruction error while using significantly fewer model parameters. A new LRT-Human (Low-Resolution Thermal Human) dataset comprised of 22k (thermal heatmap, RGB image) pairs with various thermal and non-thermal coupling is derived to support our claims. Our work explores the cross-thermal-RGB modality paradigm and poses a great opportunity for thermopile array sensors in surveillance usages.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, WACVW 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages60-69
Number of pages10
ISBN (Electronic)9798350320565
DOIs
StatePublished - 2023
Event2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, WACVW 2023 - Waikoloa, United States
Duration: 3 Jan 20237 Jan 2023

Publication series

NameProceedings - 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, WACVW 2023

Conference

Conference2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, WACVW 2023
Country/TerritoryUnited States
CityWaikoloa
Period3/01/237/01/23

Fingerprint

Dive into the research topics of 'Low-Resolution Thermal Sensor-Guided Image Synthesis'. Together they form a unique fingerprint.

Cite this