Cross-Resolution Flow Propagation for Foveated Video Super-Resolution

Eugene Lee*, Lien Feng Hsu, Evan Chen, Chen Yi Lee

*此作品的通信作者

研究成果: Conference contribution同行評審

4 引文 斯高帕斯(Scopus)

摘要

The demand of high-resolution video contents has grown over the years. However, the delivery of high-resolution video is constrained by either computational resources required for rendering or network bandwidth for remote transmission. To remedy this limitation, we leverage the eye trackers found alongside existing augmented and virtual reality headsets. We propose the application of video super-resolution (VSR) technique to fuse low-resolution context with regional high-resolution context for resource-constrained consumption of high-resolution content without perceivable drop in quality. Eye trackers provide us the gaze direction of a user, aiding us in the extraction of the regional high-resolution context. As only pixels that falls within the gaze region can be resolved by the human eye, a large amount of the delivered content is redundant as we can't perceive the difference in quality of the region beyond the observed region. To generate a visually pleasing frame from the fusion of high-resolution region and low-resolution region, we study the capability of a deep neural network of transferring the context of the observed region to other regions (low-resolution) of the current and future frames. We label this task a Foveated Video Super-Resolution (FVSR), as we need to super-resolve the low-resolution regions of current and future frames through the fusion of pixels from the gaze region. We propose Cross-Resolution Flow Propagation (CRFP) for FVSR. We train and evaluate CRFP on REDS dataset on the task of 8× FVSR, i.e. a combination of 8× VSR and the fusion of foveated region. Departing from the conventional evaluation of per frame quality using SSIM or PSNR, we propose the evaluation of past foveated region, measuring the capability of a model to leverage the noise present in eye trackers during FVSR. Code is made available at https://github.com/eugenelet/CRFP.

原文English
主出版物標題Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023
發行者Institute of Electrical and Electronics Engineers Inc.
頁面1766-1775
頁數10
ISBN(電子)9781665493468
DOIs
出版狀態Published - 2023
事件23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023 - Waikoloa, United States
持續時間: 3 1月 20237 1月 2023

出版系列

名字Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision, WACV 2023

Conference

Conference23rd IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023
國家/地區United States
城市Waikoloa
期間3/01/237/01/23

指紋

深入研究「Cross-Resolution Flow Propagation for Foveated Video Super-Resolution」主題。共同形成了獨特的指紋。

引用此