Spatiotemporal super-resolution with cross-task consistency and its semi-supervised extension

Han Yi Lin, Pi Cheng Hsiu, Tei Wei Kuo, Yen Yu Lin

研究成果: Conference contribution同行評審

摘要

Spatiotemporal super-resolution (SR) aims to upscale both the spatial and temporal dimensions of input videos, and produces videos with higher frame resolutions and rates. It involves two essential sub-tasks: spatial SR and temporal SR. We design a two-stream network for spatiotemporal SR in this work. One stream contains a temporal SR module followed by a spatial SR module, while the other stream has the same two modules in the reverse order. Based on the interchangeability of performing the two sub-tasks, the two network streams are supposed to produce consistent spatiotemporal SR results. Thus, we present a cross-stream consistency to enforce the similarity between the outputs of the two streams. In this way, the training of the two streams is correlated, which allows the two SR modules to share their supervisory signals and improve each other. In addition, the proposed cross-stream consistency does not consume labeled training data and can guide network training in an unsupervised manner. We leverage this property to carry out semi-supervised spatiotemporal SR. It turns out that our method makes the most of training data, and can derive an effective model with few high-resolution and high-frame-rate videos, achieving the state-of-the-art performance. The source code of this work is available at https://hankweb.github.io/STSRwithCrossTask/.

原文English
主出版物標題Proceedings of the 29th International Joint Conference on Artificial Intelligence, IJCAI 2020
編輯Christian Bessiere
發行者International Joint Conferences on Artificial Intelligence
頁面615-622
頁數8
ISBN(電子)9780999241165
出版狀態Published - 2020
事件29th International Joint Conference on Artificial Intelligence, IJCAI 2020 - Yokohama, Japan
持續時間: 1 一月 2021 → …

出版系列

名字IJCAI International Joint Conference on Artificial Intelligence
2021-January
ISSN(列印)1045-0823

Conference

Conference29th International Joint Conference on Artificial Intelligence, IJCAI 2020
國家/地區Japan
城市Yokohama
期間1/01/21 → …

引用此