Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report

Andrey Ignatov; Radu Timofte; Cheng Ming Chiang; Hsien Kai Kuo; Yu Syuan Xu; Man Yu Lee; Allen Lu; Chia Ming Cheng; Chih Cheng Chen; Jia Ying Yong; Hong Han Shuai; Wen Huang Cheng; Zhuang Jia; Tianyu Xu; Yijian Zhang; Long Bao; Heng Sun; Diankai Zhang; Si Gao; Shaoli Liu; Biao Wu; Xiaofeng Zhang; Chengjian Zheng; Kaidi Lu; Ning Wang; Xiao Sun; Hao Dong Wu; Xuncheng Liu; Weizhan Zhang; Caixia Yan; Haipeng Du; Qinghua Zheng; Qi Wang; Wangdu Chen; Ran Duan; Mengdi Sun; Dan Zhu; Guannan Chen; Hojin Cho; Steve Kim; Shijie Yue; Chenghua Li; Zhengyang Zhuge; Wei Chen; Wenxu Wang; Yufeng Zhou; Xiaochen Cai; Hengxing Cai; Kele Xu; Li Liu; Zehua Cheng; Wenyi Lian; Wenjing Lian

doi:10.1007/978-3-031-25066-8_6

Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report

Andrey Ignatov^*, Radu Timofte, Cheng Ming Chiang, Hsien Kai Kuo, Yu Syuan Xu, Man Yu Lee, Allen Lu, Chia Ming Cheng, Chih Cheng Chen, Jia Ying Yong, Hong Han Shuai, Wen Huang Cheng, Zhuang Jia, Tianyu Xu, Yijian Zhang, Long Bao, Heng Sun, Diankai Zhang, Si Gao, Shaoli LiuBiao Wu, Xiaofeng Zhang, Chengjian Zheng, Kaidi Lu, Ning Wang, Xiao Sun, Hao Dong Wu, Xuncheng Liu, Weizhan Zhang, Caixia Yan, Haipeng Du, Qinghua Zheng, Qi Wang, Wangdu Chen, Ran Duan, Mengdi Sun, Dan Zhu, Guannan Chen, Hojin Cho, Steve Kim, Shijie Yue, Chenghua Li, Zhengyang Zhuge, Wei Chen, Wenxu Wang, Yufeng Zhou, Xiaochen Cai, Hengxing Cai, Kele Xu, Li Liu, Zehua Cheng, Wenyi Lian, Wenjing Lian

^*此作品的通信作者

研究成果: Conference contribution › 同行評審

摘要

Video super-resolution is one of the most popular tasks on mobile devices, being widely used for an automatic improvement of low-bitrate and low-resolution video streams. While numerous solutions have been proposed for this problem, they are usually quite computationally demanding, demonstrating low FPS rates and power efficiency on mobile devices. In this Mobile AI challenge, we address this problem and propose the participants to design an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption. The participants were provided with the REDS training dataset containing video sequences for a 4X video upscaling task. The runtime and power efficiency of all models was evaluated on the powerful MediaTek Dimensity 9000 platform with a dedicated AI processing unit capable of accelerating floating-point and quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 500 FPS rate and 0.2 [Watt/30 FPS] power consumption. A detailed description of all models developed in the challenge is provided in this paper.

原文	English
主出版物標題	Computer Vision – ECCV 2022 Workshops, Proceedings
編輯	Leonid Karlinsky, Tomer Michaeli, Ko Nishino
發行者	Springer Science and Business Media Deutschland GmbH
頁面	130-152
頁數	23
ISBN（列印）	9783031250651
DOIs	https://doi.org/10.1007/978-3-031-25066-8_6
出版狀態	Published - 2023
事件	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, 以色列持續時間: 23 10月 2022 → 27 10月 2022

出版系列

名字	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
卷	13803 LNCS
ISSN（列印）	0302-9743
ISSN（電子）	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
國家/地區	以色列
城市	Tel Aviv
期間	23/10/22 → 27/10/22

UN SDG

此研究成果有助於以下永續發展目標

存取文件

10.1007/978-3-031-25066-8_6

其它文件與鏈接

Link to publication in Scopus

引用此

Ignatov, A., Timofte, R., Chiang, C. M., Kuo, H. K., Xu, Y. S., Lee, M. Y., Lu, A., Cheng, C. M., Chen, C. C., Yong, J. Y., Shuai, H. H., Cheng, W. H., Jia, Z., Xu, T., Zhang, Y., Bao, L., Sun, H., Zhang, D., Gao, S., ... Lian, W. (2023). Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report. 於 L. Karlinsky, T. Michaeli, & K. Nishino (編輯), Computer Vision – ECCV 2022 Workshops, Proceedings (頁 130-152). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 卷 13803 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-25066-8_6

Ignatov, Andrey ; Timofte, Radu ; Chiang, Cheng Ming 等. / Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge : Report. Computer Vision – ECCV 2022 Workshops, Proceedings. 編輯 / Leonid Karlinsky ; Tomer Michaeli ; Ko Nishino. Springer Science and Business Media Deutschland GmbH, 2023. 頁 130-152 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{bca4d95121cf4a98809983c46026f30f,

title = "Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report",

abstract = "Video super-resolution is one of the most popular tasks on mobile devices, being widely used for an automatic improvement of low-bitrate and low-resolution video streams. While numerous solutions have been proposed for this problem, they are usually quite computationally demanding, demonstrating low FPS rates and power efficiency on mobile devices. In this Mobile AI challenge, we address this problem and propose the participants to design an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption. The participants were provided with the REDS training dataset containing video sequences for a 4X video upscaling task. The runtime and power efficiency of all models was evaluated on the powerful MediaTek Dimensity 9000 platform with a dedicated AI processing unit capable of accelerating floating-point and quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 500 FPS rate and 0.2 [Watt/30 FPS] power consumption. A detailed description of all models developed in the challenge is provided in this paper.",

keywords = "AI Benchmark, Deep learning, MediaTek, Mobile AI, Mobile AI challenge, Mobile NPUs, Video super-resolution",

author = "Andrey Ignatov and Radu Timofte and Chiang, {Cheng Ming} and Kuo, {Hsien Kai} and Xu, {Yu Syuan} and Lee, {Man Yu} and Allen Lu and Cheng, {Chia Ming} and Chen, {Chih Cheng} and Yong, {Jia Ying} and Shuai, {Hong Han} and Cheng, {Wen Huang} and Zhuang Jia and Tianyu Xu and Yijian Zhang and Long Bao and Heng Sun and Diankai Zhang and Si Gao and Shaoli Liu and Biao Wu and Xiaofeng Zhang and Chengjian Zheng and Kaidi Lu and Ning Wang and Xiao Sun and Wu, {Hao Dong} and Xuncheng Liu and Weizhan Zhang and Caixia Yan and Haipeng Du and Qinghua Zheng and Qi Wang and Wangdu Chen and Ran Duan and Mengdi Sun and Dan Zhu and Guannan Chen and Hojin Cho and Steve Kim and Shijie Yue and Chenghua Li and Zhengyang Zhuge and Wei Chen and Wenxu Wang and Yufeng Zhou and Xiaochen Cai and Hengxing Cai and Kele Xu and Li Liu and Zehua Cheng and Wenyi Lian and Wenjing Lian",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2023",

doi = "10.1007/978-3-031-25066-8_6",

language = "English",

isbn = "9783031250651",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "130--152",

editor = "Leonid Karlinsky and Tomer Michaeli and Ko Nishino",

booktitle = "Computer Vision – ECCV 2022 Workshops, Proceedings",

address = "德國",

}

Ignatov, A, Timofte, R, Chiang, CM, Kuo, HK, Xu, YS, Lee, MY, Lu, A, Cheng, CM, Chen, CC, Yong, JY, Shuai, HH, Cheng, WH, Jia, Z, Xu, T, Zhang, Y, Bao, L, Sun, H, Zhang, D, Gao, S, Liu, S, Wu, B, Zhang, X, Zheng, C, Lu, K, Wang, N, Sun, X, Wu, HD, Liu, X, Zhang, W, Yan, C, Du, H, Zheng, Q, Wang, Q, Chen, W, Duan, R, Sun, M, Zhu, D, Chen, G, Cho, H, Kim, S, Yue, S, Li, C, Zhuge, Z, Chen, W, Wang, W, Zhou, Y, Cai, X, Cai, H, Xu, K, Liu, L, Cheng, Z, Lian, W & Lian, W 2023, Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report. 於 L Karlinsky, T Michaeli & K Nishino (編輯), Computer Vision – ECCV 2022 Workshops, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 卷 13803 LNCS, Springer Science and Business Media Deutschland GmbH, 頁 130-152, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, 以色列, 23/10/22. https://doi.org/10.1007/978-3-031-25066-8_6

Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report. / Ignatov, Andrey; Timofte, Radu; Chiang, Cheng Ming 等.
Computer Vision – ECCV 2022 Workshops, Proceedings. 編輯 / Leonid Karlinsky; Tomer Michaeli; Ko Nishino. Springer Science and Business Media Deutschland GmbH, 2023. p. 130-152 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); 卷 13803 LNCS).

研究成果: Conference contribution › 同行評審

TY - GEN

T1 - Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge

T2 - 17th European Conference on Computer Vision, ECCV 2022

AU - Ignatov, Andrey

AU - Timofte, Radu

AU - Chiang, Cheng Ming

AU - Kuo, Hsien Kai

AU - Xu, Yu Syuan

AU - Lee, Man Yu

AU - Lu, Allen

AU - Cheng, Chia Ming

AU - Chen, Chih Cheng

AU - Yong, Jia Ying

AU - Shuai, Hong Han

AU - Cheng, Wen Huang

AU - Jia, Zhuang

AU - Xu, Tianyu

AU - Zhang, Yijian

AU - Bao, Long

AU - Sun, Heng

AU - Zhang, Diankai

AU - Gao, Si

AU - Liu, Shaoli

AU - Wu, Biao

AU - Zhang, Xiaofeng

AU - Zheng, Chengjian

AU - Lu, Kaidi

AU - Wang, Ning

AU - Sun, Xiao

AU - Wu, Hao Dong

AU - Liu, Xuncheng

AU - Zhang, Weizhan

AU - Yan, Caixia

AU - Du, Haipeng

AU - Zheng, Qinghua

AU - Wang, Qi

AU - Chen, Wangdu

AU - Duan, Ran

AU - Sun, Mengdi

AU - Zhu, Dan

AU - Chen, Guannan

AU - Cho, Hojin

AU - Kim, Steve

AU - Yue, Shijie

AU - Li, Chenghua

AU - Zhuge, Zhengyang

AU - Chen, Wei

AU - Wang, Wenxu

AU - Zhou, Yufeng

AU - Cai, Xiaochen

AU - Cai, Hengxing

AU - Xu, Kele

AU - Liu, Li

AU - Cheng, Zehua

AU - Lian, Wenyi

AU - Lian, Wenjing

PY - 2023

Y1 - 2023

N2 - Video super-resolution is one of the most popular tasks on mobile devices, being widely used for an automatic improvement of low-bitrate and low-resolution video streams. While numerous solutions have been proposed for this problem, they are usually quite computationally demanding, demonstrating low FPS rates and power efficiency on mobile devices. In this Mobile AI challenge, we address this problem and propose the participants to design an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption. The participants were provided with the REDS training dataset containing video sequences for a 4X video upscaling task. The runtime and power efficiency of all models was evaluated on the powerful MediaTek Dimensity 9000 platform with a dedicated AI processing unit capable of accelerating floating-point and quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 500 FPS rate and 0.2 [Watt/30 FPS] power consumption. A detailed description of all models developed in the challenge is provided in this paper.

AB - Video super-resolution is one of the most popular tasks on mobile devices, being widely used for an automatic improvement of low-bitrate and low-resolution video streams. While numerous solutions have been proposed for this problem, they are usually quite computationally demanding, demonstrating low FPS rates and power efficiency on mobile devices. In this Mobile AI challenge, we address this problem and propose the participants to design an end-to-end real-time video super-resolution solution for mobile NPUs optimized for low energy consumption. The participants were provided with the REDS training dataset containing video sequences for a 4X video upscaling task. The runtime and power efficiency of all models was evaluated on the powerful MediaTek Dimensity 9000 platform with a dedicated AI processing unit capable of accelerating floating-point and quantized neural networks. All proposed solutions are fully compatible with the above NPU, demonstrating an up to 500 FPS rate and 0.2 [Watt/30 FPS] power consumption. A detailed description of all models developed in the challenge is provided in this paper.

KW - AI Benchmark

KW - Deep learning

KW - MediaTek

KW - Mobile AI

KW - Mobile AI challenge

KW - Mobile NPUs

KW - Video super-resolution

UR - http://www.scopus.com/inward/record.url?scp=85151167744&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-25066-8_6

DO - 10.1007/978-3-031-25066-8_6

M3 - Conference contribution

AN - SCOPUS:85151167744

SN - 9783031250651

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 130

EP - 152

BT - Computer Vision – ECCV 2022 Workshops, Proceedings

A2 - Karlinsky, Leonid

A2 - Michaeli, Tomer

A2 - Nishino, Ko

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 23 October 2022 through 27 October 2022

ER -

Ignatov A, Timofte R, Chiang CM, Kuo HK, Xu YS, Lee MY 等. Power Efficient Video Super-Resolution on Mobile NPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report. 於 Karlinsky L, Michaeli T, Nishino K, 編輯, Computer Vision – ECCV 2022 Workshops, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 130-152. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-25066-8_6