Deep Video Prediction Through Sparse Motion Regularization

Yung Han Ho, Chih Chun Chan, Wen-Hsiao Peng

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review


This paper introduces data-dependent sparse motion regularization for dense flow-based video prediction. To achieve video prediction (a form of extrapolation from past frames), the dense flow-based model estimates a motion vector for every pixel in a target frame for backward warping. Due to the sheer amount of motion vectors to be estimated, the model tends to be complex, thereby calling for proper regularization to avoid over-fitting. Most flowbased models adopt smoothness regularization. However, the smoothness requirement is detrimental to preserving the discontinuity of the motion field, which often appears in videos with distinct object motion. To address this issue, our sparse motion regularization discovers distinct sparse motion via weighted K-means clustering and regularizes the model based on minimizing clustering errors in the predicted motion field. When incorporated in an end-to-end trainable deep video prediction model, our scheme outperforms smoothness regularization, achieving superiority over direct generation-based video prediction on UCF-101 and Common Intermediate Format (CIF) datasets.

Original languageEnglish
Title of host publication2020 IEEE International Conference on Image Processing, ICIP 2020 - Proceedings
PublisherIEEE Computer Society
Number of pages5
ISBN (Electronic)9781728163956
StatePublished - Oct 2020
Event2020 IEEE International Conference on Image Processing, ICIP 2020 - Virtual, Abu Dhabi, United Arab Emirates
Duration: 25 Sep 202028 Sep 2020

Publication series

NameProceedings - International Conference on Image Processing, ICIP
ISSN (Print)1522-4880


Conference2020 IEEE International Conference on Image Processing, ICIP 2020
Country/TerritoryUnited Arab Emirates
CityVirtual, Abu Dhabi


  • Flow-based video prediction
  • clustering
  • weighted K-means


Dive into the research topics of 'Deep Video Prediction Through Sparse Motion Regularization'. Together they form a unique fingerprint.

Cite this