Robust RGB-D Hand Tracking Using Deep Learning Priors

Jordi Sanchez-Riera, Kathiravan Srinivasan, Kai Lung Hua*, Wen-Huang Cheng, M. Anwar Hossain, Mohammed F. Alhamid

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

30 Scopus citations


With the irruption of inexpensive depth sensor devices, hand gesture tracking has become a topic of great interest. Two main problems to face respect other tracking algorithms are the high complexity of the hand structure, which translate in a very large amount of possible gestures, and the rapidness of the movements we are able to make when moving the hand or just the fingers. Recent approaches try to fit a 3D hand model to the observed RGB-D data by an optimization function that minimizes the error between the model and the data. However, these algorithms are very dependent on the initialization point, which are impractical to run in a natural environment. To solve these kinds of problems, it is common to use an offline data set with prelearned gestures that will serve as a first rough estimate. In concrete, we present an algorithm that uses an articulated ICP minimization function that is initialized by the parameters obtained from a data set of hand gestures trained through a deep learning framework. This setup has two strong points. First, deep learning provides a very fast and accurate estimate of performed hand gestures. Second, the articulated ICP algorithm allows capturing the possible variability of a gesture performed by different persons or slightly different gestures. Our proposed algorithm is evaluated and validated in several ways. Independent evaluations for the deep learning framework and articulated ICP are performed. Moreover, different real sequences are recorded to validate our approach and, finally, quantitative and qualitative comparisons are conducted with state-of-the-art algorithms.

Original languageEnglish
Article number7955084
Pages (from-to)2289-2301
Number of pages13
JournalIEEE Transactions on Circuits and Systems for Video Technology
Issue number9
StatePublished - 1 Sep 2018


  • deep learning
  • Hand gesture
  • hand recognition
  • iterative closest point algorithm
  • tracking


Dive into the research topics of 'Robust RGB-D Hand Tracking Using Deep Learning Priors'. Together they form a unique fingerprint.

Cite this