GMRV Publications

Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A. Otaduy, Dan Casas, Christian Theobalt
ACM Trans. on Graphics (Proc. of ACM SIGGRAPH), Volume 38, Number 4 - july 2019
Download the publication : mueller_SIGGRAPH2019.pdf [2.1Mo]  
We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction.

Images and movies

im1.png [339Ko]
im2.png [484Ko]
im3.png [615Ko]
TwoHands_SIGGRAPH2019.mp4 [164.1Mo]

BibTex references

  author       = "Mueller, Franziska and Davis, Micah and Bernard, Florian and Sotnychenko, Oleksandr and Verschoor, Mickeal and Otaduy, Miguel A. and Casas, Dan and Theobalt, Christian",
  title        = "Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera",
  journal      = "ACM Trans. on Graphics (Proc. of ACM SIGGRAPH)",
  number       = "4",
  volume       = "38",
  month        = "july",
  year         = "2019",
  url          = ""

Other publications in the database

» Mickeal Verschoor
» Miguel A. Otaduy
» Dan Casas