GMRV Publications

Visuo-Haptic Mixed Reality with Unobstructed Tool-Hand Integration

Francesco I. Cosco, Carlos Garre, Fabio Bruno, Maurizio Muzzupappa, Miguel A. Otaduy
IEEE Transactions on Visualization and Computer Graphics, Volume 19, Number 1, page 159--172 - 2013
Download the publication : TVCG13Cosco.pdf [2.8Mo]  
Visuo-haptic mixed reality consists of adding to a real scene the ability to see and touch virtual objects. It requires the use of see-through display technology for visually mixing real and virtual objects, and haptic devices for adding haptic interaction with the virtual objects. Unfortunately, the use of commodity haptic devices poses obstruction and misalignment issues that complicate the correct integration of a virtual tool and the userís real hand in the mixed reality scene. In this work, we propose a novel mixed reality paradigm where it is possible to touch and see virtual objects in combination with a real scene, using commodity haptic devices, and with a visually consistent integration of the userís hand and the virtual tool. We discuss the visual obstruction and misalignment issues introduced by commodity haptic devices, and then propose a solution that relies on four simple technical steps: color-based segmentation of the hand, tracking-based segmentation of the haptic device, background repainting using image-based models, and misalignment-free compositing of the userís hand. We have developed a successful proof-of-concept implementation, where a user can touch virtual objects and interact with them in the context of a real scene, and we have evaluated the impact on user performance of obstruction and misalignment correction.

Images and movies

amosaic.png [2Mo]
VHMR with Unobstructed Tool-Hand Integration.avi [44.6Mo]

BibTex references

  author       = "Cosco, Francesco I.  and Garre, Carlos and Bruno, Fabio and Muzzupappa, Maurizio and Otaduy, Miguel A.",
  title        = "Visuo-Haptic Mixed Reality with  Unobstructed Tool-Hand Integration",
  journal      = "IEEE Transactions on Visualization and Computer Graphics",
  number       = "1",
  volume       = "19",
  pages        = "159--172",
  year         = "2013",
  keywords     = "Mixed Reality, Visuo-Haptic Mixed Reality, Occlusion Handling, Haptic Interfaces, Image-Based Rendering",
  url          = ""

Other publications in the database

» Francesco I. Cosco
» Carlos Garre
» Fabio Bruno
» Maurizio Muzzupappa
» Miguel A. Otaduy