GMRV Publications
Visuo-Haptic Mixed Reality with Unobstructed Tool-Hand Integration
IEEE Transactions on Visualization and Computer Graphics, Volume 19, Number 1, page 159--172 - 2013
Download the publication :
Visuo-haptic mixed reality consists of adding to a real scene the ability to see and touch virtual objects. It requires the
use of see-through display technology for visually mixing real and virtual objects, and haptic devices for adding haptic interaction
with the virtual objects. Unfortunately, the use of commodity haptic devices poses obstruction and misalignment issues that
complicate the correct integration of a virtual tool and the user’s real hand in the mixed reality scene. In this work, we propose
a novel mixed reality paradigm where it is possible to touch and see virtual objects in combination with a real scene, using
commodity haptic devices, and with a visually consistent integration of the user’s hand and the virtual tool. We discuss the visual
obstruction and misalignment issues introduced by commodity haptic devices, and then propose a solution that relies on four
simple technical steps: color-based segmentation of the hand, tracking-based segmentation of the haptic device, background
repainting using image-based models, and misalignment-free compositing of the user’s hand. We have developed a successful
proof-of-concept implementation, where a user can touch virtual objects and interact with them in the context of a real scene,
and we have evaluated the impact on user performance of obstruction and misalignment correction.
Images and movies
BibTex references
@Article\{CGBMO13,
author = "Cosco, Francesco I. and Garre, Carlos and Bruno, Fabio and Muzzupappa, Maurizio and Otaduy, Miguel A.",
title = "Visuo-Haptic Mixed Reality with Unobstructed Tool-Hand Integration",
journal = "IEEE Transactions on Visualization and Computer Graphics",
number = "1",
volume = "19",
pages = "159--172",
year = "2013",
keywords = "Mixed Reality, Visuo-Haptic Mixed Reality, Occlusion Handling, Haptic Interfaces, Image-Based Rendering",
url = "http://gmrv.es/Publications/2013/CGBMO13"
}
Other publications in the database