Leap Through

Seeing "through" the Oculus Rift with the Leap Motion Controller

Posted by Jacob H Clarkson on May 25, 2016

Leap + Oculus

The image below shows a Leap Motion controller attached to the front of an Oculus Rift DK2. There are several reasons one may wish to do this. Perhaps the most obvious is to use the hand tracking capabilities of the Leap Motion to facilitate interactions in VR - the wearer of the Head Mounted Display would be able to use their own hands as "controllers" in a game or virtual environment.

However, directly accessing the twin IR cameras that the Leap Motion uses for hand tracking allows users to see through the HMD. Combined with the hand tracking capabilities of the Leap, this setup allows for the implementation of a technique called "Minimal Blending" Augmented Virtuality. Whenever the user looks down at their hands, or holds their hands up in front of their face, the Leap detects the hands. At the positions of the hands, view ports into the real world are opened. This unobtrusive and intuitive addition of the real into the virtual makes it easy to interact with real world objects without ever having to remove the HMD.

The video below shows an early prototype of such a system in action. Note the ease with which the user is able to locate and interact with the objects on his desk.