VR Controller Camera Body Tracking

Carnegie Mellon researchers built a body tracking system using cameras on VR controllers.

Consumer VR systems today only track the position of your head and hands. No system on the market comes with tracking for your torso or limbs, but some PC VR enthusiasts attach HTC’s Vive Trackers to their hips and feet, a use case supported by software like VRChat and LIV. Since Vive Trackers cost $139 each, the total extra cost is over $700 if you don’t already have SteamVR base stations. There are unofficial body-tracking solutions that use old hardware with community-created drivers, such as for Microsoft’s Kinect, but this only works from a frontal angle and the tracking quality isn’t great.

Additional cameras on headsets pointing at your body could theoretically track limbs, but the view of your legs would often be obscured by your torso or arms, especially as headsets get thinner. Companies like Meta could also release wireless remote tracking stations with a camera, battery and mobile chipset running computer vision algorithms, but developers couldn’t rely on users to buy and install such a costly accessory.

ControllerPose is a prototype solution with two fisheye cameras attached to each VR controller, facing you. The software merges the view of the two cameras to achieve an even wider field of view, to see as much of your body as possible. Their pose estimates are accurate to 8.59 cm, the researchers report.

To process the cameras, the researchers tried two approaches: streaming to a nearby PC and onboard processing using a Raspberry Pi Compute Module 4 and Google Coral. Using a PC (or the standalone headset) would be cheaper, but onboard processing would result in lower latency.

In September last year, screenshots emerged showing the upcoming Project Cambria controllers with three built-in cameras for self-contained inside-out tracking. These cameras don’t appear to be aimed at you, but a leaked instructional video did appear to show body tracking. It’s possible that Cambria or a future Meta headset could perform body tracking using a technique similar to ControllerPose.

The downside to ControllerPose, the researchers say, is that 32% of the time in “typical VR experiences” the controllers don’t have a good view of the body — they’re either too close or to the side. Future systems may be able to integrate cameras on the controllers and cameras on the headset, but it seems that only a third-party solution can achieve completely reliable body tracking. But could such a solution work in enough scenarios to be useful for most people?

Leave a Comment