New AI hand tracking in Vrgineers’ XTAL headsets

Vrgineers and ManoMotion have developed an artificial intelligence (AI) hand-tracking algorithm designed to enhance the immersive experience of pilot training scenarios in virtual reality (VR) and mixed reality (MR) environments.

The feature is exclusive to Vrgineers’ XTAL Mixed Reality headsets and enables real-time hand-tracking functionality specifically optimized for cockpit settings.

This solution integrates pass-through RGB cameras and the AI algorithm, which together eliminate the background and allows for a focused display of the hands within virtual environments. Using the RGB camera stream, ManoMotion leverages this data to visualize the hands in both VR and MR simulations.

The goal of the MR cockpit hand-tracking project is to create a proof-of-concept, AI-driven hand-visualization functionality for pilot cockpit training. By utilizing the real-time RGB camera stream, the system detects and isolates the pilot’s hands, seamlessly overlaying them onto the VR scene.

Subsequently, the real-time data stream undergoes processing, leveraging a pre-trained hand-tracking model to separate the hands from the live video stream and render them in a visual representation.

Lars Österberg, CEO of ManoMotion, said: “Enabling pilots to see and interact with their real hands in VR helps make the cockpit pilot training solution even more immersive.”

The immersive experience offers enhanced realism and interactivity, bridging the gap between virtual simulations and real-world applications.

“As a virtual and mixed reality simulation specialist, we always try to find new innovative ways to increase the immersion of our solutions. Vrgineers was the first company to fully integrate infrared hand-tracking into a VR headset. And now, together with ManoMotion AI recognition using visible light, we are doing the same for the MR,” said Marek Polcak, CEO of Vrgineers.

Entrol has also recently launched an MR solution for training in the special missions space.