How Microsoft and Volkswagen got HoloLens 2 to work within vehicles

Microsoft partnered with Volkswagen to get the HoloLens 2 headset to work while in a moving vehicle. Up to this point, mixed reality headsets have been unable to work when on the move due to conflicting sensor data. Now, the HoloLens headset can work while on large ships, and Microsoft is working on expanding to more vehicles in the future.

The main challenge that prevents headsets like the HoloLens from working when on the move is that sensory input can cause confusion. The visible-light cameras and the inertial measurement unit (IMU) gather data for the HoloLens. When information from these sensors conflicts, the system doesn’t work properly. Microsoft equates this to humans getting seasick.

Microsoft addressed this issue by creating an algorithm that models any differences between a headset’s sensors.

Volkswagen and Microsoft started working together in 2018 to enable the HoloLens 2 to work within a moving vehicle but ran into logistical challenges along the way.

The global pandemic presented a unique roadblock that made it difficult to test prototypes. “We had to do a lot of testing in my apartment. These aren’t ideal development conditions,” said Joshua Elsdon, a Microsoft senior software engineer. “All of this stuff was done remotely and distributed across different countries, which was interesting.”

Eventually, more testing was done near Microsofts’ Redmond campus. Initial tests in the area were done on Puget Sound, which is a body of water near Redmond. Later, Microsoft worked with Volkswagen to test things out within vehicles.

“We connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car,” said Michael Wittkämper, augmented reality expert at Volkswagen. “This [opened] up completely new possibilities to not only display holograms within the driver’s forward-facing field of view, but also wherever the user wearing the glasses is looking.”

This isn’t Volkswagen’s first foray into mixed reality. The automaker introduced a head-up display in 2020 that projects navigation arrows, lane markings, and other information onto the windshield of a car.

At the moment, the moving platform feature for HoloLens works with large ships. In the future, Microsoft plans to get it to work with elevators, trains, cars, and more.