Typing on a virtual keyboard floating in mid-air sounds cool until your arms give out. That’s the reality of many AR (augmented reality) and MR (mixed reality ) headsets today. You’re either fumbling with handheld controllers or holding your hands up for too long, causing fatigue.
Researchers at Tohoku University think they’ve found a better way. The team has developed a system that turns any regular flat surface into a touch input area for AR and MR headsets. No extra sensors. No special markers. Nothing to set up.
How does it work?
When you press a fingertip against a hard surface, the skin under your fingertip turns white for a brief moment. This phenomenon is known as the blanching effect, and researchers have developed a method to identify it using the headset cameras.
The researchers trained an AI model to detect this color change in real time. When the camera recognizes a blanching effect, it registers a touch input, which maps to whatever is displayed on the surface in the AR environment.
“This research means that ordinary surfaces all around us, walls, desks, or partitions, can be used as a touch input area,” said Guanghan Zhao, who led the study. “Moreover, this method doesn’t require special sensors, markers, or additional devices. Anyone can use it easily.”
Does it work well?
User studies in the research found that the system reliably detected touch inputs across several common surface materials, and participants completed interaction tasks with accuracy.
“Our primary objective was to develop a technology that allows touch input on everyday physical surfaces for AR and MR without the need for special hardware,” said Guanghan. And resting your fingers on a surface is a whole lot more comfortable than holding your hands in the air.
The findings were presented at the 33rd IEEE Conference on Virtual Reality and 3D User Interfaces in South Korea last week. The paper has also been approved for publication in the IEEE Computer Society Digital Library.