Researchers at MIT Media Lab have found a genuinely jaw-dropping use for the LiDAR sensor sitting inside your iPhone and iPad Pro. It can detect and track objects that are completely outside the camera’s field of view. Yes, that means seeing around corners.
This kind of imaging, called non-line-of-sight (NLOS) imaging, is not a new concept. But past demonstrations relied on powerful, expensive lab-grade lasers with little application in the real world.
What makes this research exciting is that the MIT team pulled it off using the same low-power LiDAR sensor already embedded in our smartphones.
How does it work?
The team is using the LiDAR sensor to allow us to look beyond corners at objects that are not directly in our line of sight. The secret sauce is motion. As your device moves, the system simultaneously tracks the object’s shape, the object’s position, and the camera’s position over time.
The team calls this an aperture sampling model, and it essentially stitches together a series of noisy, imperfect readings into something meaningful. The outputs are not crisp photos of what is hiding around the corner. Instead, you get progressively richer inferences. The system can tell you something is there, how it is moving, and what shape it roughly has. Think of it like echolocation, but with light.
What can it actually do?
The team demonstrated four specific capabilities: tracking a single object, reconstructing its shape, tracking multiple objects at once, and something particularly interesting for robotics, which is camera self-localization using hidden landmarks.
That last one is a big deal. A robot or autonomous system that can orient itself using objects it cannot directly see has a massive advantage in the real world. It can also help improve the self-driving tech or delivery drones for things like accident avoidance.
Sadly, you cannot try this on your smartphone right now, “as that would require these companies to release their raw data, which they often don’t do,” said Siddharth Somasundaram, one of the researchers on this project. That said, the researchers have made their code publicly available, and the sensor hardware can be assembled for under $50.