Apple reportedly buys real-time green screen developer Spektral

If there’s a holy grail for augmented reality, developing a magic-quality way to instantly merge the real and digital worlds would be it — and a newly unearthed Apple acquisition could bring that one step closer to happening. Danish newspaper Børsen reports that Apple quietly spent approximately $30 million to purchase Spektral, a startup that uses machine learning to eliminate backgrounds from videos in real time on smartphones.

While the idea of using software techniques to separate people or objects from backgrounds is far from new — Apple added a similar feature to iChat back in 2006 as Backdrops — actually getting it to work well in real time is tricky. iChat Backdrops forced users to step out of the camera’s view and then superimposed whatever was different onto a fake background. Even at a low resolution, the effect always suffered from obvious pixel-level issues that made it clear the effect was artificial.

Before its acquisition, Spektral said it used “state-of-the-art machine learning and computer vision techniques” to “process images and video from the camera in real time (60 FPS) directly on the device.” Originally debuted under the name CloudCutout, its technology claims to intelligently separate people and objects as “cutouts,” seamlessly overlaying them on a new background. Demonstration videos evince what the company called “high quality details,” betraying no obvious sign of pixel-level issues.

The trick could have major implications for Apple’s AR initiatives. Being able to properly isolate specific elements in live video could enable ARKit applications where digital assets properly move in front of or behind real people, or where real people appear to be in fully or partially artificial surroundings. Like iChat Backdrops, it could also be used for cute special effects, akin to the company’s recent Animoji and Clips apps.

LinkedIn currently shows Spektral cofounder Toke Jansen as a San Francisco-based Apple manager of computational imaging, while another cofounder, Henrik Paltoft, continues to list one of his jobs as CEO of CloudCutout. As with all Apple acquisitions, it’s unclear when the acquired technology will make its way into shipping products, but the time frame after acquisitions like this has tended to be within two years.