Miscellaneous helper robots have been whirring, rolling, and bumping around for years. They generally take the form of something adorable, with big doe eyes and a chunky white plastic body that is friendly and completely non-threatening. Or they’ll go in the opposite direction, like Boston Dynamics’ agile robots that move in an uncanny valley way but lack things like faces. A robot called Misty falls squarely in the former category; even the name is cute. Misty Robotics designed it to be extensible and flexible, with technologies like computer vision on board the robot itself, and ripe for iteration by enterprising developers. At Microsoft Build 2019, we got an extended demonstration of Misty and a long chat with the company behind it.

A man is standing in a room

Ben Edwards, Misty Robotics’ head of developer engagement, plopped Misty on a coffee table in front of me. At just a couple of feet tall or so, Misty needs the extra height. To demonstrate Microsoft’s computer vision-enabled scene description, the robot rolled over to me and instructed me to say “cheese,” which I did. Misty blinked its robot doe eyes, and I heard the digitized sound of a camera shutter. Click. “A man is standing in front of a mirror,” said Misty.

That wasn’t quite right. I wasn’t standing, and there was no mirror in front of or behind me. Maybe the glass-covered painting over my shoulder was especially reflective and caused the confusion. “Some of that’s a little bit on Microsoft, and some of it’s on us,” Edwards chuckled. “We’re sending kind of a small image right now, but she generally gets that it’s a person [who is] doing something, or what they’re holding.”

We tried it again. Click. “A man is standing in a room,” said Misty. Again, technically not quite true, but getting closer.

“‘A man standing is a room’ is kind of her go-to with Microsoft’s [scene description],” said Edwards. The Misty Robotics crew has also hooked it up to other cognitive services from Amazon and IBM. The results vary, Edwards said, but the notable piece is that Misty can connect to any similar sort of third-party service.

But Misty heavily relies on Microsoft technologies. Its operating system is Windows IoT Core — it uses Android 8 Oreo for navigation and computer vision and C# code where all the skills are written. “We’re moving toward .NET Core, which is not far off from C# anyways. But we think that’s going to give us true cross-platform capabilities on Linux, Windows, and macOS. People can just use it and get into it,” said Edwards. They’ve also integrated with Azure Cognitive Services and the text-to-speech engine, and there’s a Misty SDK extension for Visual Studio.

Parts and pieces

Misty is laden with sensors, cameras, and miscellaneous computer vision technologies, with Qualcomm Snapdragon 820 and 410 chips and the Qualcomm Snapdragon Neural Processing Engine. The other hardware bits are designed to be extremely extensible.

Edwards and an assistant showed me more of Misty’s abilities, including how it listens when you call it and turns to face whoever’s speaking, thanks to its three Qualcomm Fluence Pro-powered far-field mics. When it looks at you, you can see a line of sensors across its forehead. The sensors are actually all one unit, made by Occipital, that includes a 166-degree wide-angle camera, a 4K camera, and IR depth sensors. It also employs SLAM sensor fusion and Occipital’s own Bridge Engine for spatial computing.

One of the advantages of this sensor bar is that it helps Misty navigate back to its charging station, even in the dark. In a pinch, though, Misty could turn on the flashlight that’s mounted near its right “ear.” The head has capacitive touch sensors for additional controls, and there’s a Nebula-style panel that you can swap out for a different type of camera or a laser pointer or any other tool you can think up. The (patent pending) three degrees of freedom (3DoF) neck lets Misty emote a little by cocking its head to the side like a curious puppy.

There are speakers in the robot’s chest. All four corners of Misty’s base have time-of-flight bump sensors so it doesn’t run into objects and obstacles or fall off of things like coffee tables. The arms don’t do anything, although again, they’re meant to be extensible, so someone could dream up useful ideas with specially designed appendages. The most useful arm on this particular Misty offered a cupholder.

A backpack module that’s primed for the likes of a Raspberry Pi or Arduino can add bespoke functionality to Misty. “We did a demo at CES where we had a temperature sensor on the back,” Edwards said. If Misty was somewhere that got too hot, it could be programmed to send an alert to a dashboard. “Imagine — you could get readings all around in a warehouse,” he said.

A robot’s purpose

Misty has a lot of neat gadgetry on board, but this type of robot’s usefulness is not immediately clear. It’s not exactly agile, and it’s small. No one would mistake it for a robot that’s designed to stack warehouse boxes or tow vehicles, for example.

But Edwards said “it exists as the physical manifestation of the platform we’re creating for developers. We wanted to introduce robots to people who didn’t typically think that they were going to be into robots.” The idea is that Misty makes it easy for someone who can, say, write JavaScript to program a robot in a sophisticated way without having to deal with learning about and building hardware. That ideally means developers can come up with actual uses much faster.

The company is beginning to see some of those usage trends for Misty, Edwards said. In health care — elder care, especially — Misty can be fairly helpful. In addition to simple tasks like reminding someone to take their medication, a friendly robot that interacts with you in basic ways can offer a form of companionship, he said. More importantly, remote family members can interact with a local family member through a robot, to an extent. Maybe you can get a notification on a dashboard on your phone when your loved one takes their medicine, for instance. Edwards said a number of companies are working on applications in that area.

Misty can also be useful for building management. (This is an area Microsoft has expressed interest in, Edwards said.) You can deploy a robot like Misty around the shop or warehouse to perform status reporting or similar tasks that are time-consuming and thereby costly for humans to perform.

Perhaps a system of IoT sensors could work just as well and arguably more gracefully than a little robot buzzing around on treads. Edwards noted that there can be gaps in those systems, though. The robot may also alleviate some concerns people have about being constantly surveilled, like in a room that’s monitored by cameras. “[A] robot could roam around and only be present a certain amount of time and then take a picture [for reporting].”

There are potential security applications, too, like performing simple anomaly detection, or even looking to see if a person is supposed to be in a given place at a given time. Edwards and a partner demonstrated this particular use case for me. Misty looked at me and hollered, “Intruder! Intruder! You better leave now. I have already recorded you.” Apparently, one Misty Robotics engineer has already used Misty’s anomaly detection to thwart coworkers who were surreptitiously borrowing his tools.

Edwards mentioned that a sound technician had the idea to use Misty to roll around and help him check sound levels in music venues, which could save him maybe half an hour of setup time. “We’re not ever going to think of those scenarios,” Edwards said. “But if we get this in someone’s hands, maybe they think up a killer app for some percentage of the population.”

Misty is currently rather pricey at $2,400. It’s a tough sell without that killer application, and Edwards knows this. “We think that robots need to have more really valuable use cases in homes and offices before people are going to be like, ‘Oh yeah, I’ll buy a $1,000, [or] $2,000 robot [for] my house,'” he said. But the company is banking on the idea that this will happen once developers start dreaming up new ways to use Misty.

Microsoft Build 2019: Click Here For Full Coverage