GamesBeat: You wind up getting the Star Wars: Secrets of the Empire experience in VR. But I could see very interesting things like Dr. Grordbort’s Invaders in a location-based setting with all the cool things they designed there for the room.
Laidacker: The physical world, especially if you take something like Disney or Universal–those people are experts at set design. Set design is so important for world-building and storytelling. A lot of the work we’ve done with Magic Leap is to show the possibility space of interacting with physical worlds, not just with digital content.
For example, one of our developers gave a talk at LeapCon where he talked about experimental inputs. He was showing something where we weren’t using any digital content at all. It was just a bunch of clocks on a wall. People would be able to put up their hands and control the speed of time. People were saying, “You can do that in mixed reality?” It’s using inputs from mixed reality, but communicating with Arduinos and Raspberry Pis.
In another example we had smart lights with a bunch of LEDs reacting to the inputs. We were able to change the color and emit particles from the physical lights. Every time I show this demo to someone, people actually lift up their glasses. They think the whole thing is fake. “Are those real lights?!” Yeah, they’re smart lights. It’s another example of combining the physical with the digital.
GamesBeat: The ILM folks showed the Porgs with the smart speakers.
Laidacker: My team worked on that. I was very proud of that demo.
GamesBeat: It seems like there’s a lot of cross-pollination or linkage going on between these MR experiences and escape rooms and things like that. It seems like that could be the best way to get your tech in front of a lot of people more quickly.
Laidacker: It’s a way to make it accessible, at least in the beginning, so that more and more people can try out the technology and see those possibilities, based on what mixed reality can be. But on the flip side we’re seeing a lot of developers come in and ask, “How can I use mixed reality to better people’s lives? How can I use it in the medical space?”
Or even on the entertainment side, I think a lot of the game devs who’ve come to this platform–there’s a lot more focus on interaction with digital characters and creating these intelligent characters that have agency and that adapt to the real world around you. You build this relationship, which is something you don’t necessarily see in traditional video games. Usually you have your weapon and you go out and do a bunch of fighting. The interaction space with mixed reality, because it’s in your home and feels much more personal, is making designers think about experiences in a more humanistic way.
GamesBeat: I was especially impressed by the quality of Dr. Grordbort’s Invaders, and then the Mica demo. Mica made more sense in the context of the digital assistants talk they did on stage.
Laidacker: What I liked when they described it is they said it’s not necessarily an assistant. When you think of it as an assistant, that’s a character who’s kind of there to serve you. We’re looking at it more as a companion. What I loved with Mica is she had agency. She was not there just to serve you in the experience. If anything, you really want to see how to bond with her and interact with her, but she has full agency.
On the flip side, what I love with Mica — and we stressed this with digital characters in general — is focusing on the awareness and the reactions. Are you making eye contact? If someone else walks into the room and I start having a conversation, the magic is when the digital character is aware of these environmental changes and reacting to them.
I always say that the new uncanny valley for mixed reality is not going to be around visual fidelity. We’ve solved that. If anything, people were blown away by the volumetric capture of MICA. But it’s really on that agency and AI reaction side. That’s where we really want developers to focus. Which is refreshing, because when you compare to game characters, game characters are kind of zombies. They only react when something happens.
GamesBeat: Do you think it’s more believable when it’s AR as opposed to VR? VR scenes, to me–you get things like eyes looking at you. But you also often get the sense that people are too close to you. It feels like they’re crossing into your space. Whereas with Mica, if she’s in your house, sitting on your furniture, and she’s occluded properly, it almost feels more like a real person.
Laidacker: She’s grounded in reality. That’s the bit that makes things more believable. You mentioned something interesting. She’s in your house. It’s very different from–let’s say I watch a character on television. I’m watching them in their world. But say I had that actor or digital character come to my home. The way they interact with me and my space should be very different, because they’re a guest.
One thing I love with MICA is this wasn’t just programmed by a bunch of AI developers. We brought in a behavioral scientist, someone who’s on the human instinct and behavior side, to design how MICA should be developed. That’s informing a lot of the tools we want to expose as far as how digital characters should behave in the real world.
GamesBeat: In the presentation, we got to see all these dimensions of the companion. The dragon sitting on your shoulder, or the Grordbort characters flying around with you. But if you think of something that’s more for games, do other ideas come to mind? How would you use Mica in a game?
Laidacker: What ILM was starting to do with the Porgs is a good example of that. It’s characters, but it’s these mischievous–I’m here to cause trouble and have fun with you in your environment, that type of thing. I know what they’re looking at, thinking about–there’s these characters who have AI and awareness, but then what are all the different interactive games we can play together?
It’s almost like me coming home to play with my pet, play with my kids, something like that. They’re designing around the kind of games you would play with a pet. Whereas with MICA, I would think about what I’m going to do with a friend when they come over. Maybe we’d play a tabletop game. Maybe we’ll watch something and interact with each other in different ways. All the design themes we’re trying to get developers to think about–if this was the real world, if this was a human, if this was a pet or something like that, what would you do? That informs a lot of the interaction elements in play.
GamesBeat: Do you actually want to see something like an escape room in AR?
Laidacker: I would love to see escape rooms in AR. We’ve had a few escape room developers come in and talk to us. This was before the release. One of the reasons is, they always talk about how in an escape room, you have to have someone staring on a camera the whole time. They come on a microphone saying, “That is not interactive. Please do not try to take that puzzle off the wall.” When people get stuck, someone has to be watching and saying, “Move on! This is your clue.”
All of these things are solved in video games. It’s all solved by giving haptic feedback, digital feedback, clues of different kinds. I know that for the escape room side, a lot of designers would love to be able to incorporate mixed reality, mostly just to help guide their users. It takes away that need for a physical piece.
GamesBeat: Hey, what’s that glowing red thing on the wall there?
Laidacker: Exactly! It takes away a lot of the physical build-out and resetting at the end of each stage. It also opens up the space toward what we do in video games, where games aren’t necessarily played out just one time, in one way. Especially systemic games, which you can play in many different ways. It opens up a large space for escape room designers to think about the replayability, where people would want to come back and interact with the digital elements again in different ways.