Virtual reality is in its infancy, and we’re still trying to figure out what kind of experience it can deliver for the masses at an affordable price. So, we assembled some people who are thinking about this every day, as they try to figure out the next few decades of this new medium.
I moderated a panel on “Virtual Reality Entertainment Experiences” at the close of the Experiential Technology Conference this week in San Francisco. The event, organized by Zack Lynch, was all about how digital technology and neuroscience are coming together to improve human life.
We covered a lot of ground in our panel. We started with what we’ve learned from the first generation of VR games to what lies ahead for VR startups in 2017. We covered the experience, both visual and physiological, that we feel in VR. And then, we discussed where the platform should go in the future.
Here’s an edited transcript of our conversation.
Margaret Wallace: I’m the CEO of Playmatics. We’ve been around since 2009, located in Brooklyn and here in San Francisco.
Theresa Duringer: I’m a cofounder of Temple Gates Games. We make multiplatform VR games for Gear and Vive.
Noah Falstein: I’m chief game designer at Google. If you don’t know who we are, well, you could look them up on your favorite search engine.
Shiraz Akmal: I’m CEO of Spaces. We’re a company spun out of Dreamworks Animation, focused on creating next-generation theme park attractions and theme parks that are VR-enabled. We have some fantastic investors, including Comcast and the Venture Reality Fund.
GamesBeat: I’d like to talk about what we’ve learned so far about VR, where we are now, and where it’s all going. For 2016, what would you say you learned that was different from what was predicted? What’s a piece of conventional wisdom about VR, and how might you have learned that it was wrong?
We also saw certain things reinforced, like my belief in the real viability of VR and AR in healthcare settings and other more specific settings, beyond just the at-home experience.
Duringer: We always think of VR as this immersive space, and VR has shown itself to be good at delivering a world that you can step into. One thing that I think we maybe thought would be better was the ability to put yourself in the world. We have all kinds of input systems. We have wands and remotes and gamepads, all these methods for interacting with the world. But they’re all fairly clunky. Nothing really lets you be in the world in a one-to-one way. You can’t just step into the world and, say, play a piano. Ultimately, you’re still pressing buttons. It’s very abstract.
What we’re seeing that you can do really well is simply communicate with other people. You still have that one-to-one presence in the form of your voice. I see a lot of potential in the next year for VR as a huge social platform.
Falstein: One easy thing is that we’ve seen a lot of hunger for longer-form VR. The conventional wisdom among some people is that it’s a short-form medium, that people don’t want to be too intensely immersed in VR for too long. But as it turns out, even applications for just watching regular movies in VR have been much more popular than expected. Of course, virtually everyone with a headset also has a big-screen TV that can show those movies in high resolution, but for whatever reason, that’s been very popular.
Another thing that’s intriguing is that I think people are making assumptions about motion in VR that are unwarranted. We’re starting to find ways to move around in VR that are more natural. Just as with every other form of interface control we’ve worked with in the game industry, people will get used to something new very quickly. That’s encouraging.
Akmal: From my perspective — our company is only a year old, but as a team, we’ve been working for four or five years on VR and mixed reality with all different devices. This last year is the first year I’ve felt that it’s no longer special to be a VR company. We have a crystal-clear focus on a category of VR-type products that we’re building. This year is going to be the shakeout year. If you’re not focused on specific categories and products, whether it’s entertainment or something else, it’ll be difficult to survive and separate yourself from the pack.
It reminds me of the ‘90s, when it was cool to be a dot-com company for a little while. Sooner or later, that’s not enough. Then the question becomes, “What do you really do?”
GamesBeat: The first games have come out and everyone’s experienced them. Ubisoft said of their game Eagle Flight that when they were designing it, they were told that the experience has to be seven minutes or less. That was supposedly as long as people would have the patience to wear a headset. They found that people actually stayed in for longer than an hour. We have a different kind of knowledge now.
Falstein: CCP reported that Valkyrie was the same way. They had people playing for very long periods of time.
GamesBeat: For 2017, what’s your take on where the different parties in the ecosystem are? The investors, the brands, the platforms, and the startups.
Wallace: At Playmatics, we’re not a VR company exclusively. We’ve done some work in VR and more work in AR. I happened to be speaking with an analyst yesterday. We had a long chat about the funding landscape. His opinion reflected what Shiraz was saying: There’s going to be a slowdown in investment. There’s more supply than demand as far as investment money. I’m reminded of 2008 and the social gaming revolution, where there was a lot of overinflated companies. When they go back for their next round, it’s going to be a flat round or a down round. That’s not very attractive. Even investment money from China, which was an excellent source for some time, is drying up. People are looking toward Europe and trying to shore up those deficits.
GamesBeat: And there are more newly funded VR companies that are competing for this same dwindling amount of money.
Wallace: Exactly. There’s going to be a reckoning.
Duringer: Microsoft is in the game now. Google, Sony, Facebook, a bunch of other giants. There’s a tension I’m seeing from the development side. These companies want to fund exclusive content because that makes their platforms more attractive. But what’s working out very well is the social stuff, which needs people coming to it from a variety of different platforms. There’s a tension between — do you take money to cycle off in one direction, or do you play the long game and be poorer in the near term but possibly open yourself up to a broader experience for a lot more people in the future?
Falstein: Speaking to the platforms question, I’ve been thinking quite a bit about analogies to the early days of the games industry. People are asking if the current VR and AR platforms are the Newtons of this category, or are they the first smartphones?
What’s interesting is that, compared to every other generation in the games industry, there are so many different types of platforms out there. They range from Cardboard to Vive in terms of price and capability. The platform companies are all furiously working on improving their own platforms and building what they can do. I’m not sure that it holds all that well. It’s a bit of a scary time, when you have to pick one platform or aim for the lowest common denominator and spread your bets. It’s a hard time to be trying to pick right. But technology has always had those moments.
Wallace: You could compare it to the early days of console.
Falstein: But there weren’t seven or eight different types of consoles out there. It’s hard to see who’s going to win the horse race.
Akmal: From what I have a sense of, there’s still a lot of investment pouring in from the platform holders. Seven-figure deals are happening. But the difference, at least from my perspective, is that [the] early days were more of a spray-and-pray kind of situation. “Hey, you’re developing for VR? Here’s some money. Go make something.” Some of that is still happening, but now, it’s more targeted. “Hey, we’ve invested billions in this platform, and now, we need a title that can help us sell the numbers that everyone was projecting a year ago.”
The competition for those dollars is more fierce. There are bigger stakes in the development community, especially those studios that have bet on VR. Consumer adoption is — it’s adopting but not as fast as we’d all like.
GamesBeat: I feel like there’s a flip side to every answer we just had here. You can appreciate the fact that there are lots of platforms when you consider that you don’t want to be beholden to just one platform. There were days when the iPhone was the only game in town. Apple could muscle everyone as much as it wanted to. Now that Google and Android have offered competition, it’s at least a two-horse town. In that sense, developers in VR have more choices.
A slow start to this generation could also mean that the next generation can come in sooner, be better, and displace everything that’s come before. Do you envision that?
Duringer: What I’m seeing, what I predict is going to be a game changer, Android is the open platform. Apple’s is more curated, more on top of curating apps, but also provides a little more security for the consumer. I feel like we’ll see this in VR as well. We have a lot of platforms, but I think we’ll see it shake out into which one is more open, but higher risk, with a plethora of content from different people self-publishing. Steam and the Vive are moving in that direction, which makes it very attractive to developers.
Wallace: I would echo the enthusiasm for Steam. As a small company, when we think of the ideal place to deploy — we’re pretty platform agnostic, but we are appreciative of the opportunities on Steam. The divide, in our experience, is fun for a designer.
Akmal: I spend a lot of time in China, about two weeks out of every month. We have a huge, growing business there, and the rest is in the U.S. What’s interesting is the penetration of the different types of devices. HTC is dominant over other Chinese manufacturers and headsets. Oculus is almost nonexistent in China for a variety of reasons. In the U.S., you have a lot of penetration from PlayStation. From a content developer’s perspective, that gives a lot of directions for you to go.
In China in particular, this is something I hear over and over — we have the hardware; we just need great content. We need great content, so people will buy our hardware. I think there’s still a huge opportunity. The platform holders are in it to win it, but they need developers to create cool products that resonate with the audience.
Falstein: Making good content is tough. You look at consoles when they come out. The games that come out when a console’s been around for two or three years look like they were done on totally different hardware. Everybody learns how to get better at it as they get into each generation. We’re barely getting into the first generation of VR and AR content. Already, a lot of the systems are changing.
It’s going to be a tough path there, to gain knowledge fast enough to prove it out and not be outpaced by the hardware evolutions. But the flip side is that makes it a good time for consumers because it’s causing a lot of people to try so many different things. I appreciate that even here I’m seeing a range of different applications, stuff that’s purposefully trying to be creative and not just clone something that’s worked on another system. We have a bright future there with a lot of variety. I agree that it’s good to see open platforms out there that are encouraging people to try whatever they like.
Akmal: We’ve talked about content and platforms. James Blaha, you may know, originally started his company Vivid Vision and built his game Diplopia because he was moved to fix his own vision issues. I was in China last month, and I happened to look over and see James because it turned out we were co-investors in the same company.
I remember years ago at a conference — I think it was SPVR when I first met him — he was showing this little app he built that solved a problem using VR. I was so impressed with the fact that he continued honing in on that product and that solution. It’s grown into a huge business. I was proud to know him and know that he could — we’re doing theme parks and all this stuff, but I told him, “This is the coolest thing ever. You solved something. Your business uses VR, but it isn’t just about business.” I thought that was a great thing to see. VR will survive whatever goes on between platforms and change things for the better.
GamesBeat: Have we figured out at this point what makes people sick in VR, and why, and what it does to your brain? We’re not a medical panel, but is this a solved problem?
Falstein: We were discussing this earlier. When I saw the presentation yesterday from the magician [Stephen Macknik], he was showing how magic works through distraction. Because this is a neuroscience-oriented audience, I love that he puts up a diagram showing how you’re focused on something, and the neurons receiving that image have other neurons that suppress the input from other areas. It’s not only that your brain highlights the thing you’re focused on, but it’s also turning off everything else. That’s why you don’t see things happening around you when you concentrate on one thing very intently.
That struck a chord. I’m sure many of you have seen tunneling come up as a way to solve the motion problem in VR, where you contract the vision to a very small circle and either blur or blacken what surrounds it. Then, you can move that section over quite quickly. Unlike moving everything in VR, which feels very disconcerting, it’s more comfortable. Then, it zooms back out. When I first heard this described to me, I thought it sounded terrible, but every time I’ve seen it in action, it felt totally natural. I’m realizing that it mimics what those neurons are doing. We suppress everything else, and you naturally have this feeling that you’re looking at that one spot of your own volition. Much like directors learn to draw your attention to things as a way to get around that problem.
GamesBeat: Eagle Flight was a game where you could move very fast, flying through Paris, but you didn’t have as much of that peripheral vision problem because they blurred all that out. Blurring it also reduced how much processing they had to do. It’s not as hard to render. It’s high speed, so it’s fun, but you still don’t throw up.
Falstein: When we get eye tracking built into headsets, that will mean that you won’t have to render the entire scene in full detail. When you know exactly where people are looking, you’ll only have to render that part of the view. That’ll be a huge win as well.
Akmal: I like to think of that technique. We always talk about VR breaking the frame and bringing new immersion into the world. This is like putting the frame back in. The Heist, one of the first PlayStation VR launch titles, had a chase sequence when you were in a pickup truck. You can’t see behind the truck, but you can look ahead. There’s a frame. Even though you’re flying down the freeway, it reduced the motion sickness you’d normally have if you were in, say, a convertible, because your view frame is limited. With Eagle Flight, Ubisoft got that just right.
GamesBeat: Theresa, your second VR game was Ascension VR, and that had a physiological aspect. Can you tell us about that game and why you made it?
Duringer: One of the reasons I was working on this game was I wanted to make a game that would help with my fear of flying. I had noticed on a previous game I had made that it helped when I took it on a flight — when I was in VR, it replaced the sensation of being on a plane with something more pleasant that could distract me.
I worked on this game Ascension VR, and I felt like — after making it and reaching such high notes, I still felt like I did have a fear of flying that I hadn’t fully resolved. But I’m still hammering on it. I’m releasing an expansion that makes the game more difficult, which is helping a bit more. There are a lot of things people can do.
You were mentioning the magician’s trick, the use of distraction. Going back to my studies in psychology, your brain and your eyes aren’t processing a complete picture of the world. We make up so much of it. The receptors in your eye don’t actually detect color around the periphery of your vision. You just fill that in with assumptions. As VR developers, we can release the fact that we might have to create an entire world, which has huge performance implications, and just focus on the things that we actually need to give to the user in order to have the experience of another world.
Wallace: We’ve seen that people who are predisposed to motion sickness often feel nauseous in VR environments. There have been a lot of experiments with focusing only on forward motion, although that can be very limiting. I also saw a recent spate of articles directed toward consumers, a sort of effort at education. There’s an educational component. Maybe that’s overkill — maybe you don’t want to have to educate your audience. Maybe you just want them to experience whatever you’re making, but there are all these tips and tricks about how to avoid getting sick in a VR environment.
I think there is a level of that novelty where you can raise an audience’s awareness. You don’t necessarily want to have that friction, but I think it goes hand in hand. It reminds me of the early days of photography or the introduction of film. I think we’re at that stage, and it’s very exciting.
GamesBeat: There’s a lot of creativity in the way you use this new medium. The Walk is a very interesting demo, where you’re on a tightrope between the World Trade Center towers. There, you’re really trying to make people uncomfortable, make them feel afraid of the height. It’s trying to create the physiological effect that other games try to suppress.
Akmal: We have a philosophy of trying to create categories of experiences that work with various motion profiles. Certain kinds of experiences work well when you’re walking around or sitting at a desk. For theme parks, we’re looking at what vehicles and machinery — what can you do that, if you tried it at home, might make you sick? We’re trying to work with what physiologically feels good.
Falstein: It’s amazing to see people repurposing old roller coasters and giving you a VR experience that matches the actual motion. If you’re fine with the regular roller coaster, you’ll be fine with the VR version. But instead of the real-world environment, you’re seeing something like Jurassic Park.
Akmal: I usually don’t get motion sick at all, but I’ve tried something like that. It can be pretty taxing. It’s hard to get repeat customers if everyone walks off the ride sick.
Falstein: Every team is finding the one person on the team who’s most susceptible, and they become the guinea pig.
GamesBeat: I remember a speech from Mike Abrash, the chief scientist at Oculus. He gave a very interesting keynote at F8 about how good VR quality will have to get before the audience accepts it, likes it, and thinks of it on the same level as reality. They get so immersed that they can’t tell the difference. Just like the magic trick, though, he reminded everyone that the brain is not seeing the world. It sees data coming in that the brain interprets and processes and presents. What that says to me is that we don’t have to get to the point of absolutely perfect graphics. We just have to get to a point where we can fool the brain.
Akmal: Outside of the theme park business, we worked with Stanford to help their law group develop a VR study that put you in — if I’m a white man, it turns me into an African-American woman. You spend five minutes in this experience. The thing they were testing — it’s in peer review right now, but the results look positive — was whether we can reduce implicit bias in a courtroom among jurors and judges.
What was interesting to me as a developer was how important the idea of activating all your senses could be. In this experience, you put on a headset, you see the mirror, and hey, I’m this nice-looking African-American woman. They have you do a bunch of activities like stacking blocks and painting, Tilt Brush style. But [what] made the difference was how they made me use my right hand to touch my left hand. After doing these activities for a while, it felt like this was actually my arm.
Afterwards, they have people look at a crime scene and take an implicit bias test based on who they felt was guilty. Five days later, they come back in and take a slightly modified version of the same test regardless of the graphics, there was a difference, a positive difference in reducing implicit bias, which I thought was really interesting. It didn’t rely on graphics as much as the activities and making use of all your senses.
Duringer: Sometimes, you’re really going to benefit by creating abstract graphics. For one, it makes your application stand out from the crowd. A very stylized look can also have less texture frequency, which is a total shortcut way to reduce nausea if you don’t have a lot of high-frequency textures in your game.
To the question of how you convince someone to try VR, you can look back at movies. If you watch a movie from the ‘80s now, it’s just cheese city. It looks so bad. But if you watched it at the time, it looked great. As game-makers and as a consumer audience, we’re going to grow with the technology. We’ll look back at some of these early VR projects, and they’re going to be super cringey. But we’ll become savvier as the tech develops.
Falstein: That reminds me of a story from the early days of filmmaking. D. W. Griffith had this idea of cutting in close and just showing the face of an actress. He did that and showed it to movie executives to show how some things could change. One of the executives was really outraged, though. He said, “I paid for the whole actress, and you’re only giving me her head!”
One thing I find that’s emotionally involving about VR is when characters are in your physical proximity. It feels like you can reach out and touch them. We have a whole bunch of neurons that only activate when people are really close to us. We’re always hyper aware of people who could be within arm’s reach or even closer.
I’d love to see researchers look into this kind of thing, do comparisons to see if the same parts of your brain are lighting up when you’re physically close to someone compared to when you’re in VR with someone. There are reports of immersion that people are talking about, and I think part of it is the feeling, for the first time — if there’s just a picture up here that’s meant to trick you into thinking I’m closer to you, it doesn’t really work. If we were in VR and I looked like I was really just three feet in front of you, you’d feel more of that sense of being right in the virtual space with someone. The potential for storytelling and emotional involvement is amazing.
Wallace: This has huge ethical implications down the line. There were studies early on around people’s attachments to Warcraft characters and other game avatars. We’re seeing that same transference of empathy. This isn’t a panel about robots, but I was recently at Softbank Robotics interacting with Pepper, and I went there immediately. It took me about 10 seconds to empathize and feel for this robot.
GamesBeat: It seems like we’re in agreement that we don’t have to have perfect graphics in VR, but the platform should still move forward in some way. A lot of other people have been here talking about how it should go wireless. We don’t want to be hooked up to a PC. What else do you see that the platform should have going forward?
Akmal: The major ones are weight, wireless, a wider field of view. For more commercial-grade VR enterprise, that’ll be needed for applications like construction sites. Durability is important. The hardware right now is so fragile. That’s our biggest challenge to overcome, making headsets more durable but still comfortable to use.
Duringer: From the software side, we need faster Wi-Fi, so we can stream wirelessly at higher speeds. We need standards. We need to be able to rely on a gesture API, on open SDKs and open engines, so developers don’t have to keep reinventing the wheel.
Falstein: In the early ‘80s, I did some work on Atari VCS games. Back then, the processors were so slow that you couldn’t have a frame buffer. You had to refresh the same frame. The beam was just past that point. We’re starting to do that with VR now because even though we have much faster processors, there are so many more pixels and so many more colors. Not just processing speed but the whole pipeline — with Daydream, we had to revamp a lot of what was going on in the operating system as well as in the hardware itself to get all the latency that we could out of the system.
We still have a ways to go. We’ve gotten sloppy with computer games because the computer games have gotten so fast that we don’t have to worry about a lot of that stuff — certainly not on the same level that we used to. With VR, we do have to worry, and so, we need to figure out ways to make it faster and faster.
Duringer: In particular, we need more CPU power, so we can do voice blending and animation blending, which is what we’re all running up against in mobile VR.
Wallace: Wireless is definitely a huge barrier. I also look forward to seeing further improvements to firmware technology. A lot of those updates are super significant. I’m looking forward to seeing improvements in technology that help mixed reality become more of a reality. And because I work so much in a healthcare setting — honestly, that’s what I care about the most — I look forward to having opportunities for structured events that let you explore what the current technology can do, rather than just waiting for the hardware platforms to improve the devices and firmware.
GamesBeat: I wonder where some of the interesting dividing points are going to be between generations of VR hardware — 2.0, 3.0, 4.0. Some of the things that bug me a little are these incremental improvements that create problems in other areas and end up requiring more improvements.
The Vive Tracker is coming out this year at $100 each for each device. You can put a Tracker on each foot and on your back, and then, all of a sudden, you have the ability to track a human body’s movement. But maybe you’ll need a hundred of these things to do it really well. The Leap Motion has very good finger tracking, so you can get rid of touch controls entirely, but when you try to grab something, you can’t because you don’t know exactly where it is in space. You don’t feel yourself touching it. Now, we need haptics. One improvement causes you to need another, and the expense just keeps rising.
Akmal: Everyone’s cobbling stuff together because we want it all to work. We want these amazing experiences and applications. But let’s take off the rose-tinted glasses and look at the reality of what hardware does well right now and work within those constraints. It’s painful, but for anyone in the space, you have to be aspirational about it. Yes, it would be great if we had haptics and touch and nobody ever got sick. But then, there’s the reality of 20 wires and five batteries that have to be charged for a 20-minute experience.
Duringer: The real solution in the long term, I think, is going to be infrared or optical. It’s going to use your hands and your body. Everyone has that. We already have a standard. We just don’t have the tech there at scale yet, so we’ll have to build all these expensive hardware solutions in the meantime.
Question: What’s your favorite VR experience so far, for each of the panelists? You’ve all seen a great deal, so I’m curious about what you see as the best direction VR has shown so far.
Akmal: For us, it’s Robo Recall from Epic Games, which just came out. We’ve been playing it at the office nonstop. It’s polished. It’s what everyone says it is.
Falstein: Virtual Virtual Reality. I just played that, and it’s very funny. It’s a kind of VR take on the Portal experience. It’s been the most enjoyable game I’ve tried so far in VR.
Duringer: I’m in love with Medium. I love the creativity, the sculpting in an analog way.
Wallace: I’m a huge Mr. Robot fan, so here’s where narrative takes over for me. I love the Mr. Robot experience. I also like Robo Recall. The quality there can’t be beat.
GamesBeat: I liked Dead and Buried on Oculus. It’s a two-on-two Wild West shootout game.
Question: Asking about the work you did for Stanford, has anybody talked about going beyond that? Things like experiencing crime in the body of the victim?
Akmal: There’s a history of people doing research like this on all different hardware. It’s emboldening people to do more. There’s all kinds of combinations of putting different people in different situations. It’s certainly testable.
Wallace: My understanding, from being on a few NIH grants — I’m sure things like that are happening. They have pretty strict human subject criteria you have to pass, though.
Falstein: Chris Milk got a lot of press for doing a 360 video tour of some of the camps in Syria. There was an interesting serendipity in that everyone there is staring at this weird 360-camera rig. When you experience walking through this relocation camp, you see everyone looking at you in this really uncomfortable way. It feels like they’re staring at you as an intruder. They showed it to a bunch of people from the UN, and they all had a very empathic response.
Akmal: The Santa Monica police department and a couple of other police departments have seen the work we did. Running up to the election, implicit bias was a big topic, a national debate. There was a lot of interest from police departments asking if this was something they could use in both their communities, to help regular folks have an appreciation for cops, and vice versa, putting cops in the position of regular people. What was interesting about those discussions — police officers experience a lot of pressure and feelings of bias toward them. The thought was that this might be helpful for them to cope with perceptions of what the community thinks of them. I’m not sure where the business is for that, but it’s certainly a subject for research.
Question: We hear a lot of talk about killer apps, but what do you think is going to be the killer pricing model for VR, what’s most favorable to both developers and consumers?
Akmal: The way I think about our business and how we price things — we’re not selling VR. What we’re trying to sell is an entertainment experience. What would you be willing to pay to go to a theme park and have an amazing experience that might happen to involve VR? We model our business around — it’s not VR; it’s a theme park. Everybody could look at their business in that way and find a pricing model that makes sense for their particular vision.
Falstein: I hesitate to guess what will win, but it’s clear that we have a sort of Cambrian explosion of experiments. Everything is being tried. All the different pricing models that have worked before are being tried, and I’m sure new ones will be invented. As usually happens, whatever seems to work, people will start copying. But we don’t know exactly what it is.
Duringer: The era of the inevitability of freemium — a lot of people are not seeing tons of freemium content, but they’re worried about that. How is that going to work in VR? That could be very invasive if a product is asking you to fork over money, or if a kid is in there, and you don’t have access to what they’re experiencing.
Wallace: That’s one reason, once again, why I’m super excited about the applications of VR in healthcare and clinical settings versus having to go direct to consumers only. That’s still kind of dry as yet.