Without sound, visuals are nothing.
This week, German audio company Sennheiser reinforced that point as it launched its Ambeo AR One headphones for use with Magic Leap‘s augmented reality glasses. The new technology is aimed at making AR more immersive, with “transparent audio” effects that let you mix sound from the real world with music and sound coming in from your headphones.
The German company revealed the in-ear headphones at Magic Leap’s L.E.A.P. developer conference in Los Angeles. The Ambeo was the first major accessory to get the “Works with Magic Leap” certification. I spoke with Andreas Sennheiser, who, along with is brother Daniel, is part of the third generation of the family to run the company.
Sennheiser also launched the accompanying Ambeo Augmented Audio Lab app. The headphones will let developers and creators craft their own spatial computing experiences in which real sounds blend with virtual audio. With the Ambeo AR One, developers can mix it up, deciding how much of the outside world’s sound, captured by the headset’s built-in microphones, should blend into the spatial audio experience.
Here’s an edited transcript of our interview.
GamesBeat: And this is your headset?
Andreas Sennheiser: Yes, that’s the product we announced four hours ago. It’s been in the making for more than a year. We basically followed the development of Magic Leap, what they’ve done, to make sure that when they’re ready, we’re ready with the best possible AR audio solution. The outcome is this product.
It has microphones on the outside, so the transparent hearing functionality will allow you to have all the spatial information going on around you. After a minute or two you forget you’re in there. You’re just passing through the world. With the glasses you can look through, you get the sonic overlay of all the virtual elements. You get an augmented audio experience using this product.
GamesBeat: To what degree can you make it interactive? If you look at an object, can it interpret sounds from there?
Sennheiser: This is coming through the Magic Leap device. When objects are detected, you get the audio cues. Because this has a perfect binaural 3D representation, if something happens behind you, you get the right cues from behind. You can turn around and see the source of the sound. It’s fully integrated with the visual part of the Magic Leap glasses.
GamesBeat: I did the Star Wars Porg demo. They had a real speaker in the room, playing the music.
Sennheiser: Right, you probably didn’t wear any headset in that demo. That’s more just for the demo. You probably won’t install speakers in your home exclusively for Magic Leap. The other thing for that, the audio position in the room has to be right. A speaker has a fixed position. But if you have the head-mounted visual device and a head-mounted audio device, because of their interaction, the visuals and the audio are fully combined. There’s no discrepancy between the two.
GamesBeat: What do you call the transparent audio?
Sennheiser: The transparent hearing? We call that spatial audio. In a very simplistic way, it’s just making note of what you hear and bridging this one-centimeter gap, giving you the perfect 3D audio. But there’s more intelligence than that, just passing it through. You can also set it to active noise cancellation mode, so it cancels out more of the environment, or other different settings depending on how much of the environment you want to hear.
If you decide to watch a movie using your AR goggles, for instance, of course you don’t want to hear everything else around you. You can shut that off using the ANC. But if you’re playing a game or doing something that interacts with the environment, you want to hear all of the physical objects around you as well. Then you can put it to the highest level of transparency.
GamesBeat: If somebody is programming with this to do something coordinated around a visual apps, what might they do?
Sennheiser: We only have some suggestions and first ideas of what is possible. The universe will be opening up as soon as creators start to use it. We have launched a demo app, an audio app, called the Ambeo Lab. It’s available in the Magic Leap store. It allows you to experiment a bit with how the sound is being changed by positioning it behind physical objects, for instance.
If someone creates something, they respect three different levels of the audio. One is the audio providing a cue, somewhere to look at — it provides information about what happens around you. The second is it can create an atmosphere. The atmosphere, of course, needs to somehow reflect what you see. If you’re in a room like this, this zen-like atmosphere needs to sound different from if you’re in a railway station, say. Also, the integration between the Magic Leap product and this product ensures that all of the reflective behaviors of the walls are being correctly represented through the device.
If you take a sound source and place it behind this sofa, for instance, then you’ll hear the damping effect of that physical object in the sound. That’s part of what the creators need to experience — what it does with the sound when they begin to interact with the real world.
GamesBeat: Have you done anything like this before in other products, where you have the real world and virtual audio blending together?
Sennheiser: We have one product positioned in a similar way, our Ambeo smart headset. In addition to what this can do, it’s also a binaural recording device. If you plug it into your smartphone, you can have perfect 3D sound recordings. You can also set it to transparency mode, so if you’re riding your bike, you can hear cars and other things around you while you’re listening to your music. Likewise, there are different levels of transparency that you can use depending on the situation.
The other product we launched more than three years ago was the VR microphone. It pretty quickly became the standard for single-point VR production. It comes with software that does the AB conversion to allow the perfect 3D sound, if you have a 360-degree movie. A lot of these things are happening at Sennheiser.
I’d say for the last 15 years we’ve been researching the field. At the time it was just because we liked the idea. There wasn’t any commercial interest in it. But now, with the rise of AR and VR, we’ve seen how relevant this technology will be, blending and interacting with real and virtual objects.
GamesBeat: What’s in that part that you’re holding?
Sennheiser: This includes all the electronics for the ANC and the transparency. This is basically the brain. These are just microphones and transducers. All the algorithms are running in here. Of course, this carries a battery. You slide it open this way. The other product I mentioned, the Ambeo smart headset, is powered by your smartphone.
GamesBeat: It’s an interesting combination of possibilities. Sometimes you want the sound from the real world and sometimes you want to shut it out.
Sennheiser: When I met Magic Leap and Rony for the first time, they were a small company, only 40 people. It was almost five years ago. The only thing we were sharing at the time was the vision to create something that allowed people to perfectly blend reality and reproduction. In other words, we wanted the brain to not know or feel that there’s a wall between the reproduction and reality.
What he was trying to achieve on the visual part, we’ve been trying to achieve on the audio part. We saw that this was the perfect match. We bring visuals and audio together. It creates an experience, a real, lasting memory. If you see something that feels real, and you hear something that sounds real, it becomes part of your memory. That’s the intention behind both of our companies, to make a lasting impression.
GamesBeat: It’s a little like what happened when we first had the Sony Walkman. Suddenly the world had a soundtrack.
Sennheiser: It’s interesting you mention that. You may not know, but at the time, when Sony launched their first Walkman–Sennheiser invented the first headphones in 1968. We had 100 percent market share at the time with the HD414. We also had a patent that would last for almost 20 years on the open-design headphone. Sony and Sennheiser collaborated on that, because Sony needed our technology for the Walkman to become a success.
I don’t know if it’s comparable, but it feels a bit like Magic Leap is trying to do something really new, giving something to people on the move that so far wasn’t available. We’re adding the sound. It’s similar to Sony back in the ‘70s.
GamesBeat: Do you imagine any particular scenario where somebody interacts with something and you want to bring in sound from the real world, but then shut that off and segue into virtual sounds? Going back and forth like that?
Sennheiser: The closest thing in our world is live performances, music performances. With our technology and Magic Leap’s technology, it’s easy to resurrect bands that no longer exist. You could have the three remaining members of Queen playing live and Freddy Mercury blended in. For you it could just feel like you’re at a Queen concert in 2018.
That’s close to our business. Also, a lot of our customers and creators, they’re looking in this direction. But I could also envision this being perfectly suited to things like virtual teleconferencing applications. If I just want to chat with a friend who’s somewhere else in the world, and at the same time I want to have people in the room with me joining that chat, I need to hear both them and that virtual person, blended together in the right positions at the table. We could be playing cards with four people, two of them in the room and two virtual. There are endless possibilities.
We’ve seen a little bit of this already. I think it was 2013 when Coachella had Tupac on stage with Dr. Dre. It was a pretty sketchy hologram. It wasn’t perfect. I think it was a 3D projector on some kind of surface. But with the Magic Leap goggles something like that could become like a real person. I don’t know if you’ve had a chance see Michael, their avatar?
GamesBeat: Yeah, I have.
Sennheiser: So you know how real it can feel.
GamesBeat: They didn’t have any sound to that yet.
Sennheiser: Sure. But now she speaks. With that, she’ll speak as if she’s right there with you.
GamesBeat: How much effort do you plan to put into this in the long term? Will you be helping developers execute on their ideas?
Sennheiser: We’re providing a tool, a platform, an app, and a piece of hardware which should allow developers to let their creativity go wild. We’re at the beginning of something great. It’s difficult to imagine what happens from here.
Sennheiser has always been in the background behind the real stars. Our parts have always been tools, the means to connect an audience to the artist. This, again, is a tool, a means to connect the creative person and let them make something that they want someone else to consume.
GamesBeat: How much time do you expect this to take before you see people really start running with it?
Sennheiser: I think it’s down to two things. One is the form factor of the products. Of course we’re all working on making things smaller, Magic Leap and us and everyone else. Eventually this will reach a form factor where it’s like a pair of glasses, almost invisible. That’s one step that will take a couple of years. But the other thing is the acceptance of using these products. That will also take some years. You always have the early adopters for things like VR goggles in gaming, the people who love it and rave about it, but it’s not yet a social and interactive thing. You’re just sitting there with your goggles on.
It’ll take another two, three, four years until it becomes a mass phenomenon. But in the meantime we need to create the content for people to consume. In the end it’s a chicken and egg problem. If there’s no content, people have no desire for the product. That’s why we’re joining forces and working with big content producers to get them into the field. We can make something to build for the future.