Crum often pauses in intense silences or starts sentences without finishing them while she grapples with the problem of how to describe a concept. Precision is important to her. While she clearly loves her work, Crum started off as a violinist, not a neuroscientist. At music school she took an elective course in neuroscience. “One of the papers was from Professor Eric Knudsen at Stanford who studied representations of what we hear and see and how we form a multi-sensory map of an object based on these two senses. When he reared owls with prism glasses on (so what they saw was inconsistent with what they heard), they developed a secondary map that integrated auditory and visual space.”

Crum has absolute pitch, which she describes as hearing sound as other people see color, but the names you give to those auditory colors depends on a learned pitch centre. In Western music, for example, the note A is assigned to the frequency 440 Hertz, whereas in early music — music from the medieval and Renaissance periods —  it’s 415 Hertz, a whole semi-tone lower. “I had a map of pitch that was at 440. Playing early music was driving me crazy for about six months. Every time I played a note, I saw it on the page and heard something different in my brain. One day I tuned up my early music violin at 415 and realized that I had absolute pitch at 415. I had developed an entirely new map of pitch. At that point, I decided that I had to understand my brain a little bit more.”

The science of sound

Sound poses a complex problem for the brain. If you hear a dog barking, your brain needs to figure out which sound frequencies are associated with the dog, integrate those sounds with what you see, and determine where the dog is. “It’s like monitoring just one square foot of the waves on a lake and figuring out how many boats are on that lake, how fast they are moving, and where they are located. That’s the problem our ears are having to solve.”

Our brain is constantly reweighting the information it receives from different senses. “The ability to locate something visually has less error, so we weight what we see more than what we hear, but in terms of when something happens, we weight our sonic information much more than our visual information. As you get to lower light levels, the model starts to changes and auditory information becomes more relevant.”

Different sounds constantly mask and obscure other sounds. Our brain uses context and expectation to make sense of it all. “When you listen to an old record, it’s got scratches everywhere, but you can still hear the music. The experience of the music and even the emotion is often just as strong. When there’s a big scratch on a record, to our brain something is still there. Information is missing, but we fill it in dynamically — in frequency, in time, and in spatial position.”

Sensory Superpowers

Dolby is constantly developing new sound and imaging technologies, and an understanding of how the brain perceives is vital to doing that effectively. “All of our products take advantage of perception in some way. Our codecs are a computational neural model which reduce information but maintain the perceptual experience by getting rid of information which I wouldn’t experience in real life.” One of the products in development is an imaging technology that can produce up to 20 thousand nits (a measure of light emitted per unit area as perceived by the human eye), as opposed to the 450-1000 nits emitted by a typical HDTV display. When Crum watched a video of fire on one of these new screens, something strange happened.

“I was watching a variety of content, all of which was producing the same amount of nits, but when the content was fire, I experienced my cheeks get warm,” Crum explains. “So I used thermal imaging cameras to track people’s faces, and there were changes when they saw flame. When we see flame in real life, our bodies are already preparing to expel heat based on the luminescence which is reaching our retina in conjunction with the fact that we know it’s fire. I ran the same test on HD displays, and you don’t see anything like this. This technology is truly creating a realistic experience by tricking the body.”

If we can create technology that can trick the body, all kinds of new sensory experiences become possible. “You don’t just want to create reality, you want to create something that’s even better. By using the synergistic effects of our senses on each other, we can amplify them so we have heightened experiences and potentially heightened emotional responses. Many species have superpowers, like bats and their ability to navigate. You can look at these species and how their brains have solved problems and use technology to create an experience that is not limited by the physical capabilities of our senses.”

Musical Brain Training

Crum is not only concerned with understanding and manipulating the way the brain perceives, but also with how we can use sensory input to alter the structure of the brain itself.

She teaches a class on Neuroplasticity and musical gaming at Stanford on how to create games that alter the organization of the brain in a targeted manner. “You have cells that have what is called a receptive field, the stimulus set which optimally produces a response from that cell out of some vast number of stimuli. With neuroplasticity, you might find different cells change the stimulus they respond to, or that they become more selective so you are developing more specificity.”

Crum’s students have developed all kinds of sound-based games to target different skills. One game trains people to detect phonetic differences that are usually lost in adulthood in order to allow them to become bilingual more quickly. Students have created games to increase accuracy in localizing sound and train digit span to improve memory.

One application of this kind of gaming is compensating for age-related sensory decline. “If I have hearing loss, I am relying too much on sonic information. I’ve had students re-weight how we integrate audio and visual information to allow people to hear better in noisy environments, to rely more on their visual cues when their auditory cues are compromised. In that case, they were training people to detect phonetic differences in lip reading and then progressing to different words.”

Our future may be full of devices that can create the impression of reality or even heighten it, manipulate our sensory experiences and emotions and even reshape our brains. This is less of a leap than it seems. “The brain’s job is not to get it right, it’s to be robust – the most impervious to error in action,” says Crum.”We are usually experiencing some kind of illusion.”

---

HealthBeat 2013 is a new conference showcasing how technology is transforming health care. We'll explore how IT is driving out inefficiencies on the hospital, practice, and patient levels. Check out full event details here, and register here.

---

Sound Uncovered is a free iPad app that features 12 experiences about auditory illusions and acoustic phenomena. To test the app, I went to the construction site for the interactive museum’s new home on Pier 15 and indulged my curiosity.

“We are using these tools to help people be more in tune with the real world,” said executive associate director Rob Semper during an interview in his office. “The idea that individuals can be in charge of their own learning was a trend in the 1960s and promoted by the museum’s founder, [particle physicist] Frank Oppenheimer. He wanted to create an environment where individuals could be inspired and inquisitive about the world, through hands-on investigatory experiences. This app is a continuation of that idea.”

The app is, frankly, super-cool. Whether you are a curious kid, a parent or a educator, or a museum-obsessed journalist, Sound Uncovered provides an intuitive, interactive, compelling exploration of auditory sensations. It presents you with experiments to engage with in-app and/or in the real world as well as informative articles and multimedia clips. The primary goal is to stimulate the mind using the unique technological features of the tablet, like motion sensors and touchscreens.

“Sound Uncovered is not a substitution for our physical museum,” said Jean Cheng, the project director of the online engagement group at the Exploratorium. “A tablet is a little black box where you can do anything. It has all these sensors and inputs; it is like a mini-laboratory. We want you to feel with learning that you can dig a little deeper. IPads are inherently social devices, and we want to spark conversation. It stays true to the bread-and-butter of the Exploratorium, which is all about perception and experiences.

First-person experiments include “Find the Highest Note,” which prompts you with a circular keyboard to confuse notions of high and low pitch, and “Eyes vs. Ears,” which shows how the perception of sound changes with and without visual cues. It also has a pretty trippy video of singing sand dunes in Morocco and an unexpected glimpse into how car manufactures engineer engine sounds to suit their customers. Hint: Nissan hybrid vehicles sound identical to vehicles from sci-fi film Blade Runner – and this is not a coincidence.

“Oppenheimer said the way to measure learning is if you enjoy it enough to want to share it with someone else,” Cheng said. “We are simply giving you access to the magic of these existing phenomena.”

The Exploratorium is committed to making its unique educational model as accessible as possible and to help people better understand the world around them. Sound Uncovered is the Exploratorium’s second digital product. The museum released the e-book Color Uncovered in 2011, which readers have been downloaded more than a million times. Its exhibits are on display at more than 1,000 science centers, museums, and public spaces around the world, and its digital consultancy group works with global institutions that want to integrate this type of programming.

It’s a busy time for the Exploratorium right now. The museum recently moved out of its location at the historic Palace of Fine Arts and will officially open at Pier 15 on April 17. The new location will contain 330,000 square feet of space, including 1.5 acres of public outdoor space on the waterfront.

Sound Uncovered was made possible by the Gordon and Betty Moore Foundation, as well as by the team of developers, designers, educators, artists, scientists, and museum staff dedicated to making learning enjoyable.

---

HealthBeat 2013 is a new conference showcasing how technology is transforming health care. We'll explore how IT is driving out inefficiencies on the hospital, practice, and patient levels. Check out full event details here, and register here.

---

“We’re launching uncomfortably early,” said chief officer Alexander Ljung of the new app. He said getting feedback from SoundCloud’s community is necessary to get the app into fighting shape and ready for its public beta in a few months.

Four-year-old SoundCloud has over 15 million users sharing songs, podcasts, family moments, and any other audio recording they can think up. Major musicians use the platform (Snoop Dogg recorded a special SoundCloud shout-out for the event), as well as news publications (The Economist, NPR), and politicians (Mayor Bloomberg). The company’s new mission is to “unmute the web,” making sound on the Internet as rich as it is in real life.

The new web app is optimized to work on all browsers and mobile devices. It has improved social-sharing, including the ability to repost sounds you like (a feature inspired by Tumblr and Pinterest) and real-time notifications when someone adds a new sounds or likes one of yours.

The interface is updated with a sleeker waveform player, profile pages, and easier navigation. But the features SoundCloud users are probably most pumped for are related to the audio itself. You can create and share sets of your favorite files, which appear as one clean waveform, and listen to any file in the background sans interruption while browsing the site. There is also continuous playing, so you can listen to file after file without having to hit play.

Other cool new features are set to roll out in the coming months.

“A lot of the cool things we are able to do are under the HTML5 umbrella,” said fo-founder Eric Wahlforss. The founders cited the improvement in browser technology as a reason they’re able to focus on a web app instead of having to make stand-alone apps for each platform.

July 9-10, 2013
San Francisco, CA

Early Bird Tickets on Sale

Smartphone company HTC is investing $300 million in digital sound company Beats Electronics, the company told All Things Digital.

Along with the investment, HTC will form a strategic partnership with Beats to begin integrating Beats technology into its entire line of products.

The partnership is likely part of HTC’s effort to boost its brand awareness in the U.S. While it has a strong manufacturing business, HTC isn’t as well-known as a consumer brand compared to competitors like Samsung and Motorola.

Consumers can expect to see the first HTC product with Beats integrated technology before the end of 2011, according to the company.

The HTC partnership with Beats is exclusive on all smartphones, according to the company. Beats already has a deal with Hewlett Packard to integrate its technology within HP PCs and the HP TouchPad tablet.

The site launched in late 2008, and compared to other media sharing services like Instagram, it may seem like hitting the 5 million mark within a few years isn’t a huge deal. But SoundCloud’s growth is quickly beginning to pick up steam. The site added 4 million users in the last year, co-founder and CEO Alex Ljung told the NYT.

The site lets anyone upload and share audio via its website or mobile applications. Listeners can comment on your audio files at specific points using SoundCloud’s waveform player.

SoundCloud is free for most, but there are paid plans for more hardcore customers. Ljung tells the NYT that the site was initially focused on offering music sharing to professional musicians, but it has since opened up its service to pretty much any audio file. SoundCloud now houses podcasts, audiobooks and more, but it’s still being used by some musicians to share their work.

“People share photos through Flickr and videos through YouTube, but sound hasn’t had a similar moment,” Ljung told the paper.

Indeed, it’s easier than ever now to create music and professional sounding podcasts with simple tools on computers, smartphones and tablets. SoundCloud is well positioned to see its popularity grow as consumers become as used to creating and sharing audio as they are with pictures.

SoundCloud is based in Berlin, but it has also just recently opened an office in San Francisco. The site has around 50 employees and has received $13 million in funding from Union Square Ventures and others.

It’s a large-scale platform for people to share the sounds they create and has quietly conquered the professional music community, amassed 4 million users and expanded into all kinds of sound, from podcasts and public earnings calls to cat impersonations. We covered the Berlin-based startup back in 2009 when it pulled in over $3M in funding, but it’s been mostly under the radar since then.

I recently talked to CEO and co-founder Alexander Ljung about how sound will become the sixth sense of the social graph. “We want to unmute the web,” said Ljung. “We think the web is very silent at the moment.”

SoundCloud started off as a way for Ljung and his co-founder Eric Wahlforss to share a piece of music with someone privately and get feedback. This led to the SoundCloud waveform; a visual representation that allows listeners to add comments at any point in the recording.

“We both came from the creator side of music. Eric was an artist creating music. I worked as a sound designer creating sounds and music for TV and films. We had both been tech geeks all our lives and were really into the whole social web movement around the time of Flickr. But there was no Flickr for music. There was nothing built for artists.”

SoundCloud had an API from the beginning so that developers could build various apps of their own on the platform. It now has around 150 apps. I asked Ljung about some of his favorites. “I made some beats on the Gorillaz beatbox app (for iPad) this weekend and then 50 Cent put up a vocal track where it’s only him rapping and he asked people to produce it. So now I have a track that I produced for 50 Cent!”. The Berkley school of music, one of the premier music schools in the world, recently rebuilt its environment to run its online courses entirely from SoundCloud.

Initially, SoundCloud was a set of tools for professional and semi-professional music creators. It’s now the standard way for many DJs and music makers to send and receive music. But Ljung soon noticed that people were using the platform in other ways too. “We saw how people were using SoundCloud for podcasts, radio interviews, sound effects. It started to dawn on us that the distinction between sound and music is fake. Music is one genre of sound, but there are many others.”

Ljung points out that many services now make it easy to create and capture sounds. “If you think of Twitter, it’s the simplest thing you can imagine. 140 characters. Even a sentence is an object.” Combine this simplicity and granularity with the emotional impact of sound and you get something special.

“If you want to stop being scared when watching a horror movie, you mute the sound. Sound is the emotional carrier,” Ljung told me. Human beings derive richer layers of meaning from a voice than from a written sentence. The background sounds when you call someone give them an instant impression of your situation. So why not do status updates in sound? Ljung calls these “social sounds”. SoundCloud has a variety of mobile apps to capture and share sound anywhere and anyplace.

This is part of the thinking behind one of SoundCloud’s latest services. “Last week we launched a service called TakesQuestions. You can record a question for me and I will record an answer. It’s like this asynchronous conversation. Somebody from the X-factor in the UK is doing it for instance,” Ljung said.

SoundCloud doesn’t seem to have any direct competitors. Like Flickr, its business model is based on a hierarchy of accounts from free to premium. The company opened an office in San Francisco recently, and Ljung now splits his time between the U.S. and SoundCloud’s headquarters in Berlin. Union Square Ventures invested $10 million in SoundCloud in 2010.

I asked Ljung for his thoughts on the future of the music business.“The general discussion is still that the music industry is in so much trouble, which is really not true. There are more people today than ever before who are involved in creating music. It’s time for a different definition of what the music industry is. The musical instrument market is twice as large as the recorded music market, and that’s been stable for 30 years. You need to look at what people are actually spending on engaging with music, not just listening to it. A guitar costs $1,000 or $1,500. That’s a lot of iTunes tracks.” Amen to that.