Huawei’s Facing Emotions app uses AI to help visually impaired users ‘hear’ facial expressions

Blindness, both partial and total, is incredibly common. The World Health Organization estimates that approximately 1.3 billion people live with some form of distance or near-vision condition and that as many as 217 million people suffer from moderate to severe problems with distance vision. Those impairments not only complicate everyday chores like walking the dog and checking the mail, they can make it difficult to pick up on emotional cues, like facial expressions.

To that end, electronics giant Huawei has tapped artificial intelligence (AI) to help blind people “see” the faces of those with whom they’re speaking. The Beijing company today debuted Facing Emotions, an app for the Huawei Mate 20 Pro that can “translate” seven universal emotions — anger, fear, disgust, happiness, sadness, surprise, and contempt — into corresponding sounds.

Huawei says it partnered with the Polish Blind Association to aid in the Android app’s development and that it recruited a group of blind testers and interviewed blind community representatives about their needs. The company claims those insights helped shape the app’s user experience — even down to its colors and sounds.

“Facing Emotions was created to push the possibilities of AI technology. By asking ‘how can AI help the blind and visually impaired to experience or “see” more of the world?'” Huawei wrote in a press release. “[It makes] it possible for the blind and visually impaired to have a richer experience and understanding when communicating with others.”

Facing Emotions taps the Mate 20 Pro’s back cameras to scan the faces of conversation partners, identifying facial features like eyes, nose, brows, and mouth, and their positions in relation to each other. An offline, on-device machine learning algorithm interprets the detected emotions as sounds, which the app plays on the handset’s loudspeaker.

Blind composer Tomasz Bilecki’s sound clips — which Huawei describes as “short, concise, simple, and non-invasive” — were tested by the aforementioned team of blind consultants. And designer Janek Kochanski developed a 3D-printed phone holder that allows visually impaired people with canes to use the app hands-free, and which is now available for free on Shapeways and partners in select countries.

The Facing Emotion app’s debut follows Huawei’s StorySign, which taps AI tools such as image recognition and optical character recognition (OCR) to translate popular children’s books into sign language.

“The Facing Emotions campaign embodies Huawei’s mission to bring new technology to everyone, especially those that need it most,” Wang Tao, president of Huawei’s Consumer Business Group, said in a statement.

Accessibility is a burgeoning sector of what’s been coined “AI for good” research.

Microsoft committed $25 million to its AI for Accessibility program in May, with the goal of “[helping] people with disabilities with work, life, and human connections,” and Google subsidiary DeepMind is using AI to generate closed captions for deaf users. Meanwhile, chipmaker Intel recently supported Wheelie, a startup working on a wheelchair that enables riders to navigate with facial expressions, through its AI for Social Good initiative. And just this month, Montreal startup Stradigi AI demonstrated a game that uses computer vision to help people learn American Sign Language (ASL).

“[Devices like the] Echo Dot make me feel included,” said Ellie Southwood, chair at the Royal National Institute of the Blind, in November 2017 at TechShare Pro, a U.K.-based conference on AI, disability, and inclusive design. “I spend far less time searching for things online; I can multitask while online and be more productive.”