Presented by Qualcomm Technologies, Inc.
One of the primary benefits of AI today, whether you develop AI software, shoot video, or take photos, is that AI can accelerate the process and bring your product to reality quicker. Specifically, for software developers, the AI tools at their disposal for edge devices like smartphones are opening the way for new cutting-edge features and applications.
“As a developer or as a creative, would you rather spend your time on the mundane challenges of programming or getting right to the creative side?” says Gary Brotman, senior director and head of AI strategy and product planning at Qualcomm Technologies. “Applying AI, through neural networks, to today’s smartphone cameras means that we can produce photos as good as those you’d expect from a high-end DSLR camera.”
Plus, as technology evolves, all those familiar features like scene recognition, night mode photography, super resolution, and more can be applied in real time rather than during post-processing as they are today. Eventually, most image quality and enhancement functions will be aided by AI in some way — or more accurately, the class of functions in the neural network realm.
“We are in the nascent stages of what’s possible with AI in imaging,” says Judd Heape, senior director of product management for camera, computer vision and video at Qualcomm Technologies. “The potential could open doors for developers and entrepreneurs to go far beyond our current thinking and understanding of recognition and detection.”
Camera features on the cutting edge
Currently, image modifications are done as post-processing, after the photo has already been captured. Oftentimes that editing is done with separate software programs or using the cloud. That’s all changing with mobile phones powered by technology like the Qualcomm® SnapdragonTM Mobile Platforms. Those devices are capable of processing these neural networks on-device without needing to go to the cloud. In addition, these algorithms can be applied to not just still images, but also to video. And as we go forward, more and more of the AI networks will be able to understand the camera shot in real time. For example, once it understands the scene and the objects in the scene, the AI network can instantly decide what enhancements need to be made to improve the photo or video.
“With advances in compute processing, you’ll be able to preview the effects you’re adding in real time,” Heape says. “As a consumer, you’ll effectively have cinema-grade video and still-image capture capabilities in your hand that were not possible before. You won’t need to hassle with loading a video to your PC and then going into an editing program to do painstaking modifications. All of that can happen locally, in real time, on the device itself.”
What’s under the hood
How are these new capabilities being made possible? Traditionally, hand-crafted scripts and feature detectors were manually developed for computer vision, but as the technology evolves, they’re being replaced by neural networks that are trained on data that is relevant to each particular function. That includes HDR, adding a filter, and other abilities currently in the camera pipeline — all of which are in some way evolving from traditional computer vision techniques to using new, state-of-the-art AI.
In deep learning, computers learn to perform tasks by analyzing training examples, usually hand-labeled in advance. For instance, an object recognition neural network would be fed thousands of varieties of labeled images, and then begin to learn visual patterns.
Neural nets are modeled loosely on the human brain, with thousands or even millions of densely interconnected processing nodes. Each node assigns a weight to each of its incoming connections. In an active network, the node receives a different piece of data over each of its connections and multiplies it by the associated weight. When it adds all those weights together, the resulting number must pass a threshold value in order to pass that number along all its outgoing connections, signaling a match between the image being evaluated and whether it correlates with a specific filter.
As they learn, neural networks can do anything from identifying a cat, to setting the right auto white balance, to zooming in on an item to maintain sharpness by filling in bits that didn’t exist in the image.
How to get 25X – 30X faster
“This is all possible because of huge advancements in hardware,” says Heape. “Now that we have hardware that’s more optimized for AI, and able to handle these networks, we can do this a lot more efficiently, using less power and time.”
AI is becoming far more prevalent in mobile or embedded devices; and the hardware technology is also keeping pace. The computational capabilities and architectures of device chipsets are being designed to efficiently process these AI algorithms.
The Qualcomm Snapdragon Mobile Platforms include CPUs, GPUs, DSPs, and dedicated AI accelerators. These compute cores work together to provide developers with a variety of options for processing AI algorithms far more efficiently than in the past, Brotman says.
“These AI-based camera capabilities would not have been possible had we not optimized all of the processor architectures in Snapdragon for power-efficient AI. We’ve seen the biggest advances in AI performance in the Vector Extensions in our Qualcomm® HexagonTM DSP, and our dedicated Hexagon Tensor Accelerator,” he explains. “Multi-core processor enhancements are hastening the path toward what will likely be a full AI-enabled camera.”
“Developers can achieve a 5x performance gain when running a vision-based neural network model on the GPU versus a CPU,” explains Brotman. “When you run the same neural network model on the Hexagon Vector Extensions or the Hexagon Tensor Accelerator, you’re looking at performance improvements of anywhere between 25x to 30x versus a CPU, plus a considerable improvement in energy efficiency.”
The future of AI and imaging
“We don’t really know where this is all going to end up, but developers are going to find lots of new and interesting things to do with this technology that maybe some of us never imagined,” says Heape. “It won’t just be for still-image and video quality. It’ll be for everything from understanding the environment for AR to better scene detection, to intuitive visual search of the photos and videos in your gallery. The future is very much unlimited.”
Content sponsored by Qualcomm Technologies, Inc. Qualcomm Snapdragon and Qualcomm Hexagon are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. Content produced by our editorial team is never influenced by advertisers or sponsors in any way. For more information, contact firstname.lastname@example.org.