Antutu is one of the most popular smartphone benchmarking apps around, with more than 100 million downloads globally. It attempts to measure hardware performance objectively with a battery of tests, the results of which it averages together to generate an overall score. Before this week, the gauntlet touched on CPU, GPU, UI, and memory. Starting today, though, it’s expanding to a new category: artificial intelligence (AI).
As spotted by XDA-Developers, Antutu this morning announced AI Review, a new benchmarking suite that aims to quantify smartphones’ AI performance. It’s available in public beta, and consists of two tests — an image classification test and an object recognition test — that favor accuracy as opposed to speed. (Devices that finish the tasks faster score higher, but receive a penalty for errors.)
Devices have to process 200 images using Google’s Inception v3 model in the image classification component. In the object recognition test, they’re tasked with identifying items using Google’s MobileNet SSD. Importantly, neither benchmark favors any one hardware platform over others: AI Review taps performance-boosting vendor SDKs where available, including MediaTek’s NeuroPilot, HiSilicon’s HiAI, and Qualcomm’s Snapdragon Neural Processing Engine. For manufacturers who have yet to supply a public AI SDK, like Samsung, the app defaults to TensorFlow Lite, a slimmed-down version of Google’s open source TensorFlow framework.
It’s a work in progress. Some OEMs, like HiSilicon, use TensorFlow Lite for certain AI tasks, which might hurt their performance. And as optimization work progresses on Android’s system-level AI support, including the Android Neural Network API, Antutu says that scores are likely to improve — or regress — with each version update.
Still, it’s an encouraging step toward an empirical benchmark for mobile AI performance across platforms like Samsung’s Exynos neural processing unit, MediaTek’s NeuroPilot AI platform, Qualcomm’s Hexagon DSP AI acceleration chip, and Huawei’s Kirin NPU — a benchmark that has thus far remained elusive.
A recent attempt by researchers at ETH Zurich — an app for Android devices called AI Benchmark — evaluates smartphones’ performance across a suite of open source algorithms that perform image classification, facial recognition, image super-resolution, and photo enhancement, segmentation, and deblurring.
In early tests, the results were mixed. According to the researchers, Qualcomm’s Snapdragon lacked drivers that would have improved performance at test time; Huawei achieved “quite outstanding” scores; Samsung had no acceleration support to speak of; and Mediatek had “good results” for midrange devices.
It’ll be interesting to see if Antutu’s AI Review corroborates those findings or exposes deltas in other areas.