Connect with top gaming leaders in Los Angeles at GamesBeat Summit 2023 this May 22-23. Register here.

Antutu is one of the most popular smartphone benchmarking apps around, with more than 100 million downloads globally. It attempts to measure hardware performance objectively with a battery of tests, the results of which it averages together to generate an overall score. Before this week, the gauntlet touched on CPU, GPU, UI, and memory. Starting today, though, it’s expanding to a new category: artificial intelligence (AI).

As spotted by XDA-Developers, Antutu this morning announced AI Review, a new benchmarking suite that aims to quantify smartphones’ AI performance. It’s available in public beta, and consists of two tests — an image classification test and an object recognition test — that favor accuracy as opposed to speed. (Devices that finish the tasks faster score higher, but receive a penalty for errors.)

Devices have to process 200 images using Google’s Inception v3 model in the image classification component. In the object recognition test, they’re tasked with identifying items using Google’s MobileNet SSD. Importantly, neither benchmark favors any one hardware platform over others: AI Review taps performance-boosting vendor SDKs where available, including MediaTek’s NeuroPilot, HiSilicon’s HiAI, and Qualcomm’s Snapdragon Neural Processing Engine. For manufacturers who have yet to supply a public AI SDK, like Samsung, the app defaults to TensorFlow Lite, a slimmed-down version of Google’s open source TensorFlow framework.


Above: AI Review’s image classification task.

Image Credit: Antutu

It’s a work in progress. Some OEMs, like HiSilicon, use TensorFlow Lite for certain AI tasks, which might hurt their performance. And as optimization work progresses on Android’s system-level AI support, including the Android Neural Network API, Antutu says that scores are likely to improve — or regress — with each version update.


Intelligent Security Summit On-Demand

Learn the critical role of AI & ML in cybersecurity and industry specific case studies. Watch on-demand sessions today.

Watch Here

Still, it’s an encouraging step toward an empirical benchmark for mobile AI performance across platforms like Samsung’s Exynos neural processing unit, MediaTek’s NeuroPilot AI platform, Qualcomm’s Hexagon DSP AI acceleration chip, and Huawei’s Kirin NPU — a benchmark that has thus far remained elusive.


Above: The image detection component of Antutu’s AI Review app.

Image Credit: Antutu

A recent attempt by researchers at ETH Zurich — an app for Android devices called AI Benchmark — evaluates smartphones’ performance across a suite of open source algorithms that perform image classification, facial recognition, image super-resolution, and photo enhancement, segmentation, and deblurring.

In early tests, the results were mixed. According to the researchers, Qualcomm’s Snapdragon lacked drivers that would have improved performance at test time; Huawei achieved “quite outstanding” scores; Samsung had no acceleration support to speak of; and Mediatek had “good results” for midrange devices.

It’ll be interesting to see if Antutu’s AI Review corroborates those findings or exposes deltas in other areas.

VentureBeat's mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.