The benchmark consists of 21 Computer Vision and AI Tests performed by Neural Networks running on your smartphone. It is measuring over 50 different aspects of AI performance, including the speed, accuracy, initialization time, etc. Considered Neural Networks comprise a comprehensive range of architectures allowing to assess the performance and limits of various approaches used to solve different AI tasks. A detailed description of 11 Benchmark Sections is provided below.

Section 1:   Object Recognition / Classification

Neural Network:   MobileNet - V2   |   CPU + NPU + DSP

Image Resolution:   224 x 224 px

Accuracy on ImageNet:   71.9 %

Paper & Code Links:   paper / code

A very small yet already powerful neural network that is able to recognize 1000 different object classes based on a single photo with an accuracy of ~72%. After quantization its size is less than 4Mb, which together with its low RAM consumption allows to lanch it on almost any currently existing smartphone.

Section 2:   Object Recognition / Classification

Neural Network:   Inception - V3   |   CPU + NPU + DSP

Image Resolution:   346 x 346 px

Accuracy on ImageNet:   78.0 %

Paper & Code Links:   paper / code

A different approach for the same task: now significantly more accurate, but at the expense of 4x larger size and tough computational requirements. As a clear bonus - can process images of higher resolutions, which allows more accurate recognition and smaller object detection.

Section 3:   Face Recognition

Neural Network:   Inception ResNet V1   |   CPU + NPU + DSP

Image Resolution:   512 x 512 px

LFW Score:   0.987

Paper & Code Links:   paper / code

This task probably doesn't need an introduction: based on the face photo you want to identify the person. This is done in the following way: for each face image, a neural network produces a small feature vector of size 128 that encodes the face and is invariant to its scaling, shifts and rotations. Then this vector is used to retrieve the most similar vector (and the respective identity) from your database that contains the same information about hundreds or millions of people.

Section 4:   Playing Atari Games

Neural Network:   LSTM RNN   |   CPU only

Image Resolution:   84 x 84 px

Average Total Reward:   168

Paper & Code Links:   paper / code / code

AI playing computer games like human... Why not? Originally proposed by DeepMind, this network is learning to play Atari games by interacting with their environment and observing the resulting rewards. And it's more than successful: 168 vs. 31 points got by a professional human player in Breakout game:  AI wins!

Original Image Blurred
Modified Image Restored

Section 5:   Image Deblurring

Neural Network:   SRCNN 9-5-5   |   DSP + NPU + GPU

Image Resolution:   384 x 384 px

Set-5 Score (x3):   32.75 dB

Paper & Code Links:   paper / code

Remember taking blurry photos using your phone camera? So, this is the task: make them sharp again. In the simplest case, this kind of distortions is modeled by applying a Gaussian blur to uncorrupted images, and then trying to restore them back using a neural network. In this task, blur is removed by one of the oldest, simplest and lightest neural networks - by SRCNN with only 3 convolutional layers, though in this case it still shows quite satisfactory results.

Original Image Zoomed
Modified Image Restored

Section 6:   Image Super-Resolution

Neural Network:   VGG - 19   |   DSP + NPU + GPU

Image Resolution:   256 x 256 px

Set-5 Score (x3):   33.66 dB

Paper & Code Links:   paper / code

Have you ever zoomed you photos? Remember artifacts, lack of details and sharpness? Then you know this task from your own experience: make zoomed photos look as good as the original images. In this case, the network is trained to do an equivalent task: to restore the original photo given its downscaled (e.g. by factor of 4) version. Here we consider a deep VGG-19 network with 19 layers. While its performance is currently not amazing and it is not able to reconstruct high-frequency components, it is still an ideal solution for paintings and drawings: it makes them sharp but smooth.

Original Image Zoomed
Modified Image Restored

Section 7:   Image Super-Resolution

Neural Network:   SRGAN   |   CPU x 2

Image Resolution:   512 x 512 px

Set-5 Score (x4):   29.40 dB

Paper & Code Links:   paper / code

The same task, but with new tricks: what if we train our neural network using... another neural network? Yes, two network performing two tasks: network A is trying to solve the same super-resolution problem as above, but network B observes its results, tries to find there some drawbacks and then penalizes network A. Sounds cool? In fact it is cool: while this approach has its own issues, the produced results are often looking really amazing.

Original Image Original
Modified Image Bokeh

Section 8:   Bokeh Simulation

Neural Network:   U-Net   |   CPU only

Image Resolution:   128 x 128 px

DIC-HeLa (IoU):   0.7756

Paper & Code Links:   paper / code

Probably the most well-known and popular AI task on smartphones - blurring the background like high-end DSLRs: just select Portrait Mode in your camera to see how it works on your device. In this section, a large U-Net CNN capable of doing heavy image processing is used to simulate bokeh effect without the need of multiple cameras: after being pre-trained, it can be applied to any image to add an artistic blur. Not always flawlessly, but quite impressive.

Original Image Original
Modified Image Segmented

Section 9:   Semantic Segmentation

Neural Network:   ICNet, 2X parallel   |   NPU + GPU

Image Resolution:   768 x 1152 px

CityScapes (mIoU):   69.5 %

Paper & Code Links:   paper / code

Running Self-Driving algorithm on your phone? Yes, that's possible too, at least you can perform a substantial part of this task – detect 19 categories of objects (e.g. car, pedestrian, road, sky, etc.) based on the photo from the camera mounted inside the car. In the right image you can see the results of such pixel-size segmentation (each color correpsonds to each object class) for a quite recent ICNet network designed specifically for low-performance devices.

Original Image Original
Modified Image Enhanced

Section 10:   Photo Enhancement

Neural Network:   ResNet - 12   |   NPU + GPU

Image Resolution:   128 x 192 px

DPED PSNR i-Score:   18.11 dB

Paper & Code Links:   paper / paper / code

Still struggling when looking at photos from your old phone? This can be fixed: a properly trained neural network can make photos even from an ancient iPhone 3GS looking nice and up-to-date. To do this, it observes and learns how to transform photos from a low-quality device into the same photos from a DSLR camera. Of course there are some obvious limits for this magic (e.g. the network should be retrained for each new phone model), but the resulting images are looking quite good, especially for old devices.

Section 11:   Memory Limits

Neural Network:   SRCNN 9-5-5   |   NPU + GPU

Image Resolution:   4 MP

# Parameters:   69.162

Paper & Code Links:   paper / code

You should already recognize it from section 5: SRCNN, one of the lightest and simplest neural networks... But even it can bring the majority of phones to their knees while handling high-resolution photos: to process HD-images the phone should generally have at least 6GB of RAM. This test is aimed at finding the limits of your device: how large images can it handle with this simplest network?

Copyright © 2019 by A.I.

ETH Zurich, Switzerland