AI-Benchmark, February updates:
The fastest AI chip, Qualcomm SDM6150, Samsung Galaxy S10 scores, and the recent ranking changes
During the past months, AI Benchmark scores were used in a number of events and publications, rising many questions regarding the performance of some newly presented chipsets and phones. We are providing our official explanation of the results and score updates for February 2019 below.
AI rush: Snapdragon 855, MediaTek P90, Kirin 980 or Exynos 9820 — who rules the game?
Snapdragon 855, currently placed on top of our ranking, is without a doubt one of the fastest chipsets available on the market. It is demonstrating very strong AI performance and provides hardware acceleration for both float and quantized neural networks: in the first case inference is done on Adreno 640 GPU, while quantized networks are running on its built-in Hexagon 690 DSP. This combination of GPU and DSP allows Qualcomm to omit the necessity of using a separate NPU for accelerating AI computations, which leads to smaller SoC size and its easier development. However, this decision also has its costs - Snapdragon's GPU cannot be fully utilized for running neural networks as its design was originally developed for pure computer graphics tasks, and thus only a small amount of its power can be used when running AI computations. This might also cause some difficulties in their future products development, as there are generally two ways of improving Snapdragon's AI capabilities: increasing GPU performance or radically changing its design, though the latter will also cause the change of the whole graphical system and drivers. And the third option is to introduce a separate dedicated AI chip, which actually might be the case in the next Qualcomm high-end SoC.
MediaTek P90 was quite a surprise for the market. Why this mid-range chipset from MediaTek is on top of our rating? The answer is simple - because its AI performance is completely comparable to the one of Snapdragon 855. In contrast to Qualcomm, MediaTek decided to go for a separate AI chip that was built based on their in-house GPU design significantly modified for deep learning tasks. The results are more than impressive - though P90's theoretical GMACs performance is notably lower compared to Snapdragon 855, their real speed in AI tasks is almost the same. We should also mention that the accuracy of the computations was not sacrificed for the sake of speed - sometimes it is even higher than with default Android drivers. The only downsides that this SoC has are its 30% lower CPU performance compared to Qualcomm's and Kirin's flagship SoCs and quite mediocre GPU, though it is not used in AI tasks and thus is actually irrelevant for our tests.
HiSilicon Kirin 980 was presented almost half a year ago and is showing somewhat lower scores than SDM855 and Helio P90. Does this mean that it is doing a worse job? Not really. Its float performance is almost the same as in case of the above SoCs, which means that you will get comparable speed when running float neural networks. We should emphasize that this is still the main type of models used in AI research and development: every network architecture can be trained as a float model. Instead, only some of them can be transformed into quantized models as this is often associated with a huge accuracy drop not acceptable for tasks like face recognition, image super-resolution or photo enhancement. And here the performance of Kirin chipset is still very strong.
But why its scores are then lower? From the beginning this SoC had two main issues: inability to process large data and a lack of acceleration for quantized neural networks. While the first problem was already fixed in the latest beta firmware (which results are available in the prototypes section), the second issue remains open. Due to technical problems it became impossible to use Kirin's NPU for accelerating integer computations via NNAPI, and HiSilicon went for GPU-based acceleration using Arm NN drivers, though in this case the performance is lower than the best results from Qualcomm and MediaTek. The latter also results in lower scores in the benchmark. So, at this moment Kirin 980 is still a very decent chip, but with some notable flaws with quantized computations.
During the last year Samsung was clearly falling behind in AI game - while Qualcomm was developing its first NNAPI drivers for Snapdragon's DSP, Huawei was presenting Kirin's 970 NPU, and MediaTek was making first attempts to accelerate neural networks on its Helio P60 SoC, all Samsung phones were lacking any acceleration support for AI, though in promotional materials there was some information about their Vision Processing Unit, which performance, specifications and SDK were never revealed to public. Just the same as happened to Pixel Visual Core AI chip - due to its insufficient performance Google decided not to provide any SDK or drivers for external developers despite the original promises.
Will the new Samsung Exynos 9820 SoC radically change this situation? We have already tested S10 phones with this chipset, and our first impressions are quite mixed. For now, we really hope that Samsung will be able to significantly improve the drivers before the release of S10 devices, and Qualcomm, Huawei and MediaTek chipsets will finally get accompanied by Samsung SoCs in our ranking.
It should be also noted that Samsung will anyway have an option to achieve quite good AI performance - as was shown in our previous analysis, Mali GPUs can considerably speed-up both float and quantized neural networks, and Samsung can just integrate Arm NN drivers to enable this acceleration. Though, in this case this will likely not happen before the release of the next Android Q firmware for Samsung devices, which will be not earlier than at the end of this year.
Prototypes & ranking updates: Snapdragon SDM6150 and Samsung S10 scores
The results of Qualcomm SDM6150 SoC were updated this month: its latest drivers brought acceleration support for float networks, and now this chipset can get up to 12000 points in AI Benchmark. According to our tests, its single-core CPU performance is comparable to the results of Snapdragon 845, though in multi-core tests it demonstrates 20-30% lower speed. Quantized and float tests suggest that SDM6150 might feature the same Hexagon 685 DSP as in SDM845, though its GPU should be around 2-2.5x slower compared to Adreno 630. In any case, this SoC will be a very strong player in its segment if shipped with the latest drivers and reasonably priced.
Samsung Galaxy S10 and S10+ with Snapdragon 855 finally appeared in our ranking. As expected, their results are very close to previously published scores of Lenovo Pro GT phones with the same chipset: with almost 21000 points they might be the fastest smartphones at the time of their official release.
LG is testing Android P update for their high-end phones with Snapdragon 845. This update will include the latest Qualcomm NNAPI drivers bringing phones' AI capabilities to a new level: LG G7 ThinQ with these drivers can run neural networks up to eight times faster than with the previous Android Oreo firmware. The same drivers are already included in the latest Android 9.0 update for Samsung Galaxy S9+ phones with Snapdragon 845, and should also appear soon in Vivo phones with the same chipset. Unfortunately, many other devices with SDM845 are still having either outdated Qualcomm drivers, or default Android NNAPI drivers not providing any hardware acceleration for deep learning tasks.
UNISOC SC9863 - an entry-level chipset from a recently rebranded Spreadtrum company has reached the performance of Snapdragon 650 in AI Benchmark. Though it lacks any hardware AI accelerators, it contains eight Cortex-A55 cores supporting Arm v8.4-A dot-product instructions that allow to run many neural networks faster. This SoC might be a good fit for budget smartphones with basic AI capabilities.
05 February 2019 Andrey Ignatov | AI Benchmark
