Deep Learning on Smartphones, an In-Depth-Dive:  Frameworks and SDKs, Hardware Acceleration with NPUs and GPUs, Models Deployment, Performance and Power Consumption Analysis

07:00 Pacific Time   ┈   Andrey Ignatov   ┈   AI Benchmark Project Lead, ETH Zurich

Biography: Andrey Ignatov is the project lead of the AI Benchmark initiative targeted at performance evaluation of mobile, IoT and desktop hardware at ETH Zurich, Switzerland. His PhD research was aimed at designing efficient image processing models for smartphone NPUs / DSPs and developing the next-generation deep learning based mobile camera ISP solution. He is the lecturer on Deep Learning for Smartphones course at ETH Zurich and the main author of the AI Benchmark papers describing the current state of deep learning and AI hardware acceleration on mobile devices. He is a co-founder of the AI Witchlabs and co-organizer of the NTIRE and AIM events. His main line of research is focused on image restoration and automatic image quality enhancement, adaptation of AI applications for mobile devices and benchmarking machine learning hardware.

Abstract: In this tutorial, we will cover all basic concepts, steps and optimizations required for efficient AI inference on mobile devices. First, you will get to know how to run any NN model on your own smartphone in less than 5 minutes. Next, we will review all components needed to convert and run TensorFlow or PyTorch neural networks on Android and iOS smartphones, as well as discuss the key optimizations required for fast ML inference on the edge devices. In the second part of the talk, you will get an overview of the performance of all mobile processors with AI accelerators released in the past five years. We will additionally discuss the power consumption of the latest high-end smartphone SoCs, answering the question of why NPUs and DSPs are so critical for on-device inference.

Edge AI Technology – from Development to Deployment

08:20 Pacific Time   ┈   Dr. Allen Lu   ┈   Senior Director, Computing and AI Technology Group at MediaTek

Biography: Dr. Lu is the Senior Director of Computing and Artificial intelligence Technology Group at MediaTek. He is responsible for the ML/DL algorithm/tool development for mobile phone, camera, tablet, and TV products. The application covers video and image object detection, picture quality enhancement, noise reduction, etc.

Prior to MediaTek, Dr. Lu served as the General Manager of Video IoT (iVoT) Business Unit at Novatek. He was responsible for business and technology planning and execution. iVoT is the leader in dash cam market over years. In 2017, iVoT developed the first SoC integrating deep learning accelerator with 4K-resolution ISP (image signal processing) and video codec for surveillance camera. The SoC replaced the discrete solution and achieve a commercial success.

Prior to Novatek, Dr. Lu funded Afatek, a fabless company developing RF, digital modulator, and demodulator chips. Afatek developed the first silicon integrating RF front-end with demodulator for digital TV in 2006, and subsequently developed the first silicon integrating RF front-end with multi-standard modulator for surveillance application. Afatek was acquired by iTE in 2008.

Dr. Lu worked in Silicon Valley from 1997-2002. He joined the Excess Bandwidth Corp. funded by Stanford professors specializing in signal processing. He was responsible for the communication front-end and circuit-design. The start-up delivered the best performance symmetrical high speed DSL (SHDSL) modem prototype in six months. They subsequently developed integrated chipset for SHDSL modem in twelve months. Excess Bandwidth Corp. was acquired by Virata Corp (now Conexant) in 2000. Dr. Lu was a member of technical staff at Hewlett Packard prior to Excess Bandwidth.

Dr. Lu received his M.S. and Ph.D. in Electrical Engineering from Stanford University. He led two projects in WDM MAN and LAN testbed development funded by Sprint and ARPA. He published more than 30 papers in the optical communication and photonic switching fields.

Abstract: This presentation will share Mediatek’s experience in 5G+edge AI technology development and deployment. We use smart phone as an example to illustrate the edge AI technology evolution, from feature extraction for face unlock, to image / instance segmentation for video bokeh, to pixel-level AI algorithm for image and video noise reduction and super resolution. The picture and video quality are significantly better than the conventional method in the low light and high dynamic range photo scenarios. While the state of the art 5G+AI SoC packs tens of billions of transistors on a single centimeter by centimeter silicon chip for this achievement, the SoC complexity increases over time with a higher slope than the Moore’s law can offer. We will discuss the technology improvement to bridge this gap. It’s the combination of process improvement, HW architecture improvement, and SW/algorithm complexity reduction to enable new edge AI applications and enrich our digital life experience.

Imagination Technologies Approach to Overcame the Challenges of Deploying AI in Mobile

09:10 Pacific Time   ┈   Gilberto Rodriguez   ┈   Director of AI Product Management at Imagination Technologies

Biography: Expert in embedded solutions for telecommunications, video processing and artificial intelligence. Co-founded start-ups and lead teams of more than 70 engineers distributed worldwide. 20 years of experience leading global teams and organizations, Executive MBA from London Business School and MEng in Telecommunications.

Abstract: Mobile devices are nowadays very powerful heterogeneous engines, but they still have limitations of what you can achieve of them. With a growing number of applications using AI as part of their solution and the concerns about privacy of data, having efficient hardware and tools for deploying AI is the only path to success in the field. In this workshop, I will explain some of the techniques that can be used to optimise Neural Networks for a better deployment and how Imagination Technologies develops efficient hardware architectures for mobile devices.

Samsung Exynos Mobile NPUs and SDK: Hardware Design, Performance, Models Deployment and Efficient Inference

09:50 Pacific Time   ┈   Jihoon Bang   ┈   NPU and DSP Software Development Management at Samsung SLSI

Biography: Jihoon Bang had worked for Harman international and Nvidia before he joined Samsung SLSI division in 2018 to manage NPU and DSP software. He has Electrical Engineering BS and MS degree from Seoul National University in Korea, and Software Engineering MS degree from Carnegie Mellon University in the US.

Abstract: Samsung Exynos NPUs are now being widely used in numerous mobile, IoT and autonomous driving systems. This talk will provide an overview of their hardware design as well as the full software NN stack used for efficient models deployment. It will show how to perform models profiling and optimization with the Exynos Tools allowing to achieve the best performance on Exynos-based devices. Finally, several real-world applications that are using Samsung NPUs will be presented, and the future mobile NN development directions and challenges will be discussed.

Android Neural Networks API - What's New and Best Practices

10:40 Pacific Time   ┈   Przemysław Szczepaniak   ┈   Software Engineer at Google

10:55 Pacific Time   ┈   Mika Raento   ┈   Software Engineer at Google

Biography: Przemysław Szczepaniak is a Tech Lead / Manager in Android ML Platform, focusing on making the Android Neural Networks API (NNAPI) updatable. He's been working on NNAPI since Android Pie.

Mika Raento works as a Staff Software Engineer at Google, co-leading Android ML Platform. From 2014 to 2018 he worked as a generalist consultant at McKinsey & Company, in McKinsey Digital, leaving as an Engagement Manager. From 2010 to 2013 Mika Raento was the VP of R&D at ZenRobotics Ltd, leading the development of the world's first waste-sorting robot. Before joining ZenRobotics, Mika worked at Google on the first versions of Google Latitude. He joined Google via the acquisition of Jaiku Ltd, where he created the Jaiku S60 mobile client. Mika has a PhD in Computer Science from the University of Helsinki and a MSc in the same from the University of Jyväskylä. He lives in Käpylä, Helsinki, Finland.

Abstract: The Android Neural Networks API (NNAPI) was introduced in Android O-MR1, late 2017. NNAPI provides a standard hardware abstraction layer for ML accelerators on Android as part of the operating system, allowing accelerated computation over a graph of neural network layers. NNAPI has been successful in allowing access to previously hard-to-access hardware. However, its slow speed of evolution and lack of performance and correctness guarantees have made adoption by applications limited. In this talk we share recent advances for both of those problems. We'll describe how we are making NNAPI updatable outside Android OS releases through Google Play Services. We'll also talk about how to automatically deal with performance and correctness through a mini-benchmark embedded in TFLite models.

AI on the Edge @Synaptics : HW and SW Products and Development

11:30 Pacific Time   ┈   Abdel Younes   ┈   Technical Director, Smart Home Solutions Architecture at Synaptics

Biography: Abdel Younes is currently leading the AI/ML system architecture team at Synaptics and the development of the SyNAP framework that supports AI on Synaptics edge SoC. From Aerospace Research to AI, going through OpenSource, Deeply Embedded SW development and Multimedia, Abdel Younes has 20+ years of experience leading projects and teams in various industries. He received his MS in Telecommunications and Electronics and his Ph.D. in Signal Processing and Telecommunication from the French Civil Aviation Academy (ENAC).

Abstract: New AI-at-the-edge processors with improved efficiencies and flexibility are unleashing a huge opportunity to democratize computer vision broadly across all markets, enabling edge AI devices with small, low-cost, low-power cameras. Synaptics has embarked on a roadmap of edge-AI DNN processors targeted at a range of real-time computer vision and multimedia applications. In this talk, we will describe the hardware architecture used in Synaptics VideoSmart™ VS600 family with embedded built-in Neural Processor Unit and we will show how the SyNAP™ Software framework enables lightweight and optimized AI applications such as real-time Video SuperResolution upscaling.

AI Deployment from Hardware to Software – Challenges and Opportunities

12:30 Pacific Time   ┈   Emilia Tantar   ┈   Chief Data and Artificial Intelligence, Black Swan LUX

Biography: Dr. Emilia Tantar is Chief Data and AI Officer at Black Swan LUX, spin-off of the University of Luxembourg, she co-founded in 2015 and supports with AI expertise Huawei EU-SID, Consumer Business Group. She previously held the Chief Data and Artificial Intelligence Officer position at INCERT (public agency in Luxembourg) and led the Artificial Intelligence (AI) efforts of PwC Luxembourg. Emilia is member of the European Observatory of ICT Standardization WG on AI, European Commission Digital SME Focus Group on AI, and has over 15 years' experience delivering AI to the business market as researcher and strategic consultant. She has edited 7 volumes in Springer and has authored more than 20 journal papers and scientific articles. Emilia is also a Luxembourg national delegate on the standardization of AI in CEN CENELEC JTC 21 and Focus Group on AI and ISO/IEC JTC1/SC 42 for several years.

Abstract: AI has become a basic capability of consumer electronics and is continuously changing the user experience. Higher peak computing power and higher energy efficiency depend on the collaborative optimization of software and hardware. HUAWEI through the HiAI Foundation continuously provides leading solutions in key technologies such as NPU hardware architecture, heterogeneous computing framework, and quantitative search tools. Through a complete approach spanning from hardware to AI software modules will showcase a better intelligent experience for consumers on multiple devices.

Learning to See the World Clearer

13:15 Pacific Time   ┈   Jie Cai   ┈   Research Scientist at OPPO

Biography: Jie Cai is the research scientist at OPPO US R&D. He received a Ph.D. degree in computer science and engineering from University of South Carolina in 2019. And he received a Master's degree in electrical and computer engineering from University of Illinois at Chicago in 2015. His research interests include computer vision and deep learning，with a focus on face analysis，image processing, and image/video recovery. He is working on developing novel algorithms for image/video enhancement and super-resolution for mobile devices. He also works to accelerate and compress deep learning models.

Abstract: Recent progress in video super-resolution has been remarkable. However, since most of the recent approaches are deep learning based, they are way too computationally intensive to be deployed on edge devices. Besides, it is still a challenge to recover real-world videos with a variety of degradations. In this talk, an approach for real-time video super-resolution on mobile devices is presented, wihch is able to deal with a wide range of degradations. In addiation, it achieves state-of-the-art performance on a public video super-resolution dataset and has been delivered to OPPO smartphones.

Hate it or Love it, Your SW Stack Defines Application Performance and Reach: An Overview or our HW and SW

14:30 Pacific Time   ┈   Felix Baum   ┈   Director Product Management, Qualcomm Technologies

Biography: Felix Baum is responsible for AI Software Products at Qualcomm Technologies Inc. (QTI). Felix has spent 20+ years in the embedded industry, both as an embedded developer and as a product manager. He previously led QTI product management for the Hexagon Software supporting DSPs with scalar, vector and tensor accelerators for camera, video, machine learning and audio verticals. Prior to that, he led marketing and product management efforts for various real-time operating system technologies. His career began at NASA's Jet Propulsion Laboratory at the California Institute of Technology, designing flight software for various spacecraft and managing integration for a launch campaign of the GRACE mission. Felix holds a master's degree in computer science from the California State University at Northridge and a Master of Business Administration from the University of California at Los Angeles.

Abstract: Qualcomm has worked extensively on its state-of-the-art Qualcomm Neural Network and software stack including the Qualcomm Neural Processing SDK and the AI Model Efficiency Toolkit. By doing so Qualcomm has helped many developers take their applications to the next level with extreme efficiency and performance. In this talk, we will take you through some of the most common misconceptions related to software stacks and AI frameworks, and how these have a direct and indirect impact to application performance and reach. If you are interested in improving your application performance or even porting over to ML this talk is for you.