Qualcomm vision platform purpose-built for IoT
April 18, 2018
Qualcomm has announced a vision intelligence platform, featuring the California company's first family of systems-on-chip (SoCs) purpose-built for the IoT using its 10nm FinFet process technology.
The QCS605 and QCS603 SoCs are engineered to deliver computing for on-device camera processing and machine learning, with power and thermal efficiency, across a wide range of IoT applications. They integrate an image signal processor (ISP) and artificial intelligence (AI) engine, along with a heterogeneous compute architecture including an Arm-based multicore CPU, vector processor and GPU.
The platform also includes the company’s camera processing software, machine learning and computer vision software development kits (SDKs), as well as connectivity and security technologies. It is optimised for industrial and consumer smart security cameras, sports cameras, wearable cameras, virtual reality 360° and 180° cameras, robotics, and smart displays.
Kedacom and Ricoh Theta are planning to develop products based on the platform.
"Our goal is to make IoT devices significantly smarter as we help customers bring powerful on-device intelligence, camera processing and security," said Joseph Bousaba, vice president of product management at Qualcomm Technologies. “AI is already enabling cameras with object detection, tracking, classification and facial recognition, robots that avoid obstacles autonomously, and action cameras that learn and generate a video summary of your latest adventure, but this is really just the beginning.”
He described the platform as “the culmination of years of advanced research and development that brings together breakthrough advancements in camera, on-device AI and heterogeneous computing. The platform is a premier launch pad for manufacturers and developers to create a new world of intelligent IoT devices."
The platform integrates the company’s AI engine, which is comprised of several hardware and software components integrated to accelerate on-device AI. The engine includes the Snapdragon NPE neural processing engine software framework which includes analysis, optimisation, and debugging tools for developments using Tensorflow, Caffe and Caffe2 frameworks, Open Neural Network Exchange interchange format, as well as Android neural networks API and Qualcomm Hexagon neural network library, all of which are designed to allow developers and OEMs to port trained networks into the platform.
With the AI engine and Snapdragon NPE software framework, the platform delivers up to 2.1 tera operations per second of compute performance for deep neural network inferences.
The platform supports up to 4k video resolution at 60 frames per second (fps), or 5.7k at 30fps, as well as multiple concurrent video streams at lower resolutions. For image quality, the platform integrates what the company says is its most powerful camera processor ever created, a dual 14bit Spectra 270 ISP supporting dual 16Mpixels sensor – the evolution of ISP capabilities that consistently top DxOMark benchmarks over the past several generations.
In addition, the platform includes vision processing capabilities for IoT segments such as staggered HDR to prevent the ghost effect in high-dynamic range video, electronic image stabilisation, de-warp, de-noise, chromatic aberration correction, and motion compensated temporal filters in hardware.
The QCS605 heterogeneous computing architecture uses eight Qualcomm Kryo 360 CPU cores, the Adreno 615 GPU and Hexagon 685 vector processor. The platform's integrated display processor provides for a range of display options up to WQHD resolution touch display with hardware accelerated composition, 3D overlays, and support for the major graphics APIs including Open GL, Open CL and Vulkan.
The architecture supports various high-level operating systems engineered to let developers and manufacturers build differentiating features into their products such as on-device stitching for VR 360 cameras, autonomous robot navigation and obstacle avoidance, and video summarisation for action cameras.
The platform supports up to 2x2 802.11ac wifi with MU-mimo and dual band simultaneous transmission, Bluetooth 5.1, Qualcomm 3D audio suite, Aqstic audio technologies and AptX audio. The platform also uses the firm’s noise and echo cancellation technology, as well as on-device audio analytics and processing to support natural language processing, audio speech recognition, and barge-in capability for a reliable voice interface even in loud or noisy environments or when users are far from the device.
The platform's hardware-based security is designed to help support trustworthy IoT devices with features such as secure boot from hardware root of trust, trusted execution environment, hardware crypto engines, storage security, debug security with lifecycle control, key provisioning and wireless protocol security.
To accelerate development and differentiate products, manufacturers can turn to an ecosystem of technology providers offering technology that is complementary to the vision intelligence platform. These include AI providers such as SenseTime for face, image and object recognition, Pilot.ai for a variety of vision tasks such as detection, classification and tracking of actions and objects, and MM Solutions for image quality tuning services.
The QCS605 and QCS603 are sampling now, with multiple SKUs designed to meet various technical and cost effectiveness requirements. QCS605-based VR 360 camera reference designs from Qualcomm and Altek are available today, and QCS603-based industrial security camera reference designs are expected to be available in the second half of 2018.