When the new generation Snapdragon 8 mobile platform was released this year, Qualcomm translated it again. What does this mean?

By identifying the diseases that users may have, such as depression and asthma, let the mobile phone learn to "auscultate";

Let the mobile phone achieve "anti-voyeurism" and realize automatic screen locking by recognizing the sight of strange users;

Let the mobile game get super resolution, and move the image quality that the PC can only run in the past to the mobile phone to experience.

More importantly, Snapdragon 8 has the ability to run these AI functions simultaneously!

Qualcomm claims that the performance of the seventh generation artificial intelligence engine on Snapdragon 8 is four times that of the previous generation.

This means that it is no problem to "open more" AI applications while playing mobile phones. More importantly, it is not only a simple AI performance improvement, but more importantly, it brings a smooth application experience to users.

Today, it is so difficult to upgrade the hardware technology. How did Qualcomm "dig out" so many new tricks in the performance and application of the 7th generation AI engine?

We read some research papers and technical documents published by Qualcomm and found some "clues":

In the AIMET open source tool document released by Qualcomm, there is information about "how to compress AI super-resolution model";

In a technical blog related to "anti-voyeurism", this paper introduces how to use object detection technology on the premise of protecting privacy.

These documents and the summit papers behind the technical blog all come from an institution-Qualcomm Institute of Artificial Intelligence.

It can be said that Qualcomm hides many AI papers published by research institutes in the seventh generation AI engine.

Summit paper "hiding" mobile phone AI

Let's take a look at the improvement of the seventh generation AI engine on the camera algorithm.

For this point of intelligent recognition, Qualcomm has increased the number of facial feature recognition points to 300 this year, which can capture more subtle expression changes.

But at the same time, Qualcomm increased the speed of face detection by 300%. How is this done?

In a study published by CVPR Qualcomm, we found the answer.

In this paper, Qualcomm proposed a new convolution layer, called Skip-Convolutions, which can subtract the two images before and after, and only convolution the changed part.

Yes, just like human eyes, it is easier to notice the "moving part".

This enables Snapdragon 8 to focus more on the target object itself when doing real-time video stream detection algorithms such as target detection and image recognition, and at the same time improve the accuracy by using redundant computing power.

You may ask, what's the use of such a detailed face recognition photo?

In addition, Qualcomm and Leica jointly launched the LeicaLeitz filter, which uses an intelligent engine based on artificial intelligence, including algorithms such as face detection, so that users can intelligently take more artistic photos without thinking.

Not only face detection, but also Qualcomm's functions in intelligent shooting include super-resolution, multi-frame noise reduction and local motion compensation.

However, the video stream in high-resolution shooting is usually real-time. How does the AI engine intelligently process such a large amount of data?

Also a CVPR paper, Qualcomm proposed a neural network composed of multiple cascaded classifiers, which can change the number of neurons used in the model with the complexity of video frames and control the calculation amount by itself.

In the face of the "huge and complicated" process of intelligent video processing, AI can now hold it.

Besides intelligent photography, Qualcomm's voice technology is also a highlight this time.

As mentioned at the beginning, the seventh-generation AI engine supports the accelerated analysis of users' voice patterns with mobile phones to determine the risks of health conditions such as asthma and depression.

So, how does it accurately distinguish the voice of users and does not involve collecting data?

Specifically, Qualcomm proposed a federated learning method for mobile phones, which can not only use the voice training model of mobile phone users, but also ensure that the privacy of voice data is not leaked.

Many artificial intelligence functions like this can be found in the papers published by Qualcomm Institute of Artificial Intelligence.

What can also be found is the theoretical support of AI mentioned at the beginning to improve the performance of mobile phones. This has to ask a question:

* * Running so many AI models at the same time, how does Qualcomm improve the processing performance of hardware? **

Here we have to mention "quantification", a key research direction of Qualcomm in recent years.

According to the latest technology roadmap published by Qualcomm, model quantification is one of the core technologies that AI Research Institute has been studying in recent years, aiming at "slimming" AI models.

Due to the limitations of power, computing power, memory and heat dissipation, the AI model used on mobile phones is very different from that on PCs.

On PC, GPU runs at several hundred watts, and AI model can be calculated by 16 or 32-bit floating point number (FP 16, FP32). However, the SoC of a mobile phone has only a few watts of power, so it is difficult to store a large-volume AI model.

At this time, it is necessary to simplify the FP32 model into an 8-bit integer (INT8) or even a 4-bit integer (INT4), and at the same time ensure that the model accuracy cannot be lost too much.

Taking AI matting model as an example, we can usually achieve very accurate AI matting with the help of the computing power of computer processors, but in contrast, if we want to achieve "almost effective" AI matting with mobile phones, we have to use the method of model quantification.

In order to load more AI models on mobile phones, Qualcomm has done a lot of quantitative research. The papers published at the summit include * * no data quantization * * DFQ * *, rounding mechanism **AdaRound**, joint quantization and pruning technology **BayesianBits, etc.

Among them, DFQ is a data-free quantization technology, which can reduce the time of training AI tasks and improve the performance of quantization accuracy. On MobileNet, the most common visual artificial intelligence model on mobile phones, DFQ achieves the best performance beyond all other methods:

AdaRound can quantize the weights of complex Resnet 18 and Resnet50 networks into 4 bits, which greatly reduces the storage space of the model and only loses less than 1% accuracy:

As a new quantization operation, Bayesian bits can not only double the bit width, but also quantize the residual error between the full precision value and the previous rounding value on each new bit width, thus providing a better trade-off between precision and efficiency.

These technologies can not only make more AI models run on mobile phones with lower power consumption, such as the game AI Super Resolution * (similar to DLSS)* which can only run on computers before, but now it can run on Snapdragon 8;

Even some of these AI models can "run at the same time", such as gesture detection and face recognition:

In fact, the thesis is only the first step.

In order to quickly put AI capabilities into more applications, more platforms and open source tools are needed.

Release more artificial intelligence capabilities to applications.

In this regard, Qualcomm has an open mind.

The methods and models for efficiently building AI applications in these papers have been shared by Qualcomm AI Research Institute to more developer communities and partners through cooperation and open source, so we can experience more interesting functions and applications on Snapdragon 8.

* * On the one hand, Qualcomm cooperates with Google to share with developers the ability to quickly develop more AI applications. **

Qualcomm installed Google's VertexAINAS service on Snapdragon 8, which is updated once a month, which means that the model performance of AI applications developed by developers on the 7th generation AI engine can also be updated quickly.

With NAS, developers can automatically generate appropriate models with AI, including smart camera algorithm, voice translation and super resolution announced by Qualcomm at the summit, which can be included in the "filtering range" of AI and automatically match the best model for developers.

Qualcomm's motion compensation and frame interpolation algorithms are used here. And AI technologies like these can be implemented by NAS, which can better adapt to Snapdragon 8, and there is no problem of "poor training".

Imagine that in the future, when you play games with a mobile phone equipped with Snapdragon 8, you will feel that the picture will be smoother, but it will not consume more power (referring to increasing power consumption):

At the same time, it is easier to mAIntain the ai model. According to Google, compared with other platforms, the number of lines of code required for VertexAINAS training model can be reduced by nearly 80%.

* * On the other hand, in recent years, Qualcomm has also opened his tools for studying quantity accumulation. **

Last year, Qualcomm launched an "efficiency improvement" tool model called Aimet *.

It includes a large number of compression and quantization algorithms, such as neural network pruning and singular value decomposition (SVD), many of which are the results of summit papers published by Qualcomm Institute of Artificial Intelligence. After using AIMET tools, developers can directly use these algorithms to improve their AI model and make it run more smoothly on mobile phones.

Qualcomm's quantification ability is not only open source for ordinary developers, but also enables more AI applications of head AI enterprises to be implemented on Snapdragon 8.

On the new Snapdragon 8, they cooperated with HuggingFace, a well-known company in NLP field, so that smart assistants on mobile phones can help users analyze notifications and recommend which ones can be handled first, so that users can see the most important notifications at a glance.

When running their emotional analysis model on Qualcomm artificial intelligence engine, they can be 30 times faster than ordinary CPU.

It is the precipitation of technical research and the open attitude maintained in technology that makes Qualcomm constantly refresh various AI "new brain holes" in the mobile phone industry:

From the previous video intelligent "elimination" and intelligent conference mute, to this year's anti-peeping screen and mobile phone super resolution.

There are more AI applications implemented by papers, platforms and open source tools, which are also carried in this AI engine.

Qualcomm Institute of Artificial Intelligence, hidden behind these studies, surfaced again with the emergence of the seventh generation artificial intelligence engine.

Ai's "carrot and stick"

Many times, our impression of Qualcomm AI seems to remain in the "hardware performance" of the AI engine.

After all, since the first artificial intelligence project was launched in 2007, Qualcomm has been improving its ability to handle artificial intelligence models in terms of hardware performance.

However, Qualcomm's research on artificial intelligence algorithm is also "carefully planned".

On 20 18, Qualcomm established the AI research institute, headed by MaxWelling, a well-known theoretical scholar in the field of AI, who is a student of Hinton, the father of deep learning.

According to incomplete statistics, since the establishment of Qualcomm AI Research Institute, dozens of papers have been published in top AI academic conferences such as NeurIPS, ICLR and CVPR.

At least four model compression papers have been implemented on the AI side of mobile phones, and there are also many papers involving computer vision, speech recognition and privacy computing.

The seventh generation AI engine mentioned above can be said to be just a microcosm of the achievements made by Qualcomm in AI algorithm research in recent years.

Through the AI research achievements in Qualcomm, Qualcomm has also successfully extended the AI model to many cutting-edge technology application scenarios.

In the aspect of autonomous driving, Qualcomm has launched the Snapdragon digital platform for automobiles, which "covers" a one-stop solution from chips to artificial intelligence algorithms. At present, it has reached cooperation with more than 25 car companies, and the number of networked cars using its solutions has reached 200 million.

Among them, BMW's next-generation assisted driving system and automatic driving system will adopt Qualcomm's automatic driving scheme.

On XR, Qualcomm released the development platform of SnapdragonSpacesXR, which is used to develop devices and applications such as head-mounted AR glasses.

Through cooperation with WannaKicks, Snapdragon 8 also brought the function of the seventh generation artificial intelligence engine to the AR try-on application.

On the UAV, Qualcomm released the FlightRB55G platform this year. Many of its functions, such as 360 obstacle avoidance and image stabilization, can be realized by the AI model carried on the platform. Among them, the first unmanned aerial vehicle "Wit" that arrived on Mars was equipped with a processor and related technologies provided by Qualcomm.

Looking back, it is not difficult to find that Qualcomm no longer emphasizes the improvement of hardware computing power (TOPS) in AI performance this time, but through the integration of software and hardware, obtains the data that AI performance is improved by four times, and further strengthens the all-round landing of AI application experience.

This not only shows that Qualcomm pays more attention to users' actual experience, but also shows Qualcomm's confidence in its own software strength, because the hardware is not completely the embodiment of Qualcomm's AI capability.

It can be said that the upgrade of the seventh generation AI engine of Snapdragon 8 marks the beginning of the integration of AI software and hardware in Qualcomm.

Recently, Qualcomm put forward several latest researches on codecs, which are listed in ICCV202 1 and ICLR202 1 respectively.

In these papers, Qualcomm also used artificial intelligence algorithm, showing a new idea of codec optimization.

In a study using GAN principle, Qualcomm's latest encoding and decoding algorithm makes the image not only clearer, but also smaller in each frame, which can be obtained only by 14.5KB:

In contrast, after the original coding and decoding algorithm is compressed to 16.4KB per frame, the forest will become extremely blurred:

In another paper that combines the idea of frame insertion with neural codec, Qualcomm chooses to combine P-frame compression and frame insertion compensation based on neural network, and uses AI to predict the motion compensation after frame insertion.

After testing, the algorithm is better than the SOTA record previously kept by Google on CVPR2020, and it is also better than the compression performance of the current open source codec based on H.265 standard.

It is not the first time that Qualcomm has applied the AI model to more fields, such as the application of video coding and decoding, which is a new direction.

If these models can be successfully put on the platform or even applied, we can really watch videos on the device without getting stuck.

With the continuation of the "soft and hard integration" plan, we may actually see these latest artificial intelligence achievements applied to smart phones in the future.

Combine Qualcomm's "muscle show" in PC, automobile and XR fields.

Predictably, Qualcomm you are familiar with and Snapdragon you are familiar with will never stop at mobile phones, and their AI ability will also stop at mobile phones.