Meet the AI Powering Today’s Smartest Smartphones

Artificial intelligence is increasingly becoming the differentiator that makes a new generation of smartphones truly clever. The ability of phones to generalize and determine what might happen next based on previous patterns and datasets—what’s known as machine learning—is, according to some analysts, becoming an “essential part” of users’ experiences. In 2017, specialty microprocessors that enabled AI were in just 3% of all smartphones. As of 2020, more than one-third of the world’s three billion smartphones were equipped with processors conducting trillions of operations quickly and with less power. 

The Snapdragon® mobile platform*, with its built-in AI engine, is one such system—and it’s more than likely not something that crosses the mind of most smartphone users: “There’s actually a lot of AI involved that you don’t know about. It’s running in the background to help us take a professional-level photo or quell background noise while we’re on a call,” says Ziad Asghar, VP of Product Management at Qualcomm Technologies, Inc.

And while it may operate out of eyesight or thought, the AI engine that powers the aspects of a smartphone we’ve come to rely on (think: the autofocus of the camera, the ability to keep the sound on our calls crisp and even translate them from one language to another) is what makes our phones smart—and trains them to get smarter.

What’s in a photo?

Phones equipped with Snapdragon operate a series of neural networks, the brains of the device that helps various phone functions perform in cutting-edge ways. 

Take those triple-camera smartphones, which are rapidly becoming the new normal: By the end of 2021, some research suggests more than half of new smartphones will have three cameras. In the background, as photos and videos are being taken with these cameras, the Qualcomm® AI engine is applying AI techniques that allow for that much-loved bokeh effect—it is helping to shoot videos at night and even helping social media filters to work better in the background. 

“We work with social media apps to enhance some of their AI filters because our camera knows where your eyes, your nose, your ears, and where your hair are, and correctly pastes those effects on your face,” Asghar says.

Artificial intelligence (AI) is hard at work capturing photo and video thanks to the Snapdragon mobile platform which streamlines and enhances the ability to create professional-level shots.

The AI at work is delving beyond filters, too, allowing users to focus the cameras on a specific subject and follow them as a cinematographer would. There’s much technological gymnastics at play here as the engine jumps from camera to camera to keep the subject in focus. The seamlessness of this process can be attributed to the powerful processing technology on the Snapdragon platform.

Clear communication—in any language

Taking a professional quality photo or video isn’t the only way AI capabilities on the Snapdragon get involved in today’s smartphones, though. On video calls, there are multiple neural networks running simultaneously to make the call as sharp as possible. One neural network polishes up the speaker’s voice to crystal clarity, while another runs background noise suppression. After all, who wants the barking dog in the upstairs apartment to interrupt an important conversation with your boss? 

And because the noise suppression algorithm is running on an AI-powered neural network, it can be trained over time to understand different types of sounds and “de-noise” even better—successfully separating (and canceling) outside noises from the voices of the speakers having a phone conversation. That sort of sound quality is crucial on long-distance calls, especially when callers are speaking different languages. In these instances, usually a translator would be required: Previous technologies just couldn’t do real-time translation between languages. But this is another instance where the new platform sets the next generation of smartphones apart from their predecessors. 

The Snapdragon processor, for instance, begins translating speech as soon as it enters the microphone. Inside the device, neural networks knock out three important tasks: automatic speech recognition; language translation (Transformer network or BERT); and text-to-speech. Say two people are talking, but one speaks Mandarin and the other speaks English. Without ever knowing, a person’s English speech is transcribed into text, and then translated at the construct level to capture different meanings of different words across languages, ensuring the original context is kept intact. Lastly, that transcribed text is then converted to Mandarin speech. Voila: cross-border communication with ease.

Snapdragon real-time translation is cutting-edge. It’s complex but follows three general steps that are enabled by AI—and it all happens on device. 

“Everything is done on the device. It’s not only faster, but more secure, because your voice signature stays on the phone,” says Asghar.

Providing much-needed context

Of course, the true benefits of AI-enabled smartphones are only realized when machine intelligence is running continuously. Given the myriad ways people use their phones, neural networks have to be ready to go into action at a moment’s notice. New smartphones don’t only power the robust neural networks that enable functions like real-time translation.  They also run what’s called tiny AI, smaller neural nets that are always on, running on low power. 

“We call it contextual awareness,” says Asghar. “It’s sensing the world around you by using audio, and this is on 24/7. It can open up a lot of cool use cases.”

These tiny AI use cases are enabled on the Qualcomm® Sensing Hub inside the Snapdragon platform, capable of running these always-on AI algorithms. It can sense music playing in the background when you enter a coffee shop, tell you what song it is automatically, and then even match the ring volume of your phone to your environment, depending on how noisy your environment is. It can also be powered to do other conditional tasks, such as turning a phone into a baby monitor. With the Qualcomm® Sensing Hub, a new parent can set up their phone to automatically turn on lights or play music when their baby starts crying after waking up from a nap.

Regardless of surroundings, callers can now enjoy a clear call without the interruption of background noise. The AI-powered algorithms on the platform train neural networks to understand all types of noises (from keyboard typing to dog barking to sirens) all while tracking these noises and suppressing them.

By the end of 2021, analysts predict the number of smartphones in the world will equal half of the world’s current population. But the new smartphones coming off the production line will only be good to consumers if they’re better at understanding how we use our phones today, and what we need our phones to do. The way forward is by enabling AI to its full potential—and making our smartphone experiences finally live up to their name.

This story was produced by WIRED Brand Lab for Qualcomm Technologies Inc.

Snapdragon, Qualcomm AI Engine and Qualcomm Sensing Hub are  products of Qualcomm Technologies, Inc. and/or its subsidiaries.