The promise of open clouds: Faster innovation, faster delivery

Learn how open clouds reduce latencies to client device, improve customer and device (IoT) interactions and speed up innovations from the edge to the data center.

Zammo.ai today launched a conversational AI platform that makes it simpler to engage customers via multiple voice assistants, interactive voice response (IVR)/telephony, and chatbots without having to write any code.

That no-code approach, provided via the integrations the company has embedded within its software-as-a-service (SaaS) platform, enables organizations to create workflows that span multiple conversational AI technologies without the aid of an internal IT team or a systems integrator, said company CEO Alex Farr. “No one from IT is required,” he said.

That approach provides the added benefit of eliminating the need to force customers to embrace a specific conversational AI platform, noted Farr. Organizations can add support for conversational AI platforms based on customer preferences, he said.

Over time, most customers are likely going to prefer to engage organizations using some sort of conversational interface such as Apple Siri or Amazon Alexa. As these platforms become more widely employed by consumers, it’s only a matter of time before additional customer experiences are going to be routinely employed to, for example, find the nearest location to have a car serviced.

At this point, it doesn’t appear any single conversational AI platform will emerge to dominate all others. Many users already switch between platforms multiple times a day depending on whether they are standing in a room or using their smartphone to send a text. Organizations that provide customer service, however, can never be sure what conversational AI platform might be employed at any given time.

Conversational AI platforms are, of course, core to many digital business transformation initiatives. The simpler it becomes for a customer to make a purchase or request a service, the more likely it becomes they will engage. Many an instinct to impulsively acquire something has been ignored simply because it takes too long to log into a website to order it. Today some organizations are even sending reminders to end customers to reorder goods and services via a conversational AI platform as an alternative to text messages or emails.

Not having this capability potentially creates an impression in customers’ minds about how modern an organization is as they begin to incorporate conversational AI into their everyday lives. However, a request made by a speech interface doesn’t always mean the customer wants their request answered via the same medium. Sometimes they may be asking an organization to send them a form via email.

Low code, big impact

Regardless of how speech interfaces are employed, it’s clear that line of business units are becoming more empowered to automate tasks without any help from internal IT teams. The goal is not so much to eliminate the need for IT teams as much as it is to reduce the backlog of projects that IT teams are being asked to take on and then maintain. In most cases that is being accomplished using no-code tools embedded within a platform. However, power users, also known as citizen developers, are employing low-code tools to build more complex applications. The relationship between IT teams and end users is rapidly evolving.

It may be a while before conversational AI platforms become the dominant user interface, but in certain use cases they are already having a profound impact. Most end users for the foreseeable future will continue to employ a mix of speech and graphical interfaces to interact with applications. However, the more conversational AI platforms learn about the habits and interests of an end user, the more proactive the platforms can become, especially if end users have opted into a service that provides that capability.

The challenge organizations now face is striking the right balance between being actually helpful versus overreaching in a way that some customers might perceive to be intrusive or, for that matter, simply downright creepy.

The promise of open clouds: Faster innovation, faster delivery

Learn how open clouds reduce latencies to client device, improve customer and device (IoT) interactions and speed up innovations from the edge to the data center.

This article is part of the VB Lab Microsoft / NVIDIA GTC insight series.

With the rapid pace of change taking place in AI and machine learning technology, it’s no surprise Microsoft had its usual strong presence at this year’s NVIDIA GTC event.

Representatives of the company shared their latest machine learning innovations in multiple sessions, covering inferencing at scale, a new capability to train machine learning models across hybrid environments, and the debut of the new PyTorch Profiler that will help data scientists be more efficient when they’re analyzing and troubleshooting ML performance issues.

In all three cases, Microsoft has paired its own technologies, like Azure, with open source tools and NVIDIA’s GPU hardware and technologies to create these powerful new innovations.

Inferencing at scale

Much is made of the costs associated with collecting data and training machine learning models. Indeed, the bill for computation can be high, especially with large projects — into the millions of dollars. Inferencing, which is essentially the application of a trained model, is discussed less often in the conversation about the compute costs associated with AI. But as deep learning models become increasingly complex, they involve huge mathematical expressions and many floating point operations, even at inference time.

Inferencing is an exciting wing of AI to be in, because it’s the step at which teams like Microsoft Azure are delivering an actual experience to a user. For instance, the Azure team worked with NVIDIA to improve the AI-powered grammar checker in Microsoft Word. The task is not about training a model to offer better grammar checking; it’s about powering the inferencing engine that actually performs the grammar checking.

Given Word’s massive user base, that’s a computationally intensive task — one that has comprised billions of inferences. There are two interrelated concerns: one is technical, and the other is financial. To reduce costs, you need more powerful and efficient technology.

NVIDIA developed the Triton Inference Server to harness the horsepower of those GPUs and marry it with Azure Machine Learning for inferencing. Together, they help you get your workload tuned and running well. And they support all of the popular frameworks, like PyTorch, TensorFlow, MXNet, and ONNX.

ONNX Runtime is a high-performance inference engine that leverages various hardware accelerators to achieve optimal performance on different hardware configurations. Microsoft closely collaborated with NVIDIA on the TensorRT accelerator integration in ONNX Runtime for model acceleration on NVIDIA GPUs. ONNX Runtime is enabled as one backend in Triton Server.

Azure Machine Learning is a managed platform-as-a-service platform that does most of the management work for users. This speaks to scale, which is the point at which too many AI projects flounder or even perish. It’s where technological concerns sometimes crash into the financial ones, and Triton and Azure Machine Learning are built to solve that pain point.

Making ML model training across on-premise and multi-cloud, or hybrid and multi-cloud, easier with Kubernetes

Creating a hybrid environment can be challenging, and the need to scale resource-intensive ML model training can complicate matters further. Flexibility, agility, and governance are key needs.

The Azure Arc infrastructure lets customers with Kubernetes assets apply policies, perform security monitoring, and more, all in a “single pane of glass.” Now, the Azure Machine Learning integration with Kubernetes builds on this infrastructure by extending the Kubernetes API. On top of that, there’s native Kubernetes code concepts like operators and CI/CDs, and an “agent” runs on the cluster and enables customers to do ML training using Azure Machine Learning.

Regardless of a user’s mix of clusters, Azure Machine Learning lets users easily switch targets. Frameworks that the Azure Machine Learning Kubernetes native agent supports include SciKit, TensorFlow, PyTorch, and MPI.

The native agent smooths organizational gears, too. It removes the need for data scientists to learn Kubernetes, and the IT operators who do know Kubernetes don’t have to learn machine learning.

PyTorch Profiler

The new PyTorch Profiler, an open source contribution from Microsoft and Facebook, offers GPU performance tuning for popular machine learning framework PyTorch. The debugging tool promises to help data scientists and developers more efficiently analyze and troubleshoot large-scale deep learning model performance to maximize the hardware usage of expensive computational resources.

In machine learning, profiling is the task of examining the performance of your models. This is distinct from looking at model accuracy; performance, in this case, is about how efficiently and thoroughly a model is using hardware compute resources.

It builds on the existing PyTorch autograd profiler, enhancing it with a high-fidelity GPU profiling engine that allows users to capture and correlate information about PyTorch operations and detailed GPU hardware-level information.

PyTorch Profiler requires minimal effort to set up and use. It’s fully integrated, part of the new Profiler profile module, new libkineto library, and PyTorch Tensorboard Profiler plugin. You can also visualize it all Visual Studio Code. It’s meant for beginners and experts alike, across use cases from research to production, and it’s complementary to NVIDIA’s more advanced NSight.

One of PyTorch Profiler’s key features is its timeline tracing. Essentially, it shows CPU and GPU activities and lets users zoom in on what’s happening with each. You can see all the operators that are typical PyTorch operators, as well as more high-level Python models and the GPU timeline.

One common scenario that users may see in the PyTorch Profiler is instances of low GPU utilization. A tiny gap in the GPU visualization represents, say, 40 milliseconds when the GPU was not busy. Users want to optimize that empty space and give the GPU something to do. PyTorch Profiler enables them to drill down and see what the dependencies were and what events preceded that idle gap. They could trace the issue back to the CPU and see that it was the bottleneck; the GPU was sitting there waiting for data to be read by another part of the system.

Examining inefficiencies at such a microscopic level may seem utterly trivial, but if a step is only 150 milliseconds, a 40-millisecond gap in GPU activity is a rather large percentage of the whole step. Now consider that a project may run for hours, or even weeks at a time, and it’s clear why losing such a large chunk of every step is woefully inefficient in terms of getting your money’s worth from the compute cycles you’re paying for.

PyTorch Profiler also comes with built-in recommendations to guide model builders for common problems and possible. In the above example, you may simply need to tweak DataLoader’s number of workers to ensure the GPU stays busy at all times.

Don’t miss these GTC 2021 sessions. Watch on demand at the links below:

• Enterprise ready ML Model Training on NVIDIA GPUs across Hybrid Cloud, leveraging Kubernetes (Saurya Das)
• Profiling PyTorch Models for NVIDIA GPUs (Geeta Chauhan PyTorch Partner Engineering Lead Facebook AI Gisle Dankel Software Engineer Facebook Maxim Lukiyanov Product Manager, PyTorch Profiler Microsoft​)
• Inferencing at Scale with Triton Inference Server, ONNX Runtime, and Azure Machine Learning (David Langworthy Architect Microsoft, Mahan Salehi Deep Learning Software PM NVIDIA, Emma Ning SPM Microsoft)

VB Lab Insights content is created in collaboration with a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. Content produced by our editorial team is never influenced by advertisers or sponsors in any way. For more information, contact [email protected].