SAP AI's strategy unfolds - on foundation models, cloud migrations, and how customers should respond

(Dr. Philipp Herzig, SAP Chief AI Officer)

My prior piece on SAP's AI direction brought clarifications I wasn't expecting (Behind SAP's 2024 AI strategy - Chief AI Officer Philipp Herzig on why cloud is tied to AI results, and the rise of the prompt engineer). But I ran out of space for two important topics:

• The emerging role of the Knowledge Graph in SAP's Foundation Model.
• Advice for SAP customers on AI next steps - though I did go into detail on how the prompt engineering role fits into SAP Business AI.

Since that article, I also issued a fresh podcast with ASUG CEO Geoff Scott and analyst Josh Greenbaum: What does SAP’s AI push mean for customers? We challenged ourselves to avoid the fun temptation of debating SAP's ups and downs - and instead to surface next steps customers should think about.

SAP deepens its NVIDIA partnership, and spells out the steps for SAP Business AI adoption

In part one of this article, I issued a rundown of AI news in and around SAP. Since that time, the EU AI Act has been formally ratified (you can check Herzig's views on this landmark legislation in part one). SAP has also amped up its partnership with NVIDIA: SAP and NVIDIA to Accelerate Generative AI Adoption.

But I thought the big takeaways last time were: Herzig's candid take on cloud and AI, and how he sees the near-term value of Business AI for customers. As Dr. Philipp Herzig, is now SAP's Chief AI Officer, said to me:

This is also why we made the decision that AI and cloud only come together, because otherwise you are dying in complexity. And you will not reap the benefits, or the cost to produce outweigh the benefits, and there's no return on investment.

Herzig shared how customers can achieve value from SAP Business AI in the near and mid-term:

Herzig and I discussed SAP's "mid-term" AI architecture, including the timeframes for the HANA Vector Engine and SAP's Foundational Model. But Herzig was quick to emphasize: he would like to see SAP customers achieve significant AI value, even without all those pieces. He sees AI near-term value in three categories:

1. Out-of-the-box - AI that just works. Using Joule as a workplace assistant might be one example, and/or SAP's job description generator.
2. Prompt engineering - Adjusting the AI output, but in a way that just about any SAP customer can do, without needing a data science team (there are different levels of prompt engineering; more on this shortly)
3. Retrieval Augmented Generation (RAG) type use cases, infused with customer or real-time data. This can sometimes require a data scientist to set up the proper data architecture (the first one of these is now running in SuccessFactors via Joule, pulling in data from help.sap.com).

The SAP Knowledge Graph - the last piece of the puzzle for SAP's Foundation Model

I won't rehash all the timeframes and AI adoption numbers again, but I do want to get into the SAP Knowledge Graph. Originally, I was expecting to hear more about SAP's Foundation Model early in 2024. But reading between the lines, it's clear now that the Knowledge Graph is a key piece of the puzzle, still under development.

As Herzig explained, the HANA Vector Engine already provides a means for incorporating customer-specific documents into AI output. But the Knowledge Graph will add richer context to the LLM:

How do you ground a Large Language Model -  doesn't matter which one - in the realities of the business? The documents, the master data, the business configuration. That determines the context of the business process that you're currently in. The next step, which looks already looks promising, is the Knowledge Graph.

Taking full advantage of enterprise data is the driver:

It comes later this year. The first thing we're doing: we will take the entire master data - so basically, the Data Dictionary of S/4, and put it into a comprehensive Knowledge Graph, and that gives us even more capabilities to ask questions and to reason about the business data, so to speak. Because now you understand, not only the data, but you have relationships between all the entities.

Herzig is optimistic about the relevance of output this AI architecture will generate:

The foundational model we are building is a prediction engine, based on tabular data. So you could look, for example, at open items in finance. And you can ask the model, 'Predict when the customer would actually pay.' So it becomes a prediction engine, in the context of the application.

Obviously, you also need the Knowledge Graph to do this. We have the first experiments right now that are very, very promising. But this is a topic that we are still working on. We plan to show some of those capabilities potentially at Sapphire. But to be honest, it's early days.

AI next steps - how should SAP customers respond?

However, Herzig points out plenty of AI-related features are already available to SAP customers. So what is Herzig's advice for those who want to move ahead with AI now? I had a customer ask my view on this, so I ran it by Herzig:

1. Start by consuming the things that can have some immediate impact, where you can see the benefits without a lot of deep data transformation - such as job description generation (the "low hanging fruit," if we can live with this buzzword that is well past its shelf life). Start with projects/features where you already have some advantages, where out-of-the-box AI services should plug in easily, or where your data is very clean.
2. At the same time, look hard at the so-called clean core transformation, and evaluate why it matters and really look at your cloud journey as a whole, not just SAP but overall. (Another way of looking at this is: examine your technical debt in a new light: access to analytics and AI services, not just for one app, but across applications).
3. Take an unflinching look at the state of your data platform and data governance, and why that's going to matter. Evaluate SAP Datasphere and data platform options. Line up your operational apps with a data platform journey.

Looking back, I should have added: establish an internal set of principles on AI ethics, as well as a means to audit and review AI output and results, whether from external vendors or your own in-house AI apps. Nevertheless, how does this stack up with Herzig's views?

I think we're pretty much aligned on that. The low hanging fruits usually correlate with our SaaS properties. Everything that is more coming from the past and lifting you up in the cloud, requires the respective cloud transformation.

This is also why we made the decision that AI and cloud only come together because otherwise you are dying in complexity, right? And you will not reap the benefits, or the cost to produce outweighs the benefits. And then there's no return on investment.

Datasphere specifically will be, going forward, our underlying data platform to bring SAP data, but also non-SAP data together in a standardized way that's of course, relevant for analytics. But it's also, the data foundation for all of SAP's Business AI.

What should customers do with AI? How about streamline their own SAP projects?

During our What does SAP’s AI push mean for customers? podcast, we got into this from a different angle: what AI/data challenges do SAP customers face, and how should they respond?  One topic Greenbaum and Scott both pointed to: applying AI and automation to upgrades and technical project management. Josh Greenbaum launched in with:

I want throw an idea on the table where I believe companies should start looking for opportunities. It really comes down to the following concept: find relatively new places in your enterprise where your data is new and clean, where you're doing new things. Don't try to take the old process and run with it and expect a result. Tomorrow, you should be cleaning up that data. But in the meantime, get started in some places where you have finite data and much greater control.

Greenbaum puts streamlining SAP implementations high on that list:

I'm going to throw AIOps into that mix. I'm very interested in what's happening in implementation. Before we started recording this, we were talking about the transformation suite at SAP - the mix of Cloud ALM, Signavio, and LeanIX - how SAP is really pushing that forward, as their strategy for doing streamlined implementations. 

But when you look at what you can do with the data that you start collecting, from the conference room pilot, through the implementation into the go-live and then the renewal, that's an extremely clean data set - that's finite. And you can do a tremendous amount of change detection, monitoring, analyzing customizations and project documentation with AI. That's going to really make things sing, and be a very important part of your infrastructure, and a very important part of your innovation cycle. So it's not that sexy, but it's a very doable place for AI-driven innovation.

Geoff Scott added:

I think it's a wonderful walk-before-you-run kind of analogy. Can we use some of these tools to help do data cleansing? Can we use these tools to help drive test scripts? Can they maybe do testing on our behalf? Can they help elevate the quality of our S/4 migrations, when we think about moving to the next generation of cloud ERP?

Can AI help customers get to the cloud? Scott:

Can we use these things to do some of that which is better in our control as technology, you know, leaders and practitioners, to understand we got the output we want, before we try to unload these tools on a bunch of data that we don't really understand? We worry about how it's going to give you predictive results.

I think there's a lot of value in using these tools to help us do the one thing that's on most SAP customers minds: more than 40% yet to go to cloud. Can you get me there faster, and can I use these tools to push the cost of migration down, and make it match my business needs more closely? It's a win-win, if it's done accurately.

My take

It's an interesting juxtaposition between SAP's emphasis on cloud-based AI - up against the points Scott and Greenbaum made about using AI to actually get to the cloud in the first place. I'm not sure those points are in conflict, however. I know that SAP is taking a hard look at how to ease migrations to S/4 in particular.

But all software vendors could stand to invest more heavily in AI for project delivery and customer success monitoring. If we're going to have "co-pilots," shouldn't we be able to ask those co-pilots questions about our licensing status, acquiring more user seats, or digital/data access and API costs?

SAP's strong stance on "AI must be in cloud" will warrant further discussion in 2024. It's hard to say which stance is right when so much about the ROI of AI is still unfolding. I see where SAP is coming from here; I'm a big advocate of cloud applications, particularly of the multi-tenant/SaaS variety. The ability of cloud applications to consume AI services is just one of the reasons.

However, when it comes to AI, my current take is that it's the caliber of your internal data structure, and level of overall systems messiness and customizations, that is the biggest determinant of your AI success. That's why I've argued that SAP focus more on delivering AI value through BTP rather than RISE, so that any SAP customer on any release can access AI services via SAP's BTP cloud platform. I'm not alone in those views, but Herzig's points are also important.

Herzig is drawing directly on SAP's own hard-won AI experience, and where projects have been successful. He's not wrong to focus on ROI. AI projects can get expensive quickly, and data quality obstacles are high on the list. It's also easy to see how deploying AI functionality to virtually all SuccessFactors or S/4HANA public cloud customers, on the same releases, with relatively standard configurations, will be much more efficient than trying to cater to a wide range of on-prem customizations.

I would advise customers to provide feedback to SAP (and its partners!) on what they want to see. SAP has evolved its views on this over the last year, so make sure you are heard. Make no mistake: SAP is clear on this point for a reason. You or I might disagree with a roadmap or deployment decision, but it's undeniably helpful to have such candid/clear communications of intent from a vendor. That is far preferable to vague statements of inclusiveness without specifics. Herzig's level of on-the-record depth here sets the right tone.

Most enterprise software vendors are trying to make today's generative AI tools more accurate and relevant. This is why SAP's push around foundation models and knowledge graphs is worth tracking. Being able to work with tabular data is not necessarily easy for LLMs (even with RAG); SAP is pursuing some interesting breakthroughs here that are worth watching.

The use of knowledge graphs to enhance foundation models is on the cutting edge of AI research overall (see the October 2023 research paper, Towards Foundation Models for Knowledge Graph Reasoning). If I'm an SAP customer, I'm encouraged to see SAP pressing into new areas that could make AI more relevant/accurate for enterprise concerns. But such research takes time, and runs into obstacles that must be overcome. This is the purpose of these AI articles - to lay out where SAP is headed, but also note what's available now (or soon). I hope these reviews are helpful; you can expect more in the SAP Sapphire timeframe.

As for the podcasts, you can expect another this spring as well: I want to press Scott and Greenbaum on where SAP's partners should fit into this innovation storyline. Stay tuned...