Enterprises’ urgent need is for startups to help solve getting more machine learning (ML) models into production. That’s because 87% of data science projects never make it into production. Algorithmia’s 2021 enterprise trends in machine learning of 750 business decision-makers found 22% say it takes between one and three months to deploy an ML model into production before it can deliver business value. Furthermore, 18% say it takes over three months to get a model into production. Delays getting ML models into production are symptoms of larger, more complex problems, including lack of production-ready data, integrated development environments, and more consistent model management. According to IDC, 28% of all AI and machine learning projects fail because of these factors. Closing the gaps in MLOps and across the entire model lifecycle process creates a lucrative new market opportunity for startups, valued at $4 billion by 2025. According to Dr. Ori Cohen’s research, there’s been $3.8 billion in funding already.

The state of MLOps shows startups in the lead

Cohen, lead researcher at New Relic, recently published an exhaustive analysis of the MLOps landscape, The State of MLOps. He hosts the analysis on AirTable for ease of viewing and querying the data set he’s created. Selecting the Category option under the Views menu shows the five categories of companies included in his analysis. Cohen’s analysis is shown below, with companies sorted by category.
The following are insights from the State of MLOps analysis:

• 88% of the State of MLOps are startups, dominating every category in the analysis and leading funding. ML Platform startups lead all categories on funding with $3.4 billion. Databricks, DataRobot, and Algorithmia have together raised $2.9 billion alone. Data Monitoring is the second-most funded area of MLOps, with $116.3 million raised to date. ML Monitoring is the third-most funded MLOps category with $105 million. The average funding level by MLOps startup is $110 million, based on the State of MLOps analysis.
• Data Ops/Data Engineering is the dominant persona MLOps companies concentrate on today. Half of all MLOps companies are concentrating on Data Ops/Data Engineering as their primary persona. 14 of the 17 companies concentrating on this persona are startups. Amazon SageMaker and Google Vertex AI are the largest MLOps products to attract and sell their solutions to this persona. $3.5 billion in funding is driving new solutions for this persona, 93% of all funding in MLOps. Data Scientist/ML Engineer is the second-most targeted persona, with 13 companies focusing on these roles’ needs. Microsoft Azure and IBM OpenScale concentrate on the Data Scientist/ML Engineer persona in their solution development and messaging.
• Most MLOps startups are concentrating on Tabular Data first and then expanding into other data types to differentiate. The State of MLOps shows a common progression MLOps startups make from mastering Tabular Data with their unique Data Governance, Data Monitoring, ML Monitoring, ML Platforms, and Serving Platforms first, then expanding into other data types. In addition, startups most often add in Data Quality, Data Integrity, and Pipeline Integrity to further differentiate themselves from the many startups who start with Tabular Data as their main data focus.
• MLOps is a market ripe for Private Equity investors looking for M&A opportunities and investors looking to get into AI. Cohen predicts vendor consolidation in the MLOps space, with the largest competitors buying mid-size companies. He predicts that mid-size MLOps companies will begin buying the smallest ones to become more valuable to the largest companies. His analysis of the state of MLOps shows three acquisitions already. The gaps enterprises face moving models into production require a scale level that favors mid-tier and larger startups. Look for Private Equity investors to fund mid-tier MLOps leaders into aggregator roles, acquiring multiple MLOps startups at once to create valuable acquisition for larger vendors who need the Intellectual Property (IP) and patents smaller, faster-innovating startups can provide.

The goal of MLOps is to manage and accelerate the lifecycle for analytics and ML models from development into production. Enterprises aren’t getting the yield rates or scale from ML models they’re spending months creating because they are too many data quality, data integrity, data model management, and a series of other challenges that block their progress. Startups bring much-needed insight, innovation, and urgency to solving these problems, receiving $3.4 billion in funding to date. Vendor consolidation in MLOps is inevitable as larger, slower-moving companies look to startups for the innovative spark and insight they need to energize their platforms and deliver the scale and solutions their enterprise customers need to get more value from their ML models.

This article is part of a Gaming Insights series paid for by Facebook.

Gaming is a highly competitive industry. At Facebook, we see that games advertisers are often competing for the same audience, using similar creative strategies.

Whether they’re producing many ads to fight fatigue, optimizing successful concepts, or copying the competition, these popular creative strategies might lead to some success but are reactive in nature. It also makes it hard to stand out from the crowd, with many ads looking the same. In fact, one study showed that 56% of gamers say nearly all or many of the ads they see are repetitive[1].

By moving towards a more proactive, structured approach to ad creative you can understand what works and — more importantly — why. You can gather unique data to formulate distinct points of view that can be scaled within your organization and ultimately help drive long-term, sustainable growth.

Expand beyond executional strategies

But before I get into our new framework for experimenting with ad creative, I want to explain why it’s more important than ever. The changing ads ecosystem means that businesses will find it harder to deliver personalized ads and accurately measure campaign performance.

Reduced signals also lessen the ability to target specific audiences and optimize ad creative based on their behavior. Therefore games advertisers need to evolve their strategic approach, placing learning at its core, to develop creative that’s more broadly applicable to audiences.

Creative Prototyping: Experiment to drive success

Developed by Facebook’s Creative Shop and Marketing Science teams, Creative Prototyping is a way to intentionally experiment, uncover new creative territories and drive success. At a tactical level, it involves conducting structured experiments using the Ask, Make, Learn, Adapt framework.

Some of you might recognize this from The Big Catch Playbook, where we showed advertisers how to use the Creative Prototyping framework to test motivation-led ad creative.

This approach has an immediate impact on media efficiency: A meta-analysis of creative experiments on Facebook platforms found winning assets developed through testing and learning had a measurably lower average cost per ad recall, cost per action intent, and cost per action compared with the alternative assets.[2]

Let’s take a deeper look at the different stages of the framework.

1. Ask: Develop a learning agenda and craft creative hypotheses

The first step is to create a learning agenda, ie. decide what questions to ask. Your agenda should prioritize long-run structural creative learnings over short-run, campaign-specific ones.

Once you have a defined agenda of what you want to learn, you can prioritize the items and build out hypotheses to test.

2. Make: Design the creative assets based on your hypotheses

Once you have your hypotheses, it’s time to build the assets. There are three key rules to follow when creating your assets including isolating your creative variables and finding the balance between distinction and similarity.

Remember, you’re not designing final campaign assets, but prototypes to help you learn how to build the final campaign.

3. Learn: Design your creative testing methodology and analyze results

Now you need to design a test that will either prove or reject the hypotheses. Tests should be consistent, executed across all groups and should consider historical learnings, among other best practices.

There are multiple test methodologies, depending on the hypotheses but once you’ve executed, you’ll want to explore primary and secondary learnings to iterate.

4. Adapt: Determine future implementations of learnings

At this stage, you’ll decide whether you need to do another round of Creative Prototyping, or if you’re ready to implement your learnings into business-as-usual campaigns.

You should create a “Creative Prototyping log” to record all learnings. This should become your central repository for insights and best practices. Remember, the goal is to build a learning muscle that’ll help drive success over time.

Case Study: How Babil Games elevated the impact and creativity of their campaign through Creative Prototyping

Best known for its titles Nida Harb and the Strike of Nations series, Babil Games wanted to take a more structured approach to testing different ad creative to help them make more informed decisions about their advertising and player acquisition. Working closely with Creative Shop, Babil Games used Creative Prototyping to test the best ways to create compelling Facebook ads.

For the “Ask” part of the process, they developed a learning agenda for Strike of Nations, narrowing down their questions and hypotheses according to position, concept, and execution.

Following this, they landed on a hypothesis that sought to explore the effect on performance of placing their brand logo into their ads. To test this. they developed four unique creative approaches (“Make”).

While each creative looked slightly different, they all had a similar composition in terms of the gameplay, aspect ratio, length, and end cards. The only variable that changed was how they integrated their brand logo, thus allowing them to effectively attribute performance changes to its presence.

Once the test was run, and they were in the “Learn” phase, they saw that the card and logo creative had a 24% lower CPI and 27% lower cost per registration than no logo (primary metric). It also drove higher registration volume and purchases (secondary KPI).

Therefore, Babil validated their hypothesis that incorporating the brand logo was good for their performance, and understood more about the best creative strategy to do this.

But they didn’t stop there. As part of the “Adapt” stage, they tested a new agenda item to understand how integrating gameplay into CGI footage impacted performance. They saw that incorporating gameplay into their creative outperformed pure CGI, thus validating their hypothesis and providing them with an additional learning to incorporate into their business-as-usual creative strategy.

Key learnings

As highlighted above, the advertising ecosystem is changing and ad creative has never been more important in driving effective campaign performance. By taking a proactive and structured approach to creative development game advertisers can get ahead of the competition and ultimately drive success.

Dig deeper: Learn more about the Creative Prototyping framework, including step-by-step instructions, best practices, and case studies in our new playbook, available to download for free here.

[1] Source: Online survey responses of 728 established mobile gamers in US, from “Mobile Gaming Behavior Post COVID-19” by Interpret (Facebook IQ-commissioned online survey of 13,246 mobile gamers ages 18+ across BR, CA, DE, FR, JP, KR, UK, US, VN Jul–Oct 2020).

[2] Source: Facebook Internal Data, December 2019

Lewis Tutssel is Head of Gaming, EMEA Creative Shop at Facebook.

VB Lab Insights content is created in collaboration with a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. Content produced by our editorial team is never influenced by advertisers or sponsors in any way. For more information, contact [email protected].