As the Cookpad recipe service grows to serve more than 100 million people worldwide, with thousands of new recipes being published by our Authors (Cookpad’s content creators) each day, we are constantly seeking to improve how we determine which recipes to put in front of our users. Our intention is to help them find the recipes that they might like with considerably less effort. People come to Cookpad to help them solve the question of what should I cook?, and in some cases, users may not know exactly what they are looking for and are instead looking for some inspiration.

At Cookpad, a big part of our focus goes towards answering this question. The Cookpad Inspiration-Feed is the space that hosts all of the newly published recipes created by Authors from a particular region outside of a user’s immediate follow network. We know that some of our users come to the platform in search of new ideas and the Inspiration-Feed is designed precisely for this. It also serves to provide ongoing exposure to our Authors.

Figure 1: The Cookpad Inspiration-Feed.

Up until recently, Cookpad’s strategy has been to update the Inspiration-Feed with new content using a recency-based approach. The main purpose behind this was to support our Authors, providing them (and their recipes) with the highest possible opportunity for exposure to receive feedback on their efforts. Gaining quick exposure is an extremely important factor, as the recipes published are the fuel for our platform. As our platform gains in popularity however, particularly in our more active regions, the number of new recipes being published on any given day is now too large to practically consume. The ML team aims to capitalise on the value machine learning can bring to this use-case, so that our users can go from inspiration to cooking more seamlessly. To this end, we have been working on personalising the Inspiration-Feed to solve this challenge of supply and demand by both providing exposure for Authors, and matching this to the most appropriate audiences that are likely to engage with and express interest in those recipes.

The purpose of this blog is to give a high level overview of our approach and journey to launching and scaling our first machine learning powered personalisation system. So, without further ado, let’s dive into some of the details.

How do we implicitly learn from our user’s behaviours?

For any personalisation system to work effectively, you must first identify and extract how to best represent users’ preferences or tastes. At Cookpad, there are different types of engagements and interactions that users make when using the platform. These interactions allow us to implicitly learn about their interests and tastes. The most relevant of these signals have been dubbed by the ML Team as ‘Cookability’ signals. For example, there is the Cooksnap — where the user uploads a screenshot of their completed version of a previously published recipe. Other examples include saving a recipe and revisiting a recipe across different days and finally, taking a screenshot of a recipe to save for later. Each of these interactions serves as a signal of intent or interest in a particular recipe. We use these signals as well as more traditional click signals (in an anonymised manner) to understand what our users are interested in.

A brief overview of modelling and deployment

Given the task at hand, personalising the Inspiration-Feed can therefore be framed as a ranking problem. By carrying out this personalised re-ranking on only the newly published recipes each day, we can also effectively balance our Authors’ recipe exposure, ensuring that this key aspect isn’t affected. Candidate selection, therefore, does not constitute part of the solution and we are only interested in a fixed candidate (recipe) set over a given time frame.

Our current approach is to assign an equal weight to each of the ‘Cookability’ interactions. We chose this approach, in the first instance, for the purposes of simplicity. We may wish to change this at a later stage however, to incorporate different weights that are based around a particular business objective. In this way, further experiment iteration will allow us to learn how modifications to the various weights can impact key metrics.

Another important operational project decision was to start with a straightforward approach that has a high likelihood to yield positive results, in order to mitigate risks and avoid lengthy development cycles. We exploited the use of a listwise loss to optimise our model. Each training sample is constructed with one positive from the aforementioned ‘Cookability’ interactions, coupled with sampled negatives; the latter defined as recipes a user has not interacted with. Building on the principle of top one probability to mitigate the n! permutations of possible ranked lists (see here for more details) — our loss function can therefore be defined as:

This allows us to represent the distance between any two ordered lists of scores y(i) and z(i) respectively. The model we used is a simple three-layer Feed Forward network. This is trained to minimize the above defined loss. It takes as input both computed user and recipe features, with the latter composed of a combination of both text and image representations tuned on our recipe corpus data. The predicted output scores from the model can then subsequently be used to rank and recommend the appropriate candidate recipes for each user.

When considering the deployment of our personalisation engine, we also chose a similar philosophy to walk before you can run, eventually working our way up to a more advanced service design. A brief overview of the high-level architecture for our deployed system is shown in Figure 2. This demonstrates the evolution of the service architecture. Our current iteration is phase 2, where the streaming ingestion service now consumes recipe edit and publish events from Kafka, generates recipe features, and stores them in our feature store.

Figure 2: Evolution of Service Architecture.

The feature store is built on DynamoDB and houses all of our pre-computed recipe and user features. Recipe features, including both text and image representations, are processed in real-time and ingested via a streaming ingestion service that allows us to score and process the latest recipes published by users on our platform.

The recipe embeddings relating to those recipes that the user has interacted with, are averaged to generate the user features and this is updated on a daily basis and executed as a batch job. An outcome of this design choice is that our user representations are currently one day behind the latest user data. Simply increasing the frequency of how often the user embeddings creator job runs, would reduce this discrepancy and will be considered for our phase 3 design, but was not deemed to be a necessity at this stage.

Results

As with any personalisation project, the success of any developed system is assessed by determining its impact on key user and business measured metrics. Using the above detailed system, we extensively make use of A/B testing and run multiple iterations of online experiments across our active regions. The most pertinent metrics are detailed below:

• % Daily FAU (Feed Active Users) — Number of daily active users engaging on Feed.
• % Retention — Number of users visiting Inspiration-Feed.
• Number. of Bookmarks — Average number of bookmarks per user.
• Recipe Visits — Average number of recipes visited on Feed per user.
• Position — Average click position on Feed per experiment group.
• View duration — Average view duration of recipes across users per experiment group.

Looking into the performance numbers in Figure 3 below, you can see results from one of the latest experiments. The numbers shown represent the computed average of daily figures across the duration of the experiment, which typically operates over a four week period.

Figure 3: Experiment Results.

The five listed experimental features above detail the various approaches included in this particular experiment and are described further below:

• Aux_control: Current recency-based Feed.
• ML_cosine: Baseline personalisation model.
• ML_randomized: Sorting recipes randomly.
• ML_tf_content_clicks: Model optimised on click data.
• ML_tf_content_cookability: Model optimised on ‘Cookability’ signals.

Comparing the five experimental features side by side in each region allows us to assess the performance of our latest approach. In this particular experiment, we are interested in how the tf_content models perform against our recency-based Feed (aux_control), the simplest ML baseline model (ml_cosine), and a random model (ml_randomized). The latter is key in ensuring any deployed ML model outperforms a randomly sorted Feed.

As described previously, the tf_content models are optimised to re-rank the order of recipes based on the probability of generating a particular action or signal. The tf_content_clicks model ranks recipes with the highest probability of generating clicks and the tf_content_cookability works to generate the aforementioned ‘Cookability’ actions (bookmarks, cooksnaps, repeat recipe visits and screenshots).

For each of the five features, users are randomly assigned to each group upon their first visit to the Cookpad app at the start of the experiment. This allocation ensures fair results and comparisons between groups, with each user then remaining in their respectively assigned group for the entire duration of the experiment.

Looking at the numbers, we can see that for ID Release, the tf_content_cookability model performs best across the key metrics. We can see that compared with the recency-based Feed (aux_control), the ‘Cookability’ personalisation model yields:

• Approx 7% increase in user retention
• 10% improvement in average click position (lower means less scrolling)
• 12% increase in average recipe view duration
• 18% increase in the average number of recipe visits
• And finally a 3% increase in the average number of bookmarks

For RU region, our tf_content_clicks model performs best with:

• 13% increase in retention
• 48% increase in the average number of bookmarks
• 21% increase in the average number of recipe visits
• Approx 6% improvement in average click position
• Approx 3% increase in average recipe view duration

Following these positive performance numbers we have established ML Inspiration-Feed as the default experience with 100% rollout across both regions from the above experiment. We anticipate that this exciting development should inevitably lead to similar performance gains across the entire region as well as contribute to higher Premium service conversion rates. We hope to begin tracking this key business metric very soon as we scale our efforts.

We are now actively planning next steps, with further experimentation and rollouts to new regions coming soon! The ML Team feel that these improvements are just the beginning, as we grow our personalisation project across more regions and also integrate our learnings and developments with Cookpad Search. So, stay tuned for future exciting and impactful ML developments.

I would like to acknowledge the members of the ML team at Cookpad who all contributed to this work: Andrey Ponikar, Jose Navarro, Niall Twomey, Nadine Sarraf, Akbar Gumbira, Prayana Galih and Krystal Zeng. I would also like to acknowledge Alina hryshchuk, Petra Mala and Lutfi hernandi for all their help with web integrations, coordinating and conducting experiments as well as data analysis.

Note: We are also always on the lookout for talented engineers and applied researchers to join the Machine Learning team at Cookpad. Checkout our open roles here!