Doctrine has been founded 5 years ago, as many fast-growing tech startup, its codebase has evolved at a high pace. Thanks to a strong core value, the challenge of the status quo, we have successfully overtaken some technical challenges in our history like our migration from Angular to React in 2017.

Despite that, during 2020, we started facing a limitation on the SEO side described in the following section.

Our Custom pre-rendering system

At that time, our client architecture was based on a React Single Page App with a client-side routing based navigation. This was fine for the user experience but not for search engines bots.

Indeed with that pattern, the bots needed to load the page and process the entire JavaScript to acquire its content. We discovered that the more the bots waited for content the fewer pages they could index.

A solution to this issue was to introduce a pre-rendering step when a search engine bot was detected.

As this is a common issue, some companies like prerender.io sells it as a service.

For a company like Doctrine with the vision of opening justice around the world, we quickly faced a scaling and pricing issue with those services considering the millions of pages we had to expose to search engines and so pre-render.

That’s why we decided to create a custom system based on a scheduled job in Airflow that can visit URLs of interest in a headless browser (Puppeteer), get the complete HTML rendered, transform into a static HTML (without any JavaScript dependency) and then store it in S3.

Thanks to this system, when search engines were detected, we only had to get the content from S3 and respond with the static HTML content. It was blazing-fast (under 200ms).

Unfortunately, after a year of use, it turned out that we needed a dedicated engineer to maintain this system. Even with this investment, we reached a glass ceiling of around a million pre-rendered pages, which was far from our initial objective of multiple million.

As shown in the chart below, reporting bot requests, we didn’t reach 100% of the pages served with the snapshot version.

Server Side rendering

This is when Victor Monginet entered the game joining Doctrine a year ago. He drafted an RFC around how Server-Side Rendering (SSR) could solve this pain.

Helped by the comments of the entire Doctrine tech crew, we quickly identified some challenges that we needed to crack to confirm SSR was viable for our SEO constraints.

To avoid re-inventing the wheel, we decided to use a framework. At that time, thanks to its large open-source community around the project and the company behind (Vercel), Next.js was the de facto React.js framework to build a complete server-side rendered web application with a best-in-class developer experience and many built-in features. That’s why we decided to choose it for the migration.

Which migration strategy?

In a nutshell, Doctrine’s front-end was around 100+ HTML page templates implemented using 100k lines of TypeScript. It was crystal clear to us that in order to be able to move forward we needed an incremental migration strategy. That meant we needed a way to make previous and new systems live together for months.

If we put aside the necessity to separate new code and old code in the codebase, the main challenge was to handle this migration at the infrastructure level.

For pages, we reached out to the new Next.js server. When the page had already been migrated it was returned otherwise there was a fallback to the old responsible EJS rendering view.

For API REST end-points we decided to keep the architecture simple so we proxied all the requests to the old server through the new one with the Next.js’ rewrite feature. 6 months later, it’s now handled at the ingress level.

The drawback of adding an extra HTTP call was small enough to pay the cost during the migration.

Is server-side rendering fast enough?

One of the main reasons to do this migration was to perform better on the SEO side. Meaning responding quickly to a lot and a growing number of pages known by search engines bots.

To double down this point we created a benchmark of 357 decisions URLs that we considered important to us and we computed an average response time.

We then deployed a Proof Of Concept version on a custom environment in our Private Network. We were aware of the extra cost that exposing this environment to the internet would involve, so considering that we added an overhead of 50ms to the SSR version.

To be sure where the time was spent on the server we split the trace into 3 steps:

SSR → Pure Server-side rendering part

SANITIZE → SEO Optimization of the HTML generated by SSR

REFDECISIONS → Loading of recommendations on the right side of the page which are normally rendered asynchronously

As expected, we found out that the new system was slower than the first one based on cached HTML with a cold pre-rendering. However the results were good enough to get rid of the complexity and the maintenance cost of the previous one.

From an Ambitious Idea to Execution

Backed by all this knowledge learned during the last month of 2020 we launched our migration with a first milestone on the 11th of January 2021. It was time to test all our theories against the real world with the most important page of our intelligence platform: the decision page.

After some UI glitches and tracking issues fixes, the migrated decision page was completely functional.

It was then time to onboard all the developers to migrate the code base as soon as possible without blocking the product roadmap.

After four intensive months with hundreds of commits from 4 developers, we achieved the migration on April 2, 2021.

We honored our contract with the product to keep shipping features during the migration. Now it was time to check the improvements on the SEO side.

Speed matters in business

Search engines are often considered black boxes as we don’t exactly know what could impact their algorithms. To overtake this problem, you should monitor all relevant metrics and see how they impact your ranking. The main metric monitored for this migration was the number of days between the time we allow a crawler to visit an URL and the time it does.

With the benefit of hindsight, we can see the number of days to discover a decision page drastically dropped during this year.

Conclusion

Six months after this migration we couldn’t be happier than we are now, easily onboarding new engineers on this codebase and smoothly scaling our SEO system.

In 2020, it took 62 days on average to discover 75% of freshly whitelisted content that we wanted to index on search engines. After the migration in 2021, it went down to 15 days.

We are even prouder of our value about challenging the status quo that fosters an ambitious initiative backed by RFC. No matter if it’s hard or long to get it done if it helps us to scale.

We also used this migration to streamline some front-end practices improving our code quality as well as cleaning up some flows to improve our product UX.

A key bonus is the pleasure of using cutting-edge technologies for current or future talents that will join us.