From zero to hero: Building and scaling Groupon Russia

Evil Martians spent six years with Groupon Russia in total: 18 months from the early days of a small Russian group-buying startup to acquisition by the global corporation and the next 4.5 busy years of worldwide expansion. Martians described this project as “work inside the eye of a tornado” (meaning its insane growth)—and we had liked this skydiving in a core tech team role. This article reveals the secret of solving scaling problems and implementing promising features in record-breaking or even last-minute time, and fighting against frequent DDoS attacks under that pressure.

Business model to hit the market

Groupon is considered a web phenomenon and a household name for collective discount services and unprecedented (even for ecommerce startups) growth rates. The startup got the breaks in the credit markets free-fall when an increase of client base was a tickler. The business model was found to be extremely appealing for customers and surprisingly scalable. It combined the social idea of providing a way to help local businesses with cash flow and sales generation when banks were not lending, let people in local communities find interesting things to do in their city, and a practical goal to secure profits. Founders didn’t want to sell overstock gadgets or deal with shipping and returns—and focused on services.

While pushing hard for growth, Groupon won one US state market after another, soon expanded outside the United States, and settled in dozens of countries. The main strategy of grabbing the international market was to buy already successful local discount services. It enabled fast-track and higher integration pace in a competitive battle. The most successful projects helped avoid waste of time in trying to guess the local consumers’ preferences.

Berries of Darberry

Groupon Russia started in 2010 as Darberry, a discount service for products. Back then, similar services popped up all over, but Darberry had some advantages.

First, it started earlier than others and at the right moment—the Christmas shopping period. In a week after the launch, there were over 100 daily orders. In two weeks, the total turnover reached $17K and the margin turned out to be two times higher than expected. These numbers kept growing, so the founders started to envisage the project’s evolution. They researched the US market and met Groupon, focused on the service coupons sector—with no logistics, warehouses, couriers, and returned items problems but with the higher 50% margin.

Darberry adopted that approach to build a profitable and dynamic business. That idea took off: a couple of months later, the project worked in seven cities and reached up to $250,000 in monthly turnover.

Darberry interface back then

Second, Darberry stood to build a legal business from scratch, with transparent processes available for any audits. Third, Darberry adopted the aggressive marketing strategy. Finally, they had a strong and tech-savvy platform built together with Martians (when Groupon evaluated Russian coupon platforms for acquisition, they named Darberry’s tech among the best).

In August 2010, Groupon Inc. acquired Darberry, renamed it, and rebranded it as Groupon Russia. The corporation’s target was to cover all large regions, including the largest European geographies like Russia. They chose this acquisition among over 30 similar projects: some had investments from professional players or a large audience. But Darberry won with the skilled team, business transparency, and the platform that could support rapid scaling.

Darberry offers

The Russian team kept developing the new brand and covered 22 Russian cities and 3 Ukrainian ones soon.

Step-by-step rebranding

Team for growth

When Darberry shifted from products to services, it became clear that it was driving the nail because they met an exponential progression. It was vital to support this growth by tech. But the main thing the project needed was team strengthening.

“We lacked strong people to implement our ideas. Our colleagues in CityDeal, a European project acquired by Groupon, opened offices in 140 European cities in three months. They slept in sleeping bags in their office. We were looking for people who can show the same blazing performance.”—Elena Masolova, ex-Board Member, Groupon Russia

Evil Martians were the team that was going to thrive in that environment. Earlier, Martians helped one of the Darberry founders with fast performance optimization for one of his previous projects.

“We met in the spring of 2010. Back then, I launched a dating app similar to Mark Zuckerberg’s FaceMash. It was a hobby, but the project hit the market quickly: the app started to get over 200 requests per second. My tech knowledge wasn’t enough to support it: sometimes, the app was down for hours. I needed specialists who knew Ruby on Rails and were able to deliver results faster. I found the team through Google search. On the same day, I called them, and Martians made a number of changes that allowed the application to work like fury. That’s then I realized that I wanted to work with these guys.”—Oleg Kozyrev, ex-CTO, Groupon Russia (from Inc. Russia’s interview)

Solving scaling and high load challenges

Immediately after joining the Darberry team, we learned two obvious facts. First, we started seeing insane growth. Second, we expected even more boost soon due to audience and competition increase.

Later, when Darberry turned to Groupon Russia, the growth rate metric (in addition to the revenue metric) remained the key. According to it, Russia kept pace with large economies like China and India—and of course, these growth rates outstripped the local competitors. The project grew by 25-30% every month for several years, and it took much effort to maintain this colossal growth.

Ruby on Rails to support scaling

The platform’s first version was minimalistic—just to make sure the idea was viable. Martian’s main challenge was to kick-off its technical evolution. We had a lot of creative freedom; the founders essentially handed us a blank check to implement a number of make-it-work hacks on the technical side. We had to build the tech stack that fit perfectly the “startup mode” technologies which allowed the client to grow faster, and the architecture that supported changes in any component independently.

In a month and a half after joining the project, we rebuilt the Ruby on Rails-based platform from scratch. Back then, many startups had become successful, choosing Ruby on Rails as part of their tech stack, and Martians nailed it as one of the few experts. Despite the Darberry’s seeming simplicity—listing page, product page, order page, and payment processing—the project happened to be technically sophisticated due to many integrations, constant refinements, and the high load with tens of millions of requests per day. In the middle of our cooperation, the codebase had over 150K lines of code.

Martians helped solve the key scalability and flexibility challenges and brought “real” smart Ruby on Rails, with all its capabilities to prototype and implement ideas swiftly and cutting-edge tools for building the modern web. We can call those old days the Golden era of Ruby on Rails, so the Groupon Russia tech success story is, in fact, the technology success story.

One of the Rails’ technological benefits was its focus on time-to-delivery, code simplicity, and developer friendliness. It offered an extraordinary development speed, so startups could build and rebuild everything rapidly to test their hypotheses. Besides, the enthusiastic Ruby and Rails community already had the necessary infrastructure and on-trend features like easy automatic deployment, the ability to run an immediate deployment rollback in case of failures, automated testing as a first-class citizen, great frontend development support, convention over configuration, and many others.

At the peak

The second task was to optimize the peak loads. Popular promotions generated a tremendous demand. For example, two successful offers—free talk-times from a popular instant messenger and a huge discount on the world’s leading antivirus—yielded a record number of requests: dozens of thousands per second. In America, users of Reddit and similar forums user posted the screenshots of the offers in Russian and asked how to get these promotions.

We focused on meticulous and accurate work with our databases—Darberry used PostgreSQL as a primary RDBMS—because high load problems backfired on them in the first wave. Due to this, we managed to “rescue” the project several times. We frequently profiled our queries, analyzed our database structure, sped up and measured performance. We aimed to provide a sustainable infrastructure for high scalability and reliability with the primary and standby database servers running on highly performant bare metal machines for the best “bang per buck”.

“This collaboration with Evil Martians allowed us to grow faster and earn more.”—Elena Masolova, ex-Board Member, Groupon Russia

“Native” DevOps

In our early days, we were regularly engaged in projects where the load was more or less predictable and four to five bare metal or virtual servers were enough to serve any foreseeable surge in demand. We could maintain them together with a third-party developer operations team. But as Groupon Russia grew it kept needing more servers. When we reached dozens, we faced the challenge of managing them all in one, using the “Infrastructure as Code” (IaC) approach (e.g., to deploy repeatable configurations instead of treating every machine in a unique way, setting it up and updating the configuration manually). We decided to nail it in a “native” way—with the help of our own Martian squad.

We began to recruit a team ready for high availability to place the current infrastructure on repeatability footing. Chef configuration management tool gathered momentum, providing the opportunity to describe a configuration as a code and apply it automatically. We migrated the project to Chef.

It allowed us to bring new server configurations into production in minutes instead of hours and days and update configuration instantly and smoothly for several bare metal servers at once.

Later, we also used a dynamic service discovery based on Consul. That allowed us to kick out failed or out-of-service applications and servers from rotation automatically in a matter of seconds and re-add them later when they were back online.

The new DevOps team, envisioned by Martians as a new unit to cover a new service, also initiated 24x7 monitoring based on Zabbix for server metrics and New Relic RPM for application metrics and helped with provisioning new servers and maintaining the infrastructure at a scale.

Later, we detected and responded to all service issues and downtime alerts in a 24/7 mode since every minute of lagging or downtime could affect Groupon’s sales significantly, bearing in mind the number of active users.

It helped shape a systematic approach in building an infrastructure that could work longer under pressure. We mixed Groupon’s own bare-metal servers with custom configurations, dedicated servers from our hosting provider, and cloud ones to optimize costs and performance. We arranged benchmarks to pick out the less performing and less necessary machines.

Acquiring new users

As competition in the daily deals space became more intense, Groupon Russia blew its competitors out of the water, garnering as many successful promotions as possible. Every day there were dozens of diversified offers, typically a combination of a low price and a large number of coupons or a higher price for a smaller coupons’ volume.

Personalization

Offering personalization brought a significant surge in revenue. We upgraded it gradually: initially, we suggested offers depending on a city, later—on an audience’s segment, and eventually, we learned how to show promotions specifically tailored for each user. A special algorithm, based on a history of purchases and geolocation, picked the most relevant offers to display.

Groupon Russia offers

Emailing

Groupon Russia relied on aggressive marketing, where mailing campaigns played a critical role: all registered users received daily emails with appealing offers. Their hallmark was a funny copy with short stories about each of promoted services. Many subscribers confessed they were addicted to these stories and read them every day.

Adjusting the interface

The platform’s smart and convenient interface with many features helped attract new users. The project team stuck to performance marketing, so we used to develop several design versions for each landing page, each button, and each element.

We A/B tested them all the time, and a variant with a higher conversion rate and a better attraction for users went live.

Short time-to-market

Groupon country teams communicated a lot and shared successful ideas. It helped Groupon Russia to be a trendsetter in successful promotions. Competitors copied almost all offers immediately, so the launching speed was crucial. We had to implement changes extremely fast. Sometimes we designed and launched a promotion with a new partner and new logic in three evening hours, developed major interface iterations in three days, and deployed many features in just one day. Once, we completely rewrote one key website section three times in just a week just to make sure it performs its best.

Insane competition

Groupon Russia was a revenue-driven business, so every project’s engineer clearly understood the impact that speed and each feature or iteration had on revenue. That business-oriented approach later became a must for all Martian customer projects.

Animations to watch

Revolutionary frontend technologies and tools hit the market, allowing us to experiment with smart and sophisticated animations, which were not very common in the web of old. Our team was one of the first to use them meaningfully in the ecommerce product.

It was a way not only to attract the buyers’ attention but also to solve specific interface problems: for instance, to provide a user some time between action and its result, to show additional objects correlation, and to create a more natural feeling. E.g., data-entry forms “shivered” when the field was filled-in incorrectly, indicating that something went wrong.

For sure, we used a lot of animations for landings and promotional campaigns. We designed animations for every holiday. For Christmas, we created snowflakes that started to fall when users hovered over a logo (in order not to distract users from the main flow). We made two snowflakes sprites: in focus and unfocused, moving at different speed rates to create a parallax effect. We designed a beating heart in the Groupon Russia logo for Valentine’s Day and measured the time between beats to animate it correctly.

One of the Christmas landings

Those tricks were so successful that we often met similar animations in rival projects and even discovered our own code copied and pasted.

The Christmas animation code was copied by one of the Russian competitors with our huge JS file that contained many libraries. The competitor used it for its critical landing page that they were widely promoting to get more traffic. We replaced falling snowflakes with small Groupon Russia’s logos for them to enjoy.

And, of course, we tried animations for Easter eggs. For instance, one of the landing pages allowed applying the Konami Code—a sequence of buttons to enable a cheat or other effects, popular in the early Nintendo games era. Then used, you could see the Chuck Norris animated image.

Social network app

We also integrated with Odnoklassniki, one of the most popular social networks in Russia. We developed the app’s first version in an insane record time, despite a complex engineering process. The application was a version of Groupon Russia with an isolated frontend and a partially isolated backend, created according to the network’s strict requirements. The app became complex and sophisticated since we had to integrate with their unique payment system. Odnoklassniki had virtual points that could be bought for real money and exchanged for games and services like Groupon Russia. The app smashed popularity records and gained almost the same audience—around 15 million users—as the Groupon Russia website.

Launching the Martian frontend

It was the Groupon Russia project that kick-started the Martians’ frontend team. Initially, we focused on Ruby on Rails exclusively, with mainly the backend development in mind. All other frontend, design, and DevOps tasks were given to third-party teams, while we were focusing on our core competency. It worked fine for small startups, but the Groupon Russia project was always aching for more and more new designs, features, and interfaces. We clearly understood the correlation of good interface and good sales, and the best and fastest way to implement the design was teamwork and an own frontend team.

Andrey Sitnik, a frontend and a Ruby software engineer back in the day, and who is now a well-known frontend pundit and the author of PostCSS, Autoprefixer, Browserslist, and Logux, joined Evil Martians to launch and recruit the Martian frontend team. The number of frontend tasks grew every day, so each frontend newcomer gained combat experience with Groupon Russia. Today, the Martian frontend team’s size is equal to the backend one, and this situation reflects the modern trends in engineering (years ago, a 1:2-1:3 proportion was quite typical).

We shaped our frontend fundamentals: frontend engineers, backend engineers, and UI designers are always “in the same boat” and “tied together” at all work stages. They choose the best ways of feature implementation, terms and requirements, visual effects, the flow between screens, and so on. Simultaneously, we refused to hire full-stack engineers to keep the focus and engineers’ specialization: code, application logic, data and databases, deployment—for backend engineers; markup, interfaces, JavaScript, styles, animations—for frontend specialists.

Frontend tech stack for system building

The main frontend tasks were to build dozens of landing pages and change the interface constantly to reach a higher conversion. This intensive work required a simple tech stack that could nevertheless support all modern frontend features like animations, curves, and gradients—we needed cutting-edge tools and libraries for that.

Back then, the Ruby on Rails community acted as a catalyst for frontend evolution. Rails was in the groove when, so everything about new advanced web development features appeared there ​​first. With Rails, Martians could build the best flow and UX using new programming and markup languages like CoffeeScript, Haml, Sass, and automatic assets compilation. (Only a few years later, the priority in frontend innovations passed to Node.js.) Thanks to the community and dozens of new tools and approaches, we had the best and the most optimized frontend among all coupon platforms.

To simplify the project’s frontend code with a large codebase, we began to break styles and scripts into blocks. We even designed the Evil Blocks JS framework—it’s out-of-date as of today, but at its time, it started the whole generation of Martian “nano-libraries”. We were inspired by BEM, a modular approach to web development, although we did not use that stack—and we anticipated dividing a user interface into independent blocks in React. We started implementing automatic assembly from scratch via third-party libraries since it was a long time before the standard Rails Assets Pipeline.

Addressing the business analytics challenges

When the project ended its startup phase and started to work as part of a large corporation (which, however, did not pause the rapid growth), we began to set the ground for the platform’s technical evolution on analytics.

Features for the core team played an essential role. We built an easy-to-use admin dashboard for merchants with many useful features and stats. Managers from local Groupon projects in other countries recognized it as one of the best. Merchants could copy a successful promotion from one city to another in a couple of clicks, change some parameters and leave the same images and design. We built shareable statistics tools and a vertical and horizontal reporting system.

Groupon Russia’s office

Martians tried many external services for sending emails and eventually integrated with the one that fitted the best and built a customized interface to facilitate email management. We implemented an application to monitor advertising campaigns in VKontakte (another extremely popular social network in Russia). We required access to the social network’s API to test before the official campaign’s launch. Since we got more effective campaigns, the social network began to provide a similar service to other businesses.

Undertaking a mobile mission

Today, over 50% of sales of almost all ecommerce platforms are generated from mobile apps, and Groupon, with its 75%, is no exception. But in 2010, we had the most traffic from desktops, and even marketing campaign schemes were desktop-focused: with links in emails and website banners. Therefore, our goal was to optimize the marketplace for mobile devices.

End-to-end mobile UX was the brand new concept back then. The first iPhone and 3G penetration had spurred its development. Before that, simple and light HTML was a standard, and we didn’t miss that stage as well.

Groupon Russia was the first large marketplace to work with mobile apps’ SPA and a smart mobile interface. The iPhone couldn’t provide an API for native development at that time. Therefore, we built the mobile version with Zepto.js—a minimalist JS library for mobile devices with an API that resembled jQuery. One of Martian engineers discovered and fixed many bugs in it, so the library’s authors invited him to join the core team. We designed a mobile frontend with a special design and lighter features. Due to this, we could profile the mobile version and make it fast enough.

Facing billing tasks

One of the trickiest and most critical challenges for the project was billing and payment systems integration—a large Martian team was involved in those tasks. Unlike simple ecommerce payment schemes, Groupon Russia engaged more complex mechanisms, which included the client’s business, service fees, and a certain payment policy. In addition, we had to provide users with the greatest possible range of options to pay for coupons.

Right after the project launch back in 2010, Russia could not yet boast the wide bank cards adoption. Soon, users started mastering online card payments, and the penetration rate of electronic payments grew even faster than in other countries. But at that stage, we had to come up with alternative payment schemes like payment kiosks.

A kiosk interface

Kiosk payment billing was extremely tricky—for example, you could deposit only a limited round sum of money—100, 500, 1000 rubles. We produced large portions of code to handle all edge cases correctly. The system generated an invoice number for a user, but after payment there was often a leftover that we needed to transfer somewhere or a shortage that froze the payment.

At the height of the service, we had up to fifteen very different payment gateways and providers, including banks, telecoms, cash desks, kiosks, ATMs, e-money systems, and more. The integrations were not an easy ride since it was a new mechanism for many payment systems. The integration manuals typically were incomplete, so we invented many schemes on the fly. We also faced completely new kinds of payment like cryptocurrency. Return process with its million details was another troublemaker.

In addition, our billing systems and the platform were a perennial target of cyber abuse and DDoS attacks.

Fixing a DDoS problem

The main security goals were to combat frequent DDoS attacks—it seemed that most of them originated in unfair business practices. The peak of the attacks fell within the Groupon Russia active growth stage. Most of them were professional SYN-flood attacks to clog main channels and more simple but the same annoying HTTP attacks.

To prevent simple attacks, we constantly monitored traffic, optimized the areas targeted for attacks, and blocked IP addresses and network segments. For more complex attacks, we used firewalls and a third-party smart anti-DDoS service that filtered all the traffic through an HTTP proxy and hid our servers’ original IP addresses.

Meeting support needs

For the first time, we started solving helpdesk tasks. To bring tech support to the higher level, we integrated with Zendesk, a popular helpdesk software (also written in Ruby on Rails). Fast and clear feedback from users was critical, so among the variety of solutions—from simple forms to fill to forums and widgets—we opted for a third-party SaaS service. We even built a Ruby gem—a wrapper around the Zendesk API—for a better UX.

Also, Groupon Russia decided that it would be the best practice for Groupon’s support specialists and engineers to work in a single team. Engineers could access customer’s tickets to fix their bugs and solve issues directly. This approach helped save our and customer’s time and address the challenge better.

Nailing the business integration

One of Martian’s goals was the platforms’ integration meaning its transferring from one business to another. The first was integration with the European Groupon platform, and then—with the American one. It was a sophisticated process since the systems were completely different.

One of Groupon’s managers explained it like “It looked like we took a Ferrari to pieces, inserted them into a Mercedes, and then plugged it into a Ford”.

In many cases, the technologies and features were grounded on a different architecture or approach. Therefore, we had to simplify many interface areas and fall back to simple scripts and HTML5.

Working out business handover job

Groupon Russia was the first big project where we completely mastered the entire business handover process from Martians to the core team. When the startup turned into a company that went public, Martians designed a scheme of the project’s transition to in-house engineering and maintenance.

We gradually reduced the number of our team members and activities and helped the core team with engineers and CTO recruitment: composed the job opening texts, scrutinized CVs, joined interviews, and shared our feedback.

The next step was the new team training and knowledge transfer. We taught best engineering practices, the basics of DevOps, the details of all processes and technologies. Then, we practiced this approach for each product development project we took part in.

Even when the core team began to work independently, we joined them from time to time to help with task planning and distribution, complex load issues, and database availability.

New open source goals

The project gave birth to Gon, an extremely popular Ruby gem that provided a dead-simple way to pass Ruby code variables to the frontend code. Previously, to solve that seemingly simple task, engineers used bulky self-written solutions. Then, while exploring interesting applications or estimating them in customer projects, we often met this Ruby gem in one of four Rails applications.

We can also mention Visibility.js—a javascript library to detect if a browser tab is currently active for a user, to run heavy animations and massive re-calculations if the tab is visible only.

We had many patches and commits to various open source projects, including the Rails framework itself. We helped a Ruby gem thinking_sphinx integrate with Sphinx, an open source search server. Besides, we published the Easing functions guideline and interactive cheat sheet that served as a pillar for web developers to choose animations for their interfaces.

Organized storm

The Groupon Russia team increased rapidly in tandem with business growth—from six to six hundred employees. The same was true for our team—over the years, we managed to gather up to 12 Martians to work on the project. The team’s fast expansion and the high development pace called for significant shifts in the engineering process.

In previous projects, we had a steady pace and streamlined methodology: weekly sprints, specs for each iteration, quality of code evaluation, and code review cross-checks. But in this project, many things like new payment systems, features, and promotions were implemented and rewritten several times in a matter of hours. We couldn’t afford to fix the requirements in that situation. We discussed tasks on calls or in messengers, put them in a task tracker, and implemented them immediately. We shrunk the chain to achieve faster results and switched people between tasks to reduce the workload. At the early stage, we even skipped the QA stage for the sake of speed—we deployed everything to production, and if we had too many exceptions, we rolled it back.

We also automated deployments and kept doing them often, following the Continuous Deployment principle. However, sometimes we had to deploy a major feature (with thousands of commits)—in these situations, we first ran the deployment and database migration on one or several staging servers before a production release.

Over the six and a half years, we slowly moved from short-term to long-term planning and classic agile software development. The Groupon Russia’s approach that focuses on product and business results, where each developer understood the impact on conversion and revenue—became our standard modus operandi in all of our customer projects.

We also sampled the benefits of the remote and distributed teamwork: we flew to Asia to survive the Russian winter and worked in different time zones for half a year.

Giving remote work a try

“The coolest thing about the project was a phenomenally strong team. The highest speed and growth rates were very difficult to generate and support, but the great team we managed to build coped with this challenge perfectly.”—Viktor Lysenko, ex-CEO & Founder, Groupon Russia

In the Groupon Russia era, our educational project—online and offline courses titled “Brainwashing”—gained a lot of traction. Evil Martians were one of the first to delve into the new field of professional coding courses in Russia. We were going to teach the experienced developers and help them find the right track and make significant progress in their fields. For us, it wasn’t a business project but rather a social program since we were interested in generating unique and helpful professional content.

Brainwashing courses

More to the point, we knew it worked: we witnessed our students’ great achievements and skills level-up since we had a chance to work together on some projects, supported by Martians, and meet them at Ruby-related events or job interviews.

---

Truth be told, we did not expect Darberry to score a big win at the very beginning. But then it became the project that identified the best of Evil Martians. Besides, we evolved as a business, increased and diversified our team, became even more enthusiastic in open source, and gained many well-recognized customers.

Today, successful startups can quickly achieve high performance and a surge in demand, but back at the time, Groupon Russia’s growth rate was staggering: in just 18 months, the project climbed from zero to $15 million. Once Martians joined the project, its audience grew from 200 thousand to 18 million users.

At the start, marketplace had over twenty thousand purchases per day. As Groupon Russia added new cities (sometimes a few on a same day), this figure increased to several hundred thousand sales per day. During the busy seasons like New Year holidays and in times of especially successful offers, we counted daily purchases in millions.

It was a unique experience of working inside the eye of a tornado. The hyper-growth pressure forced us to build infrastructure fast, make changes on the fly, and write high-quality code from scratch—it was no time for “trial and error”. Despite the arduousness and severity, everyone in the team had the feeling that we were working on a unique, cutting-edge project that is critical for the industry—and so it was.

“Back then, we chose Evil Martians for cooperation, and our further experience revealed that it was the right thing to do. Evil Martians is probably the strongest team in web-dev. They deliver a complex product of outstanding quality, where everything just works as expected.”—Viktor Lysenko, ex-CEO & Founder, Groupon Russia

The Groupon Russia team managed to build strong sales and create a valuable product. At the same time, they allowed engineers—both core and Martians—to deliver the impeccable work that gave substance to all the bold plans for growth.

It turned out to be the right strategy, as the challenge “from zero to millions of users” can only be solved when everyone is working toward the same goal.

Reach out to us to discuss the ways Evil Martians can support your project in a fast-growing environment and become a core or emergency tech team to scale your business rapidly and effictively.