Is Your Database Ready for Black Friday?

Is Your Database Ready for Black Friday?

With the holiday season approaching quickly, it’s an opportune time to discuss preparation for the busiest shopping season of the year, especially the typical surge of high traffic on Black Friday. Many businesses find themselves caught off guard by high holiday traffic, leading to a sluggish client experience at best, or production downtime at worst. Not only does this amount to a poor experience for your users, but it can also add up to lost revenue and diminished brand value.

Digital Reality

As our increasingly connected world becomes more reliant on digital services, expectations of speed and availability rise. Today’s users expect almost instantaneous response and feedback from your application(s), and if your environment isn’t capable of delivering, you might find yourself losing customers and revenue to your competition. 

Production outages aren’t all you should worry about during these events. You might attribute a simple slowdown to high load, but an impatient customer might perceive the situation as downtime. So even these soft events have a detrimental impact.

Black Friday: Preparation

To succeed, preparation is key, and that applies to your databases. There are a few things you can do in advance of the holiday season to be sure your systems are as ready as they can be. 

Monitoring

You should have monitoring and tooling in place well before Black Friday or the holiday shopping season. Without reliable metrics, not only is there no baseline for comparison, but you’ll also be blind to what is actually happening in your environment on the day of. We recommend Percona Monitoring and Management as a complete solution for monitoring the health and status of your systems. 

Load testing

In order to reliably determine how you will scale under various traffic conditions, you should load test your applications in advance. As always, the best time to find problems is when you’ll have ample time to address them, as finding problems during a high traffic event can be costly to your business. 

Failover

Murphy’s law is ever-present, so while the goal is to have a solid, reliable system that remains that way for the entire Black Friday period, occasionally there are occurrences beyond our control. As such, having a failover mechanism or tested recovery plan in place may end up being vital on the day of. 

Code freeze

We’ve all heard stories of the engineer who decides to implement production changes on a Friday afternoon, forcing an entire team to perform cleanup work over the weekend. The same applies to high-traffic events such as Black Friday, albeit on a larger scale. Best practice dictates a code freeze period leading up to and during large events, as stability is what matters during these times and you cannot ensure such stability amid changes leading up to the surge of traffic. 

Backups

A DBA is only as good as the last successful backup! This probably isn’t the first time you’ve read this sentence in one of my blog posts, and I promise it won’t be the last. Make sure you have a reliable and consistent backup mechanism in place, and make sure you have up-to-date and verified backups leading into any high-traffic event. With Percona XtraBackup and Percona Backup for MongoDB, we can help.

Double-check

To check and double-check that everything is as ready as it can be, it never hurts to have a second opinion from a trusted colleague. Trust, but verify — most big production outages are attributable to seemingly simple issues being overlooked and leading to a problem.

Black Friday: The Day Of

Staff for success

If you have an issue during the Black Friday period, restoring your application functionality as quickly as possible is critical. Having the right people standing by to monitor, tweak, and fix issues before those situations become outages is key to a successful high-traffic event. Small issues that are prolonged while waiting to get the right people in the room tend to turn into bigger problems. Many enterprise-level companies will make Black Friday an “all-hands-on-deck” event.

Remember the goal

In the heat of the moment during a database issue is not the time to get sidetracked trying to develop permanent fixes, or to complete a root-cause analysis to determine why the problem happened in the first place. During the Black Friday period, your goal is uptime, uptime, uptime, and there will be plenty of time after the fact for more permanent-level fixes or analysis of failures. Trying to permanently solve issues under pressure is never a good idea, and doing so can cause mistakes. Keep the system up and running during Black Friday, and just remember to come back afterwards during normal traffic periods for any permanent fixes or deep analysis. 

Failovers aren’t magic

We mentioned failovers in the Preparation section, and while having a failover mechanism or plan in place is critical to planning for high-traffic periods, keep in mind that failing over to a standby system should be a last resort. If you do have to fail over due to a production outage, remember that your database caches and buffers will need to warm up, which could lead to a period of reduced performance until the situation stabilizes.

Collect and store

If you do find yourself dealing with a database issue during Black Friday, make sure to collect and store all relevant logs, metrics, and other information that you will need later for a full root-cause analysis and review. It is easy in the heat of the moment to disregard this, but it will be incredibly important later when working toward long-term solutions. Don’t let this take precedence over the primary goal (uptime, uptime, uptime!), but don’t let it fall to the wayside either. 

Black Friday: The Aftermath

Once you have made it through Black Friday (or any other high traffic event), it is always good to analyze your traffic and usage patterns. You can use this data to help plan, enhance, and tweak your strategy for future events. 

Don’t forget to address any quick fixes put in place during the Black Friday event. The goal was to guarantee uptime, but now that traffic levels are back to normal and there is some breathing room, it’s a good time to address those quick fixes and turn them into more permanent solutions. 

Finally, holistically review how everything went, and learn from mistakes. Use this experience to make notes and build a better plan for mitigating problems and risks in the future. Whatever you do, don’t fall into complacency — there’s always another high-traffic event looming! 

Wrapping Up

For more information, download our Database Disaster Prevention Checklist, or contact us today. Percona Experts can help you plan and prepare for high-traffic events, giving you a trusted partner to help ensure your company’s success. 

Read the Database Disaster Prevention Checklist