In the wake of our web performance study with Google and Uni Hamburg, we talked about the analytics stack behind our real-user monitoring (RUM) at code.talks 2019. The abstract and video are given below.

Users leave when page loads take too long. This simple fact has complex implications on online businesses everywhere, since page speed is hardwired to user satisfaction, conversion rates, and ultimately business success. But do you know the average load time for your users? How is it correlated to your conversion rate? And is it better or worse for mobile users?

In this talk, we recount our journey from a custom-built and ETL-heavy log analysis pipeline to an easy-to-use and interactive SQL data warehouse built on AWS Services. To provide a relatable end-to-end example, we first describe our own real-user monitoring pipeline from (1) data collection in the client and CDN over (2) data storage in S3 to (3) data analysis in Athena, sharing best practices and lessons learned along the way. Having painted the big picture, we then drill into the analytics part and demonstrate how to get hourly business KPIs and their correlations with real-user performance data from raw log data stored in S3 — prepare to be amazed! Our talk concludes with a brief overview of helpful and easy-to-integrate services for powerful dashboards (QuickSight) and real-time data ingestion (Kinesis).