Making Time to Save You Time: How We Sped Up Time-Related Syscalls on Dynos

I work on Heroku’s Runtime Infrastructure team, which focuses on most of the underlying compute and containerization here at Heroku. Over the years, we’ve tuned our infrastructure in a number of ways to improve performance of customer dynos and harden security.

We recently received a support ticket from a customer inquiring about poor performance in two system calls (more commonly referred to as syscalls) their application was making frequently: clock_gettime(3) and gettimeofday(2) .

In this customer’s case, they were using a tool to do transaction tracing to monitor the performance of their application. This tool made many such system calls to measure how long different parts of their application took to execute. Unfortunately, since these two system calls were very slow for them. Every request was impacted waiting for the time to return, slowing down the app for their users.

To help diagnose the problem we first examined our existing clocksource configuration. The clocksource determines how the Linux kernel gets the current time. The kernel attempts to choose the "best" clocksource from the sources available. In our case, the kernel was defaulting to the xen clocksource, which seems reasonable at a glance since the EC2 infrastructure that powers Heroku’s Common Runtime and Private Spaces products uses the Xen hypervisor under the hood.

Unfortunately, the version of Xen in use does not support