Avoid overloading your servers while rolling out new versions

We’ve just rolled out a feature in Galaxy to allow containers to clean up resources as they are stopped, and we’ve published a package to allow Meteor apps to easily use this feature to gradually close client connections during deploy to avoid spikes of traffic to new containers. Read on for more detail on the feature who might find this useful.

Avoiding CPU usage spikes

When you deploy a new version of your Meteor app to Galaxy, we’ve always used a rolling restart to transition clients connected via Meteor’s Websocket-based DDP protocol from old containers to new containers. This means that all clients connected to a single container moved to new containers simultaneously when that container finishes shutting down and clients reconnect.

Some of our larger customers have found that this can lead to a large spike in CPU usage in their containers and database as many Livequery publications are started at once. Many of our users have found that transitioning their data layer to Apollo GraphQL from DDP/Livequery has reduced the statefulness of their servers and helped with these performance issues. But we know that many of you still enjoy using DDP/Livequery, so we built this feature to help you avoid these spikes during deploys and other container restarts.

Starting today, when Galaxy stops an app container (for example, because you’ve deployed a new version, or because we’re upgrading the underlying hardware), it will first send your container a SIGTERM signal and wait 30 seconds before killing the container. You can adjust that grace period from 10 seconds to 10 minutes on your app’s settings page.

We’ve published an npm package called @meteorjs/ddp-graceful-shutdown which you can easily add to any Meteor app (as long as you’re on at least Meteor 1.4). This package tracks incoming client connections on your server, and closes them one at a time, evenly spread out across your app’s grace period. Assuming the feedback is positive, we will likely make this be the default behavior of a future release of Meteor.

We hope this helps our larger users avoid overloading their servers when rolling out new versions!

One more thing: Improved Node 8 support for performance tuning

While we were improving our container environment to allow apps to receive signals, we’ve also updated the way that you can specify specific flags to Node in Galaxy, such as the V8 garbage collection flags that we recommend for performance. We now allow you to set these flags with the environment variable $GALAXY_NODE_OPTIONS as well as $NODE_OPTIONS, because Node 8 (which ships with Meteor 1.6) interprets the latter variable itself but exits with an error if it contains any of the V8 GC flags.