shell script to graceful restart gunicorn with source code reloading

shell script to graceful restart gunicorn with source code reloading

Posted 2021-08-21 | stdout

I always thought restart gnicorn is as easy as kill -HUP <pid>

But it's not as easy as it seems.

First of all the code might be loaded in master process then fork'ed into workers, so HUP signals won't reload the source code.

Then there might be some bugs and your code won't boot correctly, the worker will quit and drags the master process with it.

I discovered a gem from github issues that can spawn new worker first, check if new code loads correctly then restart the whole rack.

According to the docs you need to send several signals and blah blah, the author tried to explain in great detail of how it works rather than providing a simple option to do so.

So I wrote a shell script to do exactly that instead.

#!/bin/bash

oldpid=$(<./deploy/gunicorn.pid)
# forget the old kill -HUP 
kill -USR2 $oldpid
sleep 1
timeout=5
while [[ $timeout -gt 0 ]]; do
  newpid=$(<./deploy/gunicorn.pid.2) && kill -0 $newpid
  if [[ $? -eq 0 ]]; then
    break
  else
    sleep 0.5
    (( timeout-- ))
  fi
done;

# shutdown old
echo "success. newpid=$newpid"
kill -WINCH $oldpid
sleep 1
kill -TERM $oldpid

The script will check for new master pid to be alive then kill the old master in a maximium timeout of 5 seconds.

It works as perfectly to replace the simple HUP signal as intended.