Subprocess returncode is not detected when running Gunicorn with Uvicorn (with f...
source link: https://github.com/encode/uvicorn/issues/894
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Checklist
- The bug is reproducible against the latest release and/or
master
. - There are no similar issues or pull requests to fix it yet.
Describe the bug
When starting Gunicorn with Uvicorn worker(s), if the app uses subprocess
to start other processes and captures the output, their returncode
is in most cases 0
, even if the actual exit code was 1
.
To reproduce
Take this minimal FastAPI app (or replace with Starlette), main.py
:
import subprocess
from fastapi import FastAPI
app = FastAPI()
@app.post("/run")
def run_subprocess():
result = subprocess.run(
["python", "-c", "import sys; sys.exit(1)"], capture_output=True
)
return {"returncode": result.returncode}
Then run it with:
$ gunicorn -k uvicorn.workers.UvicornWorker main:app
Open the browser at http:127.0.0.1:8000/docs and send a request to /run
.
Expected behavior
The detected returncode
should always be 1
, as the subprocess always exits with 1
.
Actual behavior
In most of the cases it will return a returncode
of 0
. Strangely enough, in some cases, it will return a returncode
of 1
.
Debugging material
This is because the UvicornWorker
, which inherits from the base Gunicorn worker, declares a method init_signals()
(overriding the parent method) but doesn't do anything. I suspect it's because the signal handlers are declared in the Server.install_signal_handlers()
with compatibility with asyncio
.
But the UvicornWorker
process is started with os.fork()
by Gunicorn (if I understand correctly) and by the point it is forked, the Gunicorn "Arbiter" class (that handles worker processes) already set its own signal handlers.
And the signal handlers in the Gunicorn base worker reset those handlers, but the UvicornWorker
doesn't. So, when a process started with subprocessing
is terminated, the SIGCHLD
signal is handled by the Gunicorn Arbiter
(as if the terminated process was a worker) instead of by the UvicornWorker
.
Disclaimer: why the SIGCHLD
signal handling in the Gunicorn Arbiter
alters the returncode
of a process run with subprocess
, when capturing output, is still a mystery to me. But I realized the signal handler in the Arbiter
is expected to handle dead worker processes. And worker subclasses all seem to reset the signal handlers to revert those signals set by the Arbiter
.
I'm also submitting a PR to fix this: #895. It's just 3 lines of code. But debugging it and finding it took me almost a week. 😅
Environment
- OS / Python / Uvicorn version: just run
uvicorn --version
:Running uvicorn 0.13.1 with CPython 3.8.5 on Linux
(it's actually installed from source, for debugging) - Gunicorn version (also installed from source, for debugging):
gunicorn (version 20.0.4)
- The exact command you're running uvicorn with, all flags you passed included. If you run it with gunicorn please do the same. If there is a reverse-proxy involved and you cannot reproduce without it please give the minimal config of it to reproduce.
$ gunicorn -k uvicorn.workers.UvicornWorker main:app
Additional context
I'm pretty sure this issue #584 is related to the same problem.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK