Freezing Python’s Dependency Hell in 2018

The challenge of managing dependencies in Python has been described by many different people . It’s a storied past, which has left a history of conflicting posts across the web. Even with state-of-the-art best practices, you can still end up in Dependency Hell when adding new dependencies, because there is an open issue for pip first reported in 2013 to implement dependency resolution. The usual series of workarounds ends up like this relevant XKCD:

At Instacart, we’re automating our best practices in Lore , so Data Scientists and Machine Learning Engineers can trivially replicate their work on any computer in any environment, without spending time in Dependency Hell. This helps multiple people collaborate on a single project and switch from one project to another as easily as changing directories. It also eliminates random production issues from unintentional changes to secondary dependencies.

When every App maintains it’s own virtualenv, individual contributors are empowered to manage their dependency updates reliably, without having to update the entire company’s codebase to the latest version. There are still Python 2 vs 3 debates ten years after release, partly because old monolithic code bases hold new projects back.

Best Practices (in 2018)

virtualenv
pip freeze > requirements.txt
runtime.txt

Lore’s open source command line takes care of everything required to satisfy these steps, without adding any environment variables, updating PATHs, or extra commands. It’s a natural workflow that uses the current working directory, and it’s trivial to install: pip install lore

All lore commands will pass extra arguments to their delegate. Using lore means you don’t need some combination of brew, apt-get, anaconda, miniconda, pipenv, pyenv, pyvenv, venv, virtualenv etc. Lore is lightweight and modular by design and will not add any other entries to your App’s requirements.txt . It stands on the shoulders of pip, pyenv and virtualenv behind the scenes to avoid reinventing those wheels.

Modern Software Architecture

Why not use ____?

• brew, apt-get and other OS package managers don’t allow you to specify your Python minor or patch versions, and will force upgrade you regularly.
• Docker sort of solves this problem, by freezing your OS image, but this still doesn’t allow specific control of Python or dependency versions.
• Pyenv gives us fine grained control of multiple Python versions, but doesn’t deal with package dependencies.
• Pipfile looks promising for managing package dependencies, but is under active development. We may adopt this as an alternative if/when it reaches maturity, but for the time being we use requirements.txt .
• Anaconda requires a large installation up front, and while monolithic dependency management that just works is great if you’re the only person working on the code, it makes it harder for other people to replicate your work, unless you also use requirements.txt . That means pip is all that is actually necessary for other contributors to collaborate.
• autoenv, direnv, .venv and others that automagically change your $PATH or other environment variables prevent access to your system Python (and packages) when you’re in those project directories, which will break any shell script that uses #!/usr/local/env python .
• virtualenv is great, but it’s easy to forget a workon after a cd and pip freeze the wrong requirements, or any other number of mistakes people make “just that one time”. We prefer encapsulating common workflows in bulletproof one liners.

It is a best practice to freeze all requirements to exact patch versions in requirements.txt , rather than letting pip install the latest version of dependencies, because those new versions will introduce breaking changes. It’s not worth the off chance that you’ll get a new performance optimization, or bug fix for a problem you don’t even know you have. It’s difficult to track down the source of these breakages, because it’s not in your own code and the changes were not tracked or intentional. The same logic applies to patch version changes in Python itself.

How To

If you want to use these best practices for any project, it takes about 2 minutes to complete the one-time setup:

If you’re creating a brand new App, lore init my_app will create the directory my_app with a template scaffold from scratch, --bare skips scaffold creation for existing projects. Anyone who checks out a Lore App will instantly be at home. When they change to the directory, and run lore test for the first time, all dependencies will be installed in a brand new virtualenv on their machine.

Lore produces reliable builds for CI testing and deployment as well. Python versions and virtualenv packages are Russian Doll cached on the machine for fast and efficient repeatability across many projects.

When you import lore , all dependencies will be checked to fail fast if there is a version mismatch or unsatisfied requirement. In development or test environments, new requirements are automatically added to requirements.txt and pushed down the CI pipeline.

Lore is rigorous. If you manually launch a python process from outside the virtualenv and try to import an App’s module, it will reboot Python with the correct version in the correct env with the correct dependencies, or die trying (with a helpful error message). Nobody should be wasting time chasing spurious errors caused by subtle dependency bugs.

Of course, all of this is configurable via environment variables, configuration directories, hidden .env files, or Python code. We believe strongly in convention over configuration, and also that rules are meant to be broken.

Full disclosure

Lore dependency management is limited on Windows to the currently installed system Python version, since pyenv is not Windows compatible. We’d love to fix this.

Lore adds a few hundred milliseconds to application startup, because it reboots Python into the virtualenv. If that time matters to you, launch lore directly in the correct virtualenv with the appropriate path like ~/.pyenv/versions/3.6.6/envs/my_app/bin/lore . You can find this path and more in lore env .

If your system looks like the XKCD at the beginning of the article, you might want to uninstall everything and follow the suggestions in brew doctor . Lore will work around these issues without a cleanup, but it’s nice to have a working system too. If it’s not a project you work on, but a Python script you rely on having installed system wide, you can still shove it in its own virtualenv.