Streamline Your Django Settings With Type Hints: A Pydantic Tutorial
source link: https://www.toptal.com/django/streamline-your-django-settings-with-type-hints-pydantic-tutorial
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Streamline Your Django Settings With Type Hints: A Pydantic Tutorial
Python 3.5 offers type hint support at runtime, allowing runtime-type-checking packages, like pydantic, to validate data and settings. The results? A smaller, more maintainable codebase and a scalable approach to adding new environments.
Django projects used to frustrate me because I lacked a robust and scalable way to add new environments. By bringing together pydantic and Python 3.5 type hints, I built the powerful foundation I needed.
As described in PEP 484, type hints support static analysis, but these same annotations are also available at runtime. Third-party packages like pydantic offer runtime type checking that uses this additional metadata. Pydantic uses Python type hints to help manage settings metadata and perform runtime data validation.
This pydantic tutorial will show the far-reaching, positive effects of using pydantic settings management with Django.
Our configuration adheres to the best practices described on the Twelve-Factor App website:
- Define nonconstant and secret configurations as environment variables.
- In development environments, define environment variables in a
.env
file and add the.env
to.gitignore
. - Use the cloud provider’s mechanisms to define (secret) environment variables for the QA, staging, and production environments.
- Use a single
settings.py
file that configures itself from the environment variables. - Use pydantic to read, check, validate, and typecast environment variables onto Python variables that define the Django configurations.
Alternatively, some developers create multiple settings files, like settings_dev.py
and settings_prod.py
. Unfortunately, this approach does not scale well. It leads to code duplication, confusion, hard-to-find bugs, and higher maintenance efforts.
Using the aforementioned best practices, adding any number of environments is easy, well-defined, and error-proof. Although we could explore a more complicated environment configuration, we will focus on two for clarity: development and production.
What does this look like in practice?
Pydantic Settings Management and Environment Variables
We now focus on an example in both development and production. We show how each environment configures its settings differently and how pydantic supports each.
Our example application requires a Django-supported database, so we need to store the database connection string. We move the database connection configuration information into an environment variable, DATABASE_URL
, using the Python package dj-database-url. Please note that this variable is of type str
and is formatted as follows:
postgres://{user}:{password}@{hostname}:{port}/{database-name}
mysql://{user}:{password}@{hostname}:{port}/{database-name}
oracle://{user}:{password}@{hostname}:{port}/{database-name}
sqlite:///PATH
In our development environment, we can use a Docker-contained PostgreSQL instance for ease of use, while in our production environment, we will point to a provisioned database service.
Another variable we want to define is a boolean, DEBUG
. The DEBUG flag in Django must never be turned on in a production deployment. It is intended to get additional feedback during development. For example, in debug mode, Django will display detailed error pages when an exception occurs.
Different values for development and production could be defined as follows:
Variable Name
Development
Production
DATABASE_URL
postgres://postgres:mypw@localhost:5432/mydb
postgres://foo1:foo2@foo3:5432/foo4
DEBUG
True
False
We use the pydantic settings management module to manage these different sets of environment variable values depending on the environment.
Preparatory Steps
To put this into practice, we start configuring our development environment by creating our single .env
file with this content:
DATABASE_URL=postgres://postgres:mypw@localhost:5432/mydb
DEBUG=True
Next, we add the .env
file to the project’s .gitignore
file. The .gitignore
file avoids saving potentially sensitive information in source control.
Whereas this approach works well in our development environment, our production environment specification uses a different mechanism. Our best practices dictate that production environment variables use environment secrets. For example, on Heroku, these secrets are called Config Vars and are configured through the Heroku Dashboard. They are made available to the deployed application as environment variables:
After that, we need to adjust the application’s configuration to read these values from either environment automatically.
Configuring Django’s settings.py
Let’s start with a new Django project to provide the essential structure for our example. We scaffold a new Django project with the following terminal command:
$ django-admin startproject mysite
We now have a basic project structure for the mysite
project. The project’s file structure is as follows:
mysite/
manage.py
mysite/
__init__.py
settings.py
urls.py
asgi.py
wsgi.py
The settings.py
file contains boilerplate code that allows us to manage the application configuration. It has many predefined default settings that we must adjust to the relevant environment.
To manage these application settings using environment variables and pydantic, add this code to the top of the settings.py
file:
import os
from pathlib import Path
from pydantic import (
BaseSettings,
PostgresDsn,
EmailStr,
HttpUrl,
)
import dj_database_url
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
class SettingsFromEnvironment(BaseSettings):
"""Defines environment variables with their types and optional defaults"""
DATABASE_URL: PostgresDsn
DEBUG: bool = False
class Config:
"""Defines configuration for pydantic environment loading"""
env_file = str(BASE_DIR / ".env")
case_sensitive = True
config = SettingsFromEnvironment()
os.environ["DATABASE_URL"] = config.DATABASE_URL
DATABASES = {
"default": dj_database_url.config(conn_max_age=600, ssl_require=True)
}
DEBUG = config.DEBUG
This code does the following:
- Defines a class
SettingsFromEnvironment
, inheriting from pydantic’sBaseSettings
class. - Defines
DATABASE_URL
andDEBUG
, setting their type and optional default using Python type hints. - Defines a class
Config
telling pydantic to look for the variables in a.env
file if not present in the system’s environment variables. - Instantiates the
Config
class into the objectconfig
; the desired variables become available asconfig.DATABASE_URL
andconfig.DEBUG
. - Defines the regular Django variables
DATABASES
andDEBUG
from theseconfig
members.
The same code runs in all environments, and pydantic takes care of the following:
- It looks for the environment variables
DATABASE_URL
andDEBUG
.- If defined as environment variables, like in production, it will use those.
- Otherwise, it pulls those values from the
.env
file. - If it doesn’t find a value, it will do the following:
- For
DATABASE_URL
, it throws an error. - For
DEBUG
, it assigns a default value ofFalse
.
- For
- If it finds an environment variable, it will check field types and give an error if either of them are wrong:
- For
DATABASE_URL
, it verifies that its field type is aPostgresDsn
-style URL. - For
DEBUG
, it verifies that its field type is a valid, nonstrict pydantic boolean.
- For
Please note the explicit setting of the operating system’s environment variable from the configuration value for DATABASE_URL
. It may seem redundant to set os.environ["DATABASE_URL"] = config.DATABASE_URL
because DATABASE_URL
is already defined as an external environment variable. However, this allows pydantic to parse, check, and validate this variable. If the environment variable DATABASE_URL
is missing or formatted incorrectly, pydantic will give a clear error message. These error checks are invaluable as the application moves from development to subsequent environments.
If a variable is not defined, either a default will be assigned or an error will prompt for it to be defined. Any generated prompts also detail the desired variable type. A side benefit of these checks is that new team members and DevOps engineers more easily discover which variables need to be defined. This avoids the hard-to-find issues that result when the application runs without all the variables defined.
That’s it. The application now has a maintainable, settings-management implementation using a single version of settings.py
. The beauty of this approach is that it allows us to specify the correct environment variables in a .env
file or any other desired means available through our hosting environment.
The Scalable Path
I have been using Django settings management with pydantic runtime typing in all of my Django projects. I have found it leads to smaller, more maintainable codebases. It also provides a well-structured, self-documenting, and scalable approach to adding new environments.
The next article in this series is a step-by-step tutorial on building a Django application from scratch, with pydantic settings management, and deploying it to Heroku.
The Toptal Engineering Blog extends its gratitude to Stephen Davidson for reviewing the code samples presented in this article.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK