Zeitgeist: Dependency Management for DevOps
source link: https://blog.florentdelannoy.com/blog/2020/introducing-zeitgeist/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Developers are spoiled.
Every single package management system these days will allow a developer to define their project's dependencies in a simple format, whether that's a cargo.toml in Rust, Gemfile for Ruby apps, a pom.xml for Maven-based projects, package.json for NodeJS, composer.json in PHP…
Declaring your dependencies and their desired version in a standard, easily-parseable language allows you to track outdated dependencies, make sure you keep up to date with security updates, ensures other developers working on the same project use the same version of dependencies, lets you set up reproducible builds… The benefits are immense, and it's universally acknowledged as a best practice in modern software development.
But when it comes to the subfield of software engineering known as Infrastructure-as-Code (also variously referred to as DevOps, or SRE, but I'll use infrastructure-as-code through this post) though, the alternatives become rare.
It is possible, as most sysadmins are used to, to follow individual projects through mailing lists, or periodically check projects’ websites for new releases and updates. But this doesn't scale well as you add extra dependencies, and is at its core a very manual process.
Chef probably has the most full-featured dependency management system that I've encountered so far, the Chef Supermarket . But it only ensures Chef cookbooks are up-to-date, and only covers the subset of the infrastructure-as-code that Chef will support. What about your cloud orchestration software? What about Chef itself?
Part of the problem is that a modern infrastructure-as-code pipeline will rely on several separate, distinct, pieces of software, depending on which part of the infrastructure is being set up: Terraform , CloudFormation , Chef / Ansible / Puppet , Helm … These incompatible systems are written in different languages and follow different versioning schemes, so an overarching dependency management system isn't easy to come by.
But dependency management is a worthy goal, so let's take a crack at it!
Based on the findings above and lessons from the Kubernetes project's handling of dependencies , I'd like to introduce my attempt at solving this issue: Zeitgeist .
Introducing Zeitgeist
Zeitgeist is a language-agnostic dependency checker.
Zeitgeist will let you define your dependencies in a YAML file, dependencies.yaml
, and help you ensure:
- These dependencies versions are consistent within your project
- These dependencies are up-to-date
The key concept that allows Zeitgeist to be language-independent is that dependencies are declared with a reference to another plain-text file where that dependency is defined for the underlying tool , so any infrastructure-as-code tool that lets you define your dependencies in plain-text will be supported.
For example, let's say you build your infrastructure via Terraform, and use a pinned version of Terraform to create a Docker container that applies your infrastructure in a reproducible fashion. Your Dockerfile may look something like:
FROM alpine:3.10 ENV TERRAFORM_VERSION="0.12.12" RUN curl -LO https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip \ && unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip \ && mv terraform /bin # Install more tools, etc
You can define this version of Terraform as a dependency:
- name: terraform version: 0.12.12 upstream: flavour: github url: hashicorp/terraform refPaths: - path: Dockerfile match: TERRAFORM_VERSION
The important information for dependency checking here is the refPaths
: path
refers to a file, and match
is a string that will be searched in this file.
Zeitgeist will read the file at path
, searching for a line matching match
, and when found, confirm that the version declared on this line is identical to the version declared in your dependencies.yaml
.
The upstream
block is used to check if your local version is up-to-date with the project it references. Here, Zeitgeist will check the local version against Terraform Releases on Github
, and let you know when an update is available.
You may then define an AMI image
for an AutoScaling group in a Terraform source file
that we'll call my-asg.tf
as:
resource "aws_launch_template" "foobar" { name_prefix = "foobar" image_id = "ami-1a2b3c" instance_type = "t2.micro" } resource "aws_autoscaling_group" "bar" { availability_zones = ["us-east-1a"] desired_capacity = 1 max_size = 1 min_size = 1 launch_template { id = "${aws_launch_template.foobar.id}" version = "$Latest" } }
Now, how do you ensure this AMI stays up to date? What if a new version gets released?
Zeitgeist can also take care of that for you! Define the dependency as:
- name: my-example-ami version: ami-1a2b3c scheme: random upstream: flavour: ami owner: somebody name: "terraform-example-*" refPaths: - path: my-asg.tf match: image_id
And Zeitgeist will let you know when a new version of the AMI is available, based on the owner and a regexp.
Another common issue is when a dependency is used in two different contexts, and these need to be kept in sync. This was the Kubernetes project's most important requirement when implementing their solution, and I've followed the same yaml structure when designing Zeitgeist.
For example, Helm (up to version 3) needs two components:
-
tiller
, the server-side component of Helm, deployed on your Kubernetes cluster -
The
helm
CLI, used locally to interact withtiller
But using different versions of helm
and tiller
will often result in errors
, so it's better to keep them in sync.
This can be expressed as a dependency with two refPaths
:
- name: helm version: 2.12.2 upstream: flavour: github url: helm/helm constraints: <3.0.0 refPaths: - path: bootstrap/tiller.yaml match: gcr.io/kubernetes-helm/tiller - path: helper-image/Dockerfile match: HELM_LATEST_VERSION
And Zeitgeist will ensure these stay in sync with each other, as well as check the upstream for updates.
Working with Zeitgeist
We have introduced Zeitgeist in our infrastructure-as-code pipeline late last year to check dependencies on deployment and give us a nightly dependency report of available updates.
Since then, Zeitgeist has been fulfilling its stated aim of helping us keep our infrastructure up-to-date, but its major benefit in my opinion has been the peace of mind gained by knowing that pinned versions of dependencies are managed , and we don't have to worry about them silently falling behind.
Zeitgeist supports Github, AMIs, and Helm as upstreams for version checking, with more in the pipeline!
How do I use it in my own projects?
To use Zeitgeist in your own project, create a dependencies.yaml
and run the zeitgeist
command-line tool. Pre-build binaries are available on Github
.
Zeitgeist is written in Go and hosted on Github under the Apache 2.0 license. Contributions, bug reports, and feature suggestions are very welcome!
Recommend
-
223
In just a few years Huawei has gone from a brand unfamiliar to consumers outside of China to the world's second largest smartphone maker. Even if you've ne... by Ryan Whitwam in Huawei, Mate 10, News, Reviews
-
3
Zeitgeist 的定义由当时的思想和信仰所显示的特定历史时期的决定性精神或情...
-
4
Zeitgeist:一个去中心化 DAO 的预测市场 | PW 专访加入 PolkaWorld 社区,共建 Web 3.0!Zeitgeist 是为 Polkadot 和 Kusama 生态系统设计的 Web3.0 预测市场枢纽。它使用 Substrate 构建,预计在 2021 年底竞拍 Kusama 的平行链插槽!Polk…
-
5
Zuckerberg’s Zeitgeist: A culture of shame, addiction, and dishonestyAnd how better UX could help solve the problem.Yet another public controversy: Here’s Zuckerberg back in 2018 during international data...
-
4
Sharp Type co-founders on navigating the aesthetic zeitgeist of typographyDuring September’s Nicer Tuesdays, Sharp Type co-founders Chantra Malee and Lucas Sharp took us inside the workings of their wildly succes...
-
4
How Hemingway Gradually—Then Suddenly—Defined the ZeitgeistPlus: Early attempts at writing fiction online, the many sequels to Hackers, and Canada’s pessimism.
-
1
Invitation to Coding Events, Zeitgeist'23 — IIT Ropar Invitation to Coding Events, Zeitgeist'23 — IIT Ropar ...
-
100
README.md ...
-
98
This is Part 1 of a series of blog posts I entitled “It depends. Harden your Javascript deployment process”, where I’d like to shed some light on how one can greatly minimize the risk of Javascript…
-
69
This series of videos explains Go's new 'vgo' dependency management tool in detail.
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK