Progressive Delivery With Jenkins X: Automatic Canary Deployments
source link: https://www.tuicool.com/articles/hit/veEZ7ja
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
This is the third post in a Progressive Delivery series, see the previous ones:
- Progressive Delivery in Kubernetes: Blue-Green and Canary Deployments .
- Progressive Delivery with Jenkins X
Progressive Delivery is used by Netflix, Facebook, and others to reduce the risk of deployments. But you can now adopt it when using Jenkins X .
Progressive Delivery is the next step after Continuous Delivery, where new versions are deployed to a subset of users and are evaluated in terms of correctness and performance before rolling them to the totality of the users and rolled back if not matching some key metrics.
In particular, we focused on Canary releases and made it really easy to adopt them in your Jenkins X applications. Canary releases consist of sending a small percentage of traffic to the new version of your application and validate there are no errors before rolling it out to the rest of the users. Facebook does it this way, delivering new versions first to internal employees, then a small percentage of the users, then everybody else, but you don't need to be Facebook to take advantage of it!
You can read more on Canaries at Martin Fowler's website .
Jenkins X
If you already have an application in Jenkins X you know that you can promote it to the "production" environment with jx promote myapp --version 1.0 --env production
. But it can also be automatically and gradually rolled it out to a percentage of users while checking that the new version is not failing. If that happens, the application will be automatically rolled back. No human intervention at all during the process.
NOTE: This new functionality is very recent and a number of these steps will not be needed in the future as they will also be automated by Jenkins X.
As the first step, three Jenkins X add-ons need to be installed:
- Istio: a service mesh that allows us to manage traffic to our services.
- Prometheus : the most popular monitoring system in Kubernetes.
- Flagger : a project that uses Istio to automate canarying and rollbacks using metrics from Prometheus.
The addons can be installed (using a recent version of the jx
CLI) with:
1 jx create addon istio
2 jx create addon prometheus
3 jx create addon flagger
This will enable Istio in the jx-production
namespace for metrics gathering.
Now get the IP of the Istio ingress and point a wildcard domain to it (e.g. *.example.com
), so we can use it to route multiple services based on hostnames. The Istio ingress provides the routing capabilities needed for Canary releases (traffic shifting) that the traditional Kubernetes ingress objects do not support.
The cluster is configured, and it's time to configure our application. Add a canary.yaml
to your helm chart, under charts/myapp/templates
.
Then append to the charts/myapp/values.yaml
the following, changing myapp.example.com
to your hostname or names:
Soon, both the canary.yaml
and the values.yaml
changes won't be needed when you create your app from one of the Jenkins X quickstarts, as they will be Canary enabled by default.
That's it! Now when the app is promoted to the production environment with jx promote myapp --version 1.0 --env production
, it will do a Canary rollout. Note that the first time it is promoted it will not do a Canary as it needs a previous version data to compare to, but it will work from the second promotion on.
With the configuration in the values.yaml
file above it would look like:
- minute 1: send 10% of the traffic to the new version
- minute 2: send 20% of the traffic to the new version
- minute 3: send 30% of the traffic to the new version
- minute 4: send 40% of the traffic to the new version
- minute 5: send 100% of the traffic to the new version
If the metrics we have configured (request duration over 500 milliseconds or more than 1% responses returning 500 errors) fail, Flagger then will note that failure, and if it is repeated 5 times, it will rollback the release, sending 100% of the traffic to the old version.
To get the Canary events run:
$ kubectl -n jx-production get events --watch\ --field-selector involvedObject.kind=Canary 23m ... New revision detected! Scaling up jx-production-myapp.jx-production 22m ... Starting canary analysis forjx-production-myapp.jx-production 22m ... Advance jx-production-myapp.jx-production canary weight 10 21m ... Advance jx-production-myapp.jx-production canary weight 20 20m ... Advance jx-production-myapp.jx-production canary weight 30 19m ... Advance jx-production-myapp.jx-production canary weight 40 18m ... Advance jx-production-myapp.jx-production canary weight 50 18m ... Copying jx-production-myapp.jx-production template spec to jx-production-myapp-primary.jx-production 17m ... Promotion completed! Scaling down jx-production-myapp.jx-production
Dashboard
Flagger includes a Grafana dashboard for visualization purposes as it is not needed for the Canary releases. It can be accessed locally using Kubernetes port forwarding
Then accessing http://localhost:3000 using admin/admin
, selecting the canary-analysis
dashboard and
jx-production jx-production-myapp-primary jx-production-myapp
would provide us with a view of different metrics (CPU, memory, request duration, response errors,...) of the incumbent and new versions side by side.
Caveats
Note that Istio, by default, will prevent access from your pods to the outside of the cluster (a behavior that is expected to change in Istio 1.1). Learn how to control the Istio egress traffic .
If a rollback happens automatically because the metrics fail, the Jenkins X GitOps repository for the production environment becomes out of date, still using the new version instead of the old one. This is something planned to be fixed in upcoming releases.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK