In this post, I want to demonstrate how to set up ephemeral environments for pull requests created on GitHub. This can be applied to any SCM supported by Argo CD pull request generator. There is not one way and one set of tools to implement this kind of feature.

I will use two different Kubernetes flavors as pull request environments, the first one is Kubernetes namespaces and the second one is virtual Kubernetes clusters which can provide better isolation.

Tools Used

1. Argo CD Pull Request Generator: Argo CD Pull Request Generator is one of the applications generators implemented to watch your SCM for new pull requests and then generate necessary application definitions according to these pull requests.
2. Github Actions: Github actions is used for the CI part of this feature. Whenever a new pull request is generated, a Github Action is triggered, builds and pushes the container image for the pull request’s feature branch.
3. Vcluster: Vcluster is used for the virtual Kubernetes cluster part of this feature. It is a great tool developed by Loft Labs on top of Rancher’s lightweight Kubernetes distribution, K3S.
4. ArgoCD Secret Synchronizer: A custom operator I developed for experimental purposes which synchronizes Vcluster’s created secrets to Argo CD Cluster secrets. By using this operator manual intervention requirement is taken away and created virtual cluster is automatically added to the list of Argo CD managed clusters.

In addition to the above ones, you need to have a Kubernetes cluster with admin privileges (Katacoda can be used, look at the artifacts section for sample scenario), Github repositories for your application code, helm charts, Argo CD CRDs, and Argo CD application resource definitions.

Github Repositories

1. ephemeralenvironments: This git repository contains Argo CD CRDs, pull request applicationset definitions, and Argo CD secret synchronizer resource definitions. All of these resources are managed by kustomization. kustomization.yaml files can be seen in each folder.

If you want to run this sample, clone this git repository and execute two commands below:

• kubectl apply -k ephemeralenvironments/managementstack/argocd
• kubectl apply -k ephemeralenvironments/managementstack/argocdapplications

The first one applies Argo CD CRDs and resources to the management cluster pointed by kubeconfig file and the second one creates an applicationset the definition which generates Argo CD applications for five folders above. A Katacoda scenario is constructed to demonstrate these steps if you want to follow.

2. ephemeralcluster:This git repository contains helm chart to deploy Vcluster declaratively. There is an applicationset definition pointing this repository in the pullrequests directory of ephemeralenvironments repository above.

3. app-ephemeraltest: This repository contains sample application’s code(a simple dotnet api), dockerfile, helm chart, GitHub actions definition to build and push images for each pull request and a sample pull request to be demonstrated on a sample scenario. f-ephemeralpullrequest branch is used for this sample and a sample pull request is created from this branch to be merged to main branch.

Pull Request Generator

A pull request generator is one of the Argo CD template generators which uses SCM APIs to get pull request information and generates necessary application definitions. In default, applicationset controller polls SCM APIs with 30 seconds interval which can be changed if needed. In addition to this if wanted SCMs webhook can also be configured to trigger applicationset controller to eliminate this wait time.

Deploy to a Virtual Cluster

The Pull request generator that is used to deploy the sample application to the virtual cluster is given below:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: sampleapps
  namespace: argocd
spec:
  generators:
  - pullRequest:
      github:
          owner: a1tan
          repo: app-ephemeraltest
      requeueAfterSeconds: 120
  template:
    metadata:
      name: 'sampleapp-{{branch}}-{{number}}'
    spec:
      source:
        repoURL: 'https://github.com/a1tan/app-ephemeraltest.git'
        targetRevision: '{{head_sha}}'
        path: charts/sampleapp
        helm:
          parameters:
          - name: "image.tag"
            value: "pull-{{head_sha}}"
          - name: "environment"
            value: "PR-Vcluster-Dev"
      project: default
      destination:
        server: https://cluster-{{branch}}-{{number}}.cluster-{{branch}}-{{number}}-namespace.svc.cluster.local
        namespace: 'sampleapp-{{branch}}-{{number}}-namespace'
      syncPolicy:
          automated:
              allowEmpty: true
              prune: true
              selfHeal: true

This pull request generator listens for pull requests on Github account “a1tan” and repository app-ephemeraltest.

It also changes the polling interval to 120 seconds which can be annoying in a real-world scenario. When it detects a new pull request it creates an application named with sampleapp-{{branch}}-{{number}} pattern. It will take the latest commit on that branch with targetRevision: ‘{{head_sha}} and path with path: charts/sampleapp. It also sets the image tag as pull-{{head_sha}} which points to the latest pull request image built by the Github Actions CI pipeline. There is also another helm parameter to distinguish this deployment from namespace version below.

For the destination, this applicationset uses cluster reference as server: https://cluster-{{branch}}-{{number}}.cluster-{{branch}}-{{number}}-namespace.svc.cluster.local which is different than the main kubernetes cluster. It applies this application to a newly created, ephemeral virtual cluster instance.

This instance is also deployed by the below pull request generator:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ephemeralcluster
  namespace: argocd
spec:
  generators:
  - pullRequest:
      github:
          owner: a1tan
          repo: app-ephemeraltest
      requeueAfterSeconds: 120
  template:
    metadata:
      name: 'cluster-{{branch}}-{{number}}'
    spec:
      source:
        repoURL: 'https://github.com/a1tan/ephemeralcluster.git'
        targetRevision: 'main'
        path: charts/vcluster
        helm:
          parameters:
          - name: "syncer.extraArgs[0]"
            value: "--out-kube-config-server=https://cluster-{{branch}}-{{number}}.cluster-{{branch}}-{{number}}-namespace.svc.cluster.local"
          - name: "syncer.extraArgs[1]"
            value: "--tls-san=cluster-{{branch}}-{{number}}.cluster-{{branch}}-{{number}}-namespace.svc.cluster.local"
      project: default
      destination:
        server: https://kubernetes.default.svc
        namespace: 'cluster-{{branch}}-{{number}}-namespace'
      syncPolicy:
          automated:
              allowEmpty: true
              prune: true
              selfHeal: true

In this applicationset definition server reference is different, server: https://kubernetes.default.svc. Vcluster is deployed to a namespace in the main cluster. It has also two helm parameters — out-kube-config-server and — tls-san to make created Vcluster to be accessible from the Argo CD instance.

This application creates all Vcluster resources on cluster-{{branch}}-{{number}}-namespace namespace of main cluster. A secret containing kubeconfig to access this virtual cluster instance is also created on this namespace.

ArgoCDSecretSynchronizer operator takes this kubeconfig and creates necessary Argo CD cluster secret. In this way, the Argo CD instance on the main cluster can manage sample application on the virtual cluster.

Deploy to a Namespace

There is also a third pull request generator definition which is used to deploy sample applications to a namespace on the main cluster.

If you need a simpler solution and don’t want to create a virtual cluster for each pull request, you can go with this setup.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: sampleapps-ns
  namespace: argocd
spec:
  generators:
  - pullRequest:
      github:
          owner: a1tan
          repo: app-ephemeraltest
      requeueAfterSeconds: 120
  template:
    metadata:
      name: 'sampleapp-ns-{{branch}}-{{number}}'
      # namespace: 'myapp-{{branch}}-{{number}}-namespace'
    spec:
      source:
        repoURL: 'https://github.com/a1tan/app-ephemeraltest.git'
        targetRevision: '{{head_sha}}'
        path: charts/sampleapp
        helm:
          parameters:
          - name: "image.tag"
            value: "pull-{{head_sha}}"
          - name: "environment"
            value: "PR-Namespace-Dev"
      project: default
      destination:
        server: https://kubernetes.default.svc
        namespace: 'sampleapp-ns-{{branch}}-{{number}}-namespace'
      syncPolicy:
          automated:
              allowEmpty: true
              prune: true
              selfHeal: true

The only difference from other sample applicationset definition is the server selector, server: https://kubernetes.default.svc. By this server selector sample application is deployed to sampleapp-ns-{{branch}}-{{number}}-namespace namespace on main cluster. Helm environment parameter is also set differently to validate if the application is deployed correctly.

Tips

One thing to keep in mind, if you use a branch name as your service name inside of your helm chart, be careful with your branch naming strategy, it has to be a valid DNS name. Therefore, instead of using feature/blabla kind of naming you can prefer “f-blabla” kind of naming.

You can find the error I received because of this mistake below.

Github rate limits are also a critical issue here. For the first run of this sample, I have used the default value which is 30 seconds. After too many unsuccessful synchronization attempts, the rate limit error below is received then I have increased requeue period to 120 seconds. Webhook configuration is a better choice for these rate limits.

Don’t forget to use ${{ github.event.pull_request.head.sha }} on github actions for pull request commit sha. Tag your image with this expression to successfully get specific pull request image on Argo CD.

Conclusion

Creating pull request environments is a critical step if you want your features to adhere to the definition of done before merging into the main branch.

You can use ephemeral environments for acceptance testing, penetration testing, end-to-end testing, or load testing. Management of dependencies is another crucial issue we haven’t mentioned in this post.

Of course, it is not enough to deploy a simple application to an environment. This application has lots of dependencies including a database, database tables, distributed caching service, key vault, identity provider, message broker, and even other services according to the scenario we want to test.

All these pieces need to be defined in a way that can be applied to this ephemeral environment. Packaging methods and tools have also been determined to manage such a requirement. In a later post, I want to elaborate on this topic.

Artifacts

References

Previous