Use within gitlab-ci.yml¶

The ci-fairy tool is available by extending the .fdo.ci-fairy template:

include:
  - project: 'freedesktop/ci-templates'
    file: '/templates/ci-fairy.yml'

check-mr:
  extends:
    - .fdo.ci-fairy
  script:
   - ci-fairy check-merge-request --require-allow-collaboration

The .fdo.ci-fairy template guarantees the following preinstalled packages: bash, git, python and pip.

The distribution the template is based on is subject to change without notice.

Installation¶

As the ci-fairy is likely used within a CI job, installation via pip3 directly from git is recommended:

pip3 install git+https://gitlab.freedesktop.org/freedesktop/ci-templates@12345deadbeef

The @12345deadbeef suffix specifies the git sha to check out. The git sha should match the sha of the included templates.

Or where a local checkout is available:

pip3 install .

Alternatively, you may directly invoke the ci-fairy tool from within the git tree without prior installation. Note that pip3 will take care of all dependencies, for local invocations you must install those manually.

Use of the GitLab CI Environment¶

ci-fairy will make use of the predefined environment variables set by GitLab CI where possible. This includes but is not limited to CI_PROJECT_PATH, CI_SERVER_URL, and CI_JOB_TOKEN.

In most cases ci-fairy does not need arguments beyond what is specific to the interaction with the project at the time.

Authentication¶

Authentication is handled automatically through the $CI_JOB_TOKEN environment variable. Where the ci-fairy tool is called outside a CI job, use the --authfile argument to provide the path to a file containing the value of a GitLab private token with ‘api’ access.

For example, if your private token has the value abcd1234XYZ, authentication can be performed like this:

$ echo "abcd1234XYZ" > /path/to/authfile
$ ci-fairy --authfile /path/to/authfile <....>

Where --authfile is used within a CI job, specificy the token as a GitLab predefined environment variable of type “File”. The “Key” is the file name given to --authfile, the value is the token value of your private token.

Linting¶

GitLab has an online linter but it requires copy/pasting the .gitlab-ci.yml file into the web form. ci-fairy makes this simpler:

$ cd /path/to/project.git
$ ci-fairy lint

ci-fairy will find the .gitlab-ci.yml file and upload it to the CI linter and print any errors.

Deleting registry images¶

To delete an image from your container registry, use the delete-image subcommand. This subcommands provides three modes - deleting a specific image tag, deleting all but an image tag or deleting all images.

ci-fairy delete-image --repository fedora/30 --all
ci-fairy delete-image --repository fedora/30 --exclude-tag 2020-03.11.0
ci-fairy delete-image --tag "2020-03-*"

During testing, use the --dry-run argument to ensure that no tags are actually deleted.

Templating .gitlab-ci.yml¶

The .gitlab-ci.yml file can become repetitive where multiple jobs use the same setups (e.g. such as testing a build across different distributions). ci-fairy provides a command to generate a .gitlab-ci.yml (or any file, really) from a Jinja2 template, sourcing the data from a YAML file.

For example, let’s assume the YAML file .gitlab-ci/config.yml with a list of distributions:

distributions:
  fedora: [32, 31]
  ubuntu: [19.04]

packages:
  fedora:
    needed: ['gcc', 'valgrind']
    use_qemu: false
  ubuntu:
    needed: ['curl', 'wget', 'valgrind']
    use_qemu: true
  debian:
    needed: ['curl', 'wget']

And the template .gitlab-ci/ci.template that uses those to generate jobs (see the Jinja2 documentation for details):

{% for distro in distributions %}
{% for version in distributions[distro] %}
.job.{{distro}}.{{version}}:
   extends: .fdo.container-build@{{distro}}@x86_64
   variables:
      FDO_DISTRIBUTION_VERSION: "{{version}}"
      FDO_DISTRIBUTION_PACKAGES: "{{" ".join(packages[distro].needed)}}"
{% endfor %}
{% endfor %}

Now let’s generate the .gitlab-ci.yml file:

$ ci-fairy generate-template

and our file will look like this:

.job.fedora.32:
   extends: .fdo.container-build@fedora@x86_64
   variables:
      FDO_DISTRIBUTION_VERSION: "32"
      FDO_DISTRIBUTION_PACKAGES: "gcc valgrind"

.job.fedora.31:
   extends: .fdo.container-build@fedora@x86_64
   variables:
      FDO_DISTRIBUTION_VERSION: "31"
      FDO_DISTRIBUTION_PACKAGES: "gcc valgrind"

.job.ubuntu.19.04:
   extends: .fdo.container-build@ubuntu@x86_64
   variables:
      FDO_DISTRIBUTION_VERSION: "19.04"
      FDO_DISTRIBUTION_PACKAGES: "curl wget valgrind"

This command does not need a connection to the GitLab instance.

$ ci-fairy generate-template --config somefile.yml mytemplate.tmpl

$ cat template.tpl
qemu needed? {{use_qemu}}
$ ci-fairy generate-template --root=/packages/fedora --config distributions.yml template.tpl
qemu needed? true
$ ci-fairy generate-template --root=/packages/ubuntu --config distributions.yml template.tpl
qemu needed? false

Checking commits¶

Git commit messages have a few standard requirements that apply across projects. ci-fairy can verify the commits to be merged for those standards.

$ ci-fairy check-commits master..HEAD

By default, this checks for any leftover fixup!/squash! commits and some cosmetic requirements. Other options to check for is whether a Signed-off-by: line is present (--signed-off-by) or ensure that such a line is not included (--no-signed-off-by). See the --help output for a list fo checks available.

ci-fairy uses the CI environment where available and will do the right thing without requiring options. It does need the $FDO_UPSTREAM_REPO variable to be set to the upstream project’s full name. An example .gitlab-ci.yml section:

variables:
  FDO_UPSTREAM_REPO: libinput/libinput

commit message check:
   image: something_that_has_pip
   before_script:
     - pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
   script:
     - ci-fairy check-commits --signed-off-by

ci-fairy uses $CI_MERGE_REQUEST_TARGET_BRANCH_NAME if available, otherwise it defaults to the master branch of the $FDO_UPSTREAM_REPO. For the above snippet, the commit range checked will thus be upstream/master..HEAD.

Checking merge requests¶

ci-fairy can be used to check merge requests for common requirements:

$ ci-fairy check-merge-request --require-allow-collaboration

The --require-allow-collaboration flag checks for the merge request to allow edits from maintainers.

This can be used in the CI to fail a merge request pipeline if that checkbox is not set. There are two ways to invoke this through the CI: either through a detached merge pipeline or by searching for a merge request with the same commit sha as our branch. The detached merge pipeline has side-effects you should familiarize yourself with.

variables:
  FDO_UPSTREAM_REPO: 'project/name'

check-merge-request:
  image: golang:alpine
  before_script:
    - apk add python3 git
    - pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
  script:
     - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml
  artifacts:
    when: on_failure
    reports:
        junit: check-merge-request.xml

# work around the detached merge request pipeline issues by defining
# rules for all jobs
workflow:
  rules:
    - when: always

check-merge-request:
  image: golang:alpine
  before_script:
    - apk add python3 git
    - pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
  script:
     - ci-fairy check-merge-request --require-allow-collaboration --junit-xml=check-merge-request.xml
  rules:
     - if: '$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
       when: always
     - when: never  # override the workflow's always rule
  artifacts:
    when: on_failure
    reports:
        junit: check-merge-request.xml

Waiting for a pipeline¶

ci-fairy can be used to wait for the completion of a pipeline.

$ ci-fairy wait-for-pipeline

In the default invocation with no arguments ci-fairy will guess the gitlab project path based on the $GITLAB_USER_ID if set, or $USER otherwise, combined with the basename of the current directory. The pipline to wait for defaults to the pipeline matching the current git sha:

$ whoami
bob
$ cd myproject
$ git push -q gitlab mybranch
$ ci-fairy wait-for-pipeline
Pipeline https://gitlab.freedesktop.org/bob/myproject/-/pipelines/12345
status: success   | 90/90 | created:  0 | pending:  0 | running:  0 | failed:  1 | success: 89

Please see the --help output for more details on controlling which pipeline to wait for.