Ansible, Molecule, Docker and GitHub Actions

Last year, I got familiar with Ansible, the automation platform I now use to install and configure my Linux installations. I must thank Jeff Geerling and his excellent book “Ansible for DevOps“, which I highly recommend!

I have already started blogging about Ansible and its testing framework, Molecule. However, in the first blog post, I used Ansible and Molecule to demonstrate Gitpod with a minimal example.

In this blog post, I’d like to document the use of Ansible and Molecule with a slightly more advanced example and how to test an Ansible role against 3 main Linux distributions, Fedora, Ubuntu, and Arch. To test the Ansible role, we will use Molecule and Docker. Finally, I’ll show how to implement a continuous integration process with GitHub Actions. The example consists of a role for installing zsh, setting it as the user’s default shell, and creating an initial “.zshrc” file. It will be a long post because it will be step-by-step.

The source code used in this tutorial can be found here: https://github.com/LorenzoBettini/ansible-role-zsh. The GitHub repository is configured to be used with Gitpod (see my other blog post concerning using the online IDE Gitpod).

Install ansible and molecule

I’m assuming Docker, Python, and Pip are already installed.

First, let’s install Ansible and Molecule (with Docker support). We’ll use pip to install these tools. This method works on all distributions since it’s independent of the ansible and molecule packages provided by the distribution (Ubuntu does not even provide a package for molecule):

This will install ansible and molecule in “$HOME/.local/bin,” so this path must be in your PATH (it should already be the case in most distributions).

At the time of writing, these are the versions of the installed tools:

Create the role

This is the command to initialize a role with the directories and files also for molecule (with docker):

In this example, I’ll run:

This is the resulting directory structure of the created project (note that, at the time of writing, the official guide, https://molecule.readthedocs.io/en/latest/getting-started.html, is not updated with the directory and file structure):

What I do next is to enter the directory, remove “.travis.yml” (since we want to build on GitHub Actions), and create a Git repository (with “git init”). I’m also pushing to GitHub.

First, let’s adjust the file meta/main.yml with the information about this role and author:

The role’s name should be the same as the one specified in the “init” command (I don’t know why this file has not been generated with the role_name already set). Otherwise, the other generated files for Molecule will not work.

The role’s main tasks are defined in tasks/main.yml. Currently, the generated file does not execute any task.

Manual tests

The “init” command also created a tests directory to manually and locally test the role. We are interested in automatically testing the role. However, since the role is currently empty, it is safe to try to run it against our own machine. At least, we can check that the syntax of the role is OK, and we can perform a “dry-run” without modifying anything on our machine.

The current contents of the files generated in the “tests” directory will not work out of the box.

First, the tests/test.yml playbook:

Correctly refers to our role, but ansible will not be able to find the role in the default search path (because the role is the project’s path).

We can change the role reference with a relative path:

Then, we can try to run it, checking the syntax and doing a “dry-run”:

The “dry-run” (–check) fails because, on my machine, there’s no SSH server, and by default, the tests/inventory file (specifying “localhost”) would imply an SSH connection:

To avoid SSH, we can change the file as follows:

Let’s try again with the “–check” argument, and now it works.

Run the complete Molecule default scenario

The “init” command created a default Molecule scenario in the file default/molecule.yml:

As we can see from this file, the Docker image used by Molecule is centos:stream8. For the moment, we’ll stick with this image.

Molecule will execute a playbook against a Docker container of this Docker image. We’re implementing a role, not a playbook. The playbook is defined in the file default/converge.yml:

In fact, “converge” is the action of performing the playbook against the Docker image, the “instance”. As you see, the “init” command generated this file automatically based on the role that we created.

There’s also a default/verify.yml file that is used to verify that some expected conditions are true once we run the playbook against the Docker instance. We’ll get back to this file later to write our own assertions. The contents of this generated file are as follows (the assertion is always verified):

To check that the scenario already works, we can run it end-to-end with the command “molecule test” issued from the project’s root. Remember that Molecule will download the Docker image during the first run, which takes time, depending on your Internet connection. This is the simplified output:

As reported in the first line, this is the entire lifecycle sequence:

dependency, lint, cleanup, destroy, syntax, create, prepare, converge, idempotence, side_effect, verify, cleanup, destroy

Thus running the entire scenario always implies starting from scratch, that is, from a brand new Docker container (of course, the pulled image will be reused). Note that after “converge,” the scenario checks “idempotence,” which is a desired property of Ansible roles and playbooks. After verification, the Docker instance is also destroyed. Of course, if any of these actions fail, the lifecycle stops with failure.

Setup the CI on GitHub Actions

Our role doesn’t do anything yet, but we verified that we could run the complete Molecule scenario. Before going on, let’s set up the GitHub Actions CI workflow. We’ll use the Ubuntu runner, where Docker and Python are already installed. We’ll have first to install ansible and molecule with pip, and then we run the “molecule test”.

Concerning the pip installation step, I created the file pip/requirements.txt in the project with these contents (they correspond to the pip packages we installed on our machine):

Then, I create the file .github/workflows/molecule-ci.yml with these contents:

Now that our CI is in place, GitHub Actions will run the complete Molecule test scenario at each pushed commit. The environment variables at the end of the file will allow for colors in the GitHub Actions build output:

Familiarize with Molecule commands

While implementing our role, we could run single Molecule commands instead of the whole scenario (which, in any case, will be executed by the CI).

With “molecule create,” we create the Docker instance. Unless we run “molecule destroy” (which is executed by the entire scenario at the beginning), the Docker container will stay on our machine. Once the instance is created, you can enter the container with “molecule login“. This is useful to inspect the state of the container after running the playbook (with “molecule converge“) or to run a few commands before writing the tasks for our role:

The “login” command is more straightforward than running a “docker” command to enter the container (you don’t need to know its name). Remember that unless you run “molecule destroy,” if you exit the container and back in, you’ll find the same state.

Once you run “molecule converge“, you can run “molecule verify” to check that the assertions hold.

To get rid of the instance, just run “molecule destroy“.

Let’s start implementing our role’s tasks

To start experimenting with Molecule for testing Ansible roles, the official Fedora Docker image is probably the easiest. In fact, such an image comes with “python” already installed (and that’s required to run Ansible playbooks). Moreover, it also contains “sudo”, another command typically used in Ansible tasks (when using “become: yes”).

Thus, let’s change the image in the file default/molecule.yml:

You can commit and push and let GitHub Actions verify that everything is still OK.

Now it’s time to edit the primary role’s file, tasks/main.yml. Let’s add the task to install ZSH. In this example, I’m using “ansible.builtin.package module – Generic OS package manager” so that we are independent of the target OS. This is useful later because we want to test our role against different Linux distributions. This Ansible module is less powerful than the specific package manager modules, but for our goals, it is sufficient. Moreover, in the Linux distributions that we’ll test, the name of the package for ZSH is always the same, “zsh”.

If we had already created the instance, we first need to run “molecule destroy” to avoid errors due to the previous Docker container.

Let’s run “molecule converge“. If you don’t have the “fedora:36” Docker image already in your cache, this command will take some time the first time. Moreover, also the task of installing the “zsh” package might take some time since the package must be downloaded from the Internet, not to mention that dnf is not the fastest package manager on earth. In fact, the Ansible package module will use the distribution package manager, that is, dnf in Fedora. Here’s the output:

Let’s enter the container with “molecule login“. Now, zsh should be installed in the container:

Of course, you could always run the entire “molecule test,” but that takes more time, and for the moment, we don’t have anything to verify yet. The idempotency of the Anslibe package module implies Idempotency.

Change the user’s shell and verify it

Now, we want to change the user’s shell to zsh, and we will verify it. Let’s follow a Test-Driven Development approach, which I’m a big fan of. We first write the verification tasks in verify.yml, make sure that “molecule verify” fails, and then implement the task in our role to make the test succeed.

First, how to get the user’s shell? In the Docker container, the $SHELL environment variable is not necessarily set, so we directly inspect the contents of the file “/etc/passwd” and some shell commands to get the user’s current shell. To write the shell commands, we can enter the container (molecule login), assuming we have already created the instance, and perform some experiments there. Remember that when we’re inside the container, we are “root”, so in our experiments, we’ll try to get the root’s shell.

So, we have our shell piped command to get the root’s shell:

In verify.yml, we want to get the shell of the user who’s executing Ansible. In our molecule tests, it will be root, but the user will be different in the general use case. Thus, we use Ansible’s fact “ansible_user_id”:

Then, we’ll compare it against the desired value, NOT “/bin/bash”, but “/bin/zsh”. Note that, by default, the generated molecule/verify.yml has “gather_facts: false”. We need to remove that line or set it to true so that Ansible populates the variable with the current user. Here are the contents (we must use the module “shell” and not “command” because we need the “|”):

Since we have already created the instance and converged that, let’s run “molecule verify“:

As expected, it fails.

Let’s add the task in our role to set the current user’s shell to zsh (we rely on the Ansible user module):

Let’s run “molecule converge” (since we had already converged before adding this task, the installation of zsh does not change anything):

And let’s run “molecule verify“, and this time it succeeds!

As usual, we commit, push, and let the CI run the whole scenario.

The verification would not be really required since we should rely on the correctness of the Ansible user module). However, I thought this could be the moment to experiment with Molecule verification.

Note that if you enter the container, the “/etc/passwd” has been modified, but you’re still on bash. That’s because the change becomes effective when you log out and log in as a user. In a Docker container, that’s not possible, as far as I know. However, since log out and login are expected in a real system, as long as the shell is modified in “/etc/passwd”, we’re fine.

Add the file .zshrc

Since we want to set up zsh for the user, we should also add to the converged system a “.zshrc” with some reasonable defaults. For example, if you enter the container and run zsh, you’ll see that you have no command history. The history should be enabled in the file “.zshrc”.

Files are searched for in the directory “files” of the project, which the “init” command created for us. I had an existing small “.zshrc” with the enabled history, command completion, and a few aliases:

I put such a file in files/zshrc (I prefer not to have hidden source files, so I removed the “.”). In the role, I added this task, which copies the source file into the converged system in the current user’s home directory with the name “.zshrc”:

Of course, the “copy module” is idempotent and performs the action only if the source and the target files differ.

Let’s converge, enter the Docker container and run “zsh”. Now, the command history works.

Linting

Let’s enable linting to the Molecule scenario. Remember that the scenario has an initial phase for linting.

First of all, we have to install the two additional pip packages, yamllint, and ansible-lint. In our system, we run the following:

Of course, for the CI, we have to update pip/requirements.txt accordingly, adding these two packages.

Then, we have to enable the “lint:” section in default/molecule.yml:

Now we can run “molecule lint“:

The output also suggests how to skip some of these issues. However, let’s try to fix these problems.

Concerning the meta/main.yml this modified version fixes the reported issues (we’ll deal with Ubuntu and Arch later):

Then, we adjust the other two files, “converge.yml” and “verify.yml”. These are the relevant changed parts, respectively:

Now, “molecule lint” should be happy.

Commit and push. Everything should work on GitHub Actions.

Note: when we created the project with “molecule init”, the command also created a “.yamllint” configuration file in the root:

This configuration file enables and disables linting rules. This is out of the scope of this post. However, to experiment a bit, if we remove the last line “truthy: disable” and run “molecule lint,” we get new linting violations:

Because “become: yes” should be changed to “become: true”. I guess it’s a matter of taste whether to enable such a linting rule or not. I’ve seen many examples of Ansible files with “become: yes”. After fixing “main.yml”, there is still a warning (not an error) on the YAML file of our GitHub Actions in correspondence with the “on” line:

You can find a few issues on this being considered false positive or not. A simple workaround is to add a comment in the “molecule-ci.yml” file to make yamllint skip that:

Testing with Ubuntu, the “prepare” step

Let’s say that, besides “fedora:36“, we also want to test our role against the Docker image “ubuntu:jammy“. Instead of creating a new scenario (i.e., another directory inside the directory “molecule”), let’s parameterize the molecule.yml with an environment variable, e.g., MOLECULE_DISTRO, which defaults to “fedora:36”, but that can be passed on the command line with a different value. This is the interesting part:

Nothing changes if we run molecule commands as we did before: we still use the Fedora Docker image. If we want to try with another image, like “ubuntu:jammy”, we prefix the molecule command with that value for our environment variable.

IMPORTANT: Before trying with another Docker image, make sure you run “molecule destroy” since now we want to use a different Docker image.

Let’s try to converge with the Ubuntu Docker image…

What could go wrong?

As I anticipated, while the Fedora Docker image comes with python preinstalled, the Ubuntu Docker image does not. The latter does not even have “sudo” installed, which is required for running our tasks with “becomes: yes”. The converge failed, but the Ubuntu image has been created so you can enter the Docker container and verify that these packages are not pre-installed.

One could try to add the tasks in the role to install python and sudo (not by using “package” because such Ansible modules require python already installed). However, this would not make sense: our role is meant to be executed against an actual distribution, where these two packages are already installed as base ones. Jeff Geerling provides a few Docker images meant for Ansible, where python and sudo are already installed. However, instead of using his Ubuntu image, let’s explore another Molecule step: prepare.

As documented:

The prepare playbook executes actions which bring the system to a given state prior to converge. It is executed after create, and only once for the duration of the instances life. This can be used to bring instances into a particular state, prior to testing.

So, let’s modify this part in molecule.yml (this modification is not strictly required because if the directory of molecule/default contains a file “prepare.yml,” it will be automatically executed; it might still be good to know how to specify such a file, in case it’s in a different directory or it has a different name):

Now, in molecule/prepare.yml we create the preparation playbook. This is kind of challenging because we cannot rely on Ansible facts nor on most of its modules (remember: they require python, which we want to install in this playbook). We can rely on the “ansible.builtin.raw module – Executes a low-down and dirty command”. And looking at its documentation, we can see that it fits our needs:

This is useful and should only be done in a few cases. A common case is installing python on a system without python installed by default.

So, here’s the prepare.yml playbook:

Of course, we must run this task only when we are using Ubuntu (see the condition). We also specify “changed_when: false” to avoid linting problems (“no-changed-when # Commands should not change things if nothing needs doing.”).

Running “molecule converge” now succeeds (note the “prepare” step):

Of course, also “MOLECULE_DISTRO=ubuntu:jammy molecule verify” should succeed:

But it doesn’t. That’s because the “pipefail” we added to make lint happy works in bash but not in sh, which is used by default in Ubuntu (in Fedora, it was bash). It’s just a matter of adjusting that verification’s task accordingly:

And now verification succeeds in Ubuntu as well.

If you run this against Fedora (remember that you must destroy the Ubuntu instance first), the task “Install python in Ubuntu” will be skipped.

Note that if you run

Let’s try to converge with the Ubuntu Docker image…

and then execute

Molecule will reuse the image just created: it does not recreate the Docker image even if you haven’t specified any environment variable. This means that, as mentioned above, if you want to test with another value of the environment variable (including the default case), you first have to destroy the current image. By defining several scenarios, as we will see in a minute, there’s no such limitation.

Add a GitHub Actions build matrix

Let’s modify the GitHub Actions workflow to test our role with Fedora and Ubuntu in two jobs using a build matrix. These are the relevant parts to change to use the environment variable MOLECULE_DISTRO that we introduced in the previous section:

Now GitHub Actions will execute two jobs for each pushed commit:

Using different scenarios

We now see a different technique to test with different Linux distributions. Instead of using an environment variable to parameterize Molecule, we create another Molecule scenario. To do that, it’s enough to create another subdirectory inside the “molecule” directory. We’ll use the “Ubuntu” example to see this technique in action. (Before doing that, remember to run “molecule destroy” first).

First, let’s undo the modification we did in the file “molecule.yml”:

And let’s create another subdirectory, say “ubuntu”, inside “molecule”, where we create this “molecule.yml” file (it’s basically the same as the one inside “default” where we specify “ubuntu:jammy” and a different name for the “image”):

Let’s copy the “default/verify.yml” and “default/converge.yml” into this new directory, and let’s move the “default/prepare.yml” into this new directory, where we change the contents as follows (that is, we get rid of the “when” condition since this will be used only in this new scenario):

To summarize, this should be the layout of the “molecule” directory (we’ll get rid of duplicated contents in a minute):

Now, running any molecule command will use the “default” scenario. If we want to execute molecule commands against the “ubuntu” scenario, we must use the argument “-s ubuntu” (where “-s” is the short form of the command line argument “–scenario-name”).

For example

So we can converge, verify, and experiment with the two scenarios without destroying a previously created instance.

Of course, we adapt the GitHub Actions workflow accordingly to use scenarios instead of environment variables:

Now, let’s clean up our files to avoid duplications. For example, “verify.yml” and “converge.yml” are duplicated in the “default” and “ubuntu” directories. We take inspiration from the official documentation https://molecule.readthedocs.io/en/latest/examples.html#sharing-across-scenarios.

Let’s move the shared files “verify.yml” and “converge.yml” to a new subdirectory, say “shared”. So the layout should be as follows:

The last part of both “molecule.yml” files in “default” and “ubuntu” must be changed to refer to files in another directory (note that the “verifier” part has been removed since it’s specified in the “provisioner” part):

Now we reused the common files between the two scenarios. Of course, we verify that everything still works in both scenarios.

Testing with Arch, a custom Dockerfile

Let’s now test this simple role also with Arch Linux. The idea is to create another scenario, e.g., another subdirectory, say “arch”. We could follow the same technique that we used for Ubuntu because also the Arch Docker image has to be “prepared” with “python” and “sudo”. However, to try something different, let’s rely on a custom Docker image specified with a Dockerfile.

The “molecule.yml” in the “arch” directory is as follows:

NOTE: the specification “platform: linux/amd64” is not required because we use a custom Dockerfile. It is required if you want to test this scenario on a Mac m1 (by the way, see my other blog post about Docker on a Mac m1): while Ubuntu and Fedora also provide Docker images for the aarch64 (arm) architecture, Arch Linux does not. So we must force the use of the Intel platform on Arm architectures (of course, on Mac m1, the Docker container will be emulated).

And in the same directory, we create the Dockerfile for our Arch Docker image:

For this example, the Docker image for Arch is simple because we only need “python” and “sudo” to test our role.

Now the directory layout should be as follows:

Now, when Molecule creates the instance, it will use our custom Dockerfile.

We verify that the “arch” scenario also works and update the GitHub Actions workflow by adding “arch” to the scenario matrix.

I hope you find this post helpful in getting started with Ansible and Molecule.

Stay tuned for more posts on Ansible!