It’s impossible to work with docker containers without docker images. In this post I want to talk about what makes docker images possible: the overlay filesystems.
I’ll start with a brief description of overlay filesystems. Then we will see how it applies to docker images and how docker builds an image from a dockerfile. I’ll conclude with layers cache and OCI format for container images.

As usual I’ll try to make the blog post as practical as possible.

What’s an overlay filesystems

Overlay filesystems (also called union filesystems) allow creating a union of two or more directories: a list of lower directories and an upper directory. The lower directories of the filesystem are read only, whereas the upper directory can be used for both reads and writes.

Let’s see what that means in practice by mounting one.

Create an overlay fs

Let’s create a few folders and combine them.
First I’ll create a folder called “mount” which will contain the union of all the other folders. Then I’ll create a bunch of folders called “layer-1”, “layer-2”, “layer-3”, “layer-4”. Finally a folder called “workdir” which is needed by the overlay filesystem, well, to work properly. You can call any of the folders as you wish, but calling them “layer-1”, “layer-2”, etc. will make easier to understand the parallel with docker images as we shall see.
cd /tmp && mkdir overlay-example && cd overlay-example

[2020-04-19 16:02:35] [ubuntu] [/tmp/overlay-example]  
> mkdir mount layer-1 layer-2 layer-3 layer-4 workdir[2020-04-19 16:02:38] [ubuntu] [/tmp/overlay-example]  
> ls
layer-1  layer-2  layer-3  layer-4 mount  workdir

Let’s also create some files into layer-1, layer-2 and layer-3 folders.
We will leave the layer-4 (our upper folder) empty. Again, that’s not necessary, it will just make easier our parallel with docker images.

[2020-04-19 16:02:40] [ubuntu] [/tmp/overlay-example]  
> echo "Layer-1 file" > ./layer-1/some-file-in-layer-1[2020-04-19 16:03:36] [ubuntu] [/tmp/overlay-example]  
> echo "Layer-2 file" > ./layer-2/some-file-in-layer-2[2020-04-19 16:03:53] [ubuntu] [/tmp/overlay-example]  
> echo "Layer-3 file" > ./layer-3/some-file-in-layer-3

Finally, let’s mount the filesystem:

> sudo mount -t overlay overlay-example \ -o lowerdir =/tmp/overlay-example/layer-1:/tmp/overlay-example/layer-2:/tmp/overlay-example/layer-3,upperdir=/tmp/overlay-example/layer-4,workdir=/tmp/overlay-example/workdir \ /tmp/overlay-example/mount

Now let’s look inside the mount folder:

As expected the content of the folders layer-1, layer-2 and layer-3 have been mounted/combined in the mount folder.
Sure enough if we look at the content of the files, we’ll find what we have written in the previous step.

[2020-04-19 16:13:33] [ubuntu] [/tmp/overlay-example/mount] > cat some-file-in-layer-3 
Layer-3 file

Let’ try to create a file in the mount folder:

[2020-04-19 16:23:31] [ubuntu] [/tmp/overlay-example/mount]  
 > echo "new content" > new-file

[2020-04-19 16:27:33] [ubuntu] [/tmp/overlay-example/mount]  
> ls
new-file  some-file-in-layer-1  some-file-in-layer-2  some-file-in-layer-3

Where should the new file be? In the upper layer, which in our case is the folder called “layer-4”:

[2020-04-19 16:23:49] [ubuntu] [/tmp/overlay-example]  
 pactvm > tree
.
├── layer-1
│   └── some-file-in-layer-1
├── layer-2
│   └── some-file-in-layer-2
├── layer-3
│   └── some-file-in-layer-3
├── layer-4
│   └── new-file
├── mount
│   ├── new-file
│   ├── some-file-in-layer-1
│   ├── some-file-in-layer-2
│   └── some-file-in-layer-3
└── workdir
    └── work [error opening dir]7 directories, 8 files

Let’s try to delete a file:

[2020-04-19 16:27:33] [ubuntu] [/tmp/overlay-example/mount] > rm some-file-in-layer-2 
[2020-04-19 16:28:58] [ubuntu] [/tmp/overlay-example/mount] > ls 
new-file some-file-in-layer-1 some-file-in-layer-3

What do you think happened to the original file in the “layer-2” folder?

[2020-04-19 16:29:57] [ubuntu] [/tmp/overlay-example]  
 pactvm > tree
.
├── layer-1
│   └── some-file-in-layer-1
├── layer-2
│   └── some-file-in-layer-2
├── layer-3
│   └── some-file-in-layer-3
├── layer-4
│   ├── new-file
│   └── some-file-in-layer-2
├── mount
│   ├── new-file
│   ├── some-file-in-layer-1
│   └── some-file-in-layer-3
└── workdir
    └── work [error opening dir]7 directories, 8 files

A new file called “some-file-in-layer-2” was created in “the layer-4”. The weird thing is that the file is a character file. These kinds of files are called “whiteout” files and are how the overlay filesystem represents a file being deleted:

[2020-04-19 16:31:09] [ubuntu] [/tmp/overlay-example/layer-4]  
 pactvm > ls -la
total 12
drwxr-xr-x 2 napicell domain^users 4096 Apr 19 16:28 .
drwxr-xr-x 8 napicell domain^users 4096 Apr 19 16:07 ..
-rw-r--r-- 1 napicell domain^users   12 Apr 19 16:23 new-file
c--------- 1 root     root         0, 0 Apr 19 16:28 some-file-in-layer-2

Now that we have finished with it, let’s unmount the filesystem and remove the folder we created:

[2020-04-19 16:37:11] [ubuntu] [/tmp/overlay-example]  
> sudo umount /tmp/overlay-example/mount && rm -rf *

Wrapping up the overlay filesystems

As we said at the beginning, the overlay filesystem allows to create a union of directories. In our case the union was created in the “mount” folder and it was the result of combining the “layer-{1, 2, 3, 4}” folders. Changes to files, deletion or creation will be stored in the upper dir, which in our case is “layer-4”. This is why this layer is also called “diff” layer.
Files from upper layer shadow the ones in lower layers, i.e. if you have a file with the same name and relative path in layer-1 and layer-2, the layer-2 file is going to end up in the “mount” folder.

In the next section we will see how this is used with docker images.