In my previous post we could see how to get an external IP for load balancing on a k3s cluster running in multipass VMs and I promised to show you how MetalLB can work with k3d launched k3s clusters on Mac too (Linux users are lucky, MetalLB works out of the box with k3d).

k3d is the default and in most cases the preferred deployment tool for k3s clusters on our machines, it uses docker to run a k3s cluster within few seconds in docker containers. If you want to run RIO on k3s on your machine, you need to have k3s running on VMs, e.g. on multipass VMs.

But if it comes to exposing a service with the type LoadBalancer on k3s, we’ll face some problems, since docker desktop on Mac doesn’t support network routing into the host virtual machine which is created using hyperkit on Mac and the main reason is due to the fact that the network interface options used to create the hyperkit instance does not create a bridge interface between the Physical Machine and the Host Virtual Machine. To overcome this issue Almir Kadric has developed a tuntap support shim installer for Docker on Mac and this post is about how to use this nice implementation with MetalLB on a k3d launched k3s cluster on your machine.

Step Zero: Prerequisites

Docker: you need docker running on your machine:

$ docker --version
Docker version 19.03.5, build 633a0ea

Step One: k3s deployment with k3d

To install k3d on Mac or Linux using Homebrew or curl, you need only to run:

$ brew install k3d
or
$ curl -s https://raw.githubusercontent.com/rancher/k3d/master/install.sh | TAG=v1.3.4 bash$ k3d -vk3d version v1.3.4$ k3d -h# help is helpful :-)

Step Two: Deploy k3s with k3d

If you’re using k3d for the first time, deploying k3s with k3d is a breeze away, you won’t believe how fast you can get a k3s made k8s cluster running within a few seconds. The following commands create a 4 node cluster, get and export the kubeconfig, list the cluster and show the nodes:

$ k3d create --workers 3$ k3d get-kubeconfig --name=’k3s-default’$ export KUBECONFIG=~/.config/k3d/k3s-default/kubeconfig.yaml$ k3d list$ kubectl get nodes

Step Three: Deploy MetalLB

With the following commands we will clone our repo, deploy MetalLB controller and speakers, find the external IP of the traefik service, which is exposed with the external IP 172.20.0.2 in my case, and adapt the IP range in the MetalLB ConfigMap an apply it.

$ git clone https://github.com/arashkaffamanesh/k3d-k3s-metallb.git$ cd k3d-k3s-metallb
$ kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.3/manifests/metallb.yaml$ kubectl get svc -n kube-system | grep traefik | awk '{ print $4 }'
172.20.0.2$ vim metal-lb-layer2-config.yaml
# adapt the addresses field to something like this 172.20.0.3–172.20.0.254 in metal-lb-layer2-config.yaml$ kubectl create -f metal-lb-layer2-config.yaml
$ curl 172.20.0.2# this can’t work as expected on MacOS, docker needs some tweaking
# on Linux this works out of the box

As we can see the last command `$ curl 172.20.0.2` can’t work on Mac, we might expect to get a 404 from the traefik default backend, since we don’t have any network route to 172.20.0.0/24 network yet.

Again, as mentioned at the beginning, this is due to the fact that:

Docker desktop on Mac doesn’t support network routing into the host virtual machine which is created using hyperkit on Mac and the main reason is due to the fact that the network interface options used to create the hyperkit instance does not create a bridge interface between the Physical Machine and the Host Virtual Machine.

Now we need a TunTap bridge and some tweaking by routing to overcome this issue and here comes the magic and the great implementation by Almir Kadrik.

Step Four: Docker TunTap to rescue

To get a bridge as a gateway into the hyperkit instance of docker, we need to create a tap interface and initialize it, create a static route to the BareMetal network defined previously in the ConfigMap.

Again, the 172.20.0.0 network might have another value on k3d launched k3s cluster and needs to get adapted accordingly before running the `route add` command below, you can find the network with:

$ kubectl get svc -n kube-system | grep traefik | awk '{ print $4 }'
172.20.0.4

At this point, before installing TunTap and bringing up the interface, we’ll stop the cluster, add a static route and start the cluster and verify if we can get into the cluster via the tap interface by curling the external IP of the traefik backend:

$ k3d stop k3s-default$ ./docker-tuntap-osx/sbin/docker_tap_install.sh# wait till docker is restarted$ ./docker-tuntap-osx/sbin/docker_tap_up.sh$ ifconfig# you should see now tap1 is up with:
# inet 10.0.75.1 netmask 0xfffffffc broadcast 10.0.75.3$ ping -c1 10.0.75.1$ ping -c1 10.0.75.2$ sudo route -v add -net 172.20.0.0 -netmask 255.255.255.0 10.0.75.2$ k3d start k3s-default$ kubectl get svc -n kube-system | grep traefik | awk '{ print $4 }'
172.20.0.4$ curl 172.20.0.4404 page not found :-)

Step Five: Deploy the `whoareyou-service` with type LoadBalancer

The `404 page not found` output above is fine, it means we are hitting the default backend and are ready to go and deploy a service with the type LoadBalancer in the default namespace, print the external IP of the service and curl it:

$ cat whoareyou.yml
$ kubectl create -f whoareyou.yml$ kubectl get svc | grep whoareyou-service | awk '{ print $4 }'
172.20.0.3$ curl 172.20.0.3Hostname: whoareyou-deployment-7db6694b98–7f7lc...# run curl again:$ curl 172.20.0.3Hostname: whoareyou-deployment-7db6694b98–86r2d...

As we can see, after firing the curl command several times, we hit another container running in a different docker worker node and therefore we get a different host name, which means load balancing works with MetalLB on k3s with k3d as well.

Step 6: Cleanup

To cleanup the tap interface and delete the cluster, please run:

$ ./docker-tuntap-osx/sbin/docker_tap_uninstall.sh
$ k3d delete k3s-default

Step 7: Learn more about the awesomeness of k3s

This post was NOT about k3s, if you’d like to learn about k3s under the hood, please enjoy Darren Shepherd’s talk about “K3s under the Hood” at KubeCon in San Diego and visit the k3s Github page and give it a star :-)

We’re hiring!

We are looking for engineers who love to work in Open Source communities like Kubernetes, Rancher, Docker, etc.

If you wish to work on such projects please do visit our job offerings page.