Bare Metal in a Cloud Native World

What do you think of when you hear the term “bare metal”? To me, it conjures up images of something that’s very tangible, rows upon rows of servers connected together with colorful cables, with loud fans spinning and locked away in a basement or warehouse.

As an Ambassador for the Cloud Native Computing Foundation , I spent most of my time much higher up in the stack , at “level 7” (application) and at “level 8” (people). I’ve become accustomed to using APIs and tools like Terraform to get compute whenever I need it. One of the defining parts of my career has involved persuading people to  forget about servers .

So what does bare metal have to do with “cloud native,” and if you’re new to this space, what do you need to know about it?

I’m spending some time getting familiar with bare-metal provisioning projects. What’s your favourite, and why? #baremetal #provisioning #oss #devops #itoperations #hybridcloud #privatecloud #onprem #pxe #firmware pic.twitter.com/ClOpwugPbW

— Alex Ellis (@alexellisuk) April 14, 2020

I asked my followers on Twitter to share their favorite bare-metal tooling and that resulted in a curated list: awesome bare metal. Read on to understand how bare metal compares to cloud native, and some of the concepts involved and challenges faced by administrators.

The Cloud Native Horizon

When it comes to deploying a piece of code, let’s say a REST API — there’s nothing simpler than heading over to your AWS dashboard and pasting the code into a text box and seeing your code execute with per-second billing and autoscaling from AWS Lambda. The overall interface for Lambda functions has changed very little since its inception, making it a relatively stable platform to work with.

AWS Lambda is a SaaS product, which is closed-source, so if you are worried about the risk of getting locked into a single vendor, you may be looking at Kubernetes.  Kubernetes, in contrast to AWS Lambda, allows portability between clouds by leveraging Docker or OCI-format container images. Once packaged in an image and pushed to a registry, your code can be moved between Kubernetes clusters with relative ease. If you enjoy using managed products for the reduced maintenance burden, there are plenty of Kubernetes services available that are quick and easy to set up.

One of the drawbacks of a platform like Kubernetes is that it changes at a rate of knots. Between each minor release, you may see breaking changes and have to rewrite any integrations you have, not to mention the third-party tools and code generation tools, which change at a similar cadence. For this reason, managed clouds like Amazon EKS often run several releases behind the community.

See also:  your team may not need Kubernetes

The most popular software in the Cloud Native Computing Foundation (CNCF) focuses on the top level of the stack. Kubernetes was the first project in the foundation and many of the subsequent projects are complementary to it — at an operational level, or at an organizational one. Tools like Prometheus and AlertManager increase operational efficiency and allow for monitoring of our many services.  NATS provides high-speed messaging across local networks and the internet.  Linkerd builds a mesh between containers to add metrics, policies, and end-to-end encryption.

As a practitioner and maintainer of a popular Serverless project , I often get told “but there are servers in serverless.” It’s almost like they thought I didn’t know. For me, serverless has always been about the developer’s focus: it is not about hardware specs and network VLANs, but about APIs. Reasoning in terms of APIs is the cloud native way.

In a similar way, bare-metal servers are the substrate for Kubernetes, and for all cloud native applications, whether accessed directly, via a hypervisor, or through an IaaS provider’s API.

You should know that “they are in there somewhere,” and for many that’s as far as it needs to go unless you have a specific need for them. Some companies like Cherry Servers ,  AWS and  Packet provide a nice balance between the  advantages of bare-metal isolation and performance, and the powerful APIs usually associated with VMs.

For the rest of the post, please assume that we’re talking about self-managed bare metal.

With bare metal: what you see is what you get, quite literally. There are no confusing marketing terms like “serverless,” instead you get GBs of RAM, Gbps of network bandwidth and GHz of CPU. That is quite refreshing, and it’s also something that hasn’t changed very much since inception.

Around 20 years ago I worked with bare-metal computers on a daily basis at my school. I helped out the network admin staff by installing Operating Systems onto i386 and i486 and newer Intel Pentium processors. The machines back then still had hard disks, RAM, network cards, and storage, just like they do today.

We had around five labs with 30 computers+ in each and sometimes that would involve walking up with a CD-ROM and reinstalling the operating system, but other times it would involve booting the computer over the network to deploy an image to it remotely. This was much more scalable and took much less time than walking around with a handful of CDROMs.

That’s where it all starts with bare metal, with the operating system, and once it has been installed, you probably won’t be able to tell the difference between it and an EC2 Virtual Machine running on AWS.

I mentioned that Kubernetes moves at a relatively rapid pace. It turns out that hardware does not. We’re using the same tools and technology to boot systems on the network that I was using in the lab 20 years ago.

Bare Metal Glossary

Just as Kubernetes and Cloud have their own set of jargon, bare metal has its own terminology. I’ve put together a quick glossary on concepts and tools:

• Network card — where you physically connect a computer to a network with a cable. This may be copper, or in some instances fiber optic . Some computers have more than one network card or port.
• Management port — this is a server-specific concept. For efficiency, administrators need to manage computers remotely without plugging in a keyboard and mouse.
• Intelligent Platform Management Interface (IPMI) — management interfaces tend to be vendor-specific and are accessed over the network using a client as a Java interface.
• Wake on Lan (WoL) — rather than allowing remote-management WoL can be used to turn on a computer remotely.
• PXE (Preboot Execution Environment) — used to boot computers over a local network with only a network card being required. iPXE can be used to extend an existing PXE firmware using TFTP, or can be flashed to some network cards directly.
• iPXE — a newer open source network booting firmware that also allows for booting over the HTTP and the internet.
• Netbooting — booting from the network means that you don’t need physical access to a computer to configure it or install an operating system.
• DCHP — assigning an IP address and other metadata like the primary DNS server to a network interface
• TFTP (trivial file transfer) — a UDP-based file-server used for fetching firmware to boot over the network.
• NFS (Network File System) — NFS is one of the most common filesystems used with netbooting or file-sharing to allow a Linux computer to work without its own disk. Unfortunately, NFS is not compatible with the overlay filesystems that are used with containers
• iSCSI (Internet Small Computer Systems Interface) — an alternative to NFS which provides block-level devices instead of a networked filesystem. You can format the disk however you like, with an ext4 filesystem and even run Docker.
• Thin client — projects like the Linux Terminal Server Project LTSP  allow you to turn any PC into a thin client without any local storage required. This could be useful for IoT devices like the  Raspberry Pi  which need to rely on flash storage that has a relatively short life-time before it wears out
• Operating System — whether deploying Windows, Linux, or something else. The operating system typically has to be installed using an interactive UI, CLI, or via a predefined configuration.

Not all bare metal is equal. Consumer equipment like a workstation, home PC, an Intel NUC, or a Raspberry Pi is unlikely to come with a management port for IPMI.

Netbooting Example

Here’s a sample workflow for booting a computer over the network:

• The bare-metal server is turned on.
• The network card attempts to boot over the network using PXE.
• A DHCP request is sent for an IP address.
• A DHCP response is received with an IP and a hint of where to look for boot firmware.
• The PXE process now fetches the firmware from the TFTP server and loads it.

At that point a filesystem may be mounted over the networking using NFS and an operating system will be run remotely, or a temporary environment will be used to install an OS to the local system. On a subsequent boot, the hard disk will be used to load the OS.

Some of the concepts like netbooting and network interfaces have a crossover with virtual machines. The equivalent of IPMI on AWS would be your EC2 Dashboard where you can select a disk image to boot up, and customise the behaviour of the machine.

Why Do We Need Bare-Metal Provisioning Tools?

I recently asked Twitter what their favorite bare-metal provisioning tools were. There were many different answers, with gravity around a set of 5-10 projects with some of them being newer, and others more established.

It turns out that whilst bare metal and the low-level tools like DHCP, TFTP and NFS haven’t changed much over 20 years, people are trying to make them easier to automate. Many data centers contain heterogeneous hardware, some with RAID arrays, some without, some with one disk, some with two, along with varying firmware and capabilities.

Provisioning tools need to help us with:

• Servers — software servers to provide or configure DHCP, TFP, NFS, HTTP and so forth.
• Inventory — listing and collecting server and their capabilities.
• Image store — a store of OS images ready to be deployed to computers over the network. These images often need to be customized, so may be built with a tool like Packer.
• Delivery — chaining together the legacy tools to create a secure way to install an operating system, some projects refer to this as “workflow,” others use a state machine.

I’m spending some time getting familiar with bare-metal provisioning projects. What’s your favourite, and why? #baremetal #provisioning #oss #devops #itoperations #hybridcloud #privatecloud #onprem #pxe #firmware pic.twitter.com/ClOpwugPbW

— Alex Ellis (@alexellisuk) April 14, 2020

You can see the results of the Tweet in my curated awesome bare-metal GitHub repository . It covers bare-metal provisioning software along with tools that make it easier to work with low-level networking such as  MetalLB and  inlets .

Here’s a short selection of projects that the community mentioned:

• Digital Rebar — “Digital Rebar is the data center automation, provisioning and infrastructure as code (IaC) platform designed with a cloud native architecture replacing Cobbler, Foreman, MaaS or similar technologies.”
• MAAS “Metal as a Service” by Canonical.
• Ironic — “A service for managing and provisioning bare-metal servers,” from OpenStack Foundation.
• Netboot.xyz — “a way to PXE boot various operating system installers or utilities from one place within the BIOS without the need of having to go retrieve the media to run the tool.”
• Plundr — “Plunder is a single-binary server that is all designed in order to make the provisioning of servers, platforms and applications easier.” Written by Dan Finneran.
• Tinkerbell — “Tinkerbell is a bare-metal provisioning engine. It’s built and maintained with love by the team at Packet.”

The most popular tools help automate the various tools we looked at in the sample workflow above: PXE, DHCP, TFTP, NFS, hard-disk preparation, and OS installation. Once installed, a Kubernetes cluster could be bootstrapped, but these tools are designed to be more general purpose. Once your Operating System is installed, a management tool like SSH, Ansible, Puppet or Chef is generally used to manage the computer and its software packages.

Now a number of newer tools that the community shared are focused on building upon these tools to bring up a Kubernetes cluster, and so we close the loop on bare metal vs. Kubernetes. Examples include:

• Metalk8s — “MetalK8s was launched to make it easier to run Kubernetes (K8s) on bare-metal servers that need persistent storage” by Scality.
• Metal Stack — “We believe Kubernetes runs best on bare metal. We build an api to manage bare metal hardware and Kubernetes on top of that.”
• Metal³ — “Bare Metal Host Provisioning for Kubernetes” — with ClusterAPI support and built with Ironic.

Of the Kubernetes tools mentioned, some go a step further and embrace an abstraction called Cluster API (CAPI) . CAPI aims turn a set of VMs or servers into a functional Kubernetes cluster.

The Cluster API is a Kubernetes project to bring declarative, Kubernetes-style APIs to cluster creation, configuration, and management. It provides optional, additive functionality on top of core Kubernetes to manage the lifecycle of a Kubernetes cluster.

More on Cluster API:

• A list of Cluster API providers.
• An example of the Docker provider in YAML.

Looking Forward

When considering modern cloud computing there’s a spectrum of control and portability. At one end, we have AWS Lambda, a closed-source SaaS product that’s highly proprietary, and offers little control, but high utility and efficiency, moving up we have Kubernetes which brings a scale-out superpowers, but at a cost, and finally we have bare-metal servers that need to be managed thoughtfully with specialist tools. Personally, I think the intersection of the two is interesting, which is why I’m excited about tools like Metal³ and Metal Stack which aim to smooth out the experience of going from heterogeneous bare-metal hardware to Kubernetes.

I’m also watching the Tinkerbell and  Plundr projects closely. Plundr takes a monolithic approach, bundling as much as possible into a single binary and simple workflow engine —  “turning bare-metal into cloud” .

Tinkerbell is a set of microservices-based upon the work of Packet over the past six years designed with security and heterogeneous hardware in mind. I like the way that each step in a Tinkerbell provisioning workflow is defined as a Docker image. You define a Dockerfile and then build an image and store it in a registry, enabling portability, versioning and reproducible artifacts.

You may also like:

• Fork or Star  awesome bare metal on GitHub.
• Read the  ClusterAPI docs.
• K3sup — bootstrap k3s clusters over SSH.
• How to  Netboot a Raspberry Pi 3.
• Netbooting fundamentals.

Have you got comments, questions, or suggestions? Email  [email protected] or  reach-out on Twitter .

Amazon Web Services and the Cloud Native Computing Foundation are sponsors of The New Stack.

Feature image by Jarkko Mänty from  Pixabay .