How to deploy chef server

Overview of the content

Our very own Aleks Kuzminsky, have blogged about the next set of challenges we face in the MySQL World. Getting our hands dirty, the first step is to have a solid and reliable way to manage our servers. This is a special need of any service that must deal with persistent data.

In this article, I will show you how to deploy a chef server of your own aws infrastructure. This isn’t a simple task by any means, a chef server includes a lot of moving parts. However, I will try to get you to a reasonable state today. We will touch on these components:

Creating an EC2 instance
IAM roles and profiles
Cloudinit settings
Chef cookbook to run on the chef server itself

We will be using two separate tools to do this:

Terraform will build the instance
RevDB public chef server cookbook will configure it

In the future, if you are interested in any of the following topics, please let me know in the comments section and I’ll make sure to write up an article on them.

Automated backup and restore
Build and distribute version controlled recipes (CI/CD)
Running your own chef supermarket
Managing secrets dynamically without databags, using terraform
DNS, Route53 and SES configurationn

Creating the instance

This part will be the simplest of all. There are only a few non-standard components here. Because of the the load, you might considering running the chef server on a larger instance, in our case a t3.large. Fortunately, in terms of disk size, it doesn’t require as much, this example setup will run on only 32gb. For any other components, not mentioned here, take a look at the terraform documentation.

resource "aws_instance" "chef-server" {
  ami           = "ami-02eac2c0129f6376b" # AMI ids are region specific
  instance_type = "t3.large"
  subnet_id     = "subnet-034abffb3cc259c69" # In our example, this is a public subnet
  key_name      = "deployer_key"
  tags = {
    Name : "chef-server"
  }
  root_block_device {
    volume_size = "32gb"
  }
  iam_instance_profile = aws_iam_instance_profile.chef_server.name # See the IAM section below
  user_data = data.template_cloudinit_config.config.rendered # See the cloudinit section below
}

IAM roles and profiles

The IAM settings you really required to setup are, with links to the documentation are:

Now, I’m sure I won’t be completely alone when I say, this was really confusing first. But here is the break down as I understand it:

You can assign an iam_instance_profile to an aws_instance
Any iam_instance_profile requires a aws_iam_role
An aws_iam_role requires a name and an assume_role_policy
And this is the shaky part, because we have to use aws_iam_role_policy_attachment to attach the aws_iam_policy to an aws_iam_role

data "aws_iam_policy_document" "chef_server" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

data "aws_iam_policy_document" "chef_server_access" {
  # this section will be used for backups later.
  statement {
    actions = [
      "s3:*" 
    ]
    resources = [
      aws_s3_bucket.backups.arn,
      "${aws_s3_bucket.backups.arn}/*"
    ]
  }
}

resource "aws_iam_policy" "chef_server_access" {
  name   = "chef_server_permissions"
  policy = data.aws_iam_policy_document.chef_server_access.json
}

resource "aws_iam_role_policy_attachment" "chef_server_permissions" {
  policy_arn = aws_iam_policy.chef_server_access.arn
  role       = aws_iam_role.chef_server.name
}

resource "aws_iam_role" "chef_server" {
  name               = "chef_server"
  assume_role_policy = data.aws_iam_policy_document.chef_server.json
}

resource "aws_iam_instance_profile" "chef_server" {
  role = aws_iam_role.chef_server.name
}

Cloudinit

This is the next big step, because how the instance will become an actual chef server, gets configured here.

The cloud init user data have three different components:

Configuring Attributes for the chef server. It will use these attributes to bootstrap itself.
Deploy a basic chef server configuration such as installing packages, placing certificates etc
Bootstrap the server itself

As a disclaimer, the cookbook below is tailored to our specific needs. It uses artifactory, SecretsManager and other components. Therefore I’d recommend to fork it and adjust bits of it to your needs. Any feedback, improvements, bugfixes are highly appreciated.

This code snippet here, referenced on the top as:

user_data = data.template_cloudinit_config.config.rendered

data "template_cloudinit_config" "config" {
  gzip          = true
  base64_encode = true

  part {
    content_type = "text/part-handler"
    content      = file("${path.module}/part-handler.py")
  }

  part {
    content_type = "text/chef-attributes"
    content = jsonencode(
      {
        "chef-server" : {
          "accept_license" : true,
          "admins" : var.admins,
          "ssh_public_keys" : var.admins_ssh_keys,
          "cookbook_revision" : var.chef_server_cookbook_version
        },
        "run_list" : ["recipe[chef-server::default]"]
      }
    )
  }

  part {
    content_type = "text/chef-solo"
    content      = "cookbook_path '/var/cache/cookbooks'"
  }

  part {
    content_type = "text/cloud-config"
    content = format(
      "#cloud-config\n%s",
      yamlencode(
        {
          write_files : [
            {
              content : file("${path.module}/RPM-GPG-KEY-CHEF"),
              path : "/etc/pki/rpm-gpg/RPM-GPG-KEY-CHEF"
            },
            {
              content : "export CHEF_LICENSE=accept",
              path : "/etc/profile.d/chef-license.sh"
            },
            {
              content : "",
              path : "/etc/chef/accepted_licenses/chef_dk"
            },
            {
              content : "",
              path : "/etc/chef/accepted_licenses/chef_infra_client"
            },
            {
              content : "",
              path : "/etc/chef/accepted_licenses/inspec"
            }
          ]
          yum_repos : {
            chef-stable : {
              baseurl : "https://packages.chef.io/repos/yum/stable/el/7/\\$basearch/",
              name : "chef-stable",
              enabled : true,
              gpgkey : "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CHEF"

            }
          }
          packages : [
            "chef",
            "chefdk",
            "git"
          ]
          # This uses our publicly available cookbook-chef-server repository
          runcmd : [
            "export HOME=/var/cache/cookbooks",
            "mkdir -p /var/cache/cookbooks/",
            "git clone https://github.com/revenants-cie/cookbook-chef-server.git /var/cache/cookbooks/chef-server",
            "cd /var/cache/cookbooks/chef-server",
            "git checkout ${var.chef_server_cookbook_version}",
            "chef exec berks vendor ../",
            "chef-solo -j /etc/chef/node.json -c /etc/chef/solo.rb --logfile /var/log/chef-solo.log"
          ]
        }
      )
    )
  }
}

Chef Server Cookbok

We are almost there, but we have to cover a few aspects of the cookbooks repository. This repository is public, because we believe, with this and future posts attached, it can get people through some difficult early problems with chef. However, it is tailored to our own specific needs and this is what we are currently using to bootstrap our own chef server.

For the sake of this article, to make it work for you after forking the repository, make sure you update the link in the template_cloudinit_config resource above to your own git repo, and modify the default.rb like this:

#
# Cookbook:: chef-server
# Recipe:: default
#
# Copyright:: 2019, Revenants CIE, LLC, All Rights Reserved.
# As per https://docs.chef.io/install_server.html
# Chef Software requirements
# https://docs.chef.io/install_server_pre.html#software-requirements

include_recipe 'chef-server::awscli'
include_recipe "chef-server::aws_config"
include_recipe "chef-server::packages"
include_recipe "chef-server::chef-license"
include_recipe "chef-server::cookbook"
include_recipe "chef-server::users"
include_recipe "chef-server::chef-server"
include_recipe "chef-server::chef-solo"
include_recipe "chef-server::chef-cleanup"
include_recipe "chef-server::healthcheck"

We use Artifactory for repository management and Datadog for our monitoring needs. On demand, I would be happy to share how to active the rest of the cookbooks, such as emails, node cleanup and such. Let me know in the comments.

Once the instance is up, you should be able to navigate to it’s IP address in your browser.

Conclusion

Building a chef server is difficult. Even this example isn’t a clear cut copy-paste solution, but do I hope it helps to get going and bring up your own version.

There are endless challenges in this situation and we are facing new ones all the time. There are interesting ones, such as how can you avoid using databags, how to build a CI/CD environment and there are others which are rather hurtful, such as configuring terraform to create a public domain on a different aws account and pass SES verification.

Nonetheless, this is fascinating for me and if you’d like to hear more, let me know.

Overview of the content

Creating the instance

IAM roles and profiles

Cloudinit

Chef Server Cookbok

Conclusion

Recommend

How to backup and restore chef server

PCI DSS compliance with Infrastructure as Code

How to set up MySQL RDS with Terraform

Highly available Chef Server

Why manage GitHub with Terraform

Invoking REST APIs asynchronously with Quarkus

Comparing synchronous and asynchronous Access to Postgres

Using the OpenJ9 JVM for Quarkus Applications

Workshops: Reactive Apps with Quarkus and OpenShift

Deploying CodeReady Workspaces on the IBM Cloud

About Joyk