Puppet: Puppet Enterprise installation and self-signed Intermediate CA

Puppet Enterprise installation and self-signed Intermediate CA (Certificate Authority)

This article is about how to install Puppet Enterprise using your own self-signed Intermediate CA (Certificate Authority).

In some environments, regulations require you to intercept and inspect all SSL traffic to detect malicious activities that could otherwise masquerade as legitimate encrypted traffic. This requires the ability to decrypt and re-encrypt the stream in real time, which can only be done with the proper certificates installed.

Puppet Enterprise in its default installation creates its own Root and Intermediate CA, and maintains them. But in most cases where SSL inspection is mandatory, this is done by using a company-owned CA instead of a self-signed one. To solve this challenge, Puppet Enterprise can use your own Intermediate CA during installation.

Topics we cover in this post:

• Installing Puppet Enterprise with your own Intermediate CA
• Maintenance work needed when using your own Intermediate CA
• Puppet task and Puppet DSL examples to simplify maintenance work

This article will not guide you through the process of creating your own Root and Intermediate CA. I added links with sample openssl.cnf files for the Root CA and the Intermediate CA as well as shell scripts for doing the setup of a Root and an Intermediate CA. All of these links are available at the end of this article.

If your company already has a public key infrastructure (PKI) in place, please contact the administrators of that PKI to get an Intermediate CA to use with Puppet Enterprise.

A quick word about the CAs

For this article we will assume to have a Root CA from which an Intermediate CA for Puppet Enterprise is created. Please refer to the example files linked at the end of this article if you are unsure which options to set in your openssl.cnf files. Keep in mind the settings in these files are the ones I used and these settings might not meet the requirements of your company or environment.

Important:
All certificates, private keys and certificate revocation lists (CRLs) are required in PEM format.

Installing Puppet Enterprise

You will need the following to setup Puppet Enterprise with your own Intermediate CA:

• Root CA public cert
• Intermediate CA public cert(s)
• Intermediate CA private key with at least 4096 bits and without passphrase
• Puppet Enterprise Installer (2019.8.9 used for this article)
• A VM or server to install Puppet Enterprise on. Make sure the VM/server complies to the hardware requirements. For this article a VM installed with latest patched Ubuntu 18.04 LTS was used.

A documentation on how to install Puppet Enterprise with your own Intermediate CA is available here. Please read carefully about what you have to take care of yourself when using your own Intermediate CA.

Create a Cert Bundle

Please keep in mind the order of the certificates. It is starting with the most significant certificate and more common certificates are added afterwards. In our case the certificate of the Intermediate CA comes first and the certificate for the Root CA was added after the certificate of the Intermediate CA. For later use we create a directory called /root/puppet_install and create all bundles in this directory for later use.

mkdir -p /root/puppet_install
cat puppet.cacert.pem > /root/puppet_install/cert_bundle.pem
echo >> /root/puppet_install/cert_bundle.pem
cat cacert.pem >> /root/puppet_install/cert_bundle.pem

puppet.cacert.pem is the Intermediate CA Puppet Enterprise shall use. cacert.pem is the public certificate of the Root CA that signed our Intermediate CA certificate.

Create a CRL Bundle

Please keep in mind the order of the certificates. It is starting with the most significant CRL and more common CRLs are added afterwards. Do not add a CRL for the Puppet Intermediate CA as Puppet manages this CRL itself. So in our case we only have the Root CA CRL to take care of.

cp crl.pem /root/puppet_install/crl_bundle.pem

crl.pem is the current certificate revocation list for the Root CA. Depending on your environment it might be possible to download that crl from a server.

Remove the passphrase from the Intermediate CA private key

Puppet Enterprise needs the Intermediate CA's private key without a passphrase. If there is a passphrase on the private key we need to remove it.

Please verify that the private key file in the puppet_install directory is unprotected.

cat puppet.cakey.pem

If the file starts with lines like the ones below, the key is protected with a passphrase.

-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: AES-256-CBC,A82767871A4E6723433FF6D2C5F743E6

AYEld9mejvxLegCd0hV/dUitJcwdM98lvRQ3KOLGuTGWzH2eGFTXa2BrWSNQEmsy

If the first lines of the key look like the lines below, it is unprotected.

-----BEGIN RSA PRIVATE KEY-----
MIIJKwIBAAKCAgEArmCJkhcke2meOH1eZGtTQmwh7Vil2SJH6lDqJtjudww33aaI

The following command is an example of how you can remove the passphrase protection. The command reads the protected key and writes it to a new location without protecting it with a passphrase.

You need the passphrase to run this command.

openssl rsa -in puppet.cakey.pem \
        -out /root/puppet_install/puppet-ca.key.pem

Now we have all necessary files gathered to install Puppet Enterprise. Copy, if needed, the whole puppet_install directory to the server where Puppet Enterprise will be installed. For this example it was copied to the /root directory.

Puppet Enterprise installation

Now we are preparing our pe.conf file. The pe.conf file contains configuration information for the Puppet Enterprise installer.
Create a file /root/pe.conf with at least the following lines below. If you need some more configurations in the pe.conf file please consult the Puppet Enterprise Documentation about the available configuration options.

{
  "console_admin_password": "<your very secret admin password>"
  "puppet_enterprise::puppet_master_host": "<puppet server fqdn>"
  "pe_install::signing_ca": { 
    "bundle": "/root/puppet_install/cert_bundle.pem" 
    "crl_chain": "/root/puppet_install/crl_bundle.pem" 
    "private_key": "/root/puppet_install/puppet-ca.key.pem" 
  }
}

Change into the Puppet Enterprise installer directory and run the installer as follows:

./puppet-enterprise-installer -c /root/pe.conf

If there are any problems with the certificate bundles or the pe.conf file you will see error messages during the installation. Please consult the installer's logfiles in the /var/log/puppetlabs/installer directory.

Check auth.conf file

After the Puppet Enterprise install is finished check the /etc/puppetlabs/puppetserver/conf.d/auth.conf file. Open the file and search for entries for the url /puppet-ca/v1/certificate_revocation_list. Make sure that the type is set to regex. Here is an example of what these entries should look like:

"match-request": {
  "method": "put",
  "path": "^/puppet-ca/v1/certificate_revocation_list$",
  "type": "regex"
},

 "match-request": {
  "method": [
    "put"
  ],
"path": "^/puppet-ca/v1/certificate_revocation_list$",
  "query-params": {},
  "type": "regex"
},
"name": "puppetlabs certificate revocation list",
"sort-order": 500

If you change these entries, please restart your Puppet Enterprise server.

Maintain CRLs in Puppet Enterprise

As you use your own Intermediate CA and therefore your own Root CA(s) you should take care to upload new CRLs for the Root CA(s) to Puppet Enterprise periodically. Make sure to upload any CRLs before they expire. The Puppet Enterprise Console shows the CRLs and their expiration dates.

Certification revocation list overview in Puppet Enterprise with expiration date and days until expiration.

There are more ways to maintain CRLs in Puppet Enterprise. We will focus on the following three methods:

• Shell script
• Puppet task
• Puppet DSL

Shell script

Here is a short example script to upload CRLs to Puppet Enterprise.

#!/bin/bash

if [ $# -ne 1 ] ; then
    prog=`basename $0`
    echo "usage: $prog <crl file>"
    exit 2
fi

type_header='Content-Type: text/plain'
cert="$(puppet config print hostcert)"
cacert="$(puppet config print localcacert)"
key="$(puppet config print hostprivkey)"
uri="https://$(puppet config print server):8140/puppet-ca/v1/certificate_revocation_list"
curl --insecure --cert "$cert" --cacert "$cacert" --key "$key" \
     --header "Content-Type: text/plain" --header "Accept: text/plain" \
     --request PUT --data-binary "@$1" "$uri"
ret=$?
echo

exit $ret

The above script is available for download. Please look at the links section below.

Puppet task

From the above shell script it's easy to create a Puppet task. We assume that in your control repository there is a directory site-modules/adhoc/tasks. We will extend the script above to only run on the Puppet Primary server and add some useful metadata.

Create two files upload_new_crl.sh and upload_new_crl.json in the *site-modules/adhoc/tasks folder. The files should look like these below. You can download the files from the links section.

#!/bin/bash

BINDIR=/opt/puppetlabs/bin

if [ -x /bin/hostname ] ; then
  hostcmd='/bin/hostname'
else
  hostcmd='/usr/bin/hostame'
fi

if ${BINDIR}/puppet config print server | grep -v -q `${hostcmd}`; then
  echo "This task can only be run on the PE Primary server!"; 
  exit 1
fi

crl_file=$PT_crl_file

if [ -f "$crl_file" ] ; then

    type_header='Content-Type: text/plain'
    cert="$(puppet config print hostcert)"
    cacert="$(puppet config print localcacert)"
    key="$(puppet config print hostprivkey)"
    uri="https://$(puppet config print server):8140/puppet-ca/v1/certificate_revocation_list"
    curl --insecure --cert "$cert" --cacert "$cacert" --key "$key" \
            --header "Content-Type: text/plain" \
            --header "Accept: text/plain" \
            --request PUT --data-binary "@$crl_file" "$uri"

    echo

else
    echo "The file ${crl_file} is not available or not readable."
    exit 2
fi

exit 0

Puppet task metadata:

{
  "description": "Upload a new crl file to the Puppet Primary server. The crl file must be provided in PEM format.",
  "input_method": "environment",
  "parameters": {
    "crl_file": {
      "description": "File with new certificate revocation lists to upload.",
      "type": "String[1]"
    }
  }
}

Tip:
The task metadata will show up in the Puppet Enterprise Console. This can help other people to understand the task and the parameters.

Puppet DSL

The third method is to write a class in Puppet's declarative language (DSL) and let the Puppet agent start an upload of a new CRL as soon as the CRL file changes. The class below is written as a profile which can be added to your control repository. This Puppet code is intended to run on the Puppet primary server.

# @summary Upload new crl to Puppet Primary
#
# This code example used the same shell script as the task to upload a new crl
# into Puppet Primary server. Keep in mind that running this class the first time
# will fire the upload task and upload the crl file in this repository.
#
class profile::crl_upload (
  Stdlib::Absolutepath $crl_base_dir = '/usr/share/crl_upload',
) {
  $crl_file = "${crl_base_dir}/crl/crl.pem"

  file { $crl_base_dir:
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0700',
  }

  file { "${crl_base_dir}/bin":
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0700',
  }

  file { "${crl_base_dir}/crl":
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0700',
  }

  file { $crl_file:
    ensure => file,
    source => 'puppet:///modules/profile/crl.pem',
    owner  => 'root',
    group  => 'root',
    mode   => '0600',
    notify => Exec['upload_crl'],
  }

  file { "${crl_base_dir}/bin/upload_new_crl.sh":
    ensure  => file,
    content => epp('profile/upload_new_crl.sh.epp', {
        crl_file => $crl_file,
    }),
    owner   => 'root',
    group   => 'root',
    mode    => '0700',
  }

  exec { 'upload_crl':
    command     => "${crl_base_dir}/bin/upload_new_crl.sh",
    path        => ["${crl_base_dir}/bin"],
    refreshonly => true,
  }
}

The Puppet class together with the upload_new_crl.sh.epp template can be downloaded from the links section.

Next steps

You have to take care about your CRLs and your Puppet agents. If you downloaded the openssl.cnf files I provided, the CRLs are valid for 30 days. You have to take care to create new CRLs in time and upload them to your Puppet server. It is important that you take care of this and make sure the process works. An expired CRL anywhere in the chain causes certificate validation failures!. Choose one of the three methods above or create your own automation workflow to keep your CRLs up to date.

Configure your Puppet agents to download CRL updates by setting crl_refresh_interval in the puppet_enterprise::profile::agent class to a suitable value. You can do this either in the Puppet Enterprise Console or within your Hiera configuration files.

Recap

The Puppet Enterprise installer normally creates a Root CA and an Intermediate CA during installation. After the installation, Puppet Enterprise maintains the CAs itself.

In some environments it is necessary to intercept SSL connections due to regulations. In that case you can use your own Root and Intermediate CA with the Puppet Enterprise installer. But using your own CAs will hand the responsibility to take care of CRLs over to you.

Links

• Read more on Root CA on Github here.
• Read more on Intermediate CA on Github here.
• Get the CA Setup script. This shell script contains all the commands to create a Root CA and an Intermediate CA. The script will also create the files needed for the Puppet Enterprise installer. You are strongly advised to review it and change it to your needs before you use it!
• Learn how to unload_new_crl.sh.
• Learn how to upload_new_crl.json.
• Get the crl_upload.