Run Secure Containerized MongoDB Deployments Using the MongoDB Community Kubernetes Operator

First introduced earlier this year, the MongoDB Community Kubernetes Operator now allows you to run secure MongoDB deployments in your Kubernetes cluster.

The Community Operator is open source, and ideally suited for experimentation, testing, and lightweight production use cases. For larger or more mission-critical workloads with requirements around monitoring, alerting, and data recovery, we recommend the MongoDB Enterprise Kubernetes Operator, available with MongoDB Enterprise Advanced.

This blog tutorial will show you how to deploy and configure a fully secure MongoDB deployment inside Kubernetes from scratch, using the MongoDB Community Kubernetes Operator and cert-manager.

The Community Operator is available here.

Installation

The MongoDB Community Kubernetes Operator allows you to deploy secure MongoDB Replica Sets in your Kubernetes cluster.

1. Before we can deploy MongoDB, we need to ensure that we have created the required CustomResourceDefinition. Note: This operation requires cluster admin permissions.

kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes-operator/master/deploy/crds/mongodb.com_mongodb_crd.yaml

1. Create a namespace for our deployment.

kubectl create namespace mongodb

1. Install the latest version of the operator.

kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-kubernetes-operator/master/deploy/operator/operator.yaml

*Note: If using OpenShift, make sure to reference the OpenShift samples instead.

Deploying a SCRAM Enabled Replica Set

The Community Operator creates secure SCRAM-SHA-256 enabled deployments by default. This means that we need to define our user and what roles we want them to have, alongside a set of credentials for the user to use.

We can create the user's credentials in the form of a Kubernetes Secret

kubectl create secret generic my-mongodb-user-password -n mongodb --from-literal="password=TXs3ZsuIqT-pQFvwxOec"

Once we have created the secret, we can deploy a MongoDB replica set.

---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: mongodb-replica-set
  namespace: mongodb
spec:
  members: 3
  type: ReplicaSet
  version: "4.4.0"
  security:
    authentication:
      modes: ["SCRAM"]
  users:
    - name: my-mongodb-user
      db: admin
      passwordSecretRef: 
        name: my-mongodb-user-password # the name of the secret we created
      roles: # the roles that we want to the user to have
        - name: readWrite
          db: myDb
      scramCredentialsSecretName: mongodb-replica-set

Note: If your application is in the same namespace, it can use this secret to connect to the MongoDB instance. If your application gets the credentials some other way, this secret can be deleted.

If you want to change this user's password in the future, you can simply create a new secret with the same name, or reference a different secret in the resource definition.

Once the MongoDB resource has been created, the operator will create and configure a StatefulSet for this replica set.

You'll notice that each pod consists of 2 containers, the mongod itself, and the mongodb-agent which runs in a sidecar and handles automation of the mongod processes.

Once the MongoDB resource has been created, we can wait for the replica set to get into the “Running” state.

NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-5d757df5c8-d6ll7   1/1     Running   0          5m21s
mongodb-replica-set-0                          2/2     Running   0          2m56s
mongodb-replica-set-1                          2/2     Running   0          2m10s
mongodb-replica-set-2                          2/2     Running   0          72s

Connecting to the Replica Set

Once the resource has been successfully created, we can connect and authenticate to the MongoDB replica set as the user we defined in the resource specification.

Now you can connect to the replica set from your application using the following connection string:

USERNAME="my-mongodb-user"
PASSWORD="$(kubectl get secret my-mongodb-user-password -o  jsonpath='{.data.password}' | base64 -d)"

CONNECTION_STRING="mongodb://${USERNAME}:${PASSWORD}@mongodb-replica-set-0.mongodb-replica-set-svc.mongodb.svc.cluster.local:27017,mongodb-replica-set-1.mongodb-replica-set-svc.mongodb.svc.cluster.local:27017,mongodb-replica-set-2.mongodb-replica-set-svc.mongodb.svc.cluster.local:27017"

We can also connect directly through the mongo shell.

MONGO_URI="$(kubectl get mdb mongodb-replica-set -o jsonpath='{.status.mongoUri}')"

kubectl exec -it mongodb-replica-set-0 -c mongod -- mongo ${MONGO_URI} --username "${USERNAME}" --password "${PASSWORD}"

Note: As our user only has access to the "myDb" database, we only have permissions to read and write to this database.

use myDb
db.col.insert({
 "hello": "world"
})

Configure TLS with Jetstack's Cert Manager

Cert-manager is a Kubernetes add-on from Jetstack which automates the management and issuing of TLS certificates.

The Community Operator is fully compatible with the cert-manager certificate format.

1. Install cert-manager into your cluster: https://cert-manager.io/docs/installation/kubernetes/
2. Generate a certificate authority that will issue the certificates for our replica set.

openssl genrsa -out ca.key 2048

1. Generate a CA certificate, or use your own. Note: If using your own CA certificate, you'll need to make sure a few requirements are met.

• It has a filename of "ca.crt"
• The common name either matches the domain name of all replica set members or has the domain name of all replica set members. View the docs for more details.

When generating the CA certificate, we'll use a wildcard Common Name that matches the domain name of all of the replica set members.

COMMON_NAME="*.mongodb-replica-set-svc.mongodb.svc.cluster.local"

openssl req -x509 -new -nodes -key ca.key -subj "/CN=${COMMON_NAME}" -days 3650 -reqexts v3_req -extensions v3_ca -out ca.crt

1. Create a kubernetes configmap containing the CA.

kubectl create configmap ca-config-map --from-file=ca.crt --namespace mongodb

1. Create a kubernetes secret containing the signing pair in the mongodb namespace.

kubectl create secret tls ca-key-pair  --cert=ca.crt  --key=ca.key --namespace=mongodb

1. Once we have created our key pair, we can create a cert-manager issuer resource from our key pair which will issue the certificates for our MongoDB deployment. Create the following cert-manager issuer custom resource definition and save it as cert-manager-issuer.yaml.

---
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
  name: ca-issuer
  namespace: mongodb
spec:
  ca:
    secretName: ca-key-pair

1. Next, apply the resource definition.

kubectl apply -f cert-manager-issuer.yaml

1. Create a cert-manager certificate resource definition which references the newly created issuer into cert-manager-certificate.yaml.

---
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: cert-manager-certificate
  namespace: mongodb
spec:
  secretName: mongodb-tls
  issuerRef:
    name: ca-issuer
    kind: Issuer
  commonName: "*.mongodb-replica-set-svc.mongodb.svc.cluster.local"
  organization:
  - MongoDB

1. Apply the certificate resource.

kubectl apply -f cert-manager-certificate.yaml

Shortly after we create the Certificate resource, we should see the "mongodb-tls" secret which was created by cert-manager.

kubectl get secret mongodb-tls

NAME          TYPE                DATA   AGE
mongodb-tls   kubernetes.io/tls   3      1m

Without making any modifications to the secret, we can update our MongoDB resource to configure TLS.

We just need to reference both the configmap containing the ca, and the secret that cert-manager generated containing the certificates for our deployment.

---
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
  name: mongodb-replica-set
  namespace: mongodb
spec:
  members: 3
  type: ReplicaSet
  version: 4.4.0
  security:
    tls:
      enabled: true
      certificateKeySecretRef:
        name: mongodb-tls
      caConfigMapRef:
        name: ca-config-map
    authentication:
      modes:
        - SCRAM
  users:
    - name: my-mongodb-user
      db: admin
      passwordSecretRef:
        name: my-user-password
      roles:
        - name: readWrite
          db: myDb
      scramCredentialsSecretName: mongodb-replica-set

NAME                                         READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-5d757df5c8-d6ll7   1/1     Running   0          3h16m
mongodb-replica-set-0                        2/2     Running   1          13m
mongodb-replica-set-1                        2/2     Running   1          13m
mongodb-replica-set-2                        2/2     Running   1          12m

Shortly after the updated MongoDB resource was applied, we should see the members of the replica set back in the “Running” state. Each member will have been restarted once as changing TLS configuration results in a rolling restart of the deployment.

Once the changes have all been applied, we can test our connection over TLS by connecting to any of the mongod containers:

kubectl exec -it mongodb-replica-set-0 -c mongod -- bash

And using the mongo shell to connect using TLS:

mongo --tls --tlsCAFile /var/lib/tls/ca/ca.crt --tlsCertificateKeyFile /var/lib/tls/server/*.pem --host mongodb-replica-set-0.mongodb-replica-set-svc.mongodb.svc.cluster.local

In this blog tutorial we:

• Deployed the MongoDB Community Kubernetes Operator into our Kubernetes cluster
• Created a secure, SCRAM-SHA enabled MongoDB resource and our password in the form of a Kubernetes secret
• Used cert-manager to create TLS certificates for our MongoDB deployment
• And finally, configured our MongoDB resource to enable TLS for our deployment

To get started yourself, download MongoDB Community Kubernetes Operator here.