Cross-site replication has technical preview status and is not recommended for production environments.

Exposing instances of the MongoDB cluster¶

You need to expose all Replica Set nodes (including Config Servers) through a dedicated service to ensure that Main and Replica can reach each other, like in a full mesh:

This is done through the replsets.expose, sharding.configsvrReplSet.expose, and sharding.mongos.expose sections in the deploy/cr.yaml configuration file as follows.

spec:
  replsets:
  - rs0:
    expose:
      enabled: true
      exposeType: LoadBalancer
    ...
  sharding:
    configsvrReplSet:
      expose:
        enabled: true
        exposeType: LoadBalancer
      ...

The above example is using the LoadBalancer Kubernetes Service object, but there are other options (ClusterIP, NodePort, etc.).

To list the endpoints assigned to Pods, list the Kubernetes Service objects by executing kubectl get services -l "app.kubernetes.io/instance=CLUSTER_NAME" command.

Configuring cross-site replication on Main site¶

The cluster managed by the Operator should be able to reach external nodes of the Replica Sets. You can provide needed information in the replsets.externalNodes and sharding.configsvrReplset.externalNodes subsections of the deploy/cr.yaml configuration file. Following keys can be set to specify each external Replica, both for its Replica Set and Config Server instances:

• set host to URL or IP address of the external replset instance,
• set port to the port number of the external node (or rely on the 27017 default value),

Optionaly you can set the following additional keys:

• priority key sets the priority of the external node (2 by default for all local members of the cluster; external nodes should have lower priority to avoid unmanaged node being elected as a primary; 0 adds the node as a non-voting member),
• votes key sets the number of votes an external node can cast in a replica set election (0 by default, and 0 for non-voting members of the cluster).

Here is an example:

spec:
  unmanaged: false
  replsets:
  - name: rs0
    externalNodes:
    - host: rs0-1.percona.com
      port: 27017
      priority: 0
      votes: 0
    - host: rs0-2.percona.com
    ...
  sharding:
    configsvrReplSet:
      size: 3
      externalNodes:
        - host: cfg-1.percona.com
          port: 27017
          priority: 0
          votes: 0
        - host: cfg-2.percona.com
        ...

The Main site will be ready for replication when you apply changes as usual:

$ kubectl apply -f deploy/cr.yaml

$ kubectl get secret my-cluster-name-secrets -o yaml > my-cluster-secrets.yaml

Configuring cross-site replication on Replica instances¶

When the Operator creates a new cluster, a lot of things are happening, such as electing the Primary, generating certificates, and picking specific names. This should not happen if we want the Operator to run the Replica site, so first of all the cluster should be put into unmanaged state by setting the unmanaged key in the deploy/cr.yaml configuration file to true. Also you should set updateStrategy key to OnDelete and backup.enabled to false, because Smart Updates and backups are not allowed on unmanaged clusters.

Here is an example:

spec:
  unmanaged: true
  updateStrategy: OnDelete
  replsets:
  - name: rs0
    size: 3
    ...
  backup:
    enabled: false
  ...

Main and Replica sites should have same Secrets objects, so don’t forget to apply Secrets from your Main site. Names of the corresponding objects are set in the users, ssl, and sslInternal keys of the Custom Resource secrets subsection (my-cluster-name-secrets, my-cluster-name-ssl, and my-cluster-name-ssl-internal by default).

Copy your secrets from an existing cluster and apply each of them on your Replica site as follows:

$  kubectl apply -f my-cluster-secrets.yaml

The Replica site will be ready for replication when you apply changes as usual:

$ kubectl apply -f deploy/cr.yaml

Enabling multi-cluster Services¶

Kubernetes multi-cluster Services (MCS) is a cross-cluster discovery and invocation of Services. MCS-enabled Services become discoverable and accessible across clusters with a virtual IP address.

This feature allows splitting applications into multiple clusters combined in one fleet, which can be useful to separate logically standalone parts (i.e. stateful and stateless ones), or to address privacy and scalability requirements, etc.

Multi-cluster Services should be supported by the cloud provider. It is supported by Google Kubernetes Engine (GKE), and by Amazon Elastic Kubernetes Service (EKS).

Configuring your cluster for multi-cluster Services includes two parts:

• configure MCS with your cloud provider,
• make needed preparations with the Operator.

To set up MCS for a specific cloud provider you should follow official guides, for example ones from Google Kubernetes Engine (GKE), or from Amazon Elastic Kubernetes Service (EKS).

Setting up the Operator for MCS results in registering Services for export to other clusters using the ServiceExport object, and using ServiceImport one to import external services. Set the following options in the multiCluster subsection of the deploy/cr.yaml configuration file to make it happened:

• multiCluster.enabled should be set to true,
• multiCluster.DNSSuffix string should be equal to the cluster domain suffix for multi-cluster Services used by Kubernetes (svc.clusterset.local by default).

The following example in the deploy/cr.yaml configuration file is rather straightforward:

...
multiCluster:
  enabled: true
  DNSSuffix: svc.clusterset.local
...

Apply changes as usual with the kubectl apply -f deploy/cr.yaml command.

The initial ServiceExport creation and sync with the clusters of the fleet takes approximately five minutes. You can check the list of services for export and import with the following commands:

$ kubectl get serviceimport
NAME                     TYPE           IP                  AGE
my-cluster-name-cfg      Headless                           22m
my-cluster-name-cfg-0    ClusterSetIP   ["10.73.200.89"]    22m
my-cluster-name-cfg-1    ClusterSetIP   ["10.73.192.104"]   22m
my-cluster-name-cfg-2    ClusterSetIP   ["10.73.207.254"]   22m
my-cluster-name-mongos   ClusterSetIP   ["10.73.196.213"]   22m
my-cluster-name-rs0      Headless                           22m
my-cluster-name-rs0-0    ClusterSetIP   ["10.73.206.24"]    22m
my-cluster-name-rs0-1    ClusterSetIP   ["10.73.207.20"]    22m
my-cluster-name-rs0-2    ClusterSetIP   ["10.73.193.92"]    22m

$ kubectl get serviceexport
NAME                     AGE
my-cluster-name-cfg      22m
my-cluster-name-cfg-0    22m
my-cluster-name-cfg-1    22m
my-cluster-name-cfg-2    22m
my-cluster-name-mongos   22m
my-cluster-name-rs0      22m
my-cluster-name-rs0-0    22m
my-cluster-name-rs0-1    22m
my-cluster-name-rs0-2    22m

After ServiceExport object is created, exported Services can be resolved from any Pod in any fleet cluster as SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local.

MCS can charge cross-site replication with additional limitations specific to the cloud provider. For example, GKE demands all participating Pods to be in the same project. Also, default Namespace should be used with caution: your cloud provider may not allow exporting Services from it to other clusters.