Upgrade Process of Percona Server for MongoDB (Replica Set and Shard Cluster)

Upgrade Process of Percona Server for MongoDB (Replica Set and Shard Cluster)

Percona Server for MongoDB (PSMDB) upgrade, major and minor versions

In this blog, we will see the best practices for upgrading major and minor versions of the clusters (Replica Set and Shard Cluster) with no downtime, in a rolling manner.

Percona Server for MongoDB (PSMDB) has versions like A.B.C where A.B refers to the major version and C refers to the minor version or the revision number. Below is the way to identify the versioning and understand which one is suitable for production and development environments:

• If B is an even number then it refers to a stable release and can be deployed to production.
• If B is an odd number then it refers to a development release, which is not stable and is under testing.
• C will be an incrementing number like 0,1,2,3 and so on. Generally, it is for bug fixes and backward-compatible changes.

Before upgrading a Replica set or Sharded cluster, the below prerequisites should be taken care of.

Prerequisites:

• Always take a full backup of the database before doing an upgrade.
• Always check for any special considerations or compatibility issues specific to your PSMDB release from release notes (mostly in the Major version, but sometimes there is a rare minor-version bug fix that has backward incompatibility).
• Please inform the application team and ask them to check the current driver compatibility with the version to which an upgrade is required.
• Make sure all the nodes are in a healthy state.
• Check the FCV (feature compatibility version), it should be the same as the current database version.Below is the command to check the current FCV:
  Shell

  db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

  Below is the command to set the current FCV:
  Shell

  db.adminCommand( { setFeatureCompatibilityVersion: "version"} )

• Plan the upgrade in a maintenance window. Also, before you plan to upgrade a production environment, it is advisable to upgrade a lower environment that reproduces your production environment, to ensure that your production configuration is compatible with all changes.

Upgrade major version in a Replica Set

Before starting the upgrade (in this blog, I will upgrade from 4.4 to 5.0), take the backup/copy of the PSMDB configuration file (mongod.conf) on every node, as it will be replaced with the default one.

We are using the default location of the configuration file in this blog for backup/copy:

Start upgrading with secondaries one by one.

• Stop the service on one secondary (if not logged in as root user, make sure user is having sudo access):
  Shell

  $ sudo service mongod stop

• After the service is stopped successfully, remove the binaries:
  Shell

  $ sudo yum remove percona-server-mongodb*

• Enable the repository:
  Shell

  $ sudo percona-release enable psmdb-50

• To install a specific version of Percona Server for MongoDB, run the below command to see a list of available versions:
  Shell

  $ sudo yum list percona-server-mongodb --showduplicates

  Below is a sample output of the above command:
  Shell

  percona-server-mongodb.x86_64 5.0.2-1.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.3-2.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.4-3.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.5-4.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.6-5.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.7-6.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.8-7.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.9-8.el7 psmdb-50-release-x86_64

• Install specific version packages. For example, to install Percona Server for MongoDB 5.0.8-7, run the following command:
  Shell

  $ sudo yum install percona-server-mongodb-5.0.8-7.el7

  The below command will install the latest version available:
  Shell

  $ sudo yum install percona-server-mongodb

• Before restoring make sure to remove the content of the newly created configuration file:
  Shell

  $ > /etc/mongod.conf $ cat /etc/mongod.conf_bkp > /etc/mongod.conf

• Start the mongod service:
  Shell

  $ sudo service mongod start

• Check the installed version:
  Shell

  $ mongo --version

• Wait for the secondary to get in sync and come into the “SECONDARY” state.
• Now perform the same steps one through nine, including taking a backup/copy of the config file on other nodes.
• Once both the secondaries are upgraded, login into the Primary node and stepDown it. Once a new primary is elected, repeat the same steps one through nine.
• Wait for the sanity completion by the QA or App team.
• Then change the FCV to the current version once everything is running fine.

Upgrade major version in Sharded Cluster

• Stop the balancer.
  Run the below command to stop the balancer:
  Shell

  mongos> sh.stopBalancer()

  Run the below command to check the balancer state:
  Shell

  mongos> sh.getBalancerState() false

• Upgrade the Config server replica set first. Refer to steps 1-11 from the upgrade major version in the replica set.
• Upgrade the Shard replica sets next. Refer to upgrade major version in the replica set. Also, you can upgrade the shards in parallel in the same rolling manner, steps will be the same for all shards.
• The last mongos will be upgraded in the same rolling manner (If you have one mongos then please inform your application team for that particular downtime).
• Start the balancer.
  Shell

  mongos> sh.startBalancer()

• Now follow steps 12 and 13 from the upgrade major version in the replica set once the cluster is upgraded.

Upgrade minor version in a Replica Set

Before starting the upgrade, take the backup/copy of the PSMDB configuration file (mongod.conf) file on every node, as it will be replaced with the default one.

Using the default location of the configuration file here for backup/copy:

Start upgrading with secondaries one by one.

• • Stop the service on one secondary (if not logged in as a root user, make sure the user has sudo access):
    Shell

    $ sudo service mongod stop

  • After the service is stopped successfully, remove the binaries:
    Shell

    $ sudo yum remove percona-server-mongodb*

  • Check the available minor versions:
    Shell

    $ sudo yum list percona-server-mongodb --showduplicates

    Sample output:
    Shell

    percona-server-mongodb.x86_64 5.0.2-1.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.3-2.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.4-3.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.5-4.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.6-5.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.7-6.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.8-7.el7 psmdb-50-release-x86_64 percona-server-mongodb.x86_64 5.0.9-8.el7 psmdb-50-release-x86_64

  • Install the specific version on which you want to upgrade (I choose to upgrade from 5.0.3 to 5.0.7):
    Shell

    $ sudo yum install percona-server-mongodb-5.0.7-6.el7

  • Before restoring make sure to remove the content of the newly created configuration file:

• Start the mongod service:
  Shell

  $ sudo service mongod start

• Check the new version:
  Shell

  $ mongo --version

• Wait for the secondary to get in sync and come into the “SECONDARY” state.
• Now perform the same steps 1-10 including taking a backup/copy config file on other nodes.
• Once all the secondaries are upgraded, login into the Primary node and stepDown it. Once a new primary is elected, repeat the same steps 1-10.
• In the minor version, FCV will remain the same as we are just changing the C rather than A or B in A.B.C.

Upgrade minor version in Sharded Cluster

• Stop the balancer.
  Run the below command to stop the balancer:
  Shell

  mongos> sh.stopBalancer()

  Run the below command to check the balancer state:
  Shell

  mongos> sh.getBalancerState() false

• Upgrade the Config server replica set first. Refer to steps from the upgrade minor version in the replica set.
• Upgrade the Shard replica sets next. Refer to steps from the upgrade minor version in the replica set. Also, you can upgrade the shards parallel in the same rolling manner, steps will be the same for all shards.
• The last mongos will be upgraded in the same rolling manner (If you have one mongos then please inform your application team for that particular downtime).
• Start the balancer.
  Shell

  mongos> sh.startBalancer()

Summary

Percona Server for MongoDB is an open-source replacement for MongoDB Community Edition that combines all of the features and benefits of MongoDB Community Edition with enterprise-class features developed by Percona: LDAP Authentication and Authorization, Audit Logging, Kerberos Authentication, and hot backups

To learn more about the enterprise-grade features available in the vendor lock-in-free Percona Server for MongoDB, we recommend going through our blog MongoDB: Why Pay for Enterprise When Open Source Has You Covered?