Upgrading Percona XtraDB Cluster¶

This guide describes the procedure for upgrading Percona XtraDB Cluster without downtime (rolling upgrade) to the latest 5.7 version. A “rolling upgrade” means there is no need to take down the complete cluster during the upgrade.

Both major upgrades (from 5.6 to 5.7 version) and minor ones (from 5.7.x to 5.7.y) can be done in this way. Rolling upgrades to 5.7 from versions older than 5.6 are not supported. Therefore if you are running Percona XtraDB Cluster version 5.5, it is recommended to shut down all nodes, then remove and re-create the cluster from scratch. Alternatively, you can perform a rolling upgrade from PXC 5.5 to 5.6, and then follow the current procedure to upgrade from 5.6 to 5.7.

The following documents contain details about relevant changes in the 5.7 series of MySQL and Percona Server. Make sure you deal with any incompatible features and variables mentioned in these documents when upgrading to Percona XtraDB Cluster 5.7.

Major upgrade¶

To upgrade the cluster, follow these steps for each node:

1. Make sure that all nodes are synchronized.

2. Stop the mysql service:

   $ sudo service mysql stop
   

3. Remove existing Percona XtraDB Cluster and Percona XtraBackup packages, then install Percona XtraDB Cluster version 5.7 packages. For more information, see Installing Percona XtraDB Cluster.

   For example, if you have Percona software repositories configured, you might use the following commands:

   • On CentOS or RHEL:

   $ sudo yum remove percona-xtrabackup* Percona-XtraDB-Cluster*
   $ sudo yum install Percona-XtraDB-Cluster-57
   

   • On Debian or Ubuntu:

   $ sudo apt remove percona-xtrabackup* percona-xtradb-cluster*
   $ sudo apt install percona-xtradb-cluster-57
   

4. In case of Debian or Ubuntu, the mysql service starts automatically after install. Stop the service:

   $ sudo service mysql stop
   

5. Back up grastate.dat, so that you can restore it if it is corrupted or zeroed out due to network issue.

6. Start the node outside the cluster (in standalone mode) by setting the wsrep_provider variable to none.

   For example:

   sudo mysqld --skip-grant-tables --user=mysql --wsrep-provider='none'
   

   As of Percona XtraDB Cluster 5.7.6, the --skip-grant-tables option is not required.

   To prevent any users from accessing this node while performing work on it, you may add –skip-networking to the startup options and use a local socket to connect, or alternatively you may want to divert any incoming traffic from your application to other operational nodes.

7. Open another session and run mysql_upgrade.

8. When the upgrade is done, stop the mysqld process. You can either run sudo kill on the mysqld process ID, or sudo mysqladmin shutdown with the MySQL root user credentials.

   On CentOS, the my.cnf configuration file is renamed to my.cnf.rpmsave. Make sure to rename it back before joining the upgraded node back to the cluster.

9. Now you can join the upgraded node back to the cluster.

   In most cases, starting the mysql service should run the node with your previous configuration:

   $ sudo service mysql start
   

   For more information, see Adding Nodes to Cluster.

   As of version 5.7, Percona XtraDB Cluster runs with PXC Strict Mode enabled by default. This will deny any unsupported operations and may halt the server upon encountering a failed validation.

   If you are not sure, it is recommended to first start the node with the pxc_strict_mode variable set to PERMISSIVE in the in the MySQL configuration file, my.cnf.

   After you check the log for any experimental or unsupported features and fix any encountered incompatibilities, you can set the variable back to ENFORCING at run time:

   mysql> SET pxc_strict_mode=ENFORCING;
   

   Also switch back to ENFORCING may be done by restarting the node with updated my.cnf.

10. Repeat this procedure for the next node in the cluster until you upgrade all nodes.

It is important that on rejoining, the node should synchronize using IST. For this, it is best not to leave the cluster node being upgraded offline for an extended period. More on this below.

When performing any upgrade (major or minor), SST could be initiated by the joiner node after the upgrade if the server was offline for some time. After SST completes, the data directory structure needs to be upgraded (using mysql_upgrade) once more time to ensure compatibility with the newer version of binaries.

Minor upgrade¶

To upgrade the cluster, follow these steps for each node:

1. Make sure that all nodes are synchronized.

2. Stop the mysql service:

   $ sudo service mysql stop
   

3. Upgrade Percona XtraDB Cluster and Percona XtraBackup packages. For more information, see Installing Percona XtraDB Cluster.

   For example, if you have Percona software repositories configured, you might use the following commands:

   • On CentOS or RHEL:

   $ sudo yum update Percona-XtraDB-Cluster-57
   

   • On Debian or Ubuntu:

   $ sudo apt install --only-upgrade percona-xtradb-cluster-57
   

4. In case of Debian or Ubuntu, the mysql service starts automatically after install.

   Stop the service:

   $ sudo service mysql stop
   

5. Back up grastate.dat, so that you can restore it if it is corrupted or zeroed out due to network issue.

6. Start the node outside the cluster (in standalone mode) by setting the wsrep_provider variable to none.

   For example:

   sudo mysqld --skip-grant-tables --user=mysql --wsrep-provider='none'
   

   As of Percona XtraDB Cluster 5.7.6, the --skip-grant-tables option is not required.

   To prevent any users from accessing this node while performing work on it, you may add –skip-networking to the startup options and use a local socket to connect, or alternatively you may want to divert any incoming traffic from your application to other operational nodes.

7. Open another session and run mysql_upgrade.

8. When the upgrade is done, stop the mysqld process. You can either run sudo kill on the mysqld process ID, or sudo mysqladmin shutdown with the MySQL root user credentials.

   On CentOS, the my.cnf configuration file is renamed to my.cnf.rpmsave. Make sure to rename it back before joining the upgraded node back to the cluster.

9. Now you can join the upgraded node back to the cluster.

   In most cases, starting the mysql service should run the node with your previous configuration:

   $ sudo service mysql start
   

   For more information, see Adding Nodes to Cluster.

   As of version 5.7, Percona XtraDB Cluster runs with PXC Strict Mode enabled by default. This will deny any unsupported operations and may halt the server upon encountering a failed validation.

   If you are not sure, it is recommended to first start the node with the pxc_strict_mode variable set to PERMISSIVE in the in the MySQL configuration file, my.cnf.

   After you check the log for any experimental or unsupported features and fix any encountered incompatibilities, you can set the variable back to ENFORCING at run time:

   mysql> SET pxc_strict_mode=ENFORCING;
   

   Also switch back to ENFORCING may be done by restarting the node with updated my.cnf.

10. Repeat this procedure for the next node in the cluster until you upgrade all nodes.

Dealing with IST/SST synchronization while upgrading¶

It is important that on rejoining, the node should synchronize using IST. For this, it is best not to leave the cluster node being upgraded offline for an extended period. More on this below.

When performing any upgrade (major or minor), SST could be initiated by the joiner node after the upgrade if the server was offline for some time. After SST completes, the data directory structure needs to be upgraded (using mysql_upgrade) once more time to ensure compatibility with the newer version of binaries.

The following additional steps should be made to upgrade the data directory structure after SST (after the normal major or minor upgrade steps):

1. Shutdown the node that rejoined the cluster using SST:

   $ sudo service mysql stop
   

2. Restart the node in standalone mode by setting the wsrep_provider variable to none, for example:

   sudo mysqld --skip-grant-tables --user=mysql --wsrep-provider='none'
   

3. Run mysql-upgrade

4. Restart the node in cluster mode (e.g., by executing sudo service mysql start and make sure the cluster joins back using IST.

---