Percona Operator Volume Expansion Without Downtime

Percona Operator Volume Expansion Without Downtime

There are several ways to manage storage in Percona Kubernetes Operators: Persistent Volume (PV), hostPath, ephemeral storage, etc. Using PVs, which are provisioned by the Operator through Storage Classes and Persistent Volume Claims, is the most popular choice for our users. And one of the most popular questions is how to scale our operator storages which are based on PVs. To make PVs resize easier, the volume expansion feature was introduced as an alpha feature in Kubernetes 1.8 and eventually became a stable feature in 1.24. 

 In this blog post, I will show you how to easily increase the storage size in Percona Operators using this feature without any database downtime and explain what to do if your storage class doesn’t support volume expansion. 

Scale-up persistent volume claim (PVC) by volume expansion

• You can only resize PVC if the storage class of the PVC has set the AllowVolumeExpansion=True option. 

• Ensure that the delete PVC finalizer (delete-pxc-pvc/delete-psmdb-pvc) is not set in the custom resource, otherwise, all cluster data can be lost. 
• Please refer to volume expansion documentation for intree volume types which support volume expansion. 
• If the underlying Storage Driver can only support offline expansion, users of the PVC must take down their Pod before expansion can succeed. Please refer to the documentation of your storage provider to understand which modes of volume expansion are supported. 

Please note that expanding EBS volumes is a time-consuming operation. Also, there is a per-volume quota of one modification every six hours. 

Percona Operator for MongoDB/Percona Operator for MySQL

Under the hood of Percona Operator for MongoDB and Percona Operator for MySQL based on Percona XtraDB Cluster, StatefuSets are used, so the volume expansion task for the Operator comes down to resizing the corresponding StatefulSet persistent volume. 

Resizing a PV claimed by changing custom resource or StatefulSets (Does not work)

Only a number of specific StatefulSet fields can be modified after creation. Altering storage size in operator custom resource or StatefulSet leads to the error below:

Expansion storage by modifying persistent volume claim (PVC)

1. Change PVC size: 

2. After the process is finished, you can see the below message in the PVC description. 

If you see the following event in the PVC description (and it does not change) you need to restart pods to finish resizing. (See further notes.)

3. When PVC size was changed, you can ensure that other objects also changed.

4. Update storage size in corresponding operator custom resource (Percona Operator for MongoDB or Percona Operator for MySQL based on Percona XtraDB Cluster).

So now we have scaled storage and old storage values in the corresponding StatefulSets. As we saw above we can apply custom resources with new storage values, but the storage size field can’t be changed for StatefulSet objects. That is why we need to recreate StatefulSet with new values. To avoid downtime we delete StatefulSet without deleting the pods (–cascade=’orphan’ flag).

5. Apply cr.yml.

6. Delete StatefulSet pods one by one (optional).

According to the Kubernetes official documentation:

If your storage provider supports online expansion then no Pod restart should be necessary for volume expansion to finish.

Percona Operator for PostgreSQL (pg operator)

In contrast to Percona Operator for MongoDB/Percona Operator for MySQL based on Percona XtraDB Cluster which uses partially modifiable StatefulSet object, our Percona Operator for PostgreSQL uses Deployment instead of StatefulSet for cluster objects. Deployment is a mutable object and can be changed not only at the cluster start, but also on the running cluster. That is why the change of storage size in custom resource will be continuously applied to the running cluster. Changes in custom resources not only resize volume but also initiate pods restart. 

Moreover, in order to maintain a safe cluster configuration,  the operator keeps primary and replicas volume size exactly equal.  And hence, if the size of the primary is changed, the size of the replicas is changed too and both primary and replica pods are restarted. Otherwise, if you modify replica size first, only replica PVCs are expanded and pods are recreated. However, if you try to increase the primary size to the same value as replicas, only primary PVC is scaled (PVCs of the replicas already have the necessary value ) but both replicas and primary pods are restarted.

So you can see the situation that both primary and replica PVCs changed the size, only the storage size for the primary was increased in the custom resource.

You need to be careful and keep your custom resource file up-to-date. 

Resizing persistent volume claim (PVC) by transferring data to new PVC

Volume Expansion is the great Kubernetes feature that provides an easy and effective way to resize PVC. However, some storage classes do not support PVC volume expansion. For these classes, you must create a new PVC and move content to it. You can do it in the following steps:

1. Configure PVC size in custom resource for the corresponding operator (Percona Operator for MongoDB or Percona Operator for MySQL based on Percona XtraDB Cluster) and apply it. 

2. Delete StatefulSet. Like in the previous part, we need to recreate StatefulSet in order to apply storage changes. 

As a result, the Pods are up and the Operator recreates the StatefulSet with a new volume size.

3. Scale up the cluster (optional).

Changing the storage size would require us to terminate the Pods, which can induce performance degradation. To preserve performance during the operation we can scale up the cluster using these instructions  Percona Operator for MongoDB or Percona Operator for MySQL based on Percona XtraDB Cluster.  As long as we have changed the StatefulSet already, new operator pods will be provisioned with increased volumes. 

4. Reprovision the Pods One by One to Change the Storage.

This is the step where underlying storage is going to be changed for the database Pods.

Delete the PVC of the Pod that you are going to reprovision.

The PVC will not be deleted right away as there is a Pod using it. To proceed, delete the Pod:

The Pod will be deleted along with the PVCs. The StatefulSet controller will notice that the pod is gone and will recreate it along with the new expanded  PVC.

Check PVC size:

The CAPACITY column indicates that this PVC is increased.

Once the Pod is up, the data is synced from other nodes. The data transfer speed depends on the amount of data in the cluster and cluster utilization as a consequence synchronization might take a while. Please wait until the node is fully up and running, sync is finished, and only then proceed to the next Pod.

5. Scale Down the Cluster and clean up unnecessary PVCs. 

Conclusion

We are constantly introducing new features in the Percona Kubernetes Operators. One of them is automated volume expansion, which will be implemented in a future release.  We also plan to add the possibility to change storage size directly in a custom resource that makes it possible to add to the Private DBaaS feature in Percona Monitoring and Management (PMM) detecting the volume consumption and automatically scaling the cluster. If you have any suggestions on the topic and are willing to collaborate with us, please submit the issue to Community Forum or JIRA.