9

A Story of a Failed Percona Monitoring and Management Update

 3 years ago
source link: https://www.percona.com/blog/a-story-of-a-failed-percona-monitoring-and-management-update/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client
A Story of a Failed Percona Monitoring and Management Update

Just the other day, one of my Percona Monitoring and Management (PMM) instances failed to update and ended up in a funny state:

percona monitoring and management

No matter how many times I clicked the “check for updates” button in the lower right, PMM still showed “no updates available”. Basically, PMM forgot what version it is and because of that it thinks there are no updates available

If you run into this (or possibly other PMM  update problems), you may find the following helpful to troubleshoot or resolve the problem. 

I have deployed PMM through docker, so the first step is to get inside the docker container to be able to investigate and troubleshoot the installation.

Shell
docker exec -it pmm-server bash

Percona Monitoring and Management (as of version 2.26) learns its own version from the version of “pmm-update” package.  In my case, this did not work due to a corrupted rpm database:

Shell
[root@812fb43b3a96 opt]# yum info pmm-update
error: rpmdb: BDB0113 Thread/process 13841/140435865503552 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:
Error: rpmdb open failed

It is not clear to me why the database has become corrupted in my case, but unfortunately, it did and needed to be repaired:

Shell
mv /var/lib/rpm/__db* /tmp/
rpm --rebuilddb
rm -rf /var/cache/yum

These commands backup the current database to /tmp/ (just in case)  to rebuild the current database as well as remove already downloaded packages (it has nothing to do with database corruption but reduces reliance on the local state).

Shell
# supervisorctl start pmm-update-perform
pmm-update-perform: started

This command starts the update process.  You probably could also trigger an update through the Web GUI at this point but I want you to be aware of this option, too.

As this simply starts the background update process you do not get much information about what is going on. To find that out, you need to check the log:

Shell
tail /srv/logs/pmm-update-perform.log

Note while the update is going, the PMM instance may not be available at all. You could see this message:

image3-1024x248.png

Do not freak out – wait for the update to finish before trying to use PMM again. 

The update log does not really have a very clear message which states what update was completed successfully.  Here is, however, an example of how the end of a log from a successful update may look:

Shell
PLAY RECAP *********************************************************************
localhost                  : ok=79   changed=31   unreachable=0    failed=0    skipped=16   rescued=0    ignored=0
time="2022-04-01T15:36:39Z" level=info msg="Waiting for Grafana dashboards update to finish..."

With that, the update has been completed and my PMM instance has successfully updated to the recent version.

image2.png

I hope this blog post will help you to troubleshoot your PMM update problems too if you happen to run into one! 🙂


report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK