The Challenges of Monitoring in the Cloud

I’ve been working in cloud technologies for a little over ten years now and I just love the topic. I came into it originally because it was new and shiny but, once I got into it, it was the recognition that Platform as a Service offerings change things. Let us learn about The Challenges of Monitoring in the Cloud.

My excitement, and I see this with others too, is that when you move to AWS, Google, or Azure SQL Database, or a managed instance on Azure, you recognize the fact that suddenly you’re not spending time patching or doing backups. Instead, you can focus on the behavior, performance and data in your database, the stuff that’s really important, and you think why didn’t you move to the cloud sooner?

I’m not alone here, and that’s borne out by the 2021 State of Database DevOps report from Redgate, which showed that 58% of organizations now host some or most of their databases in the cloud, up from 46% in the same report from the year before. Interestingly, just 20% of organizations host their databases only on-premises, a drop from 26% a year earlier.

There are good reasons for that, and probably the biggest is scalability. People are jumping to the cloud because they can scale up instantly if they need to without a lot of changes to the existing code and platform. They don’t have to talk to storage vendors and have physical boxes shipped and installed, they can literally just resize the VM. Remember, they can go the other way too, and reduce the size of the VM and the ongoing costs, which makes it great for businesses with fluctuating or seasonal demand. So building platforms in the cloud and having the ability to scale elastically very quickly is a huge value.

That’s the good news. The not so good news is that database monitoring in the cloud is the same as on-premises … and a whole lot different as well.

Just like on-premises servers, you’re going to want to be monitoring your queries, your wait stats and your performance metrics. Whether it’s a physical server, a VM or PaaS, it almost doesn’t matter. It’s the same stuff you need to keep an eye on. Which is where life becomes a little complicated.

The biggest challenge is that you’re not going to be monitoring one thing. You’re going to be monitoring SQL servers running on hard iron locally, SQL servers running on VMs, Azure SQL database and probably multiple instances of VMs on Azure or AWS. However, the monitoring that comes native with cloud platforms doesn’t take you all the way to where you need to be – and it varies across platforms too.

For example, AWS exposes perfmon counters that are different from the perfmon counters exposed through Azure, so you need to gather both and some customized DMVs inside AWS that expose monitoring data that should also be collected. Instead of hopping from platform to platform and monitoring each one individually, you want to bring all that stuff back to one place to give you a single vision across your whole stack.

This is probably where you should now seriously consider a third-party monitoring solution because the rise of hybrid estates is a mixed bag for DBAs. On the one hand, it lets us demonstrate our skills and knowledge in choosing and making the most of the different options and platforms that are now out there. On the other, you’re now expected to extract data, make decisions and spend time configuring it and figuring out what other data you need from multiple sources, all at the same time.

A good monitoring tool can take all of those disconnected scenarios and present them in one simple web interface so that you can address your server estate in one place, at one time, with one view. You don’t have to do any crazy configuration, switching or dashboard reconfigurations and stuff like that. It’s a single pane of glass for the whole estate, wherever and whatever the server is.

A good example, because I’m very familiar with, it is SQL Monitor from Redgate. The following screenshot is from the live online demo of the tool, which is used to monitor a few test instances of the SQLServerCentral and Simple Talk community websites.

As you can see, SQL Monitor pulls in the status and key metrics of every server, instance and database, whether on-premises, on Virtual Machines, Azure, AWS, or Google Cloud, and be notified of problems through customizable alerts tailored to your SQL Server environment.

As your estate grows, you can bulk add servers, or automatically discover them through its API. You also get a whole host of estate-wide management tools including an overview of your installed versions and patches, SQL Server licensing, disk usage statistics, backups, and SQL Agent jobs status. All the stuff you need to keep on doing the important bits of your job, regardless of where the data is.

The online demo uses real data, by the way, and it’s not a trial version or limited in any way, so you can just go ahead and dive in and troubleshoot perf issues in real-time if you haven’t used the product before.

You, like me, are probably excited about the opportunities the cloud is providing to do more with your data, in more places, faster and smarter. But if you’re upgrading your server estate, you should also be thinking about how you upgrade your monitoring capability as well.

To find out more about why managing your entire SQL Server estate, with instant problem diagnosis, intelligent and customizable alerting has never been more vital, register for Redgate’s webinar: SQL Monitor: Monitor your performance and availability.

Reference: Pinal Dave (http://blog.SQLAuthority.com)