My blog can’t keep up: 500 errors all over

My blog is relatively minor enterprise. It is strictly non-profit (no ad). I have been posting one or two blog posts a week for about fifteen years. I have been using the same provider in all this time (csoft.net). They charge me about $50 a month. I also subscribe to Cloudflare services, which costs me some extra money.

I use wordpress. If I had to do things over, I would probably choose something else, but that is what I have today.

I estimate that I get somewhere between 30,000 and 50,000 unique visitors a month. Despite my efforts, my blog keeps on failing under the load. It becomes unavailable for hours.

I have given up on writing new blog post using the online editor. It is brittle. The old wordpress editor worked relatively well, but since upgrading wordpress, they have now pushed something called the Gutenberg editor. It tries to be clever, but half the time it just fails with a 500 error (meaning that the server just failed). So I use a client called MarsEdit. It seems to work well enough.

Several times a week, someone emails me to report that they tried to leave a comment and they got a 500 error.

I seem to get about one spam comment per minute or so. I have just now decided to close comments on posts older than 30 days, in the hope that it will relax the load on my server.

My error logs are filled with “End of script output before headers” (a few every minutes).

I used to rely on WP Super Cache, hoping that it made things better, but I have since disabled most plugins. I am hoping Cloudflare can do the work.

I had the following line in the .htaccess file at the root of my blog:

Header set Cache-Control "max-age=600, public"

The intention was that it would entice Cloudflare to cache everything. I do not think it worked.

Because my error logs showed that wp-cron.php was failing every few minutes, I added the following in my wp-config.php file:

define('DISABLE_WP_CRON', true);

I setup a separate cron job to call wp-cron.php every hour.

I now use Cloudflare with the following settings: “Caching Level: Ignore query string”, “Respect Existing Headers” and “Cache: Everything”. I pay for Argo, whatever that is, in the hope that it might improve things. With these settings, I would expect Cloudflare to cache pretty much everything. It apparently does not. My blog gets hammered. Cloudfare reports 45,000 uncached requests for the day, and most of them are in the last couple of hours.

I have asked my host provider (csoft.net) to give me more memory, but they seem unwilling to do it transparently. Though csoft.net is neither cheap nor particularly modern, they have been professional. I have purchased a service with SiteGround, as I am considering moving there because it seems more popular than csoft. I am not super excited about tuning my PHP/Wordpress setup, however. I fear that it is wrong-headed optimization.

What am I missing? How can I be in so much trouble in 2019 with such a relatively modest task?

Credit: I am grateful to Travis Downs and Nathan Kurz for an email exchange regarding my problems.