Red Hat and the Clone Wars III: The dawn of CentOS

Until the announcement that CentOS Linux 8 would be EOL at the end of 2021, CentOS users enjoyed a relatively drama-free period of stability that might suggest RHEL has always had a viable, dependable clone with predictable releases. That is, as you’re probably already guessing, very far from the truth.

In this post we’ll look at the CentOS history from inception until the CentOS Project came under Red Hat’s direct sponsorship. That’s early 2004 through January 7, 2014 when the announcement went out to the world on the CentOS-announce list. We’ll also mention a few other attempts at cloning RHEL didn’t quite thrive over the long haul.

Note, I’m mostly leaving Oracle Linux out of this even though it was introduced in 2006. As you might have guessed, it’ll be the topic of another post on its own.

A quick(ish) note first

I’ve been using, writing about, and/or working for some of the vendors in question for more than 20 years. This is one of the reasons I’ve chosen to cover this on my blog, because a lot of the participants in the current conversations around Red Hat don’t have the same history. This is a nuanced conversation that benefits from understanding how we got here.

As much as possible I’m trying not to simply work from memory. If I can, I’m trying to cite primary sources or news sites that covered these things at the time. Thank goodness for LWN, which is still doing great work and has extensive archives. Other sites, though, have gone the way of the dodo and vendors have disappeared a lot of content over the years. Some projects have also gone extinct in the interim.

If I can find something on Internet Archive, I’ll link that. If I can’t find anything to link to, I’ll go with my memory but will happily correct things if it turns out my memory is faulty. And, finally, my thinking has evolved over the years quite a bit as I’ve moved from tech reporting, to working with Linux, to working with and for vendors.

Send in the Clones

Confession: I was not a Red Hat Linux fan in the early 2000s. I was an avid Slackware user for a long time, and then dallied with SuSE (as it was capped then), Mandrake Linux, and Debian and various Debian derivatives that tried to smooth it out for desktop use, like Stormix.

But others had to have their Red Hat. The introduction of Red Hat Advanced Server (later RHEL) created a bunch of clones like White Box Linux, cAos (“community assembled operating systems”), Tao Linux, Lineox, Scientific Linux (originally High Energy Physics Linux, or HEPL), and of course CentOS. By 2011 only CentOS and Scientific Linux survived.

White Box Linux gave this reason for its creation:

Its initial creation was sponsored by the Beauregard Parish Public Library in DeRidder, USA out of self interest. We have several servers and over 50 workstations running Red Hat Linux and were left high and dry by Red Hat’s recent shift in business plan. Our choices were a difficult migration to another distribution or paying Red Hat an annual fee greater than the amortized value of our hardware. So we chose a third path, made possible by the power of open source..

The concept of “Red Hat makes money on support” and that support meant “pick up the phone and yell at somebody” came into being in this era and (IMO) calcified in people’s minds. Red Hat tries mightily to educate everyone (from employees to customers to analysts and users) on a more nuanced “value of subscription” message, but it gets boiled down to “subscription means support and support only means when I have to talk to Red Hat.”

That idea was somewhat justified in the early aughts. Expectations were modest at the time. The clones were accepted as best-effort, community run projects that would try to trail RHEL as fast as they could, but users knew that there might be long gaps between a RHEL update and seeing a package land in their clone of choice.

An additional note on the White Box Linux homepage explained that its maintainers only had access to 32-bit Intel at the time, and had no intention to port WBL to “IBM big iron.” If you wanted that you should “just BUY RHEL.”

It’s worth noting that in this era that Linux was still seen as (and actually was) a less polished, less capable UNIX-type system. The era of proprietary UNIX was not quite over, but Linux was encroaching on UNIX workloads. Red Hat couldn’t credibly claim technical parity, much less superiority, to Solaris or others.

Also, security updates? Eh. Those were for Windows users who actually had security problems, right? The Internet wasn’t quite the hellscape of constant predatory attacks on anything with an IP address that it is today. And Linux? Why target Linux when there were so, so, so many vulnerable Windows boxen to prey on?

Community ENTerprise Operating System (CentOS)

It was in this context that CentOS evolved as the early leader for clones, long before it came under Red Hat’s wing. In 2004 and 2005 I worked for a hosting company and (if memory serves) CentOS was a leading choice for shared hosting with cPanel or Plesk because it was almost RHEL but none of the cost. (We would provision RHEL for customers if they were willing to pay the subscription, of course.)

In fact here’s what I thought about it in 2005. “After spending some time with CentOS, this writer sees little difference between Red Hat’s official offerings and the CentOS offerings that are community-supported. Official support directly from Red Hat may be necessary for some organizations, but if that’s not a requirement, the CentOS distribution may be a better choice.”

In fact, CentOS offered users the magic of updates via Yum rather than Red Hat Network. It was lower friction and a lot cheaper. (Lineox actually used APT for updates, which was the fashion at the time…) What’s not to love?

CentOS learns about shared control

Well. As Linux continued to evolve and mature, expectations grew. Business needs evolved. And CentOS, as a volunteer project, occasionally stumbled.

In 2009, the main project admin for CentOS went radio silent. CentOS had been spun up in the wild west days of the Internet, when it seemed totally fine to let one person hold the domain, communications channels, and funds. Oops.

Folks depending on CentOS suddenly had a Come to Jesus moment. What happens if your production systems depend on a project where the main admin is AWOL and you have no plan for that?

Happily that was short-lived, and the public only needed to fret for a few days. At least, it was short-lived in terms of the public eye. The CentOS devs had been trying to establish contact for weeks before the open letter went out for all to see.

A few days after the other CentOS folks posted an “open letter” about the issue, the admin returned and the dev team said “a majority of issues were resolved immediately and a working agreement was reached with deadlines for remaining unresolved issues. There should be no impact to any CentOS users going forward. The CentOS project is now in control of the CentOS.org and CentOS.info domains and owns all trademarks, materials, and artwork in the CentOS distributions.”

Update uncertainty

Things more or less returned to normal for CentOS as a project and for CentOS users. Updates continued, albeit not always speedily.

For example, RHEL 5.4 (which brought KVM, support for up to 192 CPUs for Xen, and more) was released on or about 2 September 2009. The corresponding CentOS 5.4 release was announced 21 October 2009.

In 2010, Red Hat released RHEL 5.5 on 30 March. CentOS 5.5 was announced on 14 May.

Red Hat released RHEL 6 on 10 November 2010. CentOS, however, didn’t release CentOS 6 until 10 July 2011 about eight months after RHEL 6.

CentOS attempted to fix the problem with slow updates by introducing “continuing release” (CR) repos. It’s worth mentioning, again, that the landscape was very very different in the first decade of CentOS’s (and RHEL’s) life. This was before public cloud became a real player for most users. (AWS launched in 2006, but was only a few services and hardly the Goliath it is today.)

If you complained about the speed of updates on the CentOS list you were … directed to purchase a RHEL subscription “if timely patches are of importance to you.”

Scientific Linux was doing a better job of pushing out updates at the time, but it was not quite a clone of RHEL. It was the first to get a RHEL 6 clone out, and included additional packages and tools for creating custom spins of Scientific Linux. Troy Dawson, a core contributor, explained to me the project’s philosophy around updates at the time:

For non-security errata, updates are pushed out once per week into the “fastbugs” repository — with a few exceptions. According to Dawson, one exception is if the Scientific team has problems building an RPM for one reason or another. He also says that after a major release from Red Hat, updates are slowed while the team works on that. Finally, Dawson says that they hold onto kernel changes for a few days “”just to make sure we don’t see any yelling”” on the Red Hat mailing list about the new kernels. “”It generally takes a couple weeks to get their latest security errata built and tested to our satisfaction. An example would be when RHEL 6.1 came out. We didn’t release any security updates for SL 6.0 until this week. So it took a week and a half to get the security errata out.””

As I mentioned, I’m mostly ignoring Oracle Linux in this post, but it’s worth noting that Oracle Linux wasn’t super-speedy in this time period either. Major releases from Oracle Linux would trail RHEL releases by months, and minor releases would trail by weeks. Consuming package-at-a-time updates was less prominent in this time.

Red Hat flexes Trademarks, the community complains

Prepare yourself for a big shock: This isn’t the first time Red Hat has taken action trying to stop clones and copies cutting in on its business. For example, in 2002, Red Hat started cracking down on the discount CDs that cut into its boxed set sales.

As I mentioned previously, those sales cut both ways. They helped spread Red Hat Linux to users who didn’t have high speed Internet connections, who were willing to spend a few bucks to try Linux but not $29.95, and/or didn’t have friends who could hook them up with a copy.

At the time, Red Hat said, “In the Open Source space, our name is our bread and butter. Red Hat is trying to practice its own quality control.” (That was then-spokesperson Melissa London from Red Hat.)

And here’s Red Hat’s Trademark policy at the time:

Redistributing Red Hat® Linux That Has Been Downloaded From An FTP Site or Duplicated From A Red Hat-Produced CD: You may not state that your product ‘contains Red Hat Linux X.X.’ This would amount to impermissible use of Red Hat’s trademarks.

Any of this sounding familiar? Go ahead and use the source, Luke, but trying to ride Red Hat’s coattails isn’t welcome.

Red Hat relied on trademarks again versus CentOS later down the line after RHEL became the primary business. In 2005, Red Hat informed CentOS that its use of Red Hat trademarks could confuse consumers about the relationship of CentOS and Red Hat/RHEL.

The blog post about this on the CentOS site is no longer available, but it still lives on via the Internet Archive’s Wayback Machine. (Bless ’em.) In part, the email sent to the CentOS folks said:

While Red Hat permits others to redistribute the software that constitutes Red Hat Linux, Red Hat does not authorize any person to use the RED HAT marks in association with such redistribution in any fashion, except by express agreement. In this regard, our client is concerned that your use of the RED HAT marks on your web site in this manner is likely to create confusion, mistake and/or deception among consumers with respect to the source, origin, sponsorship or approval of the products sold under your company name.

Trademark enforcement wouldn’t stop CentOS or others from cloning Red Hat Enterprise Linux, but without handy tools to debrand the sources, things were less easy. Also, being deterred from uttering the name “RHEL” might have helped slow the spread of clones, since people were less apt (no pun intended) to stumble on the project if they weren’t already aware.

Of course, every time Red Hat has tried to prevent wholesale cloning of Red Hat Linux and RHEL releases, people have complained. A lot.

Lessons from the first decade of RHEL clones

What they haven’t done, to date, is topple Red Hat from its position as the de facto Linux “standard” for business. Only one company has come close, though many have tried, and that’s Canonical. While Canonical has made huge inroads in unpaid usage, its attempts to become the primary paid Linux have not panned out.

As you probably have guessed, Canonical and Ubuntu – and the rivalry between the two camps – will be a topic for another post very soon.

But it’s worth noting and emphasizing that the larger community seems to have a very rosy view of RHEL clones before CentOS was “acquired” by Red Hat.

CentOS never had speedy, reliable updates until brought into the Red Hat fold. Neither did any of the other clones. What the community has today with the releases to CentOS Stream may not be as easy to work with as before, but they seem to be far better than what the community was working with before RHEL 7.

With RHEL 7, and with CentOS brought into the fold, the first CentOS release based on sources from git.centos.org was released with a “aim” to deliver updates “within 24 to 48 hours” of upstream releases. Had Red Hat left CentOS alone, it’s hard to say what the landscape would look like today. But I’ll try, in a future post.