A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model
by on June 23, 2023
This article was originally published primarily as a response to IBM's Red Hat's change to no longer publish complete, corresponding source (CCS) for RHEL and the prior discontinuation of CentOS Linux (which are related events, as described below). We hope that this will serve as a comprehensive document that discusses the history of Red Hat's RHEL business model, the related source code provisioning, and the GPL compliance issues with RHEL.
For approximately twenty years, Red Hat (now a fully owned subsidiary of IBM) has experimented with building a business model for operating system deployment and distribution that looks, feels, and acts like a proprietary one, but nonetheless complies with the GPL and other standard copyleft terms. Software rights activists, including SFC, have spent decades talking to Red Hat and its attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts disaster and is actively unfriendly to community-oriented Free and Open Source Software (FOSS). These pleadings, discussions, and encouragements have, as far as we can tell, been heard and seriously listened to by key members of Red Hat's legal and OSPO departments, and even by key C-level executives, but they have ultimately been rejected and ignored — sometimes even with a “fine, then sue us for GPL violations” attitude. Activists have found this discussion frustrating, but kept the nature and tenure of these discussions as an “open secret” until now because we all had hoped that Red Hat's behavior would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even worse.
What Exactly Is the RHEL Business Model?
The most concise and pithy way to describe RHEL's business model is:
“if you exercise your rights under the GPL, your money is no good
here”. Specifically, IBM's Red Hat offers copies of RHEL to its
customers, and each copy comes with a support and automatic-update
subscription contract. As we understand it, this contract
clearly states
that the terms do not intend to contradict any rights to copy, modify,
redistribute and/or reinstall the software as many times and as many places
as the customer likes (see §1.4). Additionally, though, the contract indicates that
if the customer engages in these activities, that Red Hat reserves the
right to cancel that contract and make no further contracts with the
customer for support and update services. In essence, Red Hat requires their customers
to choose between (a) their software freedom and rights, and (b) remaining a Red Hat
customer. In some versions of these contracts that we have reviewed, Red
Hat even reserves the right to “Review” a customer (effectively a
Red Hat's lawyers clearly take the position that this business model complies with the GPL (though we aren't so sure), on grounds that that nothing in the GPL agreements requires an entity keep a business relationship with any other entity. They have further argued that such business relationships can be terminated based on any behaviors — including exercising rights guaranteed by the GPL agreements. Whether that analysis is correct is a matter of intense debate, and likely only a court case that disputed this particular issue would yield a definitive answer on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today, in copyleft expert circles, whether this model itself violates GPL. There is, however, no doubt that this provision is not in the spirit of the GPL agreements. The RHEL business model is unfriendly, captious, capricious, and cringe-worthy.
Furthermore, this RHEL business model remains, to our knowledge, rather unique in the software industry. IBM's Red Hat definitely deserves credit for so carefully constructing their business model such that it has spent most of the last two decades in murky territory of “probably not violating the GPL”.
Does The RHEL Business Model Violate the GPL Agreements?
Perhaps the biggest problem with a murky business model that skirts the line of GPL compliance is that violations can and do happen — since even a minor deviation from the business model clearly violates the GPL agreements. Pre-IBM Red Hat deserves a certain amount of credit, as SFC is aware of only two documented incidents of GPL violations that have occurred since 2006 regarding the RHEL business model. We've decided to share some general details of these violations for the purpose of explaining where this business model can so easily cross the line.
In the first violation, a large Fortune 500 company (which we'll call Company A), who both used RHEL internally and also built public-facing Linux-based products, decided to create a consumer-facing product (which we'll call Product P) based primarily on CentOS Linux, but P included a few packages built from RHEL sources. Company A did not seek nor ask for support or update services for this separate Product P. Red Hat later became aware that Product P contained some part of RHEL, and Red Hat demanded royalty payments for Product P. Red Hat threatened to revoke the support and update services on Company A's internal RHEL servers if such royalties were not paid.
Since Company A was powerful and had good lawyers and savvy business development staff, they did not acquiesce. Company A ultimately continued (to our knowledge) on as a RHEL customer for their internal servers and continued selling Product P without royalty payments. Nevertheless, a demand for royalties for distribution is clearly a violation as that demand creates a “further restriction” on the permissions granted by GPL. As stated in GPLv3:
You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License.
Red Hat tried to impose a further restriction in this situation, and therefore violated the GPL. The violation was resolved since no royalty was paid and Company A faced no consequences. SFC learned of the incident later, and informed Red Hat that the past royalty demand was a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree such demands would not be made in future.
In another violation incident, we learned that Red Hat, in a specific non-USA country, was requiring that any customer who lowered the number of RHEL machines under service contract with Red Hat sign an additional agreement. This additional agreement promised that the customer had deleted every copy of RHEL in their entire organization other than the copies of RHEL that were currently contracted for service with Red Hat. Again, this is a “further restriction”. The GPL agreements give everyone the unfettered right to make and keep as many copies of the software as they like, and a distributor of GPL'd software may not require a user to attest that they've deleted these legitimate, licensed copies of third-party-licensed software under the GPL. SFC informed Red Hat's legal department of this violation, and we were assured that this additional agreement would no longer be presented to any Red Hat customers in the future.
In both these situations, we at SFC were worried they were merely a “tip of the proverbial iceberg”. For years, we have heard from Red Hat customers who are truly confused. It's common in the industry to talk about RHEL “seat licenses”, and many software acquisition specialists in the industry are not aware of the nuances of the RHEL business model and do not understand their rights. We remain very concerned that RHEL salespeople purposely confuse customers to sell more “seat licenses”. It's often led us to ask: “If a GPL violation happens in the woods, and everyone involved doesn't hear it, how does anyone know that software rights have indeed been trampled upon in those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that are reported to us, and if you're aware of one, please do email us at <compliance@sfconservancy.org> immediately. We fear that be it through incompetence or malice, many RHEL salespeople and business development professionals may regularly violate GPL and no one knows about it. That said, the business model as described by IBM's Red Hat may well comply with the GPL — it's just so murky that any tweak to the model in any direction seems to definitely violate, in our experience.
Furthermore, Red Hat exploits the classic “caveat emptor” approach — popular in many a shady business deal throughout history. While, technically speaking, a careful reader of the GPL and the RHEL agreements understands the bargain they're making, we suspect most small businesses just don't have the FOSS licensing acumen and knowledge to truly understand that deal.
Why Was an Independent CentOS So Important?
Until Red Hat's “aquisition” of CentOS in early 2014, CentOS provided an excellent counterbalance to the problems with the RHEL business model. Specifically, CentOS was a community-driven project, with many volunteers, supported by some involvement from small businesses, to re-create RHEL releases using the CCS releases made for RHEL. Our pre-2014 view was that CentOS was the “canary in the murky coalmine” of the RHEL business. If CentOS seemed vibrant, usable, and a viable alternative to RHEL for those who didn't want to purchase Red Hat's updates and services, the community could rest easy. Even if there were GPL violations by Red Hat on RHEL, CentOS' vibrancy assured that such violations were having only a minor negative impact on the FOSS community around RHEL's codebase.
Red Hat, however, apparently knew that this vibrant community was cutting into their profits. Starting in 2013, Red Hat engaged in a series of actions that increased their grip. First, they “acquired” CentOS. This was initially couched as a cooperation agreement, but Red Hat systematically made job offers that key CentOS volunteers couldn't refuse, acquired the small businesses who might ultimately build CentOS into a product, and otherwise integrated CentOS into Red Hat's own operations.
After IBM acquired Red Hat, the situation got worse. Having gotten rights to the CentOS brand as part of the “aquisition”, Red Hat slowly began to change what CentOS was. CentOS Linux quickly ceased to be a check-and-balance on RHEL, and just became a testing ground for RHEL. Then, in 2020, when most of us were distracted by the worst of the COVID-19 pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely. IBM's Red Hat then used the name “CentOS Stream” to refer to experimental source packages related to RHEL. These were (and are) not actually the RHEL source releases — rather, they appear to be primarily a testing ground for what might appear in RHEL later.
Finally, Red Hat announced two days ago that RHEL CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its CCS publicly available to everyone. This is a common misconception about GPL's requirements. While the details of CCS provisioning vary in the different versions of the GPL agreements, the general principle is that CCS need to be provided either (a) along with the binary distributions to those who receive, or (b) to those who request pursuant to a written offer for source. In a normal situation, with no mitigating factors, the fact that a company moved from distributing CCS publicly to everyone to only giving it to customers who received the binaries already would not raise concerns.
In this situation, however, this completes what appears to be a decade-long plan by Red Hat to maximize the level of difficulty of those in the community who wish to “trust but verify” that RHEL complies with the GPL agreements. Namely, Red Hat has badly thwarted efforts by entities such as Rocky Linux and Alma Linux. These entities are de-facto the intellectual successors to CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations sought to build Linux-based distributions that mirrored RHEL releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.
What Should Those Who Care About Software Rights Do About RHEL?
Due to this ongoing bad behavior by IBM's Red Hat, the situation has become increasingly complex and difficult to face. No third party can effectively monitor RHEL compliance with the GPL agreements, since customers live in fear of losing their much-needed service contracts. Red Hat's legal department has systematically refused SFC's requests in recent years to set up some form of monitoring by SFC. (For example, we asked to review the training materials and documents that RHEL salespeople are given to convince customers to buy RHEL, and Red Hat has not been willing to share these materials with us.) Nevertheless, since SFC serves as the global watchdog for GPL compliance, we welcome reports of RHEL-related violations.
We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I personally remember standing with Erik Troan in a Red Hat booth at a USENIX conference in the late 1990s, and meeting Bob Young around the same time. Both expressed how much they wanted to build a company that respected, collaborated with, engaged with, and most of all treated as equals the wide spectrum of individuals, hobbyists, and small businesses that make the plurality of the FOSS community. We hope that the modern Red Hat can find their way back to this mission under IBM's control.
Please email any comments on this entry to info@sfconservancy.org.
