RHEL Panel Discussion at FOSSY 2023

by Bradley M. Kuhn on July 19, 2023

This past weekend, July 13-16th, 2023, Software Freedom Conservancy (SFC) hosted and ran a new conference, FOSSY (Free and Open Source Software Yearly) in Portland, Oregon, USA. I was glad to host the keynote panel discussion on the recent change made by Red Hat (now a subsidiary of IBM) regarding the public source code releases for Red Hat Enterprise Linux (RHEL).

The panelists included (in alphabetical order) Jeremy Alison, software engineer at CIQ (focused on Rocky Linux) and Samba co-founder, myself, Bradley M. Kuhn, policy fellow at SFC, benny Vasquez, the Chair of the AlmaLinux OS Foundation, and James (Jim) Wright, who is Oracle’s Chief Architect for Open Source Policy, Strategy, Compliance, and Alliances.

Red Hat themselves did not reply to our repeated requests to join us on this panel, but we were able to gather the key organizations impacted by Red Hat's recent decision to cease public distribution of RHEL sources. SUSE was also invited but let us know they were unable to send someone on short notice to Portland for the panel.

We're very glad to make the video available to everyone who has been following this evolving story. FOSSY is a new event, and we've hopefully shown how running a community-led FOSS event here in Portland each summer creates an environment where these kinds of important discussions can be held to explore issues impacting FOSS users around the world.

I thank our panelists again for booking last-minute travel to be with us for this exciting panel and thank all the FOSSY attendees for their excellent questions during the panel.

I hope to see all of you at next years' FOSSY!

[permalink]

Tags: conservancy, GPL, conferences, events

A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model

by Bradley M. Kuhn on June 23, 2023

This article was originally published primarily as a response to IBM's Red Hat's change to no longer publish complete, corresponding source (CCS) for RHEL and the prior discontinuation of CentOS Linux (which are related events, as described below). We hope that this will serve as a comprehensive document that discusses the history of Red Hat's RHEL business model, the related source code provisioning, and the GPL compliance issues with RHEL.

---

For approximately twenty years, Red Hat (now a fully owned subsidiary of IBM) has experimented with building a business model for operating system deployment and distribution that looks, feels, and acts like a proprietary one, but nonetheless complies with the GPL and other standard copyleft terms. Software rights activists, including SFC, have spent decades talking to Red Hat and its attorneys about how the Red Hat Enterprise Linux (RHEL) business model courts disaster and is actively unfriendly to community-oriented Free and Open Source Software (FOSS). These pleadings, discussions, and encouragements have, as far as we can tell, been heard and seriously listened to by key members of Red Hat's legal and OSPO departments, and even by key C-level executives, but they have ultimately been rejected and ignored — sometimes even with a “fine, then sue us for GPL violations” attitude. Activists have found this discussion frustrating, but kept the nature and tenure of these discussions as an “open secret” until now because we all had hoped that Red Hat's behavior would improve. Recent events show that the behavior has simply gotten worse, and is likely to get even worse.

What Exactly Is the RHEL Business Model?

The most concise and pithy way to describe RHEL's business model is: “if you exercise your rights under the GPL, your money is no good here”. Specifically, IBM's Red Hat offers copies of RHEL to its customers, and each copy comes with a support and automatic-update subscription contract. As we understand it, this contract clearly states that the terms do not intend to contradict any rights to copy, modify, redistribute and/or reinstall the software as many times and as many places as the customer likes (see §1.4). Additionally, though, the contract indicates that if the customer engages in these activities, that Red Hat reserves the right to cancel that contract and make no further contracts with the customer for support and update services. In essence, Red Hat requires their customers to choose between (a) their software freedom and rights, and (b) remaining a Red Hat customer. In some versions of these contracts that we have reviewed, Red Hat even reserves the right to “Review” a customer (effectively a BSA-style audit) to examine how many copies of RHEL are actually installed (see §10) — presumably for the purpose of Red Hat getting the information they need to decide whether to “fire” the customer.

Red Hat's lawyers clearly take the position that this business model complies with the GPL (though we aren't so sure), on grounds that that nothing in the GPL agreements requires an entity keep a business relationship with any other entity. They have further argued that such business relationships can be terminated based on any behaviors — including exercising rights guaranteed by the GPL agreements. Whether that analysis is correct is a matter of intense debate, and likely only a court case that disputed this particular issue would yield a definitive answer on whether that disagreeable behavior is permitted (or not) under the GPL agreements. Debates continue, even today, in copyleft expert circles, whether this model itself violates GPL. There is, however, no doubt that this provision is not in the spirit of the GPL agreements. The RHEL business model is unfriendly, captious, capricious, and cringe-worthy.

Furthermore, this RHEL business model remains, to our knowledge, rather unique in the software industry. IBM's Red Hat definitely deserves credit for so carefully constructing their business model such that it has spent most of the last two decades in murky territory of “probably not violating the GPL”.

Does The RHEL Business Model Violate the GPL Agreements?

Perhaps the biggest problem with a murky business model that skirts the line of GPL compliance is that violations can and do happen — since even a minor deviation from the business model clearly violates the GPL agreements. Pre-IBM Red Hat deserves a certain amount of credit, as SFC is aware of only two documented incidents of GPL violations that have occurred since 2006 regarding the RHEL business model. We've decided to share some general details of these violations for the purpose of explaining where this business model can so easily cross the line.

In the first violation, a large Fortune 500 company (which we'll call Company A), who both used RHEL internally and also built public-facing Linux-based products, decided to create a consumer-facing product (which we'll call Product P) based primarily on CentOS Linux, but P included a few packages built from RHEL sources. Company A did not seek nor ask for support or update services for this separate Product P. Red Hat later became aware that Product P contained some part of RHEL, and Red Hat demanded royalty payments for Product P. Red Hat threatened to revoke the support and update services on Company A's internal RHEL servers if such royalties were not paid.

Since Company A was powerful and had good lawyers and savvy business development staff, they did not acquiesce. Company A ultimately continued (to our knowledge) on as a RHEL customer for their internal servers and continued selling Product P without royalty payments. Nevertheless, a demand for royalties for distribution is clearly a violation as that demand creates a “further restriction” on the permissions granted by GPL. As stated in GPLv3:

You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License.

Red Hat tried to impose a further restriction in this situation, and therefore violated the GPL. The violation was resolved since no royalty was paid and Company A faced no consequences. SFC learned of the incident later, and informed Red Hat that the past royalty demand was a violation. Red Hat did not dispute nor agree that it was a violation, and did informally agree such demands would not be made in future.

In another violation incident, we learned that Red Hat, in a specific non-USA country, was requiring that any customer who lowered the number of RHEL machines under service contract with Red Hat sign an additional agreement. This additional agreement promised that the customer had deleted every copy of RHEL in their entire organization other than the copies of RHEL that were currently contracted for service with Red Hat. Again, this is a “further restriction”. The GPL agreements give everyone the unfettered right to make and keep as many copies of the software as they like, and a distributor of GPL'd software may not require a user to attest that they've deleted these legitimate, licensed copies of third-party-licensed software under the GPL. SFC informed Red Hat's legal department of this violation, and we were assured that this additional agreement would no longer be presented to any Red Hat customers in the future.

In both these situations, we at SFC were worried they were merely a “tip of the proverbial iceberg”. For years, we have heard from Red Hat customers who are truly confused. It's common in the industry to talk about RHEL “seat licenses”, and many software acquisition specialists in the industry are not aware of the nuances of the RHEL business model and do not understand their rights. We remain very concerned that RHEL salespeople purposely confuse customers to sell more “seat licenses”. It's often led us to ask: “If a GPL violation happens in the woods, and everyone involved doesn't hear it, how does anyone know that software rights have indeed been trampled upon in those woods?”. As we do for as many GPL violation reports as we can, we zealously pursue RHEL-related GPL violations that are reported to us, and if you're aware of one, please do email us at <compliance@sfconservancy.org> immediately. We fear that be it through incompetence or malice, many RHEL salespeople and business development professionals may regularly violate GPL and no one knows about it. That said, the business model as described by IBM's Red Hat may well comply with the GPL — it's just so murky that any tweak to the model in any direction seems to definitely violate, in our experience.

Furthermore, Red Hat exploits the classic “caveat emptor” approach — popular in many a shady business deal throughout history. While, technically speaking, a careful reader of the GPL and the RHEL agreements understands the bargain they're making, we suspect most small businesses just don't have the FOSS licensing acumen and knowledge to truly understand that deal.

Why Was an Independent CentOS So Important?

Until Red Hat's “aquisition” of CentOS in early 2014, CentOS provided an excellent counterbalance to the problems with the RHEL business model. Specifically, CentOS was a community-driven project, with many volunteers, supported by some involvement from small businesses, to re-create RHEL releases using the CCS releases made for RHEL. Our pre-2014 view was that CentOS was the “canary in the murky coalmine” of the RHEL business. If CentOS seemed vibrant, usable, and a viable alternative to RHEL for those who didn't want to purchase Red Hat's updates and services, the community could rest easy. Even if there were GPL violations by Red Hat on RHEL, CentOS' vibrancy assured that such violations were having only a minor negative impact on the FOSS community around RHEL's codebase.

Red Hat, however, apparently knew that this vibrant community was cutting into their profits. Starting in 2013, Red Hat engaged in a series of actions that increased their grip. First, they “acquired” CentOS. This was initially couched as a cooperation agreement, but Red Hat systematically made job offers that key CentOS volunteers couldn't refuse, acquired the small businesses who might ultimately build CentOS into a product, and otherwise integrated CentOS into Red Hat's own operations.

After IBM acquired Red Hat, the situation got worse. Having gotten rights to the CentOS brand as part of the “aquisition”, Red Hat slowly began to change what CentOS was. CentOS Linux quickly ceased to be a check-and-balance on RHEL, and just became a testing ground for RHEL. Then, in 2020, when most of us were distracted by the worst of the COVID-19 pandemic, Red Hat unilaterally terminated all CentOS Linux development. Later (during the Delta variant portion of the pandemic in late 2021) Red Hat ended CentOS Linux entirely. IBM's Red Hat then used the name “CentOS Stream” to refer to experimental source packages related to RHEL. These were (and are) not actually the RHEL source releases — rather, they appear to be primarily a testing ground for what might appear in RHEL later.

Finally, Red Hat announced two days ago that RHEL CCS will no longer be publicly available in any way. Now, to be clear, the GPL agreements did not obligate Red Hat to make its CCS publicly available to everyone. This is a common misconception about GPL's requirements. While the details of CCS provisioning vary in the different versions of the GPL agreements, the general principle is that CCS need to be provided either (a) along with the binary distributions to those who receive, or (b) to those who request pursuant to a written offer for source. In a normal situation, with no mitigating factors, the fact that a company moved from distributing CCS publicly to everyone to only giving it to customers who received the binaries already would not raise concerns.

In this situation, however, this completes what appears to be a decade-long plan by Red Hat to maximize the level of difficulty of those in the community who wish to “trust but verify” that RHEL complies with the GPL agreements. Namely, Red Hat has badly thwarted efforts by entities such as Rocky Linux and Alma Linux. These entities are de-facto the intellectual successors to CentOS Linux project that Red Hat carefully dismantled over the last decade. These organizations sought to build Linux-based distributions that mirrored RHEL releases, and it is now unclear if they can do that effectively, since Red Hat will undoubtedly capriciously refuse to sell them exactly-one RHEL service and update “seat license” at a reasonable price. It appears that, as of this week, one must have at least that to get timely access to RHEL CCS.

What Should Those Who Care About Software Rights Do About RHEL?

Due to this ongoing bad behavior by IBM's Red Hat, the situation has become increasingly complex and difficult to face. No third party can effectively monitor RHEL compliance with the GPL agreements, since customers live in fear of losing their much-needed service contracts. Red Hat's legal department has systematically refused SFC's requests in recent years to set up some form of monitoring by SFC. (For example, we asked to review the training materials and documents that RHEL salespeople are given to convince customers to buy RHEL, and Red Hat has not been willing to share these materials with us.) Nevertheless, since SFC serves as the global watchdog for GPL compliance, we welcome reports of RHEL-related violations.

We finally express our sadness that this long road has led the FOSS community to such a disappointing place. I personally remember standing with Erik Troan in a Red Hat booth at a USENIX conference in the late 1990s, and meeting Bob Young around the same time. Both expressed how much they wanted to build a company that respected, collaborated with, engaged with, and most of all treated as equals the wide spectrum of individuals, hobbyists, and small businesses that make the plurality of the FOSS community. We hope that the modern Red Hat can find their way back to this mission under IBM's control.

[permalink]

Tags: conservancy, GPL, law

John Deere's ongoing GPL violations: What's next

by Denver Gingerich on March 16, 2023

I grew up on a farm. My parents worked hard to grow crops and manage the farm business. My parents also found additional jobs to make ends meet. As farmers have done for millennia, my family used tools to farm. Some of those tools were tractors. Farmers now, as they have for thousands of years, rely on their ability and right to fix their tools. Perhaps that's bending a hand rake back into shape. Maybe they need to weld a broken three-point hitch back together. Agriculture was humanity's first truly revolutionary technological advancement. Since its inception, each generation of farmers exercised their right to repair their tools. This has allowed agriculture to grow and improve immeasurably. We take for granted the benefits that this has given us, and the abundance of food it provides.

The right to repair farm tools is now in serious jeopardy, not because farmers haven't fought to maintain this right, and not even because farmers haven't chosen to use tools that guarantee their right to repair their tools. In fact, most farmers are still buying tools that have a right to repair built into them, not by their intrinsic nature, but by the software that the toolmakers have chosen to include as part of the tools they sell to the farmers.

Sadly, farm equipment manufacturers, who benefit immensely from the readily-available software that they can provide as part of the farming tools (tractors, combines, etc.) they sell to farmers, are not complying with the right to repair licenses of the software they have chosen to use in these farming tools. As a result, farmers are cut off from their livelihood if the farm equipment manufacturer does not wish to repair their farming tools when they inevitably fail, even when the farmer could easily perform the repairs on their own, or with the help of someone else they know.

In particular, John Deere, the largest manufacturer of farm equipment in North America and one of the largest worldwide, has been failing to meet the requirements of the software right to repair licenses they use for some time. While we have worked for years with John Deere to try and resolve their compliance problems, they have still not complied with these licenses for the software that they use, which would give farmers the right, and technical details, to repair their own farm tools if Deere complied. This is a serious issue that goes far beyond one person wanting to fix their printer software, or install an alternative firmware on a luxury device. It has far-reaching implications for all farmers' livelihoods, for food security throughout the world, and for how we as a society choose to reward those who make our lives better, or stand in the way of empowering everyone to improve the world.

As we have been doing privately for multiple years, we now publicly call on John Deere to immediately resolve all of its outstanding GPL violations, across all lines of its farm equipment, by providing complete source code, including "the scripts used to control compilation and installation of the executable" that the GPL and other copyleft licenses require, to the farmers and others who are entitled to it, by the licenses that Deere chose to use. What Deere has provided to SFC as of today falls far short of the requirements of the GPL, with respect to both this quoted text, and many other parts of the license. And that speaks only of the products for which Deere has started to engage with us about - for many of almost a dozen requests we've made (each for a different product) Deere has yet to provide anything to us at all. In addition to failing to respond at all to others who have requested source code, Deere's inability to provide complete corresponding source to us for all requested products more than 2 years after our first request is beyond unacceptable, which is why we are making this public statement today - to more strongly encourage Deere to do the right thing and comply with the licenses they use, and to let others know about these serious problems so they have a more complete picture of Deere's attempts to stifle farmers' right to repair their farm equipment.

We stand with all the other organizations that are taking John Deere to task for its various violations of other agreements and laws, including antitrust, and we hope these organizations succeed in bringing fairness to farmers. We each help in our own ways, which is the true strength of the right to repair movement.

If you are a farmer concerned by Deere's practices, or personally affected by them, please reach out to us at compliance@sfconservancy.org. By working together, we can give farmers back their rights, allowing them to repair their own farm tools again, by themselves or using their friend or shop of choice, improving their lives and the lives of everyone on earth who depends on them every day.

[permalink]

Tags: conservancy

Call for Community-Led Tracks at FOSSY

by Daniel Takamori on January 31, 2023

Today Software Freedom Conservancy is officially opening our call for track proposals for our first annual FOSSY conference! We will be holding the conference in Portland, Oregon July 13-16, 2023 at the Oregon Convention Center. We are looking for community driven tracks that can balance important and in depth technical and non-technical issues, while uplifting contributors of all experiences. Tracks will be modeled after the DevRooms at FOSDEM and the miniconfs at linux.conf.au. They may be between 1 and 4 days, and the organizers of the tracks will be in charge of outreach, calls for submissions, communicating with potential speakers in the track, determining the schedule and hosting the track in person at FOSSY.

We're looking for organizers who can give us a really good idea of what we can expect from their track. The description should give a detailed explanation of the topic, ideally along with some of the issues you expect to cover. Example talks you expect, what kind of audience are you aiming for, and how this topic fits into the larger FOSS ecosystem are good things to mention.

You'll note that we ask for two people to be listed as organizers for the track. It's easy to underestimate the work involved so having more than two organizers could also really help to take care of all of the work. We'll be there to help and support you, but this will be your show!

We'd like you to tell us why the organizers are the right ones for the job. Do they have experience running conferences, unique perspectives due to involvement with the topic? Conference organizing is a demanding job that requires a balance of logistics, people centered concerns and technical skills. We trust you to assemble a group of people that can cater to those needs and want to put on a great event.

Given that this is the first FOSSY, we will be creating this space together! How is the topic you are proposing beneficial for the FOSS community and how does it fit into this new space? The hope is to have a balance of technical and non-technical topics, and we want to hear from you about what's important on those issues. Given that we want to shape the conference into something that uplifts contributors of all levels and experience, how will you approach a varied audience?

How long will your track be? Are you planning a quick and deep dive into a single topic or do you dream of having a 4 day long track dealing with tough issues that you want attendees to sit with and reflect on over the weekend? We don't need you to lock yourself into this choice, but we do need a rough figure how much participation and space you'll need if you are hoping to do something specific.

Anything that gives us a sense of the organization and spirit of your tracks will be helpful.

Please use our submission page or email us at conference@sfconservancy.org if you have any questions.

The deadline for application is Sunday March 19th, so be sure to reach out soon!

We're very excited to hear from you about how we can shape this conference into something for us all. Thanks so much for your interest and we hope to see you in July!

[permalink]

Tags: conservancy, conferences