Copyleft compliance misconception #2: Anyone can easily fix the incomplete source releases that companies provide

by Denver Gingerich on December 11, 2018

This post is part of a series on copyleft compliance misconceptions. For our previous post, see Copyleft compliance misconception #1.

As Conservancy's FLOSS License Compliance Engineer, I receive many reports of copyleft noncompliance every week, and the people reporting them are often rightly concerned that the compliance issues are not fixed quickly. These issues often arise on devices such as media players, Android phones, broadband routers, and even vehicles. In each case there is some copylefted software that a user wishes to inspect or modify (which is their right, according to both the license and the morality of software freedom) but they are unable to, due to noncompliance. Usually, those who understand software observe noncompliance as a real-world, practical problem of incomplete (or entirely missing) source code and build information.

In the course of resolving these copyleft noncompliance incidents, Conservancy normally receives a candidate source release (or "CCS (Complete Corresponding Source) candidate") from the company whose device or product is out of compliance. You might think that it is a straight-forward task for us to then to bring that CCS candidate into compliance with a few fixes. Unfortunately, that is normally not the case. There are a variety of reasons for that, which no one can resolve themselves without assistance from the software distributor. I've outlined the most common problems here in hopes that companies (and others) who wish to do right and comply correctly with copyleft licenses can make their own fixes earlier -- ideally before shipping a product at all, but at the very least, quickly at the time they are first contacted about a compliance problem. Learning about these issues and being prepared before a compliance problem is found can make a big difference! For example, clauses like GPLv3§8, the Linux Kernel Enforcement Statement, and Red Hat's Cure Commitment have a very short window of 30 days to fix violations. It's rare these days that companies get their CCS in order in 30 days after a violation is reported, but following the advice below could definitely change that!

We don't stand on ceremony with regard to copyleft compliance; our goal in any copyleft enforcement action is to bring whatever of our own (meager) resources to bear to aid a company in producing compliant CCS. In an ideal world, we'd just provide patches or a list of fixes when receive the first "round" (as we call it) of CCS candidates.

Trying to do this, in fact, is a big part of my job. When I evaluate a candidate source release for correctness, I normally create a list of suggested fixes (often including patches the company can programmatically apply) alongside the list of issues that I send to the company if I find the source release is incomplete. I then send these lists to the company in the hope that they will take our suggestions and fix their source release.

However, the result of sending the list of issues and associated fixes seldom results in the company responding with a source release that is complete and correct (especially in the case of a first "round" check). Over the years, I've noticed the causes for this generally tend to fall into one of these categories:

A. The company believes they don't need to provide source that is derivative of and/or has been combined with work licensed under a strong copyleft license like GPL

Thankfully, this situation is relatively rare, but it is probably one of the more public causes for incomplete source that people are aware of (see, for example, Christoph Hellwig's case against VMware that Conservancy is supporting and (partially) funding). In this case, Conservancy has notified the company that their source release is missing source code for certain binary files that are derivative of a copylefted work. Normally this comes up with driver modules for the Linux kernel -- the company does not wish to release (or does not have, because their upstream failed to provide) the source code for a driver they are shipping, but the driver is clearly combined with Linux. We generally try to find out who provided the driver to the company to begin with, and then follow up with them to try and obtain the source code.

In this situation we are naturally blocked from achieving complete corresponding source, as part of the source is clearly missing and the company is refusing to provide it to us. Short of reverse engineering their driver, this is no way for Conservancy or anyone else, other than the authors of the proprietary driver, to provide a fix for this situation.

The best approach for companies in this situation is to assure at the time they source a product that no proprietary Linux modules are involved. Often, downstream companies don't invest the resources to hire their own engineer (or a third-party engineering service organization) to rebuild their Linux-based firmwares from scratch. We strongly recommend anyone building Linux-based products do this, as problems like this can easily be found before you make a deal with an upstream vendor. While in this negotiating phase, the vendor will want your business so you'll have a lot more power to get the upstream vendor to provide CCS and even to indemnify you down the road if there turns out to be a problem, as described in D, below.

Solving this problem admittedly needs more lead time than any of the others. Fortunately, there are many (and more common) situations that come up, and while they don't generate impasse, a basic approach of "planning ahead" can go a long way to avoid repeated attempts at compliance.

B. The company applies one or two of the fixes then sends back a new known-incomplete candidate

One of the most common causes of a company not correcting its source release in the first "round" of CCS checks is simply that they did not implement all of the fixes we provided to them. This wastes both their time and Conservancy's, since anyone reviewing that "round" needs to figure out which fixes they implemented and which they didn't, and then send back a new report to them highlighting the fixes that were already provided in the last report. Then the company needs to go back and (hopefully) implement more of the fixes to send yet another candidate back to Conservancy.

Conservancy has tried to determine why this cause of incomplete source is so common, but unfortunately few past violators are able to explain clearly why this occurs. We suspect that the number of fixes that a company needs to make feels overwhelming to them so they simply don't bother with most of them. Or perhaps they think we won't notice that they only implemented a few of the fixes; we have seen situations where individuals who complain about CCS to a violator often run out of time and patience seeking corrected CCS, so perhaps these companies have misunderstood that incomplete CCS to be satisfactory merely because the individual never followed up. In any case, we hope that by highlighting this problem in blog posts like this, it might become less of a problem in the future.

C. The company fixes all the issues we highlighted, but there are new issues in their candidate

This is perhaps the next most common cause of a company not providing a complete source release on their second try, but is less insidious. An example of where this might happen is when a source release clearly requires a cross-compiler in order to build, but the company does not specify which one (and the code makes it difficult for Conservancy to guess, so Conservancy can't attempt to use a potential cross-compiler itself). When the company does respond with the name of the cross-compiler they used (or the source and binaries of the cross-compiler itself), Conservancy often then finds a number of issues with the rest of the source that cause the build to fail even when the correct cross-compiler is used.

There are a range of other reasons for this (such as a component that now builds revealing that a dependent component never built properly to begin with), but all result in the same outcome: additional rounds of CCS checking that cause Conservancy and the company to expend significantly more time on trying to resolve the problem. Naturally, this problem is exacerbated when combined with B above, which we see much more often than we'd like.

D. The company is as frustrated as we are, but has been intimidated by their upstream.

We have some degree of sympathy for many violators we encounter, because we know their upstream vendors are the bad actors and they've been caught in the middle. By the time an embedded Linux electronics product makes it to the mass market -- when a user might purchase it and report a violation -- the name-brand vendor has made various complex business deals with one or more upstream vendors. Those upstream vendors often knowingly violated the GPL on Linux, but do not inform their customers about the presence of Linux, or the incompleteness of the firmware sources. Most downstream vendors never rebuild the main firmware; they develop a user-space application on top of it quickly and push it to market. When the violation is discovered, the main vendor is in a very weak negotiating position, and fears problems for future products. Many violators have lamented to us this difficult problem, but are confused about what to do about -- they simply become stuck in the middle sending us CCS candidates from an upstream vendor who has already decided to violate the GPL knowingly and willfully.

Conservancy has tried many creative solutions to this problem, including asking the violator to join us in informal or formal legal action against the bad actor. Sadly, few companies care enough about the GPL to take this stand, and we're often left with incomplete CCS and an indefinitely continuing GPL violation. This is what occurred with Tesla, which is why we applauded them for going public with this as they try to address their ongoing GPL violations of incomplete source.

Conclusion

It is extremely rare for me to find that a company has fixed all of the problems in their source release on the first attempt, even when I provide a complete list of all the problems and patches to fix each of them. That's why I wanted to highlight some of the causes for this that we normally see, both so that people using copylefted software in devices they've bought know what we're up against, and so that companies are aware of the common pitfalls and can correct their source releases accordingly.

[permalink]

Tags: conservancy