Linaro Connect, Volkswagen and Developer Ethics

by Karen Sandler on September 30, 2015

Last week I had the privilege of delivering Friday's keynote address at Linaro Connect. I was so excited and pleased that I had been asked to speak about compliance there. As Linaro is a consortium for Linux kernel related initiatives on ARM, I was excited and curious as to what the conference was like and thrilled to be given the chance to talk about why copyleft and GPL compliance are so fundamental to the success of collaborative engineering initiatives like Linaro. The fact that the conference is so developer focused was a huge bonus.

One of the topics I touched on, given its newsworthiness was the situation with Volkswagen. Many people have talked about the implications of so-called dieselgate and its implications for free and open source software. In my talk I focused on another aspect of this - engineer and developer culture.

When I was in engineering school at The Cooper Union we had a mandatory course during our first year where we read the book To Engineer Is Human (which incidentally, if you buy you can sign up to support Conservancy on Amazon Smile first). The book discusses prominent engineering failures (including the dramatic Tacoma Narrows Bridge collapse “Galloping Gertie”), why they failed and how such failure is ultimately a part of successful societal engineering. In the class we talked about the culture of engineering ethics and how engineers ultimately have a special responsibility in society on behalf of the people who are impacted by the work they do.

In the recent case of Volkswagen, the failure of the company to behave ethically not only caused a negative impact on the environment and alienated VW's customer base, but also had a massively negative effect on the company's bottom line and financial outlook. How many engineers at the company felt horribly about what was happening and felt powerless to do anything about it? And in that case, the failure of Volkswagen to do the right thing was bad for the company in a number of levels.

As we see that copyleft and best security are linked (I talked about the Honeymoon Effect during the talk, and you can read my old paper on medical device safety plus many great discussions by folks like Matthew Garrett and even Bruce Schneier) and we embark upon an Internet of Things network, the ethical implications of software freedom become all the more poignant. In addition to the ethical aspects inherent in sharing code and the ethical considerations of following a license under which you received software for your use, there's an additional ethics layer in the safety implications of keeping GPL'd code closed. Because software so often interacts in complex ways (as shown in the car vulnerability demonstrations that go through the wheel maintenance system to exploit the critical ignition and brake systems), it's impossible to predict which software the next failure will be based on.

We need companies to understand that complying with the GPL isn't just good community participation or a safeguard from lawsuits - it is fundamental to their longterm financial success in a myriad of ways. Developers play a key role in that process. It's not always easy to stand up for the right thing in a corporate context. Doing so can cause reprisal in the form of some penalty. Obviously, if an engineer had been able to take action at Volkswagen, they would have saved the company a lot of embarrassment and lost revenue but without the hindsight of seeing how that situation actually played out it's likely that there was a real fear of penalty for speaking up.

Fortunately, where copylefted software is involved there are external mechanisms to help with some of these issues. Because companies must make good on providing source when they distribute, an outsider could determine that a company is not meeting its obligations. This is the main reason why having the option of participating anonymously in our coalition of developers who want to enforce the GPL is so important. In software development, coming out in favor of enforcement may not cause you any negative repercussions with your current employer but many developers rightly worry that other future employers may negatively view their participation in the coalition.

In the same vein as my ethical education in engineering school, developers should include the long term ethical considerations in their core technical analysis of what free and open source software licenses their companies should use and how they comply with it on a long term basis. While failures are terrible to have, they're essential to learn from and work towards better technical and ethical infrastructure.

Video of the full talk is available. You can sign up to our GPL Compliance Project for Linux Developers by emailing <linux-services@sfconservancy.org>.

[permalink]

Tags: conservancy, GPL

How Would Software Freedom Have Helped With VW?

by Bradley M. Kuhn on September 29, 2015

Would software-related scandals, such as Volkswagen's use of proprietary software to lie to emissions inspectors, cease if software freedom were universal? Likely so, as I wrote last week. In a world where regulations mandate distribution of source code for all the software in all devices, and where no one ever cheats on that rule, VW would need means other than software to hide their treachery.

Universal software freedom is my lifelong goal, but I realized years ago that I won't live to see it. I suspect that generations of software users will need to repeatedly rediscover and face the harms of proprietary software before a groundswell of support demands universal software freedom. In the meantime, our community has invented semi-permanent strategies, such as copyleft, to maximize software freedom for users in our current mixed proprietary and Free Software world.

In the world we live in today, software freedom can impact the VW situation only if a few complex conditions are met. Let's consider the necessary hypothetical series of events, in today's real world, that would have been necessary for Open Source and Free Software to have stopped VW immediately.

First, VW would have created a combined or derivative work of software with a copylefted program. While many cars today contain Linux, which is copylefted, I am not aware of any cars that use Linux outside of the on-board entertainment and climate control systems. The VW software was not part of those systems, and VW engineers almost surely wrote the emissions testing mode code from scratch. Even if they included some non-copylefted Open Source or Free Software in it, those licenses don't require disclosure of any source code; VW's ability to conceal its bad actions with non-copylefted code is roughly identical to the situation of proprietary VW code before us. As a thought experiment, though, let's pretend, that VW based the nefarious code on Linux by writing a proprietary Linux module to trick the emissions testing systems.

In that case, VW would have violated the GPL. But that alone is far from enough to ensure anyone would catch VW. Indeed, GPL violations remain very prevalent, and only one organization, Conservancy, enforces the GPL for Linux. Conservancy has such limited enforcement resources (only three full-time people on staff, and enforcement is one of many of our programs), I suspect that years would pass before Conservancy had the resources to pursue the violation; Conservancy currently has hundreds of Linux GPL violations queued for action. Even once opened, most GPL violations take years to resolve. As an example, we are currently enforcing the GPL against one auto manufacturer who has Linux in their car. We've already spent hundreds of hours and the company to date continues to fail in their GPL compliance efforts. Admittedly, it's highly unlikely that particular violator has a GPL-violating Linux module specifically designed to circumvent automotive regulations. However, after enforcing the GPL in that case for more than two years, I still don't have enough data about their use of Linux to even know which proprietary Linux modules are present — let alone whether those modules are nefarious in any way other than as violating Linux's license.

Thus, in today's world, a “software freedom solution” to prevent the VW scandal must meet unbelievable preconditions: (a) VW would have to base all its software on copylefted Open Source and Free Software, and (b) an organization with a mission to enforce copyleft for the public good would require the resources to find the majority of GPL violators and ensure compliance in a timely fashion. This thought experiment quickly shows how much more work remains to advance and defend software freedom. While requirements of source code disclosure, such as those in copyleft licenses, are necessary to assure the benefits of software freedom, they cannot operate unless someone exercises the offers for source and looks at the details.

We live in a world where most of the population accepts proprietary software as legitimate. Even major trade associations in the Open Source community laud companies who make proprietary software, as long as they adopt and occasionally contribute to some Free Software too. Currently, it feels like software freedom is winning, because the overwhelming majority in the software industry believe Open Source and Free Software is useful and superior in some circumstances. Furthermore, while I appreciate the aspirational ideal of voluntary Open Source, I find in my work that so many companies, just as VW did, will cheat against important social good policies unless someone watches and regulates. Mere adoption of Open Source won't work alone; we only yield the valuable results of software freedom if software is copylefted and someone upholds that copyleft.

Indeed, just as it has been since the 1980s, very few people believe that software freedom is of fundamental importance for all software users. Scandals, like VW's use of proprietary software to hide other bad acts, might slowly change opinions, but one scandal is rarely enough to permanently change public opinion. I therefore encourage those who support software freedom to take this incident as inspiration for a stronger stance, and to prepare yourselves for the long haul of software freedom advocacy.

[permalink]

Tags: conservancy, GPL

Exercising Software Freedom in the Global Email System

by Bradley M. Kuhn on September 15, 2015

In this post, I discuss one example of how a choice for software freedom can cause many strange problems that others will dismiss. My goal here is to explain in gory detail how proprietary software biases in the computing world continue to grow, notwithstanding Open Source ballyhoo.

Two decades ago, nearly every company, organization, entity, and tech-minded individual ran their own email server. Generally speaking, even back then, nearly all the software for both MTAs and MUAs were Free Software⁰. MTA's are the mail transport agents — the complex software that moves email around from one Internet domain to another. MUAs are the mail user agents, sometimes called mail clients — the local programs with which users manipulate their own email.

I've run my own MTA since around 1993: initially with sendmail, then with exim for a while, and with Postfix since 1999 or so. Also, everywhere I've worked throughout my entire career since 1995, I've either been in charge of — or been the manager of the person in charge of — the MTA installation for the organization where I worked. In all cases, that MTA has always been Free Software, of course.

However, the world of email has changed drastically during that period. The most notable change in the email world is the influx of massive amounts of spam, which has been used as an excuse to implement another disturbing change. Slowly but surely, email service — both the MTA and the MUA — have been outsourced for most organizations. Specifically, either (a) organizations run proprietary software on their own computers to deal with email and/or (b) people pay a third-party to run proprietary and/or trade-secret software on their behalf to handle the email services. Email, generally speaking, isn't handled by Free Software all that much anymore.

This situation became acutely apparent to me this earlier this month when Conservancy moved its email server. I had plenty of warning that the move was needed¹, and I'd set up a test site on the new server. We sent and received some of our email for months (mostly mailing list traffic) using that server configured with a different domain (sf-conservancy.org). When the shut-off day came, I moved sfconservancy.org's email officially. All looked good: I had a current Debian, with a new version of Postfix and Dovecot on a speedier host, and with better spam protection settings in Postfix and better spam filtering with a newer version of SpamAssassin. All was going great, thanks to all those great Free Software projects — until the proprietary software vendors threw a spanner in our works.

For reasons that we'll never determine for sure², the IPv4 number that our new hosting provide gave us was already listed on many spam blacklists. I won't debate the validity of various blacklists here, but the fact is, for nearly every public-facing, pure-blacklist-only service, delisting is straightforward, takes about 24 hours, and requires at most answering some basic questions about your domain name and answering a captcha-like challenge. These services, even though some are quite dubious, are not the center of my complaint.

The real peril comes from third-party email hosting companies. These companies have arbitrary, non-public blacklisting rules. More importantly, they are not merely blacklist maintainers, they are MTA (and in some cases, even MUA) providers who sell their proprietary and/or trade-secret hosted solutions as a package to customers. Years ago, the idea of giving up that much control of what happens to your own email would be considered unbelievable. Today, it's commonplace.

And herein lies the fact that is obvious to most software freedom advocates but indiscernible by most email users. As a Free Software user, with your own MTA on your own machine, your software only functions if everyone else respects your right to run that software yourself. Furthermore, if the people you want to email are fully removed from their hosting service, they won't realize nor understand that their hosting site might block your emails. These companies have their customers fully manipulated to oppose your software freedom. In other words, you can't appeal to those customers (the people you want to email), because you're likely the only person to ever raise this issue with them (i.e., unless they know you very well, they'll assume you're crazy). You're left begging to the provider, whom you have no business relationship with, to convince them that their customers want to hear from you. Your voice rings out indecipherable from the spammers who want the same permission to attack their customers.

The upshot for Conservancy? For days, Microsoft told all its customers that Conservancy is a spammer; Microsoft did it so subtly that the customers wouldn't even believe it if we told them. Specifically, every time I or one of my Conservancy colleagues emailed organizations using Microsoft's “Exchange Online”, “Office 365” or similar products to host email for their domain⁴, we got the following response:

Sep  2 23:26:26 pine postfix/smtp[31888]: 27CD6E12B: to=, relay=example-org.mail.protection.outlook.com[207.46.163.215]:25, delay=5.6, delays=0.43/0/0.16/5, dsn=5.7.1, status=bounced (host example-org.mail.protection.outlook.com[207.46.163.215] said: 550 5.7.1 Service unavailable; Client host [162.242.171.33] blocked using FBLW15; To request removal from this list please forward this message to delist@messaging.microsoft.com (in reply to RCPT TO command))

Oh, you ask, did you forward your message to the specified address? Of course I did; right away! I got back an email that said:

Hello ,

Thank you for your delisting request SRXNUMBERSID. Your ticket was received on (Sep 01 2015 06:13 PM UTC) and will be responded to within 24 hours.

Once we passed the 24 hour mark with no response, I started looking around for more information. I also saw a suggestion online that calling is the only way to escalate one of those tickets, so I phoned 800-865-9408 and gave V-2JECOD my ticket number and she told that I could only raise these issues with the “Mail Flow Team”. She put me on hold for them, and told me that I was number 2 in the queue for them so it should be a few minutes. I waited on hold for just under six hours. I finally reached a helpful representative, who said the ticket was the lowest level of escalation available (he hinted that it would take weeks to resolve at that level, which is consistent with other comments about this problem I've seen online). The fellow on the phone agreed to escalate it to the highest priority available, and said within four hours, Conservancy should be delisted. Thus, ultimately, I did resolve these issues after about 72 hours. But, I'd spent about 15 hours all-told researching various blacklists, email hosting companies, and their procedures³, and that was after I'd already carefully configured our MTA and DNS to be very RFC-compliant (which is complicated and confusing, but absolutely essential to stay off these blacklists once you're off).

Admittedly, this sounds like a standard Kafkaesque experience with a large company that almost everyone in post-modern society has experienced. However, it's different in one key way: I had to convince Microsoft to allow me to communicate with their customers who are paying Microsoft for proprietary and/or trade-secret software and services, ostensibly to improve efficiency of their communications. Plus, since Microsoft, by the nature of their so-called spam blocking, doesn't inform their customers whom they've blocked, I and my colleagues would have just sounded crazy if we'd asked our contacts to call their provider instead. (I actually considered this, and realized that we might negatively impact relationships with professional contacts.)

These problems do reduce email software freedom by network effects. Most people rely on third-party proprietary email software from Google, Microsoft, Barracuda, or others. Therefore, most people, don't exercise any software freedom regarding email services. Since exercising software freedom for email slowly becomes a rarer and rarer (rather than norm it once was), society slowly but surely pegs those who do exercise software freedom as “random crazy people”.

There are a few companies who are seeking to do email hosting in a way that respects your software freedom. The real test of such companies is if someone technically minded can get the same software configured on their own systems, and have it work the same way. Yet, in most cases, you go to one of these companies' Github pages and find a bunch of stuff pushed public, but limited information on how to configure it so that it functions the same way the hosted service does. RMS wrote years ago that Free Software cannot properly succeed without Free Documentation, and in many of these hosting cases: the hosting company is using fully upstreamed Free Software, but has configured the software in a way that is difficult to stumble upon by oneself. (For that reason, I'm committing to writing up tutorials on how Conservancy configured our mail server, so at least I'll be part of the solution instead of part of the problem.)

BTW, as I dealt with all this, I couldn't help but think of John Gilmore's activism efforts regarding open mail relays. While I don't agree with all of John's positions on this, his fundamental position is right: we must oppose companies who think they know better how we should configure our email servers (or on which IP numbers we should run those servers). I'd add a corollary that there's a serious threat to software freedom, at least with regard to email software, if we continue to allow such top-down control of the once beautifully decentralized email system.

The future of software freedom depends on issues like this. Imagine someone who has just learned that they can run their own email server, or bought some Free Software-based plug computing system that purports to be a “home cloud” service with email. There's virtually no chance that such users would bother to figure all this out. They'd see their email blocked, declare the “home cloud” solution useless, and would just get a gmail.com, outlook.com, or some other third-party email account. Thus, I predict that software freedom that we once had, for our MTAs and MUAs, will eventually evaporate for everyone except those tiny few who invest the time to understand these complexities and fight the for-profit corporate power that curtails software freedom. Furthermore, that struggle becomes Sisyphean as our numbers dwindle.

Email is the oldest software-centric communication system on the planet. The global email system serves as a canary in the coalmine regarding software freedom and network service freedom issues. Frighteningly, software now controls most of the global communications systems. How long will it be before mobile network providers refuse to terminate PSTN calls or SMS's sent from devices running modified Android firmwares like Replicant? Perhaps those providers, like large email providers, will argue that preventing robocalls (the telephone equivalent of SPAM) necessitates such blocking. Such network effects place so many dystopias on software freedom's horizon.

I don't deny that every day, there is more Free Software existing in the world than has ever existed before — the P.T. Barnum's of Open Source have that part right. The part they leave out is that, each day, their corporate backers make it a little more difficult to complete mundane tasks using only Free Software. Open Source wins the battle while software freedom loses the war.

---

⁰Yes, I'm intimately aware that Elm's license was non-free, and that the software freedom of PINE's license was in question. That's slightly relevant here but mostly orthogonal to this point, because Free Software MUAs were still very common then, and there were projects to actively rewrite the ones whose software freedom was in question

¹For the last five years, one of Conservancy's Director Emeriti, Loïc Dachary, has donated an extensive amount of personal time and in-kind donations by providing Cloud server for Conservancy to host its three key servers, including the email server. The burden of maintaining this for us became too time consuming (very reasonably), and Loïc's asked us to find another provider. I want, BTW, to thank Loïc his for years of volunteer work maintaining infrastructure for us; he provided this service for much longer than we could have hoped! Loïc also gave us plenty of warning that we'd need to move. None of these problems are his fault in the least!

²The obvious supposition is that, because IPv4 numbers are so scarce, this particular IP number was likely used previously by a spammer who was shut down.

³I of course didn't count the time time on phone hold, as I was able to do other work while waiting, but less efficiently because the hold music was very distracting.

⁴If you want to see if someone's domain is a Microsoft customer, see if the MX record for their domain (say, example.org) points to example-org.mail.protection.outlook.com.

[permalink]

Tags: conservancy

Summary of My DebConf 15 Keynote

by Bradley M. Kuhn on August 17, 2015

In my keynote I outlined the advantages of copyright aggregation for community-oriented projects like Debian. Not only does copyright aggregation assure that a well-equipped organization can enforce copyleft licenses, but also the organization can handle future relicensing requests and cooperate with other Free Software communities who need license exceptions. Holding copyright is a privilege, but it is also a burden, since copyright gives the copyright holder excessive power. In the Free Software community, we mitigate that power by choosing a Free Software license (as I explained in the essay that I cowrote with RMS in 2001). But copyright grants yet another power — which ultimately becomes an obligation. The copyright holder must, on behalf of users, ensure compliance with copyleft so that the users' software freedom is always respected. Conservancy can now help Debian with that arduous task.

In my keynote, I announced an exciting new project that Debian is undertaking with Conservancy, called the Debian Copyright Aggregation Project to address these issues. Debian contributors who choose to can assign their copyrights to Conservancy so that we may shoulder this burden on behalf of the Debian community.

For those Debian contributors who find copyright assignment too heavy-weight or otherwise problematic for their principles, Conservancy's enforcement agreement process, already in use by Conservancy's Samba, BusyBox, and GPL Compliance Project for Linux Developers, allows Debian copyright holders to delegate a revocable license enforcement authority to Conservancy. Furthermore, both these rights delegation programs are purely voluntary and optional for all Debian copyright holders.

I and my colleagues at Conservancy look forward to providing Debian to ongoing access to Conservancy's Free Software licensing and enforcement expertise. Conservancy is available to handle questions and concerns from the Debian community. For efficiency and streamlined access to this service, Debian community members who have such questions should channel them through the DPL, who will manage the communication path with Conservancy staff on these matters.

Finally, and slightly off topic but quite important, I thank the Debian community for their years of excellent work. Conservancy uses Debian heavily for its own daily work, and all Conservancy's staff are delighted to provide these services to Debian.

[permalink]

Tags: conservancy