Exercising Software Freedom in the Global Email System

by Bradley M. Kuhn on September 15, 2015

In this post, I discuss one example of how a choice for software freedom can cause many strange problems that others will dismiss. My goal here is to explain in gory detail how proprietary software biases in the computing world continue to grow, notwithstanding Open Source ballyhoo.

Two decades ago, nearly every company, organization, entity, and tech-minded individual ran their own email server. Generally speaking, even back then, nearly all the software for both MTAs and MUAs were Free Software⁰. MTA's are the mail transport agents — the complex software that moves email around from one Internet domain to another. MUAs are the mail user agents, sometimes called mail clients — the local programs with which users manipulate their own email.

I've run my own MTA since around 1993: initially with sendmail, then with exim for a while, and with Postfix since 1999 or so. Also, everywhere I've worked throughout my entire career since 1995, I've either been in charge of — or been the manager of the person in charge of — the MTA installation for the organization where I worked. In all cases, that MTA has always been Free Software, of course.

However, the world of email has changed drastically during that period. The most notable change in the email world is the influx of massive amounts of spam, which has been used as an excuse to implement another disturbing change. Slowly but surely, email service — both the MTA and the MUA — have been outsourced for most organizations. Specifically, either (a) organizations run proprietary software on their own computers to deal with email and/or (b) people pay a third-party to run proprietary and/or trade-secret software on their behalf to handle the email services. Email, generally speaking, isn't handled by Free Software all that much anymore.

This situation became acutely apparent to me this earlier this month when Conservancy moved its email server. I had plenty of warning that the move was needed¹, and I'd set up a test site on the new server. We sent and received some of our email for months (mostly mailing list traffic) using that server configured with a different domain (sf-conservancy.org). When the shut-off day came, I moved sfconservancy.org's email officially. All looked good: I had a current Debian, with a new version of Postfix and Dovecot on a speedier host, and with better spam protection settings in Postfix and better spam filtering with a newer version of SpamAssassin. All was going great, thanks to all those great Free Software projects — until the proprietary software vendors threw a spanner in our works.

For reasons that we'll never determine for sure², the IPv4 number that our new hosting provide gave us was already listed on many spam blacklists. I won't debate the validity of various blacklists here, but the fact is, for nearly every public-facing, pure-blacklist-only service, delisting is straightforward, takes about 24 hours, and requires at most answering some basic questions about your domain name and answering a captcha-like challenge. These services, even though some are quite dubious, are not the center of my complaint.

The real peril comes from third-party email hosting companies. These companies have arbitrary, non-public blacklisting rules. More importantly, they are not merely blacklist maintainers, they are MTA (and in some cases, even MUA) providers who sell their proprietary and/or trade-secret hosted solutions as a package to customers. Years ago, the idea of giving up that much control of what happens to your own email would be considered unbelievable. Today, it's commonplace.

And herein lies the fact that is obvious to most software freedom advocates but indiscernible by most email users. As a Free Software user, with your own MTA on your own machine, your software only functions if everyone else respects your right to run that software yourself. Furthermore, if the people you want to email are fully removed from their hosting service, they won't realize nor understand that their hosting site might block your emails. These companies have their customers fully manipulated to oppose your software freedom. In other words, you can't appeal to those customers (the people you want to email), because you're likely the only person to ever raise this issue with them (i.e., unless they know you very well, they'll assume you're crazy). You're left begging to the provider, whom you have no business relationship with, to convince them that their customers want to hear from you. Your voice rings out indecipherable from the spammers who want the same permission to attack their customers.

The upshot for Conservancy? For days, Microsoft told all its customers that Conservancy is a spammer; Microsoft did it so subtly that the customers wouldn't even believe it if we told them. Specifically, every time I or one of my Conservancy colleagues emailed organizations using Microsoft's “Exchange Online”, “Office 365” or similar products to host email for their domain⁴, we got the following response:

Sep  2 23:26:26 pine postfix/smtp[31888]: 27CD6E12B: to=, relay=example-org.mail.protection.outlook.com[207.46.163.215]:25, delay=5.6, delays=0.43/0/0.16/5, dsn=5.7.1, status=bounced (host example-org.mail.protection.outlook.com[207.46.163.215] said: 550 5.7.1 Service unavailable; Client host [162.242.171.33] blocked using FBLW15; To request removal from this list please forward this message to delist@messaging.microsoft.com (in reply to RCPT TO command))

Oh, you ask, did you forward your message to the specified address? Of course I did; right away! I got back an email that said:

Hello ,

Thank you for your delisting request SRXNUMBERSID. Your ticket was received on (Sep 01 2015 06:13 PM UTC) and will be responded to within 24 hours.

Once we passed the 24 hour mark with no response, I started looking around for more information. I also saw a suggestion online that calling is the only way to escalate one of those tickets, so I phoned 800-865-9408 and gave V-2JECOD my ticket number and she told that I could only raise these issues with the “Mail Flow Team”. She put me on hold for them, and told me that I was number 2 in the queue for them so it should be a few minutes. I waited on hold for just under six hours. I finally reached a helpful representative, who said the ticket was the lowest level of escalation available (he hinted that it would take weeks to resolve at that level, which is consistent with other comments about this problem I've seen online). The fellow on the phone agreed to escalate it to the highest priority available, and said within four hours, Conservancy should be delisted. Thus, ultimately, I did resolve these issues after about 72 hours. But, I'd spent about 15 hours all-told researching various blacklists, email hosting companies, and their procedures³, and that was after I'd already carefully configured our MTA and DNS to be very RFC-compliant (which is complicated and confusing, but absolutely essential to stay off these blacklists once you're off).

Admittedly, this sounds like a standard Kafkaesque experience with a large company that almost everyone in post-modern society has experienced. However, it's different in one key way: I had to convince Microsoft to allow me to communicate with their customers who are paying Microsoft for proprietary and/or trade-secret software and services, ostensibly to improve efficiency of their communications. Plus, since Microsoft, by the nature of their so-called spam blocking, doesn't inform their customers whom they've blocked, I and my colleagues would have just sounded crazy if we'd asked our contacts to call their provider instead. (I actually considered this, and realized that we might negatively impact relationships with professional contacts.)

These problems do reduce email software freedom by network effects. Most people rely on third-party proprietary email software from Google, Microsoft, Barracuda, or others. Therefore, most people, don't exercise any software freedom regarding email services. Since exercising software freedom for email slowly becomes a rarer and rarer (rather than norm it once was), society slowly but surely pegs those who do exercise software freedom as “random crazy people”.

There are a few companies who are seeking to do email hosting in a way that respects your software freedom. The real test of such companies is if someone technically minded can get the same software configured on their own systems, and have it work the same way. Yet, in most cases, you go to one of these companies' Github pages and find a bunch of stuff pushed public, but limited information on how to configure it so that it functions the same way the hosted service does. RMS wrote years ago that Free Software cannot properly succeed without Free Documentation, and in many of these hosting cases: the hosting company is using fully upstreamed Free Software, but has configured the software in a way that is difficult to stumble upon by oneself. (For that reason, I'm committing to writing up tutorials on how Conservancy configured our mail server, so at least I'll be part of the solution instead of part of the problem.)

BTW, as I dealt with all this, I couldn't help but think of John Gilmore's activism efforts regarding open mail relays. While I don't agree with all of John's positions on this, his fundamental position is right: we must oppose companies who think they know better how we should configure our email servers (or on which IP numbers we should run those servers). I'd add a corollary that there's a serious threat to software freedom, at least with regard to email software, if we continue to allow such top-down control of the once beautifully decentralized email system.

The future of software freedom depends on issues like this. Imagine someone who has just learned that they can run their own email server, or bought some Free Software-based plug computing system that purports to be a “home cloud” service with email. There's virtually no chance that such users would bother to figure all this out. They'd see their email blocked, declare the “home cloud” solution useless, and would just get a gmail.com, outlook.com, or some other third-party email account. Thus, I predict that software freedom that we once had, for our MTAs and MUAs, will eventually evaporate for everyone except those tiny few who invest the time to understand these complexities and fight the for-profit corporate power that curtails software freedom. Furthermore, that struggle becomes Sisyphean as our numbers dwindle.

Email is the oldest software-centric communication system on the planet. The global email system serves as a canary in the coalmine regarding software freedom and network service freedom issues. Frighteningly, software now controls most of the global communications systems. How long will it be before mobile network providers refuse to terminate PSTN calls or SMS's sent from devices running modified Android firmwares like Replicant? Perhaps those providers, like large email providers, will argue that preventing robocalls (the telephone equivalent of SPAM) necessitates such blocking. Such network effects place so many dystopias on software freedom's horizon.

I don't deny that every day, there is more Free Software existing in the world than has ever existed before — the P.T. Barnum's of Open Source have that part right. The part they leave out is that, each day, their corporate backers make it a little more difficult to complete mundane tasks using only Free Software. Open Source wins the battle while software freedom loses the war.

---

⁰Yes, I'm intimately aware that Elm's license was non-free, and that the software freedom of PINE's license was in question. That's slightly relevant here but mostly orthogonal to this point, because Free Software MUAs were still very common then, and there were projects to actively rewrite the ones whose software freedom was in question

¹For the last five years, one of Conservancy's Director Emeriti, Loïc Dachary, has donated an extensive amount of personal time and in-kind donations by providing Cloud server for Conservancy to host its three key servers, including the email server. The burden of maintaining this for us became too time consuming (very reasonably), and Loïc's asked us to find another provider. I want, BTW, to thank Loïc his for years of volunteer work maintaining infrastructure for us; he provided this service for much longer than we could have hoped! Loïc also gave us plenty of warning that we'd need to move. None of these problems are his fault in the least!

²The obvious supposition is that, because IPv4 numbers are so scarce, this particular IP number was likely used previously by a spammer who was shut down.

³I of course didn't count the time time on phone hold, as I was able to do other work while waiting, but less efficiently because the hold music was very distracting.

⁴If you want to see if someone's domain is a Microsoft customer, see if the MX record for their domain (say, example.org) points to example-org.mail.protection.outlook.com.

[permalink]

Tags: conservancy

Summary of My DebConf 15 Keynote

by Bradley M. Kuhn on August 17, 2015

In my keynote I outlined the advantages of copyright aggregation for community-oriented projects like Debian. Not only does copyright aggregation assure that a well-equipped organization can enforce copyleft licenses, but also the organization can handle future relicensing requests and cooperate with other Free Software communities who need license exceptions. Holding copyright is a privilege, but it is also a burden, since copyright gives the copyright holder excessive power. In the Free Software community, we mitigate that power by choosing a Free Software license (as I explained in the essay that I cowrote with RMS in 2001). But copyright grants yet another power — which ultimately becomes an obligation. The copyright holder must, on behalf of users, ensure compliance with copyleft so that the users' software freedom is always respected. Conservancy can now help Debian with that arduous task.

In my keynote, I announced an exciting new project that Debian is undertaking with Conservancy, called the Debian Copyright Aggregation Project to address these issues. Debian contributors who choose to can assign their copyrights to Conservancy so that we may shoulder this burden on behalf of the Debian community.

For those Debian contributors who find copyright assignment too heavy-weight or otherwise problematic for their principles, Conservancy's enforcement agreement process, already in use by Conservancy's Samba, BusyBox, and GPL Compliance Project for Linux Developers, allows Debian copyright holders to delegate a revocable license enforcement authority to Conservancy. Furthermore, both these rights delegation programs are purely voluntary and optional for all Debian copyright holders.

I and my colleagues at Conservancy look forward to providing Debian to ongoing access to Conservancy's Free Software licensing and enforcement expertise. Conservancy is available to handle questions and concerns from the Debian community. For efficiency and streamlined access to this service, Debian community members who have such questions should channel them through the DPL, who will manage the communication path with Conservancy staff on these matters.

Finally, and slightly off topic but quite important, I thank the Debian community for their years of excellent work. Conservancy uses Debian heavily for its own daily work, and all Conservancy's staff are delighted to provide these services to Debian.

[permalink]

Tags: conservancy

Understanding Conservancy Through the GSoC Lens

by Bradley M. Kuhn on September 11, 2014

[ A version of this post originally appeared on the Google Open Source Blog. ]

Software Freedom Conservancy, Inc. is a 501(c)(3) non-profit charity that serves as a home to Open Source and Free Software projects. Such is easily said, but in this post I'd like to discuss what that means in practice for an Open Source and Free Software project and why such projects need a non-profit home. In short, a non-profit home makes the lives of Free Software developers easier, because they have less work to do outside of their area of focus (i.e., software development and documentation).

As the summer of 2014 ends, Google Summer of Code (GSoC) coordination work exemplifies the value a non-profit home brings its Free Software projects. GSoC is likely the largest philanthropic program in the Open Source and Free Software community today. However, one of the most difficult things for organizations that seek to take advantage of such programs is the administrative overhead necessary to take full advantage of the program. Google invests heavily in making it easy for organizations to participate in the program — such as by handling the details of stipend payments to students directly. However, to take full advantage of any philanthropic program, the benefiting organization has some work to do. For its member projects, Conservancy is the organization that gets that logistical work done.

For example, Google kindly donates $500 to the mentoring organization for every student it mentors. However, these funds need to go “somewhere”. If the funds go to an individual, there are two inherent problems. First, that individual is responsible for taxes on that income. Second, funds that belong to an organization as a whole are now in the bank account of a single project leader. Conservancy solves both those problems: as a tax-exempt charity, the mentor payments are available for organizational use under its tax exemption. Furthermore, Conservancy maintains earmarked funds for each of its projects. Thus, Conservancy keeps the mentor funds for the Free Software project, and the project leaders can later vote to make use of the funds in a manner that helps the project and Conservancy's charitable mission. Often, projects in Conservancy use their mentor funds to send developers to important conferences to speak about the project and recruit new developers and users.

Meanwhile, Google also offers to pay travel expenses for two mentors from each mentoring organization to attend the annual GSoC Mentor Summit (and, this year, it's an even bigger Reunion conference!). Conservancy handles this work on behalf of its member projects in two directions. First, for developers who don't have a credit card or otherwise are unable to pay for their own flight and receive reimbursement later, Conservancy staff book the flights on Conservancy's credit card. For the other travelers, Conservancy handles the reimbursement details. On the back end of all of this, Conservancy handles all the overhead annoyances and issues in requesting the POs from Google, invoicing for the funds, and tracking to ensure payment is made. While the Google staff is incredibly responsive and helpful on these issues, the Googlers need someone on the project's side to take care of the details. That's what Conservancy does.

GSoC coordination is just one of the many things that Conservancy does every day for its member projects. If there's anything other than software development and documentation that you can imagine a project needs, Conservancy does that job for its member projects. This includes not only mundane items such as travel coordination, but also issues as complex as trademark filings and defense, copyright licensing advice and enforcement, governance coordination and mentoring, and fundraising for the projects. Some of Conservancy's member projects have been so successful in Conservancy that they've been able to fund developer salaries — often part-time but occasionally full-time — for years on end to allow them to focus on improving the project's software for the public benefit.

Finally, if your project seeks help with regard to handling its GSoC funds and travel, or anything else mentioned on Conservancy's list of services to member projects, Conservancy is welcoming new applications for membership. Your project could join Conservancy's more than thirty other member projects and receive these wonderful services to help your community grow and focus on its core mission of building software for the public good.

---

[permalink]

Tags: conservancy, Google Summer of Code

Why Conservancy's Kallithea Project Exists

by Bradley M. Kuhn on July 15, 2014

Eleven days ago, Conservancy announced Kallithea. Kallithea is a GPLv3'd system for hosting and managing Mercurial and Git repositories on one's own servers. As Conservancy mentioned in its announcement, Kallithea is indeed based on code released under GPLv3 by RhodeCode GmbH. Below, I describe why Conservancy chose to serve as non-profit home to an obvious fork (as this is the first time Conservancy ever welcomed a fork as a member project).

The primary impetus for Kallithea is that more recent versions of RhodeCode GmbH's codebase contain a very unorthodox and ambiguous license statement, which states:

(1) The Python code and integrated HTML are licensed under the GPLv3 license as is RhodeCode itself.
(2) All other parts of the RhodeCode including, but not limited to the CSS code, images, and design are licensed according to the license purchased.

Simply put, this licensing scheme is — either (a) a GPL violation, (b) an unclear license permission statement under the GPL which leaves the redistributor feeling unclear about their rights, or (c) both.

When members of the Mercurial community first brought this license to Conservancy's attention about ten months ago, the first focus was to form a formal opinion regarding (a). Of course, Conservancy did form such an opinion, and you can probably guess what that is. However, I realized a few weeks later that this analysis really didn't matter in this case; the situation called for a more innovative solution.

Indeed, I recalled at that time the disputes between AT&T and University of California at Berkeley over BSD. In that case, while nearly all of the BSD code was adjudicated as freely licensed, the dispute itself was painful for the BSD community. BSD's development slowed nearly to a standstill for years while the legal disagreement was resolved. Court action — even if you're in the right — isn't always the fastest nor best way to push forward an important Free Software project.

In the case of RhodeCode's releases, there was an obvious and more productive solution. Namely, the 1.7.2 release of RhodeCode's codebase, written primarily by Marcin Kuzminski was fully released under GPLv3-only, and provided an excellent starting point to begin a GPLv3'd fork. Furthermore, some of the improved code in the 2.2.5 era of RhodeCode's codebase were explicitly licensed under GPLv3 by RhodeCode GmbH itself. Finally, many volunteers produced patches for all versions of RhodeCode's codebase and released those patches under GPLv3, too. Thus, there was already a burgeoning GPLv3-friendly community yearning to begin.

Conservancy's primary contribution, therefore, was to vet and verify a completely indisputable GPLv3'd version of the codebase. This was extensive and time consuming work; I personally spent over 100 hours to reach this point, and I suspect many Kallithea volunteers have already spent that much and more. Ironically, the most complex part of the work so far was verifying and organizing the licensing situation regarding third-party Javascript (released under a myriad of various licenses). You can see the details of that work by reading the revision history of Kallithea (or, you can read an overview in Kallithea's LICENSE file).

Like with any Free Software codebase fork, acrimony and disagreement led to Kallithea's creation. However, as the person who made most of the early changesets for Kallithea, I want to thank RhodeCode GmbH for explicitly releasing some of their work under GPLv3. Even as I hereby reiterate publicly my previously private request that RhodeCode GmbH correct the parts of their licensing scheme that are (at best) problematic, and (at worst) GPL-violating, I also point out this simple fact to those who have been heavily criticizing and admonishing RhodeCode GmbH: the situation could be much worse! RhodeCode could have simply never released any of their code under the GPLv3 in the first place. After all, there are many well-known code hosting sites that refuse to release any of their code (or release only a pittance of small components). By contrast, the GPLv3'd RhodeCode software was nearly a working system that helped bootstrap the Kallithea community. We're grateful for that, and we welcome RhodeCode developers to contribute to Kallithea under GPLv3. We do note, of course, that RhodeCode developers sadly can't incorporate any of our improvements in their codebase, due to their problematic license. However, Conservancy extends again our offer (also made privately last year) to work with RhodeCode GmbH to correct its licensing problems.

[permalink]

Tags: conservancy, GPL, Kallithea