Open source community split over offer of 'corporate' welfare for critical dev tools

Linux Foundation presents IT and help to key volunteers – and some wonder if this is a deal with the Devil

Special report The free and open source software (FOSS) community is caught in a love triangle of sorts.

Sourceware, a volunteer group that has been supporting various critical FOSS developer tools for more than two decades, is being courted by The Linux Foundation's Open Source Security Foundation (OpenSSF). The OpenSSF aims to improve open source software security by providing Sourceware projects with more modern IT infrastructure.

But some members of the Sourceware community fear that accepting the help of the OpenSSF would give the corporate Linux world more leverage over FOSS developer tools. They would prefer to seek support from the Software Freedom Conservancy, a charitable non-profit that they believe is better aligned with software freedom.

The Linux Foundation, also a non-profit entity, is sponsored by, among others, Microsoft, Google, and Verizon; the conservancy is supported by Google, Mozilla, and others.

This clumsy community courtship has been contemplated for years and coalesced into a proposal in September, leading participants in the FOSS community to debate what patronage is appropriate and desirable. The outcome – how developer tools like the GNU Compiler Collection (GCC) are hosted and who pays for it – will have consequences, for better or worse.

Lighting the touch paper

At the end of August, systems software developer Frank Ch. Eigler sent a note to the Sourceware overseers mailing list announcing that the 24-year-old open source project had reached out to the Software Freedom Conservancy (SFC) for financial support.

Sourceware hosts a variety of free and open source (FOSS) software projects, including developer tools for the GNU Project, such as GCC, GDB, glibc, Binutils, and GNATS, and others. These are critical components in the open source ecosystem.

Eiger insisted the project's current infrastructure, provided by IBM's Red Hat, is fine. "Things are stable, new services are coming online, and users seem to be happy," he wrote. "However, it is always good to think about any future needs."

Planning for those needs is well under way. In a presentation at OpenJS World 2022 back on June 24, Brian Behlendorf, general manager of OpenSSF, described Sourceware projects as if they were held together by spit and string – something of a common trope for the trade.

"The build servers and the really critical pieces that are involved in the development of GCC, glibc, GDB – the fundamentals that enable things like the Linux kernel and the Linux operating systems and almost all the other interpreters for other languages that we build on top of – needed a bit more rigor," he said.

"It needed a bit more of what we have for the Linux kernel in terms of build systems. And so we've been working with that community to better support the build systems that everybody depends upon to be locked down and hardened."

Over the past few years, it has become apparent that the open source ecosystem – which provides the software to run much of the internet, the economy, and our critical infrastructure – would benefit from a bit more rigor.

The OpenSSF was formed in August 2020 to raise the bar for open source security, and subsequent cyberattacks like the SolarWinds supply chain fiasco, the Apache Log4j vulnerability, and Colonial Pipeline ransomware infection, to name a few, have drawn more attention to the organization's mission – something that hasn't been top of mind in the FOSS community.

By September 18, the GNU Toolchain Infrastructure (GTI) initiative cited in Behlendorf's presentation had gelled into a proposal that discussed in somewhat heated terms at the 2022 GNU Cauldron conference. Nine days later, further details were published to the Sourceware mailing list.

Specifically, the GTI aims to provide Sourceware projects with IT services managed by the Linux Foundation, such as git repositories, email systems, issue tracking systems, patch review systems, a website, documentation, CI/CD, software artifact management, and software supply chain best practices. The plan is to use free software and to work with supporters to develop that code for missing components.

Asked whether GTI represents a shift away from the volunteer era of open source toward something more professionalized, Behlendorf suggested that amateur and professional participation in FOSS can coexist.

"The reason why a lot of this kind of volunteer infrastructure has worked has been because by being transparent, by being focused on voluntary collaboration, [these projects] have been able to go far with very little resources," he explained in an interview with The Register.

"It's people contributing spare servers and it's companies in some cases – Sourceware was supported substantially by Red Hat in the form of both engineers and and and other financial resources. But really, it's this participatory kind of thing that has to work."

"In this new infrastructure, everything is still done very publicly," Behlendorf said.

"The code that is used to deliver it is all still free software. There's still room for volunteers to be able to step in and help with things either on the periphery or on pioneering new service offering or helping improve the tooling that's used in the delivery of the infrastructure. So I think there's perhaps a phase change towards greater professionalization in some ways, but it's not in a way that loses what has made open source not only very powerful but also a lot of fun to be a part of."

Open source security

GTI, or the notion of GNU Toolchain improvements, has been percolating in community discussions for several years and was mentioned by Behlendorf in a May 9 letter [PDF] to Congress about efforts to secure the open source ecosystem.

The project, explained Carlos O'Donell, a distinguished engineer at Red Hat who works on the GNU C Library for Red Hat Enterprise Linux, Fedora, and other projects, involves a collaboration with the Linux Foundation's OpenSSF to fund infrastructure and supply chain security.

According to O'Donell, "key stakeholders in the GNU Toolchain community" were briefed on the proposal and shaped it.

"The key stakeholders consulted include GNU Toolchain project leadership, GNU Toolchain project release managers, GNU Toolchain project core developers, major vendors, active Sourceware / Overseers administrators, and both John Sullivan and Zoë Kooyman of the Free Software Foundation," he wrote in the mailing list announcement.

But not everyone involved agrees with that assessment or is satisfied with the representations that have been made. And therein lies the problem: open source governance consists of herding cats. Members of the community have different ideas about how things should work and consensus building isn't easy or necessarily possible in every situation.

"Open source projects have this complex history of how do you get decision making done with so many disparate views," said O'Donell in an interview with The Register.

"And one of the ways you do this is you put together a proposal, you start sharing that proposal with people you know, and you trust with leadership in the community. And then you expand that proposal to a public discussion."

Show me the money

That's where we are now, said O'Donell, who added that some Sourceware admins have asked the SFC to open a bank account for them so they can look for alternative sources of fiscal sponsorship – even though Sourceware continues to be supported by Red Hat.

The central issue here is whether either the GTI or the application to involve SFC – which are not necessarily mutually exclusive – will change how the various projects hosted by Sourceware are governed or licensed. Those providing funding and support to open source projects – $285,000 in the first year under the GTI plan – often have the opportunity to shape those projects.

"It's really hard amongst a group of 20-ish people to get complete unanimity around things," said Behlendorf. "And especially when a subset of those folks had been voluntarily maintaining the Sourceware infrastructure, it's going to be hard to hear that some folks feel it's insufficient when a hard decision like this is really needed. So I saw this as a community making a really tough decision but they were making it with the right process."

The Linux Foundation is like loggers who claim to speak for the trees

Critics of the GTI deride it as a corporate takeover – a charge those involved with the GTI emphatically deny. There's a part of the FOSS community that believes the Linux Foundation, funded by major tech companies including Microsoft and Oracle, favors corporate interests over those of the community.

As Bruce Perens, one of the founders of the open-source movement, put it several years ago, "The Linux Foundation is like loggers who claim to speak for the trees."

"Ultimately, this is a classic discussion of what kinds of governance and organizations should be the homes for FOSS projects," said Bradley M. Kuhn, Policy Fellow at SFC, in an email to The Register.

"At SFC, we believe strongly that governance and organizational structure matter. Specifically, there are substantial governance differences between 501(c)(3) charities (such as the SFC) and 501(c)(6) non-profits (such as the Linux Foundation and OpenSSF)."

This is a classic discussion of what kinds of governance and organizations should be the homes for FOSS projects

Essentially, Kuhn argues, there's a difference between 501(c)(6) organizations, referred to as trade associations, which serve for-profit companies and promote common business interests, and 501(c)(3) organizations, referred to as charities, which promote activities like education and advocacy for the public good.

"This governance difference is stark in this particular situation," Kuln elaborated.

"While the details of the OpenSSF proposal to control the GCC, GDB, glibc, and Binutils' infrastructure remains hazy, they've stated that the governing body will be a group of companies, who buy seats on a committee that will control the projects' infrastructure. While that committee may well sometimes act in the interest of the community (by taking advice from a technical advisory committee, which apparently gets collectively only one vote), it's not guaranteed."

Mark Wielaard, a Sourceware overseer and senior principal engineer with Red Hat's Platform Tools group, told The Register in an email that Sourceware has been going for almost 25 years and those involved would love to have that continue for another 25 years.

"We work directly with the communities and while we're always working on improvements, and the users always have requests, there was not a groundswell of interest to move to a different hosting platform," said Wielaard. "As far as we understand the situation, a proposal to move services is not generally supported by various projects, including some of the GNU projects, hosted by Sourceware."

Wielaard said the above mentioned discussions with stakeholders were "done very selectively, with minimal information given about the structure of the new governance structure or actual technical plans." He noted that the SFC held public consultations on supporting Sourceware and asked the Linux Foundation to participate.

"The Linux Foundation chose to not join any of these public discussions over the last year," said Wielaard. "Even now that they've presented the plan publicly, they've still only provided minimal details."

Zoë Kooyman, executive director of the Free Software Foundation (FSF), told The Register that the FSF has been presented with the Linux Foundation / OpenSSF proposal and has engaged with the OpenSSF, the SFC, and the Sourceware volunteers. She said the FSF conducted a community conversation on the subject just this week.

"The focus of the discussion has been on infrastructure for the GNU Toolchain projects," said Kooyman.

"While the FSF provides overall fiscal sponsorship, including raising and holding funds for the projects, we have not had to provide the infrastructure since the volunteers at Sourceware have done (and continue to do) such a good job of that.

"Generally the FSF is supportive of projects receiving resources and using infrastructure from other sources, as long as that support comes without strings, is supported by the community, and is consistent with the mission of fully free software everywhere. We have shown ourselves willing to explore improvements to the infrastructure, but we have so far not made any endorsements.

"Decisions have to be made in accordance with decision-making practices of the different GNU Toolchain packages by their maintainers. We know people expect freedom whenever they see the GNU name, and it's an important part of the FSF's job to make sure that expectation is met. That means we are looking for a set of standard practices and guarantees, and we are exploring if we feel this proposal could satisfy those. Now, and in the future."

No change

Asked whether any of the GTI discussions have touched on the possibility of changing the software licensing of the GNU Toolchain, Kooyman said, "No, we have not been asked that during any part of these conversations and the FSF is committed to copyleft."

We aren't politicians; we just want to volunteer doing our work to help the community, and we're frustrated by all this

The community conversation, said Wielaard, "didn't answer most of the questions the community had," and pointed to questions asked during the session though a chat box that were never addressed.

"As such, that session earlier this week didn't really provide many more answers and just generated more questions. So there remain unanswered questions about specifics and there hasn't really been much community discussion about the Linux Foundation plans."

Wielaard said the discussion with the SFC is ongoing, public, and positive, as the SFC's engagement with the FSF has been.

"We aren't politicians; we just want to volunteer doing our work to help the community, and we're frustrated by all this," Wielaard said. "The SFC and FSF understand these politics so we've reached out to them to help us deal with all of it.

"We are also in talks with the FSF tech-team, which is responsible for some parts of the GNU projects which also get some services from Sourceware. We expect to work more closely with them in the future to share resources, backups, software releases, etc."

Asked where the GTI stands, O'Donell said it's just getting started: "We're going to be putting together infrastructure proposals, and we're going to be iterating with the community on those proposals to get their feedback. This is just the very, very, very beginning. Some people feel like it's the end, but it's totally the opposite of the end." ®