The unanswered question at CentOS community Q&A: How can we trust you now?

Project will share more resources with Fedora and use a single login

The CentOS board conducted a public Q&A just ahead of last week's FOSDEM 2021 open source conference – and there was an awkward silence when someone asked whether changing the end-of-life (EOL) date for a released project is something that might happen again.

This killer query came about half an hour into the session, which can be viewed here. "A question has come in about the change of the EOL for a community deliverable during a release being very unusual. Is this [a] thing that could theoretically happen in the future?" said moderator Brian Exelbierd from Red Hat (and also a board member).

A lengthy silence ensued before Pat Riehecky from Fermi National Accelerator Laboratory, a CentOS board member since April 2020, offered: "It's hard to predict the future."

Indeed, but this question was not really about the future, nor was it about CentOS Stream, though the board members (eventually) chose to answer as if it were.

Er, anybody want to take this one?

CentOS 8.0 Linux, built from the same sources as Red Hat Enterprise Linux (RHEL) 8, was released on 24 September 2019 with a projected EOL of May 2029.

In December 2020, community manager Richard Bowen posted about the "shifting focus" to CentOS Stream, announcing that "CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021."

The EOL for CentOS 7 remains 30 June 2024. This drastically curtailed EOL was disruptive for CentOS users, some of whom were rolling out CentOS 8 in production with the expectation of 10 years of updates. "This is a breach of trust from the already published timeline of CentOS 8 where the EOL was May 2029. One year's notice for such a massive change is unacceptable," said a user at the time of the announcement.

Crossing streams

CentOS Linux was massively popular as it offered a community-supported and compatible build of RHEL and was trusted for production use. The community was happy with it, but Red Hat was not, figuring that it got little benefit from all those CentOS users especially as it knew little about them, as Exelbierd told us last month. Red Hat said it would no longer put engineering effort into CentOS Linux but would shift all its effort to CentOS Stream, which tracks just ahead of RHEL, in effect forcing the CentOS board to go along with the plans.

During the virtual Q&A, board members assured us that they were "very excited" about Stream and did not show any expressions of regret or disappointment, unless via body language. Riehecky said Stream would help him get code into RHEL. "Historically when I've wanted to make some changes in the RHEL release, the plan was I get them into Fedora and hope that they filter down. That's not a great way for something I care about to make it into the release … Stream gives me a place where I can make them happen."

Johnny Hughes, board member and CentOS software engineer, added: "Before we were just building stuff as it came out. Now with the RHEL engineers accepting communications and patch inputs, when that gets turned on, we have capability for the community to interact directly and get things in which not only helps CentOS Stream but helps RHEL and we start operating more like a true community."

We also learned that the integration of Stream into the Red Hat development process is a big deal for Red Hat, that the process is not yet complete, and that CentOS will share more resources with Fedora, Red Hat's leading edge Linux distro.

"We already share infrastructure with Fedora," said Hughes. "We're moving the authentication to shared authentication. The community platform engineering team takes care of both the CentOS hardware and the Fedora hardware. We are moving in that direction. We're going to have a GitLab for CentOS Stream and Bugzilla.redhat.com is going to be the place where you interact with and report bugs. You already use that for Fedora... we're combining a bunch of things."

The actual code repos for Fedora and CentOS will not be brought together. This "is something that has been considered but for a couple of technical and honestly some political reasons Fedora and CentOS have not chosen to combine their repos at this time," said Exelbierd. "That doesn't mean it may not happen sometime in future."

Should I be concerned about the stability of Stream in production? asked a user. The answer is not straightforward. "The packages that go into Stream have gone through the internal QA before they land in Stream. They should be as safe as when they land in RHEL," said Riehecky, adding that this is in part a matter of users contributing tests for things they care about.

Riehecky added: "If you've got something that needs a test, we can add that into the RPM so it is self-tested and into the upstream that it's based on. If you are concerned about testing, I'd encourage you to build tests."

"Red Hat engineers are not allowed to commit code to Stream that is work in progress. It is expected to be finished code," said Exelbierd.

The whole point of Stream is that it is not yet RHEL, it is to become integral to the process of developing RHEL. "I wouldn't expect Stream to go anywhere as it's becoming fundamental to the release of RHEL itself," said Riehecky.

Community spirit

Red Hat knows that RHEL is better suited to production workloads than Stream, which is why it is trying to mitigate the loss of CentOS 8 by offering more options for free RHEL deployment.

Listening to the board speak, it is apparent that the focus on Stream has real benefits for RHEL and for the RHEL community. It is opening up the development of RHEL, making it more transparent and making it easier for users to get fixes and new features into the RHEL code.

The people it is not good for are the existing CentOS users who want a community build of RHEL, which is why there is intense interest in Rocky Linux, with the involvement of Gregory Kurtzer, co-founder of the CentOS project. Another contender is AlmaLinux, formerly Project Lenux, a "1:1 binary compatible fork of RHEL 8 built by the creators of the established CloudLinux OS."

A beta version of AlmaLinux was released on 1 February. "The folk that are doing RHEL 8 rebuilds to keep 8 alive, we wish them the best. We love our open-source friends," said Riehecky.

"The board and Red Hat want Stream to succeed. If we make it terrible, it will not. It's on us and on Red Hat to make it a working solution for people," said Mike McLean, Red Hat lead developer for the Koji build system and CentOS board member.

It may indeed succeed on the terms set for it by Red Hat, but it does not meet the same need as CentOS Linux. That question to the CentOS board about changing the EOL of a released project was really about trust, and it was one that board members chose not to answer. ®