How Citrix dropped the ball on Xen ... according to Citrix

How to win friends and admit how you lost them earlier?

Open Source Summit What's the difference between the Citrix Hypervisor and Xen? Well, one has quite a big crowd of upset current and former community members.

One of the more interesting talks at the Open Source Summit was from Jonathan Headland, software development manager at Citrix, with the unusual title "How to Disengage the Open-Source Community: The Citrix Hypervisor Experience." Given all the usual fist-pumping so many companies' marketing teams like to engage in, especially at an event like the Open Source Summit, The Reg FOSS desk was intrigued.

Among other things, these days Citrix offers the Citrix Hypervisor, the product formerly known as XenServer, which it has owned since it acquired XenSource in 2007. The focus of Headland's talk [PDF] was how Citrix mismanaged the relationships between its commercial version of XenServer, the free open source version, and both its upstream and its user community. His opening line was:

How would you feel if you were in a relationship for 15 years and it wasn't going very well?

He went on to carefully itemize the mistakes the company made, and the four lessons he suggests for others to avoid doing the same.

Citrix originally offered XenServer under the "freemium" model: one product was free, the other commercial for enterprise users. Only paying customers received maintenance. The model was successful, and the revenue funded a full-time team of eight engineers and a community manager who worked on the upstream project.

According to Headland, Citrix's first significant misstep was in 2011, when it decided to open source the full feature set of the enterprise product, with revenue to be made from support. The goal, he said, was to get more community buy-in, but the company learned some tough lessons: "We had a very poor understanding of our customers and what they'd actually pay for. Customers happily took the new features, but it turned out that they weren't so keen to pay for the maintenance."

The result was crashes: in revenue, in the reputations of the people who made the decisions, and in that of open source itself within the company.

When you open source your software, there's no way back. You'd better be sure before you do it.

Another problem was that when Citrix gave away the source code to the enterprise product, it didn't provide the accompanying tooling. "Even if we had remembered, or thought, to make all these tools available, we'd still have needed to teach people how to use them."

The result was disappointing Xen enthusiasts, and "rather than increasing contributions, it inhibited them."

Revolutions often result in counter-revolutions, and the pendulum swings back.

In 2017, in "an atmosphere of mistrust of open source projects, and with the feeling that many of its customers were free-riding and not making any contributions," the company announced a change of direction. Product management reintroduced limits, cutting or reducing the features available for free, some to lower than the free product had back in 2010. For example, the number of hosts in a server cluster went from 64 down to just three.

However, this hit one particular sub-community of free users that one engineer Headland interviewed called the "weird systems users": hobbyists who offer virtualization to non-profit and charity users, using old, out-of-maintenance hardware that had been inherited or passed on to them. Enthusiast users, with no funds to buy licenses, but who had been among the most important finders and fixers of bugs. Unable to use the free version any more, they were forced to move to other products … or create them.

You may ask, how can you force a change in an open source product? The answer is: you can't. We got forked.

The result was a whole new product: XCP-ng. "We did regain revenue from our paying customers, the big ones, the enterprises who were never going to make contributions – but we lost the ones who were keeping the project alive."

Headland's talk ended with some confessions, and the four lessons he felt were the most important things for any company selling products based on open source.

He said that Citrix misunderstood and underestimated the breadth and richness of the community, and the number and types of stakeholders in it. That it hadn't identified the most important members, and didn't do enough to support even the ones it did know about. It also never thought about working with the other "commercial peer contributors to Xen: Red Hat, Oracle, SUSE, and Amazon, to name a few."

We didn't mean to cause any harm. We wanted to be good citizens.

His takeaways?

1. Commercial organizations must think about protecting their revenue.
2. Remember to share the tools as well as the code itself.
3. Make sure that any free editions have enough value to entice enthusiasts, because they're the most valuable external contributors.
4. Understand and support the whole community: not just upstream, but downstream and other stakeholders.

It is both fascinating and wonderful to see such openness and honesty from any commercial entity. Even before Citrix changed its name, Xen wasn't the most well-known commercial hypervisor – that's always been VMware, the company that pretty much created the industry. But as the references to Amazon and EC2 hint, Xen has some very big users and is a more important competitor in the space than you might think.

Whether Citrix's candor is going to win it more trust is uncertain, but it's an astonishingly big olive branch, and we applaud it. ®