This article is more than 1 year old

DevOps still 'rarely done well at scale' concludes report after a decade of research

Puppet Field CTO Nigel Kersten speaks to the Reg about 10 years of studying DevOps adoption

Interview On the 10th anniversary of its first survey, the multi-vendor State of Devops report concludes that DevOps is still "rarely done well at scale," and that "almost everyone is using the cloud, but most people are using it poorly."

The report is opinionated and there is no vendor pitch, perhaps because it is sponsored by a variety of DevOps companies including CircleCI, Puppet, bmc, New Relic, Snyk and Splunk. The first survey was in 2011 so this the eleventh report but a decade after the first.

"We don't want to produce something lightweight, there's a lot of crappy surveys out there, that I don't think are particularly robust in their methodology or insightful in their findings," Puppet's Field CTO Nigel Kersten, who has been involved with the State of DevOps Report from the beginning, told us.

The survey was completed by 2,657 individual DevOps professionals across the world, with around 75 per cent from the USA, Canada or Europe, and covering a wide range of organisation sizes.

The general approach is to divide organizations into those that are highly evolved, mid-level performers, or low performers. "Highly evolved" means frequent deployments, very short lead time for application changes ("less than an hour"), quick response to failures ("less than an hour"), and a change failure rate of less than 5 per cent. "Low" is the other extreme, with a lead time for changes of "between a week and six months." According to the report, "the vast majority of organizations are stuck in the middle."

This division informs the rest of the analysis, which compares the responses of high, mid, and low performers on questions like "Members of my team have clear roles, plans and goals for their work." 89 per cent of high performers agreed with this, compared to just 46 per cent of low performers.

Have some standards!

Another striking example: 57 per cent of high performers felt their IT systems conformed with all five of the US NIST (National Institute of Standards and Technology) essential characteristics of cloud computing, these being "on-demand self-service, broad network access, resource pooling, rapid elasticity or expansion, and measured service," whereas just 5 per cent of low performers could tick these boxes.

"Almost everyone is using the cloud, but most people are using it poorly," the report says. Asked to expand on why, Kersten told us that "most people are not taking advantage of those things, and essentially lifted and shifted their existing processes and practices into the cloud."

Has the way businesses do DevOps improved in 10 years? "It's definitely improved," said Kersten; yet reading the report one gets the sense that progress is slow. Over the past four years, the percentage of high performers has gone up from 10% to 18%, the survey said, and low performers down from 11% per cent to 4 per cent ; yet most remain in the middle group (78 per cent or 79 per cent).

Kersten distinguishes between "start-ups and tech companies" which "are not really talking about DevOps, they're just doing it," and "the enterprise, the mass market of IT" where "DevOps has become really successful at the team level, but because of the interactions between adjacent teams and departments, and the weight of process built up sometimes over centuries, the interaction between teams becomes critical, and that's where folk are really failing. It's not really a DevOps problem."

It is a familiar story for those who have studied software development and delivery: the methodology counts for less than the interaction between people. "Well-run companies set clear goals, we know which team does what, people communicate, they have autonomy. They tend to do really well at DevOps."

The report reads more like a paper on what it takes to do DevOps (or perhaps to run a business) well, than a typical survey, which is mostly a good thing.

Another point that comes out of the report is the importance of what it calls self-service internal platforms. "What we mean is not just centralizing IT," said Kersten, "but moving away from the anti-pattern of lots of small cross-functional teams all doing their own thing … developers shouldn't each be reinventing the wheel on how do I provision a virtual machine or create a testing environment of a CI/CD pipeline."

Having internal platforms that do this is a good thing he said – provided they really are self-service. "Where we see people fail is where they still use what we call ticket ops, where to do this thing I need to describe what I need and then someone will give me almost what I asked for."

A team for all functions

A cross-functional team is one that spans the whole application lifecycle from code to deployment, as opposed to a more specialist team that might only be concerned with database administration, for example. Are cross-functional teams a good thing?

"It depends," Kersten said. "There are underlying strata of technology that are better off centralized, particularly if you've got regulatory burdens, but that doesn't mean you shouldn't have cross-functional teams … too far in either direction is definitely terrible. The biggest problem we see is if there isn't a culture of sharing practices amongst each other."

One thing to avoid, said Kersten, is a DevOps team. "I think we've broken the term DevOps team inside organisations," he told us. "I think it has passed beyond useful … calling your folk DevOps engineers or cloud engineers, these sorts of imprecise titles are not particularly useful, and DevOps is particularly broken."

Automation is critical to "fast flow" teams

Automation is critical to "fast flow" teams

What if an organization reads the report and realises that it is not good at public cloud and not effective at DevOps, what should it do? "First optimize for the team," said Kersten.

"How much time you're spending in manual work doing toil, and focus on automating those things away. Then start optimizing processes across teams, and this is where middle management have to step up and acknowledge that this is their role … let's try and optimize the whole system rather than just each individual team doing the best they can. We don't operate as islands inside these large organisations."

That said, Kersten highlights one area which he described as low hanging fruit. "We've seen that legacy environments are a drag upon everyone's evolutionary journey. Managerial focus on actually optimizing your legacy environments and not just thinking of them as a sunk cost that you've got to keep paying the bill for, is one of the most impactful things you can do."

A section in the report on "overcoming the burden of legacy" has more detail on this, saying "invest in your legacy environments so they are no longer a constant inhibitor of progress."

The full report can be found here

More about

More about

More about


Send us news

Other stories you might like