Putting security into DevOps is tougher than it looks

Cultural issues are the first thing to overcome, says Red Hat

Paid Feature If software development has absorbed a single lesson in the last two decades it’s that there’s an urgent need to integrate security at an early stage rather than leaving flaws to rot dangerously inside compiled code. Optimistically, dubbed shift left, the trick has been working out what this means in an era undergoing an historic transformation of development models.

In monolithic, linear development, implementing shift left was about adding a security checking stage earlier in the coding lifecycle. This was never easy and, the complaint went, slowed everything down, but the pileup of vulnerability disasters told the industry something had to change. One case study was Microsoft’s then novel Security Development Lifecycle (SDL) of the early 2000s.

Today, however, coding is increasingly defined by cloud native applications, agile development, and infrastructure as code (IaC), coordinated using continuous deployment platforms such as Kubernetes. In this world of high-velocity DevOps, the idea of early intervention is stretched to breaking point. Code is created, tested, and deployed at incredible speed.

Suddenly, vulnerability management becomes challenging in an environment focused on rapid turnaround and collaboration across teams of developers, each working on their own sub-pipeline. Even with centralized security testers on hand, DevOps can look like a new world where the need for speed and the demand for security have become opposing forces.

DevSecOps – adding the security function to development operations – was designed to fix this even if what it means in practice has evolved over time. Armed with frameworks such as common weakness enumeration (CWE) and automated testing tools, DevSecOps is about integrating security as a primary development objective from the start. In a sense, there is no disruptive intervention in the pipeline, because security is embedded in every process.

The bigger question is how DevSecOps meshes with DevOps in environments such as Kubernetes, whether development teams understand how to make it work, and what cultural and economic barriers might stand in its way. Without answers to these questions, there is a danger that DevOps will repeat the mistakes of the pre-agile era with its confetti of vulnerable code, zero days, and public exploits.

Decentralized security

Providing context, Red Hat recently questioned 500 developers and IT professionals on their views and experience of security in Kubernetes environments, with 55 percent confirming they had delayed an application because of security concerns.

Almost all, 94 percent, had experienced at least one security incident in the previous 12 months, with 59 percent mentioning misconfiguration, and 32 percent security incident at runtime. Some of this might be expected except that 32 percent rated the security vulnerability as ‘major’ with 20 percent saying it resulted in a failed audit.

A central finding was that Kubernetes security is now spread across different development roles, with DevOps the largest at 27 percent, security Ops second on 21 percent and DevSecOps third on 18 percent. The positive news was that 25 percent of respondents said their DevSecOps processes were now at an advanced stage with another 47 percent saying this had started within their organization.

Each group brings with it a different background of experience, distinct skills, and a strong view of why they get out of bed each morning. Some see security as fundamental while others see it is merely an element of their job. How collaboration happens, and whether it happens, depends on their world view and that of their peers and employers.

No more silos

“Developers are not security experts. They don’t think of security naturally,” says Lucy Huh Kerner, Red Hat’s Director of Security Global Strategy and Evangelism. “Security is often looked at as an inhibitor by them and not something that’s going to make a company money. It’s about doing eleventh-hour pen testing.”

You need to have a tight collaboration between developers, operations, and security teams because if you don’t you end up with everyone using their own tools to different ends

So, what are the fundamental principles of DevSecOps and how can organizations know whether they are meeting them?

“Underneath the covers, there is a lot you have to do to be successful at DevSecOps,” says Kerner. “You need to have a tight collaboration between developers, operations, and security teams because if you don’t you end up with everyone using their own tools to different ends.”

She likens the arrival of cloud-native infrastructure and environments such as Kubernetes to an exciting new world in which exuberance about the possibilities has often left security as a secondary concern. Much of this was structural and unavoidable, the fact that security is traditionally the job of a centralized team, rather than something everyone had to worry about.

Ideally, in the decentralized world implied by Kubernetes + DevOps, security should become everyone’s problem. But the forces working against this are formidable. Development teams are made up of people working to different objectives. Embracing DevSecOps has not turned out to be as easy a win as it looked on paper.

According to Kerner, collaboration is the first battle worth having because without it security and DevSecOps is doomed. “There is a massive cultural aspect to this. If your organization is siloed you’re not going to make much progress. This might even come down to the different tools each silo is used to.”

How does an organization assess whether its departments and functions have become silos? Most obviously, if people in one department – infrastructure and IT security – never or rarely talk to the coders and testers in another. It’s the same problem that exists in DevOps but with the added twist of security to create an atmosphere of jeopardy.

An example of how communication can go wrong is where an application vulnerability ends up being exploited in a live application. The IT security people have the problem of cleaning up something whose effects are communicated to the coders long after the fact. But this is way too late, a case of cure before prevention.

Kerner’s view is that breaking down silos usually requires direct intervention by management. “It starts at the top with the CIO, CTO or CISO. They must make sure that the teams under them are forced to work together. They must force the issue even to the extent of creating roles that cross the boundaries between security and development,” she says.


DevOps is built around the idea of automation, the core principle that drives the ‘robot sysadmin’ of Kubernetes and the whole IaC philosophy that eschews manual processes. The task for DevSecOps is to integrate security automation and testing into a workflow where code iterations might happen dozens of times each day. This means embedding controls and the tools used to impose and manage them so that every developer can access them. This is also why decentralization is essential because without it the development pipeline goes back to the hierarchical model of a security team doing all the work, itself nigh impossible when changes are being committed frequently.

DevSecOps must, logically, embed automation at its core. “Fundamentally, DevSecOps, the cloud and containerization, is about automation,” argues Kerner. But the infrastructure required for testing can be complex, requiring a plethora of tools for versioning and defect tracking, many of which will be unfamiliar to DevOps people.

“Take things in baby steps. You’re not going to wake up one day and find that you’ve become a DevSecOps organization. This is about getting used to having processes like testing automated.”

Continuous security, compliance

Another DevOps struggle is how compliance checks are carried out. In some organizations this is still seen as a distinct process reserved for a later stage in the pipeline. “Security teams can stop your application from going to production. In some organizations, the security team has a large say, for example the public sector or financial sector, where they must comply with regulations,” says Kerner.

The DevSecOps approach is that this should be conducted on an ongoing basis by security teams or individual developers. With continuous compliance in place, DevOps should experience few or no major patching interruptions. Compliance-as-code is the essential final element of DevSecOps because it sets down the rules used by automated scanners to detect code breaching an organization’s parameters before it is committed.

Kerner is certain that the industry won’t regress. Cloud-native application development is here to stay, and Kubernetes has grown for the simple reason that it makes the provisioning element of cloud development a lot easier. But organizations and people don’t embrace the future at the same rate. The challenge is that the price of not changing is starting to mount.

“Not all organizations think there is a problem. Many don’t find out for years that they have been breached. Or security researchers tell them. There is no feedback loop.”

And the feedback loop of the public telling an organization its applications have leaked when the damage is done is unsustainable. The mistake would be to think it’s someone else’s problem or that developers will magically solve it on their own. In a world increasingly built and defined by software processes, teams and skills must be integrated using the same principles and technologies.

The industry builds software and software must now rebuild the industry, says Kerner. “Every organization’s software development lifecycle looks slightly different but having a successful SDL requires an organization to sit down and think about what this should look like.”

This article is sponsored by Red Hat.

Biting the hand that feeds IT © 1998–2021