One of the more wickedly astonishing findings from the current DevOps Report is that change review or advisory boards have little effect on a company’s performance. In fact CABs – as they are called – tend to slow down IT’s ability to release software quickly and regularly, negatively affecting organisational performance.
I don’t think many people would say they like or even believe in change review boards – except the architects on them … well, at least some of them hopefully.
Nonetheless, if continued existence is proof faith in a concept, we in the IT industry seem to believe fervently in review boards: I encounter them at almost every organisation I speak to. When IT moved more slowly and we were delighting ourselves with ITIL and other PRINCEs (PRojects IN Controlled Environments) of process, review boards seemed like a good idea. After all, not too long ago we’d just emerged from the switch-over to the “distributed computer” (read: Wintel boxes).
That huge mess of new hardware and software spawned entire clean-up crew industries in systems management, breathing new life into aging mainframe management companies once they had acquired the rascally new-comers like Tivoli. We certainly didn’t want some runaway IT projects built by a bunch of cowboys who’d leave us city-folks behind to clean up the mess. We needed a process to assure our future selves' sanity and which would allow us to get home in time to watch Seinfeld.
In recent times, though, the need to ship software more frequently has created a new set of expectations for IT and, thus has been a driver for innovation in the software release cycle. For many, IT’s goal is now to ship software weekly, if not daily, giving their organisations the capabilities to operate like software companies. So a review board that itself meets monthly to look over a huge pile of changes becomes a massive road-block… and if they don’t seem to be effective, why have them?
There’s no escaping reviewing
Of course, the trick is that “reviewing” is still occurring, but since everyone started following the Toyota Way principle of "lean thinking", the reviewing is now done closer to the actual work. Instead of relying on change review boards, the application teams themselves do peer review with some even going as extreme as doing paired programming. There are many practices and technologies that help accomplish the original goals of those review boards too.
Standardized testing is also done more and more by the actual application team and also has become highly automated. It’s not like these fast-moving DevOps people are just shipping code gleefully, they’re testing and reviewing at almost a nauseating level for old timers who enjoyed throwing the testing tasks over the wall to QA. A recent Gartner study on agile practices in enterprises found that 75 per cent of organisations were doing unit tests and a third had automated acceptance testing. That said, pair programming was only in place 23 per cent of the time: that’s apparently still a weird meal for most to swallow despite the praises its practitioners sing.
To be a bit hand-wavy about it, the way we write and run applications is picking up much of the review board’s work as well. The actual cloud platforms used to run applications are creating much more resilient software that with things like the ability to roll-back problems and isolating poorly behaving services. Meanwhile architectural practices like microservices and 12 factor app principles are describing how to design and write software that’s designed for this resilience and speed of delivery.
So what’s an enterprise architect to do?
The role of the enterprise architect seems to be evolving as work is pushed down to the actual software teams and as staff on those teams become more “balanced” with all the roles needed on the team beyond just developers. There’s a certain kind of architecture needed to sustain independently operating applications teams, and it looks like “architects” are well situated to be those enablers. This, of course, is in subtle but important contrast to being the change review “approvers”.
This all reminds me of an old anecdote from the lean manufacturing world. As one US car factory that was going lean, trying to “catch up” with the Japanese, one of the senior presidents observed that the factory engineers were always very busy in their offices, doing some sort of work. “I do not think the problems are in that office,” he told the factory GM, “I think they are on the factory floor.”
The implication for us in IT, of course, being that problems are not solved, nor product created in change review board meetings, but by the teams who are creating and struggling with the software every day. Findings from studies like the DevOps report are now showing this, and it’s large companies that seem to suffer the most.
When that same study sliced up the findings by company size, it found that organisations with more than 10,000 people were 40 per cent less likely to be high performers than 500-people outfits. There are many other factors causing that friction; if you’re one of those large organisations, it’s worth revisiting CABs. ®