Software engineering performance monitoring: Do your metrics measure up in the modern DevOps age?

The days of counting LoC are long gone, thankfully, so what now?

Sponsored How do you measure software engineering productivity and performance? It seems a fair question, but – it turns out – a slippery one to answer.

It is easy to find fault with the most obvious approaches. Lines of code written, or keystrokes recorded? This is hopeless, because it is not what anyone cares about. Shorter code is often better, and code written to solve an intricate problem cannot be compared to more everyday programming.

A good developer may spend hours reducing the amount of code in a project, perhaps by devising more efficient algorithms, or perhaps by application of the DRY (Don’t Repeat Yourself) principle, fixing someone’s use of clipboard inheritance by refactoring repeated code.

What then can be usefully measured? A 2017 report claimed that “only four key metrics differentiate between low, medium and high performers.” These are:

• Lead time: how long does it take for an item of work to get from first start to production deployment.
• Deployment frequency: how often is the team able to deploy an update? Smaller and more frequent updates are preferable to fewer and larger updates.
• Mean time to restore: if a problem is discovered, how long does it take to revert to a fault-free deployment?
• Change fail percentage: how often does a change result in a defect? Some teams measure the defect escape rate how many defects are discovered in testing versus how many make it through to production.

Another interesting metric is code complexity. Cyclomatic complexity can be measured and refers to the number of different paths of execution in the code. High complexity may indicate a greater likelihood of defects and lower maintainability.

Come again?

Why are we even discussing this? Every so often, the higher ups come to us with some new, business-school-driven concept of “measuring” productivity or the output of software engineering. Schools of thinking come and go, but the direction of travel is consistent: to quantify our performance and productivity in some unimpeachable way. But, rather than dismisses measurement outright it’s worth taking the time to consider “what are” the best ways to measure productivity. Indeed, is there an approach that can be taken – or that you can propose – that will finally have the higher-ups off your back and that’s able to keep everybody satisfied?

DevOps means that, more than ever, there’s a great advantage in the gathering and analyzing of metrics. The advantage is that it enables “us” – depending on who the “us” is, of course – to set goals and to measure whether we hit them and to quantify any necessary improvements. DevOps tooling can assist with this by tracking and logging issues and statistics at each point in the CI/CD pipeline.

The important thing to remember is that creating software is not just about code. In 2001 the Agile movement was born, beginning with the Agile Manifesto, a short set of principles to guide development of software. Agile is perhaps best known for the fourth statement in the manifesto: “Responding to change over following a plan,” which favors iterative development rather than planning every single detail up front as you might with waterfall. The second principle, though: “Customer collaboration over contract negotiation,” may be even more important. The big insight here is that the development “team” is not just the developers, but – rather – all the stakeholders, everyone who cares about the outcome. Successful teams, therefore, are those who engage and respond to every member. That includes the end users.

Both technology and software methodology have evolved since the early days of Agile, but these core concepts have been largely accepted. It is not so much the productivity of developers that counts, but the productivity of the software development and delivery process of which they are part. This is important because you can now propose to the higher ups a system of management based on a bigger set of goals rather than putting individuals under a microscope.

Of course the code has to be written, and any high-performing team needs its fair share of excellent programmers, but from a business perspective it is the productivity of the whole DevOps process that counts. It’s also true, that the best developers cannot be productive on their own, but only as part of a functional and effective process that ensures their code is solving the right problems, and that fixes and improvements can travel rapidly from code editor to deployed application.

Foundations of measurement

A decent DevOps infrastructure and process are therefore important – but what are their key characteristics and how do they count towards measuring the “team’s” productivity?

One feature of a strong DevOps system is it does not constrain developers to a specific way of working but integrates well with a variety of tools and approaches. Constraining workers to the corporate way of working rather than their own way is an automatic fail in terms of productivity. So, for example, your choice of code repository should not dictate the CI/CD (Continuous Integration/Continuous Delivery) platform you use, but – rather – your CI/CD platform should integrate well with your preferred code repository. Similarly, the choice of language, framework and runtime should be made on the basis of the best fit for the project and developer skills, not determined by the CI/CD system.

Another boost for productivity is giving developers the ability to test their work locally before committing code. The advent of containers makes this easier, particularly for developers working on multiple projects who may otherwise struggle to create a local environment that supports all of them effectively.

A modern approach to DevOps will, therefore, be container-aware and is able to include definitions as part of a project. This makes sense given that this is critical to the successful execution of the code. This should mean an end to the old problem of “it works on my machine” in response to bug reports.

In a sense, DevOps is about orchestration of all the events and facets of the code–test–build–test–deploy cycle. Flexibility is key, here. Take “test”, for example – this may be applied to from static code analysis, unit tests, load and performance tests, compatibility tests for the API presented by a microservice... and then, of course, there are security tests. A robust DevOps process should therefore allow for all these different kinds of tests to be plugged in at the appropriate point in the pipeline in the interest of productivity.

Another important aspect is visibility. A DevOps process that’s optimized for measuring productivity should include a dashboard that shows such things as the status of builds, items that are succeeding/failing, and that – overall – helps to identify small problems before they become really big problems. The kinds of problems that could be considered project show-stoppers and that will weight against teams when it comes to measuring productivity. That said, it’s important to note that passive visibility is not enough. You must think about, for example, how will administrators be notified in the event of a breakage? How easy it is to rollback quickly to a working application? These may be small individually but collectively may combine to trip up overall delivery of a project in a way that will, once more, go against the team’s productivity score.

Something else that can help when it comes to measuring productivity is the idea of selective and canary – limited rollout for sanity checking – deployments. Deployment to a limited region or to a subset of users has value not only for quality testing, but also for gathering data on user acceptance and whether the trial deployment is more or less effective than what it replaces. Including these in a projects metrics can be good practice.

Working as one

If productivity is measured at the broader level of the engineering team’s output and that team depends on a collaborative process, it is important to use tools that enable and encourage collaboration. That might sound like a blindingly simple statement but a long-standing issue with software development has been that its technical aspects make it hard for less-technical stakeholders to understand and track what is happening.

It is therefore great to have issue tracking that is clear and accessible enough for all project members to follow. More than that though, why not integrate with easy-to-use collaboration tools that are familiar to all?

Collaboration is more than tools, though – it’s about attitude, too. What are the characteristics of the most productive developers? The ability to listen and respond to feedback is high on the list. It is all about good communication, something that the best DevOps tools can enable and encourage, but which unfortunately cannot be enforced.

The bottom line? Yes, you can measure the performance and productivity of software engineering but this is a rather pointless exercise. Greater value comes from systems that measure development, operations, security, and collaboration across the entire software process. At the end of that day, that comes down to the efficiency of your DevOps processes.

Sponsored by CircleCI.