I’ve just been representing Reg Developer as one of only three journalists at an interesting roundtable discussion devoted to “Software Quality, Best Practice and Governance” (which seems to mean Testing, in the widest sense).
Among those present were Sarah Saltzman of Compuware, the sponsor of the roundtable event; Teresa Jones of Butler Group; Brian Wells, chair of something called the TMMi foundation (of which more later); Geoff Thompson, of Experimentus; and, representing the businesses using this stuff, Philip Griffiths, Global Applications Architect at Heinz. Most noticeable, by the way, was Sarah’s reluctance to talk about Compuware’s products too much – well done!
The discussion started off conventionally – but got more controversial later. The importance of early defect removal was stressed and the need for business resources to be made available, so that testers could relate the evolving system to the needs of the business as well as (or instead of) what the IT group said it was going to deliver. Clearly, it helps if the testers contribute to the project from inception; rather than being brought in at the end, as a "barrier to going live".
But I've heard all this before, at various times over the last 30 years or so. We even touched on the fact that if your project is behind schedule and testing is being squeezed, shipping in "testing mercenaries" on short term contracts probably puts you further behind schedule (someone on the team has to stop what they're doing and get these people up to speed). But this is one of Fred Brook's insights from 30 years or so ago. Why is it still relevant? And, since we really do know a lot about managing software quality why do we still accept bugs in software that is only marginally fit for purpose, sometimes just because it is popular (the Microsoft syndrome)?
And here’s where the discussion turned interesting. You won’t get blamed for buying popular (or well-marketed) software even if it isn't high quality, and bugs in features you don’t use don’t matter much, so perhaps we then come to accept bugs in features that do (after all, if popular software contains bugs, perhaps they don’t matter). Then, sometimes the problems are not in the software but the spec it was written to, so you can’t safely blame anyone (“the user is always right”), even though using software which automates the wrong process is seriously expensive and disruptive.
Once again, it’s the people/culture/change issues that bite. Phillip Griffiths pointed out that getting the resources for managing quality out of a management often preoccupied with short-term stock market performance isn’t easy (in the long term, quality is free, but only if you are prepared to invest in it up front in the shorter term). And people often have a vested interest in the status quo.
“Hero management” – by people who get a lot of fun (and, probably overtime and bonuses) out of waiting for things to go wrong and then fire-fighting with flair and enthusiasm - is rife. And it’s a very expensive approach to managing quality. It was even noted that a profitable testing market is evolving – full of people who’ve done a course on testing but who have little experience of “defect risk management” and who "don't know what they don't know" – but are paid high rates to help with last minute quality panics. And, often a system will go live even though people at the sharp end know it can’t work, because no-one is brave enough to tell management the bad news; and people believe that they can manage their way out of the mess anyway – hero management again.
Griffiths sees a need for a business model fundamentally based on “fitness for purpose” quality. The CEO must define a way of working that is process-oriented, not technology-focussed and must support the quality champions in middle management actually managing change. And s/he must allocate business resources to testing and analysis.
It all comes down to organisational maturity and process improvement, it seems to me. Without this, you can even “get it right” in response (usually) to some crisis – and then slip back into old ways. Geoff Thompson described a firm which had done just that. In a situation of extreme pressure with no time for rework, quality was built in from the start and the developers/testers were given the time they needed – on condition that that was all they got – and, on delivery, the product "just worked" without any problems. But now that things are less stressed, techniques like “pair testing” (with the testers embedded in development alongside programmers), that helped achieve this, are less popular.
Which is where TMMi comes in. It’s based on the TMM (Test Maturity Model) process model, from around 2003, which adds testing to the Capability Maturity Management model (CMM; now Capability Maturity Management Integration, CMMI); which doesn’t go into much detail on actually achieving demonstrable quality. According to Brian Wells, testing is what demonstrates Quality and it covers all forms of defect removal
Now, the TMMi foundation is taking the text book TMM model and "refactoring" it (simplifying it , adding stuff, moving stuff around - training is moving from level 3 to level 2, for example). The aim of this independent foundation is to produce a "generic but detailed enough" public domain reference model for the testing process – which (amongst other things) should help developers choose tools without the fear that their tool choice is being driven by a vendor agenda embodied in a proprietary process.
The TMMi model is non-prescriptive – it won’t mandate an independent test team, for example, but suggest that there is a need for independence in at least part of the testing process – and it is research driven, not commercial. The first draft of the new TMMi model should be available for review in 1Q 2007; delivery of version 1 is expected at the end of 2007 and I hope to look at it in more detail in Reg Developer on due course.
And Compuware? Well, it is looking at mapping TMMI onto the requirements-driven “risk based testing” process models it is developing for itself (see, for example, the Risks Are For The Weekend Executive Guide, at the bottom of the page, registration needed). That is something else we may look at in Reg Developer – and perhaps we’ll look at some of Sarah’s testing and requirements management tools too.
David Norfolk is the author of IT Governance, published by Thorogood. More details here.