Is anything more important than stopping defects from escaping into live operations and disrupting users? In almost half of organisations, it seems that reducing the costs of development, testing and bug-fixing are more important. This insight came from a 2023 survey Acutest commissioned from IDC/ Foundry. Of the 109 organisations surveyed across various business sectors and sizes, almost half committed less than 20 per cent of their development budgets to testing.
In our previous article (Housing Technology, March 2024), we discussed some approaches to take to avoid joining the 93 per cent of organisations that release severe defects into live operations. These include:
Effective requirement and design reviews;
Leveraging generative AI for streamlined requirement engineering;
Enhancing non-functional coverage to avoid performance issues and downtime.
In this second article of our three-part series, we emphasise the importance of prioritising assurance and testing within organisational change programmes. Our insights draw from real-world experiences of what drives up the costs of testing and how to rein in costs without compromising on quality.
- Almost half of organisations commit less than 20 per cent of their development budgets to testing.
- 47 per cent of organisations consider reducing the costs of development, testing and bug-fixing as the most important aspect of testing.
Problems cost more to fix the later they are found
Our previous article highlighted that over 60 per cent of defects identified during testing stem from issues introduced during requirement specification and solution design. Early reviews find defects almost as soon as they are introduced, significantly reducing the cost of fixing them. According to ‘Software Engineering: A Practitioner’s Guide’ (auth. R. S. Pressman), preventing defects at this stage is 15 times more cost-effective than addressing them during testing. Waiting until live operations to uncover a defect is even more expensive (60-100 times more expensive, according to Pressman).
Early reviews not only enhance documentation quality but also yield substantial cost-savings in project and test delivery.
Drive out duplication
If a registered engineer serviced the boiler in a property, you wouldn’t reinspect the flue and disassemble the boiler to check the internal components for wear yourself. However, this is what organisations do when they test IT solutions. Running identical tests through the same workflows to obtain the same results consumes expensive QA resources and doesn’t surface defects missed in earlier phases of testing.
For one client, Acutest reviewed a test suite of 400 tests and found more than 50 per cent were essentially duplicates. Though each test used different data, the functional behaviour remained identical. If one test passed, the duplicate tests would inevitably pass as well.
Duplicated tests consume time and cost money throughout development and release cycles. Writing then repeatedly running duplicate tests through multiple cycles and analysing the results all contribute to the cost. When multiple functionally-identical tests fail, the resulting alarm leads to further resource wastage in fixing what may be a minor issue.
Look for duplication when reviewing the requirement and design – collaborate with business users, suppliers and testers to develop ‘user stories’. These stories not only avoid any confusion between suppliers and customers but also reveal duplicate functional behaviours that only require testing once.
Don’t waste time, save cost
To reduce test costs while preventing serious problems from reaching live operations, organisations must improve their test efficiency:
- Write fewer, better tests – perversely, large numbers of tests lead to more duplicates and higher costs. Our cross-client analysis has found that doubling the number of tests triples the duplication rate. Don’t measure test-team productivity by the numbers of tests they write because spending time writing more tests doesn’t guarantee better outcomes or coverage.
- Don’t write tests too early – in Acutest’s experience, if a test isn’t executed within three months of development, it’s unlikely to be run at all due to the world changing by the time execution arrives. Our analysis of clients’ projects shows that for test suites with over 1,000 tests, typically 30 per cent of those tests are never run. Writing tests just before they are needed minimises the risk of obsolescence and ensures their relevance.
- Monitor test-team efficiency – track key metrics such as unexecuted tests, time from test scripting to execution, test reuse and earned confidence. These indicators will alert you to inefficient and money-wasting projects delivering the same level of quality that others may be delivering for less.
Know what is good enough
Achieving ‘100 per cent of tests completed and passed’ is often a key milestone for deployment and just as frequently a source of unnecessary cost. Executing the final 10 per cent of tests may well take as much effort as the preceding 30 per cent and not significantly boost confidence. For example, in one of Acutest’s reviews, the stakeholders agreed that passing 104 of the 258 tests originally planned would provide sufficient confidence to go live.
Our analysis shows that in test reviews, the key stakeholders often conclude that many tests either couldn’t fail or the impact of failure would be so low that it didn’t justify the expense of running them. Implementing the right risk-based approach means that decisions on going live can be made earlier and with more confidence, even before all testing has been completed. This saves considerable cost without significantly compromising the delivered solution’s quality.
Simple changes give big returns
The survey highlighted how important the cost of assurance is to many organisations, often more so than quality, especially where there is such a small percentage of overall budget available. Significant cost-savings can be achieved by internal delivery and test teams making simple but important changes to the way they work.
- Fail fast by removing defects during the requirement and design phases rather than when they become much more expensive to fix.
- Avoid incurring the cost of writing and running duplicate tests or tests that will never be run.
- Focus on executing tests that will build confidence and enable decisions about going live to be taken earlier.
By following these principles, you will never have to pay someone to run that type of test again.
Kristen Hansen is the principal consultant at Acutest (part of Trustmarque).
