QA Strategy
Value-driven test automation: what really pays off
Not every test is worth automating. How a points-based scoring matrix helps you choose the right tests – with an example.
Why not simply automate everything?
"We automate all tests" sounds like a good plan – and often ends in a large, unstable test suite that costs more to maintain than it saves. Every automated test has to be built, maintained and investigated when it raises false alarms. If the user interface changes frequently, this effort grows quickly.
Value-driven test automation turns the question around: not "What can we automate?" but "Where does automation deliver the greatest value relative to the effort?" A simple scoring matrix, in which every test case receives points, provides the answer.
The criteria of the scoring matrix
Each test case is assessed against four criteria, each with 1 to 3 points. 3 points always mean: this favours automation.
- Frequency: how often is the test run? With every commit or release it scores 3 points, once a year only 1.
- Risk: how much damage would a failure of exactly this function cause – to revenue, customers or compliance?
- Stability: how stable is the function? A user interface that changes every week scores 1 point – its test would need constant adjustment.
- Feasibility: how easy are creation and maintenance? External systems, difficult test data or results that are hard to verify reduce the score.
The points are added up. The result lies between 4 and 12 – the higher the total, the more automation pays off.
In addition, there are knockout criteria that apply regardless of the score: a test that runs only once, or a result that cannot be assessed unambiguously and automatically, stays manual.
What the score means
The total assigns each test case to one of four recommendations:
10 to 12 points
Automate first
Run often, business-critical, stable and easy to implement. This is where automation pays off fastest.
8 to 9 points
Plan deliberately
Worth it, but with a catch – usually the effort, for example because of external systems or test data. Belongs in the next expansion stage.
6 to 7 points
When time allows
Little benefit relative to the effort. Useful once the important tests are in place – or reassess as soon as something changes.
4 to 5 points or knockout criterion
Keep manual
Rare, unstable or hard to verify. Manual or exploratory testing is more economical here.
An example from practice
The assessment of typical test cases for a customer portal shows what this looks like in practice:
| Test case | Frequency | Risk | Stability | Feasibility | Points | Recommendation |
|---|---|---|---|---|---|---|
| Login | 3 | 3 | 3 | 3 | 12 | Automate first |
| Price calculation via the API | 3 | 3 | 3 | 3 | 12 | Automate first |
| Search with filters | 3 | 2 | 3 | 2 | 10 | Automate first |
| Order with payment | 3 | 3 | 2 | 1 | 9 | Plan deliberately |
| Registration with email confirmation | 2 | 2 | 3 | 1 | 8 | Plan deliberately |
| Invoice PDF export | 1 | 2 | 3 | 1 | 7 | When time allows |
| New feature, UI still changing | 2 | 2 | 1 | 2 | 7 | When time allows, reassess later |
| One-off data migration | 1 | 3 | 1 | 1 | 6 | Keep manual (knockout: one-off) |
| Design and usability | 2 | 1 | 1 | 1 | 5 | Keep manual |
The result is clear: login, price calculation and search are automated first – they run constantly, are critical and easy to implement. The order with payment is at least as important but needs the payment provider's test environment and clean test data; it is planned for the next expansion stage. The one-off migration drops out despite its 6 points because of the knockout criterion, and the design stays manual anyway.
Important: the matrix is not an end in itself. It makes the decision traceable and ensures that everyone involved applies the same criteria. If a test case changes – for example because a new feature has become stable – it is reassessed.
Does it pay off? A sample calculation
Whether an automated test pays for itself can be roughly estimated:
Savings = manual effort per run × number of runs − (creation + maintenance)
A manual test that takes 30 minutes and is run 50 times a year costs 25 hours. If automation costs 6 hours to build and 4 hours of maintenance per year, that leaves 15 hours saved – and more with every further year. If the same test only runs twice a year, automation does not pay off.
The greatest value cannot even be measured in hours: automated tests in the pipeline report defects minutes after a change instead of weeks later.
Common mistakes
- The inverted pyramid: almost everything is tested through the user interface, although API and unit tests would be faster and more stable.
- Automating too early: functions whose user interface is still changing mainly create maintenance effort.
- Ignoring false alarms: a test suite that regularly turns red without a real defect quickly loses all trust.
- The wrong metric: the number of automated test cases says little. What matters is how much risk they cover and how much manual effort they save.
- Nobody feels responsible: without clear ownership of maintenance and evaluation, tests quickly become outdated.
How we support you
We assess your test cases together with you, prioritise them with the scoring matrix and build automation where it pays off – with tools such as Playwright, Selenium, Robot Framework or Tricentis Tosca, integrated into your pipeline. Regular evaluations show whether automation delivers on its promise.
Frequently asked questions
What does value-driven test automation mean?
Value-driven test automation selects tests based on how much value their automation delivers relative to the effort. Criteria include execution frequency, business risk, stability of the function and feasibility.
How does a scoring matrix for test automation work?
Each test case receives points for several criteria, for example 1 to 3 for frequency, risk, stability and feasibility. The total shows how much automation pays off: the higher, the better. Knockout criteria such as a one-off execution rule out automation regardless of the score.
Which tests should be automated first?
Tests that run frequently, check business-critical functions, are stable and are easy to implement – for example login, core calculations or important interfaces.
Which tests should remain manual?
Rarely executed tests, checks of functions that are still changing significantly, and anything that cannot be assessed unambiguously and automatically, such as design, clarity or exploratory testing.
How do you measure the success of test automation?
Meaningful measures are manual effort saved per release, the number of defects found early by automated tests, the stability of the test suite and maintenance effort – not the mere number of automated test cases.