Skip to main content

QA Strategy

Value-driven test automation: what really pays off

Not every test is worth automating. How a points-based scoring matrix helps you choose the right tests – with an example.

Module in digitaler Form

Why not simply automate everything?

"We automate all tests" sounds like a good plan – and often ends in a large, unstable test suite that costs more to maintain than it saves. Every automated test has to be built, maintained and investigated when it raises false alarms. If the user interface changes frequently, this effort grows quickly.

Value-driven test automation turns the question around: not "What can we automate?" but "Where does automation deliver the greatest value relative to the effort?" A simple scoring matrix, in which every test case receives points, provides the answer.

The criteria of the scoring matrix

Each test case is assessed against four criteria, each with 1 to 3 points. 3 points always mean: this favours automation.

  • Frequency: how often is the test run? With every commit or release it scores 3 points, once a year only 1.
  • Risk: how much damage would a failure of exactly this function cause – to revenue, customers or compliance?
  • Stability: how stable is the function? A user interface that changes every week scores 1 point – its test would need constant adjustment.
  • Feasibility: how easy are creation and maintenance? External systems, difficult test data or results that are hard to verify reduce the score.

The points are added up. The result lies between 4 and 12 – the higher the total, the more automation pays off.

In addition, there are knockout criteria that apply regardless of the score: a test that runs only once, or a result that cannot be assessed unambiguously and automatically, stays manual.

What the score means

The total assigns each test case to one of four recommendations:

10 to 12 points

Automate first

Run often, business-critical, stable and easy to implement. This is where automation pays off fastest.

8 to 9 points

Plan deliberately

Worth it, but with a catch – usually the effort, for example because of external systems or test data. Belongs in the next expansion stage.

6 to 7 points

When time allows

Little benefit relative to the effort. Useful once the important tests are in place – or reassess as soon as something changes.

4 to 5 points or knockout criterion

Keep manual

Rare, unstable or hard to verify. Manual or exploratory testing is more economical here.

An example from practice

The assessment of typical test cases for a customer portal shows what this looks like in practice:

Example assessment of a customer portal (1 to 3 points per criterion, 3 = favours automation)
Test caseFrequencyRiskStabilityFeasibilityPointsRecommendation
Login333312Automate first
Price calculation via the API333312Automate first
Search with filters323210Automate first
Order with payment33219Plan deliberately
Registration with email confirmation22318Plan deliberately
Invoice PDF export12317When time allows
New feature, UI still changing22127When time allows, reassess later
One-off data migration13116Keep manual (knockout: one-off)
Design and usability21115Keep manual

The result is clear: login, price calculation and search are automated first – they run constantly, are critical and easy to implement. The order with payment is at least as important but needs the payment provider's test environment and clean test data; it is planned for the next expansion stage. The one-off migration drops out despite its 6 points because of the knockout criterion, and the design stays manual anyway.

Important: the matrix is not an end in itself. It makes the decision traceable and ensures that everyone involved applies the same criteria. If a test case changes – for example because a new feature has become stable – it is reassessed.

Does it pay off? A sample calculation

Whether an automated test pays for itself can be roughly estimated:

Savings = manual effort per run × number of runs − (creation + maintenance)

A manual test that takes 30 minutes and is run 50 times a year costs 25 hours. If automation costs 6 hours to build and 4 hours of maintenance per year, that leaves 15 hours saved – and more with every further year. If the same test only runs twice a year, automation does not pay off.

The greatest value cannot even be measured in hours: automated tests in the pipeline report defects minutes after a change instead of weeks later.

Common mistakes

  • The inverted pyramid: almost everything is tested through the user interface, although API and unit tests would be faster and more stable.
  • Automating too early: functions whose user interface is still changing mainly create maintenance effort.
  • Ignoring false alarms: a test suite that regularly turns red without a real defect quickly loses all trust.
  • The wrong metric: the number of automated test cases says little. What matters is how much risk they cover and how much manual effort they save.
  • Nobody feels responsible: without clear ownership of maintenance and evaluation, tests quickly become outdated.

How we support you

We assess your test cases together with you, prioritise them with the scoring matrix and build automation where it pays off – with tools such as Playwright, Selenium, Robot Framework or Tricentis Tosca, integrated into your pipeline. Regular evaluations show whether automation delivers on its promise.

Frequently asked questions

What does value-driven test automation mean?

Value-driven test automation selects tests based on how much value their automation delivers relative to the effort. Criteria include execution frequency, business risk, stability of the function and feasibility.

How does a scoring matrix for test automation work?

Each test case receives points for several criteria, for example 1 to 3 for frequency, risk, stability and feasibility. The total shows how much automation pays off: the higher, the better. Knockout criteria such as a one-off execution rule out automation regardless of the score.

Which tests should be automated first?

Tests that run frequently, check business-critical functions, are stable and are easy to implement – for example login, core calculations or important interfaces.

Which tests should remain manual?

Rarely executed tests, checks of functions that are still changing significantly, and anything that cannot be assessed unambiguously and automatically, such as design, clarity or exploratory testing.

How do you measure the success of test automation?

Meaningful measures are manual effort saved per release, the number of defects found early by automated tests, the stability of the test suite and maintenance effort – not the mere number of automated test cases.

Industries

Industry expertise

Every industry has its own rules, risks and auditors. We know them from practice – and test so that your software not only works but stands up to every audit.

  • Financial services

    Online banking, core banking systems and regulated processes. We test against high security and compliance requirements – so releases stay predictable, traceable and auditable.

    • BFSG
    • DORA
    • BAIT
  • Public sector

    Specialist applications, citizen portals and SAP landscapes. Accessibility under BITV, clean acceptance testing and documentation that stands up to any audit.

    • BITV 2.0
    • EN 301 549
    • SAP
  • IT & Software

    For software vendors and IT service providers: test strategy, automation in the CI/CD pipeline and quality assurance of AI features – from MVP to release.

    • CI/CD
    • Test automation
    • EU AI Act