How test impact analysis works
For the full index of agent-friendly pages, see llms.txt.
Test impact analysis (TIA) answers one question from your AI coding agent: which of my existing mabl tests does this code change affect? This article explains how mabl arrives at that answer. For the step-by-step workflow, see validate code changes with test impact analysis.
From a code change to a description
When you ask your coding agent which mabl tests are impacted by your changes, you don’t need to describe the changes yourself. TIA handles it for you. Your mabl tests never mention the files in your code, so TIA doesn’t look for them. Instead, it translates those code changes into a description written in the same language your mabl tests are written in, looking for related pages, controls, and flows. If the change includes updates to a shared component, TIA will look for every test that interacts with that component.
Each analysis is scoped to one application in mabl. A change that touches more than one application is analyzed once per application and the results are combined.
Finding related tests
mabl searches the content of your tests against that description, matching on both wording and meaning. For example, a test that says “checkout” can match a change described as “purchase flow”. The search reads test names, descriptions, and steps.
Based on this search, mabl returns a prioritized set of tests:
- Each test comes with the reason it was included. when using TIA interactively, outside of CI, you can review why mabl included tests before them
- Each test includes details the skill needs to decide whether to run it: whether it’s enabled, its reliability history, and the run configuration recorded on it, such as environment and credentials.
- The set of tests is not exhaustive. The analysis returns the most relevant tests. When a change matches more than it returns, TIA may suggest running a narrower follow-up analysis.
- The result also names coverage gaps: user-facing areas in the description with no apparent test coverage. These are the candidates the skill offers to hand off to the test authoring skill.
A large set of results generally indicates that your code changes are broad, often because they touch shared code. In this case, TIA organizes and groups the set of tests rather than trimming it to look tidy.
With a knowledge graph
Note: Early access
Knowledge graphs are available in early access, and pricing and packaging are subject to change. We will be rolling them out to customers over the coming months. If you’d like access sooner, reach out to your customer success manager.
The search described above works on any workspace. If your application has a knowledge graph, TIA uses it alongside the search. The knowledge graph maps your application’s pages, workflows, and features, and records which tests cover each one and which steps do the covering. mabl builds and maintains the graph for your application.
With a knowledge graph, TIA can produce higher confidence results:
- Tests ranked by impact - for example, tests that exercise the changed area directly come before tests that only pass through it.
- Explaining each result with the graph - mabl cites the part of your application that connects the test to your change.
Without a graph, the analysis still returns the related tests it finds by searching, but without that ranking.
Deciding which tests to run
The analysis returns candidates for execution. Before running anything, TIA sorts each test into one of three groups:
| Group | Which tests | What TIA does |
|---|---|---|
| Running now | Tests that only read data, or clean up after themselves | Runs one first as a canary to confirm the target, then the rest. |
| Needs your approval | Tests that write shared data, whose cleanup TIA can’t verify, or need a billable feature enabled to run locally | Asks you before running them. |
| Not running | Tests that are disabled, unreliable, or outside the scope you set | Skips them and says why. |
What the analysis can’t tell you
When you read a report generated by TIA, bear in mind the following:
- An analysis that reports no gaps is not proof of coverage. TIA finds gaps by comparing the change against your existing tests, so a brand-new surface has nothing to match. If you want to be sure about a specific area, you can ask it directly.
- API and performance tests may be missing from results. The search is tuned for browser flows described in user terms. If your change touches API or performance behavior, treat those tests as possibly missing from the set rather than as considered and excluded.
- A test can match on its description rather than its steps. Search reads test descriptions and notes as well as steps. A test with a good write-up and no executable steps may surface, though the skill does check the step count and never leads an execution plan with a stub.
- The analysis looks at one application at a time. Tests that cover the same shared UI from a different application are invisible to a single analysis. When a change touches shared UI, ask whether another application covers the same ground before treating the analysis as complete.
- Two analyses of the same change won’t return identical lists. The search matches on meaning as well as wording, so the analysis can vary from run to run. TIA always tells you why a test was included, so use those reasons to judge the returned set of tests rather than comparing it to a previous run.
- TIA finds tests; it doesn’t write them. If you want to cover a testing gap, TIA will offer to hand off to the test authoring skill.