
Missing data is a failure, never a pass
28 October 2026 · AI engineering · Testing · Agentic workflows
3 rules written first5 verdict words1 real bug caught5 to 2 days per feature
1. The problem
For four years I owned one area of a workforce product: the web app, the backend service, the mobile app and the end-to-end tests. Almost every feature in that area touched permissions. A request can be made by an admin or by a normal user. It can be for yourself or for someone else. It can need an approval step or not. Multiply those, and every release has a matrix of cases that all have to behave.
Before each release I clicked through that matrix by hand. It took days. It was also where mistakes slipped through, because by the twentieth case you stop reading carefully. You see the warning you expect to see.
More unit tests were not the answer. We already had good coverage at that layer, with Vitest, Testing Library and a mock service worker. The cases I was missing only showed up when the real interface talked to the real backend. A test in jsdom cannot see the real address bar, the real browser history, or a real session with real permission entries. Only a browser can.
More end-to-end tests in the normal suite were not the answer either. A normal test reads the records it needs from a fixed list of ids. That list goes stale every time the sandbox data changes, and then the test fails for a reason that has nothing to do with the feature. I wanted a check that looks for the data it needs at run time, and that says so when nothing matches.
A manual test can see all of that, but it is slow and it leaves no record. I wanted something that sees what a person sees and writes down what it saw.
2. What I built
I built agents in Claude Code that click through the app the way I did. They drive the real interface with Playwright, in a headed browser, against a sandbox backend. Each one writes a report for every case with the evidence attached: the URL, the network request that fired or did not fire, and the message shown on screen.
The main design decision is that no agent does two jobs.
A worker validates exactly one scenario. It gets a scenario name, a target user and a folder for the run. It drives the browser, checks what happened, writes one report, and returns a one-word verdict. It never starts another agent.
An orchestrator runs the matrix. It creates one folder with a timestamp, starts the worker once per scenario, one after another, and keeps going past failures. At the end it reads every report and writes one summary with a pass and fail table. It never drives the browser itself.
One run. The workers run one after another in one browser, on purpose. Every scenario goes through the same steps, so two runs can be compared line by line.
The split helps three ways. Every scenario runs through the same steps, so two runs can be diffed. A worker that crashes costs one scenario, not the whole matrix. And there is one place for each policy: the orchestrator owns "continue on failure", the worker owns "fail loudly, never false-pass".
3. Three rules, written before the first run
It is a big ask to let an agent run against a real backend. The danger is not that it breaks the app. It is that it reports everything as passing when it never really checked. That is the objection I would have raised myself, so I wrote the rules against it first.
-
Each agent gets only the tools it needs. Two of them cannot write a file at all. A test agent that can edit source code can "fix" the test instead of the bug. The real risk, though, is data, not code. These agents create and delete real records in the sandbox, and git cannot restore a deleted database row. So every scenario works in its own date window far in the future, the worker cleans that window before it seeds anything, and cleanup after the run is off by default. The evidence stays on disk where a person can inspect it.
-
Find elements through the accessibility tree, never by coordinates. A click at a pixel position can hit the right button today and the wrong one after a layout change. The agents locate elements by role, label or test id, the same way a screen reader would. A side effect I like: every control in my area now has an accessible name, because the agents cannot find it otherwise.
-
Missing or unclear data is a failure, never a pass. An assertion passes only if the expected request really fired, came back with a 2xx status, and the response body contains the id it should contain. An empty response, a missing element or an ambiguous screen is a failure or an inconclusive result. It is never a pass.
4. What counts as evidence
A run passes when a specific, machine-checkable thing happened. A screenshot that looks right does not count. The things the agents are allowed to point at:
- the URL, with the parameters that must be present or absent
- the network request that fired, with its query string and its HTTP status
- the message rendered on screen, matched against the expected copy
- the browser history length before and after a change, to prove the app replaced the entry instead of adding a new one
- a value the app stored in the browser
There are also must-not-fire checks, and they matter as much as the positive ones. When a user is not allowed to change a record, the form shows a warning and disables the save. The screen looks correct either way. So the only honest assertion is that no request was sent at all. A person does not catch this case, because the screen already looks right.
Every case ends with one of five words, and each word has a fixed meaning.
- PASS: the expected observable happened.
- FAIL: it did not, or the wrong thing happened.
- INCONCLUSIVE: the evidence was missing or ambiguous, so no verdict is possible.
- SKIPPED: a precondition for the case was absent, for example a feature flag state or a required data setup.
- OBSERVED: the case documents accepted behaviour and carries no pass or fail.
The last three words are the important ones. They are how an agent says "I don't know" and "I could not run this" without lying. Without them, every problem with the test data would come back as a passing row.
The agents also discover what they need at run time instead of reading it from a hardcoded list. A worker that needs a record with a certain setup looks for one in the sandbox. If none matches, it reports SKIPPED with the reason. It does not guess, and it does not pass.
5. What the runs found
Three moments from real runs show the system working as designed, rather than only producing passing rows.
It refused to pass. The first full run ended with 7 PASS, 0 FAIL, 1 INCONCLUSIVE and 5 SKIPPED. The sandbox had no record with the configuration those five cases needed. The agents reported "skipped, no matching data" instead of guessing. I added the missing configuration and ran those five again.
It found a real bug. In that re-run one scenario failed first. A backend response had lost one field. The frontend treated the missing field as "no permission", so every row showed "no permission to change" and a real admin was blocked. The agent expected the create request to fire, none did, so it said FAIL and pointed at the response with the missing field. I fixed the backend the same day, and all five then passed. That was a real regression, not a flaky test.
It reported a limit instead of hiding it. One planned case could not be built in the sandbox at all, because two permission sets there were identical. The report says so, names the one configuration change that would make the case testable, and marks it as a configuration gap, not a failure. The lesson I wrote down for myself: read the test data while writing the matrix, not while running it.
6. What it does not do
The runs are serial and share one browser. That keeps them comparable, and it keeps them slow. The design is open to parallel runs, but I have not needed them yet.
The agents run against a sandbox only, never against production.
They do not replace the unit and integration layer. That layer answers "does this component do the right thing with this input". The agents answer "does the whole thing work for this user in this browser". Different questions.
The "five days to two" number is my own estimate for a permission-heavy feature. The part I can show is where the checking happens: end-to-end validation now runs in the same cycle as the code, not in a phase after it.
7. What changed in how I work
The report goes into code review next to the code. Instead of me saying "I checked the permission cases", the reviewer opens the report and sees the request that was sent, or the request that was not sent. For me this is the next step in code quality. It costs one slow run per feature, and I take that trade.
It changed how I plan, too. "How would an agent prove this works?" is now a question I ask while writing the plan. It forces the requirements to get specific before anyone writes code, because an agent cannot check "the form should feel right".
And it fixed my own opinion on test tooling. A green report you cannot trust costs more than a red one, because you stop looking. Any agent that tests on my behalf must be allowed to say "I don't know". That rule came first, and it is the reason the reports are worth reading.
Fact table: every number in this post, and where it comes from
The agent definitions and the run reports are private files in a work repository, so they are not linked. The counts come from the summary files those runs wrote.
| Claim | Source |
|---|---|
| Four years, one area, four repositories | my role from July 2022 to August 2026, as on my CV |
| Two agents cannot write a file | the tool list in those two agent definitions |
| Three rules before the first run | the agent definitions and the shared report contract they read first |
| Five verdict words | the report contract |
| Every control has an accessible name | my own account, confirmed when this post was reviewed |
| First full run: 7 PASS, 1 INCONCLUSIVE, 5 SKIPPED | summary file of that run |
| Re-run of the five skipped: 5 PASS | summary file of the re-run |
| The dropped field | the failing report in that re-run, and the fix the same day |
| The case that could not exist | my memory note from that run, and its summary file |
| About five days to about two | my own estimate for a permission-heavy feature; not a measurement |
| Hero image | photo by Lukas Kosc on Pexels, pexels.com/photo/35192687 |