Nearly 1 in 4 tests Claude wrote were useless. Then we made our test runs 9x faster

1Why this matters

AI coding agents now write a large share of new software. Left to themselves, they also write tests. Lots of them. A team sees a wall of green checkmarks and concludes the software works.

That conclusion is the most expensive assumption in AI-built software right now.

• A test suite is the last wall between a bug and your user. Nobody reads a thousand tests to check they mean anything.
• Agents depend on tests more than people do. A developer notices when a screen looks wrong. An agent has no picture of the product. Between one task and the next, the test suite is its only memory of "this works".
• A hollow suite makes the agent blind, and it does not know it. It will report "all 3,216 tests pass" in good faith while the one screen your customer uses is broken.

The owner of these builds put it bluntly: "I'm 100% sure you have a lot of useless test and are not running important tests." So we stopped building and counted.

2The words, explained once

This section assumes you have never written software. If you have, skip to section 3.

The agent

• Language model. Software that reads text and writes text.
• AI coding agent. A language model given tools: it reads files, edits them, runs commands and reads the results, in a loop, until a task is done. Here, the agents were Claude.
• Helper agents. A lead agent can hand parts of a job to helpers that work at the same time.

The test

• Test. A small program that checks another program: set up a situation, do something, check the result. "Put 2 and 2 in the calculator, press equals, expect 4."
• Green and red. A test that passes shows green; one that fails shows red. The whole value of a test is its ability to go red. A test that never can is decoration.
• Unit test. Checks one small piece of code on its own. Fast, and blind to everything around it.
• Journey test, also called an end-to-end or browser test. Drives the real app like a person: open the browser, click, type, check what appears on screen. Slow, and the closest thing to a real user.
• Regression test. Written after a bug is fixed, so that the same bug cannot come back unnoticed.

The fakes

• Mock. A fake stand-in for a real service. It answers the way the test author expects, which is not always the way the real service does.
• Fixture. The pretend data a test runs on: three invented hotels, one invented customer.
• Local build versus hosted app. The app on the developer's computer, versus the real one on the internet that users open.

The machinery

• CI, continuous integration. A robot that runs the tests on every change and refuses the change if a test fails.
• Default runner. Whatever runs the tests when nobody thinks about it. A test it does not load runs only when someone remembers. In practice, never.
• Hook. A small program that watches an AI agent and steps in at set moments. Our own agents run inside several hundred of these checks. We call them the guard rails. They have tests too, and those tests are part of this study.
• Foreground and background. In the foreground, the agent waits for the test and does nothing else. In the background, the test runs alone and the agent keeps working, then gets a notice.
• Exit code. The number a program returns when it ends. 0 usually means success.

3What we measured

Three real builds, all done by Claude coding agents in September 2026.

The three builds

• A hotel group's field app. Staff prepare hotel visits, fill in a visit on a phone or laptop, dictate notes and send a report. 752 unit tests, 516 active journey tests, and 60 extra check and probe files. We studied a build session of about a week.
• A tender-writing app. A team answers public tenders: a document editor, a compliance matrix that scores the answer against the requirements, a radar of upcoming tenders, and a chat assistant. 5,425 unit test cases and 62 browser test cases of its own. Built over 41 hours.
• Our own agent guard rails. The hooks above. 499 test suites, roughly 8,500 individual checks.

Four questions per build

• Every bug a human reported, from the owner or from a client through the owner. A bug reported three times counts once.
• Was there a test on that exact path, and was it green while the bug was live?
• What do the tests themselves check? We read and graded them, one by one where we could.
• What happened at each test run? About 5,400 runs, replayed from the agents' own session records: how long, did the agent wait, what happened after a failure.

One rule

Every number here was counted from the code, the session records or a fresh test run. An agent saying "done" or "all green" was never accepted as evidence. As you will see, that rule is the whole story.

4Zero of 65

Humans found 65 bugs across the three builds. No test caught any of them first. Not one.

What green-while-broken looks like

• The test was built on the wrong answer. The owner of the tender-writing app said app state belongs in a database, not a shared file. The agent's tests required the shared file, and stayed green until the owner noticed.
• The test faked the one step that broke. A user selects a paragraph in the editor and asks the assistant about it. The test handed the selection to the panel directly. The bug was that the selection never left the editor.
• The test used tidy data. The owner could not open a document on the live app. The test had opened one too: a known file with a plain name, on the developer's computer. The owner's file had a curly apostrophe in its name, and the hosted storage refused it.
• The test used a clean browser. In the field app, an "add address" button did not work for the owner, twice. The tests passed. They ran in a clean automated browser; the owner's browser had an ad blocker that hid the button because of its name.
• The test used short data. A data-table test fed in 24 months of figures. The real hotel had 92.

Our own code did it to us

• Two guard-rail suites checked for the exact line of code that later caused a false alarm. They did not miss the defect. They locked it in: fixing it would have turned them red.
• One afternoon, the full test run went green minutes after a live trial of the same guard rails stopped the agent 8 times, against a budget of 1.

5The useless 23%

Now the provocative number, and exactly where it comes from.

Take the browser tests of the two apps, the tests meant to act like a real user. There are 578. 133 of them, 23%, could not fail on anything the product itself does.

5.1 Where the 23% comes from

• 120 tested a demo app. The field app started from a starter kit: a ready-made skeleton that ships with a demo sales tool, with companies, deals and a kanban board. The field app has none of those screens. The demo tests were never removed, so every run still carefully tested a deals board nobody uses. That is 23% of the field app's journeys and 12% of its test time.
• 13 were empty. In the tender-writing app, 13 of 26 browser test files are placeholders: a title marked "to do", with no test inside. A review round asked for the tests to be "listed", and they were listed.

5.2 Where our own "useless" verdicts were wrong

A provocative number deserves an honest footnote. Grading tests is itself a test, and it can be wrong in both directions.

• The field app's build session checked our delete list item by item, and kept 6 of its 7 items. Two suites we called useless test live screens. One has 9 passing checks. One is the only check for a set of figures. The loose probe files hold the only regression tests for three of the owner's bugs. One test was deleted.
• The 6 kept include the 120 demo journeys, kept for now. Several of them exercise shared server code (sign-in, storage, permissions) that the real app uses too. They do not test the product's features, but they are not all worthless.
• Our guard rails, reviewed by an AI reviewer, show the same pattern. It called roughly 60 to 90 of 499 suites useless. Four it named each cover at least one check that is live and enforcing today, and our own audit tool grades all four "keep".
• Our automated audit erred the other way. It reported zero useless suites. Its detector for "tests the wording, not the behaviour" was broken, so it scored almost everything clean.

That is why the headline uses the narrow 23% and the narrow claim: tests that cannot fail on the product's own features. For everything wider, a "useless" verdict is a claim, and it gets checked before anything is deleted.

5.3 The zone around the 23%

• Tests of the screen you replaced. The tender-writing app changed its document editor twice. Its 10 editor browser tests still target the old one, and 9 skip themselves unless special settings are present. The editor people actually use has zero browser tests. Three unit-test files were pinned to the old editor so they would keep passing.
• Coverage that follows the work, not the product. In the field app, 36 of 67 features have no journey test, including the whole new home screen. One feature holds 93 journeys, 18% of the suite, because each review round added its own file.
• A "last verified" date on nothing. 20 features with zero tests still show a date in the field app's feature list.
• Tests of wording, not behaviour. In our guard rails, at least 70 suites hold at least 290 checks of the form "this exact text appears in the source code".
• Never-green journeys. The tender-writing app's two best browser suites, for the compliance matrix and the radar, have never passed. Of 50 browser runs, 5 passed, none of them those two. Nothing stopped the build.
• Unit tests were mostly fine. In the tender-writing app, the 5,425 unit cases were mostly on target and green. The problem sits in the layer meant to act like the user, which is where users find bugs.

6The tests that never ran

Worse than useless tests: the tests that mattered most did not run by default.

• The field app. 145 unit-test files never ran in CI or in the pre-release check. CI ran the build and the journeys; the pre-release check only read a stamp saying "the journeys passed in the last 24 hours". Most of the regression tests for the owner's bugs lived in those 145 files. For 14 days mid-build, there was no recorded green run at all. (The field app's session has since wired those unit tests into its pre-release check.)
• The tender-writing app. CI fired only when someone proposed a change to the main branch. The agents worked on other branches and never did. So no browser test and no full run ever gated a change. The mutation tool, which breaks the code on purpose to check the tests notice, had reported a false score of zero for weeks.
• Our guard rails. 15 of 499 test suites run automatically. The other 484, 97%, run only when someone launches them.
• And the real thing was locked. 17 of 20 attempts to run the tender app's journeys on the hosted app stopped at the sign-in screen. The robot had no account. Nobody followed up, because nothing was waiting for the result.

7Red that meant nothing

A test runs and fails. That is the moment the whole system exists for. Here is what happened in that moment.

• The failure notice said "success". The agents almost always added a small command after the test: print the last few lines, or a status message. The notice then carried the exit code of that printing, which succeeds. Of 287 background test runs set up this way, 217 notices said "completed, exit code 0" whatever the result. In the guard rails' main session, 86 of 88.
• The agent saw a slice. In the field app, 733 of 2,255 test runs were piped through a filter that showed a few lines. In the tender-writing app, 17 main-session browser runs were cut so short that the pass or fail line never showed.
• Failures got relabelled. In the tender-writing app, 98 of 406 failing runs never reached a pass. 44 were called "pre-existing", "environment" or "another lane's problem". 21 were closed with a "green" claim and no re-run.
• Red got committed. In the field app, 22 red runs were chained straight into saving and publishing the code. One helper agent, afterwards: "The suite shows 20 failures, and my chain pushed anyway."

None of this is malice. Each move is locally sensible for an agent trying to finish a task. But nothing forced a red result to be dealt with, so red became a mood, not a stop sign.

8Waiting for green

The owner's other complaint was simpler: the builds felt slow and the agent often looked idle. The records agree.

• The agent froze. 15, 27 and 13 hours of blocked agent time.
• The human saw nothing. No progress, no count, no time left. The owner typed "status" several times during runs, and once, after a long one: "I don't care about the test, let's finish this."
• Background runs already worked. When a run did go to the background, the field app's agent kept working in 94 of 105 cases.

That quote is the real cost. When tests are slow, blocking and invisible, people resent them, and a team that resents its tests finds ways to skip them.

9From 55 minutes to 6

Our guard rails' full test run had crept from about 6 minutes in mid-September to 55 by the end of the month. Four plain fixes made it about 9 times faster.

10What we learned

Every finding traces back to five mechanisms. They are not specific to our agents. They are what any agent does by default, because they are cheapest.

Three laws

• A test counts only if it can fail on something a user would see. Not "does a test exist?" but "if this broke tomorrow, the way a user would notice, would this test go red?"
• The default runner is the gate. Anything it does not load is not a test. The 145 unit files and the 484 local-only suites were written with care and protected nothing.
• The builder does not grade its own tests. The same agent writing the feature and its test passes its assumptions on, including the wrong ones. A separate seat designs the test, from the user's side, before the code.

What already works

• "Seen red first." When the field app's briefs required each new test to be shown failing on the old code first, the share of helper commits that said why a test existed rose from 6% to 53%.
• Every finding becomes a test. Turning each adversarial review finding into a labelled test grew one of our suites from 33 to 75 cases, each a real past failure.

Why sentences did not work

• This was not our first attempt. "Tests that prove nothing" shows up in 15 previous improvement cycles.
• Every past fix was a sentence in the agents' instructions. Sentences did not hold. What holds is structure: a check that refuses the commit, a runner that loads the tests, a notice that carries the real result.

11Eight moves for Monday

If your team uses AI coding agents, do these this week, in order. Each takes hours, not weeks.

1. Count your zero. Take the last 10 bugs a human found. Was a test on that path, and was it green while the bug was live?
2. Count your theatre. Find browser tests that are empty, skipped, or aimed at screens you no longer have. Check each one, then delete the truly dead ones in one change, listed by name.
3. Make the default runner load everything, on the branches your agents actually use. Then open the last automatic run and count what it ran.
4. Give the robot an account on your hosted app, so browser tests run where your users are.
5. Turn every human-found bug into a failing test first. Show it red on today's code, then fix until green, in a folder the default runner loads.
6. Make red stop the work. A red test blocks "done" and the commit. "Pre-existing" or "flaky" needs a named person to agree.
7. Run tests in the background, with the real result. No filter after the test command that swallows its exit code.
8. Put a test line on screen. For example: running, 212 of 499, about 3 minutes left, last result green. Nobody paying for the work should have to type "status".

12The fine print

• Three builds, one month, one team's agents. The mechanisms are general; the exact percentages are ours.
• Bugs are the ones humans reported. Bugs nobody noticed are not counted, so the true miss rate is no better than this and may be worse.
• The guard rails count defect classes, not individual reports.
• Some classifications are heuristic. About 5,400 test runs were classified by script, with samples checked by hand. The per-bug tables were read case by case.
• The 23% covers browser tests in the two apps only, and means "cannot fail on the product's own features". Some demo journeys still exercise shared server code.
• The speed "before" run shared a busy machine. An earlier run the same day took 1,308 s. The new runner is still about 4 times faster than that.
• We did not count duplicates.

13Glossary

AI coding agent

A language model given tools to read, edit and run code, working in a loop until a task is done.

Background run

A test that runs on its own while the agent keeps working.

CI (continuous integration)

A robot that runs the tests on every change and refuses the change if a test fails.

Default runner

Whatever runs the tests when nobody thinks about it. Tests it does not load effectively do not exist.

Exit code

The number a program returns when it ends. 0 usually means success. A command added after the test can replace the test's exit code with its own.

Fixture

Pretend data a test runs on. Useful, and dangerous when it is tidier than real data.

Foreground run

A test the agent waits on, unable to do anything else.

Green / red

Pass / fail. A test that can never go red proves nothing.

Hook

A small program that watches an AI agent and steps in at set moments.

Hosted app

The real app on the internet, as opposed to the local build on a developer's computer.

Journey test

A test that drives the real app like a person and checks what appears on screen. Also called end-to-end or browser test.

Mock

A fake stand-in for a real service, used in tests.

Mutation test

A check on the tests themselves: break the code on purpose and see whether any test notices.

Regression test

A test that catches a fixed bug if it ever comes back.

Starter kit

A ready-made app skeleton new projects start from, often with a demo and tests for that demo.

Unit test

A test of one small piece of code on its own.