Haiku 5.5 matches Sonnet 5.5 on retrieval. It costs a fifth as much.

1What we tested, from zero

Every answer our AI assistant gives about our own company starts with a search. It reads our notes, our past projects and our playbooks before it writes a word. After the work is done, another agent files what was learned. Those two jobs run all day. They were running on Sonnet. We wanted to know if a smaller model could do them as well.

Two jobs we gave the models

• Retrieval. A request comes in, such as "what do we know about this client's tender process?". A retrieval agent plans the search, reads the files and writes a short report of what it found, with sources.
• Filing. A working session ends. A filing agent reads what happened and writes the lasting lessons into the right place: private company notes, or a shared copy with every client detail removed.

Agents, subagents and readers

An agent is a model with instructions and tools. It can read files, run searches and start other agents. An agent started by another agent is a subagent.

For a wide question, our retrieval agent splits the search. It starts reader subagents, one per area, and merges what they send back. A filing agent also starts a reader first, to see what is already filed. More readers means more coverage, and more cost.

Effort

Effort is a setting that tells the model how much to think before it acts. Claude models take five levels: low, medium, high, extra high and max. Higher effort uses more tokens, so it costs more. It also changes how the agent works: at higher effort it tends to start more readers and run more searches.

The three models

• Haiku 4.5. The small Claude model we tried before. It has no effort setting.
• Haiku 5.5. The new small model.
• Sonnet 5.5. The mid sized model. Our agents ran on Sonnet until now.

How a score works

Each question has an answer key: the facts a good answer must contain. A blind judge reads each answer and marks every fact found, half found or missed. The score is the share of the key the answer earned. 72% means 72% of the facts.

2How we tested

We built the test from our own work, then paid for every model and effort level to run it. Here is the whole method, in the order it ran.

1. Collect. Take real requests from our own past work.
2. Key. Write the facts a good answer must hold, and check each one against its source.
3. Copy. Give every run its own fenced copy of our knowledge workspaces.
4. Pin. Set one model and one effort level for every agent in the run.
5. Run. Start the agent with the original request, word for word.
6. Verify. Read every agent's transcript and confirm the model and effort it ran on.
7. Judge. A blind judge scores each answer against its key.
8. Compare. Average score and cost per setup, on the same questions.

The two jobs

• Retrieval. The retrieval agent reads the request and plans the search. For a wide question it starts reader subagents, one per area of our workspaces. Each reader searches its area and reports back. The retrieval agent merges what they found and writes a report with a source for every fact.
• Filing. The filing agent reads a finished working session. It first starts a reader to see what is already filed. It writes private company notes itself. Then it can start three filers, one per shared library: tools, team patterns and use cases. Each filer writes a shared copy with every client detail removed.

Where the tasks came from

• 18 retrieval questions. Each is a real request our retrieval agent answered in the six weeks before the test. They cover a contact's details, a brand canon for a new site, past decisions behind a proposal, how a client runs a tender, our own positioning, and similar company questions.
• Answer keys checked against their sources. We started from the original answer and any later correction from the person who asked. A reader then opened every source file. A fact no source confirmed was dropped. Each key holds 4 to 12 facts, 196 in all.
• Partial credit. A found fact counts 1, a half found fact 0.5, a missed fact 0. The score is the mean over the key.
• Facts outside the workspaces, scored apart. 17 of the 196 facts live only in our CRM, a mailbox or a local file. The fence blocks those, so they sit in a separate column and stay out of every chart.
• 5 questions excluded. Their answers live in another organisation's workspace, which was outside the copy. No setup could have found them. We replaced them with 5 reserve questions from the same pool.
• 10 filing jobs. Each comes from a real working session, cut at the moment the filing agent was first started. Its key lists what must be filed, where it must go, and what must never appear, such as a secret or a client name in a shared copy. Two jobs should file nothing, to test restraint.

The workspace copy

• 4 drives, 8,943 files: company notes (6,463), tools (809), team patterns (1,534) and use cases (137).
• A fresh copy per run. Every run started from the same snapshot, so no run could see another run's writes.
• A fence. A guard checked every command an agent ran. It blocked our live workspaces, the CRM, the mailbox, other organisations and any write outside the run's own folder. We also compared the live workspaces before and after each run. No real data could change.
• Filing starts before the original filing. For each job, we rolled the copy back to the moment the original filing agent started. Files it created were removed. Files it changed were rolled back to their earlier text, as closely as the workspace history allowed. The agent then saw only what existed at that moment.

Pinning the model and the effort

• One definition per setup. Each setup got its own copy of the agent instructions. Only the model line and the effort line differ. A diff over all 11 setups found no other change.
• Subagents inherit. When an agent starts another, the guard removes any model or effort it asks for. Every subagent runs on the setup's model and effort.
• Checked after the run. A script read every agent's transcript and confirmed the model and effort it ran on. A run that failed this check was not graded.

The 11 setups

Haiku 4.5 has no effort control, so it runs once, as it comes.

The blind judge

• The judge. claude-opus-5-5 at high effort, with no tools and no project settings.
• What it saw. The answer, or the files a filing run wrote, and the key. Model names, effort words, run names and agent names were removed first.
• The rubric. Each fact in the key is found, half found or missed. For filing, a fact filed in the wrong place keeps its credit and costs a fixed 0.25 once, for placement. A job that should file nothing scores 1 when nothing is filed, else 0.
• Consistency. We gave the judge the same answer twice, on two cells. Each time one fact moved from half found to found. Each cell moved by 0.045, under 0.05.
• A human read. A person on our team reviewed 10 graded answers, chosen to cover close calls, a zero, a perfect score and a job that should file nothing, and found no grade to change.

Why some points use a shared sample

The costly setups could not run every question inside our usage limits. Sonnet 5.5 at high, extra high and max, and Haiku 5.5 at max, ran out first. So the retrieval chart compares every setup on the same shared sample of 6 questions that all eight of them finished. Section 3 then shows the 7 setups that finished all 18.

Run to run variance

One run per question can swing. So we ran Sonnet 5.5 low and medium 3 times on each of the 6 shared questions. Section 5 has the result.

Cost

• Billed cost per run. Each run reports what it cost, including every subagent it started. Two cells had no billed figure, so we used the list price of their tokens.
• The judge is counted apart. Section 9 lists it.

What failed

We started 477 runs for the 11 setups. 223 were graded. 237 were cut: by usage limits, outages, a failed model check, budget caps, or agents still working when the run closed. 14 repeated a question already graded, and 3 filing runs started from the wrong state. None of these was scored. 7 filing jobs at Sonnet 5.5 max effort never ran.

3Retrieval: score against cost

On the shared sample

• Haiku 5.5 high: 71.7% at $1.15 per attempt.
• Sonnet 5.5 high: 71.3% at $5.57. That is 4.8× the cost for a score 0.4 points lower.
• Haiku 5.5 extra high: 73.6% at $2.54, above Sonnet 5.5 medium (70.9%) and high (71.3%).
• Haiku 4.5: 40.5% at $0.69. Every Haiku 5.5 level beats it, and two of them cost less.

Over all 18 questions

The shared sample is small. Seven setups finished all 18 questions, so we can compare them on the whole set.

• Sonnet 5.5 keeps a lead on the full set. Medium scores 74.5%, low 74.2%.
• Haiku 5.5 medium scores 68.8% at $0.37 per attempt. Sonnet 5.5 medium costs 6.2× as much for 5.7 more points.
• Haiku 5.5 high drops to 66.4% on the full set. Section 8 explains part of that: two of its answers were sent before the readers reported.
• Haiku 4.5 scores 43.9% at $0.74. Haiku 5.5 medium beats it by 24.9 points at half the cost.

4Filing: score against cost

Filing is harder to score. A good filing run writes the right lesson, in the right place, with client details removed from any shared copy. It also files nothing when nothing is worth filing.

• Haiku 5.5 high scores 69.1% at $1.47 per job.
• Sonnet 5.5 medium scores 71.9% at $5.20. That is 2.8 more points for 3.5× the cost.
• Sonnet 5.5 low scores 70.9% at $2.87, 1.9× the cost of Haiku 5.5 high.
• Haiku 5.5 swings. Low scores 53.2%, medium 47.9%, high 69.1%, extra high 47.9%.
• Haiku 4.5 scores 28.9%. It filed little of what the keys asked for.

On the 3 filing jobs every setup ran, Haiku 5.5 high scored 87.7% and Sonnet 5.5 low 86.8%. Three jobs are too few to rank on, so we chart the full 10.

5More effort, mixed results

You would expect more thinking to give better answers. On these jobs the score moves up and down as effort rises.

• Retrieval, Haiku 5.5: 56.6% at low, 68.8% at medium, 66.4% at high, 69.1% at extra high.
• Filing, Haiku 5.5: 53.2% at low, 47.9% at medium, 69.1% at high, 47.9% at extra high.
• Sonnet 5.5, retrieval: low scores 74.2% and medium 74.5% on all 18 questions. Low is the cheaper of the two.

How much of that is noise

Sonnet 5.5 low scored above medium. That looked wrong, so we ran both again: 3 runs on each of the 6 shared questions.

• The gap is noise. Low averages 72.5%, medium 70.9%. Low minus medium is 1.5 points, with a 95% interval from minus 2.5 to plus 5.7 points. The interval includes zero.
• One question can swing a lot. The largest spread between runs of the same question was 0.32, on one question at low effort.
• A typical question swings 6.2 points between runs. That is the mean standard deviation within a question, 0.062 on the 0 to 1 scale.

6The race: one question, two models

The video at the top replays two real runs on one retrieval question. The question asks what we know about our own app hosting product, and the facts sit across two of our workspaces. Both models ran at medium effort.

• Sonnet 5.5 split the search. It started 5 readers, opened 65 files and finished first, at 6 min 13 s.
• Haiku 5.5 searched alone. It opened 24 files and finished at 7 min 07 s.
• Haiku found more. 5 facts in full and 2 in half, against 3 in full and 4 in half.
• Sonnet cost 9.7× as much: $4.34 against $0.45.

7How the agents work

Cost follows the work an agent does. Higher effort started more readers and ran more tool calls.

• Haiku 5.5 low and medium rarely start readers. They answer most questions alone, with 13 to 23 tool calls.
• Haiku 5.5 extra high starts 4.9 readers on average and runs 183 tool calls. It is the slowest setup in the table, at 14.5 minutes.
• Sonnet 5.5 high starts 2.7 readers and runs 85 tool calls.

Sonnet 5.5 at max effort

We started 25 retrieval runs of Sonnet 5.5 at max effort. Every one was cut by a usage limit, an outage or a time limit before it closed. So max effort was not graded, and every cost below is a partial cost: a lower bound. The runs still show how the agent behaves.

• In 13 of the 25 runs, the retrieval agent started 8 readers.
• Those 13 runs cost $16.61 each at the median and $16.02 on average, at list price, up to $23.75. These are lower bounds. A finished run would have cost more.
• The cost is real reading. In the three runs we traced, each of the 8 readers was started once and made 9 to 90 tool calls in its own area. No reader was restarted.

8A defect the test found

Reading the answers turned up a defect in our own retrieval agent. It is a fault in how the agent waits, and it hides real quality.

• What happened. The agent started its readers, then wrote its report before their results came back. Every need in the report read "pending".
• How we counted. A graded run counts when at least half of the needs in its report read "pending" and the judge scored it 0.
• How often. 3 graded runs: 2 at Haiku 5.5 high; 1 at Haiku 5.5 extra high. Each started 3 or 5 readers. Each scored 0.
• Why it matters. The readers did the work and the answer dropped it. Those zeros pull down the Haiku 5.5 high and extra high averages in sections 3 and 5.
• What we do about it. The agent must wait for every reader it started before it writes. That is an instruction change, and it applies to every model.

9What it cost

This was an expensive test. Here is what the useful part cost: the runs we graded for the 11 setups, the repeats for the noise study, and the judge.

• 246 graded tests and repeats cost $513.56 in all.
• Sonnet 5.5 cost $2.93 per retrieval test and $3.80 per filing test. Haiku 5.5 cost $1.24 and $1.16.
• The judge cost $66.56. That is $0.27 per answer read.

10What we are changing

We are moving both agents to Haiku 5.5.

Three rules we take from this:

• Test the small model on your own work. The model sheet does not tell you this. Haiku 5.5 matched Sonnet 5.5 on our shared sample and came within 5.7 points on the full set.
• Pick effort per job. Medium suits our retrieval. High suits our filing. Extra high cost more, scored within the noise of medium on retrieval and lower on filing.
• Read the answers. Three zeros in our data came from the agent, and the model was blameless.

11The fine print

• Small sets. 18 retrieval questions and 10 filing jobs. Section 5 shows how much one question can swing.
• One judge. Every score comes from one blind judge, claude-opus-5-5. It moved one fact per cell when asked twice.
• Single runs. Each cell is one run, except the variance study on Sonnet 5.5 low and medium.
• Our own tasks. The questions and jobs come from our own work and our own knowledge workspaces. Yours will differ.
• Partial grids. Sonnet 5.5 high, extra high and max, and Haiku 5.5 max, did not finish every task. Their points use the shared sample, or are left out.
• Facts outside the workspaces. CRM and mailbox facts were fenced off and excluded from every score here.
• We run these agents. They search and file our own knowledge every day. That is a stake in the result, so read the conclusions with it in mind.
• Our setup favours the incumbent. The agent instructions were written and tuned while the agents ran on Sonnet. Haiku ran them unchanged.
• Cost is per run, as reported. Each run's own billed total, with its subagents. Two cells use the list price of their tokens.
• Dates. The runs and grading took place on 7 and 8 October 2026.

12Glossary

Agent

A model with instructions and tools. It can read files, run searches and start other agents.

Answer key

The facts a good answer must contain, written before any model ran.

Blind judge

A model that scores each answer without knowing which model or effort level wrote it.

Bootstrap interval

A range found by resampling the measured runs many times. The true value probably sits inside it.

Cost per attempt

What one run of one question or job cost, including every subagent it started.

Effort

A setting that tells the model how much to think before it acts: low, medium, high, extra high or max.

Fence

The guard that kept every run inside its own copy of our data.

Filing agent

The agent that writes what a working session taught us into the right workspace.

Partial credit

A found fact counts 1, a half found fact 0.5, a missed fact 0.

Reader subagent

A subagent the retrieval agent starts to search one area and report back.

Retrieval agent

The agent that plans a search across our workspaces and writes a report with sources.

Run to run noise

How much the score of the same setup on the same question moves between runs.

Shared sample

The 6 retrieval questions every setup finished, used to compare them fairly.

Subagent

An agent started by another agent.

Token

A small piece of text, a few characters long. Models are billed per token.

Workspace

A shared drive of files that holds our company knowledge.