We hid 8 traps in two software contracts and ran Haiku, Sonnet, Opus and Fable 24 times. All found all 8. Costs, list lengths and the flaw most runs missed.
[ KEY TAKEAWAYS ]
For a first scan of a standard vendor contract, the cheapest Claude model is enough: Haiku found all 8 traps we planted in every run, for under a cent. For a contract your firm will live with for years we'd use Sonnet or Opus, because their lists were shorter. That last part is a judgment call, and two contracts can't prove it.
reviews found all 8 planted traps, across four models and two contracts.
is the price gap between Haiku and Fable on the harder contract: $0.003 against $0.23 a review.
paid for all 24 reviews together, at API list prices.
runs noticed the unplanned flaw in the first contract, on any model.
We wrote a software contract of the kind a law firm signs for its case management system, 12 sections, mostly boring, and hid a trap in eight of them. Each model got the same instruction: you're reviewing this for a 12-lawyer firm about to sign it, list every risky clause. Three runs per model, default settings, no special instructions.
Those traps were obvious on purpose. Real contracts tend to put the bad part in a definition at the top and use it twenty paragraphs later, so we wrote a second contract with the same 8 dangers hidden that way. There, a trap only counted when the answer named the actual danger. Citing the right section wasn't enough.
All four models found all 8 traps in all 24 runs. The most expensive review cost 78 to 80 times the cheapest, for the same 8 traps. What changed between models was the number of issues a lawyer has to read through.
| Contract 1 | Traps found | Cost a review | Issues listed |
|---|---|---|---|
| Haiku 5.5 | 8 of 8 | $0.002 | 16 to 19 |
| Sonnet 5.5 | 8 of 8 | $0.03 | 15 to 16 |
| Opus 5.5 | 8 of 8 | $0.05 | 8 to 10 |
| Fable 5.1 | 8 of 8 | $0.16 | 10 to 13 |
| Contract 2, hidden traps | Traps found | Cost a review | Issues listed |
|---|---|---|---|
| Haiku 5.5 | 8 of 8 | $0.003 | 23 to 24 |
| Sonnet 5.5 | 8 of 8 | $0.05 | 16 to 17 |
| Opus 5.5 | 8 of 8 | $0.08 | 13 to 18 |
| Fable 5.1 | 8 of 8 | $0.23 | 15 to 16 |
Average cost per review at API list prices, three runs per contract per model. Per million tokens, input and output: Haiku $0.10 and $0.50, Sonnet $2 and $10, Opus $4 and $20, Fable $10 and $50. Check the cost calculator for your own volumes.
On the first contract Opus listed 8 to 10 points, almost all of them the real traps. Haiku listed 16 to 19. Some of its extras were fair points to negotiate, and others were clauses most firms sign without a second look: the 99.5% uptime promise, the right to cancel if the vendor breaks the contract, Delaware law. At these prices the AI is the cheapest part of the review, so the shorter list usually matters more than the price of the model. That gap narrowed on the second contract, where Opus listed up to 18.
Both contracts cap the vendor's liability at the fees paid in the month before the claim, and both have the firm pay once a year. So in most months the cap is zero. We wrote that by accident. No model noticed it in the first contract. In the second, three models caught it in every run and Haiku in one of three.
| Model | Contract 1 | Contract 2 |
|---|---|---|
| Haiku 5.5 | 0 of 3 runs | 1 of 3 runs |
| Sonnet 5.5 | 0 of 3 runs | 3 of 3 runs |
| Opus 5.5 | 0 of 3 runs | 3 of 3 runs |
| Fable 5.1 | 0 of 3 runs | 3 of 3 runs |
A run counts when the answer says the cap falls to zero or near zero because fees are paid annually. Answer key in the repo.
Fable's second run put it plainly: "the cap is zero in eleven months of the year." Haiku's first run said the cap "is a very small amount", which is true and misses the point.
It's tempting to read that as proof the expensive models see more, and two contracts don't support it. The second contract spells the cap out through a "Fee Period" definition, which may be what made the flaw easier to spot, and on the first contract Fable missed it in all three runs just as Haiku did. We'd treat it as a reason to look closer. It's also a reminder that a clean result on the traps you planted says nothing about the ones you didn't think of.
These are worth knowing whether or not you ever use AI on a contract.
| Trap | How it read in contract 1 | Ask for |
|---|---|---|
| The long renewal | Renews for 3 years unless you cancel 4 months before the end, by letter only | 12-month renewals and notice by email |
| Price changes by website | The vendor raises the price by updating its pricing page, mid-contract, with no notice | Fixed fees for the term, a capped yearly increase, written notice |
| Client files train their AI | A permanent right to use your data to build its products, surviving the end of the contract | Delete the clause. Your data runs the service and nothing else |
| Paying for your own data | A full export costs $4,000 a time, takes up to 60 days and comes in the vendor's format | A free full export, at any time, in a format other software can open |
| Locked out over one invoice | One payment past due and they can cut access, including to your files, with no warning | Written notice, a cure period, and read access to your files no matter what |
| The vendor owes one month's fee | If they leak client data, the most they owe is what you paid in the prior month | A cap of at least 12 months of fees, higher for data breaches |
| You pay for their mistakes | You cover the vendor's losses from your use of the software, even when the vendor caused them | Each side covers its own fault |
| Deleted a week after you leave | Seven days after the contract ends, everything is gone | At least 60 days to get it out, and written confirmation before deletion |
The second contract keeps every danger and moves it out of sight. Four examples from it:
Every model connected those. Haiku did the $250 times 16 math itself, and it spotted that "Losses" quietly includes the vendor's own fault.
Before you use a real contract Whatever you review goes to Anthropic to be processed. Check your confidentiality duties and your agreement with Anthropic before you put a client document through this, and read which Claude plan fits a law firm and how zero data retention works first. Nothing here is legal advice.
Download the contracts as PDFs: contract 1 and contract 2. Both are fictional. The GitHub repo has those, the answer keys, all 24 answers and a small Python script that sends any PDF or text contract to the models you pick and saves what each one said and what it cost. You need an Anthropic API key, and reviews are charged to it. The script shows a rough cost and asks before it spends anything.
Our test didn't separate them on the traps we planted: Haiku, Sonnet, Opus and Fable each found all 8 in all 24 runs. The differences were cost and list length. Haiku cost about $0.003 a review and listed up to 24 issues. Opus cost about $0.08 and listed 8 to 18. For a first scan Haiku is enough. For a contract the firm will live with for years we'd use Sonnet or Opus, which is a judgment call the test doesn't settle.
It found every trap we planted, in both contracts, in every run. It caught the flaw we didn't plant in only 1 of 3 runs on the second contract, and it listed up to twice as many issues as Opus for a lawyer to read. That makes it fine for a first scan and weak as the only read of a contract that matters.
Our two contracts cost between $0.002 and $0.23 per review at API list prices, depending on the model. All 24 reviews together cost $1.80. A longer contract costs proportionally more. The lawyer reading the result costs far more than any of these.
Check your confidentiality duties and your agreement with Anthropic first, because the contract text goes to Anthropic to be processed. Which terms apply depends on the plan. Team doesn't train on your content by default. The API can run under a zero data retention agreement, which also limits which models you can use.
No. We wrote both contracts ourselves, and a contract drafted by a vendor's lawyers is harder. No lawyer reviewed our answer keys or the model answers, and finding a trap isn't the same as giving good advice. What the test shows is what a lawyer can expect to get as a first pass.
Yes. The script, both contracts, the answer keys and all 24 answers are public on GitHub under an MIT license. You need Python and an Anthropic API key, and reviews are charged to your key. The script shows a rough cost and asks before it spends anything.
Choosing a model for your firm's documents?
Thirty minutes on what your firm reviews, which plan you're on and where the documents can safely go. You leave with a model and a data path you can defend.
NEWSLETTER
[ WRITTEN BY ]
Saša Sladić
Co-Founder & CEO
Co-Founder and CEO at Oktopeak. Works with founders in legal, healthcare and fintech to get stalled, broken and inherited products into production.
[ LEGAL TECH ]
LEGAL TECH
Jun 29, 2026 · 18 min read
Claude Team vs Enterprise for a law firm: the bill at 2, 5, 10 and 20 people, the 20-seat floor, who sees spend, and where Pro, Max and the API fit. Checked Nov 2026.
Read ArticleLEGAL TECH
Oct 7, 2026 · 7 min read
A partner's words on the call. Projects, skills, bounded context, tool approval, caching, a usage view, training: what each saves and how to set it.
Read ArticleLEGAL TECH
Jun 3, 2026 · 11 min read
ZDR is not available on Claude Team or Pro chat. It is available on the Anthropic API at the organization level, which a small firm can obtain. Here is the real difference between the chat tiers and the API, why a Canadian or healthcare firm hits a residency wall, and the architecture we use to keep privileged data off retention.
Read Article[ GET STARTED ]
30-minute call. No pitch deck. Just an honest conversation about your project.
