One pass of my eval bills $9.14 on the API and $0 through the CLI

That is 27 calls, and it is not an estimate. The CLI prints a total_cost_usd in its envelope: what the run would have cost on the API. It bills the subscription instead, so the number is a receipt for money nobody spent.

The switch fixed something better than the bill

Running with --output-format json and --json-schema rides the same structured-output machinery the API does, an internal forced tool call. Format reliability on my suite went from 7 out of 15 to 15 out of 15.

The schema needs relaxing first: strip pattern, minLength, maxLength, minItems, maxItems, format and the $schema meta-ref, because the CLI validator rejects draft-2020-12. The strict version stays in Zod on the caller side, so nothing is actually loosened, the validation just moves to where it can run.

One trap is worth the whole post

If ANTHROPIC_API_KEY sits in the child process environment, the CLI quietly bills the API account rather than the subscription. Nothing errors. Nothing warns. The invoice arrives. It gets stripped explicitly at spawn.

This is the failure mode I would look for first in anyone else's runner: the money leak is silent, and the only symptom is a bill at the end of the month for a run you believed was free.

And the limit, which matters more than the savings

This is a dev-loop tool. Anthropic's consumer terms prohibit automated access "except when you are accessing our Services via an Anthropic API Key or where we otherwise explicitly permit it", and the commercial terms governing API use do not cover consumer subscriptions. An eval runner on my own machine is the CLI used as designed. A shipped service is not.

Iterate on the CLI, ship on the API. What does one pass of your eval suite cost, and does anyone know it?