# Claude Haiku 5.5 — Bid Bench 1.1 method

The exact `claude-haiku-5-5` model is selected in Claude Code 2.1.299 through
Claude.ai Max subscription OAuth. Low, Medium, High, XHigh and Max receive the
same frozen cases. This is a Claude Code CLI experiment; historical API and
other CLI comparisons describe these measured configurations and do not isolate
model weights from serving, context or orchestration differences. CLI selection
and, where a provider response exists, streamed model identity are recorded;
effort is explicit CLI configuration.

Each setting receives 30 delivery cases, four deliberately blocked tasks and
eight forecast batches, each repeated three times: 126 attempts per setting,
630 total. Delivery weights Documents, Takeoffs, Analysis, Knowledge and Apps
equally, with six distinct cases per workflow. Each of six difficulty tiers has
five distinct cases repeated three times. Ninety delivery attempts are not 90
independent problems, and 15 Stress attempts are not 15 independent problems.

Every attempt has a 300-second deadline, eight provider responses including tool
discovery, 32768 output tokens per response including thinking, and 32 native
benchmark tool calls. Four Haiku workers run concurrently. Incorrect answers
and resource failures remain in the denominator; no selective answer retry is
permitted. Transport failures stop dispatch when they cannot be classified
safely. Underlying CLI transport retries are disabled.

Two early output-limit failures exposed the CLI's automatic continuation after
`max_tokens`. They remain failed, with original records preserved. A narrowly
reviewed controller correction stops at the first output-limit event; only
never-started attempts resumed. Two isolated startup connection resets also
remain infrastructure failures, with no genuine provider response, observed
served identity, token total or cost. Synthetic CLI error-message zeros are
excluded from telemetry. Known aggregate costs remain lower bounds. A later
allowance pause drained active attempts before the final unstarted work resumed.
No completed or failed attempt was retried.

Fresh isolated homes and directories contain only public task sources and
trusted runtime code. A macOS file-content allowlist excludes private graders,
prior runs, account files and unrelated workspace contents. Only `read_source`
for exact declared IDs and callback-free QuickJS `javascript` are exposed.
JavaScript has no filesystem, process, import or network bindings, with one
second CPU and 32 MiB memory. Builtin tools, hooks, skills, plugins, memory,
browsers and agent delegation are disabled. Native model-visible tool payloads
must exactly match trusted bridge results. The largest declared source is
124754 bytes and 864 rows. All five actual configuration canaries check exact
source and nonce preservation, parallel native returns, 400000-character
JavaScript output, absent host globals and denied private reads.

The bridge permits 1.5 MB serialized results, but this Claude client retains at
most 500000 text characters inline. A declared-source spill stops the run;
trusted oversized JavaScript output is a failed client resource attempt.
No extra file-reading tool or compressed replacement source is supplied.

Private deterministic semantic-v4 acceptance checks run outside inference.
Frontier Apps must also preserve test inputs. Executable function tests and
structured text, geometry and schedule inputs do not represent scanned-plan
takeoffs or the full Bidlo application. Final publication requires an
independent audit and fresh grading of all 630 attempts.

Forecasts are separate from delivery. Bidder error is mean Brier score on the
provided candidate panels, which cover 18 of 61 actual bidder/project appearances.
Price error is the mean of four group-level weighted absolute percentage
errors. Historical baselines use the identical frozen panels and outcomes.
An incomplete 12-batch cohort has no complete forecast point; valid-output
subsets are labeled separately. These retrospective checks do not establish
prospective forecasting accuracy or describe Bidlo's production methodology.

Chart cost is an estimated uncached API equivalent, not a subscription invoice.
For each provider request, prompt tokens equal uncached input plus cache-read
and cache-creation input. At up to 100000 prompt tokens, input/output are
$0.10/$0.50 per million tokens; above 100000 they are $0.50/$2.50. Output includes
thinking. Cache-hit tokens are priced at the uncached input rate to match the
historical chart convention. The threshold is applied separately to every
request, including requests within the same tool loop.

Cost includes all delivery attempts and is normalized per 100 accepted tasks.
Known native usage is retained on cancellation. An unfinished response's final
output usage remains unknown; partial costs are lower bounds and marked ≥.
Unknown complete token measurements remain blank. Latency is median elapsed
time among accepted attempts. Delivery confidence intervals resample the 30
case clusters, preserving three repeats per case, using 10000 deterministic
bootstrap samples. Small authored samples and saturation do not establish
equal capability or reliability on unseen work.

Official pricing and effort sources:
[Haiku 5.5 model documentation](https://platform.claude.com/docs/en/models/haiku-5-5/overview),
[effort documentation](https://platform.claude.com/docs/en/build-with-claude/effort).