# Claude Sonnet 5.5 on Bid Bench 1.1: method

Bid Bench evaluates heavy civil construction and estimating work. This run uses the frozen Bid Bench 1.1 case set, public inputs and deterministic semantic grading rules. Its delivery score, forecasts and stopping behavior measure different outcomes and are reported separately.

## Cohort and scoring

The five settings are Low, Medium, High, XHigh and Max. Each setting runs 30 delivery cases, four deliberately blocked cases and eight forecast cases, repeated three times: 126 attempts per setting and 630 attempts overall. The delivery score uses 90 attempts per setting. Equal numbers of cases make Documents, Takeoffs, Analysis, Knowledge and Apps equally weighted.

There are six difficulty tiers, with five distinct delivery cases per tier. Each tier therefore contains 15 attempts per setting. Repeats are observations of the same authored problem, not independent new problems. Confidence intervals resample distinct cases as clusters, retaining their three repeats, using 10,000 bootstrap draws. They describe variation within this authored suite and do not establish general performance on unseen work.

Delivery answers are checked against frozen contracts. Numeric comparisons use the frozen tolerances. App answers are executed in isolated QuickJS against private tests; the Frontier app also must preserve its input. Semantic normalization accepts only the predetermined equivalent representations in the frozen grader. A wrong answer, refusal on solvable work, deadline failure or other execution failure remains a failed delivery attempt. Answers are never retried to obtain a pass.

The blocked cohort tests whether the model declines tasks missing essential information. The report distinguishes a blocked decision from naming the exact required blocker. False stops are blocked decisions on solvable delivery tasks.

## Forecasts

Bidder forecasts are scored with the Brier score; chart error is that score multiplied by 100. Unit-price forecasts use mean group weighted absolute percentage error. Lower error is better. The two measures have different scales and are not directly comparable.

There are 12 bidder batches and 12 price batches per setting. A full-cohort metric appears only if all 12 relevant batches produce valid predictions; otherwise the full metric is null and coverage is explicit. Any valid-output subset metric is labeled separately. Historical baselines use the same frozen group labels and grouping rules.

The bidder candidate panels cover 18 of 61 actual bidder/project appearances. Results on these panels do not evaluate complete bidder discovery. These benchmark forecasts are separate from Bidlo's production forecasting methodology.

## Model and execution

The requested model is `claude-sonnet-5-5`, selected explicitly through Claude Code 2.1.299 using an existing Claude.ai Max subscription. The run uses no Anthropic API-key inference and no fallback model. Anthropic's official documentation lists all five tested effort levels for Sonnet 5.5. The command, isolated configuration and returned model identity are preserved for every attempt; these are CLI selection records, not cryptographic provider attestations.

Every attempt has a 300 second wall-clock deadline, at most eight provider responses, 32768 output tokens per response and a separate 32 call native-tool safety ceiling. Four attempts run concurrently. Underlying transport retries are disabled. The controller counts unique provider message IDs; Claude Code's tool-oriented `num_turns` field is not substituted for provider response count. Exceeding a monitored limit kills the process group and fails the attempt. A request already in flight at cancellation may have unreported usage. A provider length stop (`max_tokens`) fails the attempt immediately; the CLI may not repair it with a synthetic continuation prompt.

The model receives the public request, source index, supplied input and two tools: `read_source` and `javascript`. Independent calls may be issued together in one provider response. `read_source` resolves exact declared source IDs and returns the whole declared object. `javascript` executes callback-free QuickJS, with 1 second CPU and 32 MiB memory, and has no filesystem, process, import or network bindings.

Each attempt starts with an empty working directory, home, Claude configuration directory, temporary directory and session. macOS content permissions deny private benchmark files, prior attempts and unrelated workspace files. Model-facing shell, browser, file, plugin, memory, skill, agent and communication tools are disabled. The trusted controller reads current subscription authentication; the benchmark tool child receives no OAuth token, and authentication is not copied into the isolated home.

## Full tool results and verification

The largest source is 124754 serialized bytes, containing 864 rows. Each source and JavaScript result is paired with a trusted native-tool return and compared against the exact model-visible payload. Source objects are also compared with their frozen declarations. Missing, reused, invented, altered or silently truncated returns cannot support an accepted attempt.

Claude Code has a 500,000 character inline text ceiling for annotated MCP tools. Both benchmark tools declare that ceiling; the bridge retains its reference 1,500,000 byte serialized-output guard. All public sources fit below the client ceiling. A source spill is an infrastructure failure that stops dispatch. An oversized JavaScript result is an explicit client resource failure, not a silently accepted partial result. No auxiliary file-reading or source lookup tool is added to bypass this limit.

Unscored canaries verify the largest full source, an independent nonce read batched in the same provider response, a 400,000 character JavaScript return, denied private access and unavailable host globals. Local regressions test caps, process cancellation, QuickJS limits and deliberately damaged payloads. After the scored run, raw identity, source, tool and budget evidence is audited and every answer is independently regraded before publication.

The evidence retains the initial seal and a linked controller amendment. After 225 terminal attempts, the length-stop watchdog was tightened to prevent CLI continuation prompts. None of those 225 contained a length stop; all original records and traces were preserved byte for byte. Only the remaining 405 previously unstarted attempts used the amended controller. Public tasks, tools, budgets and model settings did not change, and no scored answer was retried.

## Resource accounting and comparison limits

Cost is an estimate at [Anthropic's standard uncached API rates](https://platform.claude.com/docs/en/models/sonnet-5-5/overview): 2 USD per million input tokens and 10 USD per million output tokens. It is not the subscription bill. This normalization matches historical Bid Bench charts.

Anthropic reports ordinary input, cache-creation input and cache-read input separately. The estimate sums all three at the uncached input rate for each provider request. Output tokens already include thinking and are counted once. Repeated assistant content blocks with the same provider message ID do not multiply usage. Recorded final usage is used where available; fixed input counts from message-start events can supply missing final input fields. Unknown components remain unknown, rather than becoming fabricated zero observations.

The chart plots average estimated cost across all delivery attempts. The comparison table reports cost per 100 accepted delivery tasks, including observed cost from failed delivery attempts in the numerator. Incomplete usage is a lower bound, marked ≥. Externally killed attempts are conservatively treated as incomplete even when every observed message has final usage, because a further CLI request may already be in flight without a recorded message-start event. A failed answer can still have complete resource telemetry if the CLI exits normally, its successful final result reconciles with all observed request usage, and all native calls have returns. A complete mean token count is shown only with complete usage for all 90 delivery attempts. Latency is median elapsed wall time on accepted delivery attempts; failure rates remain in the delivery score.

Historical peer rows retain their original evidence and coverage. They were measured under different API or CLI serving, context and orchestration conditions. These comparisons are descriptive, especially for token use and latency, and do not isolate model capability from transport differences.

Account monitoring is read-only. Paid extra usage must be disabled, and new dispatch stops at the allowance threshold or if paid spending changes. Subscription allowance consumption is separate from the estimated API-equivalent costs shown in the charts.

Independent audit fingerprint (record bytes and fresh grades): `d0b33c3511c20312e6ae2883a238127095f87fb39c9fa223b0fa23500148cb26`.