# Bid Bench 1.1 — measurement protocol

This edition evaluates models in heavy civil construction and estimating workflows. It measures frozen tasks and tools through both the Bidlo API harness and subscription CLIs. These are not measurements of the full Bidlo application. Differences in orchestration, budgets and serving conditions limit comparisons.

## Roster and execution

The roster covers 18 model entries and 80 model/thinking configurations. OpenAI Fast variants are excluded uniformly; standard Astra remains included. Grok 4.7 has Low, Medium, High and XHigh settings; previously omitted effort is retained as the documented High default. Grok Build is measured only at its app default because adjustable controls were not established in the reviewed provider documentation. Corrected Claude settings include Sonnet 5 Max, Fable 5.1 Max and Opus 5 Off. Fable Off is excluded because it is unsupported. Off is explicitly disabled for Sonnet 5 and Opus 5. Effort labels record requested controls, not equal internal compute budgets.

Every case runs three times per configuration. All routes use the same frozen task contracts and declared sources. Original API runs used eight model steps, a 32,768 requested output-token cap per step and a five-minute deadline, with up to two recorded transient retries. Recovery API runs retained eight steps and the requested cap with a twenty-minute deadline. Gateway caps were lower for some models: Fable 27,000, Sol 29,000 and Astra 25,000. Claude subscription CLI recovery used eight CLI turns, a requested 32,768 output cap and twenty minutes; eight CLI turns are not eight SDK steps. One OpenAI infrastructure replacement used Codex CLI with a twenty-minute deadline and 32 tool calls. Its output limit followed the CLI runtime. Recovery concurrency ranged from two to six; original concurrency reached twelve. There is no cross-model fallback. These budget and runtime differences are disclosed rather than treated as a uniform causal experiment.

JavaScript runs in QuickJS with 32 MiB memory, a one-second CPU budget, and no network, filesystem or process access. The model can read only sources declared for its case. Source reads and calculation calls do not share automatic data bindings: the model supplies calculation inputs explicitly. Large-source results therefore include the ability to transfer and reconcile data within this tool interface. Model answers, tool traces, routing metadata, usage, timing and metered Gateway cost are retained internally. They are not included in the public aggregate downloads.

## Tasks and acceptance

Delivery includes 30 controlled cases: six each for Documents, Takeoffs, Analysis, Knowledge and Apps. The five categories have equal weight. A case passes only when its entire acceptance contract passes; partial answers do not earn fractional credit. Published workflow percentages use 18 attempts per category and configuration. Difficulty tiers run from Routine through Multi-step, Complex, Expert, Frontier and Stress. They describe authored complexity and have not been independently calibrated with human contractors.

Documents reconcile issued amendments, withdrawals, subsidiary payment and field-level cancellation precedence against structured extracted source text. Takeoffs integrate cross-section schedules, overlapping exclusions and reuse zones, then convert volume states and apply truck limits. These cases do not test scanned-plan recognition or scale extraction from drawings.

Analysis tests letting-date selection, local-date filters, approved revisions, composite record identities and historical cutoffs. Knowledge uses closed-answer construction and estimating scenarios. Apps tests executable JavaScript domain functions for order quantities, quote leveling, interval takeoffs, procurement and crew scheduling. Hidden cases compare outputs to deterministic reference calculations. These are component functions, not complete graphical applications.

Numerical tolerances, required keys, exact choice tokens and behavioral tests are set before inference. Execution or identity failures cannot pass. Optional prose is accepted. When a response contains preliminary status objects and a final answer, grading extracts the last complete JSON object containing status. A uniform parser correction prevents an earlier fenced status stub from hiding that final answer; the original strict delivery pass counts are retained internally. The Frontier Analysis prompt leaves its projects field untyped. A county project list is therefore accepted as equivalent to a count only when every member exactly matches the expected county membership, without omissions or duplicates. A uniform normalization handles this ambiguity; parser-only and original strict pass counts remain archived internally. Grading corrections do not change numerical tolerances. Recovery execution budgets are disclosed above. No subjective model judge assigns scores.

Four separate stopping cases contain a missing input, an unavailable outcome or an infeasible constraint without disclosing the correct blocker in the question. Correct stops must identify the actual blocker. On the future submitted-price case, either outcome or chronology identifies the same essential blocker; both are accepted uniformly. This prevents an arbitrary choice between overlapping reason codes from being scored as a reasoning error. False stops are blocked answers to solvable delivery cases. Stop time is the median for correct stops, excluding timeouts.

## Retrospective predictions

Bidder forecasts cover 12 TXDOT projects and 20 candidates per project. The candidate universe is selected from pre-cutoff participation; it is not a complete roster of every possible bidder. These panels contain 18 of 61 actual bidder/project appearances (29.5%). Brier error therefore measures calibration within a restricted candidate panel, not recall of every actual bidder. Each target has multiple possible positive bidders, so probabilities do not need to sum to one. Brier score is the mean squared probability error. The baseline uses prior district participation frequency, falling back to overall frequency when district history is absent.

Price forecasts cover 16 project-item targets in four exact item-code/unit groups. The target is the median submitted contractor unit price. Each group's WAPE is the sum of absolute prediction errors divided by the sum of actual prices; the report averages group WAPE rather than pooling incompatible units. The baseline is the median of supplied prior project-item medians in the same code/unit group.

History spans January 2024 through April 2026. Targets fall between May 1 and July 21, 2026. Engineer-estimate pseudo-bidders, unidentified contractors and nonpositive or unlinked bid totals are excluded. The recorded project date must match an explicit letting date. Target outcomes are held outside model-visible inputs. Public graphs show forecast error only when every required batch returned valid predictions; coverage remains in the downloadable data.

This is a small retrospective reconstruction from Bidlo's historical export, not an untouched prospective test. Effective availability of project descriptors at the original prediction date has not been independently verified. Forecast errors should not be read as validated production prediction accuracy.

## Quality and resources

Delivery pass rate is accepted attempts divided by all 90 delivery attempts for each configuration. The Cost chart estimates API-equivalent cost from recorded input/output tokens at the standard uncached rates frozen on September 28, 2026. It excludes caching discounts, priority prices, gateway fees and subscription fees; it is not an invoice or a claim about actual CLI expense. Average task cost divides the estimated cost of all attempts, including failures, by 90. Cost per 100 accepted tasks divides the same total by accepted attempts and multiplies by 100. All 80 delivery settings have complete token measurements for this estimate. Original paid receipts and excluded diagnostic costs remain in the private billing history, separate from the selected benchmark attempts.

Tokens are mean input plus output tokens per delivery attempt. Cached input is included; reasoning output is counted once. CLI usage and API usage are normalized to that convention. Latency is the median observed completion time for accepted attempts, including tool work. Nineteen settings combine API and CLI runs. Their timing includes differing orchestration overhead and is not a controlled comparison of provider speed. Conditional timing can also favor a model that succeeds only on easier tasks, so it must be read with pass rate and the per-setting harness counts in the downloads.

Pass-rate intervals use 4,000 deterministic case-cluster bootstrap draws, stratified by category, retaining all three repeats within each sampled case. They describe uncertainty on this authored set, not the population of contractor work. Ties and overlapping intervals should not be interpreted as decisive rankings. Several settings still pass all 90 delivery attempts, including the Stress tier. The resulting 100–100 bootstrap interval reflects no observed failures on this small set; it does not establish perfect reliability on unseen work. This edition cannot separate those settings by delivery accuracy.

## Frozen evidence and source corrections

The main execution batch, a uniformly applied Frontier/stopping extension, a corrected forecast batch, and a clarified Analysis batch have separate immutable input manifests. The Frontier extension was frozen before scored main-run outputs were inspected. After leading configurations still reached the ceiling on completed delivery cases, a sixth Stress tier was authored and frozen before any of its inference. It adds one case in every category for every configuration: document effective-date cutoffs, cubic excavation geometry, historical bidder withdrawals and aliases, combined estimating scenarios, and exact stock-cutting procurement. Its reference answers were checked with independent field selection, exact quadrature, an independent SQL query, and two stock-cutting solvers. This later tier is an exploratory extension, not an untouched confirmatory set. A source audit found the original forecast export contained the engineer's estimate as a pseudo-bidder. Those original forecasts are retained but excluded from publication; every configuration receives the corrected inputs and labels. A stopped response exposed an ambiguity in the original Analysis fixtures: the base estimates lacked an explicit approval status. All four affected cases were rerun for every configuration with that premise clarified and the numerical answers unchanged. Their original attempts remain unreported diagnostics. The Frontier Analysis case already specifies approved records. Four originally prompted stopping cases are retained only as instruction-following controls.

This edition reports 10,080 attempt slots: 7,200 delivery, 960 stopping and 1,920 forecasts. Earlier malformed Claude effort settings are excluded as a group and preserved; they are not relabeled as verified effort measurements. A separate frozen corrected-setting plan contains 2,772 Claude attempts, measured through 550 API and 2,222 subscription CLI attempts. Additional Grok levels and narrowly reviewed execution repairs have separate frozen plans. All earlier results, partial traces, controls and preflights remain preserved internally. Completed wrong answers and legitimate budget failures were not rerun to obtain higher scores.

Nineteen specifically reviewed infrastructure failures or interrupted starts received separately recorded replacements: fourteen through other-provider APIs, four through Claude CLI and one through Codex CLI. Original traces, receipts and paid partial usage remain immutable. All replacements have terminal records. Six different Claude forecast attempts reached the twenty-minute deadline without full final usage; three early nonstreaming runs also lack a served-model identity receipt. Their requested configuration is recorded, but the missing identity and usage are not inferred. They count as unsuccessful scheduled forecasts, receive no numeric error, and are not repeated. This limitation affects Fable Max and Opus 5.5 Max forecasts, not delivery cost/token coverage.

Validation uses automated reference checks, frozen-hash verification and primary-agent review. No independent human domain-expert review is claimed. Public JSON and CSV contain aggregate scores, coverage, normalized resource estimates and harness counts. Input snapshots, private labels and full execution traces stay private. A complete attempt roster does not imply every forecast produced a valid numeric result.