# GPT-6.1 Sol — Bid Bench 1.1 CLI extension

Measured October 13, 2026. Five settings (Low, Medium, High, XHigh, Max) each receive the 30 delivery cases, four stopping cases and eight corrected forecast batches from Bid Bench 1.1, repeated three times. All 630 attempts completed with recorded tokens and selected-model/effort evidence. Every input and private grading-label batch matches its archived September run snapshot. The original benchmark results remain unchanged.

The [Bid Bench 1.1 method](/benchmarks/bid-bench-1.1-method.md) defines the case contracts, categories, difficulty tiers, exact acceptance rules, reference calculations, source corrections, candidate coverage, forecast labels, baselines and prior execution conditions. This extension uses its semantic-v4 deterministic grader and hidden function tests, including the input-preservation requirement for the Frontier app case. It adds no post-output grading corrections. No subjective model judge or human domain-expert review is claimed.

## Subscription CLI execution

All model inference uses Codex CLI, model `gpt-6.1-sol`, with ChatGPT authentication forced and API credentials excluded from the environment. User configuration, project instructions, shell, web search, apps and multi-agent tools are disabled. The model can call only read_source for sources declared in its case and javascript for isolated QuickJS calculations. JavaScript has 32 MiB memory, one second CPU time and no network, filesystem or process access. Source reads and calculation calls do not bind data automatically.

Each attempt has a 20-minute deadline and a limit of 32 tool calls. The maximum observed is 16. Four CLI processes run concurrently for the first 67 attempts; after those drain, eight run for the remainder. There is no model fallback and no scored retry. Ultra is excluded because its automatic delegation changes this two-tool harness. Off is not an available reasoning setting in the verified CLI catalog.

Identity evidence is the Codex session's selected model and reasoning setting. The CLI does not expose a provider-response served-model field, so this is not an API served-model receipt. Per-attempt final answers, source/tool traces, start records, session evidence and grades remain private. The public CSV contains only five configuration aggregates. The JSON includes those five plus unchanged historical comparison rows; complete=true applies to the 630 extension attempts, not to missing telemetry in historical rows.

## Metrics and comparison limits

Delivery pass requires the whole acceptance contract. Each category has 18 attempts per setting and equal weight; each difficulty tier has 15. Correct stops and false stops are separate. All 120 forecast batches return valid outputs. A valid output permits numerical grading and does not imply an accurate forecast.

Bidder error is mean Brier error over the fixed candidate panels; the chart displays Brier ×100. Price error is the mean of four exact item-code/unit groups' WAPE. Both metrics average three repeats with equal fixed batch sizes. Lower error is better. The bidder baseline is 0.06768977925381477; the price baseline is 0.3567919219293358. The cohort is retrospective, the candidate panels cover only 18 of 61 actual bidder/project appearances, and original descriptor availability has not been independently established.

Tokens are mean recorded input plus output per delivery attempt, including cached input; reasoning output is already part of output and is not added twice. Latency is median completion time for accepted delivery attempts. The charts default to Tokens with Latency as the second view. No API-equivalent dollar price is assigned to this subscription run. Subscription allowance is consumed, and a stable paid-credit balance is not a general zero-cost claim.

Historical comparison rows are dated September 28–30, 2026. They use the same task contracts but different API/CLI tool loops, budgets, context overhead and serving conditions. Scores can be described side by side; latency and tokens are not controlled provider-speed comparisons. Older failed forecasts and unknown measurements retain their original coverage and are not filled in.

All five GPT-6.1 settings pass all 90 delivery attempts. This small authored set therefore reaches its ceiling and cannot separate those settings by delivery accuracy. A 100–100 case-cluster bootstrap interval reflects no observed failures on this set, not perfect reliability on unseen work. These cases do not evaluate scanned-plan recognition, full graphical apps, full Bidlo application behavior or validated prospective forecasting.