GPT-6.1 Sol — Bid Bench
Bid Bench evaluates GPT-6.1 Sol in the context of heavy civil construction and estimating workflows.
In this October 13 extension of Bid Bench 1.1, Low, Medium, High, XHigh and Max each pass all 90 delivery attempts. All five also identify every deliberately blocked task correctly. The run contains 630 attempts, with forecasts scored separately from delivery.
Low reaches the delivery ceiling
All 4 models shown. Only complete tokens measurements are plotted.
Every measured reasoning setting is shown. Select a point for its level and result.
GPT-6.1 Sol at Medium passes 100.0% of delivery attempts. The comparison settings pass 94.4% for GPT-6 Sol (Medium), 90.0% for Claude Fable 5.1 (Medium), 91.1% for Grok 4.7 (Medium).
Swipe to compare all four models →
| GPT-6.1 SolMedium | GPT-6 SolMedium | Claude Fable 5.1Medium | Grok 4.7Medium | |
|---|---|---|---|---|
| Mean tokens / task | 41,115 | 13,383 | 10,134 | 57,712 |
| Median accepted time | 25.1 s | 10.4 s | 17.5 s | 47.0 s |
| Delivery score | 100.0% | 94.4% | 90.0% | 91.1% |
| DocumentsDocument reconciliation | 100.0% | 83.3% | 83.3% | 100.0% |
| TakeoffsQuantity calculations | 100.0% | 100.0% | 100.0% | 88.9% |
| AnalysisRecord analysis | 100.0% | 88.9% | 66.7% | 100.0% |
| KnowledgeConstruction concepts | 100.0% | 100.0% | 100.0% | 100.0% |
| AppsConstruction functions | 100.0% | 100.0% | 100.0% | 66.7% |
Medium for all four models. These settings do not represent equal compute budgets. Historical peers were measured September 28–30 using different API/CLI conditions; these are descriptive comparisons. Full results: CSV · JSON.
GPT-6.1 Sol was measured through the subscription Codex CLI on October 13. Tokens include cached input; no dollar cost is assigned. Historical peers use different tool loops, context and execution conditions, so token and latency comparisons are not controlled. Method and limitations.
Low reaches the same delivery score as Max while using fewer tokens and finishing faster. It averages 37,222 input-plus-output tokens per delivery attempt, compared with Max’s 52,176. Median accepted completion time rises from 22.1 seconds at Low to 39.1 seconds at Max.
The earlier GPT-6 Sol run passed 85/90 attempts at Low and Medium, and 89/90 at High, XHigh and Max. These are descriptive comparisons: GPT-6.1 used the subscription Codex CLI, while GPT-6 Sol’s delivery results used the Bidlo API harness. Tool loops, context overhead and serving conditions differ.
Every workflow passes
Documents. Every setting passes 18/18 reconciliation attempts, including amendment precedence, withdrawals and effective-date cutoffs. None abandons a solvable document task.
Takeoffs. Every setting passes 18/18 quantity, exclusion, volume-conversion and truck-limit checks. The inputs are cross-section schedules and structured geometry, rather than scanned drawings.
Analysis. Every setting passes 18/18 record-selection and aggregation attempts, including approved revisions, letting-date filters and historical cutoffs.
Knowledge. Every setting passes all 18 construction and estimating scenarios, including the combined constraints in the stress case.
Apps. Every setting passes 18/18 executable function tests. Hidden checks cover procurement, quote leveling, scheduling and exact stock-cutting behavior.
Historical medians still beat price forecasts
Right of the dotted baseline is better. Bidder error is the Brier score × 100 (0–100); price error is a percentage. The two measures are not directly comparable.
All 120 forecast batches return valid predictions. Valid output means the forecast can be scored; it does not mean the prediction is accurate.
High has the lowest bidder Brier error, 0.06714 versus the historical baseline’s 0.06769. That is a small improvement within restricted candidate panels covering 18 of 61 actual bidder/project appearances.
Max has the lowest price error, 46.0% mean group WAPE, followed by High at 46.4%. Both remain worse than the historical-median baseline’s 35.7%. Additional thinking improves some price forecasts without establishing reliable prospective accuracy.
The stress tier no longer separates settings
15 attempts per difficulty level and thinking setting.
Every setting passes 15/15 attempts at every difficulty tier, from Routine through Stress. Each also identifies all 12 stopping cases, with zero false stops across its 90 solvable delivery attempts. The current authored set therefore cannot distinguish these settings by delivery accuracy.
Method
Each setting receives 30 frozen delivery cases, four stopping cases and eight forecast batches, repeated three times. The run uses the existing deterministic acceptance checks and hidden app tests. Only declared source reads and isolated JavaScript calculations are available, with a 32-tool-call limit and a 20-minute deadline per attempt. Concurrency increases from four to eight after the first 67 attempts. No scored attempts are retried.
Inference uses ChatGPT-authenticated Codex CLI. Tokens include cached input and output; latency is the median among accepted delivery attempts. Subscription usage is not an API invoice, so this report does not assign a dollar cost. These are controlled text, schedule and function tests, rather than the full Bidlo application. The small authored sample and retrospective forecast reconstruction limit generalization. Full method · JSON · CSV.
Choosing a setting
Low is the lightest measured setting to reach the delivery ceiling. High and Max reduce price error on this cohort, but neither beats the simple historical baseline. Harder delivery cases are needed before the benchmark can establish a benefit from more thinking.