GPT-6 Luna — Bid Bench
Bid Bench evaluates GPT-6 Luna in the context of heavy civil construction and estimating workflows.
The benchmark covers Documents, Takeoffs, Analysis, Knowledge, Apps and Predictions. Bid Bench 1.0 compares delivery quality across thinking settings, with forecasts and stopping behavior scored separately.
Low cost comes with more failures
Delivery scores weight the first five workflows equally.
All four models shown. Time is measured on accepted work; scores include failures.
Lines connect each model’s best measured trade-offs. All measurements are available in the downloads.
At Medium, Luna passes 78.9% of delivery attempts, versus 73.3% for GPT-5.6 Luna, 57.8% for Gemini 3.8 Flash and 32.2% for Claude Haiku 4.5. Luna improves on its predecessor in Documents, Takeoffs and Knowledge; the older model still leads in Analysis and Apps.
Swipe to compare all four models →
| GPT-6 LunaMedium | Claude Haiku 4.5Medium | Gemini 3.8 FlashMedium | GPT-5.6 LunaMedium | |
|---|---|---|---|---|
| Cost / 100 accepted | $0.22 | — | — | $0.46 |
| Delivery score | 78.9% | 32.2% | 57.8% | 73.3% |
| DocumentsDocument reconciliation | 72.2% | 50.0% | 61.1% | 66.7% |
| TakeoffsQuantity calculations | 83.3% | 33.3% | 88.9% | 50.0% |
| AnalysisRecord analysis | 50.0% | 16.7% | 38.9% | 66.7% |
| KnowledgeConstruction concepts | 100.0% | 55.6% | 100.0% | 88.9% |
| AppsConstruction functions | 88.9% | 5.6% | 0.0% | 94.4% |
Medium for all four models; providers do not use identical compute budgets. — means incomplete measurement. Full results: CSV · JSON.
Thinking helps some workflows more
Knowledge improves early; reliable record analysis needs more thinking in this set.
Low passes 63/90 attempts at $0.17 per 100 accepted tasks. Medium improves to 71/90 at $0.22. High also passes 71/90, with slightly more tokens and time. Max reaches 83/90, but incomplete telemetry prevents a complete cost comparison. Median accepted latency rises from 6.9 seconds at Low to 18.2 at Max.
Documents. Low passes 12/18 and Medium 13/18. Max reaches 15/18, leaving document reconciliation short of full completion.
Takeoffs. Low and Max both pass 16/18. An inspected Off response divides by 27 twice when converting cubic feet to cubic yards; additional thinking does not eliminate every quantity error.
Analysis. Low and Medium pass 9/18; Max reaches 18/18. An inspected Medium answer selects the wrong eligible winning bid.
Knowledge. Medium through Max pass 18/18, compared with Low’s 10/18. This is the clearest early gain from thinking.
Apps. Low and Medium pass 16/18; High and XHigh reach 17/18. Max returns to 16/18, so the highest setting is not uniformly strongest.
Forecasts do not beat the baselines
Both forecasts are scored against recorded outcomes. Zero error is perfect; beating the historical baseline means improving on a simple statistical forecast. The bidder baseline uses past district participation rates, with overall participation as a fallback; the price baseline uses historical median prices for the same item and unit.
Right of the dotted baseline is better. Bidder error is the Brier score × 100 (0–100); price error is a percentage. The two measures are not directly comparable.
The candidate panels cover 18 of 61 actual bidder/project appearances. Bidder error measures only that restricted panel.
Among complete settings, High has the lowest bidder error score, 6.80 out of 100, versus the baseline’s 6.77. Low has the lowest price error, 41.8%, versus 35.7%. Thinking Off supplies only 8/12 valid bidder batches and 10/12 price batches, so it has no complete forecast point. These retrospective results do not validate prospective prediction accuracy.
Hard cases benefit from higher settings
We test construction tasks at six difficulty levels, from routine work to the hardest cases. Each line shows how often the model completes them correctly at a different thinking setting. On the hardest tasks, Max passes 12 of 15 attempts; Off passes 1.
15 attempts per difficulty level and thinking setting.
High through Max identify all 12 blocked tasks correctly. False stops fall from 12/90 with thinking Off to zero at Max. The trade-off is longer time to a correct stop.
| Thinking | Correct stops | False stops | Stop time |
|---|---|---|---|
| Off | 9/12 | 12/90 | 1.7 s |
| Low | 11/12 | 10/90 | 3.6 s |
| Medium | 10/12 | 3/90 | 3.5 s |
| High | 12/12 | 3/90 | 3.7 s |
| XHigh | 12/12 | 1/90 | 4.6 s |
| Max | 12/12 | 0/90 | 6.1 s |
False stops abandon solvable delivery tasks. Time is the median for correct stops, excluding timeouts.
Method
Each setting receives 30 delivery cases, repeated three times with frozen inputs, tools and acceptance checks. Cost includes failed attempts; tokens are input plus output per attempt; latency is the median for accepted results. Unknown complete cost or token measurements remain blank. These are controlled text, schedule and function tests, not scanned-plan takeoffs or the full Bidlo app. Method, uncertainty and source corrections · JSON · CSV.
Choosing a setting
Luna fits work where low cost and explicit verification matter. Medium improves Knowledge substantially over Low; Max is stronger for Analysis.