Introducing Bid Bench
Bid Bench evaluates AI models in the context of heavy civil construction and estimating workflows.
A model can return a correct answer and still be a poor choice for a contractor's daily work. It may take too long, use too many tokens, or cost more than another setting that produces the same result. Bid Bench measures those trade-offs across the models available in Bidlo.
Six workflows
Documents tests reconciliation of plans, specifications and addenda: which source controls, what changed, and which quantities are separately paid.
Takeoffs tests quantities from cross-section schedules, exclusions and volume conversions, including truck capacity limits.
Analysis tests whether a model selects the right fields, filters and record versions. A letting date and a project start date are different inputs.
Knowledge tests construction terminology and estimating scenarios, including the conditions that change an otherwise familiar answer.
Apps tests contractor tools through executable behavior checks: quantity calculators, quote comparisons, procurement and crew scheduling.
Predictions tests bidder probabilities and unit-price estimates against recorded bid outcomes, with simple historical baselines.
Quality at a useful cost
Bid Bench 1.1 covers 18 models and 80 model/thinking configurations. The delivery score weights the first five categories equally. Predictions and stopping behavior have separate measures.
More thinking is a setting to test, not a guarantee of better output. Each report compares the settings the app actually supports and includes cost, token use and completion time. The downloads include the full roster.
Beyond routine tasks
Each delivery category progresses through six authored difficulty tiers. The hardest cases add interacting constraints: cancelling individual amendment lines, overlapping takeoff exclusions, historical record cutoffs, and exact scheduling and stock-cutting decisions.
The Stress tier was added after earlier cases showed a ceiling, then applied uniformly. Some settings still pass every delivery case; broader generalization remains unproven.
Separate cases test whether a model stops when an essential input is unavailable or the constraints cannot be satisfied. We score the identified blocker, time to a correct stop, and false stops on solvable work.
Repeatable tests
Inputs, model settings, tool budgets and acceptance checks are frozen. Every case runs three times. Checks use source references, numerical tolerances, expected records, exact choices and hidden application tests. Failed attempts remain in the denominator.
This edition uses controlled, authored delivery cases. Documents use structured extracted text; takeoffs use schedules; apps are domain functions. Predictions use a small retrospective Texas cohort, with engineer-estimate pseudo-bidders removed. These results do not establish performance on scanned plans or prospective bids.
The reports include uncertainty and downloadable measurements. Method and scope describes the tests, formulas and source correction.