CivilCADBench
Benchmarking Multimodal LLMs for Civil Engineering Drawing Sets
Real infrastructure drawing sets and source CAD files containing over 250,000 entities. CivilCADBench evaluates whether frontier models can read a set, rebuild the editable CAD file hidden behind a monochrome print, and design a new sheet under updated parameters.
Models redraw the print. They lose the file.
We evaluate seven frontier MLLMs at high reasoning effort. The strongest model reads and traces printed content accurately, yet recovers far less of the hidden structure that printing discards.
Where frontier models actually fail
Inspecting generated DXF files side by side with the designers' source files exposes three distinct failure modes that pixel metrics miss.
L1 Single-Page Reconstruction: Geometry F1 100, Layer F1 12
On the bar schedule (top), GPT-6 Astra reproduces every cell accurately while GPT-5.6 Sol mis-scales the table and loses most text. On the pier detail (bottom), both GPT-6 models trace every line (geometry F1 100), yet place reinforcement on the wrong layer (layer-aware F1 12 and 11).
Three task families, in rising difficulty
The tasks test reading drawing values, rebuilding editable CAD files from monochrome prints, and designing a new sheet from a reference CAD file and updated parameters.
Tie every printed value to the object it describes
Six-option multi-select questions (1–4 correct, chance 1/56) reviewed by three civil engineering experts. Every wrong option encodes one named misbinding refuted by the drawing.
- Single-Sheet1,800 items over one page across six binding types: instance, cross-view, table, datum, scope, and view.
- Few-Sheet383 multi-sheet parent questions given evidence pages plus two look-alike decoy sheets.
- Full-Set330 parents posed over the whole ~30-page unit, where the model searches pages with a zoomable viewer.
- ProbesShort-answer read and bind probes in Few-Sheet (1,953) and Full-Set (1,989) settings separately test fact reading and cross-page reasoning.
Rebuild the editable CAD file behind the pixels
Given only a 200 dpi monochrome raster with no vector or text layer, the agent writes Python / ezdxf code in a sandbox to output a layered DXF with live dimensions.
- L1 · Page232 bridge and tunnel sheets across small (108), medium (96), and large (28) entity-count tiers.
- L2 · Set28 complete drawing sets (5–14 pages each, 230 pages total) requiring cross-sheet consistency.
L1 input · monochrome raster
Target · 9-layer DXF file
Derive a new sheet under updated parameters
Mirroring real design-office workflows, a derived sibling sheet is withheld as ground truth while the agent receives the reference DXF, context rasters, and new member parameters.
- L3 · Design19 parametric design tasks where bar counts, lengths, geometry, dimensions, and schedule totals must all update coherently.
Given · reference sibling DXF
Withheld · designers' target sheetSandboxed agent loop, deterministic file-to-file scoring
Agents write and execute code inside an isolated container, preview their rendered DXF against the input, and submit once. Scoring uses zero LLM or VLM judges.
ezdxf tools.- Geometry F1curves sampled at 0.25 mm, matched within 1 mm on paper0.25
- Layer-aware F1matched geometry points on the exact same target layer0.30
- Layer sharefraction of entities placed on the nine allowed layers0.10
- Text F1strings within 6 mm, similarity ≥ 0.8, identical numbers0.20
- Dimension F1dimension labels matched by value and position0.15
Inspect the benchmark data yourself
A sample of every task, open in the browser: the drawing pages, the questions with their answers and the reason each wrong option is wrong, and the target CAD files coloured by layer.