AEC-Geometric-Bench: scoring geometric extraction from construction drawings

We released AEC-Geometric-Bench, a public benchmark for geometric and symbolic data extraction from PDF construction drawings.

The question it asks is the one a quantity takeoff actually needs answered: what is on the sheet, where is it, and can you name the rooms?

What is scored

Six systems are scored under one uniform condition: one PDF is submitted, one response is returned, and everything the system reports is scored.

The full corpus, AEC-Geometric-Bench-312, is 312 single-page architectural sheets issued for construction. It carries 24,771 annotated object instances, 59,328 wall shapes, and 16,112 area shapes. Fifteen of those sheets are released with their ground truth, so the result can be recomputed rather than taken on trust.

Object F1 is over eight classes at IoU 0.50, pooled across sheets. On identical ground truth Kamai reaches 0.929. The strongest general-purpose model in the set, Gemini 3.7 Flash, reaches 0.062.

Results on AEC-Geometric-Bench-312

Segmentation is more favorable to the models than object detection: they recover 0.612 to 0.799 area pixel F1 against Kamai's 0.983. Area instance F1 collapses to 0.078 to 0.227. They find the floor area without resolving it into discrete rooms, which is what a takeoff needs.

Results on the fifteen released sheets

These fifteen sit at the corpus average for Kamai (0.929 object F1). Per-class cells in a 15-sheet table rest on few instances. The corpus figures are the ones to cite.

Reproduce it

The repo includes the fifteen redacted sheets, the CVAT annotations, a frozen hashed taxonomy, and a scorer.

cd scoring
python3 score.py --pred example-predictions/kamai --name "Kamai v1 Aug26"

That command reproduces the released-subset row for objects, wall pixel, and area pixel. Area instance reads 0.941 in the paper and 0.940 in this script: the paper's harness computes that one IoU on rasterized instance maps; the script computes it analytically on the merged polygons. Every other figure is identical.

Ground truth, scoring rules, and the prediction format are in the README.

Limitations

• Ground truth is single-annotator.
• Fifteen sheets is a small sample. Cite the 312-sheet corpus.
• Two of the six systems ran through an agent harness that records neither token counts nor latency, so no cost figure is given for them.
• This benchmark was produced by the vendor of one of the systems evaluated. The protocol was fixed before measuring, the taxonomy was hashed, every system is scored on identical ground truth by the same matcher with no per-system exceptions, and the ground truth was annotated by a professional team rather than by the authors.

Data is CC BY-NC 4.0. The scoring code is Apache 2.0, so a commercial system can be scored against the benchmark.

If you want to see Kamai on one of your sheets, open the app or talk to sales.