anatoly/autocad-mcp-pro
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

bench: matrix v4 adds page_setup_truth; correctness suite grows to 32 with three settings checks (A/B vs v1.5.1: 26 -> 32 / 32)

Browse Source 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-22 13:07:31 +03:00
co-authored by Claude Opus 5
parent 02dbb133b5
commit c2942d3c94
8 changed files with 230 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/README.md
+20 -14
View File
@@ -46,7 +46,7 @@ python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --
python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
```
The release-machine ezdxf self-check is **16/16 (100.0)** on the v4 matrix
The release-machine ezdxf self-check is **17/17 (100.0)** on the v4 matrix
(`--matrix v3` / `--matrix v2` reproduce the earlier sets).
Repository stars and raw tool counts do not contribute to the score. Adapter
registration lives in `competitors.yaml`.
@@ -87,17 +87,18 @@ zero we invented would be indistinguishable from a zero we measured.
only described, and `--publish` writes the artifact-path-free file that
`results/published/` holds.
### Matrix v4 (v1.6) — the P&ID round trip
### Matrix v4 (v1.6) — the P&ID round trip and the page-setup truth
`tasks_v4.py` keeps the v3 fifteen unchanged and adds one task the reader can
fail on its own:
`tasks_v4.py` keeps the v3 fifteen unchanged and adds two tasks that can each
fail on their own:
| Task | Category | Verified against |
|---|---|---|
| `pid_roundtrip` | pid | the example spec drawn through `pid_from_spec`'s code, then read back by the graph builder, which never sees the spec: 5 nodes, 4 edges, 0 dangling ends, `confidence_min` 1.0, zero critique issues, an instrument index of exactly `FIC-101` wired to `FCV-101`, and the two authored line numbers back on the two process lines while the two unnumbered lines carry sequence-built numbers from XDATA |
| `page_setup_truth` | pagesetup | `page_setup_apply("Layout1", ISO_A3 landscape 1:1)` then `batch_plot`; the PDF's own `/MediaBox` must parse to 420 × 297 mm (±0.5), the page-setup list must say `ISO_A3` / `landscape` / `[420, 297]`, and the ctb must be a known one |
The competitor reports carry no result for it, for the same reason they carry
none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.
The competitor reports carry no result for either, for the same reason they
carry none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.
## Live competitor lane (v1.4)
@@ -216,19 +217,24 @@ python benchmarks/compare_versions.py --json results.json
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
29 checks, ezdxf backend, one subprocess per check. Machine-readable report:
32 checks, ezdxf backend, one subprocess per check. Machine-readable report:
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
| Version | Checks passing | Pass rate | Fixed | Regressed |
|---------|----------------|-----------|-------|-----------|
| **v1.5.1** (baseline) | 26 / 29 | 89.7 % | — | — |
| **v1.6.0-dev** (this branch) | 29 / 29 | 100 % | 3 | **0** |
| **v1.5.1** (baseline) | 26 / 32 | 81.2 % | — | — |
| **v1.6.0-dev** (this branch) | 32 / 32 | 100 % | 6 | **0** |
The three are the P&ID checks Track A added — `pid_block_define_attdef_roundtrip`,
`pid_tag_parse_fic`, `pid_graph_edge_count` — all `miss → pass`: v1.5.1 has
none of the methods. Every one of the 26 checks it was released on still
passes, so the track added capability without moving a number it had already
earned.
The six are all `miss → pass` — v1.5.1 has none of the methods: the three
P&ID checks track A added (`pid_block_define_attdef_roundtrip`,
`pid_tag_parse_fic`, `pid_graph_edge_count`) and the three settings checks of
track E (`settings_dimstyle_iso25_values`: the ISO-25 preset reaches the
DIMSTYLE table with ISO 129-1's numbers, read back through ezdxf and not
through the lister; `settings_layer_state_roundtrip`: save → change → restore
puts the layer table back and the state survives save/reopen as an XRECORD in
the file; `settings_pdf_mediabox_a3`: the plotted PDF's own `/MediaBox` reads
420 × 297 mm). Every one of the 26 checks it was released on still passes, so
the two tracks added capability without moving a number it had already earned.
### Result — v1.5.1 vs v1.5.0
benchmarks/adapters/autocad_mcp_pro.py
+48
View File
@@ -431,6 +431,54 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
[],
)
async def _task_page_setup_truth(self):
"""ISO A3 landscape on Layout1, plotted, and the sheet read back from the PDF.
The number that matters is parsed from the file's ``/MediaBox`` — the
setter's ``applied`` block is what we asked for, the page-setup list is
what the LAYOUT now says, and the PDF is what a printer would get. All
three have to agree with ISO 216's 420 × 297.
"""
from engineering.standards.papers import resolve_page_setup
from engineering.standards.plot import batch_plot
setup = resolve_page_setup(
"ISO_A3",
orientation="landscape",
plot_style="monochrome.ctb",
scale="1:1",
)
applied = await self.backend.page_setup_apply("Layout1", setup)
listed = (await self.backend.page_setup_list("Layout1"))[0]
out_dir = self.artifact_dir / "page_setup_truth"
out_dir.mkdir(parents=True, exist_ok=True)
report = await batch_plot(self.backend, ["Layout1"], str(out_dir), "{drawing}-{layout}.pdf")
row = report["sheets"][0]
width_mm, height_mm = (float(v) for v in row["mediabox_mm"])
size_listed = tuple(float(v) for v in listed["size_mm"])
passed = (
applied["ok"] is True
and applied["plot_style_known"] is True
and listed["paper"] == "ISO_A3"
and listed["orientation"] == "landscape"
and size_listed == (420.0, 297.0)
and abs(width_mm - 420.0) < 0.5
and abs(height_mm - 297.0) < 0.5
and row["bytes"] > 0
)
return (
passed,
{
"paper": listed["paper"],
"orientation": listed["orientation"],
"size_mm_listed": list(size_listed),
"mediabox_mm": [width_mm, height_mm],
"pdf_bytes": row["bytes"],
"plot_style_known": applied["plot_style_known"],
},
[str(row["path"])],
)
async def _task_auditable_delivery(self):
await ensure_engineering_layers(self.backend)
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
benchmarks/correctness_suite.py
+71
View File
@@ -321,6 +321,74 @@ async def pid_graph_edge_count():
return len(result["graph"]["edges"]) == 4 and result["graph"]["stats"]["dangling"] == 0
# ── Settings and environment (v1.6, track E) ─────────────────────────────────
async def settings_dimstyle_iso25_values():
"""The ISO-25 preset reaches the DIMSTYLE table with ISO 129-1's numbers.
Read back through ezdxf directly, not through `dimstyle_list`, so the
check cannot be satisfied by a list that echoes its own input.
"""
from engineering.standards.dimstyles import resolve_dimstyle
b = await _b()
await b.dimstyle_create("ISO-25", resolve_dimstyle("iso-25", None), set_current=True)
style = b._doc.dimstyles.get("ISO-25")
listed = {row["name"]: row for row in await b.dimstyle_list()}
return (
abs(float(style.dxf.dimtxt) - 2.5) < 1e-9
and abs(float(style.dxf.dimasz) - 2.5) < 1e-9
and abs(float(style.dxf.dimexe) - 1.25) < 1e-9
and abs(float(style.dxf.dimexo) - 0.625) < 1e-9
and abs(float(style.dxf.dimgap) - 0.625) < 1e-9
and int(style.dxf.dimtad) == 1
and int(style.dxf.dimdsep) == ord(",")
and str(style.dxf.dimtxsty) == "ISOCP"
and b._doc.header["$DIMSTYLE"] == "ISO-25"
and listed["ISO-25"]["current"] is True
and listed["ISO-25"]["values"]["DIMDSEP"] == ","
)
async def settings_layer_state_roundtrip():
"""Save → change → restore puts the table back, and the state survives a
save/reopen because it is an XRECORD in the file, not server memory."""
b = await _b()
await b.layer_create("LS_A", color=1)
await b.layer_create("LS_B", color=2)
saved = await b.layer_state_save("bench", description="A/B round trip")
await b.layer_modify("LS_A", color=5)
await b.layer_freeze("LS_B")
restored = await b.layer_state_restore("bench")
layers = {layer.name: layer for layer in await b.layer_list()}
path = os.path.join(tempfile.mkdtemp(), "ls.dxf")
await b.drawing_save_as(path, "dxf")
await b.drawing_open(path)
names_after_reopen = {row["name"] for row in await b.layer_state_list()}
return (
saved["ok"] is True
and restored["missing_layers"] == []
and restored["new_layers"] == []
and layers["LS_A"].color == 1
and layers["LS_B"].is_frozen is False
and "bench" in names_after_reopen
)
async def settings_pdf_mediabox_a3():
"""Page setup evidence is the PDF's own /MediaBox (spec §8.3)."""
from engineering.standards.papers import resolve_page_setup
from engineering.standards.plot import batch_plot
b = await _b()
await b.page_setup_apply("Layout1", resolve_page_setup("ISO_A3"))
out_dir = tempfile.mkdtemp()
report = await batch_plot(b, ["Layout1"], out_dir, "{drawing}-{layout}.pdf")
width_mm, height_mm = report["sheets"][0]["mediabox_mm"]
return abs(width_mm - 420.0) < 0.5 and abs(height_mm - 297.0) < 0.5
CHECKS = {
"core_line_length": (core_line_length, "Core"),
"core_circle_radius": (core_circle_radius, "Core"),
@@ -351,6 +419,9 @@ CHECKS = {
"pid_block_define_attdef_roundtrip": (pid_block_define_attdef_roundtrip, "P&ID"),
"pid_tag_parse_fic": (pid_tag_parse_fic, "P&ID"),
"pid_graph_edge_count": (pid_graph_edge_count, "P&ID"),
"settings_dimstyle_iso25_values": (settings_dimstyle_iso25_values, "Settings"),
"settings_layer_state_roundtrip": (settings_layer_state_roundtrip, "Settings"),
"settings_pdf_mediabox_a3": (settings_pdf_mediabox_a3, "Settings"),
}
benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json
+22 -4
View File
@@ -1,12 +1,12 @@
{
"summary": {
"baseline_ref": "v1.5.1",
"total": 29,
"total": 32,
"old_pass": 26,
"new_pass": 29,
"old_pct": 89.7,
"new_pass": 32,
"old_pct": 81.2,
"new_pct": 100.0,
"fixed": 3,
"fixed": 6,
"regressed": 0
},
"rows": [
@@ -183,6 +183,24 @@
"category": "P&ID",
"old": "miss",
"new": "pass"
},
{
"check": "settings_dimstyle_iso25_values",
"category": "Settings",
"old": "miss",
"new": "pass"
},
{
"check": "settings_layer_state_roundtrip",
"category": "Settings",
"old": "miss",
"new": "pass"
},
{
"check": "settings_pdf_mediabox_a3",
"category": "Settings",
"old": "miss",
"new": "pass"
}
]
}
benchmarks/tasks_v4.py
+19 -7
View File
@@ -1,10 +1,16 @@
"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A).
"""The v4 task matrix: v3 plus what 1.6 added.
``pid_roundtrip`` draws a five-item P&ID through the same code the tools call
and reads it back: the edge count, the absence of dangling ends and the
instrument index are each derived independently of the drawing code. The
published competitor reports were never asked this question; the chart shows
``not_run`` for them, not zero.
``pid_roundtrip`` (track A) draws a five-item P&ID through the same code the
tools call and reads it back: the edge count, the absence of dangling ends
and the instrument index are each derived independently of the drawing code.
``page_setup_truth`` (track E) applies an ISO A3 landscape page setup to a
layout, plots it through ``batch_plot`` and reads the sheet size back from
the PDF's own ``/MediaBox`` — the setup is verified by the file, not by the
setter's return value.
The published competitor reports were never asked either question; the chart
shows ``not_run`` for them, not zero.
"""
from __future__ import annotations
@@ -12,7 +18,7 @@ from __future__ import annotations
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
from benchmarks.tasks_v3 import TASKS_V3
#: Added in v1.6.0. The reader can fail this one on its own.
#: Added in v1.6.0. Each can fail on its own.
NEW_TASKS_V4: tuple[TaskSpec, ...] = (
TaskSpec(
"pid_roundtrip",
@@ -20,6 +26,12 @@ NEW_TASKS_V4: tuple[TaskSpec, ...] = (
"Draw a P&ID from a spec and read back a matching graph, index and line list",
1.0,
),
TaskSpec(
"page_setup_truth",
"pagesetup",
"Apply an ISO A3 page setup, plot it, and read the sheet size back from the PDF",
1.0,
),
)
TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4