diff --git a/CLAUDE.md b/CLAUDE.md index dcfad51..7c797c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -104,7 +104,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a **Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so. -**Benchmark matrix:** `benchmarks/tasks_v4.py` is the current set (16 tasks; `--matrix v3` / `--matrix v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks or the v4 `pid_roundtrip` — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (29 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`). +**Benchmark matrix:** `benchmarks/tasks_v4.py` is the current set (17 tasks; `--matrix v3` / `--matrix v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks or the v4 `pid_roundtrip` / `page_setup_truth` — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (32 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`). ### Adding a New Tool diff --git a/README.md b/README.md index 41e3f2d..55af7a4 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ Live through COM on Windows, or headless through ezdxf anywhere — one typed co -> **v1.5 release snapshot:** 204 tools · 8 resources · 5 prompt templates · 3259 collected tests. +> **v1.5 release snapshot:** 204 tools · 8 resources · 5 prompt templates · 3282 collected tests. > 204 is the **registered** count; a default install advertises 199 over `tools/list`, > because `ENABLE_3D` is unset. `system_about` is the runtime authority. @@ -155,18 +155,18 @@ Self-measurement, produced by scripts in [`benchmarks/`](https://github.com/U-C4 ### Correctness — every release re-proves itself -29 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run. +32 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run. | Version | Checks passing | Pass rate | Fixed | Regressed | |---|---:|---:|---:|---:| -| v1.5.1 *(baseline)* | 26 / 29 | 89.7 % | — | — | -| **v1.6.0-dev** *(this branch)* | **29 / 29** | **100 %** | 3 | **0** | +| v1.5.1 *(baseline)* | 26 / 32 | 81.2 % | — | — | +| **v1.6.0-dev** *(this branch)* | **32 / 32** | **100 %** | 6 | **0** | -The three fixed rows are the P&ID checks, new capability (`miss → pass`): `block_define` round-trips an ATTDEF, `pid_tag_parse` reads `FIC-101`, and `pid_graph` counts the example sheet's four edges. The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings. +The six fixed rows are new capability (`miss → pass`): the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings. ### The task matrix — six tasks that can fail -An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds a sixth that the P&ID reader can fail on its own. +An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds two more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness. | Task | Verified against | |---|---| @@ -176,6 +176,7 @@ An earlier matrix scored this server 10/10, which carried no information: every | `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 | | `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives | | `pid_roundtrip` | the example sheet drawn by `pid_from_spec`, read back by `pid_graph`, which never sees the spec: 5 nodes, 4 edges, 0 dangling, `confidence_min` 1.0, `FIC-101` wired to `FCV-101` | +| `page_setup_truth` | ISO A3 landscape applied to Layout1, plotted through `batch_plot`, and the sheet read back from the PDF's `/MediaBox`: 420 × 297 mm, not the setter's return value | ### Headless performance diff --git a/benchmarks/README.md b/benchmarks/README.md index 58d83fb..ddd7f0d 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -46,7 +46,7 @@ python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf -- python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json ``` -The release-machine ezdxf self-check is **16/16 (100.0)** on the v4 matrix +The release-machine ezdxf self-check is **17/17 (100.0)** on the v4 matrix (`--matrix v3` / `--matrix v2` reproduce the earlier sets). Repository stars and raw tool counts do not contribute to the score. Adapter registration lives in `competitors.yaml`. @@ -87,17 +87,18 @@ zero we invented would be indistinguishable from a zero we measured. only described, and `--publish` writes the artifact-path-free file that `results/published/` holds. -### Matrix v4 (v1.6) — the P&ID round trip +### Matrix v4 (v1.6) — the P&ID round trip and the page-setup truth -`tasks_v4.py` keeps the v3 fifteen unchanged and adds one task the reader can -fail on its own: +`tasks_v4.py` keeps the v3 fifteen unchanged and adds two tasks that can each +fail on their own: | Task | Category | Verified against | |---|---|---| | `pid_roundtrip` | pid | the example spec drawn through `pid_from_spec`'s code, then read back by the graph builder, which never sees the spec: 5 nodes, 4 edges, 0 dangling ends, `confidence_min` 1.0, zero critique issues, an instrument index of exactly `FIC-101` wired to `FCV-101`, and the two authored line numbers back on the two process lines while the two unnumbered lines carry sequence-built numbers from XDATA | +| `page_setup_truth` | pagesetup | `page_setup_apply("Layout1", ISO_A3 landscape 1:1)` then `batch_plot`; the PDF's own `/MediaBox` must parse to 420 × 297 mm (±0.5), the page-setup list must say `ISO_A3` / `landscape` / `[420, 297]`, and the ctb must be a known one | -The competitor reports carry no result for it, for the same reason they carry -none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly. +The competitor reports carry no result for either, for the same reason they +carry none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly. ## Live competitor lane (v1.4) @@ -216,19 +217,24 @@ python benchmarks/compare_versions.py --json results.json ### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate) -29 checks, ezdxf backend, one subprocess per check. Machine-readable report: +32 checks, ezdxf backend, one subprocess per check. Machine-readable report: [`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json). | Version | Checks passing | Pass rate | Fixed | Regressed | |---------|----------------|-----------|-------|-----------| -| **v1.5.1** (baseline) | 26 / 29 | 89.7 % | — | — | -| **v1.6.0-dev** (this branch) | 29 / 29 | 100 % | 3 | **0** | +| **v1.5.1** (baseline) | 26 / 32 | 81.2 % | — | — | +| **v1.6.0-dev** (this branch) | 32 / 32 | 100 % | 6 | **0** | -The three are the P&ID checks Track A added — `pid_block_define_attdef_roundtrip`, -`pid_tag_parse_fic`, `pid_graph_edge_count` — all `miss → pass`: v1.5.1 has -none of the methods. Every one of the 26 checks it was released on still -passes, so the track added capability without moving a number it had already -earned. +The six are all `miss → pass` — v1.5.1 has none of the methods: the three +P&ID checks track A added (`pid_block_define_attdef_roundtrip`, +`pid_tag_parse_fic`, `pid_graph_edge_count`) and the three settings checks of +track E (`settings_dimstyle_iso25_values`: the ISO-25 preset reaches the +DIMSTYLE table with ISO 129-1's numbers, read back through ezdxf and not +through the lister; `settings_layer_state_roundtrip`: save → change → restore +puts the layer table back and the state survives save/reopen as an XRECORD in +the file; `settings_pdf_mediabox_a3`: the plotted PDF's own `/MediaBox` reads +420 × 297 mm). Every one of the 26 checks it was released on still passes, so +the two tracks added capability without moving a number it had already earned. ### Result — v1.5.1 vs v1.5.0 diff --git a/benchmarks/adapters/autocad_mcp_pro.py b/benchmarks/adapters/autocad_mcp_pro.py index 92ed439..c88b523 100644 --- a/benchmarks/adapters/autocad_mcp_pro.py +++ b/benchmarks/adapters/autocad_mcp_pro.py @@ -431,6 +431,54 @@ class AutoCADMCPProAdapter(BenchmarkAdapter): [], ) + async def _task_page_setup_truth(self): + """ISO A3 landscape on Layout1, plotted, and the sheet read back from the PDF. + + The number that matters is parsed from the file's ``/MediaBox`` — the + setter's ``applied`` block is what we asked for, the page-setup list is + what the LAYOUT now says, and the PDF is what a printer would get. All + three have to agree with ISO 216's 420 × 297. + """ + from engineering.standards.papers import resolve_page_setup + from engineering.standards.plot import batch_plot + + setup = resolve_page_setup( + "ISO_A3", + orientation="landscape", + plot_style="monochrome.ctb", + scale="1:1", + ) + applied = await self.backend.page_setup_apply("Layout1", setup) + listed = (await self.backend.page_setup_list("Layout1"))[0] + out_dir = self.artifact_dir / "page_setup_truth" + out_dir.mkdir(parents=True, exist_ok=True) + report = await batch_plot(self.backend, ["Layout1"], str(out_dir), "{drawing}-{layout}.pdf") + row = report["sheets"][0] + width_mm, height_mm = (float(v) for v in row["mediabox_mm"]) + size_listed = tuple(float(v) for v in listed["size_mm"]) + passed = ( + applied["ok"] is True + and applied["plot_style_known"] is True + and listed["paper"] == "ISO_A3" + and listed["orientation"] == "landscape" + and size_listed == (420.0, 297.0) + and abs(width_mm - 420.0) < 0.5 + and abs(height_mm - 297.0) < 0.5 + and row["bytes"] > 0 + ) + return ( + passed, + { + "paper": listed["paper"], + "orientation": listed["orientation"], + "size_mm_listed": list(size_listed), + "mediabox_mm": [width_mm, height_mm], + "pdf_bytes": row["bytes"], + "plot_style_known": applied["plot_style_known"], + }, + [str(row["path"])], + ) + async def _task_auditable_delivery(self): await ensure_engineering_layers(self.backend) await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY") diff --git a/benchmarks/correctness_suite.py b/benchmarks/correctness_suite.py index 7196aac..3bc5020 100644 --- a/benchmarks/correctness_suite.py +++ b/benchmarks/correctness_suite.py @@ -321,6 +321,74 @@ async def pid_graph_edge_count(): return len(result["graph"]["edges"]) == 4 and result["graph"]["stats"]["dangling"] == 0 +# ── Settings and environment (v1.6, track E) ───────────────────────────────── + + +async def settings_dimstyle_iso25_values(): + """The ISO-25 preset reaches the DIMSTYLE table with ISO 129-1's numbers. + + Read back through ezdxf directly, not through `dimstyle_list`, so the + check cannot be satisfied by a list that echoes its own input. + """ + from engineering.standards.dimstyles import resolve_dimstyle + + b = await _b() + await b.dimstyle_create("ISO-25", resolve_dimstyle("iso-25", None), set_current=True) + style = b._doc.dimstyles.get("ISO-25") + listed = {row["name"]: row for row in await b.dimstyle_list()} + return ( + abs(float(style.dxf.dimtxt) - 2.5) < 1e-9 + and abs(float(style.dxf.dimasz) - 2.5) < 1e-9 + and abs(float(style.dxf.dimexe) - 1.25) < 1e-9 + and abs(float(style.dxf.dimexo) - 0.625) < 1e-9 + and abs(float(style.dxf.dimgap) - 0.625) < 1e-9 + and int(style.dxf.dimtad) == 1 + and int(style.dxf.dimdsep) == ord(",") + and str(style.dxf.dimtxsty) == "ISOCP" + and b._doc.header["$DIMSTYLE"] == "ISO-25" + and listed["ISO-25"]["current"] is True + and listed["ISO-25"]["values"]["DIMDSEP"] == "," + ) + + +async def settings_layer_state_roundtrip(): + """Save → change → restore puts the table back, and the state survives a + save/reopen because it is an XRECORD in the file, not server memory.""" + b = await _b() + await b.layer_create("LS_A", color=1) + await b.layer_create("LS_B", color=2) + saved = await b.layer_state_save("bench", description="A/B round trip") + await b.layer_modify("LS_A", color=5) + await b.layer_freeze("LS_B") + restored = await b.layer_state_restore("bench") + layers = {layer.name: layer for layer in await b.layer_list()} + path = os.path.join(tempfile.mkdtemp(), "ls.dxf") + await b.drawing_save_as(path, "dxf") + await b.drawing_open(path) + names_after_reopen = {row["name"] for row in await b.layer_state_list()} + return ( + saved["ok"] is True + and restored["missing_layers"] == [] + and restored["new_layers"] == [] + and layers["LS_A"].color == 1 + and layers["LS_B"].is_frozen is False + and "bench" in names_after_reopen + ) + + +async def settings_pdf_mediabox_a3(): + """Page setup evidence is the PDF's own /MediaBox (spec §8.3).""" + from engineering.standards.papers import resolve_page_setup + from engineering.standards.plot import batch_plot + + b = await _b() + await b.page_setup_apply("Layout1", resolve_page_setup("ISO_A3")) + out_dir = tempfile.mkdtemp() + report = await batch_plot(b, ["Layout1"], out_dir, "{drawing}-{layout}.pdf") + width_mm, height_mm = report["sheets"][0]["mediabox_mm"] + return abs(width_mm - 420.0) < 0.5 and abs(height_mm - 297.0) < 0.5 + + CHECKS = { "core_line_length": (core_line_length, "Core"), "core_circle_radius": (core_circle_radius, "Core"), @@ -351,6 +419,9 @@ CHECKS = { "pid_block_define_attdef_roundtrip": (pid_block_define_attdef_roundtrip, "P&ID"), "pid_tag_parse_fic": (pid_tag_parse_fic, "P&ID"), "pid_graph_edge_count": (pid_graph_edge_count, "P&ID"), + "settings_dimstyle_iso25_values": (settings_dimstyle_iso25_values, "Settings"), + "settings_layer_state_roundtrip": (settings_layer_state_roundtrip, "Settings"), + "settings_pdf_mediabox_a3": (settings_pdf_mediabox_a3, "Settings"), } diff --git a/benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json b/benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json index 45109b1..ace10dc 100644 --- a/benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json +++ b/benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json @@ -1,12 +1,12 @@ { "summary": { "baseline_ref": "v1.5.1", - "total": 29, + "total": 32, "old_pass": 26, - "new_pass": 29, - "old_pct": 89.7, + "new_pass": 32, + "old_pct": 81.2, "new_pct": 100.0, - "fixed": 3, + "fixed": 6, "regressed": 0 }, "rows": [ @@ -183,6 +183,24 @@ "category": "P&ID", "old": "miss", "new": "pass" + }, + { + "check": "settings_dimstyle_iso25_values", + "category": "Settings", + "old": "miss", + "new": "pass" + }, + { + "check": "settings_layer_state_roundtrip", + "category": "Settings", + "old": "miss", + "new": "pass" + }, + { + "check": "settings_pdf_mediabox_a3", + "category": "Settings", + "old": "miss", + "new": "pass" } ] } \ No newline at end of file diff --git a/benchmarks/tasks_v4.py b/benchmarks/tasks_v4.py index 80b37ba..e2b54e9 100644 --- a/benchmarks/tasks_v4.py +++ b/benchmarks/tasks_v4.py @@ -1,10 +1,16 @@ -"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A). +"""The v4 task matrix: v3 plus what 1.6 added. -``pid_roundtrip`` draws a five-item P&ID through the same code the tools call -and reads it back: the edge count, the absence of dangling ends and the -instrument index are each derived independently of the drawing code. The -published competitor reports were never asked this question; the chart shows -``not_run`` for them, not zero. +``pid_roundtrip`` (track A) draws a five-item P&ID through the same code the +tools call and reads it back: the edge count, the absence of dangling ends +and the instrument index are each derived independently of the drawing code. + +``page_setup_truth`` (track E) applies an ISO A3 landscape page setup to a +layout, plots it through ``batch_plot`` and reads the sheet size back from +the PDF's own ``/MediaBox`` — the setup is verified by the file, not by the +setter's return value. + +The published competitor reports were never asked either question; the chart +shows ``not_run`` for them, not zero. """ from __future__ import annotations @@ -12,7 +18,7 @@ from __future__ import annotations from benchmarks.tasks_v2 import TASKS_V2, TaskSpec from benchmarks.tasks_v3 import TASKS_V3 -#: Added in v1.6.0. The reader can fail this one on its own. +#: Added in v1.6.0. Each can fail on its own. NEW_TASKS_V4: tuple[TaskSpec, ...] = ( TaskSpec( "pid_roundtrip", @@ -20,6 +26,12 @@ NEW_TASKS_V4: tuple[TaskSpec, ...] = ( "Draw a P&ID from a spec and read back a matching graph, index and line list", 1.0, ), + TaskSpec( + "page_setup_truth", + "pagesetup", + "Apply an ISO A3 page setup, plot it, and read the sheet size back from the PDF", + 1.0, + ), ) TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4 diff --git a/tests/test_benchmark_v4.py b/tests/test_benchmark_v4.py index 240e0a6..4c33aa4 100644 --- a/tests/test_benchmark_v4.py +++ b/tests/test_benchmark_v4.py @@ -1,23 +1,28 @@ -"""Track A — the v4 task matrix. +# tests/test_benchmark_v4.py +"""The v4 task matrix: v3 plus what 1.6 added. -v4 adds one task, ``pid_roundtrip``: draw a P&ID through the same code the -tools call and read it back through the graph builder, so the edge count, the -dangling count and the instrument index are each derived independently of the -drawing code. The published competitor reports were never asked this question -and the chart shows them ``not_run`` — the same rule v3 set for its five. +Track A added ``pid_roundtrip``; track E adds ``page_setup_truth`` — apply an +ISO A3 landscape page setup, plot through ``batch_plot`` and read the sheet +size back from the PDF's own ``/MediaBox``. The published competitor reports +were never asked either question and the chart shows them ``not_run`` — the +same rule v3 set for its five. """ from __future__ import annotations +import pytest + from benchmarks.tasks_v3 import TASKS_V3 from benchmarks.tasks_v4 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V4, TASKS_V4, task_by_id def test_v4_extends_v3_without_touching_it(): assert TASKS_V4[: len(TASKS_V3)] == TASKS_V3 - assert [t.task_id for t in NEW_TASKS_V4] == ["pid_roundtrip"] + assert [t.task_id for t in NEW_TASKS_V4] == ["pid_roundtrip", "page_setup_truth"] + assert len(TASKS_V4) == 17 assert MATRICES["v3"] == TASKS_V3 and MATRICES["v4"] == TASKS_V4 and DEFAULT_MATRIX == "v4" assert task_by_id("pid_roundtrip").category == "pid" + assert task_by_id("page_setup_truth").category == "pagesetup" def test_the_adapter_implements_every_v4_task(): @@ -32,4 +37,33 @@ def test_correctness_suite_has_the_pid_checks(): for name in ("pid_block_define_attdef_roundtrip", "pid_tag_parse_fic", "pid_graph_edge_count"): assert CHECKS[name][1] == "P&ID" - assert len(CHECKS) == 29 + + +def test_correctness_suite_has_the_settings_checks(): + from benchmarks.correctness_suite import CHECKS + + for name in ( + "settings_dimstyle_iso25_values", + "settings_layer_state_roundtrip", + "settings_pdf_mediabox_a3", + ): + assert CHECKS[name][1] == "Settings" + assert len(CHECKS) == 32 + + +@pytest.mark.asyncio # asyncio_mode is strict; the four tests above are sync +async def test_page_setup_truth_passes_headlessly(tmp_path): + """The benchmark task itself, run once here so a regression in page setup + or the PDF read-back fails the suite and not only the matrix.""" + from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter + + adapter = AutoCADMCPProAdapter(backend="ezdxf") + await adapter.setup(tmp_path) + try: + await adapter._reset() + passed, metrics, _artifacts = await adapter._task_page_setup_truth() + finally: + await adapter.cleanup() + assert passed, metrics + assert abs(metrics["mediabox_mm"][0] - 420.0) < 0.5 + assert abs(metrics["mediabox_mm"][1] - 297.0) < 0.5