bench: matrix v4 adds page_setup_truth; correctness suite grows to 32 with three settings checks (A/B vs v1.5.1: 26 -> 32 / 32)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
2026-09-22 13:07:31 +03:00
parent 02dbb133b5
commit c2942d3c94
8 changed files with 230 additions and 40 deletions
@@ -104,7 +104,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a

 **Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.

-**Benchmark matrix:** `benchmarks/tasks_v4.py` is the current set (16 tasks; `--matrix v3` / `--matrix v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks or the v4 `pid_roundtrip` — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (29 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
+**Benchmark matrix:** `benchmarks/tasks_v4.py` is the current set (17 tasks; `--matrix v3` / `--matrix v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks or the v4 `pid_roundtrip` / `page_setup_truth` — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (32 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).

 ### Adding a New Tool

@@ -27,7 +27,7 @@ Live through COM on Windows, or headless through ezdxf anywhere — one typed co

 </div>

-> **v1.5 release snapshot:** 204 tools · 8 resources · 5 prompt templates · 3259 collected tests.
+> **v1.5 release snapshot:** 204 tools · 8 resources · 5 prompt templates · 3282 collected tests.
 > 204 is the **registered** count; a default install advertises 199 over `tools/list`,
 > because `ENABLE_3D` is unset. `system_about` is the runtime authority.

@@ -155,18 +155,18 @@ Self-measurement, produced by scripts in [`benchmarks/`](https://github.com/U-C4

 ### Correctness — every release re-proves itself

-29 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.
+32 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.

 | Version | Checks passing | Pass rate | Fixed | Regressed |
 |---|---:|---:|---:|---:|
-| v1.5.1 *(baseline)* | 26 / 29 | 89.7 % | — | — |
-| **v1.6.0-dev** *(this branch)* | **29 / 29** | **100 %** | 3 | **0** |
+| v1.5.1 *(baseline)* | 26 / 32 | 81.2 % | — | — |
+| **v1.6.0-dev** *(this branch)* | **32 / 32** | **100 %** | 6 | **0** |

-The three fixed rows are the P&ID checks, new capability (`miss → pass`): `block_define` round-trips an ATTDEF, `pid_tag_parse` reads `FIC-101`, and `pid_graph` counts the example sheet's four edges. The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.
+The six fixed rows are new capability (`miss → pass`): the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.

 ### The task matrix — six tasks that can fail

-An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds a sixth that the P&ID reader can fail on its own.
+An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds two more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness.

 | Task | Verified against |
 |---|---|
@@ -176,6 +176,7 @@ An earlier matrix scored this server 10/10, which carried no information: every
 | `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 |
 | `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives |
 | `pid_roundtrip` | the example sheet drawn by `pid_from_spec`, read back by `pid_graph`, which never sees the spec: 5 nodes, 4 edges, 0 dangling, `confidence_min` 1.0, `FIC-101` wired to `FCV-101` |
+| `page_setup_truth` | ISO A3 landscape applied to Layout1, plotted through `batch_plot`, and the sheet read back from the PDF's `/MediaBox`: 420 × 297 mm, not the setter's return value |

 ### Headless performance

@@ -46,7 +46,7 @@ python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --
 python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
 ```

-The release-machine ezdxf self-check is **16/16 (100.0)** on the v4 matrix
+The release-machine ezdxf self-check is **17/17 (100.0)** on the v4 matrix
 (`--matrix v3` / `--matrix v2` reproduce the earlier sets).
 Repository stars and raw tool counts do not contribute to the score. Adapter
 registration lives in `competitors.yaml`.
@@ -87,17 +87,18 @@ zero we invented would be indistinguishable from a zero we measured.
 only described, and `--publish` writes the artifact-path-free file that
 `results/published/` holds.

-### Matrix v4 (v1.6) — the P&ID round trip
+### Matrix v4 (v1.6) — the P&ID round trip and the page-setup truth

-`tasks_v4.py` keeps the v3 fifteen unchanged and adds one task the reader can
-fail on its own:
+`tasks_v4.py` keeps the v3 fifteen unchanged and adds two tasks that can each
+fail on their own:

 | Task | Category | Verified against |
 |---|---|---|
 | `pid_roundtrip` | pid | the example spec drawn through `pid_from_spec`'s code, then read back by the graph builder, which never sees the spec: 5 nodes, 4 edges, 0 dangling ends, `confidence_min` 1.0, zero critique issues, an instrument index of exactly `FIC-101` wired to `FCV-101`, and the two authored line numbers back on the two process lines while the two unnumbered lines carry sequence-built numbers from XDATA |
+| `page_setup_truth` | pagesetup | `page_setup_apply("Layout1", ISO_A3 landscape 1:1)` then `batch_plot`; the PDF's own `/MediaBox` must parse to 420 × 297 mm (±0.5), the page-setup list must say `ISO_A3` / `landscape` / `[420, 297]`, and the ctb must be a known one |

-The competitor reports carry no result for it, for the same reason they carry
-none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.
+The competitor reports carry no result for either, for the same reason they
+carry none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.

 ## Live competitor lane (v1.4)

@@ -216,19 +217,24 @@ python benchmarks/compare_versions.py --json results.json

 ### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)

-29 checks, ezdxf backend, one subprocess per check. Machine-readable report:
+32 checks, ezdxf backend, one subprocess per check. Machine-readable report:
 [`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).

 | Version | Checks passing | Pass rate | Fixed | Regressed |
 |---------|----------------|-----------|-------|-----------|
-| **v1.5.1** (baseline)     | 26 / 29 | 89.7 % | — | — |
-| **v1.6.0-dev** (this branch) | 29 / 29 | 100 % | 3 | **0** |
+| **v1.5.1** (baseline)     | 26 / 32 | 81.2 % | — | — |
+| **v1.6.0-dev** (this branch) | 32 / 32 | 100 % | 6 | **0** |

-The three are the P&ID checks Track A added — `pid_block_define_attdef_roundtrip`,
-`pid_tag_parse_fic`, `pid_graph_edge_count` — all `miss → pass`: v1.5.1 has
-none of the methods. Every one of the 26 checks it was released on still
-passes, so the track added capability without moving a number it had already
-earned.
+The six are all `miss → pass` — v1.5.1 has none of the methods: the three
+P&ID checks track A added (`pid_block_define_attdef_roundtrip`,
+`pid_tag_parse_fic`, `pid_graph_edge_count`) and the three settings checks of
+track E (`settings_dimstyle_iso25_values`: the ISO-25 preset reaches the
+DIMSTYLE table with ISO 129-1's numbers, read back through ezdxf and not
+through the lister; `settings_layer_state_roundtrip`: save → change → restore
+puts the layer table back and the state survives save/reopen as an XRECORD in
+the file; `settings_pdf_mediabox_a3`: the plotted PDF's own `/MediaBox` reads
+420 × 297 mm). Every one of the 26 checks it was released on still passes, so
+the two tracks added capability without moving a number it had already earned.

 ### Result — v1.5.1 vs v1.5.0

@@ -431,6 +431,54 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
            [],
        )

+    async def _task_page_setup_truth(self):
+        """ISO A3 landscape on Layout1, plotted, and the sheet read back from the PDF.
+
+        The number that matters is parsed from the file's ``/MediaBox`` — the
+        setter's ``applied`` block is what we asked for, the page-setup list is
+        what the LAYOUT now says, and the PDF is what a printer would get. All
+        three have to agree with ISO 216's 420 × 297.
+        """
+        from engineering.standards.papers import resolve_page_setup
+        from engineering.standards.plot import batch_plot
+
+        setup = resolve_page_setup(
+            "ISO_A3",
+            orientation="landscape",
+            plot_style="monochrome.ctb",
+            scale="1:1",
+        )
+        applied = await self.backend.page_setup_apply("Layout1", setup)
+        listed = (await self.backend.page_setup_list("Layout1"))[0]
+        out_dir = self.artifact_dir / "page_setup_truth"
+        out_dir.mkdir(parents=True, exist_ok=True)
+        report = await batch_plot(self.backend, ["Layout1"], str(out_dir), "{drawing}-{layout}.pdf")
+        row = report["sheets"][0]
+        width_mm, height_mm = (float(v) for v in row["mediabox_mm"])
+        size_listed = tuple(float(v) for v in listed["size_mm"])
+        passed = (
+            applied["ok"] is True
+            and applied["plot_style_known"] is True
+            and listed["paper"] == "ISO_A3"
+            and listed["orientation"] == "landscape"
+            and size_listed == (420.0, 297.0)
+            and abs(width_mm - 420.0) < 0.5
+            and abs(height_mm - 297.0) < 0.5
+            and row["bytes"] > 0
+        )
+        return (
+            passed,
+            {
+                "paper": listed["paper"],
+                "orientation": listed["orientation"],
+                "size_mm_listed": list(size_listed),
+                "mediabox_mm": [width_mm, height_mm],
+                "pdf_bytes": row["bytes"],
+                "plot_style_known": applied["plot_style_known"],
+            },
+            [str(row["path"])],
+        )
+
    async def _task_auditable_delivery(self):
        await ensure_engineering_layers(self.backend)
        await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
@@ -321,6 +321,74 @@ async def pid_graph_edge_count():
    return len(result["graph"]["edges"]) == 4 and result["graph"]["stats"]["dangling"] == 0


+# ── Settings and environment (v1.6, track E) ─────────────────────────────────
+
+
+async def settings_dimstyle_iso25_values():
+    """The ISO-25 preset reaches the DIMSTYLE table with ISO 129-1's numbers.
+
+    Read back through ezdxf directly, not through `dimstyle_list`, so the
+    check cannot be satisfied by a list that echoes its own input.
+    """
+    from engineering.standards.dimstyles import resolve_dimstyle
+
+    b = await _b()
+    await b.dimstyle_create("ISO-25", resolve_dimstyle("iso-25", None), set_current=True)
+    style = b._doc.dimstyles.get("ISO-25")
+    listed = {row["name"]: row for row in await b.dimstyle_list()}
+    return (
+        abs(float(style.dxf.dimtxt) - 2.5) < 1e-9
+        and abs(float(style.dxf.dimasz) - 2.5) < 1e-9
+        and abs(float(style.dxf.dimexe) - 1.25) < 1e-9
+        and abs(float(style.dxf.dimexo) - 0.625) < 1e-9
+        and abs(float(style.dxf.dimgap) - 0.625) < 1e-9
+        and int(style.dxf.dimtad) == 1
+        and int(style.dxf.dimdsep) == ord(",")
+        and str(style.dxf.dimtxsty) == "ISOCP"
+        and b._doc.header["$DIMSTYLE"] == "ISO-25"
+        and listed["ISO-25"]["current"] is True
+        and listed["ISO-25"]["values"]["DIMDSEP"] == ","
+    )
+
+
+async def settings_layer_state_roundtrip():
+    """Save → change → restore puts the table back, and the state survives a
+    save/reopen because it is an XRECORD in the file, not server memory."""
+    b = await _b()
+    await b.layer_create("LS_A", color=1)
+    await b.layer_create("LS_B", color=2)
+    saved = await b.layer_state_save("bench", description="A/B round trip")
+    await b.layer_modify("LS_A", color=5)
+    await b.layer_freeze("LS_B")
+    restored = await b.layer_state_restore("bench")
+    layers = {layer.name: layer for layer in await b.layer_list()}
+    path = os.path.join(tempfile.mkdtemp(), "ls.dxf")
+    await b.drawing_save_as(path, "dxf")
+    await b.drawing_open(path)
+    names_after_reopen = {row["name"] for row in await b.layer_state_list()}
+    return (
+        saved["ok"] is True
+        and restored["missing_layers"] == []
+        and restored["new_layers"] == []
+        and layers["LS_A"].color == 1
+        and layers["LS_B"].is_frozen is False
+        and "bench" in names_after_reopen
+    )
+
+
+async def settings_pdf_mediabox_a3():
+    """Page setup evidence is the PDF's own /MediaBox (spec §8.3)."""
+    from engineering.standards.papers import resolve_page_setup
+    from engineering.standards.plot import batch_plot
+
+    b = await _b()
+    await b.page_setup_apply("Layout1", resolve_page_setup("ISO_A3"))
+    out_dir = tempfile.mkdtemp()
+    report = await batch_plot(b, ["Layout1"], out_dir, "{drawing}-{layout}.pdf")
+    width_mm, height_mm = report["sheets"][0]["mediabox_mm"]
+    return abs(width_mm - 420.0) < 0.5 and abs(height_mm - 297.0) < 0.5
+
+
 CHECKS = {
    "core_line_length": (core_line_length, "Core"),
    "core_circle_radius": (core_circle_radius, "Core"),
@@ -351,6 +419,9 @@ CHECKS = {
    "pid_block_define_attdef_roundtrip": (pid_block_define_attdef_roundtrip, "P&ID"),
    "pid_tag_parse_fic": (pid_tag_parse_fic, "P&ID"),
    "pid_graph_edge_count": (pid_graph_edge_count, "P&ID"),
+    "settings_dimstyle_iso25_values": (settings_dimstyle_iso25_values, "Settings"),
+    "settings_layer_state_roundtrip": (settings_layer_state_roundtrip, "Settings"),
+    "settings_pdf_mediabox_a3": (settings_pdf_mediabox_a3, "Settings"),
 }


@@ -1,12 +1,12 @@
 {
  "summary": {
    "baseline_ref": "v1.5.1",
-    "total": 29,
+    "total": 32,
    "old_pass": 26,
-    "new_pass": 29,
-    "old_pct": 89.7,
+    "new_pass": 32,
+    "old_pct": 81.2,
    "new_pct": 100.0,
-    "fixed": 3,
+    "fixed": 6,
    "regressed": 0
  },
  "rows": [
@@ -183,6 +183,24 @@
      "category": "P&ID",
      "old": "miss",
      "new": "pass"
+    },
+    {
+      "check": "settings_dimstyle_iso25_values",
+      "category": "Settings",
+      "old": "miss",
+      "new": "pass"
+    },
+    {
+      "check": "settings_layer_state_roundtrip",
+      "category": "Settings",
+      "old": "miss",
+      "new": "pass"
+    },
+    {
+      "check": "settings_pdf_mediabox_a3",
+      "category": "Settings",
+      "old": "miss",
+      "new": "pass"
    }
  ]
 }
@@ -1,10 +1,16 @@
-"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A).
+"""The v4 task matrix: v3 plus what 1.6 added.

-``pid_roundtrip`` draws a five-item P&ID through the same code the tools call
-and reads it back: the edge count, the absence of dangling ends and the
-instrument index are each derived independently of the drawing code. The
-published competitor reports were never asked this question; the chart shows
-``not_run`` for them, not zero.
+``pid_roundtrip`` (track A) draws a five-item P&ID through the same code the
+tools call and reads it back: the edge count, the absence of dangling ends
+and the instrument index are each derived independently of the drawing code.
+
+``page_setup_truth`` (track E) applies an ISO A3 landscape page setup to a
+layout, plots it through ``batch_plot`` and reads the sheet size back from
+the PDF's own ``/MediaBox`` — the setup is verified by the file, not by the
+setter's return value.
+
+The published competitor reports were never asked either question; the chart
+shows ``not_run`` for them, not zero.
 """

 from __future__ import annotations
@@ -12,7 +18,7 @@ from __future__ import annotations
 from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
 from benchmarks.tasks_v3 import TASKS_V3

-#: Added in v1.6.0. The reader can fail this one on its own.
+#: Added in v1.6.0. Each can fail on its own.
 NEW_TASKS_V4: tuple[TaskSpec, ...] = (
    TaskSpec(
        "pid_roundtrip",
@@ -20,6 +26,12 @@ NEW_TASKS_V4: tuple[TaskSpec, ...] = (
        "Draw a P&ID from a spec and read back a matching graph, index and line list",
        1.0,
    ),
+    TaskSpec(
+        "page_setup_truth",
+        "pagesetup",
+        "Apply an ISO A3 page setup, plot it, and read the sheet size back from the PDF",
+        1.0,
+    ),
 )

 TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4
@@ -1,23 +1,28 @@
-"""Track A — the v4 task matrix.
+# tests/test_benchmark_v4.py
+"""The v4 task matrix: v3 plus what 1.6 added.

-v4 adds one task, ``pid_roundtrip``: draw a P&ID through the same code the
-tools call and read it back through the graph builder, so the edge count, the
-dangling count and the instrument index are each derived independently of the
-drawing code. The published competitor reports were never asked this question
-and the chart shows them ``not_run`` — the same rule v3 set for its five.
+Track A added ``pid_roundtrip``; track E adds ``page_setup_truth`` — apply an
+ISO A3 landscape page setup, plot through ``batch_plot`` and read the sheet
+size back from the PDF's own ``/MediaBox``. The published competitor reports
+were never asked either question and the chart shows them ``not_run`` — the
+same rule v3 set for its five.
 """

 from __future__ import annotations

+import pytest
+
 from benchmarks.tasks_v3 import TASKS_V3
 from benchmarks.tasks_v4 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V4, TASKS_V4, task_by_id


 def test_v4_extends_v3_without_touching_it():
    assert TASKS_V4[: len(TASKS_V3)] == TASKS_V3
-    assert [t.task_id for t in NEW_TASKS_V4] == ["pid_roundtrip"]
+    assert [t.task_id for t in NEW_TASKS_V4] == ["pid_roundtrip", "page_setup_truth"]
+    assert len(TASKS_V4) == 17
    assert MATRICES["v3"] == TASKS_V3 and MATRICES["v4"] == TASKS_V4 and DEFAULT_MATRIX == "v4"
    assert task_by_id("pid_roundtrip").category == "pid"
+    assert task_by_id("page_setup_truth").category == "pagesetup"


 def test_the_adapter_implements_every_v4_task():
@@ -32,4 +37,33 @@ def test_correctness_suite_has_the_pid_checks():

    for name in ("pid_block_define_attdef_roundtrip", "pid_tag_parse_fic", "pid_graph_edge_count"):
        assert CHECKS[name][1] == "P&ID"
-    assert len(CHECKS) == 29
+
+
+def test_correctness_suite_has_the_settings_checks():
+    from benchmarks.correctness_suite import CHECKS
+
+    for name in (
+        "settings_dimstyle_iso25_values",
+        "settings_layer_state_roundtrip",
+        "settings_pdf_mediabox_a3",
+    ):
+        assert CHECKS[name][1] == "Settings"
+    assert len(CHECKS) == 32
+
+
+@pytest.mark.asyncio  # asyncio_mode is strict; the four tests above are sync
+async def test_page_setup_truth_passes_headlessly(tmp_path):
+    """The benchmark task itself, run once here so a regression in page setup
+    or the PDF read-back fails the suite and not only the matrix."""
+    from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
+
+    adapter = AutoCADMCPProAdapter(backend="ezdxf")
+    await adapter.setup(tmp_path)
+    try:
+        await adapter._reset()
+        passed, metrics, _artifacts = await adapter._task_page_setup_truth()
+    finally:
+        await adapter.cleanup()
+    assert passed, metrics
+    assert abs(metrics["mediabox_mm"][0] - 420.0) < 0.5
+    assert abs(metrics["mediabox_mm"][1] - 297.0) < 0.5