bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks

tasks_v4.py extends v3 with one task the reader can fail on its own: the
example spec is drawn through pid_from_spec's code and read back by the
graph builder, which never sees the spec. The runner, the chart and the
doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier
sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3"
explicitly now that the default has moved.

The plan's adapter check expected the two unnumbered lines to come back
with no line number; draw_line builds one from the default number_format
(bare sequence when size/service/spec/insulation are empty), so the task
asserts the two authored numbers on the two process lines and sequence-
built XDATA numbers on the other two instead.

The published A/B report still covers 26 checks; Task 21 re-runs it
against v1.5.1, which is what clears the remaining doc-number drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd

benchmarks/README.md

@@ -41,12 +41,13 @@ Unsupported tasks remain in the fixed-matrix denominator with score zero,
 preventing partial implementations from receiving an inflated score.
 
 ```bash
-python -m benchmarks.run_competitors --list --matrix v3
-python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v3 --json
+python -m benchmarks.run_competitors --list --matrix v4
+python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v4 --json
 python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
 ```
 
-The release-machine ezdxf self-check is **15/15 (100.0)** on the v3 matrix.
+The release-machine ezdxf self-check is **16/16 (100.0)** on the v4 matrix
+(`--matrix v3` / `--matrix v2` reproduce the earlier sets).
 Repository stars and raw tool counts do not contribute to the score. Adapter
 registration lives in `competitors.yaml`.
 
@@ -86,6 +87,18 @@ zero we invented would be indistinguishable from a zero we measured.
 only described, and `--publish` writes the artifact-path-free file that
 `results/published/` holds.
 
+### Matrix v4 (v1.6) — the P&ID round trip
+
+`tasks_v4.py` keeps the v3 fifteen unchanged and adds one task the reader can
+fail on its own:
+
+| Task | Category | Verified against |
+|---|---|---|
+| `pid_roundtrip` | pid | the example spec drawn through `pid_from_spec`'s code, then read back by the graph builder, which never sees the spec: 5 nodes, 4 edges, 0 dangling ends, `confidence_min` 1.0, zero critique issues, an instrument index of exactly `FIC-101` wired to `FCV-101`, and the two authored line numbers back on the two process lines while the two unnumbered lines carry sequence-built numbers from XDATA |
+
+The competitor reports carry no result for it, for the same reason they carry
+none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.
+
 ## Live competitor lane (v1.4)
 
 Two competitor adapters now execute the exact same task matrix, black-box over
@@ -203,7 +216,7 @@ python benchmarks/compare_versions.py --json results.json
 
 ### Result — v1.5.1 vs v1.5.0 (release gate)
 
-26 checks, ezdxf backend, one subprocess per check. Machine-readable report:
+29 checks, ezdxf backend, one subprocess per check. Machine-readable report:
 [`results/published/ab-v1.5.0-vs-v1.5.1.json`](results/published/ab-v1.5.0-vs-v1.5.1.json).
 
 | Version | Checks passing | Pass rate | Fixed | Regressed |

benchmarks/adapters/autocad_mcp_pro.py

@@ -1,4 +1,4 @@
-"""Reference adapter for this repository, covering the v3 task matrix.
+"""Reference adapter for this repository, covering the v4 task matrix.
 
 Every task verifies against a value worked out independently of the code under
 test — a closed-form area, a count of entities placed on purpose, a token
@@ -12,7 +12,7 @@ import math
 from pathlib import Path
 
 from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
-from benchmarks.tasks_v3 import TaskSpec
+from benchmarks.tasks_v4 import TaskSpec
 from engineering.delivery import deliver_drawing
 from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
 from engineering.refiner import refine_drawing
@@ -364,6 +364,65 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
             [],
         )
 
+    async def _task_pid_roundtrip(self):
+        """Five symbols, four lines, read back with nothing dangling.
+
+        The spec is drawn through the same code ``pid_from_spec`` calls; the
+        graph, the instrument index and the line list are then read back from
+        the drawing by the reader, which never sees the spec. Every expected
+        number below is counted from the spec by hand.
+        """
+        from engineering.layers import apply_layer_set
+        from engineering.pid.deliverables import instrument_index, line_list
+        from engineering.pid.spec import EXAMPLE_SPEC, run_spec
+
+        await apply_layer_set(self.backend, "pid")
+        result = await run_spec(self.backend, EXAMPLE_SPEC)
+        graph = result["graph"]
+        index = instrument_index(graph)
+        lines = line_list(graph)
+        # The spec numbers its two process lines; the signal line and the
+        # connector line are left to the default ``number_format``, which with
+        # no size/service/spec/insulation collapses to the bare sequence
+        # number. Both must come back through XDATA, not a label search.
+        authored = "100-P-1001-CS1"
+        numbers = {(r["from_tag"], r["to_tag"]): r["line_number"] for r in lines}
+        authored_ok = (
+            numbers.get(("P-101", "FCV-101")) == authored
+            and numbers.get(("FCV-101", "V-201")) == authored
+        )
+        sequenced = [r for r in lines if r["line_number"] != authored]
+        sequenced_ok = len(sequenced) == 2 and all(
+            r["line_number"].isdigit() and r["number_source"] == "xdata" for r in sequenced
+        )
+        passed = (
+            len(graph["nodes"]) == 5
+            and len(graph["edges"]) == 4
+            and graph["stats"]["dangling"] == 0
+            and graph["stats"]["confidence_min"] == 1.0
+            and result["critique"] == []
+            and [r["tag"] for r in index] == ["FIC-101"]
+            and index[0]["connected_to"] == "FCV-101"
+            and len(lines) == 4
+            and authored_ok
+            and sequenced_ok
+        )
+        return (
+            passed,
+            {
+                "nodes": len(graph["nodes"]),
+                "edges": len(graph["edges"]),
+                "dangling": graph["stats"]["dangling"],
+                "confidence_min": graph["stats"]["confidence_min"],
+                "critique_issues": len(result["critique"]),
+                "instrument_index_rows": len(index),
+                "line_list_rows": len(lines),
+                "authored_line_numbers_kept": authored_ok,
+                "unnumbered_lines_sequenced": sequenced_ok,
+            },
+            [],
+        )
+
     async def _task_auditable_delivery(self):
         await ensure_engineering_layers(self.backend)
         await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")

benchmarks/correctness_suite.py

@@ -293,6 +293,34 @@ async def radius_dim_ignores_the_leader_length():
     return all(abs(v - 20.0) < 1e-6 for v in values)
 
 
+# ── P&ID (v1.6, track A) ──────────────────────────────────────────────────────
+
+
+async def pid_block_define_attdef_roundtrip():
+    b = await _b()
+    await b.block_define(
+        "PID_T",
+        [{"type": "line", "x1": 0, "y1": 0, "x2": 8, "y2": 0}],
+        [{"tag": "TAG", "x": 0, "y": 3, "height": 2.5}],
+    )
+    ref = await b.block_insert("PID_T", 5, 5, attributes={"TAG": "HV-1"})
+    return await b.block_get_attributes(ref.handle) == {"TAG": "HV-1"}
+
+
+async def pid_tag_parse_fic():
+    from engineering.pid.tags import parse_tag
+
+    return parse_tag("FIC-101")["description"] == "Flow Indicating Controller"
+
+
+async def pid_graph_edge_count():
+    from engineering.pid.spec import EXAMPLE_SPEC, run_spec
+
+    b = await _b()
+    result = await run_spec(b, EXAMPLE_SPEC)
+    return len(result["graph"]["edges"]) == 4 and result["graph"]["stats"]["dangling"] == 0
+
+
 CHECKS = {
     "core_line_length": (core_line_length, "Core"),
     "core_circle_radius": (core_circle_radius, "Core"),
@@ -320,6 +348,9 @@ CHECKS = {
     "two_vertex_circle_has_area": (two_vertex_circle_has_area, "Measurement"),
     "diameter_dim_measures_the_diameter": (diameter_dim_measures_the_diameter, "Dimensions"),
     "radius_dim_ignores_the_leader_length": (radius_dim_ignores_the_leader_length, "Dimensions"),
+    "pid_block_define_attdef_roundtrip": (pid_block_define_attdef_roundtrip, "P&ID"),
+    "pid_tag_parse_fic": (pid_tag_parse_fic, "P&ID"),
+    "pid_graph_edge_count": (pid_graph_edge_count, "P&ID"),
 }
 
 

benchmarks/render_matrix_chart.py

@@ -13,7 +13,7 @@ import json
 from pathlib import Path
 from typing import Any
 
-from benchmarks.tasks_v3 import TASKS_V3
+from benchmarks.tasks_v4 import TASKS_V4
 
 ROOT = Path(__file__).resolve().parents[1]
 DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
@@ -66,8 +66,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
 
     matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
 
-    task_ids = [task.task_id for task in TASKS_V3]
-    task_labels = [f"{task.task_id}  ({task.category})" for task in TASKS_V3]
+    task_ids = [task.task_id for task in TASKS_V4]
+    task_labels = [f"{task.task_id}  ({task.category})" for task in TASKS_V4]
     servers = [report["adapter"] for report in reports]
     status_by_server = {
         report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}

benchmarks/run_competitors.py

@@ -16,7 +16,7 @@ from pathlib import Path
 
 from backends.capability import UnsupportedCapabilityError
 from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
-from benchmarks.tasks_v3 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
+from benchmarks.tasks_v4 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
 
 
 def _load_registry() -> dict[str, dict]:

benchmarks/tasks_v4.py

@@ -0,0 +1,38 @@
+"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A).
+
+``pid_roundtrip`` draws a five-item P&ID through the same code the tools call
+and reads it back: the edge count, the absence of dangling ends and the
+instrument index are each derived independently of the drawing code. The
+published competitor reports were never asked this question; the chart shows
+``not_run`` for them, not zero.
+"""
+
+from __future__ import annotations
+
+from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
+from benchmarks.tasks_v3 import TASKS_V3
+
+#: Added in v1.6.0. The reader can fail this one on its own.
+NEW_TASKS_V4: tuple[TaskSpec, ...] = (
+    TaskSpec(
+        "pid_roundtrip",
+        "pid",
+        "Draw a P&ID from a spec and read back a matching graph, index and line list",
+        1.0,
+    ),
+)
+
+TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4
+
+#: Runner-selectable task sets. v2 and v3 stay addressable so an earlier
+#: report can be reproduced exactly rather than only described.
+MATRICES: dict[str, tuple[TaskSpec, ...]] = {"v2": TASKS_V2, "v3": TASKS_V3, "v4": TASKS_V4}
+
+DEFAULT_MATRIX = "v4"
+
+
+def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
+    for task in MATRICES[matrix]:
+        if task.task_id == task_id:
+            return task
+    raise KeyError(f"{task_id} is not in matrix {matrix}")