bench: matrix v5 adds mech_assembly; the correctness suite grows with the mechanical and sheet checks

The evidence for tracks B and G, in the shape tracks A and E used.

- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
  addressable so an earlier report reproduces exactly rather than being only
  described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
  built from the part model and the sheet standard, through the code the
  tools call (apply_titleblock, draw_part/add_view/dimension_part,
  insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
  two numbers that cannot be negotiated: drawing_critique(focus=None) returns
  zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
  the v5 matrix). The adapter returns the focuses that fired, so a
  regression names itself, and a test drops one balloon to prove the gate
  fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
  that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
  20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
  through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
  410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
  40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
  k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
  quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
  (25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
  4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
  the drawing, a linked balloon, an xref attached / listed / detached, a
  31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
  every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
  the first call after Documents.Add is refused) and tracks its documents by
  the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
  code:
  * ezdxf dimension_diameter put the text beyond the SECOND chord point;
    ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
    from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
    front-view diameter stood on its own chain dimension. ezdxf now follows
    ActiveX.
  * draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
    missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
    every target layer before the first entity.
  * bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
    (ObjectName AcDbBlockReference) - the live parts list was empty.
  * COM _entity_info reported no dimension text_position: ModelSpace.Item
    narrows to IAcadEntity, which has no TextPosition (measured
    AttributeError); read through _com_unnarrow (TextPosition measured on
    AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
    diameter collided with its own bore.
  * pid graph raised KeyError 'start' mid-critique on a LINE whose point read
    AutoCAD refused while busy; such a line is left out.
  * the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
  re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
  the task count from the v5 matrix, the token table is remeasured with a
  TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
  collected-tests figure is retaken (3990).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd

tests/test_benchmark_v4.py

@@ -48,7 +48,8 @@ def test_correctness_suite_has_the_settings_checks():
         "settings_pdf_mediabox_a3",
     ):
         assert CHECKS[name][1] == "Settings"
-    assert len(CHECKS) == 32
+    # v4 left the suite at 32; tracks B + G grew it to 39 (tests/test_benchmark_v5.py).
+    assert len(CHECKS) == 39
 
 
 @pytest.mark.asyncio  # asyncio_mode is strict; the four tests above are sync

tests/test_benchmark_v5.py

@@ -0,0 +1,112 @@
+# tests/test_benchmark_v5.py
+"""The v5 task matrix: v4 plus what tracks B and G added.
+
+``mech_assembly`` is roadmap criterion 3 — a flange-coupling sheet built from
+the tools, gated on two numbers that cannot be negotiated:
+``drawing_critique(focus=None)`` returns zero issues and ``drawing_finalize``
+scores at least 90. The published competitor reports were never asked it and
+the chart shows them ``not_run``, the rule v3 set for its five.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from benchmarks.tasks_v4 import TASKS_V4
+from benchmarks.tasks_v5 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V5, TASKS_V5, task_by_id
+
+
+def test_v5_extends_v4_without_touching_it():
+    assert TASKS_V5[: len(TASKS_V4)] == TASKS_V4
+    assert [t.task_id for t in NEW_TASKS_V5] == ["mech_assembly"]
+    assert len(TASKS_V5) == 18
+    assert MATRICES["v4"] == TASKS_V4 and MATRICES["v5"] == TASKS_V5
+    assert DEFAULT_MATRIX == "v5"
+    assert task_by_id("mech_assembly").category == "mech"
+
+
+def test_the_adapter_implements_every_v5_task():
+    from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
+
+    for task in TASKS_V5:
+        assert hasattr(AutoCADMCPProAdapter, f"_task_{task.task_id}"), task.task_id
+
+
+def test_the_suite_carries_the_mechanical_and_sheet_checks():
+    """Six from spec 14, plus mech_thread_unrepresented_is_caught: a quality
+    gate that never fires is not a gate, the way dim_overlap_critique_fires
+    already guards the dimension focus."""
+    from benchmarks.correctness_suite import CHECKS
+
+    for name in (
+        "mech_part_roundtrip",
+        "mech_section_hatch_area",
+        "mech_iso286_on_dimension",
+        "mech_thread_unrepresented_is_caught",
+        "std_part_iso4014_m12",
+    ):
+        assert CHECKS[name][1] == "Mechanical"
+    assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet"
+    assert CHECKS["bom_balloon_link"][1] == "Sheet"
+    assert len(CHECKS) == 39
+
+
+@pytest.mark.asyncio
+async def test_mech_assembly_passes_headlessly(tmp_path):
+    """The benchmark task itself, run once here so a regression in the part
+    drawer, the sheet or the critique fails the suite and not only the matrix."""
+    from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
+
+    adapter = AutoCADMCPProAdapter(backend="ezdxf")
+    await adapter.setup(tmp_path)
+    try:
+        await adapter._reset()
+        passed, metrics, _artifacts = await adapter._task_mech_assembly()
+    finally:
+        await adapter.cleanup()
+    assert passed, metrics
+    assert metrics["critique_issues"] == 0, metrics["critique_focuses"]
+    assert metrics["score"] >= 90.0
+
+
+@pytest.mark.asyncio
+async def test_mech_assembly_fails_loudly_when_the_critique_finds_anything(tmp_path, monkeypatch):
+    """The gate has to be able to fail. One parts-list row left without its
+    balloon is enough: ISO 7573 gives every row an item reference, and the
+    critique names the focus that fired rather than only shading the score."""
+    from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
+    from engineering.sheet import bom
+
+    real = bom.add_balloon
+
+    async def only_the_first(backend, *, item, **kw):
+        if int(item) == 1:
+            return await real(backend, item=item, **kw)
+        return {"ok": True, "skipped": True}
+
+    monkeypatch.setattr(bom, "add_balloon", only_the_first)
+    adapter = AutoCADMCPProAdapter(backend="ezdxf")
+    await adapter.setup(tmp_path)
+    try:
+        await adapter._reset()
+        passed, metrics, _artifacts = await adapter._task_mech_assembly()
+    finally:
+        await adapter.cleanup()
+    assert passed is False
+    assert "mech_bom_balloon_mismatch" in metrics["critique_focuses"]
+
+
+@pytest.mark.asyncio
+async def test_the_section_hatch_area_check_measures_the_drawing():
+    """1200 mm^2 is computed by hand from the model, and read back through
+    analysis_measure_entity — a number the drawing code cannot fake."""
+    from benchmarks import correctness_suite
+
+    assert await correctness_suite.mech_section_hatch_area() is True
+
+
+@pytest.mark.asyncio
+async def test_the_iso5457_frame_check_passes():
+    from benchmarks import correctness_suite
+
+    assert await correctness_suite.sheet_frame_iso5457_a3() is True

tests/test_com_backend.py

@@ -546,6 +546,27 @@ def _gate_valve_block():
     return _Block([body_left, body_right, tag])
 
 
+def test_a_dimension_reports_where_its_text_stands():
+    """The dim_overlap critique reads `text_position`; without it a diameter
+    dimension's reference point was its bounding-box centre - the circle's
+    centre - and every outside diameter collided with its own bore on the live
+    engine (measured by scripts/smoke_mech_com.py, AutoCAD 2026)."""
+    from backends.com_backend import _entity_info
+
+    outside = _Entity(
+        "AcDbDiametricDimension",
+        GetBoundingBox=lambda: ((178.0, 170.0, 0.0), (184.0, 233.0, 0.0)),
+        TextPosition=(181.0, 230.0, 0.0),
+        Handle="1C1",
+        Layer="DIM",
+        color=256,
+        Linetype="ByLayer",
+        Visible=True,
+    )
+    props = _entity_info(outside).properties
+    assert props["text_position"] == [181.0, 230.0]
+
+
 def test_com_geometry_bbox_is_the_drawn_body_without_the_attribute():
     from backends.com_backend import _entity_info
 

tests/test_dimensions.py

@@ -57,6 +57,17 @@ async def test_a_diameter_dimension_measures_the_diameter(backend):
     assert dim.get_measurement() == pytest.approx(40.0)
 
 
+async def test_the_text_stands_beyond_the_first_chord_point_as_activex_puts_it(backend):
+    """AddDimDiametric(ChordPoint, FarChordPoint, LeaderLength), measured on
+    AutoCAD 2026: a 50 mm diameter from (x, 170) to (x, 220) with leader 10
+    had its text at (x, 160) - beyond the FIRST point. Both engines follow it."""
+    info = await backend.dimension_diameter(0, 25, 0, -25, leader_length=10.0)
+    dim = backend._doc.entitydb.get(info.handle)
+    text = dim.dxf.text_midpoint
+    assert text.y == pytest.approx(35.0)
+    assert text.x == pytest.approx(0.0)
+
+
 async def test_the_leader_length_moves_the_text_not_the_measurement(backend):
     """The regression in one assertion: leader_length must not be measurable."""
     near = await backend.dimension_diameter(-20, 0, 20, 0, leader_length=5.0)

tests/test_engineering_validator.py

@@ -101,6 +101,21 @@ async def test_validator_catches_fake_dimension_text(backend, tmp_path):
     assert "fake_dimension_text" in _codes(result)
 
 
+async def test_a_standard_part_designation_is_not_a_fake_dimension():
+    """A parts-list row names a part: "ISO 4014 - M12x60" is the ISO 4014
+    designation grammar, not a measurement. Found by the v5 mech_assembly
+    benchmark, where every text parts list cost the sheet a warning. The
+    exemption is the designation only - a measurement beside it still flags."""
+    from engineering.validator import looks_like_dimension
+
+    assert not looks_like_dimension("ISO 4014 - M12x60")
+    assert not looks_like_dimension("DIN 912 - M8x20")
+    assert not looks_like_dimension("ISO 4032 - M12")
+    assert looks_like_dimension("ISO 4014 - M12x60  Ø78")
+    assert looks_like_dimension("40x60")
+    assert looks_like_dimension("Ø78")
+
+
 async def test_validator_catches_hidden_orphans(backend, tmp_path):
     await ensure_standard_linetypes(backend)
     await ensure_engineering_layers(backend)

tests/test_mech_draw.py

@@ -492,3 +492,65 @@ async def test_a_transaction_that_cannot_open_is_a_value_error(backend, monkeypa
     monkeypatch.setattr(backend, "transaction_begin", busy)
     with pytest.raises(ValueError, match="could not open a transaction"):
         await draw_from_spec(backend, {"parts": [{"part": PLATE, "at": [0.0, 0.0]}]})
+
+
+async def test_a_single_step_disk_dimensions_without_a_text_collision(backend):
+    """The largest diameter's text used to land on its own segment's chain text.
+
+    The front-view diameters start at the upper chord end, and ActiveX puts a
+    diameter's text beyond that FIRST point - above the view, clear of the
+    chain rows (measured on AutoCAD 2026). The headless engine put it beyond
+    the second point instead, so ⌀90 landed exactly on the "15" of a 15 mm
+    disk and ``dim_overlap`` fired on every single-step part the default
+    ``mech_part_draw`` produced headlessly. Found by the v5 ``mech_assembly``
+    benchmark and the live smoke together.
+    """
+    from engineering.critique import run_critique
+
+    disk = build_part(
+        {
+            "kind": "revolved",
+            "name": "FLANGE",
+            "material": "cast_iron",
+            "segments": [{"length": 15.0, "d_outer": 90.0, "d_inner": 40.0}],
+        }
+    )
+    await draw_part(backend, disk, at=(0.0, 0.0), views=("front",), dimension=True)
+    assert await run_critique(backend, ["dim_overlap"]) == []
+
+
+async def test_every_layer_exists_before_an_entity_is_put_on_it(backend, monkeypatch):
+    """ActiveX will not create a layer on assignment - the headless engine will.
+
+    Measured on AutoCAD 2026 by scripts/smoke_mech_com.py: ``entity.Layer =
+    "MECH"`` raised ``-0x7ffdfff7 ('Key not found')`` after every view of the
+    part had been drawn, so ``mech_part_draw`` could not finish on a live
+    seat. This backend is made as strict as ActiveX, and the whole part -
+    views, anchor, dimensions - has to draw on it.
+    """
+    creators = (
+        "entity_create_point",
+        "entity_create_line",
+        "entity_create_circle",
+        "entity_create_arc",
+        "entity_create_polyline",
+        "entity_create_text",
+        "entity_create_hatch",
+    )
+    for name in creators:
+        real = getattr(backend, name)
+
+        async def strict(*args, _real=real, **kwargs):
+            layer = kwargs.get("layer")
+            known = {row.name.lower() for row in await backend.layer_list()}
+            if layer and layer.lower() not in known:
+                raise RuntimeError(f"Key not found: layer {layer!r}")
+            return await _real(*args, **kwargs)
+
+        monkeypatch.setattr(backend, name, strict)
+
+    drawn = await draw_part(
+        backend, build_part(SHAFT), at=(0.0, 0.0), views=("front", "side"), dimension=True
+    )
+    assert drawn["part_id"]
+    assert ANCHOR_LAYER.lower() in {row.name.lower() for row in await backend.layer_list()}

tests/test_pid_graph.py

@@ -896,3 +896,14 @@ def test_com_block_reference_reports_mirrored_from_its_normal():
     assert _entity_info(blockref(Normal=(0.0, 0.0, -1.0))).properties["mirrored"] is True
     assert _entity_info(blockref(Normal=(0.0, 0.0, 1.0))).properties["mirrored"] is False
     assert _entity_info(blockref()).properties["mirrored"] is False
+
+
+def test_a_line_the_engine_could_not_read_is_not_an_edge():
+    """The live reader keeps a LINE whose point read AutoCAD refused, without
+    ``start``/``end`` (measured by scripts/smoke_mech_com.py on AutoCAD 2026);
+    the graph must leave it out instead of raising KeyError mid-critique."""
+    from backends.base import EntityInfo
+    from engineering.pid.graph import _vertices
+
+    blind = EntityInfo("1F", "LINE", "GEOMETRY", 256, "ByLayer", True, properties={})
+    assert _vertices(blind) == []

tests/test_sheet_bom.py

@@ -224,6 +224,30 @@ async def test_extract_reads_the_xdata_payload_and_never_touches_the_drawing(bac
     assert all(r["source"] == "xdata" for r in records)
 
 
+@pytest.mark.asyncio
+async def test_extract_reads_a_block_reference_under_the_live_type_name(backend, monkeypatch):
+    """The live engine calls an INSERT by its ObjectName, BLOCKREFERENCE.
+
+    Measured on AutoCAD 2026 by scripts/smoke_mech_com.py: a filter on
+    "INSERT" found nothing, and the parts list of a sheet holding an ISO 4014
+    bolt came back empty. The listing is made to answer the way ActiveX does.
+    """
+    handle = await _place_bolt(backend, "a", 0.0)
+    real = backend.entity_list
+
+    async def live_names(type_filter=None, **kwargs):
+        if type_filter and type_filter.upper() == "INSERT":
+            return []
+        wanted = (
+            "INSERT" if type_filter and type_filter.upper() == "BLOCKREFERENCE" else type_filter
+        )
+        return await real(type_filter=wanted, **kwargs)
+
+    monkeypatch.setattr(backend, "entity_list", live_names)
+    records = await extract_records(backend)
+    assert [r["handle"] for r in records] == [handle]
+
+
 @pytest.mark.asyncio
 async def test_extract_falls_back_to_the_block_attributes_when_there_is_no_payload(backend):
     await backend.block_define(

tests/test_token_suite.py

@@ -161,13 +161,25 @@ def test_the_cache_arithmetic_follows_the_published_multipliers():
 # -- lane 1: idle cost ------------------------------------------------------
 
 
-def test_the_idle_lane_covers_the_three_advertised_surfaces(report):
+def test_the_idle_lane_covers_the_four_advertised_surfaces(report):
+    """Default, lean, search - and TOOL_PACKS=core,mech, the mechanical client
+    that never opens a P&ID or a live seat: the default surface minus exactly
+    the pid and settings packs."""
     variants = {row["variant"]: row for row in report["idle"]}
-    assert set(variants) == {"default", "lean", "search"}
+    assert set(variants) == {"default", "lean", "packs_core_mech", "search"}
     assert variants["default"]["advertised_tools"] > 100
     assert variants["lean"]["advertised_tools"] == len(server.LEAN_TOOL_NAMES)
     assert variants["search"]["advertised_tools"] == 2
-    assert variants["default"]["chars"] > variants["lean"]["chars"] > variants["search"]["chars"]
+    dropped = len(server.PACK_TOOL_NAMES["pid"]) + len(server.PACK_TOOL_NAMES["settings"])
+    assert variants["packs_core_mech"]["advertised_tools"] == (
+        variants["default"]["advertised_tools"] - dropped
+    )
+    assert (
+        variants["default"]["chars"]
+        > variants["packs_core_mech"]["chars"]
+        > variants["lean"]["chars"]
+        > variants["search"]["chars"]
+    )
 
 
 def test_the_idle_lane_separates_the_wire_payload_from_what_the_model_pays(report):