Umut CANandClaude Opus 5.5
898e775f93
The evidence for tracks B and G, in the shape tracks A and E used.
- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
addressable so an earlier report reproduces exactly rather than being only
described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
built from the part model and the sheet standard, through the code the
tools call (apply_titleblock, draw_part/add_view/dimension_part,
insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
two numbers that cannot be negotiated: drawing_critique(focus=None) returns
zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
the v5 matrix). The adapter returns the focuses that fired, so a
regression names itself, and a test drops one balloon to prove the gate
fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
(25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
the drawing, a linked balloon, an xref attached / listed / detached, a
31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
the first call after Documents.Add is refused) and tracks its documents by
the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
code:
* ezdxf dimension_diameter put the text beyond the SECOND chord point;
ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
front-view diameter stood on its own chain dimension. ezdxf now follows
ActiveX.
* draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
every target layer before the first entity.
* bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
(ObjectName AcDbBlockReference) - the live parts list was empty.
* COM _entity_info reported no dimension text_position: ModelSpace.Item
narrows to IAcadEntity, which has no TextPosition (measured
AttributeError); read through _com_unnarrow (TextPosition measured on
AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
diameter collided with its own bore.
* pid graph raised KeyError 'start' mid-critique on a LINE whose point read
AutoCAD refused while busy; such a line is left out.
* the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
the task count from the v5 matrix, the token table is remeasured with a
TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
collected-tests figure is retaken (3990).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
113 lines
4.1 KiB
Python
113 lines
4.1 KiB
Python
# tests/test_benchmark_v5.py
|
|
"""The v5 task matrix: v4 plus what tracks B and G added.
|
|
|
|
``mech_assembly`` is roadmap criterion 3 — a flange-coupling sheet built from
|
|
the tools, gated on two numbers that cannot be negotiated:
|
|
``drawing_critique(focus=None)`` returns zero issues and ``drawing_finalize``
|
|
scores at least 90. The published competitor reports were never asked it and
|
|
the chart shows them ``not_run``, the rule v3 set for its five.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
from benchmarks.tasks_v4 import TASKS_V4
|
|
from benchmarks.tasks_v5 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V5, TASKS_V5, task_by_id
|
|
|
|
|
|
def test_v5_extends_v4_without_touching_it():
|
|
assert TASKS_V5[: len(TASKS_V4)] == TASKS_V4
|
|
assert [t.task_id for t in NEW_TASKS_V5] == ["mech_assembly"]
|
|
assert len(TASKS_V5) == 18
|
|
assert MATRICES["v4"] == TASKS_V4 and MATRICES["v5"] == TASKS_V5
|
|
assert DEFAULT_MATRIX == "v5"
|
|
assert task_by_id("mech_assembly").category == "mech"
|
|
|
|
|
|
def test_the_adapter_implements_every_v5_task():
|
|
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
|
|
|
for task in TASKS_V5:
|
|
assert hasattr(AutoCADMCPProAdapter, f"_task_{task.task_id}"), task.task_id
|
|
|
|
|
|
def test_the_suite_carries_the_mechanical_and_sheet_checks():
|
|
"""Six from spec 14, plus mech_thread_unrepresented_is_caught: a quality
|
|
gate that never fires is not a gate, the way dim_overlap_critique_fires
|
|
already guards the dimension focus."""
|
|
from benchmarks.correctness_suite import CHECKS
|
|
|
|
for name in (
|
|
"mech_part_roundtrip",
|
|
"mech_section_hatch_area",
|
|
"mech_iso286_on_dimension",
|
|
"mech_thread_unrepresented_is_caught",
|
|
"std_part_iso4014_m12",
|
|
):
|
|
assert CHECKS[name][1] == "Mechanical"
|
|
assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet"
|
|
assert CHECKS["bom_balloon_link"][1] == "Sheet"
|
|
assert len(CHECKS) == 39
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_mech_assembly_passes_headlessly(tmp_path):
|
|
"""The benchmark task itself, run once here so a regression in the part
|
|
drawer, the sheet or the critique fails the suite and not only the matrix."""
|
|
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
|
|
|
adapter = AutoCADMCPProAdapter(backend="ezdxf")
|
|
await adapter.setup(tmp_path)
|
|
try:
|
|
await adapter._reset()
|
|
passed, metrics, _artifacts = await adapter._task_mech_assembly()
|
|
finally:
|
|
await adapter.cleanup()
|
|
assert passed, metrics
|
|
assert metrics["critique_issues"] == 0, metrics["critique_focuses"]
|
|
assert metrics["score"] >= 90.0
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_mech_assembly_fails_loudly_when_the_critique_finds_anything(tmp_path, monkeypatch):
|
|
"""The gate has to be able to fail. One parts-list row left without its
|
|
balloon is enough: ISO 7573 gives every row an item reference, and the
|
|
critique names the focus that fired rather than only shading the score."""
|
|
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
|
from engineering.sheet import bom
|
|
|
|
real = bom.add_balloon
|
|
|
|
async def only_the_first(backend, *, item, **kw):
|
|
if int(item) == 1:
|
|
return await real(backend, item=item, **kw)
|
|
return {"ok": True, "skipped": True}
|
|
|
|
monkeypatch.setattr(bom, "add_balloon", only_the_first)
|
|
adapter = AutoCADMCPProAdapter(backend="ezdxf")
|
|
await adapter.setup(tmp_path)
|
|
try:
|
|
await adapter._reset()
|
|
passed, metrics, _artifacts = await adapter._task_mech_assembly()
|
|
finally:
|
|
await adapter.cleanup()
|
|
assert passed is False
|
|
assert "mech_bom_balloon_mismatch" in metrics["critique_focuses"]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_section_hatch_area_check_measures_the_drawing():
|
|
"""1200 mm^2 is computed by hand from the model, and read back through
|
|
analysis_measure_entity — a number the drawing code cannot fake."""
|
|
from benchmarks import correctness_suite
|
|
|
|
assert await correctness_suite.mech_section_hatch_area() is True
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_the_iso5457_frame_check_passes():
|
|
from benchmarks import correctness_suite
|
|
|
|
assert await correctness_suite.sheet_frame_iso5457_a3() is True
|