anatoly/autocad-mcp-pro
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

bench: matrix v5 adds mech_assembly; the correctness suite grows with the mechanical and sheet checks

Browse Source 
The evidence for tracks B and G, in the shape tracks A and E used.

- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
  addressable so an earlier report reproduces exactly rather than being only
  described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
  built from the part model and the sheet standard, through the code the
  tools call (apply_titleblock, draw_part/add_view/dimension_part,
  insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
  two numbers that cannot be negotiated: drawing_critique(focus=None) returns
  zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
  the v5 matrix). The adapter returns the focuses that fired, so a
  regression names itself, and a test drops one balloon to prove the gate
  fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
  that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
  20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
  through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
  410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
  40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
  k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
  quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
  (25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
  4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
  the drawing, a linked balloon, an xref attached / listed / detached, a
  31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
  every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
  the first call after Documents.Add is refused) and tracks its documents by
  the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
  code:
  * ezdxf dimension_diameter put the text beyond the SECOND chord point;
    ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
    from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
    front-view diameter stood on its own chain dimension. ezdxf now follows
    ActiveX.
  * draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
    missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
    every target layer before the first entity.
  * bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
    (ObjectName AcDbBlockReference) - the live parts list was empty.
  * COM _entity_info reported no dimension text_position: ModelSpace.Item
    narrows to IAcadEntity, which has no TextPosition (measured
    AttributeError); read through _com_unnarrow (TextPosition measured on
    AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
    diameter collided with its own bore.
  * pid graph raised KeyError 'start' mid-critique on a LINE whose point read
    AutoCAD refused while busy; such a line is left out.
  * the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
  re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
  the task count from the v5 matrix, the token table is remeasured with a
  TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
  collected-tests figure is retaken (3990).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-23 13:30:02 +03:00
co-authored by Claude Opus 5.5
parent 3b033033f9
commit 898e775f93
30 changed files with 1355 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/adapters/autocad_mcp_pro.py
+170 -2
View File
@@ -1,4 +1,4 @@
"""Reference adapter for this repository, covering the v4 task matrix.
"""Reference adapter for this repository, covering the v5 task matrix.
Every task verifies against a value worked out independently of the code under
test — a closed-form area, a count of entities placed on purpose, a token
@@ -12,7 +12,7 @@ import math
from pathlib import Path
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
from benchmarks.tasks_v4 import TaskSpec
from benchmarks.tasks_v5 import TaskSpec
from engineering.delivery import deliver_drawing
from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
from engineering.refiner import refine_drawing
@@ -514,6 +514,174 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
[str(row_before["path"]), str(row["path"])],
)
async def _task_mech_assembly(self):
"""A flange-coupling sheet built from the part model and the sheet standard.
Roadmap criterion 3. Two parts (a hub with a bore and a flange), a full
section through the hub, four standard fasteners with their parts-list
rows and balloons, an ISO 5457 A3 frame and an ISO 7200 title block.
Every step goes through the code the tools call: the frame and title
block through ``apply_titleblock``, the parts list is *read* off the
drawing by ``extract_records`` and drawn by ``draw_bom_table``, and each
balloon is linked to its row's inserts by ``add_balloon``.
The gate is two numbers, and both are asserted rather than reported:
``run_critique(focus=None)`` must return zero issues, and the score the
finalize gate computes - ``combine(validator.summary, critique)`` over
the saved sheet - must be at least 90. A regression in either fails the
task; the focuses that fired come back in the metrics so the failure
names itself.
No step here repairs what the critique would flag: the view engine
draws the ISO 128-23 centre lines of every circle it emits (the end
view's bore and bolt holes included), so the sheet is clean because
the tools drew it clean, not because the benchmark tidied up after them.
"""
from engineering import DrawingValidator
from engineering.critique import run_critique
from engineering.layers import apply_layer_set
from engineering.mech.draw import add_view, dimension_part, draw_part
from engineering.mech.part import build_part
from engineering.mech.stdparts import insert_std_part
from engineering.scoring import combine
from engineering.sheet.bom import (
add_balloon,
draw_bom_table,
extract_records,
rows_from_records,
)
from engineering.sheet.titleblock import (
TB_HEIGHT,
TitleBlockMetadata,
apply_titleblock,
titleblock_origin,
)
await apply_layer_set(self.backend, "mech")
await apply_titleblock(
self.backend,
size="A3",
metadata=TitleBlockMetadata(
title="FLANGE COUPLING",
drawing_no="MA-001",
drawn_by="autocad-mcp-pro",
scale="1:1",
material="S235JR",
),
projection="first",
frame=True,
zones=True,
marks=True,
)
hub = build_part(
{
"kind": "revolved",
"name": "HUB",
"material": "steel",
"segments": [
{"length": 40.0, "d_outer": 40.0, "d_inner": 20.0},
{"length": 15.0, "d_outer": 90.0, "d_inner": 20.0},
],
}
)
flange = build_part(
{
"kind": "revolved",
"name": "FLANGE",
"material": "cast_iron",
"segments": [{"length": 15.0, "d_outer": 90.0, "d_inner": 40.0}],
# Four M12 clearance holes (ISO 273 medium, 13.5) on a 65 PCD.
"features": [
{
"kind": "hole_circle",
"id": "bolts",
"x": 7.5,
"pcd": 65.0,
"count": 4,
"diameter": 13.5,
"start_angle": 45.0,
}
],
}
)
# A3 landscape: the frame runs (20, 10)-(410, 287) and the title block
# fills its lower-right corner, 180 x 60. First angle throughout: the
# hub's front view sits at x = 125 so its section lands left of it
# inside the frame; the flange's front view stands at x = 345 so its
# end view (the bolt circle) lands between the two parts; the
# fasteners take the free band below, left of the title block.
drawn_hub = await draw_part(
self.backend, hub, at=(125.0, 160.0), views=("front",), dimension=False
)
# A full longitudinal section along the hub's axis (part-local frame).
await add_view(
self.backend,
drawn_hub["part_id"],
"section",
plane={"p1": [0.0, 0.0], "p2": [55.0, 0.0], "label": "A"},
style="full",
)
await dimension_part(self.backend, drawn_hub["part_id"], style="chain")
await draw_part(
self.backend, flange, at=(345.0, 160.0), views=("front", "side"), dimension=True
)
# Two bolt-and-nut pairs: each nut sits on the thread of its bolt.
stations = (60.0, 110.0, 160.0, 210.0)
fasteners = []
for index, x in enumerate(stations):
designation = "ISO 4014 - M12x60" if index % 2 == 0 else "ISO 4032 - M12"
fasteners.append(
await insert_std_part(self.backend, designation, at=(x, 90.0), view="side")
)
# The parts list is a read of the drawing, not a list typed beside it.
records = await extract_records(self.backend)
rows = rows_from_records(records)
tb_x, tb_y = titleblock_origin("A3")
await draw_bom_table(self.backend, rows, at=(tb_x, tb_y + TB_HEIGHT))
station_of = {
str(result.get("handle")): x for result, x in zip(fasteners, stations, strict=True)
}
for row in rows:
targets = [str(handle) for handle in row.get("handles") or ()]
x = station_of.get(targets[0], stations[0]) if targets else stations[0]
await add_balloon(
self.backend,
item=int(row["item"]),
at=(x, 135.0),
leader_to=(x, 95.0),
targets=targets,
)
# drawing_finalize saves before it validates; so does the benchmark,
# or the validator's not_saved/no_path findings would grade the
# harness instead of the sheet.
sheet = self.artifact_dir / "mech_assembly.dxf"
await self.backend.drawing_save_as(str(sheet))
issues = await run_critique(self.backend, None)
validation = await DrawingValidator().run(self.backend)
summary = {"error": 0, "warning": 0, "info": 0}
for issue in issues:
summary[issue.severity] = summary.get(issue.severity, 0) + 1
score = combine(validation.summary, summary)
passed = len(issues) == 0 and float(score["score"]) >= 90.0
return (
passed,
{
"critique_issues": len(issues),
"critique_focuses": sorted({issue.focus for issue in issues}),
"score": float(score["score"]),
"grade": score["grade"],
"invalidity_ratio": score["invalidity_ratio"],
"parts_list_rows": len(rows),
"fasteners": len(fasteners),
"validator_findings": sorted(f.code for f in validation.findings),
},
[str(sheet)],
)
async def _task_auditable_delivery(self):
await ensure_engineering_layers(self.backend)
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")