bench: matrix v5 adds mech_assembly; the correctness suite grows with the mechanical and sheet checks

The evidence for tracks B and G, in the shape tracks A and E used.

- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
  addressable so an earlier report reproduces exactly rather than being only
  described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
  built from the part model and the sheet standard, through the code the
  tools call (apply_titleblock, draw_part/add_view/dimension_part,
  insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
  two numbers that cannot be negotiated: drawing_critique(focus=None) returns
  zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
  the v5 matrix). The adapter returns the focuses that fired, so a
  regression names itself, and a test drops one balloon to prove the gate
  fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
  that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
  20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
  through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
  410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
  40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
  k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
  quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
  (25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
  4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
  the drawing, a linked balloon, an xref attached / listed / detached, a
  31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
  every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
  the first call after Documents.Add is refused) and tracks its documents by
  the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
  code:
  * ezdxf dimension_diameter put the text beyond the SECOND chord point;
    ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
    from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
    front-view diameter stood on its own chain dimension. ezdxf now follows
    ActiveX.
  * draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
    missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
    every target layer before the first entity.
  * bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
    (ObjectName AcDbBlockReference) - the live parts list was empty.
  * COM _entity_info reported no dimension text_position: ModelSpace.Item
    narrows to IAcadEntity, which has no TextPosition (measured
    AttributeError); read through _com_unnarrow (TextPosition measured on
    AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
    diameter collided with its own bore.
  * pid graph raised KeyError 'start' mid-critique on a LINE whose point read
    AutoCAD refused while busy; such a line is left out.
  * the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
  re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
  the task count from the v5 matrix, the token table is remeasured with a
  TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
  collected-tests figure is retaken (3990).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-23 13:30:02 +03:00
co-authored by Claude Opus 5.5
parent 3b033033f9
commit 898e775f93
30 changed files with 1355 additions and 50 deletions
+13 -4
View File
@@ -92,10 +92,14 @@ Tools are organized into sections (counts are indicative — `system_about` is a
18. Styles (10 tools, pack `core`): `dimstyle_list/create/modify/set_current` (authored ISO-25 / ANSI presets from `engineering/standards/dimstyles.py`; `dimstyle_modify` accepts only `DIM_VARIABLE_WHITELIST` names (33) and refuses out-of-range values by name before writing; `changed` lists only variables that moved, and `rerender_required: true` says a headless DIMENSION keeps its rendered block until redrawn), `textstyle_list/create/set_current` (`font_resolved: false` is reported, never refused — DXF stores only the name), `mleaderstyle_list/create` (both engines: ezdxf on `doc.mleader_styles`, live through the `ACAD_MLEADERSTYLE` dictionary — ActiveX has no MLeaderStyle *collection*, which is not an engine boundary, so no `mleaderstyle` capability key exists), and `drawing_apply_standard("iso"|"ansi")` — dimstyle + text style + units + the `mech` layer set in one call, each item reporting `created` or already present.
19. Page setup & templates (6 tools, pack `core`): `page_setup_list/apply` (ISO 216 / ANSI Y14.1 sizes from `PAPER_SIZES`; an unknown ctb is written and reported `plot_style_known: false`), `plot_style_list` (catalog + the files actually installed on a live seat), `batch_plot` (every paper layout through `drawing_export_pdf(layout=)`; `mediabox_mm` is **read back from the PDF's own `/MediaBox`**, so the sheet size is verified, not assumed), `drawing_template_list` (the five bundled templates: `iso_a3_mech`, `iso_a1_arch`, `iso_a3_pid`, `ansi_b_mech`, `ansi_d_arch` — DXF headless, `.dwt` twin on COM, `scripts/build_templates.py --check` proves the DXFs reproducible), `drawing_template_save` (a `.dwt` request headlessly is refused with capability `dwt_write` naming the DXF route). `drawing_new(template=)` resolves a catalog name to the bundled file and reports `template: {name, source}`.
20. Environment (22 tools, pack `settings`): documents `document_list/activate/close` (both engines — the headless backend keeps a document registry; `document_close` refuses to discard unsaved work unless `discard=True`), layer states `layer_state_save/restore/list/delete` (portable: JSON chunks in an `ACADMCP_LAYERSTATES` XRECORD that travels with the file and **does not appear in AutoCAD's Layer States Manager** — `system_capabilities.layer_states` says so), named views `view_named_save/restore/list` (headless restore writes `$VIEWCTR/$VIEWSIZE` and reports `applied: "header_only"`), UCS `ucs_list/set/restore` (non-orthogonal axes refused with the measured angle; **tool coordinates stay WCS** — a UCS is stored for the operator, no tool interprets input in it), `system_variable_describe` (the 90-entry `SYSVAR_CATALOG`; an unknown name answers `known: false` plus nearest names, never an error; `system_set_variable` now refuses an out-of-range value for a catalogued name before writing), `drawing_properties_get/set` (custom properties on both engines through `$CUSTOMPROPERTYTAG`; the five summary fields are COM-only, capability `dwgprops`), and the live-only group `system_launch` (`live_application`), `system_preferences_get/set` (`preferences`; a whitelist of `Preferences.*` keys with ranges, read-only keys refused by name, writes report `old`/`new`), `user_pick_point/select`, `system_prompt_message` (`interactive_prompt`; a cancelled pick returns `{cancelled: true}`, a `COM_CALL_TIMEOUT` overrun reports `timed_out`). The COM paths were executed live once by `scripts/smoke_settings_com.py` (AutoCAD 2026), which also built the `.dwt` twins.
21. Mechanical parts (6 tools, pack `mech`): `mech_part_draw`, `mech_view_add`, `mech_dimension_part`, `mech_hole_pattern`, `mech_part_from_spec`, `mech_part_inspect`. One part model — a profile (`RevolvedPart` segments or a `PrismaticPart` outline) plus typed features — and one view engine that turns it into front / side / top / section / detail views with hidden lines, cut faces and ISO 129 dimensions. The model is written onto the drawing as `ACADMCP_MECH` XDATA, so `mech_view_add` can add a view months later without the caller re-describing the part. A feature that cannot be projected into a view is listed in `omitted: [{feature, reason}]` — never dropped.
22. Standard parts (3 tools, pack `mech`): `std_part_list`, `std_part_insert`, `std_feature_draw`. ISO 4014 / 4017 / 4032 / 7089 / 4762 fasteners, ISO 15 bearings with ISO 8826-1 representation, DIN 471/472 retaining-ring grooves, DIN 509 undercuts, DIN 332 centre holes and ISO 3601-2 O-ring grooves — authored tables, inserted as real blocks with attributes and `ACADMCP_MECH` XDATA, which is what makes the parts list a *read* of the drawing. A size outside a table's transcribed coverage is refused by name, never interpolated.
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 55 curated drafting tools (52 with `TOOL_PACKS=core`; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 60 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*` and the three `mech_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools; SECTION 18/19 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
@@ -107,7 +111,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
**Benchmark matrix:** `benchmarks/tasks_v4.py` is the current set (17 tasks; `--matrix v3` / `--matrix v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks or the v4 `pid_roundtrip` / `page_setup_truth` — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (32 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
**Benchmark matrix:** `benchmarks/tasks_v5.py` is the current set (18 tasks; `--matrix v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth` or the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (39 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
### Adding a New Tool
@@ -134,10 +138,15 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
- **`TOOL_PACKS=all|core,pid,settings`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **`TOOL_PACKS=all|core,pid,settings,mech`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
- **The part model, not primitives**: a machine part is `mech_part_draw` with a segment list or an outline plus typed features — never hand-drawn lines. The same rule as gears and keyways, for the same reason.
- **Standards tables are transcribed, never derived**: every row under `engineering/mech/standards/` carries a `SOURCE` naming standard, edition and table, and a size outside the coverage is refused by name. `engineering/mech/standards/materials.py` states which of its rows are ISO 128-50, which are ASME Y14.2M via `acadiso.pat`, and which (concrete, wood) have no mechanical standard at all.
- **Six `mech_*` critique focuses** (`mech_missing_centreline`, `mech_unhatched_section`, `mech_view_misaligned`, `mech_duplicate_dimension`, `mech_thread_unrepresented`, `mech_bom_balloon_mismatch`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_MECH` payload. `mech_view_misaligned` checks the projection axis and not the quadrant, and `mech_duplicate_dimension` catches two dimensions in the same place rather than a semantic restatement — both narrowings are in the module docstring, because the payload carries no projection angle and no engine exposes what a dimension measured.
- **Live ActiveX will not create a layer on assignment** (`entity.Layer = "X"` on a missing layer raises `Key not found`; ezdxf creates it silently). Every mechanical/annotation draw path ensures its target layers before the first entity — a new path must do the same, or it passes every headless test and dies live.
- **A diameter's text stands beyond the FIRST chord point** on both engines (the ActiveX `AddDimDiametric` rule, measured); `dimension_diameter(x1, y1, x2, y2)` annotates on the `(x1, y1)` side.
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
### Premium Drawing Rules