bench: matrix v5 adds mech_assembly; the correctness suite grows with the mechanical and sheet checks
The evidence for tracks B and G, in the shape tracks A and E used.
- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
addressable so an earlier report reproduces exactly rather than being only
described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
built from the part model and the sheet standard, through the code the
tools call (apply_titleblock, draw_part/add_view/dimension_part,
insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
two numbers that cannot be negotiated: drawing_critique(focus=None) returns
zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
the v5 matrix). The adapter returns the focuses that fired, so a
regression names itself, and a test drops one balloon to prove the gate
fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
(25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
the drawing, a linked balloon, an xref attached / listed / detached, a
31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
the first call after Documents.Add is refused) and tracks its documents by
the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
code:
* ezdxf dimension_diameter put the text beyond the SECOND chord point;
ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
front-view diameter stood on its own chain dimension. ezdxf now follows
ActiveX.
* draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
every target layer before the first entity.
* bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
(ObjectName AcDbBlockReference) - the live parts list was empty.
* COM _entity_info reported no dimension text_position: ModelSpace.Item
narrows to IAcadEntity, which has no TextPosition (measured
AttributeError); read through _com_unnarrow (TextPosition measured on
AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
diameter collided with its own bore.
* pid graph raised KeyError 'start' mid-critique on a LINE whose point read
AutoCAD refused while busy; such a line is left out.
* the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
the task count from the v5 matrix, the token table is remeasured with a
TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
collected-tests figure is retaken (3990).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
3b033033f9
commit
898e775f93
+104
@@ -21,6 +21,14 @@ views and UCS, and the live-only group (launch, preferences, document
|
||||
properties, operator prompts) refused honestly headlessly. Spec:
|
||||
`docs/superpowers/specs/2026-09-16-v1.6-settings-design.md`.
|
||||
|
||||
Tracks B and G of 1.6: the complete mechanical 2D part drawer and the sheet
|
||||
standard. 25 tools in four new sections — mechanical parts (21), standard
|
||||
parts (22), mechanical annotation (23) and sheet and delivery (24) — one part
|
||||
model and one view engine instead of a zoo of per-part generators, a new
|
||||
`mech` pack, six `mech_*` critique focuses and four new contract members
|
||||
(`xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg`). Spec:
|
||||
`docs/superpowers/specs/2026-09-22-v1.6-mech-design.md`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Styles — 10 tools, pack `core`.** `dimstyle_list/create/modify/set_current`,
|
||||
@@ -74,6 +82,28 @@ properties, operator prompts) refused honestly headlessly. Spec:
|
||||
`v1.5.1`: **26 / 32 → 32 / 32, six `miss → pass`, zero regressed**.
|
||||
`scripts/smoke_settings_com.py` runs the track once against the live seat
|
||||
(record below).
|
||||
- **Benchmark evidence for tracks B and G.** `benchmarks/tasks_v5.py` (18
|
||||
tasks, the default matrix; v4 / v3 / v2 stay addressable) adds
|
||||
`mech_assembly`: a flange-coupling A3 sheet built through the code the tools
|
||||
call — frame and title block, a hub with a full section and chain
|
||||
dimensions, a flange with its bolt-circle end view, ISO 4014 bolts and ISO
|
||||
4032 nuts, the parts list read off the drawing and linked balloons — gated on
|
||||
`drawing_critique(focus=None)` returning zero issues and a finalize score of
|
||||
at least 90 (it scores 100); a test drops one balloon and watches the gate
|
||||
fail by name. The correctness suite grows to 39 with `mech_part_roundtrip`,
|
||||
`mech_section_hatch_area` (1200 mm² read back out of the drawing),
|
||||
`mech_iso286_on_dimension`, `mech_thread_unrepresented_is_caught`,
|
||||
`std_part_iso4014_m12`, `sheet_frame_iso5457_a3` and `bom_balloon_link`.
|
||||
A/B against `v1.5.1`: **26 / 39 → 39 / 39, thirteen `miss → pass`, zero
|
||||
regressed**. The token suite measures a `TOOL_PACKS=core,mech` row: 193
|
||||
tools / 61,673 tokens against 224 / 72,163 by default.
|
||||
- **Live COM smoke.** `scripts/smoke_mech_com.py` executed once on AutoCAD
|
||||
2026 (`25.1s (LMS Tech)`): the A3 frame and ISO 7200 title block, a two-step
|
||||
shaft with a full section and 12 chain dimensions, an ISO 4014 M12x60 block
|
||||
carrying 1 ACADMCP_MECH chunk, an ISO 7573 parts list read off the drawing
|
||||
with its linked balloon, an xref attached and detached, and a real
|
||||
31687-byte `.dwg` written by ActiveX `SaveAs` — `drawing_critique(focus=None)`
|
||||
returned 0 issues (record below).
|
||||
|
||||
- **P&ID — 9 tools, pack `pid`.**
|
||||
- `pid_symbol_list` / `pid_symbol_insert`: a 43-symbol ISO 10628-2 /
|
||||
@@ -153,6 +183,33 @@ properties, operator prompts) refused honestly headlessly. Spec:
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Found by the tracks B + G evidence** (each with a test that fails on the
|
||||
old code):
|
||||
- `dimension_diameter` put its text beyond the *second* chord point
|
||||
headlessly and beyond the *first* on AutoCAD (`AddDimDiametric`'s
|
||||
ChordPoint, measured: a ⌀50 from (x, 170) to (x, 220) annotated at
|
||||
(x, 160)). The two engines annotated opposite sides of the same part, and
|
||||
headlessly every front-view diameter of `mech_part_draw` stood on its own
|
||||
chain dimension — `dim_overlap` fired on any single-step disk. The
|
||||
headless engine now follows ActiveX.
|
||||
- On a live seat `mech_part_draw` died after drawing every view: its anchor
|
||||
POINT goes on `MECH`, and ActiveX refuses `entity.Layer` on a missing
|
||||
layer (`Key not found`) where ezdxf creates it. Every mechanical draw
|
||||
path now ensures its target layers before the first entity.
|
||||
- `bom_extract` returned nothing live: ActiveX names a block reference
|
||||
`BLOCKREFERENCE`, and the parts list filtered on `INSERT` only. Both
|
||||
names are read.
|
||||
- The live `_entity_info` reported no `text_position` for a dimension (the
|
||||
entity arrives narrowed to `IAcadEntity`, which has no `TextPosition`),
|
||||
so `dim_overlap` compared bounding-box centres — a diameter and its bore
|
||||
read "0.00 mm apart". It is read through the concrete class now.
|
||||
- `pid_graph` raised `KeyError: 'start'` inside `drawing_critique(focus=None)`
|
||||
on a LINE whose point read AutoCAD had refused while busy; such a line is
|
||||
left out instead.
|
||||
- The validator's `fake_dimension_text` read a standard-part designation
|
||||
(`ISO 4014 - M12x60`) as a hand-typed dimension, so every parts list
|
||||
drawn as text cost its sheet a warning. The designation is exempt; a
|
||||
measurement beside it still flags.
|
||||
- **Track A hardening** (`tests/test_track_a_hardening.py`; the remaining
|
||||
review findings are filed with a disposition each in
|
||||
`docs/analysis/track-a-review-backlog.md`):
|
||||
@@ -716,6 +773,53 @@ The pick path was executed live earlier in the track (Task 23's measurement:
|
||||
`GetPoint` answers in WCS while `GetEntity` answers in the current UCS). Each
|
||||
plot also dropped AutoCAD's `plot.log` into the repo root; it is ignored now.
|
||||
|
||||
### Live COM smoke — mechanical parts and the sheet (tracks B + G)
|
||||
|
||||
`scripts/smoke_mech_com.py`, AutoCAD 2026 (`25.1s (LMS Tech)`), Windows 11,
|
||||
2026-09-23, in new documents only (an xref source saved and closed first, then
|
||||
the sheet) — `apply_layer_set("mech")` → `apply_titleblock` (A3, zones, marks)
|
||||
→ `draw_part` → `add_view(section)` → `dimension_part` → `insert_std_part` →
|
||||
`weld_symbol_prims` → `extract_records` → `draw_bom_table` → `add_balloon` →
|
||||
`xref_attach` / `xref_manage("list")` / `xref_manage("detach")` →
|
||||
`run_critique(focus=None)` → `drawing_export_dwg("R2018")`, all through the COM
|
||||
engine, exit 0, every scratch document closed afterwards:
|
||||
|
||||
```json
|
||||
{
|
||||
"autocad": "25.1s (LMS Tech)",
|
||||
"part_id": "103",
|
||||
"views": 2,
|
||||
"section_omitted": [],
|
||||
"dimensions": 12,
|
||||
"bolt_handle": "2F7",
|
||||
"xdata_chunks": 1,
|
||||
"parts_list_rows": 1,
|
||||
"parts_list_representation": "native",
|
||||
"balloon": "31A",
|
||||
"xref_attached": "XREF_SOURCE",
|
||||
"xref_listed": 1,
|
||||
"xref_detached": true,
|
||||
"xrefs_after_detach": 0,
|
||||
"dwg": {
|
||||
"path": "%TEMP%\\mech_smoke_*\\mech_smoke.dwg",
|
||||
"exists": true,
|
||||
"bytes": 31687,
|
||||
"magic": "AC1032"
|
||||
},
|
||||
"critique_issues": 0,
|
||||
"critique": []
|
||||
}
|
||||
```
|
||||
|
||||
The first runs found the four live defects listed under **Fixed** (the `MECH`
|
||||
layer, `BLOCKREFERENCE`, the dimension `text_position`, the refused LINE read)
|
||||
and one harness fault: the first call after `Documents.Add` can be refused with
|
||||
`RPC_E_CALL_REJECTED` while the new document initialises, and a refused
|
||||
cleanup then left the scratch document open. The smoke retries refused calls
|
||||
for up to 90 s (a refused call was not executed) and tracks its documents by
|
||||
the document list rather than by an assumed name, because `SaveAs` renames the
|
||||
document it writes.
|
||||
|
||||
## [1.5.1] — 2026-08-06
|
||||
|
||||
A patch for three defects found *after* 1.5.0 went to PyPI, two of them by the
|
||||
|
||||
@@ -92,10 +92,14 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
||||
18. Styles (10 tools, pack `core`): `dimstyle_list/create/modify/set_current` (authored ISO-25 / ANSI presets from `engineering/standards/dimstyles.py`; `dimstyle_modify` accepts only `DIM_VARIABLE_WHITELIST` names (33) and refuses out-of-range values by name before writing; `changed` lists only variables that moved, and `rerender_required: true` says a headless DIMENSION keeps its rendered block until redrawn), `textstyle_list/create/set_current` (`font_resolved: false` is reported, never refused — DXF stores only the name), `mleaderstyle_list/create` (both engines: ezdxf on `doc.mleader_styles`, live through the `ACAD_MLEADERSTYLE` dictionary — ActiveX has no MLeaderStyle *collection*, which is not an engine boundary, so no `mleaderstyle` capability key exists), and `drawing_apply_standard("iso"|"ansi")` — dimstyle + text style + units + the `mech` layer set in one call, each item reporting `created` or already present.
|
||||
19. Page setup & templates (6 tools, pack `core`): `page_setup_list/apply` (ISO 216 / ANSI Y14.1 sizes from `PAPER_SIZES`; an unknown ctb is written and reported `plot_style_known: false`), `plot_style_list` (catalog + the files actually installed on a live seat), `batch_plot` (every paper layout through `drawing_export_pdf(layout=)`; `mediabox_mm` is **read back from the PDF's own `/MediaBox`**, so the sheet size is verified, not assumed), `drawing_template_list` (the five bundled templates: `iso_a3_mech`, `iso_a1_arch`, `iso_a3_pid`, `ansi_b_mech`, `ansi_d_arch` — DXF headless, `.dwt` twin on COM, `scripts/build_templates.py --check` proves the DXFs reproducible), `drawing_template_save` (a `.dwt` request headlessly is refused with capability `dwt_write` naming the DXF route). `drawing_new(template=)` resolves a catalog name to the bundled file and reports `template: {name, source}`.
|
||||
20. Environment (22 tools, pack `settings`): documents `document_list/activate/close` (both engines — the headless backend keeps a document registry; `document_close` refuses to discard unsaved work unless `discard=True`), layer states `layer_state_save/restore/list/delete` (portable: JSON chunks in an `ACADMCP_LAYERSTATES` XRECORD that travels with the file and **does not appear in AutoCAD's Layer States Manager** — `system_capabilities.layer_states` says so), named views `view_named_save/restore/list` (headless restore writes `$VIEWCTR/$VIEWSIZE` and reports `applied: "header_only"`), UCS `ucs_list/set/restore` (non-orthogonal axes refused with the measured angle; **tool coordinates stay WCS** — a UCS is stored for the operator, no tool interprets input in it), `system_variable_describe` (the 90-entry `SYSVAR_CATALOG`; an unknown name answers `known: false` plus nearest names, never an error; `system_set_variable` now refuses an out-of-range value for a catalogued name before writing), `drawing_properties_get/set` (custom properties on both engines through `$CUSTOMPROPERTYTAG`; the five summary fields are COM-only, capability `dwgprops`), and the live-only group `system_launch` (`live_application`), `system_preferences_get/set` (`preferences`; a whitelist of `Preferences.*` keys with ranges, read-only keys refused by name, writes report `old`/`new`), `user_pick_point/select`, `system_prompt_message` (`interactive_prompt`; a cancelled pick returns `{cancelled: true}`, a `COM_CALL_TIMEOUT` overrun reports `timed_out`). The COM paths were executed live once by `scripts/smoke_settings_com.py` (AutoCAD 2026), which also built the `.dwt` twins.
|
||||
21. Mechanical parts (6 tools, pack `mech`): `mech_part_draw`, `mech_view_add`, `mech_dimension_part`, `mech_hole_pattern`, `mech_part_from_spec`, `mech_part_inspect`. One part model — a profile (`RevolvedPart` segments or a `PrismaticPart` outline) plus typed features — and one view engine that turns it into front / side / top / section / detail views with hidden lines, cut faces and ISO 129 dimensions. The model is written onto the drawing as `ACADMCP_MECH` XDATA, so `mech_view_add` can add a view months later without the caller re-describing the part. A feature that cannot be projected into a view is listed in `omitted: [{feature, reason}]` — never dropped.
|
||||
22. Standard parts (3 tools, pack `mech`): `std_part_list`, `std_part_insert`, `std_feature_draw`. ISO 4014 / 4017 / 4032 / 7089 / 4762 fasteners, ISO 15 bearings with ISO 8826-1 representation, DIN 471/472 retaining-ring grooves, DIN 509 undercuts, DIN 332 centre holes and ISO 3601-2 O-ring grooves — authored tables, inserted as real blocks with attributes and `ACADMCP_MECH` XDATA, which is what makes the parts list a *read* of the drawing. A size outside a table's transcribed coverage is refused by name, never interpolated.
|
||||
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
|
||||
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
|
||||
|
||||
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 55 curated drafting tools (52 with `TOOL_PACKS=core`; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
|
||||
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 60 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*` and the three `mech_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
|
||||
|
||||
**Tool packs:** `TOOL_PACKS=all|core,pid,settings` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools; SECTION 18/19 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
|
||||
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
|
||||
|
||||
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
|
||||
|
||||
@@ -107,7 +111,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
||||
|
||||
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
|
||||
|
||||
**Benchmark matrix:** `benchmarks/tasks_v4.py` is the current set (17 tasks; `--matrix v3` / `--matrix v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks or the v4 `pid_roundtrip` / `page_setup_truth` — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (32 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
|
||||
**Benchmark matrix:** `benchmarks/tasks_v5.py` is the current set (18 tasks; `--matrix v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth` or the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (39 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
|
||||
|
||||
### Adding a New Tool
|
||||
|
||||
@@ -134,10 +138,15 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
||||
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
|
||||
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
|
||||
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
|
||||
- **`TOOL_PACKS=all|core,pid,settings`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
|
||||
- **`TOOL_PACKS=all|core,pid,settings,mech`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
|
||||
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
|
||||
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
|
||||
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
|
||||
- **The part model, not primitives**: a machine part is `mech_part_draw` with a segment list or an outline plus typed features — never hand-drawn lines. The same rule as gears and keyways, for the same reason.
|
||||
- **Standards tables are transcribed, never derived**: every row under `engineering/mech/standards/` carries a `SOURCE` naming standard, edition and table, and a size outside the coverage is refused by name. `engineering/mech/standards/materials.py` states which of its rows are ISO 128-50, which are ASME Y14.2M via `acadiso.pat`, and which (concrete, wood) have no mechanical standard at all.
|
||||
- **Six `mech_*` critique focuses** (`mech_missing_centreline`, `mech_unhatched_section`, `mech_view_misaligned`, `mech_duplicate_dimension`, `mech_thread_unrepresented`, `mech_bom_balloon_mismatch`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_MECH` payload. `mech_view_misaligned` checks the projection axis and not the quadrant, and `mech_duplicate_dimension` catches two dimensions in the same place rather than a semantic restatement — both narrowings are in the module docstring, because the payload carries no projection angle and no engine exposes what a dimension measured.
|
||||
- **Live ActiveX will not create a layer on assignment** (`entity.Layer = "X"` on a missing layer raises `Key not found`; ezdxf creates it silently). Every mechanical/annotation draw path ensures its target layers before the first entity — a new path must do the same, or it passes every headless test and dies live.
|
||||
- **A diameter's text stands beyond the FIRST chord point** on both engines (the ActiveX `AddDimDiametric` rule, measured); `dimension_diameter(x1, y1, x2, y2)` annotates on the `(x1, y1)` side.
|
||||
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
|
||||
|
||||
### Premium Drawing Rules
|
||||
|
||||
@@ -27,22 +27,23 @@ Live through COM on Windows, or headless through ezdxf anywhere — one typed co
|
||||
|
||||
</div>
|
||||
|
||||
> **v1.5 release snapshot:** 229 tools · 8 resources · 5 prompt templates · 3976 collected tests.
|
||||
> **v1.5 release snapshot:** 229 tools · 8 resources · 5 prompt templates · 3990 collected tests.
|
||||
> 229 is the **registered** count; a default install advertises 224 over `tools/list`,
|
||||
> because `ENABLE_3D` is unset. `system_about` is the runtime authority.
|
||||
|
||||
## Why this exists
|
||||
|
||||
**A big MCP server is expensive to be connected to.** The full catalog costs a client **59,315 tokens** before it has asked for anything. Discovery mode replaces it with two tools and costs **356**.
|
||||
**A big MCP server is expensive to be connected to.** The full catalog costs a client **72,163 tokens** before it has asked for anything. Discovery mode replaces it with two tools and costs **356**.
|
||||
|
||||
**A drafter searches for `FILLET`, not `entity_fillet`.** Those command names appeared in no tool name or description — `df = 0` against a stock index, not badly ranked but *absent*. The fix was data: an authored corpus of **193 AutoCAD command names and 1019 synonym phrases** covering all 215 tools. A test refuses to let a tool exist without one.
|
||||
**A drafter searches for `FILLET`, not `entity_fillet`.** Those command names appeared in no tool name or description — `df = 0` against a stock index, not badly ranked but *absent*. The fix was data: an authored corpus of **197 AutoCAD command names and 1158 synonym phrases** covering all 229 tools. A test refuses to let a tool exist without one.
|
||||
|
||||
| Advertised surface | Tools seen | Idle cost |
|
||||
|---|---:|---:|
|
||||
| `TOOL_PROFILE=full` (default) | 199 | 59,315 tokens |
|
||||
| `TOOL_PACKS=core,settings` (full profile) | 190 | 55,593 tokens |
|
||||
| `TOOL_PACKS=core` (full profile) | 168 | 48,825 tokens |
|
||||
| `TOOL_PROFILE=lean` | 55 | 16,360 tokens |
|
||||
| `TOOL_PROFILE=full` (default) | 224 | 72,163 tokens |
|
||||
| `TOOL_PACKS=core,settings` (full profile) | 201 | 60,727 tokens |
|
||||
| `TOOL_PACKS=core,mech` (full profile) | 193 | 61,673 tokens |
|
||||
| `TOOL_PACKS=core` (full profile) | 179 | 53,959 tokens |
|
||||
| `TOOL_PROFILE=lean` | 60 | 19,484 tokens |
|
||||
| `DISCOVERY_MODE=search` | 2 | **356 tokens** |
|
||||
|
||||
> [!NOTE]
|
||||
@@ -101,9 +102,11 @@ Claude Desktop, Cursor, or any stdio MCP host. For HTTP: `autocad-mcp --transpor
|
||||
| Geometry | lines, arcs, polylines, splines, hatches, trim/extend/fillet/chamfer, handle-preserving edits |
|
||||
| Annotation | ISO 129 toleranced dimensions, ISO 286 fits (`fit="H7"`), TABLE, MLEADER, GD&T frames and datums (ISO 1101) |
|
||||
| Engineering generators | involute gears (front + section A-A), DIN 6885 keyed bores, ISO A3 title block |
|
||||
| Mechanical parts | one part model (segments or outline + typed features) and one view engine: front / side / top / section / detail with hidden lines, ISO 128-50 cut faces and ISO 129 dimensions; ISO 4014/4017/4032/7089/4762 fasteners, ISO 15 bearings, DIN 471/472, DIN 509, DIN 332 and ISO 3601-2 as real blocks with attributes; ISO 21920-1 surface texture, ISO 2553 welds, ISO 128-40 section lines |
|
||||
| Sheet & delivery | ISO 5457 frames A4-A0 with zones and trim marks, ISO 7200 title blocks for every size, revision blocks with clouds, ISO 7573 parts lists with ISO 6433 balloons linked by XDATA, CSV/XLSX extraction, xrefs and images on both engines, real `.dwg` on a live seat |
|
||||
| P&ID | catalogue blocks with ports and tags (ISO 10628-2 / ISA-5.1), port-to-port lines with ISA-5.1 classes and line numbers, `pid_graph` reads any P&ID back with confidence, instrument index / line list / equipment list, `pid_from_spec` one-call sheets |
|
||||
| Styles & standards | ISO-25 / ANSI dimension styles, ISOCP / ROMANS text styles, ISO / ANSI leader styles from authored presets; `drawing_apply_standard("iso")` sets all of it plus units and the `mech` layers in one call; `changed` reports only what moved |
|
||||
| Discovery | `search_tools` ranked over an AutoCAD command and synonym corpus — `FILLET`, `BPOLY`, `QSELECT`, `WBLOCK`, `OVERKILL`, `CHSPACE` each rank **#1** of the 199-tool advertised catalog |
|
||||
| Discovery | `search_tools` ranked over an AutoCAD command and synonym corpus — `FILLET`, `BPOLY`, `QSELECT`, `WBLOCK`, `OVERKILL`, `CHSPACE` each rank **#1** of the 224-tool advertised catalog |
|
||||
| Batching | `cad_batch` runs a step list in one round trip; `fields=` projects 11 result-heavy tools |
|
||||
| Paper space | tab lifecycle, viewports, `entity_change_space` (CHSPACE), `page_setup_apply` (ISO 216 / ANSI Y14.1 paper, ctb, scale, device — on both engines), `batch_plot` with every sheet size read back from its PDF's `/MediaBox`, `drawing_export_pdf(layout=…)` |
|
||||
| Page setup & templates | ISO 216 / ANSI Y14.1 sheets, ctb catalog, `batch_plot` verified by the PDF's own `/MediaBox`, five bundled templates built by the server's own tools and pinned reproducible (`drawing_new(template="iso_a3_mech")`, `iso_a1_arch`, `iso_a3_pid`, `ansi_b_mech`, `ansi_d_arch`), save any drawing as a template (`.dwt` on live AutoCAD, `dwt_write` refused headlessly) |
|
||||
@@ -157,28 +160,29 @@ Self-measurement, produced by scripts in [`benchmarks/`](https://github.com/U-C4
|
||||
|
||||
### Correctness — every release re-proves itself
|
||||
|
||||
32 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.
|
||||
39 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|---|---:|---:|---:|---:|
|
||||
| v1.5.1 *(baseline)* | 26 / 32 | 81.2 % | — | — |
|
||||
| **v1.6.0-dev** *(this branch)* | **32 / 32** | **100 %** | 6 | **0** |
|
||||
| v1.5.1 *(baseline)* | 26 / 39 | 66.7 % | — | — |
|
||||
| **v1.6.0-dev** *(this branch)* | **39 / 39** | **100 %** | 13 | **0** |
|
||||
|
||||
The six fixed rows are new capability (`miss → pass`): the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.
|
||||
The thirteen fixed rows are new capability (`miss → pass`). Tracks B and G add seven: `mech_part_roundtrip` (a part read back out of its own `ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a 60 x ⌀40 sleeve with a ⌀20 bore cuts 1200 mm² exactly, measured out of the drawing by `analysis_measure_entity`), `mech_iso286_on_dimension` (40 H7 is +0.025 / 0 and survives the layout), `mech_thread_unrepresented_is_caught` (the ISO 6410 focus fires on a thread drawn as a plain circle — a gate that never fires is not a gate), `std_part_iso4014_m12` (s = 18, k = 7.5, read by value), `sheet_frame_iso5457_a3` (the frame runs 20,10 to 410,287 inside the 420 × 297 sheet) and `bom_balloon_link` (two identical bolts are one row of quantity 2, and its balloon carries the row's item number). The other six are the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.
|
||||
|
||||
### The task matrix — six tasks that can fail
|
||||
### The task matrix — tasks that can fail
|
||||
|
||||
An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds two more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness.
|
||||
An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds three more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness, and one where a whole mechanical sheet has to come out clean.
|
||||
|
||||
| Task | Verified against |
|
||||
|---|---|
|
||||
| `tool_discovery` | six AutoCAD command names, each ranking #1 |
|
||||
| `token_budget` | 59,315 → 356 tokens, against a ceiling fixed in advance |
|
||||
| `token_budget` | 72,163 → 356 tokens, against a ceiling fixed in advance |
|
||||
| `hatch_islands` | 300 filled with the island, 400 ignoring it |
|
||||
| `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 |
|
||||
| `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives |
|
||||
| `pid_roundtrip` | the example sheet drawn by `pid_from_spec`, read back by `pid_graph`, which never sees the spec: 5 nodes, 4 edges, 0 dangling, `confidence_min` 1.0, `FIC-101` wired to `FCV-101` |
|
||||
| `page_setup_truth` | ANSI B then ISO A3 landscape applied to Layout1, each plotted through `batch_plot` and read back from its PDF's `/MediaBox`: 432 × 279 then 420 × 297 mm, not the setter's return value (a fresh layout is already A3, so the B sheet is what a no-op setter cannot fake) |
|
||||
| `mech_assembly` | a flange-coupling A3 sheet — hub with a full section, flange with its bolt-circle end view, two ISO 4014 bolts with ISO 4032 nuts, an ISO 7573 parts list read off the drawing, linked balloons, ISO 5457 frame, ISO 7200 title block — gated on `drawing_critique(focus=None)` returning **zero** issues and a finalize score of at least **90**; a test drops one balloon and watches the gate fail |
|
||||
|
||||
### Headless performance
|
||||
|
||||
|
||||
@@ -1346,6 +1346,18 @@ def _entity_info(entity) -> EntityInfo:
|
||||
geometry_bbox = _com_geometry_bbox(entity)
|
||||
if geometry_bbox is not None:
|
||||
props["geometry_bbox"] = geometry_bbox
|
||||
elif "Dimension" in obj_name:
|
||||
# ezdxf parity: where the dimension's text stands. Without it the
|
||||
# dim_overlap critique fell back to the bounding-box centre, which
|
||||
# for a diameter dimension is the circle's centre - so an outside
|
||||
# diameter and its bore, whose texts stand 15 mm apart, read as
|
||||
# "0.00 mm apart" (measured on AutoCAD 2026 by
|
||||
# scripts/smoke_mech_com.py; TextPosition read live on
|
||||
# AcDbDiametricDimension, WCS). ModelSpace.Item hands the entity
|
||||
# over narrowed to IAcadEntity, which has no TextPosition - measured
|
||||
# AttributeError - so it is read through the concrete class.
|
||||
text = _com_unnarrow(entity).TextPosition
|
||||
props["text_position"] = [text[0], text[1]]
|
||||
except Exception as exc:
|
||||
log.debug("Type-specific entity properties extraction failed: %s", exc)
|
||||
|
||||
|
||||
@@ -5550,7 +5550,15 @@ class EzdxfBackend(AutoCADBackend):
|
||||
cx = (x1f + x2f) / 2
|
||||
cy = (y1f + y2f) / 2
|
||||
radius = math.sqrt((x2f - x1f) ** 2 + (y2f - y1f) ** 2) / 2
|
||||
angle_rad = math.atan2(y2f - y1f, x2f - x1f)
|
||||
# The text stands `leader_length` beyond the FIRST chord point, the
|
||||
# ActiveX rule: AddDimDiametric(ChordPoint, FarChordPoint,
|
||||
# LeaderLength) measured on AutoCAD 2026 put a 50 mm diameter drawn
|
||||
# from (x, 170) to (x, 220) with its text at (x, 160). This engine
|
||||
# used to put it beyond the second point, so the same call annotated
|
||||
# opposite sides of a part on the two engines - and the mechanical
|
||||
# drawer's front-view diameters landed on their own chain dimensions
|
||||
# headlessly while clearing them live.
|
||||
angle_rad = math.atan2(y1f - y2f, x1f - x2f)
|
||||
leader = float(leader_length)
|
||||
# The measured size comes from `radius`; `location` only places the
|
||||
# text. This used to pass centre + (radius + leader) as `mpoint`,
|
||||
|
||||
+40
-9
@@ -41,13 +41,13 @@ Unsupported tasks remain in the fixed-matrix denominator with score zero,
|
||||
preventing partial implementations from receiving an inflated score.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --list --matrix v4
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v4 --json
|
||||
python -m benchmarks.run_competitors --list --matrix v5
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v5 --json
|
||||
python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
|
||||
```
|
||||
|
||||
The release-machine ezdxf self-check is **17/17 (100.0)** on the v4 matrix
|
||||
(`--matrix v3` / `--matrix v2` reproduce the earlier sets).
|
||||
The release-machine ezdxf self-check is **18/18 (100.0)** on the v5 matrix
|
||||
(`--matrix v4` / `--matrix v3` / `--matrix v2` reproduce the earlier sets).
|
||||
Repository stars and raw tool counts do not contribute to the score. Adapter
|
||||
registration lives in `competitors.yaml`.
|
||||
|
||||
@@ -100,6 +100,23 @@ fail on their own:
|
||||
The competitor reports carry no result for either, for the same reason they
|
||||
carry none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.
|
||||
|
||||
### Matrix v5 (v1.6) — the mechanical sheet
|
||||
|
||||
`tasks_v5.py` keeps the v4 seventeen unchanged and adds roadmap criterion 3:
|
||||
|
||||
| Task | Category | Verified against |
|
||||
|---|---|---|
|
||||
| `mech_assembly` | mech | a flange-coupling A3 sheet built through the code the tools call — `apply_titleblock` (ISO 5457 frame with zones and marks, ISO 7200 title block), a hub drawn by `draw_part` with a full section by `add_view` and chain dimensions by `dimension_part`, a flange with its bolt-circle end view, two ISO 4014 M12x60 bolts and two ISO 4032 M12 nuts by `insert_std_part`, the ISO 7573 parts list *read off the drawing* by `extract_records` and drawn by `draw_bom_table`, and one balloon per row linked to that row's inserts by `add_balloon`. The sheet is saved, then gated on two numbers that cannot be negotiated: `run_critique(focus=None)` returns **zero** issues and `combine(validator, critique)` scores at least **90**. The focuses that fired come back in the metrics, and `tests/test_benchmark_v5.py` drops one balloon to prove the gate fails and names `mech_bom_balloon_mismatch` |
|
||||
|
||||
Nothing in the task tidies up after the tools: the view engine draws the ISO
|
||||
128-23 centre lines of every circle it emits, so the sheet is clean because it
|
||||
was drawn clean. Building it found four defects that every headless test had
|
||||
missed — the headless diameter text on the wrong side of the chord, the
|
||||
validator reading a fastener designation (`M12x60`) as a fake dimension, and on
|
||||
the live seat a missing `MECH` layer and a parts list that could not see a
|
||||
`BLOCKREFERENCE`; each is fixed with a test. The competitor reports carry no
|
||||
result for it.
|
||||
|
||||
## Live competitor lane (v1.4)
|
||||
|
||||
Two competitor adapters now execute the exact same task matrix, black-box over
|
||||
@@ -217,15 +234,29 @@ python benchmarks/compare_versions.py --json results.json
|
||||
|
||||
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
|
||||
|
||||
32 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
39 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|---------|----------------|-----------|-------|-----------|
|
||||
| **v1.5.1** (baseline) | 26 / 32 | 81.2 % | — | — |
|
||||
| **v1.6.0-dev** (this branch) | 32 / 32 | 100 % | 6 | **0** |
|
||||
| **v1.5.1** (baseline) | 26 / 39 | 66.7 % | — | — |
|
||||
| **v1.6.0-dev** (this branch) | 39 / 39 | 100 % | 13 | **0** |
|
||||
|
||||
The six are all `miss → pass` — v1.5.1 has none of the methods: the three
|
||||
The thirteen are all `miss → pass` — v1.5.1 has none of the methods. Seven
|
||||
are tracks B and G: `mech_part_roundtrip` (a part read back from its own
|
||||
`ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a
|
||||
sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces,
|
||||
1200 mm² exactly, measured out of the drawing by `entity_measure` — never from
|
||||
the code that drew it), `mech_iso286_on_dimension` (ISO 286-1: 40 H7 is
|
||||
+0.025 / 0, and the fit survives `layout_dimensions`),
|
||||
`mech_thread_unrepresented_is_caught` (the ISO 6410 focus fires on a thread
|
||||
drawn as a plain circle and stops firing once the thin ¾ arc is there — the
|
||||
mechanical twin of `dim_overlap_critique_fires`), `std_part_iso4014_m12` (ISO
|
||||
4014:2011 table 1, M12: s = 18, k = 7.5, matched by value; the inserted block
|
||||
carries its designation back), `sheet_frame_iso5457_a3` (the frame runs 20,10
|
||||
to 410,287 inside the 420 × 297 trimmed sheet) and `bom_balloon_link` (two
|
||||
identical bolts collapse into one row of quantity 2 with a stable item number,
|
||||
and that row's balloon carries it). The other six are the three
|
||||
P&ID checks track A added (`pid_block_define_attdef_roundtrip`,
|
||||
`pid_tag_parse_fic`, `pid_graph_edge_count`) and the three settings checks of
|
||||
track E (`settings_dimstyle_iso25_values`: the ISO-25 preset reaches the
|
||||
@@ -234,7 +265,7 @@ through the lister; `settings_layer_state_roundtrip`: save → change → restor
|
||||
puts the layer table back and the state survives save/reopen as an XRECORD in
|
||||
the file; `settings_pdf_mediabox_a3`: the plotted PDF's own `/MediaBox` reads
|
||||
420 × 297 mm). Every one of the 26 checks it was released on still passes, so
|
||||
the two tracks added capability without moving a number it had already earned.
|
||||
the four tracks added capability without moving a number it had already earned.
|
||||
|
||||
### Result — v1.5.1 vs v1.5.0
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Reference adapter for this repository, covering the v4 task matrix.
|
||||
"""Reference adapter for this repository, covering the v5 task matrix.
|
||||
|
||||
Every task verifies against a value worked out independently of the code under
|
||||
test — a closed-form area, a count of entities placed on purpose, a token
|
||||
@@ -12,7 +12,7 @@ import math
|
||||
from pathlib import Path
|
||||
|
||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||
from benchmarks.tasks_v4 import TaskSpec
|
||||
from benchmarks.tasks_v5 import TaskSpec
|
||||
from engineering.delivery import deliver_drawing
|
||||
from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
|
||||
from engineering.refiner import refine_drawing
|
||||
@@ -514,6 +514,174 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
|
||||
[str(row_before["path"]), str(row["path"])],
|
||||
)
|
||||
|
||||
async def _task_mech_assembly(self):
|
||||
"""A flange-coupling sheet built from the part model and the sheet standard.
|
||||
|
||||
Roadmap criterion 3. Two parts (a hub with a bore and a flange), a full
|
||||
section through the hub, four standard fasteners with their parts-list
|
||||
rows and balloons, an ISO 5457 A3 frame and an ISO 7200 title block.
|
||||
Every step goes through the code the tools call: the frame and title
|
||||
block through ``apply_titleblock``, the parts list is *read* off the
|
||||
drawing by ``extract_records`` and drawn by ``draw_bom_table``, and each
|
||||
balloon is linked to its row's inserts by ``add_balloon``.
|
||||
|
||||
The gate is two numbers, and both are asserted rather than reported:
|
||||
``run_critique(focus=None)`` must return zero issues, and the score the
|
||||
finalize gate computes - ``combine(validator.summary, critique)`` over
|
||||
the saved sheet - must be at least 90. A regression in either fails the
|
||||
task; the focuses that fired come back in the metrics so the failure
|
||||
names itself.
|
||||
|
||||
No step here repairs what the critique would flag: the view engine
|
||||
draws the ISO 128-23 centre lines of every circle it emits (the end
|
||||
view's bore and bolt holes included), so the sheet is clean because
|
||||
the tools drew it clean, not because the benchmark tidied up after them.
|
||||
"""
|
||||
from engineering import DrawingValidator
|
||||
from engineering.critique import run_critique
|
||||
from engineering.layers import apply_layer_set
|
||||
from engineering.mech.draw import add_view, dimension_part, draw_part
|
||||
from engineering.mech.part import build_part
|
||||
from engineering.mech.stdparts import insert_std_part
|
||||
from engineering.scoring import combine
|
||||
from engineering.sheet.bom import (
|
||||
add_balloon,
|
||||
draw_bom_table,
|
||||
extract_records,
|
||||
rows_from_records,
|
||||
)
|
||||
from engineering.sheet.titleblock import (
|
||||
TB_HEIGHT,
|
||||
TitleBlockMetadata,
|
||||
apply_titleblock,
|
||||
titleblock_origin,
|
||||
)
|
||||
|
||||
await apply_layer_set(self.backend, "mech")
|
||||
await apply_titleblock(
|
||||
self.backend,
|
||||
size="A3",
|
||||
metadata=TitleBlockMetadata(
|
||||
title="FLANGE COUPLING",
|
||||
drawing_no="MA-001",
|
||||
drawn_by="autocad-mcp-pro",
|
||||
scale="1:1",
|
||||
material="S235JR",
|
||||
),
|
||||
projection="first",
|
||||
frame=True,
|
||||
zones=True,
|
||||
marks=True,
|
||||
)
|
||||
|
||||
hub = build_part(
|
||||
{
|
||||
"kind": "revolved",
|
||||
"name": "HUB",
|
||||
"material": "steel",
|
||||
"segments": [
|
||||
{"length": 40.0, "d_outer": 40.0, "d_inner": 20.0},
|
||||
{"length": 15.0, "d_outer": 90.0, "d_inner": 20.0},
|
||||
],
|
||||
}
|
||||
)
|
||||
flange = build_part(
|
||||
{
|
||||
"kind": "revolved",
|
||||
"name": "FLANGE",
|
||||
"material": "cast_iron",
|
||||
"segments": [{"length": 15.0, "d_outer": 90.0, "d_inner": 40.0}],
|
||||
# Four M12 clearance holes (ISO 273 medium, 13.5) on a 65 PCD.
|
||||
"features": [
|
||||
{
|
||||
"kind": "hole_circle",
|
||||
"id": "bolts",
|
||||
"x": 7.5,
|
||||
"pcd": 65.0,
|
||||
"count": 4,
|
||||
"diameter": 13.5,
|
||||
"start_angle": 45.0,
|
||||
}
|
||||
],
|
||||
}
|
||||
)
|
||||
# A3 landscape: the frame runs (20, 10)-(410, 287) and the title block
|
||||
# fills its lower-right corner, 180 x 60. First angle throughout: the
|
||||
# hub's front view sits at x = 125 so its section lands left of it
|
||||
# inside the frame; the flange's front view stands at x = 345 so its
|
||||
# end view (the bolt circle) lands between the two parts; the
|
||||
# fasteners take the free band below, left of the title block.
|
||||
drawn_hub = await draw_part(
|
||||
self.backend, hub, at=(125.0, 160.0), views=("front",), dimension=False
|
||||
)
|
||||
# A full longitudinal section along the hub's axis (part-local frame).
|
||||
await add_view(
|
||||
self.backend,
|
||||
drawn_hub["part_id"],
|
||||
"section",
|
||||
plane={"p1": [0.0, 0.0], "p2": [55.0, 0.0], "label": "A"},
|
||||
style="full",
|
||||
)
|
||||
await dimension_part(self.backend, drawn_hub["part_id"], style="chain")
|
||||
await draw_part(
|
||||
self.backend, flange, at=(345.0, 160.0), views=("front", "side"), dimension=True
|
||||
)
|
||||
|
||||
# Two bolt-and-nut pairs: each nut sits on the thread of its bolt.
|
||||
stations = (60.0, 110.0, 160.0, 210.0)
|
||||
fasteners = []
|
||||
for index, x in enumerate(stations):
|
||||
designation = "ISO 4014 - M12x60" if index % 2 == 0 else "ISO 4032 - M12"
|
||||
fasteners.append(
|
||||
await insert_std_part(self.backend, designation, at=(x, 90.0), view="side")
|
||||
)
|
||||
|
||||
# The parts list is a read of the drawing, not a list typed beside it.
|
||||
records = await extract_records(self.backend)
|
||||
rows = rows_from_records(records)
|
||||
tb_x, tb_y = titleblock_origin("A3")
|
||||
await draw_bom_table(self.backend, rows, at=(tb_x, tb_y + TB_HEIGHT))
|
||||
station_of = {
|
||||
str(result.get("handle")): x for result, x in zip(fasteners, stations, strict=True)
|
||||
}
|
||||
for row in rows:
|
||||
targets = [str(handle) for handle in row.get("handles") or ()]
|
||||
x = station_of.get(targets[0], stations[0]) if targets else stations[0]
|
||||
await add_balloon(
|
||||
self.backend,
|
||||
item=int(row["item"]),
|
||||
at=(x, 135.0),
|
||||
leader_to=(x, 95.0),
|
||||
targets=targets,
|
||||
)
|
||||
|
||||
# drawing_finalize saves before it validates; so does the benchmark,
|
||||
# or the validator's not_saved/no_path findings would grade the
|
||||
# harness instead of the sheet.
|
||||
sheet = self.artifact_dir / "mech_assembly.dxf"
|
||||
await self.backend.drawing_save_as(str(sheet))
|
||||
issues = await run_critique(self.backend, None)
|
||||
validation = await DrawingValidator().run(self.backend)
|
||||
summary = {"error": 0, "warning": 0, "info": 0}
|
||||
for issue in issues:
|
||||
summary[issue.severity] = summary.get(issue.severity, 0) + 1
|
||||
score = combine(validation.summary, summary)
|
||||
passed = len(issues) == 0 and float(score["score"]) >= 90.0
|
||||
return (
|
||||
passed,
|
||||
{
|
||||
"critique_issues": len(issues),
|
||||
"critique_focuses": sorted({issue.focus for issue in issues}),
|
||||
"score": float(score["score"]),
|
||||
"grade": score["grade"],
|
||||
"invalidity_ratio": score["invalidity_ratio"],
|
||||
"parts_list_rows": len(rows),
|
||||
"fasteners": len(fasteners),
|
||||
"validator_findings": sorted(f.code for f in validation.findings),
|
||||
},
|
||||
[str(sheet)],
|
||||
)
|
||||
|
||||
async def _task_auditable_delivery(self):
|
||||
await ensure_engineering_layers(self.backend)
|
||||
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
|
||||
|
||||
@@ -417,6 +417,231 @@ async def settings_pdf_mediabox_a3():
|
||||
)
|
||||
|
||||
|
||||
# ── Mechanical parts and the sheet (tracks B + G) ───────────────────────────
|
||||
|
||||
|
||||
def _field(row: dict, *names):
|
||||
"""A row's value under whichever of these names it carries.
|
||||
|
||||
The parts-list column spelling belongs to the sheet module; what this
|
||||
suite pins is the numbers, so it reads by meaning rather than by name.
|
||||
"""
|
||||
for name in names:
|
||||
if name in row:
|
||||
return row[name]
|
||||
return None
|
||||
|
||||
|
||||
def _inserted_handle(result: dict):
|
||||
return result.get("handle") or (result.get("handles") or [None])[0]
|
||||
|
||||
|
||||
async def mech_part_roundtrip():
|
||||
"""Draw a part, read it back from its own ACADMCP_MECH XDATA, compare.
|
||||
|
||||
Segments and material only: a feature's parameter spelling belongs to
|
||||
`engineering/mech/features.py` and its own tests, and a check that guessed
|
||||
at it would fail for the wrong reason.
|
||||
"""
|
||||
from engineering.mech.draw import draw_part, read_part
|
||||
from engineering.mech.part import build_part, part_to_dict
|
||||
|
||||
b = await _b()
|
||||
spec = {
|
||||
"kind": "revolved",
|
||||
"name": "SHAFT",
|
||||
"material": "steel",
|
||||
"segments": [
|
||||
{"length": 40.0, "d_outer": 30.0},
|
||||
{"length": 20.0, "d_outer": 50.0, "d_inner": 10.0},
|
||||
],
|
||||
}
|
||||
part = build_part(spec)
|
||||
drawn = await draw_part(b, part, at=(0.0, 0.0), views=("front",), dimension=False)
|
||||
back = await read_part(b, drawn["part_id"])
|
||||
return part_to_dict(build_part(back["part"])) == part_to_dict(part)
|
||||
|
||||
|
||||
async def mech_section_hatch_area():
|
||||
"""The hatched cut area of a full section, measured against a hand number.
|
||||
|
||||
A plain sleeve: length 60, outside diameter 40, bore 20. A full
|
||||
longitudinal section cuts two faces, each 60 x 10, so the hatch fills
|
||||
60 * (40 - 20) = 1200 mm^2 exactly. The number is read out of the drawing
|
||||
through entity_measure (the tool's analysis_measure_entity), never from the
|
||||
code that drew it — which is why it cannot be faked.
|
||||
"""
|
||||
from engineering.mech.draw import add_view, draw_part
|
||||
from engineering.mech.part import build_part
|
||||
|
||||
b = await _b()
|
||||
part = build_part(
|
||||
{
|
||||
"kind": "revolved",
|
||||
"name": "SLEEVE",
|
||||
"material": "steel",
|
||||
"segments": [{"length": 60.0, "d_outer": 40.0, "d_inner": 20.0}],
|
||||
}
|
||||
)
|
||||
drawn = await draw_part(b, part, at=(0.0, 0.0), views=("front",), dimension=False)
|
||||
await add_view(
|
||||
b,
|
||||
drawn["part_id"],
|
||||
"section",
|
||||
plane={"p1": [0.0, 0.0], "p2": [60.0, 0.0], "label": "A"},
|
||||
style="full",
|
||||
)
|
||||
total = 0.0
|
||||
for ent in await b.entity_list(type_filter="HATCH", limit=500):
|
||||
total += float((await b.entity_measure(ent.handle))["area"])
|
||||
return abs(total - 1200.0) < 0.01
|
||||
|
||||
|
||||
async def mech_iso286_on_dimension():
|
||||
"""A 40 H7 bore dimension carries ISO 286's +0.025 / 0 through the layout.
|
||||
|
||||
ISO 286-1 table 1: the 30-50 mm step has IT7 = 25 um, and H fixes the lower
|
||||
deviation at 0. The fit has to survive `layout_dimensions`, or the number
|
||||
is right in the table and wrong on the sheet.
|
||||
"""
|
||||
from engineering.fits import fit_lookup
|
||||
from engineering.mech.dimension import layout_dimensions
|
||||
from engineering.mech.primitives import Circle, DimIntent
|
||||
from engineering.mech.views import View
|
||||
|
||||
deviation = fit_lookup("H7", 40.0)
|
||||
if abs(deviation.upper_mm - 0.025) > 1e-9 or abs(deviation.lower_mm) > 1e-9:
|
||||
return False
|
||||
intent = DimIntent("diameter", (-20.0, 0.0), (20.0, 0.0), fit="H7", feature="bore")
|
||||
view = View(
|
||||
kind="front",
|
||||
prims=(Circle((0.0, 0.0), 20.0),),
|
||||
dims=(intent,),
|
||||
omitted=(),
|
||||
bbox=(-20.0, -20.0, 20.0, 20.0),
|
||||
)
|
||||
placed = layout_dimensions(view, [intent])
|
||||
return len(placed) == 1 and placed[0].fit == "H7" and placed[0].feature == "bore"
|
||||
|
||||
|
||||
async def mech_thread_unrepresented_is_caught():
|
||||
"""The ISO 6410 focus fires on a thread drawn as a plain circle.
|
||||
|
||||
A quality gate that never fires is not a gate; this is the mechanical twin
|
||||
of `dim_overlap_critique_fires`.
|
||||
"""
|
||||
from engineering.mech.critique import build_index, issues_for
|
||||
from engineering.mech.xdata import APP_ID, to_values
|
||||
|
||||
b = await _b()
|
||||
payload = {
|
||||
"v": 1,
|
||||
"kind": "part",
|
||||
"id": "P1",
|
||||
"view": "side",
|
||||
"part": {
|
||||
"name": "STUD",
|
||||
"material": "steel",
|
||||
"segments": [{"length": 60.0, "d_outer": 20.0}],
|
||||
"features": [{"kind": "thread", "id": "t1", "designation": "M20x1.5"}],
|
||||
},
|
||||
}
|
||||
anchor = await b.entity_create_circle(0.0, 0.0, 0.5, layer="0")
|
||||
await b.entity_set_xdata(anchor.handle, APP_ID, to_values(payload))
|
||||
await b.entity_create_circle(0.0, 0.0, 10.0, layer="GEOMETRY")
|
||||
before = issues_for("mech_thread_unrepresented", await build_index(b))
|
||||
await b.entity_create_arc(0.0, 0.0, 8.0, 0.0, 270.0, layer="GEOMETRY")
|
||||
after = issues_for("mech_thread_unrepresented", await build_index(b))
|
||||
return len(before) == 1 and after == []
|
||||
|
||||
|
||||
async def sheet_frame_iso5457_a3():
|
||||
"""ISO 5457 A3: 420 x 297 trimmed, a 20 mm filing margin on the left and
|
||||
10 mm on the other three edges, so the frame runs (20, 10) to (410, 287)
|
||||
and the drawing space is 390 x 277. The trimmed sheet edge is drawn too
|
||||
(ISO 5457 4.1), so the outermost extent is the sheet, 0,0 to 420,297 -
|
||||
and nothing may reach past it."""
|
||||
from engineering.mech.primitives import points_of
|
||||
from engineering.sheet.frames import SHEETS, frame_prims
|
||||
|
||||
if tuple(float(v) for v in SHEETS["A3"]) != (297.0, 420.0):
|
||||
return False
|
||||
prims = frame_prims("A3", orientation="landscape", zones=False, marks=False)
|
||||
points = {
|
||||
(round(float(x), 6), round(float(y), 6)) for prim in prims for x, y in points_of(prim)
|
||||
}
|
||||
frame = {(20.0, 10.0), (410.0, 10.0), (410.0, 287.0), (20.0, 287.0)}
|
||||
sheet = {(0.0, 0.0), (420.0, 0.0), (420.0, 297.0), (0.0, 297.0)}
|
||||
xs = [x for x, _y in points]
|
||||
ys = [y for _x, y in points]
|
||||
extent = (min(xs), min(ys), max(xs), max(ys))
|
||||
return frame <= points and sheet <= points and extent == (0.0, 0.0, 420.0, 297.0)
|
||||
|
||||
|
||||
async def bom_balloon_link():
|
||||
"""Two identical bolts collapse into one row of quantity 2, the item number
|
||||
is stable across a rerun, and the balloon drawn for that row carries it."""
|
||||
from engineering.mech.stdparts import insert_std_part
|
||||
from engineering.mech.xdata import APP_ID, decode
|
||||
from engineering.sheet.bom import balloon_prims, rows_from_records
|
||||
|
||||
b = await _b()
|
||||
records = []
|
||||
for x in (0.0, 60.0):
|
||||
result = await insert_std_part(b, "ISO 4014 - M12x60", at=(x, 0.0), view="side")
|
||||
raw = await b.entity_get_xdata(_inserted_handle(result), APP_ID)
|
||||
records.append(decode(raw["xdata"][APP_ID]))
|
||||
rows = rows_from_records(records)
|
||||
again = rows_from_records(records)
|
||||
if len(rows) != 1 or len(again) != 1:
|
||||
return False
|
||||
item = _field(rows[0], "item", "no", "pos")
|
||||
if item is None or item != _field(again[0], "item", "no", "pos"):
|
||||
return False
|
||||
if int(_field(rows[0], "qty", "quantity") or 0) != 2:
|
||||
return False
|
||||
texts = [
|
||||
p.text for p in balloon_prims(int(item), (0.0, 40.0), (0.0, 5.0)) if hasattr(p, "text")
|
||||
]
|
||||
return texts == [str(int(item))]
|
||||
|
||||
|
||||
async def std_part_iso4014_m12():
|
||||
"""ISO 4014's M12 row and the block it becomes.
|
||||
|
||||
ISO 4014:2011 table 1, M12: width across flats s = 18 mm, head height
|
||||
k = 7.5 mm. The row is matched by value rather than by column name — the
|
||||
spelling is the catalogue's business, the numbers are the standard's — and
|
||||
the inserted block has to carry its designation back in ACADMCP_MECH.
|
||||
The catalogue nests a row's sizes one level down (``dims``), so the
|
||||
values are read one level deep.
|
||||
"""
|
||||
from engineering.mech.stdparts import catalogue, insert_std_part
|
||||
from engineering.mech.xdata import APP_ID, decode
|
||||
|
||||
rows = [row for row in catalogue("M12") if any("4014" in str(value) for value in row.values())]
|
||||
if not rows:
|
||||
return False
|
||||
values = [
|
||||
inner
|
||||
for row in rows
|
||||
for value in row.values()
|
||||
for inner in (value.values() if isinstance(value, dict) else (value,))
|
||||
]
|
||||
numbers = {
|
||||
round(float(value), 3)
|
||||
for value in values
|
||||
if isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||
}
|
||||
if not {18.0, 7.5} <= numbers:
|
||||
return False
|
||||
b = await _b()
|
||||
result = await insert_std_part(b, "ISO 4014 - M12x60", at=(0.0, 0.0), view="side")
|
||||
raw = await b.entity_get_xdata(_inserted_handle(result), APP_ID)
|
||||
payload = decode(raw["xdata"][APP_ID])
|
||||
return payload["kind"] == "std_part" and "M12" in str(payload["designation"])
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"core_line_length": (core_line_length, "Core"),
|
||||
"core_circle_radius": (core_circle_radius, "Core"),
|
||||
@@ -450,6 +675,13 @@ CHECKS = {
|
||||
"settings_dimstyle_iso25_values": (settings_dimstyle_iso25_values, "Settings"),
|
||||
"settings_layer_state_roundtrip": (settings_layer_state_roundtrip, "Settings"),
|
||||
"settings_pdf_mediabox_a3": (settings_pdf_mediabox_a3, "Settings"),
|
||||
"mech_part_roundtrip": (mech_part_roundtrip, "Mechanical"),
|
||||
"mech_section_hatch_area": (mech_section_hatch_area, "Mechanical"),
|
||||
"mech_iso286_on_dimension": (mech_iso286_on_dimension, "Mechanical"),
|
||||
"mech_thread_unrepresented_is_caught": (mech_thread_unrepresented_is_caught, "Mechanical"),
|
||||
"std_part_iso4014_m12": (std_part_iso4014_m12, "Mechanical"),
|
||||
"sheet_frame_iso5457_a3": (sheet_frame_iso5457_a3, "Sheet"),
|
||||
"bom_balloon_link": (bom_balloon_link, "Sheet"),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from benchmarks.tasks_v4 import TASKS_V4
|
||||
from benchmarks.tasks_v5 import TASKS_V5
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
|
||||
@@ -66,8 +66,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||
|
||||
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
|
||||
|
||||
task_ids = [task.task_id for task in TASKS_V4]
|
||||
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V4]
|
||||
task_ids = [task.task_id for task in TASKS_V5]
|
||||
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V5]
|
||||
servers = [report["adapter"] for report in reports]
|
||||
status_by_server = {
|
||||
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"summary": {
|
||||
"baseline_ref": "v1.5.1",
|
||||
"total": 32,
|
||||
"total": 39,
|
||||
"old_pass": 26,
|
||||
"new_pass": 32,
|
||||
"old_pct": 81.2,
|
||||
"new_pass": 39,
|
||||
"old_pct": 66.7,
|
||||
"new_pct": 100.0,
|
||||
"fixed": 6,
|
||||
"fixed": 13,
|
||||
"regressed": 0
|
||||
},
|
||||
"rows": [
|
||||
@@ -201,6 +201,48 @@
|
||||
"category": "Settings",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_part_roundtrip",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_section_hatch_area",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_iso286_on_dimension",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_thread_unrepresented_is_caught",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "std_part_iso4014_m12",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "sheet_frame_iso5457_a3",
|
||||
"category": "Sheet",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "bom_balloon_link",
|
||||
"category": "Sheet",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -16,7 +16,7 @@ from pathlib import Path
|
||||
|
||||
from backends.capability import UnsupportedCapabilityError
|
||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||
from benchmarks.tasks_v4 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
|
||||
from benchmarks.tasks_v5 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
|
||||
|
||||
|
||||
def _load_registry() -> dict[str, dict]:
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
"""The v5 task matrix: v4 plus what tracks B and G added.
|
||||
|
||||
``mech_assembly`` (tracks B + G) is roadmap criterion 3: a flange-coupling
|
||||
sheet built from the tools — two parts with a section, a bolt circle of ISO
|
||||
4014 bolts and ISO 4032 nuts, ISO 129 dimensions, an ISO 5457 A3 frame, an
|
||||
ISO 7200 title block and an ISO 7573 parts list with balloons. The gate is two
|
||||
numbers, not an impression: ``drawing_critique(focus=None)`` returns zero
|
||||
issues and the finalize score is at least 90. Either one slipping fails the
|
||||
task outright.
|
||||
|
||||
The published competitor reports were never asked this question; the chart
|
||||
shows ``not_run`` for them, not zero.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from benchmarks.tasks_v2 import TASKS_V2
|
||||
from benchmarks.tasks_v3 import TASKS_V3
|
||||
from benchmarks.tasks_v4 import TASKS_V4, TaskSpec
|
||||
|
||||
#: Added in v1.6.0 by tracks B + G.
|
||||
NEW_TASKS_V5: tuple[TaskSpec, ...] = (
|
||||
TaskSpec(
|
||||
"mech_assembly",
|
||||
"mech",
|
||||
"Draw a flange-coupling sheet from the part model and the sheet standard, "
|
||||
"with a clean critique and a finalize score of at least 90",
|
||||
1.0,
|
||||
),
|
||||
)
|
||||
|
||||
TASKS_V5: tuple[TaskSpec, ...] = TASKS_V4 + NEW_TASKS_V5
|
||||
|
||||
#: Runner-selectable task sets. v2, v3 and v4 stay addressable so an earlier
|
||||
#: report can be reproduced exactly rather than only described.
|
||||
MATRICES: dict[str, tuple[TaskSpec, ...]] = {
|
||||
"v2": TASKS_V2,
|
||||
"v3": TASKS_V3,
|
||||
"v4": TASKS_V4,
|
||||
"v5": TASKS_V5,
|
||||
}
|
||||
|
||||
DEFAULT_MATRIX = "v5"
|
||||
|
||||
|
||||
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
||||
for task in MATRICES[matrix]:
|
||||
if task.task_id == task_id:
|
||||
return task
|
||||
raise KeyError(f"{task_id} is not in matrix {matrix}")
|
||||
@@ -488,6 +488,7 @@ async def idle_lane(counter: Any, turns: int = SESSION_TURNS) -> list[dict[str,
|
||||
"""What a client pays on connect, for each advertised surface."""
|
||||
srv = _load_server()
|
||||
previous_profile = config.settings.tool_profile
|
||||
previous_packs = config.settings.tool_packs
|
||||
rows: list[dict[str, Any]] = []
|
||||
try:
|
||||
config.settings.tool_profile = "full"
|
||||
@@ -510,6 +511,17 @@ async def idle_lane(counter: Any, turns: int = SESSION_TURNS) -> list[dict[str,
|
||||
)
|
||||
)
|
||||
config.settings.tool_profile = "full"
|
||||
config.settings.tool_packs = "core,mech"
|
||||
rows.append(
|
||||
await _idle_row(
|
||||
"packs_core_mech",
|
||||
"TOOL_PACKS=core,mech - a mechanical client that never opens a "
|
||||
"P&ID or a live seat.",
|
||||
counter,
|
||||
turns,
|
||||
)
|
||||
)
|
||||
config.settings.tool_packs = previous_packs
|
||||
srv._apply_discovery_mode("search")
|
||||
rows.append(
|
||||
await _idle_row(
|
||||
@@ -523,6 +535,9 @@ async def idle_lane(counter: Any, turns: int = SESSION_TURNS) -> list[dict[str,
|
||||
finally:
|
||||
srv._apply_discovery_mode("off")
|
||||
config.settings.tool_profile = previous_profile
|
||||
# restored here too, so a failed measurement cannot leave the process
|
||||
# on a narrowed surface
|
||||
config.settings.tool_packs = previous_packs
|
||||
await srv._apply_tool_profile(previous_profile)
|
||||
return rows
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# v1.6 Tracks B + G — the complete mechanical 2D part drawer and the sheet standard
|
||||
|
||||
**Date:** 2026-09-22 · **Status:** design approved in conversation ·
|
||||
**Date:** 2026-09-22 · **Status:** shipped (v1.6.0-dev, tracks B + G) ·
|
||||
**Parent:** `2026-09-15-v1.6-roadmap-design.md` (tracks B and G, merged into one
|
||||
spec by the user's decision) · **Baseline:** main after Track E
|
||||
(`39e6e2f`, 204 tools, 8 resources, 3283 tests)
|
||||
|
||||
@@ -423,6 +423,13 @@ def segment_key(index: int, *, bore: bool = False) -> str:
|
||||
|
||||
|
||||
def _revolved_diameters(part: RevolvedPart) -> list[DimIntent]:
|
||||
"""Each segment's outside diameter and bore, measured across the axis.
|
||||
|
||||
``p1`` is the upper chord end: a diameter's text stands beyond ``p1`` (the
|
||||
ActiveX ``AddDimDiametric`` ChordPoint rule, which both engines follow), so
|
||||
every front-view diameter is annotated above the view, clear of the axial
|
||||
chain rows below it.
|
||||
"""
|
||||
intents: list[DimIntent] = []
|
||||
bounds = zip(segment_bounds(part), part.segments, strict=True)
|
||||
for index, ((x0, x1), segment) in enumerate(bounds):
|
||||
|
||||
@@ -109,6 +109,7 @@ async def draw_prims(
|
||||
"""Draw primitive descriptions, turned by ``rotation`` about their origin, then placed at ``at``."""
|
||||
placed = tuple(prims)
|
||||
_refuse_islands_without_edge_paths(backend, placed)
|
||||
await _ensure_layers(backend, sorted({f"{layer_prefix}{layer_for(p)}" for p in placed}))
|
||||
if abs(float(rotation)) > 1e-12:
|
||||
placed = rotate(placed, float(rotation), (0.0, 0.0))
|
||||
if abs(float(at[0])) > 1e-12 or abs(float(at[1])) > 1e-12:
|
||||
@@ -195,6 +196,23 @@ async def draw_prims(
|
||||
return {"handles": handles, "counts": counts}
|
||||
|
||||
|
||||
async def _ensure_layers(backend, names) -> tuple[str, ...]:
|
||||
"""Create whichever target layers the drawing lacks, before the first entity.
|
||||
|
||||
MEASURED on AutoCAD 2026 by ``scripts/smoke_mech_com.py``: ``draw_part``
|
||||
put its anchor POINT on ``MECH``, a layer no layer set creates, and
|
||||
ActiveX answered ``entity.Layer = "MECH"`` with ``-0x7ffdfff7 ('Key not
|
||||
found')`` - after every view had already been drawn. The headless engine
|
||||
creates a missing layer on assignment, so the same call passed every test.
|
||||
Group A measured the same refusal for annotation layers; this is the same
|
||||
cure (``ensure_annotation_layers``) at the places every mechanical drawing
|
||||
path passes through.
|
||||
"""
|
||||
from engineering.mech.annotate import ensure_annotation_layers
|
||||
|
||||
return await ensure_annotation_layers(backend, [str(name) for name in names if name])
|
||||
|
||||
|
||||
def _refuse_islands_without_edge_paths(backend, prims) -> None:
|
||||
"""Refuse a hatch with islands on an engine that cannot attach them - before drawing.
|
||||
|
||||
@@ -515,6 +533,7 @@ async def draw_part(
|
||||
if index == 0:
|
||||
parent, parent_offset = view, offset
|
||||
|
||||
await _ensure_layers(backend, [ANCHOR_LAYER])
|
||||
anchor = await backend.entity_create_point(origin[0], origin[1], layer=ANCHOR_LAYER)
|
||||
payload = {
|
||||
"v": PAYLOAD_VERSION,
|
||||
@@ -839,6 +858,7 @@ async def dimension_part(
|
||||
skipped: list[dict] = []
|
||||
table_handle: str | None = None
|
||||
|
||||
await _ensure_layers(backend, [ROLE_LAYER["dim"]])
|
||||
for index, view, placed in planned:
|
||||
record = records[index]
|
||||
offset = record["offset"]
|
||||
@@ -963,6 +983,7 @@ async def draw_hole_pattern(
|
||||
prims.append(Line((hole.x - reach, hole.y), (hole.x + reach, hole.y), "center"))
|
||||
prims.append(Line((hole.x, hole.y - reach), (hole.x, hole.y + reach), "center"))
|
||||
drawn = await draw_prims(backend, prims)
|
||||
await _ensure_layers(backend, [layer, ROLE_LAYER["dim"] if dimension else None])
|
||||
if layer:
|
||||
# the role map owns the layers; an explicit override moves the circles
|
||||
# only, so the ISO 128-23 centre marks stay on CENTER.
|
||||
|
||||
@@ -209,8 +209,19 @@ class _Grid:
|
||||
|
||||
|
||||
def _vertices(info: EntityInfo) -> list[tuple[float, float]]:
|
||||
"""The entity's vertices, or ``[]`` when the engine could not report them.
|
||||
|
||||
MEASURED on AutoCAD 2026 by scripts/smoke_mech_com.py: the live engine's
|
||||
entity reader drops a LINE's ``start``/``end`` when AutoCAD refuses the
|
||||
point read (``RPC_E_CALL_REJECTED`` while busy) and keeps the entity, so
|
||||
a subscript here crashed ``drawing_critique(focus=None)`` on a sheet with
|
||||
no P&ID on it. A line without coordinates cannot be an edge; it is left
|
||||
out rather than guessed at.
|
||||
"""
|
||||
if info.type in LINE_TYPES:
|
||||
s, e = info.properties["start"], info.properties["end"]
|
||||
s, e = info.properties.get("start"), info.properties.get("end")
|
||||
if not s or not e:
|
||||
return []
|
||||
return [(float(s[0]), float(s[1])), (float(e[0]), float(e[1]))]
|
||||
return [(float(p[0]), float(p[1])) for p in info.properties.get("points") or []]
|
||||
|
||||
|
||||
@@ -111,6 +111,11 @@ COLUMN_SHARES: dict[str, float] = {
|
||||
PAGE_SIZE = 1000
|
||||
|
||||
|
||||
#: A block reference under either engine's type name: the DXF name headlessly,
|
||||
#: the ActiveX ObjectName (AcDbBlockReference) live.
|
||||
INSERT_TYPES = ("INSERT", "BLOCKREFERENCE")
|
||||
|
||||
|
||||
async def all_entities(backend: AutoCADBackend, *, type_filter=None, layer=None):
|
||||
"""Every entity of a type in the CURRENT space -- not the first 200 of them.
|
||||
|
||||
@@ -361,7 +366,18 @@ async def extract_records(backend: AutoCADBackend, *, layer=None, limit: int | N
|
||||
"""
|
||||
if limit is not None and int(limit) <= 0:
|
||||
raise ValueError(f"bom_extract: limit must be a positive number of records; got {limit!r}")
|
||||
inserts = await all_entities(backend, type_filter="INSERT", layer=layer)
|
||||
# MEASURED on AutoCAD 2026 by scripts/smoke_mech_com.py: the live engine
|
||||
# names a block reference by its ObjectName, BLOCKREFERENCE, and filters on
|
||||
# that name, so a filter on the DXF name INSERT found nothing and the parts
|
||||
# list came back empty on a sheet holding an ISO 4014 bolt. Both names are
|
||||
# read, the way engineering/pid/graph.py already does.
|
||||
inserts: list = []
|
||||
seen: set[str] = set()
|
||||
for type_name in INSERT_TYPES:
|
||||
for entity in await all_entities(backend, type_filter=type_name, layer=layer):
|
||||
if entity.handle not in seen:
|
||||
seen.add(entity.handle)
|
||||
inserts.append(entity)
|
||||
records: list[dict] = []
|
||||
for entity in inserts:
|
||||
payload: dict = {}
|
||||
|
||||
@@ -27,6 +27,22 @@ DIMENSION_LIKE_TEXT = re.compile(
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# A standard-part designation (ISO 4014 - M12x60, DIN 912 - M8x20) names a
|
||||
# part, it does not measure one: the "12x60" in it is the thread and the
|
||||
# nominal length the standard's designation grammar spells that way. A parts
|
||||
# list drawn as text carries one per row, so the designation is taken out
|
||||
# before DIMENSION_LIKE_TEXT looks - a hand-typed "Ø78" beside it still flags.
|
||||
STANDARD_DESIGNATION = re.compile(
|
||||
r"\b(?:ISO|DIN|EN)\s*\d+(?:-\d+)?\s*-\s*M\d+(?:\.\d+)?(?:\s*[x×]\s*\d+(?:\.\d+)?)?",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def looks_like_dimension(text: str) -> bool:
|
||||
"""Is this TEXT/MTEXT content masquerading as a dimension?"""
|
||||
return bool(DIMENSION_LIKE_TEXT.search(STANDARD_DESIGNATION.sub(" ", text or "")))
|
||||
|
||||
|
||||
# R27 — match SPUR / HELICAL as whole words (not substrings of e.g.
|
||||
# "SPURIOUS") when checking title/helix consistency.
|
||||
_SPUR_RE = re.compile(r"\bSPUR\b", re.IGNORECASE)
|
||||
@@ -92,7 +108,7 @@ class DrawingValidator:
|
||||
e
|
||||
for e in ents
|
||||
if e.type in ("TEXT", "MTEXT")
|
||||
and DIMENSION_LIKE_TEXT.search(getattr(e, "properties", {}).get("text", "") or "")
|
||||
and looks_like_dimension(getattr(e, "properties", {}).get("text", "") or "")
|
||||
]
|
||||
real_dims = [e for e in ents if e.type in REAL_DIMENSION_TYPES]
|
||||
if text_likes:
|
||||
|
||||
@@ -79,7 +79,7 @@ def derived() -> dict[str, object]:
|
||||
}
|
||||
|
||||
from benchmarks import correctness_suite
|
||||
from benchmarks.tasks_v4 import TASKS_V4
|
||||
from benchmarks.tasks_v5 import TASKS_V5
|
||||
|
||||
return {
|
||||
"collected_tests": _collected_tests(),
|
||||
@@ -94,7 +94,7 @@ def derived() -> dict[str, object]:
|
||||
"ab_fixed": ab["fixed"],
|
||||
"ab_regressed": ab["regressed"],
|
||||
"suite_checks": len(correctness_suite.CHECKS),
|
||||
"v4_tasks": len(TASKS_V4),
|
||||
"matrix_tasks": len(TASKS_V5),
|
||||
"perf_ms": perf,
|
||||
}
|
||||
|
||||
@@ -176,7 +176,7 @@ CHECKS: list[tuple[str, str, str, str]] = [
|
||||
"ab_fixed",
|
||||
),
|
||||
("CLAUDE.md", "A/B suite size", r"correctness_suite\.py` \((\d+) checks\)", "ab_total"),
|
||||
("CLAUDE.md", "v4 task count", r"the current set \((\d+) tasks", "v4_tasks"),
|
||||
("CLAUDE.md", "matrix task count", r"the current set \((\d+) tasks", "matrix_tasks"),
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,290 @@
|
||||
"""Exercise tracks B and G once against the live AutoCAD on this machine.
|
||||
|
||||
Creates NEW documents (never touches the operator's open drawings), then runs
|
||||
the whole track through the same code the tools call:
|
||||
|
||||
xref source: drawing_new -> a line -> drawing_export_dwg -> close
|
||||
apply_layer_set("mech") -> apply_titleblock (ISO 5457 A3 + ISO 7200)
|
||||
-> build_part -> draw_part -> add_view(section) -> dimension_part
|
||||
-> insert_std_part (ISO 4014 block + ACADMCP_MECH xdata)
|
||||
-> weld_symbol_prims -> extract_records -> rows_from_records
|
||||
-> draw_bom_table -> add_balloon
|
||||
-> xref_attach / xref_manage("list") / xref_manage("detach")
|
||||
-> run_critique(focus=None) -> drawing_export_dwg
|
||||
|
||||
and prints what came back as JSON. Exit codes:
|
||||
|
||||
0 the sheet drew, the critique returned nothing and the .dwg exists
|
||||
1 it drew but the critique found something or the DWG is missing
|
||||
2 no live CAD application reachable (pywin32 missing, or the ProgID
|
||||
cannot be created) -- the COM paths then remain fake-tested only
|
||||
|
||||
The backend's ``connect()`` is lazy on purpose (the server must start before
|
||||
AutoCAD is open), so reachability is read from ``system_status()`` rather than
|
||||
from ``connect()``. Every document this run opens is closed on the way out,
|
||||
also when a step fails, unless ``--keep`` is given.
|
||||
|
||||
AUTOCAD_MCP_BACKEND=com uv run --frozen python scripts/smoke_mech_com.py
|
||||
uv run --frozen python scripts/smoke_mech_com.py --keep # leave it open
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
|
||||
XDATA_APP = "ACADMCP_MECH"
|
||||
|
||||
#: Every DWG since R13 opens with "AC10xx" (AC1032 for the 2018 format).
|
||||
DWG_MAGIC = b"AC10"
|
||||
|
||||
#: ``RPC_E_CALL_REJECTED``: AutoCAD's message filter refused the call because
|
||||
#: it was busy (a document still initialising after ``Documents.Add`` is the
|
||||
#: case measured on this seat). A refused call was not executed.
|
||||
RPC_E_CALL_REJECTED = -2147418111
|
||||
|
||||
#: How long a refused call is retried before the smoke gives up on it.
|
||||
REJECTED_PATIENCE_S = 90.0
|
||||
|
||||
#: Names of the documents this run opened and has not closed yet.
|
||||
_OPENED: list[str] = []
|
||||
|
||||
|
||||
def _patient(backend) -> None:
|
||||
"""Retry any backend call AutoCAD refused with ``RPC_E_CALL_REJECTED``.
|
||||
|
||||
Measured on AutoCAD 2026: the first call after ``Documents.Add`` was
|
||||
refused, the smoke died, and its cleanup - refused as well - left the
|
||||
scratch document open. Refused calls are retried with backoff for up to
|
||||
``REJECTED_PATIENCE_S``; a document created twice by a retried
|
||||
``drawing_new`` is still closed, because ``_new_document`` tracks the
|
||||
document list, not a name it assumes.
|
||||
"""
|
||||
run = backend._run
|
||||
|
||||
async def patient(func, *args, **kwargs):
|
||||
deadline = time.monotonic() + REJECTED_PATIENCE_S
|
||||
pause = 0.5
|
||||
while True:
|
||||
try:
|
||||
return await run(func, *args, **kwargs)
|
||||
except RuntimeError as exc:
|
||||
cause = exc.__cause__
|
||||
hr = cause.args[0] if cause is not None and cause.args else None
|
||||
if hr != RPC_E_CALL_REJECTED or time.monotonic() + pause > deadline:
|
||||
raise
|
||||
await asyncio.sleep(pause)
|
||||
pause = min(pause * 2.0, 5.0)
|
||||
|
||||
backend._run = patient
|
||||
|
||||
|
||||
async def _names(backend) -> set[str]:
|
||||
return {row["name"] for row in await backend.document_list()}
|
||||
|
||||
|
||||
async def _new_document(backend) -> str:
|
||||
"""``drawing_new``, recording every document it added (normally one)."""
|
||||
before = await _names(backend)
|
||||
await backend.drawing_new()
|
||||
added = sorted((await _names(backend)) - before)
|
||||
_OPENED.extend(name for name in added if name not in _OPENED)
|
||||
return added[-1] if added else ""
|
||||
|
||||
|
||||
async def _close_leftovers(backend) -> None:
|
||||
open_names = {row["name"] for row in await backend.document_list()}
|
||||
for name in list(_OPENED):
|
||||
if name in open_names:
|
||||
try:
|
||||
await backend.document_close(name, discard=True)
|
||||
except Exception as exc: # best effort on the way out
|
||||
print(f"could not close {name}: {exc}", file=sys.stderr)
|
||||
_OPENED.remove(name)
|
||||
|
||||
|
||||
async def _xref_source(backend, workspace: Path) -> Path:
|
||||
"""A one-line DWG in its own scratch document, saved and closed.
|
||||
|
||||
An xref has to be a DWG, and it cannot be the host document itself -
|
||||
``SaveAs`` rebinds the host to the file it wrote.
|
||||
"""
|
||||
created = await _new_document(backend)
|
||||
await backend.entity_create_line(0.0, 0.0, 50.0, 0.0, layer="0")
|
||||
path = workspace / "xref_source.dwg"
|
||||
before = await _names(backend)
|
||||
await backend.drawing_export_dwg(str(path), "R2018")
|
||||
rebound = await _rebound(backend, created, before)
|
||||
await backend.document_close(rebound, discard=True)
|
||||
_OPENED.remove(rebound)
|
||||
return path
|
||||
|
||||
|
||||
async def _rebound(backend, created: str, before: set[str]) -> str:
|
||||
"""``SaveAs`` renames the active document to the file it wrote; track that name.
|
||||
|
||||
Read from the document list rather than ``system_status``: the latter
|
||||
swallows a refused call and answers without ``active_document``.
|
||||
"""
|
||||
after = await _names(backend)
|
||||
renamed = sorted(after - before)
|
||||
rebound = renamed[0] if created not in after and len(renamed) == 1 else created
|
||||
if created in _OPENED:
|
||||
_OPENED.remove(created)
|
||||
if rebound not in _OPENED:
|
||||
_OPENED.append(rebound)
|
||||
return rebound
|
||||
|
||||
|
||||
async def _sheet(backend, workspace: Path, xref_path: Path) -> dict:
|
||||
from engineering.critique import run_critique
|
||||
from engineering.layers import apply_layer_set
|
||||
from engineering.mech.annotate import weld_symbol_prims
|
||||
from engineering.mech.draw import add_view, dimension_part, draw_part, draw_prims
|
||||
from engineering.mech.part import build_part
|
||||
from engineering.mech.stdparts import insert_std_part
|
||||
from engineering.sheet.bom import (
|
||||
add_balloon,
|
||||
draw_bom_table,
|
||||
extract_records,
|
||||
rows_from_records,
|
||||
)
|
||||
from engineering.sheet.titleblock import (
|
||||
TB_HEIGHT,
|
||||
TitleBlockMetadata,
|
||||
apply_titleblock,
|
||||
titleblock_origin,
|
||||
)
|
||||
|
||||
created = await _new_document(backend)
|
||||
await apply_layer_set(backend, "mech")
|
||||
await apply_titleblock(
|
||||
backend,
|
||||
size="A3",
|
||||
metadata=TitleBlockMetadata(title="COM SMOKE", drawing_no="SMOKE-001", scale="1:1"),
|
||||
projection="first",
|
||||
frame=True,
|
||||
zones=True,
|
||||
marks=True,
|
||||
)
|
||||
|
||||
shaft = build_part(
|
||||
{
|
||||
"kind": "revolved",
|
||||
"name": "SHAFT",
|
||||
"material": "steel",
|
||||
"segments": [
|
||||
{"length": 40.0, "d_outer": 30.0},
|
||||
{"length": 20.0, "d_outer": 50.0, "d_inner": 20.0},
|
||||
],
|
||||
}
|
||||
)
|
||||
drawn = await draw_part(backend, shaft, at=(125.0, 170.0), views=("front",), dimension=False)
|
||||
section = await add_view(
|
||||
backend,
|
||||
drawn["part_id"],
|
||||
"section",
|
||||
plane={"p1": [0.0, 0.0], "p2": [60.0, 0.0], "label": "A"},
|
||||
style="full",
|
||||
)
|
||||
dims = await dimension_part(backend, drawn["part_id"], style="chain")
|
||||
bolt = await insert_std_part(backend, "ISO 4014 - M12x60", at=(250.0, 120.0), view="side")
|
||||
bolt_handle = bolt.get("handle") or (bolt.get("handles") or [None])[0]
|
||||
got = await backend.entity_get_xdata(bolt_handle, XDATA_APP)
|
||||
await draw_prims(backend, weld_symbol_prims(kind="fillet", size=5.0), at=(60.0, 100.0))
|
||||
|
||||
records = await extract_records(backend)
|
||||
rows = rows_from_records(records)
|
||||
tb_x, tb_y = titleblock_origin("A3")
|
||||
table = await draw_bom_table(backend, rows, at=(tb_x, tb_y + TB_HEIGHT))
|
||||
balloon = await add_balloon(
|
||||
backend,
|
||||
item=int(rows[0]["item"]),
|
||||
at=(250.0, 150.0),
|
||||
leader_to=(250.0, 125.0),
|
||||
targets=[str(h) for h in rows[0]["handles"]],
|
||||
)
|
||||
|
||||
attached = await backend.xref_attach(str(xref_path), (30.0, 30.0), 1.0, 0.0, "attach")
|
||||
listed = await backend.xref_manage(attached["name"], "list", None)
|
||||
detached = await backend.xref_manage(attached["name"], "detach", None)
|
||||
after = await backend.xref_manage(attached["name"], "list", None)
|
||||
|
||||
issues = await run_critique(backend, None)
|
||||
dwg_path = workspace / "mech_smoke.dwg"
|
||||
before = await _names(backend)
|
||||
exported = await backend.drawing_export_dwg(str(dwg_path), "R2018")
|
||||
await _rebound(backend, created, before)
|
||||
head = dwg_path.read_bytes()[:6] if dwg_path.exists() else b""
|
||||
return {
|
||||
"part_id": drawn["part_id"],
|
||||
"views": len(drawn.get("views") or []) + 1,
|
||||
"section_omitted": section.get("omitted"),
|
||||
"dimensions": int(dims.get("count") or len(dims.get("handles") or [])),
|
||||
"bolt_handle": bolt_handle,
|
||||
"xdata_chunks": len(got["xdata"].get(XDATA_APP, [])),
|
||||
"parts_list_rows": len(rows),
|
||||
"parts_list_representation": table.get("representation"),
|
||||
"balloon": balloon.get("handle"),
|
||||
"xref_attached": attached.get("name"),
|
||||
"xref_listed": len(listed.get("xrefs") or []),
|
||||
"xref_detached": bool(detached.get("ok")),
|
||||
"xrefs_after_detach": len(after.get("xrefs") or []),
|
||||
"dwg": {
|
||||
"path": exported.get("path", str(dwg_path)),
|
||||
"exists": dwg_path.exists(),
|
||||
"bytes": dwg_path.stat().st_size if dwg_path.exists() else 0,
|
||||
"magic": head.decode("ascii", "replace"),
|
||||
},
|
||||
"critique_issues": len(issues),
|
||||
"critique": [issue.to_dict() for issue in issues],
|
||||
}
|
||||
|
||||
|
||||
async def main(keep_document: bool = False) -> int:
|
||||
try:
|
||||
from backends.com_backend import ComBackend
|
||||
except ImportError as exc:
|
||||
print(f"pywin32 not available: {exc}")
|
||||
return 2
|
||||
backend = ComBackend()
|
||||
_patient(backend)
|
||||
try:
|
||||
await backend.connect()
|
||||
except Exception as exc:
|
||||
print(f"no live AutoCAD reachable: {exc}")
|
||||
return 2
|
||||
try:
|
||||
status = await backend.system_status()
|
||||
if not status.get("connected"):
|
||||
print(f"no live AutoCAD reachable: {status.get('error', status)}")
|
||||
return 2
|
||||
workspace = Path(tempfile.mkdtemp(prefix="mech_smoke_"))
|
||||
xref_path = await _xref_source(backend, workspace)
|
||||
report = {
|
||||
"autocad": status.get("autocad_version") or status.get("cad_progid"),
|
||||
**await _sheet(backend, workspace, xref_path),
|
||||
}
|
||||
print(json.dumps(report, indent=2))
|
||||
ok = (
|
||||
report["critique_issues"] == 0
|
||||
and report["dwg"]["exists"]
|
||||
and report["dwg"]["magic"].startswith(DWG_MAGIC.decode())
|
||||
and report["xrefs_after_detach"] == 0
|
||||
)
|
||||
return 0 if ok else 1
|
||||
finally:
|
||||
if not keep_document:
|
||||
await _close_leftovers(backend)
|
||||
await backend.disconnect()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(asyncio.run(main(keep_document="--keep" in sys.argv[1:])))
|
||||
@@ -48,7 +48,8 @@ def test_correctness_suite_has_the_settings_checks():
|
||||
"settings_pdf_mediabox_a3",
|
||||
):
|
||||
assert CHECKS[name][1] == "Settings"
|
||||
assert len(CHECKS) == 32
|
||||
# v4 left the suite at 32; tracks B + G grew it to 39 (tests/test_benchmark_v5.py).
|
||||
assert len(CHECKS) == 39
|
||||
|
||||
|
||||
@pytest.mark.asyncio # asyncio_mode is strict; the four tests above are sync
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# tests/test_benchmark_v5.py
|
||||
"""The v5 task matrix: v4 plus what tracks B and G added.
|
||||
|
||||
``mech_assembly`` is roadmap criterion 3 — a flange-coupling sheet built from
|
||||
the tools, gated on two numbers that cannot be negotiated:
|
||||
``drawing_critique(focus=None)`` returns zero issues and ``drawing_finalize``
|
||||
scores at least 90. The published competitor reports were never asked it and
|
||||
the chart shows them ``not_run``, the rule v3 set for its five.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from benchmarks.tasks_v4 import TASKS_V4
|
||||
from benchmarks.tasks_v5 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V5, TASKS_V5, task_by_id
|
||||
|
||||
|
||||
def test_v5_extends_v4_without_touching_it():
|
||||
assert TASKS_V5[: len(TASKS_V4)] == TASKS_V4
|
||||
assert [t.task_id for t in NEW_TASKS_V5] == ["mech_assembly"]
|
||||
assert len(TASKS_V5) == 18
|
||||
assert MATRICES["v4"] == TASKS_V4 and MATRICES["v5"] == TASKS_V5
|
||||
assert DEFAULT_MATRIX == "v5"
|
||||
assert task_by_id("mech_assembly").category == "mech"
|
||||
|
||||
|
||||
def test_the_adapter_implements_every_v5_task():
|
||||
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
||||
|
||||
for task in TASKS_V5:
|
||||
assert hasattr(AutoCADMCPProAdapter, f"_task_{task.task_id}"), task.task_id
|
||||
|
||||
|
||||
def test_the_suite_carries_the_mechanical_and_sheet_checks():
|
||||
"""Six from spec 14, plus mech_thread_unrepresented_is_caught: a quality
|
||||
gate that never fires is not a gate, the way dim_overlap_critique_fires
|
||||
already guards the dimension focus."""
|
||||
from benchmarks.correctness_suite import CHECKS
|
||||
|
||||
for name in (
|
||||
"mech_part_roundtrip",
|
||||
"mech_section_hatch_area",
|
||||
"mech_iso286_on_dimension",
|
||||
"mech_thread_unrepresented_is_caught",
|
||||
"std_part_iso4014_m12",
|
||||
):
|
||||
assert CHECKS[name][1] == "Mechanical"
|
||||
assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet"
|
||||
assert CHECKS["bom_balloon_link"][1] == "Sheet"
|
||||
assert len(CHECKS) == 39
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mech_assembly_passes_headlessly(tmp_path):
|
||||
"""The benchmark task itself, run once here so a regression in the part
|
||||
drawer, the sheet or the critique fails the suite and not only the matrix."""
|
||||
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
||||
|
||||
adapter = AutoCADMCPProAdapter(backend="ezdxf")
|
||||
await adapter.setup(tmp_path)
|
||||
try:
|
||||
await adapter._reset()
|
||||
passed, metrics, _artifacts = await adapter._task_mech_assembly()
|
||||
finally:
|
||||
await adapter.cleanup()
|
||||
assert passed, metrics
|
||||
assert metrics["critique_issues"] == 0, metrics["critique_focuses"]
|
||||
assert metrics["score"] >= 90.0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_mech_assembly_fails_loudly_when_the_critique_finds_anything(tmp_path, monkeypatch):
|
||||
"""The gate has to be able to fail. One parts-list row left without its
|
||||
balloon is enough: ISO 7573 gives every row an item reference, and the
|
||||
critique names the focus that fired rather than only shading the score."""
|
||||
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
||||
from engineering.sheet import bom
|
||||
|
||||
real = bom.add_balloon
|
||||
|
||||
async def only_the_first(backend, *, item, **kw):
|
||||
if int(item) == 1:
|
||||
return await real(backend, item=item, **kw)
|
||||
return {"ok": True, "skipped": True}
|
||||
|
||||
monkeypatch.setattr(bom, "add_balloon", only_the_first)
|
||||
adapter = AutoCADMCPProAdapter(backend="ezdxf")
|
||||
await adapter.setup(tmp_path)
|
||||
try:
|
||||
await adapter._reset()
|
||||
passed, metrics, _artifacts = await adapter._task_mech_assembly()
|
||||
finally:
|
||||
await adapter.cleanup()
|
||||
assert passed is False
|
||||
assert "mech_bom_balloon_mismatch" in metrics["critique_focuses"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_section_hatch_area_check_measures_the_drawing():
|
||||
"""1200 mm^2 is computed by hand from the model, and read back through
|
||||
analysis_measure_entity — a number the drawing code cannot fake."""
|
||||
from benchmarks import correctness_suite
|
||||
|
||||
assert await correctness_suite.mech_section_hatch_area() is True
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_the_iso5457_frame_check_passes():
|
||||
from benchmarks import correctness_suite
|
||||
|
||||
assert await correctness_suite.sheet_frame_iso5457_a3() is True
|
||||
@@ -546,6 +546,27 @@ def _gate_valve_block():
|
||||
return _Block([body_left, body_right, tag])
|
||||
|
||||
|
||||
def test_a_dimension_reports_where_its_text_stands():
|
||||
"""The dim_overlap critique reads `text_position`; without it a diameter
|
||||
dimension's reference point was its bounding-box centre - the circle's
|
||||
centre - and every outside diameter collided with its own bore on the live
|
||||
engine (measured by scripts/smoke_mech_com.py, AutoCAD 2026)."""
|
||||
from backends.com_backend import _entity_info
|
||||
|
||||
outside = _Entity(
|
||||
"AcDbDiametricDimension",
|
||||
GetBoundingBox=lambda: ((178.0, 170.0, 0.0), (184.0, 233.0, 0.0)),
|
||||
TextPosition=(181.0, 230.0, 0.0),
|
||||
Handle="1C1",
|
||||
Layer="DIM",
|
||||
color=256,
|
||||
Linetype="ByLayer",
|
||||
Visible=True,
|
||||
)
|
||||
props = _entity_info(outside).properties
|
||||
assert props["text_position"] == [181.0, 230.0]
|
||||
|
||||
|
||||
def test_com_geometry_bbox_is_the_drawn_body_without_the_attribute():
|
||||
from backends.com_backend import _entity_info
|
||||
|
||||
|
||||
@@ -57,6 +57,17 @@ async def test_a_diameter_dimension_measures_the_diameter(backend):
|
||||
assert dim.get_measurement() == pytest.approx(40.0)
|
||||
|
||||
|
||||
async def test_the_text_stands_beyond_the_first_chord_point_as_activex_puts_it(backend):
|
||||
"""AddDimDiametric(ChordPoint, FarChordPoint, LeaderLength), measured on
|
||||
AutoCAD 2026: a 50 mm diameter from (x, 170) to (x, 220) with leader 10
|
||||
had its text at (x, 160) - beyond the FIRST point. Both engines follow it."""
|
||||
info = await backend.dimension_diameter(0, 25, 0, -25, leader_length=10.0)
|
||||
dim = backend._doc.entitydb.get(info.handle)
|
||||
text = dim.dxf.text_midpoint
|
||||
assert text.y == pytest.approx(35.0)
|
||||
assert text.x == pytest.approx(0.0)
|
||||
|
||||
|
||||
async def test_the_leader_length_moves_the_text_not_the_measurement(backend):
|
||||
"""The regression in one assertion: leader_length must not be measurable."""
|
||||
near = await backend.dimension_diameter(-20, 0, 20, 0, leader_length=5.0)
|
||||
|
||||
@@ -101,6 +101,21 @@ async def test_validator_catches_fake_dimension_text(backend, tmp_path):
|
||||
assert "fake_dimension_text" in _codes(result)
|
||||
|
||||
|
||||
async def test_a_standard_part_designation_is_not_a_fake_dimension():
|
||||
"""A parts-list row names a part: "ISO 4014 - M12x60" is the ISO 4014
|
||||
designation grammar, not a measurement. Found by the v5 mech_assembly
|
||||
benchmark, where every text parts list cost the sheet a warning. The
|
||||
exemption is the designation only - a measurement beside it still flags."""
|
||||
from engineering.validator import looks_like_dimension
|
||||
|
||||
assert not looks_like_dimension("ISO 4014 - M12x60")
|
||||
assert not looks_like_dimension("DIN 912 - M8x20")
|
||||
assert not looks_like_dimension("ISO 4032 - M12")
|
||||
assert looks_like_dimension("ISO 4014 - M12x60 Ø78")
|
||||
assert looks_like_dimension("40x60")
|
||||
assert looks_like_dimension("Ø78")
|
||||
|
||||
|
||||
async def test_validator_catches_hidden_orphans(backend, tmp_path):
|
||||
await ensure_standard_linetypes(backend)
|
||||
await ensure_engineering_layers(backend)
|
||||
|
||||
@@ -492,3 +492,65 @@ async def test_a_transaction_that_cannot_open_is_a_value_error(backend, monkeypa
|
||||
monkeypatch.setattr(backend, "transaction_begin", busy)
|
||||
with pytest.raises(ValueError, match="could not open a transaction"):
|
||||
await draw_from_spec(backend, {"parts": [{"part": PLATE, "at": [0.0, 0.0]}]})
|
||||
|
||||
|
||||
async def test_a_single_step_disk_dimensions_without_a_text_collision(backend):
|
||||
"""The largest diameter's text used to land on its own segment's chain text.
|
||||
|
||||
The front-view diameters start at the upper chord end, and ActiveX puts a
|
||||
diameter's text beyond that FIRST point - above the view, clear of the
|
||||
chain rows (measured on AutoCAD 2026). The headless engine put it beyond
|
||||
the second point instead, so ⌀90 landed exactly on the "15" of a 15 mm
|
||||
disk and ``dim_overlap`` fired on every single-step part the default
|
||||
``mech_part_draw`` produced headlessly. Found by the v5 ``mech_assembly``
|
||||
benchmark and the live smoke together.
|
||||
"""
|
||||
from engineering.critique import run_critique
|
||||
|
||||
disk = build_part(
|
||||
{
|
||||
"kind": "revolved",
|
||||
"name": "FLANGE",
|
||||
"material": "cast_iron",
|
||||
"segments": [{"length": 15.0, "d_outer": 90.0, "d_inner": 40.0}],
|
||||
}
|
||||
)
|
||||
await draw_part(backend, disk, at=(0.0, 0.0), views=("front",), dimension=True)
|
||||
assert await run_critique(backend, ["dim_overlap"]) == []
|
||||
|
||||
|
||||
async def test_every_layer_exists_before_an_entity_is_put_on_it(backend, monkeypatch):
|
||||
"""ActiveX will not create a layer on assignment - the headless engine will.
|
||||
|
||||
Measured on AutoCAD 2026 by scripts/smoke_mech_com.py: ``entity.Layer =
|
||||
"MECH"`` raised ``-0x7ffdfff7 ('Key not found')`` after every view of the
|
||||
part had been drawn, so ``mech_part_draw`` could not finish on a live
|
||||
seat. This backend is made as strict as ActiveX, and the whole part -
|
||||
views, anchor, dimensions - has to draw on it.
|
||||
"""
|
||||
creators = (
|
||||
"entity_create_point",
|
||||
"entity_create_line",
|
||||
"entity_create_circle",
|
||||
"entity_create_arc",
|
||||
"entity_create_polyline",
|
||||
"entity_create_text",
|
||||
"entity_create_hatch",
|
||||
)
|
||||
for name in creators:
|
||||
real = getattr(backend, name)
|
||||
|
||||
async def strict(*args, _real=real, **kwargs):
|
||||
layer = kwargs.get("layer")
|
||||
known = {row.name.lower() for row in await backend.layer_list()}
|
||||
if layer and layer.lower() not in known:
|
||||
raise RuntimeError(f"Key not found: layer {layer!r}")
|
||||
return await _real(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(backend, name, strict)
|
||||
|
||||
drawn = await draw_part(
|
||||
backend, build_part(SHAFT), at=(0.0, 0.0), views=("front", "side"), dimension=True
|
||||
)
|
||||
assert drawn["part_id"]
|
||||
assert ANCHOR_LAYER.lower() in {row.name.lower() for row in await backend.layer_list()}
|
||||
|
||||
@@ -896,3 +896,14 @@ def test_com_block_reference_reports_mirrored_from_its_normal():
|
||||
assert _entity_info(blockref(Normal=(0.0, 0.0, -1.0))).properties["mirrored"] is True
|
||||
assert _entity_info(blockref(Normal=(0.0, 0.0, 1.0))).properties["mirrored"] is False
|
||||
assert _entity_info(blockref()).properties["mirrored"] is False
|
||||
|
||||
|
||||
def test_a_line_the_engine_could_not_read_is_not_an_edge():
|
||||
"""The live reader keeps a LINE whose point read AutoCAD refused, without
|
||||
``start``/``end`` (measured by scripts/smoke_mech_com.py on AutoCAD 2026);
|
||||
the graph must leave it out instead of raising KeyError mid-critique."""
|
||||
from backends.base import EntityInfo
|
||||
from engineering.pid.graph import _vertices
|
||||
|
||||
blind = EntityInfo("1F", "LINE", "GEOMETRY", 256, "ByLayer", True, properties={})
|
||||
assert _vertices(blind) == []
|
||||
|
||||
@@ -224,6 +224,30 @@ async def test_extract_reads_the_xdata_payload_and_never_touches_the_drawing(bac
|
||||
assert all(r["source"] == "xdata" for r in records)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extract_reads_a_block_reference_under_the_live_type_name(backend, monkeypatch):
|
||||
"""The live engine calls an INSERT by its ObjectName, BLOCKREFERENCE.
|
||||
|
||||
Measured on AutoCAD 2026 by scripts/smoke_mech_com.py: a filter on
|
||||
"INSERT" found nothing, and the parts list of a sheet holding an ISO 4014
|
||||
bolt came back empty. The listing is made to answer the way ActiveX does.
|
||||
"""
|
||||
handle = await _place_bolt(backend, "a", 0.0)
|
||||
real = backend.entity_list
|
||||
|
||||
async def live_names(type_filter=None, **kwargs):
|
||||
if type_filter and type_filter.upper() == "INSERT":
|
||||
return []
|
||||
wanted = (
|
||||
"INSERT" if type_filter and type_filter.upper() == "BLOCKREFERENCE" else type_filter
|
||||
)
|
||||
return await real(type_filter=wanted, **kwargs)
|
||||
|
||||
monkeypatch.setattr(backend, "entity_list", live_names)
|
||||
records = await extract_records(backend)
|
||||
assert [r["handle"] for r in records] == [handle]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_extract_falls_back_to_the_block_attributes_when_there_is_no_payload(backend):
|
||||
await backend.block_define(
|
||||
|
||||
@@ -161,13 +161,25 @@ def test_the_cache_arithmetic_follows_the_published_multipliers():
|
||||
# -- lane 1: idle cost ------------------------------------------------------
|
||||
|
||||
|
||||
def test_the_idle_lane_covers_the_three_advertised_surfaces(report):
|
||||
def test_the_idle_lane_covers_the_four_advertised_surfaces(report):
|
||||
"""Default, lean, search - and TOOL_PACKS=core,mech, the mechanical client
|
||||
that never opens a P&ID or a live seat: the default surface minus exactly
|
||||
the pid and settings packs."""
|
||||
variants = {row["variant"]: row for row in report["idle"]}
|
||||
assert set(variants) == {"default", "lean", "search"}
|
||||
assert set(variants) == {"default", "lean", "packs_core_mech", "search"}
|
||||
assert variants["default"]["advertised_tools"] > 100
|
||||
assert variants["lean"]["advertised_tools"] == len(server.LEAN_TOOL_NAMES)
|
||||
assert variants["search"]["advertised_tools"] == 2
|
||||
assert variants["default"]["chars"] > variants["lean"]["chars"] > variants["search"]["chars"]
|
||||
dropped = len(server.PACK_TOOL_NAMES["pid"]) + len(server.PACK_TOOL_NAMES["settings"])
|
||||
assert variants["packs_core_mech"]["advertised_tools"] == (
|
||||
variants["default"]["advertised_tools"] - dropped
|
||||
)
|
||||
assert (
|
||||
variants["default"]["chars"]
|
||||
> variants["packs_core_mech"]["chars"]
|
||||
> variants["lean"]["chars"]
|
||||
> variants["search"]["chars"]
|
||||
)
|
||||
|
||||
|
||||
def test_the_idle_lane_separates_the_wire_payload_from_what_the_model_pays(report):
|
||||
|
||||
Reference in New Issue
Block a user