bench: matrix v6 adds arch_roundtrip; four architectural correctness checks; the live arch smoke

The evidence for track F, in the shape tracks A, E and B + G used.

- benchmarks/tasks_v6.py: 19 tasks, DEFAULT_MATRIX = "v6"; v5 to v2 stay
  addressable. arch_roundtrip draws the example plan with
  arch_plan_from_spec and reads it back without the spec: rooms_detect must
  find each labelled room within 0.1 % of its label and of the net floor
  computed by hand (4825 x 5750 and 3825 x 5750), the schedules must list
  D1, D2, W1, W2, and the plan is gated on a zero-issue critique and a
  finalize score of at least 90. A test removes one room label and the gate
  fails naming arch_room_unlabelled.
- Four correctness checks, every number worked out by hand in the docstring:
  L / T / X junction corners and wall areas, a 4 x 5 m room labelled 20.00 m2
  rather than its 21.84 m2 axis area, a door interrupting both faces, and a
  plan of plain lines read as 20 and 15 m2 at confidence 0.6. The published
  A/B report is re-run over 43 checks: 26 / 43 -> 43 / 43, zero regressed.
- scripts/smoke_arch_com.py runs the whole plan once on a live seat and
  closes every document it opens; the CHANGELOG records what it returned.
- The token suite measures a TOOL_PACKS=core,arch row; check_doc_numbers
  derives the task count from the v6 matrix; CLAUDE.md, both READMEs and
  the spec status line say what shipped.
- fix(com): the first live smoke exited 1 - the room reader found 0 wall
  segments. Measured on AutoCAD 2026 (2026-09-23): with a makepy cache,
  ModelSpace.Item(i) returns the gen_py IAcadEntity wrapper
  (type(item).__name__ == "IAcadEntity") and item.StartPoint on an
  AcDbLine raises AttributeError, so every entity_list row lost its
  type-specific properties (28 wall LINEs, 0 with start/end). _entity_info
  now re-dispatches exactly that wrapper through _com_unnarrow; after the
  fix 38 of 53 rows carry start/end and the smoke exits 0 with both rooms
  at 27 743 750 / 21 993 750 mm2 and 0 critique issues. Re-dispatch
  measured at ~1.7 ms per entity (dynamic vs gencache alike).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-23 23:35:46 +03:00
co-authored by Claude Opus 5.5
parent 6e004cc658
commit 0abbe55041
20 changed files with 952 additions and 53 deletions
+10 -4
View File
@@ -96,10 +96,11 @@ Tools are organized into sections (counts are indicative — `system_about` is a
22. Standard parts (3 tools, pack `mech`): `std_part_list`, `std_part_insert`, `std_feature_draw`. ISO 4014 / 4017 / 4032 / 7089 / 4762 fasteners, ISO 15 bearings with ISO 8826-1 representation, DIN 471/472 retaining-ring grooves, DIN 509 undercuts, DIN 332 centre holes and ISO 3601-2 O-ring grooves — authored tables, inserted as real blocks with attributes and `ACADMCP_MECH` XDATA, which is what makes the parts list a *read* of the drawing. A size outside a table's transcribed coverage is refused by name, never interpolated.
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
25. Architecture (12 tools, pack `arch`): `arch_wall`, `arch_opening`, `arch_stair`, `arch_room`, `arch_rooms_detect`, `arch_schedule`, `arch_grid`, `arch_symbol`, `arch_catalogue_list`, `arch_catalogue_insert`, `arch_dimension_chains`, `arch_plan_from_spec`. One plan model — a wall network (axis polyline, thickness, justification, material) with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, cut faces hatched by material (`engineering/arch/materials.py`: ISO 128-50 general hatching for brick, the JIS A 0150 RC / stone / wood symbols, AutoCAD's AR-CONC / AR-SAND, AAC as a declared convention), and anything it cannot place listed in `omitted: [{element, reason}]`, never dropped. Rooms are *computed*: a label's area is the planar face its point lies in, and `arch_rooms_detect` reads rooms from any plan (confidence 1.0 for our walls, 0.6 for plain lines) without modifying it. Schedules are a read of the `ACADMCP_ARCH` records, drawn as real TABLEs. Labels and schedules speak English, or Turkish with `lang="tr"` (decimal comma). The COM paths were executed live once by `scripts/smoke_arch_com.py` (AutoCAD 2026).
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 60 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*` and the three `mech_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 64 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
@@ -111,7 +112,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
**Benchmark matrix:** `benchmarks/tasks_v5.py` is the current set (18 tasks; `--matrix v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth` or the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (39 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
**Benchmark matrix:** `benchmarks/tasks_v6.py` is the current set (19 tasks; `--matrix v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (43 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
### Adding a New Tool
@@ -138,7 +139,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
- **`TOOL_PACKS=all|core,pid,settings,mech`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **`TOOL_PACKS=all|core,pid,settings,mech,arch`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
@@ -147,6 +148,11 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Six `mech_*` critique focuses** (`mech_missing_centreline`, `mech_unhatched_section`, `mech_view_misaligned`, `mech_duplicate_dimension`, `mech_thread_unrepresented`, `mech_bom_balloon_mismatch`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_MECH` payload. `mech_view_misaligned` checks the projection axis and not the quadrant, and `mech_duplicate_dimension` catches two dimensions in the same place rather than a semantic restatement — both narrowings are in the module docstring, because the payload carries no projection angle and no engine exposes what a dimension measured.
- **Live ActiveX will not create a layer on assignment** (`entity.Layer = "X"` on a missing layer raises `Key not found`; ezdxf creates it silently). Every mechanical/annotation draw path ensures its target layers before the first entity — a new path must do the same, or it passes every headless test and dies live.
- **A diameter's text stands beyond the FIRST chord point** on both engines (the ActiveX `AddDimDiametric` rule, measured); `dimension_diameter(x1, y1, x2, y2)` annotates on the `(x1, y1)` side.
- **The plan model, not lines**: a wall is `arch_wall` (an axis, a thickness, a justification, a material) and a door or window is hosted in it by `arch_opening` — never hand-drawn lines. The engine resolves the junctions and cuts the openings; what it cannot place is in `omitted`.
- **Areas are measured, never typed**: a room's area is the planar face its label point lies in, written into the label and the `room` record by `arch_room`; no tool accepts an area from its caller, and `arch_plan_from_spec` refuses a typed `area` by path.
- **Plan text follows the plot scale**: `scale` is a denominator (50 = 1:50) and a paper height `h` is drawn at `h × scale` drawing units — no annotative objects. `lang="tr"` switches words and the decimal separator; any other language is refused with the list.
- **Three `arch_*` critique focuses** (`arch_room_unlabelled`, `arch_wall_gap` — two wall ends within 50 mm that do not join, `arch_opening_clash`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_ARCH` record.
- **Nominal, not standard**: the furniture and sanitary blocks carry nominal catalogue dimensions and say so; the stair tool reports the Blondel relation (2R + G within 600–650 mm) as a design rule, never as a building code; the `arch` layer colours are a repository convention (ISO 13567 fixes names, not colours).
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
### Premium Drawing Rules