bench: matrix v7 adds takeoff_roundtrip and understand_foreign; five understanding checks; the live smoke and the field-test script

The evidence for track H, in the shape tracks A, E, B + G and F used.

- benchmarks/tasks_v7.py: 21 tasks, DEFAULT_MATRIX = "v7"; v6 to v2 stay
  addressable. Both new tasks read the synthetic plant pair and nothing
  else. takeoff_roundtrip: the P&ID called schematic, every routable run's
  layout length equal to the rectilinear MST of its tags, sizes and services
  as drawn (a return drawn on the supply layer included), the cable metres
  rounded up after the allowance, an unstated power left empty.
  understand_foreign: inches over millimetres flagged, the stray entity found
  by handle, two plan copies separated, five layers classified. A test breaks
  each input and watches its gate fail by name.
- Five correctness checks, every number worked out by hand in its docstring;
  the published A/B report is re-run over 48 checks: 26 / 48 -> 48 / 48,
  zero regressed.
- scripts/smoke_understand_com.py reads the pair through the live engine -
  snapshots, the DWG route, both takeoffs and a diff - and closes every
  document it opens; the CHANGELOG records what it returned.
  scripts/field_test_takeoff.py runs the takeoffs on a user's own files,
  refuses to write inside the repository and prints no quantity by default.
- The README token table is remeasured; check_doc_numbers derives the task
  count from the v7 matrix; CLAUDE.md, both READMEs, the CHANGELOG and the
  spec status line say what shipped.

One reader defect found on the way: describe.tag_occurrences read a wiring
callout ("Wiring to CP-2") as a place its panel is written, so the scale
check called CP-2 ambiguous and read "Wiring to CP1" as a phantom tag CP1 -
10 tags and 45 pairs (13.3 % within 10 %) where the generator placed 11 and
55 (21.8 %). A callout is no longer a tag occurrence, the rule
network.tag_occurrences already kept; two tests failed before the fix.

The smoke works only in documents it opened from its own temporary folder,
known by full path; it attaches with GetActiveObject only (the backend's
Dispatch fallback would launch AutoCAD), checks in the same COM-thread call
that the active document is its own before each snapshot, the DWG save and
the move, closes its documents with Close(False) and re-activates the
document that was active when it started. No new ActiveX member: Documents,
Item, FullName, ActiveDocument, Activate and Close are the ones the backend
already calls. Measured on AutoCAD 2026 (25.1s): exit 0, model-space types
equal file / live / DWG, export 0.104 s, no document of the run left open.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-24 15:10:37 +03:00
co-authored by Claude Opus 5.5
parent 835bc4b65e
commit ba41428c88
21 changed files with 1418 additions and 39 deletions
+14 -4
View File
@@ -33,6 +33,9 @@ pip install -e ".[com]"
# With PDF/screenshot support via matplotlib
pip install -e ".[pdf]"
# XLSX workbooks (takeoffs, data_extract) via openpyxl
pip install -e ".[office]"
# Everything
pip install -e ".[full]"
```
@@ -97,10 +100,12 @@ Tools are organized into sections (counts are indicative — `system_about` is a
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
25. Architecture (12 tools, pack `arch`): `arch_wall`, `arch_opening`, `arch_stair`, `arch_room`, `arch_rooms_detect`, `arch_schedule`, `arch_grid`, `arch_symbol`, `arch_catalogue_list`, `arch_catalogue_insert`, `arch_dimension_chains`, `arch_plan_from_spec`. One plan model — a wall network (axis polyline, thickness, justification, material) with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, cut faces hatched by material (`engineering/arch/materials.py`: ISO 128-50 general hatching for brick, the JIS A 0150 RC / stone / wood symbols, AutoCAD's AR-CONC / AR-SAND, AAC as a declared convention), and anything it cannot place listed in `omitted: [{element, reason}]`, never dropped. Rooms are *computed*: a label's area is the planar face its point lies in, and `arch_rooms_detect` reads rooms from any plan (confidence 1.0 for our walls, 0.6 for plain lines) without modifying it. Schedules are a read of the `ACADMCP_ARCH` records, drawn as real TABLEs. Labels and schedules speak English, or Turkish with `lang="tr"` (decimal comma). The COM paths were executed live once by `scripts/smoke_arch_com.py` (AutoCAD 2026).
26. Understanding & QA (4 tools, pack `core`): `drawing_understand` (one read-only report of a drawing somebody else made — declared vs inferred units with the evidence, declared vs robust extents and the outlier handles behind them, clusters (sheets, plan copies), every layer's likely discipline and service from an EN / TR / RU / NL / DE vocabulary with a confidence and the deciding keyword, equipment tags, rooms, text languages, blocks, layouts, title-block attributes), `drawing_scale_check` (a drawing against a reference: tags unique within a cluster are matched — a tag seen twice is `ambiguous`, never picked — pairs at least 2 m apart, the median ratio, IQR and the share within ±10 %: `to_scale` at 80 % or more, with its factor; `schematic` at 50 % or less; `partly` between), `drawing_diff` (two revisions matched by signature, then handle, then type + layer + nearest position; `markup=True` draws revision clouds on the newer one — its only write) and `drawing_topology_check` (dangling ends, near misses, interior crossings; the three `topo_*` critique focuses run the same engine under `focus=None` on layers no semantic focus covers). Every one reads one snapshot per drawing (`engineering/understand/snapshot.py`: `ezdxf` directly, or on a live seat `Document.Export(<tmp>/snap, "DXF", ...)`, which never renames the document and carries model space, paper space and block definitions — measured).
27. Plant takeoffs (2 tools, pack `plant`): `pipe_takeoff` (topology from the P&ID — segments merged at their ends, split at T-junctions, continued through inline fittings, a double-drawn segment counted once; service from the layer overridden by supply / return words; sizes kept as drawn — `Ø51`, `SMS51`, `DN20` are never equated — direct, by continuity or `unassigned`; lengths from the layout: the rectilinear MST of the tags a run connects, labelled an estimate, the P&ID length kept beside it as a comparison; statuses A / B / C) and `cable_takeoff` (one row per load: power from the P&ID's electrical texts, never invented; the panel from the layout's `Wiring to CPn` callouts; the Manhattan length plus the allowance rounded up to the metre; a section only from a caller's `section_rules`). Both write a workbook (Metraj / Özet / Kontrol / Metodoloji, `lang` tr / en / ru, XLSX with the `office` extra, CSV always) and never the drawing. `scripts/field_test_takeoff.py` runs both on a user's own pair and writes next to it, never into this repository. The COM paths were executed live once by `scripts/smoke_understand_com.py` (AutoCAD 2026).
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 64 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 65 curated drafting tools (55 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members, and keeps `drawing_understand`; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch,plant` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools, `plant` the two SECTION 27 takeoffs (track D's Plant 3D readers join it); SECTION 18/19/24/26 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
@@ -112,7 +117,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
**Benchmark matrix:** `benchmarks/tasks_v6.py` is the current set (19 tasks; `--matrix v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (43 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
**Benchmark matrix:** `benchmarks/tasks_v7.py` is the current set (21 tasks; `--matrix v6` / `v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) or the v7 `takeoff_roundtrip` / `understand_foreign` (the synthetic plant pair of `tests/fixtures/plant_pair.py` read without its truth: the P&ID called schematic, every routable run's layout length equal to the rectilinear MST of its tags, the cable metres rounded up after the allowance; units, the outlier, both plan copies and the service layers found by one `describe`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (48 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
### Adding a New Tool
@@ -139,7 +144,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
- **`TOOL_PACKS=all|core,pid,settings,mech,arch`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **`TOOL_PACKS=all|core,pid,settings,mech,arch,plant`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
@@ -153,6 +158,11 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Plan text follows the plot scale**: `scale` is a denominator (50 = 1:50) and a paper height `h` is drawn at `h × scale` drawing units — no annotative objects. `lang="tr"` switches words and the decimal separator; any other language is refused with the list.
- **Three `arch_*` critique focuses** (`arch_room_unlabelled`, `arch_wall_gap` — two wall ends within 50 mm that do not join, `arch_opening_clash`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_ARCH` record.
- **Nominal, not standard**: the furniture and sanitary blocks carry nominal catalogue dimensions and say so; the stair tool reports the Blondel relation (2R + G within 600–650 mm) as a design rule, never as a building code; the `arch` layer colours are a repository convention (ISO 13567 fixes names, not colours).
- **Read-only by default (track H)**: `drawing_understand`, `drawing_scale_check`, `drawing_topology_check` and both takeoffs never modify the drawing; `drawing_diff` writes only with `markup=True`, and a takeoff writes a workbook file, never the drawing.
- **Lengths come from the inferred unit**: a reader measures in the unit the geometry implies (text heights, wall and door sizes, overall size) and reports metres; a disagreement with `INSUNITS` is a warning with the numbers — a layout declaring inches over millimetre geometry is measured in millimetres, never ×25.4.
- **Scale check before a P&ID length**: a P&ID is a length source only when `drawing_scale_check` calls it `to_scale`; otherwise a run's length is the orthogonal estimate on the layout and the drawn length stays a comparison column (`length_source="pid"` on a schematic P&ID is refused unless `force=True`, quoting the statistics).
- **Nothing is invented (track H)**: a power the P&ID does not state stays empty and becomes an open item, an unreadable size stays `unassigned`, and no cable-section table is authored — a section comes only from the caller's `section_rules`.
- **Client drawings never enter the repository**: every track-H fixture is the synthetic plant pair (`tests/fixtures/plant_pair.py`), whose truth is computed from the positions it placed; `scripts/field_test_takeoff.py` refuses an output folder inside the repository and prints no quantity unless `--show` is given.
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
### Premium Drawing Rules