bench: matrix v7 adds takeoff_roundtrip and understand_foreign; five understanding checks; the live smoke and the field-test script

The evidence for track H, in the shape tracks A, E, B + G and F used.

- benchmarks/tasks_v7.py: 21 tasks, DEFAULT_MATRIX = "v7"; v6 to v2 stay
  addressable. Both new tasks read the synthetic plant pair and nothing
  else. takeoff_roundtrip: the P&ID called schematic, every routable run's
  layout length equal to the rectilinear MST of its tags, sizes and services
  as drawn (a return drawn on the supply layer included), the cable metres
  rounded up after the allowance, an unstated power left empty.
  understand_foreign: inches over millimetres flagged, the stray entity found
  by handle, two plan copies separated, five layers classified. A test breaks
  each input and watches its gate fail by name.
- Five correctness checks, every number worked out by hand in its docstring;
  the published A/B report is re-run over 48 checks: 26 / 48 -> 48 / 48,
  zero regressed.
- scripts/smoke_understand_com.py reads the pair through the live engine -
  snapshots, the DWG route, both takeoffs and a diff - and closes every
  document it opens; the CHANGELOG records what it returned.
  scripts/field_test_takeoff.py runs the takeoffs on a user's own files,
  refuses to write inside the repository and prints no quantity by default.
- The README token table is remeasured; check_doc_numbers derives the task
  count from the v7 matrix; CLAUDE.md, both READMEs, the CHANGELOG and the
  spec status line say what shipped.

One reader defect found on the way: describe.tag_occurrences read a wiring
callout ("Wiring to CP-2") as a place its panel is written, so the scale
check called CP-2 ambiguous and read "Wiring to CP1" as a phantom tag CP1 -
10 tags and 45 pairs (13.3 % within 10 %) where the generator placed 11 and
55 (21.8 %). A callout is no longer a tag occurrence, the rule
network.tag_occurrences already kept; two tests failed before the fix.

The smoke works only in documents it opened from its own temporary folder,
known by full path; it attaches with GetActiveObject only (the backend's
Dispatch fallback would launch AutoCAD), checks in the same COM-thread call
that the active document is its own before each snapshot, the DWG save and
the move, closes its documents with Close(False) and re-activates the
document that was active when it started. No new ActiveX member: Documents,
Item, FullName, ActiveDocument, Activate and Close are the ones the backend
already calls. Measured on AutoCAD 2026 (25.1s): exit 0, model-space types
equal file / live / DWG, export 0.104 s, no document of the run left open.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-24 15:10:37 +03:00
co-authored by Claude Opus 5.5
parent 835bc4b65e
commit ba41428c88
21 changed files with 1418 additions and 39 deletions
+149
View File
@@ -39,6 +39,16 @@ new `arch` pack, three `arch_*` critique focuses, and English or Turkish
labels. No new contract member. Spec: labels. No new contract member. Spec:
`docs/superpowers/specs/2026-09-23-v1.6-arch-design.md`. `docs/superpowers/specs/2026-09-23-v1.6-arch-design.md`.
Track H of 1.6: understanding drawings somebody else made. Six tools in two
new sections — understanding and QA (26: a one-call report, a scale check
between a P&ID and its layout, a revision diff, topology checks) and plant
takeoffs (27: pipe and cable quantities whose topology comes from the P&ID
and whose lengths come from the layout) — one DXF snapshot per drawing (a
live document through `Document.Export`, never renamed), a vocabulary in five
languages, three `topo_*` critique focuses, a new `plant` pack and the
`office` extra. Read-only by default; nothing invented. No new contract
member. Spec: `docs/superpowers/specs/2026-09-24-v1.6-understand-design.md`.
### Added ### Added
- **Styles — 10 tools, pack `core`.** `dimstyle_list/create/modify/set_current`, - **Styles — 10 tools, pack `core`.** `dimstyle_list/create/modify/set_current`,
@@ -139,6 +149,33 @@ labels. No new contract member. Spec:
2026 (`25.1s (LMS Tech)`): the example plan drawn in one transaction, both 2026 (`25.1s (LMS Tech)`): the example plan drawn in one transaction, both
rooms read back within 0.1 % of their labels and of the hand-computed net rooms read back within 0.1 % of their labels and of the hand-computed net
floor, and `drawing_critique(focus=None)` returned 0 issues (record below). floor, and `drawing_critique(focus=None)` returned 0 issues (record below).
- **Understanding foreign drawings — 4 tools, pack `core`; plant takeoffs — 2
tools, pack `plant`.** `drawing_understand`, `drawing_scale_check`,
`drawing_diff`, `drawing_topology_check`, `pipe_takeoff`, `cable_takeoff`.
Every one reads a snapshot of frozen WCS records
(`engineering/understand/snapshot.py`); labels are read by one parser
(`%%c`, `\U+2205`, the U+2300 sign, `ø` and MTEXT format runs decoded,
Cyrillic look-alike tags folded to Latin) and layer names and texts by one
EN / TR / RU / NL / DE vocabulary. The takeoff workbook is written as CSV
always and as XLSX with the new `office` extra (`openpyxl`, also in `full`).
Lean gains `drawing_understand` and reaches 65.
- **Benchmark evidence for track H.** `benchmarks/tasks_v7.py` (21 tasks, the
default matrix; v6 to v2 stay addressable) adds `takeoff_roundtrip` and
`understand_foreign`, both on the synthetic plant pair of
`tests/fixtures/plant_pair.py`, whose truth is computed from the positions
it placed; each gate is shown to fail on a broken input. The correctness
suite grows to 48 with `scale_check_detects_schematic`,
`pipe_takeoff_rmst_exact`, `cable_takeoff_roundup`,
`diff_detects_known_edits` and `topology_known_defects`. A/B against
`v1.5.1`: **26 / 48 → 48 / 48, twenty-two `miss → pass`, zero regressed**.
- **`scripts/field_test_takeoff.py`** runs both takeoffs on a user's own P&ID
and layout and writes the report next to them; it refuses an output folder
inside the repository and prints no quantity unless `--show` is given.
- **Live COM smoke.** `scripts/smoke_understand_com.py` executed once on
AutoCAD 2026 (`25.1s (LMS Tech)`): the live snapshots matched
their files type for type in model space, the DWG route read the same
P&ID, both takeoffs matched the truth and the diff reported exactly the
one entity moved (record below).
- **P&ID — 9 tools, pack `pid`.** - **P&ID — 9 tools, pack `pid`.**
- `pid_symbol_list` / `pid_symbol_insert`: a 43-symbol ISO 10628-2 / - `pid_symbol_list` / `pid_symbol_insert`: a 43-symbol ISO 10628-2 /
@@ -318,8 +355,40 @@ labels. No new contract member. Spec:
`KeyError: 'start'` recorded above as a busy-refused point read (not `KeyError: 'start'` recorded above as a busy-refused point read (not
re-measured). re-measured).
- **Track H, found after the groups landed.** Six defects in the readers,
each with a test that failed before its fix:
- A truncated DXF (an interrupted save, a partial copy) made ezdxf raise
`StopIteration`, which `asyncio.to_thread` cannot hand back to an
awaiting caller, so every track H tool reading one would have hung. The
snapshot now refuses it as "not a readable DXF file".
- `take_snapshot` of a DWG the operator already had open opened it again
read-only and closed "it" unsaved - possibly their copy, with their
unsaved work. An open document is now looked up first, exported as it
stands and left open; only a document the snapshot opened is closed.
- `drawing_diff`: a LINE drawn back to front and moved was reported as
reshaped; an INSERT carrying one attribute tag twice could lose an edit
of the first value (a repeated tag is now compared as the list of its
values); the cluster list is capped like the other lists
(`truncated["clusters"]`, the markup still clouds every cluster); and a
whole-drawing move keeps only the 32 nearest candidates per entity.
- A line number was read as an equipment tag: `100-P-001` yielded the pump
`P-001`. A token that is the tail of a word beginning with a digit is no
longer a tag.
- The pipe network merged ends and dropped stubs within `tol = 1.0`
drawing units - a metre on a metre P&ID. The tolerance now defaults to
1 mm in the P&ID's inferred unit, and a dropped stub is counted and said.
- A wiring callout was counted as a place its panel is written:
`Wiring to CP-2` made `CP-2` ambiguous in `drawing_scale_check` and
`drawing_understand`, and `Wiring to CP1` a phantom tag `CP1`. On the
synthetic pair the check matched 10 tags and 45 pairs (13.3 % within
±10 %) where the generator placed 11 and 55 (21.8 %). A callout is no
longer a tag occurrence - the rule the line-network reader already kept.
### Changed ### Changed
- **`MAX_DXF_BYTES` defaults to 512 MB** (was 50 MB): a real plant layout read
for track H was a 188 MB DXF. The snapshot's refusal names the file's size,
the limit and the variable; `0` still disables the check.
- Three tool groups appear in `system_about` and `docs/tool-inventory.json`: - Three tool groups appear in `system_about` and `docs/tool-inventory.json`:
`styles` (10), `page_setup` (6), `environment` (22) — 204 tools in 23 groups; `styles` (10), `page_setup` (6), `environment` (22) — 204 tools in 23 groups;
`LEAN_TOOL_NAMES` 55; capability keys `dwgprops`, `dwt_write`, `LEAN_TOOL_NAMES` 55; capability keys `dwgprops`, `dwt_write`,
@@ -949,6 +1018,86 @@ The first run exited 1: the first room label was refused because the room
reader found 0 wall segments on `A-WALL-E-N` — the `entity_list` narrowing reader found 0 wall segments on `A-WALL-E-N` — the `entity_list` narrowing
listed under **Fixed**. The run above is the one after the fix. listed under **Fixed**. The run above is the one after the fix.
### Live COM smoke — understanding (track H)
`scripts/smoke_understand_com.py`, AutoCAD 2026, Windows 11, on the synthetic
plant pair only — `take_snapshot` of `pid.dxf` opened live, the same P&ID
saved as DWG and read back through the read-only DWG route, `take_snapshot`
of `layout.dxf`, `pipe_rows` / `cable_rows` on the live snapshots against the
pair's truth, then one tag moved 10 mm and `diff_snapshots` of the layout
before and after, all through the COM engine, exit 0, every document closed:
```json
{
"autocad": "25.1s (LMS Tech)",
"export_s": 0.104,
"live_pid_source": "live:pid.dxf",
"dwg_pid_source": "live:pid.dwg",
"model_space": {
"pid_file": {
"ARC": 1,
"INSERT": 15,
"LINE": 8,
"LWPOLYLINE": 8,
"MTEXT": 8,
"TEXT": 31
},
"pid_live": {
"ARC": 1,
"INSERT": 15,
"LINE": 8,
"LWPOLYLINE": 8,
"MTEXT": 8,
"TEXT": 31
},
"pid_dwg": {
"ARC": 1,
"INSERT": 15,
"LINE": 8,
"LWPOLYLINE": 8,
"MTEXT": 8,
"TEXT": 31
},
"layout_file": {
"CIRCLE": 18,
"LINE": 25,
"LWPOLYLINE": 4,
"MULTILEADER": 8,
"TEXT": 26
},
"layout_live": {
"CIRCLE": 18,
"LINE": 25,
"LWPOLYLINE": 4,
"MULTILEADER": 8,
"TEXT": 26
}
},
"paper_space_layouts": [],
"takeoffs": {
"scale_verdict": "schematic",
"within_10pct": 0.21818181818181817,
"runs_checked": 9,
"pipe_mismatch": [],
"loads_checked": 4,
"cable_mismatch": []
},
"diff": {
"moved": "44",
"added": 0,
"removed": 0,
"changed": 1,
"moved_is_the_change": true
}
}
```
The smoke works only in documents it brought in from its own temporary
folder, known by full path; each live snapshot, the DWG save and the move
first check, in the same COM-thread call, that the active document is the
smoke's own, and on the way out it re-activates the document that was active
when it started.
## [1.5.1] — 2026-08-06 ## [1.5.1] — 2026-08-06
A patch for three defects found *after* 1.5.0 went to PyPI, two of them by the A patch for three defects found *after* 1.5.0 went to PyPI, two of them by the
+14 -4
View File
@@ -33,6 +33,9 @@ pip install -e ".[com]"
# With PDF/screenshot support via matplotlib # With PDF/screenshot support via matplotlib
pip install -e ".[pdf]" pip install -e ".[pdf]"
# XLSX workbooks (takeoffs, data_extract) via openpyxl
pip install -e ".[office]"
# Everything # Everything
pip install -e ".[full]" pip install -e ".[full]"
``` ```
@@ -97,10 +100,12 @@ Tools are organized into sections (counts are indicative — `system_about` is a
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel). 23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean. 24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
25. Architecture (12 tools, pack `arch`): `arch_wall`, `arch_opening`, `arch_stair`, `arch_room`, `arch_rooms_detect`, `arch_schedule`, `arch_grid`, `arch_symbol`, `arch_catalogue_list`, `arch_catalogue_insert`, `arch_dimension_chains`, `arch_plan_from_spec`. One plan model — a wall network (axis polyline, thickness, justification, material) with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, cut faces hatched by material (`engineering/arch/materials.py`: ISO 128-50 general hatching for brick, the JIS A 0150 RC / stone / wood symbols, AutoCAD's AR-CONC / AR-SAND, AAC as a declared convention), and anything it cannot place listed in `omitted: [{element, reason}]`, never dropped. Rooms are *computed*: a label's area is the planar face its point lies in, and `arch_rooms_detect` reads rooms from any plan (confidence 1.0 for our walls, 0.6 for plain lines) without modifying it. Schedules are a read of the `ACADMCP_ARCH` records, drawn as real TABLEs. Labels and schedules speak English, or Turkish with `lang="tr"` (decimal comma). The COM paths were executed live once by `scripts/smoke_arch_com.py` (AutoCAD 2026). 25. Architecture (12 tools, pack `arch`): `arch_wall`, `arch_opening`, `arch_stair`, `arch_room`, `arch_rooms_detect`, `arch_schedule`, `arch_grid`, `arch_symbol`, `arch_catalogue_list`, `arch_catalogue_insert`, `arch_dimension_chains`, `arch_plan_from_spec`. One plan model — a wall network (axis polyline, thickness, justification, material) with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, cut faces hatched by material (`engineering/arch/materials.py`: ISO 128-50 general hatching for brick, the JIS A 0150 RC / stone / wood symbols, AutoCAD's AR-CONC / AR-SAND, AAC as a declared convention), and anything it cannot place listed in `omitted: [{element, reason}]`, never dropped. Rooms are *computed*: a label's area is the planar face its point lies in, and `arch_rooms_detect` reads rooms from any plan (confidence 1.0 for our walls, 0.6 for plain lines) without modifying it. Schedules are a read of the `ACADMCP_ARCH` records, drawn as real TABLEs. Labels and schedules speak English, or Turkish with `lang="tr"` (decimal comma). The COM paths were executed live once by `scripts/smoke_arch_com.py` (AutoCAD 2026).
26. Understanding & QA (4 tools, pack `core`): `drawing_understand` (one read-only report of a drawing somebody else made — declared vs inferred units with the evidence, declared vs robust extents and the outlier handles behind them, clusters (sheets, plan copies), every layer's likely discipline and service from an EN / TR / RU / NL / DE vocabulary with a confidence and the deciding keyword, equipment tags, rooms, text languages, blocks, layouts, title-block attributes), `drawing_scale_check` (a drawing against a reference: tags unique within a cluster are matched — a tag seen twice is `ambiguous`, never picked — pairs at least 2 m apart, the median ratio, IQR and the share within ±10 %: `to_scale` at 80 % or more, with its factor; `schematic` at 50 % or less; `partly` between), `drawing_diff` (two revisions matched by signature, then handle, then type + layer + nearest position; `markup=True` draws revision clouds on the newer one — its only write) and `drawing_topology_check` (dangling ends, near misses, interior crossings; the three `topo_*` critique focuses run the same engine under `focus=None` on layers no semantic focus covers). Every one reads one snapshot per drawing (`engineering/understand/snapshot.py`: `ezdxf` directly, or on a live seat `Document.Export(<tmp>/snap, "DXF", ...)`, which never renames the document and carries model space, paper space and block definitions — measured).
27. Plant takeoffs (2 tools, pack `plant`): `pipe_takeoff` (topology from the P&ID — segments merged at their ends, split at T-junctions, continued through inline fittings, a double-drawn segment counted once; service from the layer overridden by supply / return words; sizes kept as drawn — `Ø51`, `SMS51`, `DN20` are never equated — direct, by continuity or `unassigned`; lengths from the layout: the rectilinear MST of the tags a run connects, labelled an estimate, the P&ID length kept beside it as a comparison; statuses A / B / C) and `cable_takeoff` (one row per load: power from the P&ID's electrical texts, never invented; the panel from the layout's `Wiring to CPn` callouts; the Manhattan length plus the allowance rounded up to the metre; a section only from a caller's `section_rules`). Both write a workbook (Metraj / Özet / Kontrol / Metodoloji, `lang` tr / en / ru, XLSX with the `office` extra, CSV always) and never the drawing. `scripts/field_test_takeoff.py` runs both on a user's own pair and writes next to it, never into this repository. The COM paths were executed live once by `scripts/smoke_understand_com.py` (AutoCAD 2026).
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 64 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning. **Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 65 curated drafting tools (55 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members, and keeps `drawing_understand`; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`. **Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch,plant` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools, `plant` the two SECTION 27 takeoffs (track D's Plant 3D readers join it); SECTION 18/19/24/26 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index. **Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
@@ -112,7 +117,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so. **Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
**Benchmark matrix:** `benchmarks/tasks_v6.py` is the current set (19 tasks; `--matrix v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (43 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`). **Benchmark matrix:** `benchmarks/tasks_v7.py` is the current set (21 tasks; `--matrix v6` / `v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) or the v7 `takeoff_roundtrip` / `understand_foreign` (the synthetic plant pair of `tests/fixtures/plant_pair.py` read without its truth: the P&ID called schematic, every routable run's layout length equal to the rectilinear MST of its tags, the cable metres rounded up after the allowance; units, the outlier, both plan copies and the service layers found by one `describe`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (48 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
### Adding a New Tool ### Adding a New Tool
@@ -139,7 +144,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out. - **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing). - **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing. - **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
- **`TOOL_PACKS=all|core,pid,settings,mech,arch`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`. - **`TOOL_PACKS=all|core,pid,settings,mech,arch,plant`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README. - **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change. - **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above). - **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
@@ -153,6 +158,11 @@ Tools are organized into sections (counts are indicative — `system_about` is a
- **Plan text follows the plot scale**: `scale` is a denominator (50 = 1:50) and a paper height `h` is drawn at `h × scale` drawing units — no annotative objects. `lang="tr"` switches words and the decimal separator; any other language is refused with the list. - **Plan text follows the plot scale**: `scale` is a denominator (50 = 1:50) and a paper height `h` is drawn at `h × scale` drawing units — no annotative objects. `lang="tr"` switches words and the decimal separator; any other language is refused with the list.
- **Three `arch_*` critique focuses** (`arch_room_unlabelled`, `arch_wall_gap` — two wall ends within 50 mm that do not join, `arch_opening_clash`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_ARCH` record. - **Three `arch_*` critique focuses** (`arch_room_unlabelled`, `arch_wall_gap` — two wall ends within 50 mm that do not join, `arch_opening_clash`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_ARCH` record.
- **Nominal, not standard**: the furniture and sanitary blocks carry nominal catalogue dimensions and say so; the stair tool reports the Blondel relation (2R + G within 600–650 mm) as a design rule, never as a building code; the `arch` layer colours are a repository convention (ISO 13567 fixes names, not colours). - **Nominal, not standard**: the furniture and sanitary blocks carry nominal catalogue dimensions and say so; the stair tool reports the Blondel relation (2R + G within 600–650 mm) as a design rule, never as a building code; the `arch` layer colours are a repository convention (ISO 13567 fixes names, not colours).
- **Read-only by default (track H)**: `drawing_understand`, `drawing_scale_check`, `drawing_topology_check` and both takeoffs never modify the drawing; `drawing_diff` writes only with `markup=True`, and a takeoff writes a workbook file, never the drawing.
- **Lengths come from the inferred unit**: a reader measures in the unit the geometry implies (text heights, wall and door sizes, overall size) and reports metres; a disagreement with `INSUNITS` is a warning with the numbers — a layout declaring inches over millimetre geometry is measured in millimetres, never ×25.4.
- **Scale check before a P&ID length**: a P&ID is a length source only when `drawing_scale_check` calls it `to_scale`; otherwise a run's length is the orthogonal estimate on the layout and the drawn length stays a comparison column (`length_source="pid"` on a schematic P&ID is refused unless `force=True`, quoting the statistics).
- **Nothing is invented (track H)**: a power the P&ID does not state stays empty and becomes an open item, an unreadable size stays `unassigned`, and no cable-section table is authored — a section comes only from the caller's `section_rules`.
- **Client drawings never enter the repository**: every track-H fixture is the synthetic plant pair (`tests/fixtures/plant_pair.py`), whose truth is computed from the positions it placed; `scripts/field_test_takeoff.py` refuses an output folder inside the repository and prints no quantity unless `--show` is given.
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator. - **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
### Premium Drawing Rules ### Premium Drawing Rules
+17 -14
View File
@@ -27,24 +27,24 @@ Live through COM on Windows, or headless through ezdxf anywhere — one typed co
</div> </div>
> **v1.5 release snapshot:** 247 tools · 8 resources · 5 prompt templates · 4942 collected tests. > **v1.5 release snapshot:** 247 tools · 8 resources · 5 prompt templates · 4960 collected tests.
> 247 is the **registered** count; a default install advertises 242 over `tools/list`, > 247 is the **registered** count; a default install advertises 242 over `tools/list`,
> because `ENABLE_3D` is unset. `system_about` is the runtime authority. > because `ENABLE_3D` is unset. `system_about` is the runtime authority.
## Why this exists ## Why this exists
**A big MCP server is expensive to be connected to.** The full catalog costs a client **78,772 tokens** before it has asked for anything. Discovery mode replaces it with two tools and costs **356**. **A big MCP server is expensive to be connected to.** The full catalog costs a client **83,758 tokens** before it has asked for anything. Discovery mode replaces it with two tools and costs **356**.
**A drafter searches for `FILLET`, not `entity_fillet`.** Those command names appeared in no tool name or description — `df = 0` against a stock index, not badly ranked but *absent*. The fix was data: an authored corpus of **206 AutoCAD command names and 1439 synonym phrases** covering all 247 tools. A test refuses to let a tool exist without one. **A drafter searches for `FILLET`, not `entity_fillet`.** Those command names appeared in no tool name or description — `df = 0` against a stock index, not badly ranked but *absent*. The fix was data: an authored corpus of **206 AutoCAD command names and 1439 synonym phrases** covering all 247 tools. A test refuses to let a tool exist without one.
| Advertised surface | Tools seen | Idle cost | | Advertised surface | Tools seen | Idle cost |
|---|---:|---:| |---|---:|---:|
| `TOOL_PROFILE=full` (default) | 236 | 78,772 tokens | | `TOOL_PROFILE=full` (default) | 242 | 83,758 tokens |
| `TOOL_PACKS=core,settings` (full profile) | 201 | 60,926 tokens | | `TOOL_PACKS=core,settings` (full profile) | 205 | 63,890 tokens |
| `TOOL_PACKS=core,mech` (full profile) | 193 | 61,872 tokens | | `TOOL_PACKS=core,mech` (full profile) | 197 | 64,836 tokens |
| `TOOL_PACKS=core,arch` (full profile) | 191 | 60,568 tokens | | `TOOL_PACKS=core,arch` (full profile) | 195 | 63,532 tokens |
| `TOOL_PACKS=core` (full profile) | 179 | 54,158 tokens | | `TOOL_PACKS=core` (full profile) | 183 | 57,122 tokens |
| `TOOL_PROFILE=lean` | 64 | 21,939 tokens | | `TOOL_PROFILE=lean` | 65 | 22,661 tokens |
| `DISCOVERY_MODE=search` | 2 | **356 tokens** | | `DISCOVERY_MODE=search` | 2 | **356 tokens** |
> [!NOTE] > [!NOTE]
@@ -106,6 +106,7 @@ Claude Desktop, Cursor, or any stdio MCP host. For HTTP: `autocad-mcp --transpor
| Mechanical parts | one part model (segments or outline + typed features) and one view engine: front / side / top / section / detail with hidden lines, ISO 128-50 cut faces and ISO 129 dimensions; ISO 4014/4017/4032/7089/4762 fasteners, ISO 15 bearings, DIN 471/472, DIN 509, DIN 332 and ISO 3601-2 as real blocks with attributes; ISO 21920-1 surface texture, ISO 2553 welds, ISO 128-40 section lines | | Mechanical parts | one part model (segments or outline + typed features) and one view engine: front / side / top / section / detail with hidden lines, ISO 128-50 cut faces and ISO 129 dimensions; ISO 4014/4017/4032/7089/4762 fasteners, ISO 15 bearings, DIN 471/472, DIN 509, DIN 332 and ISO 3601-2 as real blocks with attributes; ISO 21920-1 surface texture, ISO 2553 welds, ISO 128-40 section lines |
| Sheet & delivery | ISO 5457 frames A4-A0 with zones and trim marks, ISO 7200 title blocks for every size, revision blocks with clouds, ISO 7573 parts lists with ISO 6433 balloons linked by XDATA, CSV/XLSX extraction, xrefs and images on both engines, real `.dwg` on a live seat | | Sheet & delivery | ISO 5457 frames A4-A0 with zones and trim marks, ISO 7200 title blocks for every size, revision blocks with clouds, ISO 7573 parts lists with ISO 6433 balloons linked by XDATA, CSV/XLSX extraction, xrefs and images on both engines, real `.dwg` on a live seat |
| Architecture | one plan model — walls with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, poché by material; rooms measured from the faces the walls enclose (never typed) and read back from any plan with a confidence; door / window / room schedules as real TABLEs; a furniture and sanitary catalogue at nominal sizes; structural grid, north arrow, section / level / elevation marks; exterior dimension chains; English or Turkish labels (`lang="tr"`, decimal comma) | | Architecture | one plan model — walls with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, poché by material; rooms measured from the faces the walls enclose (never typed) and read back from any plan with a confidence; door / window / room schedules as real TABLEs; a furniture and sanitary catalogue at nominal sizes; structural grid, north arrow, section / level / elevation marks; exterior dimension chains; English or Turkish labels (`lang="tr"`, decimal comma) |
| Understanding & takeoffs | `drawing_understand` reads a drawing somebody else made in one call — declared vs inferred units, robust extents and the outlier behind them, plan copies, each layer's discipline and service in five languages, tags, rooms; `drawing_scale_check` says whether a P&ID is to scale against its layout; `pipe_takeoff` / `cable_takeoff` take the topology from the P&ID and the lengths from the layout (rectilinear MST, statuses A/B/C, power never invented) into a Turkish / English / Russian workbook; `drawing_diff` between revisions; `drawing_topology_check` for dangling ends, near misses and crossings |
| P&ID | catalogue blocks with ports and tags (ISO 10628-2 / ISA-5.1), port-to-port lines with ISA-5.1 classes and line numbers, `pid_graph` reads any P&ID back with confidence, instrument index / line list / equipment list, `pid_from_spec` one-call sheets | | P&ID | catalogue blocks with ports and tags (ISO 10628-2 / ISA-5.1), port-to-port lines with ISA-5.1 classes and line numbers, `pid_graph` reads any P&ID back with confidence, instrument index / line list / equipment list, `pid_from_spec` one-call sheets |
| Styles & standards | ISO-25 / ANSI dimension styles, ISOCP / ROMANS text styles, ISO / ANSI leader styles from authored presets; `drawing_apply_standard("iso")` sets all of it plus units and the `mech` layers in one call; `changed` reports only what moved | | Styles & standards | ISO-25 / ANSI dimension styles, ISOCP / ROMANS text styles, ISO / ANSI leader styles from authored presets; `drawing_apply_standard("iso")` sets all of it plus units and the `mech` layers in one call; `changed` reports only what moved |
| Discovery | `search_tools` ranked over an AutoCAD command and synonym corpus — `FILLET`, `BPOLY`, `QSELECT`, `WBLOCK`, `OVERKILL`, `CHSPACE` each rank **#1** of the 242-tool advertised catalog | | Discovery | `search_tools` ranked over an AutoCAD command and synonym corpus — `FILLET`, `BPOLY`, `QSELECT`, `WBLOCK`, `OVERKILL`, `CHSPACE` each rank **#1** of the 242-tool advertised catalog |
@@ -162,23 +163,23 @@ Self-measurement, produced by scripts in [`benchmarks/`](https://github.com/U-C4
### Correctness — every release re-proves itself ### Correctness — every release re-proves itself
43 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run. 48 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.
| Version | Checks passing | Pass rate | Fixed | Regressed | | Version | Checks passing | Pass rate | Fixed | Regressed |
|---|---:|---:|---:|---:| |---|---:|---:|---:|---:|
| v1.5.1 *(baseline)* | 26 / 43 | 60.5 % | — | — | | v1.5.1 *(baseline)* | 26 / 48 | 54.2 % | — | — |
| **v1.6.0-dev** *(this branch)* | **43 / 43** | **100 %** | 17 | **0** | | **v1.6.0-dev** *(this branch)* | **48 / 48** | **100 %** | 22 | **0** |
The seventeen fixed rows are new capability (`miss → pass`). Track F adds four: `arch_junction_l_t_x` (an L corner mitres at (4100, −100) / (3900, 100), a T stem stops on the near face at (2950, 100) / (3050, 100), an X cuts all four faces, and the wall areas are 1 400 000, 1 490 000 and 2 360 000 mm² — computed by hand), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls measures 20.00 m² in its label and its record, not the 21.84 m² its axes enclose), `arch_opening_cuts_wall` (both faces are interrupted across a 900 door and two jambs close it) and `arch_rooms_detect_foreign` (a plan of plain lines on a `WALLS` layer yields 20 and 15 m² at confidence 0.6, and nothing is written). Tracks B and G add seven: `mech_part_roundtrip` (a part read back out of its own `ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a 60 x ⌀40 sleeve with a ⌀20 bore cuts 1200 mm² exactly, measured out of the drawing by `analysis_measure_entity`), `mech_iso286_on_dimension` (40 H7 is +0.025 / 0 and survives the layout), `mech_thread_unrepresented_is_caught` (the ISO 6410 focus fires on a thread drawn as a plain circle — a gate that never fires is not a gate), `std_part_iso4014_m12` (s = 18, k = 7.5, read by value), `sheet_frame_iso5457_a3` (the frame runs 20,10 to 410,287 inside the 420 × 297 sheet) and `bom_balloon_link` (two identical bolts are one row of quantity 2, and its balloon carries the row's item number). The other six are the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings. The twenty-two fixed rows are new capability (`miss → pass`). Track H adds five: `scale_check_detects_schematic` (the synthetic P&ID reads **schematic** against its layout, and a uniform 2× copy of six tags reads **to scale** with factor 2), `pipe_takeoff_rmst_exact` (all eight routable runs of the pair measure exactly the rectilinear MST of their tags), `cable_takeoff_roundup` (9, 17, 19 and 10 m after the 20 % allowance, and the heater's unstated power stays empty), `diff_detects_known_edits` (a move, a text change, an attribute change, an add and a delete — and nothing else) and `topology_known_defects` (a 5 mm near miss, one interior crossing, and a clean T that is neither). Track F adds four: `arch_junction_l_t_x` (an L corner mitres at (4100, −100) / (3900, 100), a T stem stops on the near face at (2950, 100) / (3050, 100), an X cuts all four faces, and the wall areas are 1 400 000, 1 490 000 and 2 360 000 mm² — computed by hand), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls measures 20.00 m² in its label and its record, not the 21.84 m² its axes enclose), `arch_opening_cuts_wall` (both faces are interrupted across a 900 door and two jambs close it) and `arch_rooms_detect_foreign` (a plan of plain lines on a `WALLS` layer yields 20 and 15 m² at confidence 0.6, and nothing is written). Tracks B and G add seven: `mech_part_roundtrip` (a part read back out of its own `ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a 60 x ⌀40 sleeve with a ⌀20 bore cuts 1200 mm² exactly, measured out of the drawing by `analysis_measure_entity`), `mech_iso286_on_dimension` (40 H7 is +0.025 / 0 and survives the layout), `mech_thread_unrepresented_is_caught` (the ISO 6410 focus fires on a thread drawn as a plain circle — a gate that never fires is not a gate), `std_part_iso4014_m12` (s = 18, k = 7.5, read by value), `sheet_frame_iso5457_a3` (the frame runs 20,10 to 410,287 inside the 420 × 297 sheet) and `bom_balloon_link` (two identical bolts are one row of quantity 2, and its balloon carries the row's item number). The other six are the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.
### The task matrix — tasks that can fail ### The task matrix — tasks that can fail
An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds four more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness, one where a whole mechanical sheet has to come out clean, and one where a floor plan's rooms are read back off the drawing. An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds six more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness, one where a whole mechanical sheet has to come out clean, one where a floor plan's rooms are read back off the drawing, and two where a foreign plant's P&ID and layout are read: the takeoffs against lengths computed in advance, and the one-call report against the defects planted in it.
| Task | Verified against | | Task | Verified against |
|---|---| |---|---|
| `tool_discovery` | six AutoCAD command names, each ranking #1 | | `tool_discovery` | six AutoCAD command names, each ranking #1 |
| `token_budget` | 78,772 → 356 tokens, against a ceiling fixed in advance | | `token_budget` | 83,758 → 356 tokens, against a ceiling fixed in advance |
| `hatch_islands` | 300 filled with the island, 400 ignoring it | | `hatch_islands` | 300 filled with the island, 400 ignoring it |
| `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 | | `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 |
| `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives | | `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives |
@@ -186,6 +187,8 @@ An earlier matrix scored this server 10/10, which carried no information: every
| `page_setup_truth` | ANSI B then ISO A3 landscape applied to Layout1, each plotted through `batch_plot` and read back from its PDF's `/MediaBox`: 432 × 279 then 420 × 297 mm, not the setter's return value (a fresh layout is already A3, so the B sheet is what a no-op setter cannot fake) | | `page_setup_truth` | ANSI B then ISO A3 landscape applied to Layout1, each plotted through `batch_plot` and read back from its PDF's `/MediaBox`: 432 × 279 then 420 × 297 mm, not the setter's return value (a fresh layout is already A3, so the B sheet is what a no-op setter cannot fake) |
| `mech_assembly` | a flange-coupling A3 sheet — hub with a full section, flange with its bolt-circle end view, two ISO 4014 bolts with ISO 4032 nuts, an ISO 7573 parts list read off the drawing, linked balloons, ISO 5457 frame, ISO 7200 title block — gated on `drawing_critique(focus=None)` returning **zero** issues and a finalize score of at least **90**; a test drops one balloon and watches the gate fail | | `mech_assembly` | a flange-coupling A3 sheet — hub with a full section, flange with its bolt-circle end view, two ISO 4014 bolts with ISO 4032 nuts, an ISO 7573 parts list read off the drawing, linked balloons, ISO 5457 frame, ISO 7200 title block — gated on `drawing_critique(focus=None)` returning **zero** issues and a finalize score of at least **90**; a test drops one balloon and watches the gate fail |
| `arch_roundtrip` | a two-room plan — an entrance door, an interior door, two windows, a stair — drawn by `arch_plan_from_spec` and read back by `arch_rooms_detect`, which never sees the spec: each room within 0.1 % of its label and of the net floor computed by hand (4825 × 5750 and 3825 × 5750 mm), the door and window schedules listing D1, D2, W1, W2, `drawing_critique(focus=None)` at **zero** and a finalize score of at least **90**; a test removes one room label and watches the gate fail | | `arch_roundtrip` | a two-room plan — an entrance door, an interior door, two windows, a stair — drawn by `arch_plan_from_spec` and read back by `arch_rooms_detect`, which never sees the spec: each room within 0.1 % of its label and of the net floor computed by hand (4825 × 5750 and 3825 × 5750 mm), the door and window schedules listing D1, D2, W1, W2, `drawing_critique(focus=None)` at **zero** and a finalize score of at least **90**; a test removes one room label and watches the gate fail |
| `takeoff_roundtrip` | the synthetic plant pair — a P&ID stretched 1.3× in one room and rearranged in the other, Russian supply / return words, a return drawn on the supply layer, a reducer, a segment drawn twice — read by the scale check (**schematic**: 21.8 % of 55 pairs within ±10 %), `pipe_takeoff` (every routable run's layout length equal to the rectilinear MST of its tags, sizes and services as drawn, the run to a tag the layout lacks status C) and `cable_takeoff` (9 / 17 / 19 / 10 m after the 20 % allowance, the heater's power left empty); a test removes the wiring callouts and watches the cable half fail by name |
| `understand_foreign` | the pair's layout — inches declared over millimetres, a stray line 5,000 km out, two plan copies — comes back as millimetres with an `INSUNITS` warning, the stray line's handle and two clusters; the P&ID's `P_product piping` / `P_cipsupplyline` / `P_cipreturnline` / `P_ijswater` / `E_power` classify as product / CIP supply / CIP return / ice water / electrical; a test makes the layout declare millimetres and watches the units gate fail |
### Headless performance ### Headless performance
+15 -4
View File
@@ -127,6 +127,17 @@ result for it.
The competitor reports carry no result for it. The competitor reports carry no result for it.
### Matrix v7 (v1.6) — understanding foreign drawings
`tasks_v7.py` keeps the v6 nineteen unchanged and adds track H's evidence, both on the synthetic plant pair of `tests/fixtures/plant_pair.py` (invented names; its truth is computed from the positions it placed, never typed):
| Task | Category | Verified against |
|---|---|---|
| `takeoff_roundtrip` | plant | `scale_check(pid, layout)` must say **schematic** (room 1 is a 1.3× stretch, room 2 is rearranged: 21.8 % of the 55 tag pairs within ±10 % of the median ratio); `pipe_rows(..., length_source="auto")` must report all nine runs with the truth's status, each routable run's layout length equal to the rectilinear MST of its tags to a micron (PR1 5500, PR2 13200, CS1 20700 mm, …), its size and service as drawn — the CIP return drawn on the supply layer included — and both sizes across the reducer; `cable_rows` must give M11 / T102 / M12 / T202 their panels, 6900 / 14100 / 15800 / 8000 mm and 9 / 17 / 19 / 10 m after the 20 % allowance, the heater's power empty. `tests/test_benchmark_v7.py` deletes the wiring callouts and the gate names all four loads |
| `understand_foreign` | understand | `describe(layout)` must infer millimetres and warn about `INSUNITS` (the file says inches), name the stray line's handle in its extents and report two clusters (the plan is drawn twice, 100 m apart); `describe(pid)` must classify `P_product piping`, `P_cipsupplyline`, `P_cipreturnline`, `P_ijswater` and `E_power` as product, CIP supply, CIP return, ice water and electrical. `tests/test_benchmark_v7.py` makes the layout declare millimetres and the units gate fails alone |
The competitor reports carry no result for them.
## Live competitor lane (v1.4) ## Live competitor lane (v1.4)
Two competitor adapters now execute the exact same task matrix, black-box over Two competitor adapters now execute the exact same task matrix, black-box over
@@ -244,15 +255,15 @@ python benchmarks/compare_versions.py --json results.json
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate) ### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
43 checks, ezdxf backend, one subprocess per check. Machine-readable report: 48 checks, ezdxf backend, one subprocess per check. Machine-readable report:
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json). [`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
| Version | Checks passing | Pass rate | Fixed | Regressed | | Version | Checks passing | Pass rate | Fixed | Regressed |
|---------|----------------|-----------|-------|-----------| |---------|----------------|-----------|-------|-----------|
| **v1.5.1** (baseline) | 26 / 43 | 60.5 % | — | — | | **v1.5.1** (baseline) | 26 / 48 | 54.2 % | — | — |
| **v1.6.0-dev** (this branch) | 43 / 43 | 100 % | 17 | **0** | | **v1.6.0-dev** (this branch) | 48 / 48 | 100 % | 22 | **0** |
The seventeen are all `miss → pass` — v1.5.1 has none of the methods. Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven The twenty-two are all `miss → pass` — v1.5.1 has none of the methods. Five are track H: `scale_check_detects_schematic` (the synthetic P&ID is schematic against its layout; a uniform 2× copy is to scale with factor 2), `pipe_takeoff_rmst_exact` (eight routable runs, each exactly the rectilinear MST of its tags), `cable_takeoff_roundup` (9 / 17 / 19 / 10 m, the unstated power left empty), `diff_detects_known_edits` (five known edits and nothing else) and `topology_known_defects` (one near miss, one crossing, a clean T). Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven
are tracks B and G: `mech_part_roundtrip` (a part read back from its own are tracks B and G: `mech_part_roundtrip` (a part read back from its own
`ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a `ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a
sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces, sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces,
+143 -2
View File
@@ -1,4 +1,4 @@
"""Reference adapter for this repository, covering the v6 task matrix. """Reference adapter for this repository, covering the v7 task matrix.
Every task verifies against a value worked out independently of the code under Every task verifies against a value worked out independently of the code under
test — a closed-form area, a count of entities placed on purpose, a token test — a closed-form area, a count of entities placed on purpose, a token
@@ -12,7 +12,7 @@ import math
from pathlib import Path from pathlib import Path
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
from benchmarks.tasks_v6 import TaskSpec from benchmarks.tasks_v7 import TaskSpec
from engineering.delivery import deliver_drawing from engineering.delivery import deliver_drawing
from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
from engineering.refiner import refine_drawing from engineering.refiner import refine_drawing
@@ -779,6 +779,147 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
[str(sheet)], [str(sheet)],
) )
async def _task_takeoff_roundtrip(self):
"""Pipe and cable quantities off the synthetic plant pair, against its truth.
Track H's evidence (spec §15). The pair is written fresh into the
artifact folder by ``tests/fixtures/plant_pair.py``, whose truth is
computed from the positions it placed. Then only the two DXFs are read:
* ``scale_check(pid, layout)`` must say ``schematic`` - room 1 is the
layout stretched 1.3x, room 2 is rearranged, and 21.8 % of the 55 tag
pairs lie within ±10 % of the median ratio;
* ``pipe_rows(..., length_source="auto")`` must report every run of the
P&ID (the ice-water utility included, because ``services`` names it)
with the truth's status; a routable run's layout length must equal the
rectilinear MST of its tags exactly, its sizes (both across the
reducer) and its service must be the drawn ones (the CIP return drawn
on the supply layer included); the run ending on T105, which the
layout lacks, is status C with no layout length;
* ``cable_rows(pid, layout)`` must give every load its panel from the
layout's wiring callouts, its stated power (the heater's stays empty),
the Manhattan length to the panel and the metres rounded up after the
20 % allowance.
Any one slipping fails the task, and the metrics name what slipped.
"""
from engineering.understand.network import build_network
from engineering.understand.scale import scale_check
from engineering.understand.snapshot import read_snapshot
from engineering.understand.takeoff import cable_rows, pipe_rows
from tests.fixtures.plant_pair import build_plant_pair
truth = build_plant_pair(self.artifact_dir / "plant_pair")
pid = read_snapshot(truth["pid"])
layout = read_snapshot(truth["layout"])
scale = scale_check(pid, layout)
services = tuple(sorted({run["service"] for run in truth["pipe_runs"]}))
pipe = pipe_rows(
build_network(pid), pid, layout, scale=scale, services=services, length_source="auto"
)
found = {(run["service"], tuple(sorted(run["tags"]))): run for run in pipe["runs"]}
pipe_mismatch = []
for run in truth["pipe_runs"]:
got = found.get((run["service"], tuple(run["tags"])))
if got is None:
pipe_mismatch.append({"run": run["id"], "reason": "no run with these tags"})
continue
reasons = []
if got["status"] != run["status"]:
reasons.append(f"status {got['status']} != {run['status']}")
want, have = run["layout_length_mm"], got.get("layout_m")
if want is None and have is not None:
reasons.append(f"a layout length ({have} m) for an unroutable run")
elif want is not None and (have is None or abs(float(have) * 1000.0 - want) > 1e-6):
reasons.append(f"layout length {have} m != {want} mm")
if set(got["diameters"]) != set(run["diameters"]):
reasons.append(f"sizes {sorted(got['diameters'])} != {sorted(run['diameters'])}")
if reasons:
pipe_mismatch.append({"run": run["id"], "reason": "; ".join(reasons)})
cable = {row["tag"]: row for row in cable_rows(pid, layout)["rows"]}
cable_mismatch = []
for want in truth["cable_rows"]:
got = cable.get(want["tag"])
if got is None:
cable_mismatch.append({"tag": want["tag"], "wrong": ["missing"]})
continue
wrong = [key for key in ("panel", "kw") if got.get(key) != want[key]]
if got.get("cable_m") != want["roundup_m"]:
wrong.append("cable_m")
length = got.get("length_m")
if length is None or abs(float(length) * 1000.0 - want["length_mm"]) > 1e-6:
wrong.append("length_m")
if wrong:
cable_mismatch.append({"tag": want["tag"], "wrong": wrong})
passed = scale["verdict"] == truth["scale_verdict"] and not pipe_mismatch
passed = passed and not cable_mismatch
return (
passed,
{
"scale_verdict": scale["verdict"],
"within_10pct": scale.get("within_10pct"),
"length_source": pipe.get("length_source"),
"runs_checked": len(truth["pipe_runs"]),
"pipe_mismatch": pipe_mismatch,
"loads_checked": len(truth["cable_rows"]),
"cable_mismatch": cable_mismatch,
},
[truth["pid"], truth["layout"]],
)
async def _task_understand_foreign(self):
"""One call on a foreign layout reports what is wrong with it, by evidence.
Track H's evidence (spec §5, §15), on the synthetic plant pair: the
layout declares inches (``$INSUNITS = 1``) over millimetre geometry, a
stray LINE 5,000 km away stretches its declared extents, and the plan is
drawn twice, 100 m apart. ``describe`` of the layout must infer
millimetres, report that they disagree with the declared unit and warn
about ``INSUNITS``, name the stray entity's handle in its extents, and
report exactly two clusters; ``describe`` of the P&ID must classify its
four service layers and the electrical layer as the generator names them
(English and Dutch names). A warning or an extents report is searched as
JSON, so the gate holds whatever shape the evidence takes.
"""
import json
from engineering.understand.describe import describe
from engineering.understand.snapshot import read_snapshot
from tests.fixtures.plant_pair import build_plant_pair
truth = build_plant_pair(self.artifact_dir / "plant_pair")
layout = describe(read_snapshot(truth["layout"]))
pid = describe(read_snapshot(truth["pid"]))
units = layout["units"]
units_flagged = (
str(units.get("inferred")) == truth["layout_true_unit"]
and units.get("agree") is False
and "INSUNITS" in json.dumps(layout["warnings"], ensure_ascii=False)
)
outlier_found = f'"{truth["outlier_handle"]}"' in json.dumps(layout["extents"])
copies_separated = len(layout["clusters"]) == truth["cluster_count"]
read = {row["name"]: row["service"] for row in pid["layers"]}
misread = {
name: read.get(name)
for name, service in truth["layer_services"].items()
if read.get(name) != service
}
passed = units_flagged and outlier_found and copies_separated and not misread
return (
passed,
{
"units": units,
"units_flagged": units_flagged,
"outlier_found": outlier_found,
"clusters": len(layout["clusters"]),
"copies_separated": copies_separated,
"misread_layers": misread,
},
[truth["layout"], truth["pid"]],
)
async def _task_auditable_delivery(self): async def _task_auditable_delivery(self):
await ensure_engineering_layers(self.backend) await ensure_engineering_layers(self.backend)
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY") await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
+213
View File
@@ -818,6 +818,214 @@ async def arch_rooms_detect_foreign():
) )
# ── Understanding foreign drawings (track H) ───────────────────────────────
def _plant_pair() -> dict:
"""The synthetic plant pair, written to a fresh temporary folder."""
from tests.fixtures.plant_pair import build_plant_pair
return build_plant_pair(tempfile.mkdtemp(prefix="acadmcp_plant_"))
def _quoted(handle: str) -> str:
"""A handle as it appears in a JSON dump - quoted, so '2A' never matches '12A3'."""
return f'"{handle}"'
async def scale_check_detects_schematic():
"""The synthetic P&ID is schematic against its layout; a uniform 2x copy is not.
The pair: room 1 of the P&ID is the layout stretched 1.3x, room 2 is
rearranged, and of the 55 pairs of the 11 shared tags 21.8 % lie within
±10 % of the median ratio - at most 50 % is ``schematic`` (spec §6).
The control: six tags at (0, 0), (4000, 0), (8000, 0), (0, 3000),
(4000, 3000), (8000, 6000) and the same tags at twice those coordinates -
the nearest pair is 3000 mm apart, every ratio is exactly 2.0, so the
doubled drawing is ``to_scale`` against the first with factor 2.
"""
import ezdxf
from engineering.understand.scale import scale_check
from engineering.understand.snapshot import read_snapshot
truth = _plant_pair()
pair = scale_check(read_snapshot(truth["pid"]), read_snapshot(truth["layout"]))
folder = tempfile.mkdtemp(prefix="acadmcp_scale_")
positions = [(0, 0), (4000, 0), (8000, 0), (0, 3000), (4000, 3000), (8000, 6000)]
paths = []
for factor in (1.0, 2.0):
doc = ezdxf.new("R2018")
msp = doc.modelspace()
for number, (x, y) in enumerate(positions, start=1):
msp.add_circle((x * factor, y * factor), 500.0)
msp.add_text(f"T{number}", dxfattribs={"insert": (x * factor, y * factor)})
path = os.path.join(folder, f"x{factor:g}.dxf")
doc.saveas(path)
paths.append(path)
control = scale_check(read_snapshot(paths[1]), read_snapshot(paths[0]))
return (
pair["verdict"] == "schematic"
and control["verdict"] == "to_scale"
and abs(float(control["factor"]) - 2.0) < 1e-9
)
async def pipe_takeoff_rmst_exact():
"""Every routable pipe run's layout length is the rectilinear MST of its tags.
Measured on the synthetic pair with ``length_source="layout"``. The truth
is the generator's own Manhattan / Prim arithmetic over the tag positions it
placed, hand-checked in tests/test_plant_pair.py - PR1 5500 (5000 + 500),
PR2 13200 (the three distances 6000 / 7200 / 13200, spanning tree 6000 +
7200), CS1 20700 (20500 + 200). All eight routable runs must agree to a
micron; the run ending on T105, which the layout lacks, is not routable.
"""
from engineering.understand.network import build_network
from engineering.understand.snapshot import read_snapshot
from engineering.understand.takeoff import pipe_rows
truth = _plant_pair()
pid, layout = read_snapshot(truth["pid"]), read_snapshot(truth["layout"])
services = tuple(sorted({run["service"] for run in truth["pipe_runs"]}))
result = pipe_rows(build_network(pid), pid, layout, services=services, length_source="layout")
found = {(r["service"], tuple(sorted(r["tags"]))): r for r in result["runs"]}
routable = [run for run in truth["pipe_runs"] if run["layout_length_mm"] is not None]
for run in routable:
got = found.get((run["service"], tuple(run["tags"])))
if got is None or got.get("layout_m") is None:
return False
if abs(float(got["layout_m"]) * 1000.0 - run["layout_length_mm"]) > 1e-6:
return False
return len(routable) == 8
async def cable_takeoff_roundup():
"""Cable rows: the panel from the callout, Manhattan length, metres rounded up.
The truth rows of the synthetic pair, by hand: M11 (5500, 8200) to CP-1
(1200, 10800) is 6900 mm, 6.9 m x 1.2 = 8.28 -> 9; T102 (8000, 3500) to CP-1
is 14100 mm -> 16.92 -> 17; M12 (26300, 8400) to CP-2 (35000, 1300) is
15800 mm -> 18.96 -> 19; T202 (28700, 3000) to CP-2 is 8000 mm -> 9.6 -> 10.
The heater on T202 states no power, and its row must keep an empty one.
"""
from engineering.understand.snapshot import read_snapshot
from engineering.understand.takeoff import cable_rows
truth = _plant_pair()
rows = {
row["tag"]: row
for row in cable_rows(read_snapshot(truth["pid"]), read_snapshot(truth["layout"]))["rows"]
}
expected = {"M11": ("CP-1", 9), "T102": ("CP-1", 17), "M12": ("CP-2", 19), "T202": ("CP-2", 10)}
for tag, (panel, metres) in expected.items():
row = rows.get(tag)
if row is None or row.get("panel") != panel or row.get("cable_m") != metres:
return False
return rows["T202"].get("kw") is None and rows["M11"].get("kw") == 5.5
async def diff_detects_known_edits():
"""Two revisions differing by exactly five edits - and the diff says so.
A move (a LINE shifted 10 mm), a text change (PUMP A -> PUMP B), an
attribute change (TAG T101 -> T102), an add (a new LINE) and a delete (a
LINE removed); a CIRCLE stays untouched. The edits are made on a copy read
back from the first file, so every surviving entity keeps its handle. The
diff must list the added line as added, the removed line as removed, the
three edited entities as changed - and nothing else, the circle included.
"""
import json
import ezdxf
from engineering.understand.diff import diff_snapshots
from engineering.understand.snapshot import read_snapshot
folder = tempfile.mkdtemp(prefix="acadmcp_diff_")
old_path, new_path = os.path.join(folder, "rev_a.dxf"), os.path.join(folder, "rev_b.dxf")
doc = ezdxf.new("R2018")
msp = doc.modelspace()
moved = msp.add_line((0, 0), (100, 0), dxfattribs={"layer": "PIPE"})
text = msp.add_text("PUMP A", dxfattribs={"insert": (0, 50), "height": 2.5})
block = doc.blocks.new("TAGGED")
block.add_circle((0, 0), 10)
block.add_attdef("TAG", (0, 0), dxfattribs={"height": 2.5})
insert = msp.add_blockref("TAGGED", (200, 0))
insert.add_auto_attribs({"TAG": "T101"})
gone = msp.add_line((0, 100), (100, 100), dxfattribs={"layer": "PIPE"})
kept = msp.add_circle((300, 300), 20)
doc.saveas(old_path)
revision = ezdxf.readfile(old_path)
revision.entitydb[moved.dxf.handle].translate(10, 0, 0)
revision.entitydb[text.dxf.handle].dxf.text = "PUMP B"
revision.entitydb[insert.dxf.handle].get_attrib("TAG").dxf.text = "T102"
revision.modelspace().delete_entity(revision.entitydb[gone.dxf.handle])
added = revision.modelspace().add_line((500, 500), (600, 500), dxfattribs={"layer": "PIPE"})
revision.saveas(new_path)
diff = diff_snapshots(read_snapshot(old_path), read_snapshot(new_path))
changed = json.dumps(diff["changed"])
dumped = changed + json.dumps(diff["added"]) + json.dumps(diff["removed"])
return (
len(diff["added"]) == 1
and len(diff["removed"]) == 1
and len(diff["changed"]) == 3
and _quoted(added.dxf.handle) in json.dumps(diff["added"])
and _quoted(gone.dxf.handle) in json.dumps(diff["removed"])
and all(_quoted(e.dxf.handle) in changed for e in (moved, text, insert))
and _quoted(kept.dxf.handle) not in dumped
)
async def topology_known_defects():
"""Dangling ends, a near miss and a crossing, placed on purpose - and nothing else.
On layer PIPE: a closed 1000 x 1000 square (no free end, no defect); a LINE
from (500, -500) that stops at (500, -5), 5 mm short of the square's bottom
edge (a near miss at gap 10, and a dangling end); two LINEs crossing at
(2500, 500), away from all four of their ends (one interior crossing); and a
clean T - a LINE ending on the middle of another at (4500, 0), which is a
junction, not a crossing and not a near miss.
"""
import json
import ezdxf
from engineering.understand.snapshot import read_snapshot
from engineering.understand.topology import topology_findings
doc = ezdxf.new("R2018")
msp = doc.modelspace()
pipe = {"layer": "PIPE"}
ring = msp.add_lwpolyline(
[(0, 0), (1000, 0), (1000, 1000), (0, 1000)], close=True, dxfattribs=pipe
)
short = msp.add_line((500, -500), (500, -5), dxfattribs=pipe)
cross_a = msp.add_line((2000, 0), (3000, 1000), dxfattribs=pipe)
cross_b = msp.add_line((2000, 1000), (3000, 0), dxfattribs=pipe)
bar = msp.add_line((4000, 0), (5000, 0), dxfattribs=pipe)
stem = msp.add_line((4500, 0), (4500, 500), dxfattribs=pipe)
path = os.path.join(tempfile.mkdtemp(prefix="acadmcp_topo_"), "topology.dxf")
doc.saveas(path)
found = topology_findings(read_snapshot(path), layers=["PIPE"], gap=10.0)
near = json.dumps(found["near_miss"])
crossing = json.dumps(found["crossing"])
dangling = json.dumps(found["dangling"])
return (
_quoted(short.dxf.handle) in near
and _quoted(short.dxf.handle) in dangling
and _quoted(ring.dxf.handle) not in dangling
and _quoted(ring.dxf.handle) not in crossing
and len(found["crossing"]) == 1
and _quoted(cross_a.dxf.handle) in crossing
and _quoted(cross_b.dxf.handle) in crossing
and all(_quoted(e.dxf.handle) not in crossing + near for e in (bar, stem))
)
CHECKS = { CHECKS = {
"core_line_length": (core_line_length, "Core"), "core_line_length": (core_line_length, "Core"),
"core_circle_radius": (core_circle_radius, "Core"), "core_circle_radius": (core_circle_radius, "Core"),
@@ -862,6 +1070,11 @@ CHECKS = {
"arch_room_area_net": (arch_room_area_net, "Architecture"), "arch_room_area_net": (arch_room_area_net, "Architecture"),
"arch_opening_cuts_wall": (arch_opening_cuts_wall, "Architecture"), "arch_opening_cuts_wall": (arch_opening_cuts_wall, "Architecture"),
"arch_rooms_detect_foreign": (arch_rooms_detect_foreign, "Architecture"), "arch_rooms_detect_foreign": (arch_rooms_detect_foreign, "Architecture"),
"scale_check_detects_schematic": (scale_check_detects_schematic, "Understanding"),
"pipe_takeoff_rmst_exact": (pipe_takeoff_rmst_exact, "Understanding"),
"cable_takeoff_roundup": (cable_takeoff_roundup, "Understanding"),
"diff_detects_known_edits": (diff_detects_known_edits, "Understanding"),
"topology_known_defects": (topology_known_defects, "Understanding"),
} }
+3 -3
View File
@@ -13,7 +13,7 @@ import json
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from benchmarks.tasks_v6 import TASKS_V6 from benchmarks.tasks_v7 import TASKS_V7
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published" DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
@@ -66,8 +66,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1" matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
task_ids = [task.task_id for task in TASKS_V6] task_ids = [task.task_id for task in TASKS_V7]
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V6] task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V7]
servers = [report["adapter"] for report in reports] servers = [report["adapter"] for report in reports]
status_by_server = { status_by_server = {
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]} report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
@@ -1,12 +1,12 @@
{ {
"summary": { "summary": {
"baseline_ref": "v1.5.1", "baseline_ref": "v1.5.1",
"total": 43, "total": 48,
"old_pass": 26, "old_pass": 26,
"new_pass": 43, "new_pass": 48,
"old_pct": 60.5, "old_pct": 54.2,
"new_pct": 100.0, "new_pct": 100.0,
"fixed": 17, "fixed": 22,
"regressed": 0 "regressed": 0
}, },
"rows": [ "rows": [
@@ -267,6 +267,36 @@
"category": "Architecture", "category": "Architecture",
"old": "miss", "old": "miss",
"new": "pass" "new": "pass"
},
{
"check": "scale_check_detects_schematic",
"category": "Understanding",
"old": "miss",
"new": "pass"
},
{
"check": "pipe_takeoff_rmst_exact",
"category": "Understanding",
"old": "miss",
"new": "pass"
},
{
"check": "cable_takeoff_roundup",
"category": "Understanding",
"old": "miss",
"new": "pass"
},
{
"check": "diff_detects_known_edits",
"category": "Understanding",
"old": "miss",
"new": "pass"
},
{
"check": "topology_known_defects",
"category": "Understanding",
"old": "miss",
"new": "pass"
} }
] ]
} }
+1 -1
View File
@@ -16,7 +16,7 @@ from pathlib import Path
from backends.capability import UnsupportedCapabilityError from backends.capability import UnsupportedCapabilityError
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
from benchmarks.tasks_v6 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id from benchmarks.tasks_v7 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
def _load_registry() -> dict[str, dict]: def _load_registry() -> dict[str, dict]:
+71
View File
@@ -0,0 +1,71 @@
"""The v7 task matrix: v6 plus what track H added.
Track H teaches the server to read drawings somebody else made, so its two
tasks are gated on a drawing pair whose answers are known in advance - the
synthetic plant pair of ``tests/fixtures/plant_pair.py``, which reproduces
every field finding of the design spec with invented names:
* ``takeoff_roundtrip`` - the scale check calls the P&ID ``schematic``; every
routable pipe run's layout length equals the rectilinear minimum spanning
tree of its tags exactly; its size, service and status match the generator's
truth; the cable rows equal the Manhattan lengths from each load to its panel
with the allowance rounded up to the metre.
* ``understand_foreign`` - one ``describe`` of the layout flags inches declared
over millimetre geometry, finds the stray entity by handle and separates the
two plan copies; one ``describe`` of the P&ID classifies its English and
Dutch service layers.
The published competitor reports were never asked these questions; the chart
shows ``not_run`` for them, not zero.
"""
from __future__ import annotations
from benchmarks.tasks_v2 import TASKS_V2
from benchmarks.tasks_v3 import TASKS_V3
from benchmarks.tasks_v4 import TASKS_V4
from benchmarks.tasks_v5 import TASKS_V5
from benchmarks.tasks_v6 import TASKS_V6, TaskSpec
#: Added in v1.6.0 by track H.
NEW_TASKS_V7: tuple[TaskSpec, ...] = (
TaskSpec(
"takeoff_roundtrip",
"plant",
"Take pipe and cable quantities off a schematic P&ID and its layout: the P&ID "
"called schematic, every run's layout length equal to the rectilinear MST of "
"its tags, sizes and services as drawn, cable metres rounded up after the "
"allowance",
1.0,
),
TaskSpec(
"understand_foreign",
"understand",
"Read a foreign layout in one call: inches declared over millimetres flagged, "
"the stray entity found, both plan copies separated, the service layers of its "
"P&ID classified",
1.0,
),
)
TASKS_V7: tuple[TaskSpec, ...] = TASKS_V6 + NEW_TASKS_V7
#: Runner-selectable task sets. v2 to v6 stay addressable so an earlier report
#: can be reproduced exactly rather than only described.
MATRICES: dict[str, tuple[TaskSpec, ...]] = {
"v2": TASKS_V2,
"v3": TASKS_V3,
"v4": TASKS_V4,
"v5": TASKS_V5,
"v6": TASKS_V6,
"v7": TASKS_V7,
}
DEFAULT_MATRIX = "v7"
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
for task in MATRICES[matrix]:
if task.task_id == task_id:
return task
raise KeyError(f"{task_id} is not in matrix {matrix}")
@@ -1,6 +1,6 @@
# v1.6 Track H — understanding foreign drawings, takeoffs, diff and topology # v1.6 Track H — understanding foreign drawings, takeoffs, diff and topology
**Date:** 2026-09-24 · **Status:** design approved in conversation · **Date:** 2026-09-24 · **Status:** shipped (v1.6.0-dev, track H) ·
**Parent:** `2026-09-15-v1.6-roadmap-design.md` (track H; track D stays separate **Parent:** `2026-09-15-v1.6-roadmap-design.md` (track H; track D stays separate
and follows) · **Baseline:** main after track F (`0abbe55`, 241 tools) and follows) · **Baseline:** main after track F (`0abbe55`, 241 tools)
+6 -1
View File
@@ -32,7 +32,7 @@ from collections.abc import Sequence
from engineering.arch.faces import Face, face_containing, planar_faces from engineering.arch.faces import Face, face_containing, planar_faces
from engineering.pid.graph import ATTRIBUTE_TAGS, KEYWORDS from engineering.pid.graph import ATTRIBUTE_TAGS, KEYWORDS
from engineering.understand.clusters import find_clusters from engineering.understand.clusters import find_clusters
from engineering.understand.labels import parse_tag from engineering.understand.labels import parse_tag, wiring_target
from engineering.understand.snapshot import EntityRecord, Snapshot, length_of from engineering.understand.snapshot import EntityRecord, Snapshot, length_of
from engineering.understand.units import infer_units, robust_extents, wall_segments from engineering.understand.units import infer_units, robust_extents, wall_segments
from engineering.understand.vocab import classify_layer, room_label from engineering.understand.vocab import classify_layer, room_label
@@ -113,6 +113,9 @@ def tag_occurrences(snap: Snapshot, *, prep: dict | None = None) -> list[dict]:
A tag is read with `labels.parse_tag` from a TEXT / MTEXT / MULTILEADER, A tag is read with `labels.parse_tag` from a TEXT / MTEXT / MULTILEADER,
or from an INSERT attribute named like a tag (`pid.graph.ATTRIBUTE_TAGS`). or from an INSERT attribute named like a tag (`pid.graph.ATTRIBUTE_TAGS`).
A wiring callout (`labels.wiring_target`: ``Wiring to CP-2``) is not an
occurrence of the panel it names - it stands at the load, not the panel -
the rule `network.tag_occurrences` keeps too.
Two readings of one tag in one space closer than Two readings of one tag in one space closer than
:data:`TAG_MERGE_FRACTION` of the diagonal are one occurrence (the :data:`TAG_MERGE_FRACTION` of the diagonal are one occurrence (the
attribute's position and confidence win). Rows carry ``tag``, ``handle``, attribute's position and confidence win). Rows carry ``tag``, ``handle``,
@@ -128,6 +131,8 @@ def tag_occurrences(snap: Snapshot, *, prep: dict | None = None) -> list[dict]:
continue continue
readings = [] readings = []
if rec.type in _TEXT_TYPES and rec.text: if rec.type in _TEXT_TYPES and rec.text:
if wiring_target(rec.text):
continue # names the panel it points at; it is not written there
readings.append((parse_tag(rec.text), "text", TEXT_TAG_CONFIDENCE)) readings.append((parse_tag(rec.text), "text", TEXT_TAG_CONFIDENCE))
elif rec.type == "INSERT": elif rec.type == "INSERT":
for name, value in rec.attribs: for name, value in rec.attribs:
+2 -2
View File
@@ -79,7 +79,7 @@ def derived() -> dict[str, object]:
} }
from benchmarks import correctness_suite from benchmarks import correctness_suite
from benchmarks.tasks_v6 import TASKS_V6 from benchmarks.tasks_v7 import TASKS_V7
return { return {
"collected_tests": _collected_tests(), "collected_tests": _collected_tests(),
@@ -94,7 +94,7 @@ def derived() -> dict[str, object]:
"ab_fixed": ab["fixed"], "ab_fixed": ab["fixed"],
"ab_regressed": ab["regressed"], "ab_regressed": ab["regressed"],
"suite_checks": len(correctness_suite.CHECKS), "suite_checks": len(correctness_suite.CHECKS),
"matrix_tasks": len(TASKS_V6), "matrix_tasks": len(TASKS_V7),
"perf_ms": perf, "perf_ms": perf,
} }
+128
View File
@@ -0,0 +1,128 @@
"""Run both takeoffs on a real P&ID and layout, and write the report next to them.
LOCAL ONLY. This is the field test of track H (spec §15): the user points it at
a client's P&ID and layout, compares what comes out with the hand-made
workbooks, and keeps the result on their own disk. Nothing it reads or writes
may enter this repository (spec §14), so:
* the report is written next to the P&ID (or into ``--out``) and **the script
refuses any output folder inside this repository**, before reading a byte;
* the console gets file names, row counts and the scale verdict - **never a
quantity**, unless ``--show`` is given on the user's own terminal;
* it is never run in CI and never by a test on a real drawing; the test next
to it (tests/test_field_test_takeoff.py) runs it on the synthetic plant pair.
What it writes: ``pipe_takeoff.xlsx`` / ``cable_takeoff.xlsx`` (CSV beside
them always, XLSX when the ``office`` extra is installed) through the same
``write_workbook`` the tools use, and ``field_test_report.json`` with the scale
check and both results in full.
uv run --frozen python scripts/field_test_takeoff.py --pid D:/job/pid.dxf --layout D:/job/layout.dxf
uv run --frozen python scripts/field_test_takeoff.py --pid ... --layout ... --lang en --show
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
#: Exit code for a refused run (output inside the repository, a missing file).
REFUSED = 2
def _inside_repository(folder: Path) -> bool:
resolved = folder.resolve()
return resolved == ROOT or ROOT in resolved.parents
def _parse(argv) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Pipe and cable takeoffs on a real P&ID + layout (local only)."
)
parser.add_argument("--pid", required=True, help="the P&ID, a .dxf")
parser.add_argument("--layout", required=True, help="the layout, a .dxf")
parser.add_argument("--out", default=None, help="report folder (default: the P&ID's folder)")
parser.add_argument("--lang", default="tr", choices=("tr", "en", "ru"))
parser.add_argument("--allowance", type=float, default=0.20)
parser.add_argument(
"--show", action="store_true", help="also print the totals (quantities) to the console"
)
return parser.parse_args(argv)
def main(argv=None) -> int:
args = _parse(argv)
pid_path, layout_path = Path(args.pid), Path(args.layout)
out = Path(args.out) if args.out else pid_path.resolve().parent
if _inside_repository(out):
print(
f"refused: {out} is inside the repository ({ROOT}); client drawings and "
"anything derived from them never enter it. Pass --out outside it."
)
return REFUSED
for path in (pid_path, layout_path):
if not path.is_file():
print(f"refused: {path} does not exist")
return REFUSED
from engineering.understand.network import build_network
from engineering.understand.report import write_workbook
from engineering.understand.scale import scale_check
from engineering.understand.snapshot import read_snapshot
from engineering.understand.takeoff import cable_rows, pipe_rows
out.mkdir(parents=True, exist_ok=True)
pid = read_snapshot(str(pid_path))
layout = read_snapshot(str(layout_path))
scale = scale_check(pid, layout)
pipe = pipe_rows(build_network(pid), pid, layout, scale=scale, allowance=args.allowance)
cable = cable_rows(pid, layout, allowance=args.allowance)
written = [
write_workbook(pipe, kind="pipe", lang=args.lang, path=str(out / "pipe_takeoff.xlsx")),
write_workbook(cable, kind="cable", lang=args.lang, path=str(out / "cable_takeoff.xlsx")),
]
report = out / "field_test_report.json"
report.write_text(
json.dumps(
{"scale": scale, "pipe": pipe, "cable": cable, "written": written},
indent=2,
ensure_ascii=False,
default=str,
),
encoding="utf-8",
)
print(
json.dumps(
{
"scale_verdict": scale["verdict"],
"pipe_rows": len(pipe["rows"]),
"cable_rows": len(cable["rows"]),
"report": str(report),
"workbooks": [
path
for item in written
for path in ([item["xlsx"]] if item["xlsx"] else []) + list(item["csv"])
],
},
indent=2,
)
)
if args.show:
print(
json.dumps(
{"pipe": pipe["rows"], "cable": cable["rows"]},
indent=2,
ensure_ascii=False,
default=str,
)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+386
View File
@@ -0,0 +1,386 @@
"""Exercise track H once against the live AutoCAD on this machine.
Builds the synthetic plant pair (tests/fixtures/plant_pair.py) in a temporary
folder and reads it through the live engine - never an operator's drawing:
open pid.dxf -> take_snapshot(backend) (Document.Export, "DXF")
-> drawing_save_as(pid.dwg), close it
-> take_snapshot(backend, pid.dwg) (opened read-only, exported, closed)
open layout.dxf -> take_snapshot(backend)
pipe_rows / cable_rows on the two live snapshots, against the pair's truth
entity_move one tag text by 10 mm -> take_snapshot -> diff_snapshots
against the snapshot taken before the move
and prints what came back as JSON. Exit codes:
0 every live snapshot holds the same model-space entities (by type) as the
file it came from, the takeoffs match the truth, and the diff reports
exactly the one moved entity
1 it ran, but something disagreed (the JSON says what)
2 no live CAD application reachable (pywin32 missing, or no running
instance to attach to) -- the COM paths then remain fake-tested only
**The operator's drawings are never touched.** AutoCAD may have other
documents open, with unsaved work in them, while this runs. So:
* the smoke only ever attaches to a running instance (``GetActiveObject``);
it never creates one, because ``Dispatch`` would launch AutoCAD;
* every document it works in is one it brought in itself from the files it
just wrote into its own temporary folder, and it knows them by full path -
never by a name another document could share;
* every call that reads or writes *the current document* (a live snapshot, the
DWG save, the move) first checks, inside the same COM-thread call, that the
active document is the smoke's own - and refuses otherwise;
* on the way out (also when a step fails) it closes its own documents with
``Close(False)`` and re-activates the document that was active when it
started; unless ``--keep`` is given, then its documents stay open;
* nothing it prints names a document other than its own.
AUTOCAD_MCP_BACKEND=com uv run --frozen python scripts/smoke_understand_com.py
uv run --frozen python scripts/smoke_understand_com.py --keep # leave them open
"""
from __future__ import annotations
import asyncio
import dataclasses
import json
import sys
import tempfile
import time
from collections import Counter
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT))
#: ``RPC_E_CALL_REJECTED``: AutoCAD's message filter refused the call because
#: it was busy (a document still initialising after ``Documents.Open``). A
#: refused call was not executed.
RPC_E_CALL_REJECTED = -2147418111
#: How long a refused call is retried before the smoke gives up on it.
REJECTED_PATIENCE_S = 90.0
#: Full paths (normalised) of the documents this run opened and has not closed.
_OPENED: list[str] = []
#: The run's state shared with the COM thread: the folder its files live in,
#: the document that must be active for the next current-document call (None:
#: the call does not touch the current document), and the document that was
#: active when the run started (a COM object, used on the COM thread only).
_STATE: dict = {"folder": None, "expect": None, "start": None}
class NotOurDocument(RuntimeError):
"""The active document is not the one the smoke is about to read or write."""
def _key(path) -> str:
return str(Path(str(path)).resolve()).lower()
def _ours(path) -> bool:
folder = _STATE["folder"]
return bool(path) and folder is not None and Path(_key(path)).parent == Path(_key(folder))
def _check_active() -> None:
"""COM thread: refuse unless the active document is the expected own one."""
expected = _STATE["expect"]
if expected is None:
return
from backends import com_backend as com
app = com._acad_app()
active = str(app.ActiveDocument.FullName) if int(app.Documents.Count) else ""
if not active or _key(active) != expected:
# Never name the other document: it may be an operator's drawing.
raise NotOurDocument(
f"refused: the active document is not the smoke's own {Path(expected).name}"
)
def _patient(backend) -> None:
"""Retry any call AutoCAD refused with ``RPC_E_CALL_REJECTED``, and run the
active-document check in the same COM-thread call as the call it guards."""
run = backend._run
async def patient(func, *args, **kwargs):
def guarded(*a, **kw):
_check_active()
return func(*a, **kw)
deadline = time.monotonic() + REJECTED_PATIENCE_S
pause = 0.5
while True:
try:
return await run(guarded, *args, **kwargs)
except RuntimeError as exc:
cause = exc.__cause__
hr = cause.args[0] if cause is not None and cause.args else None
if hr != RPC_E_CALL_REJECTED or time.monotonic() + pause > deadline:
raise
await asyncio.sleep(pause)
pause = min(pause * 2.0, 5.0)
backend._run = patient
class _Current:
"""``async with _Current(path):`` - calls inside it must find ``path`` active."""
def __init__(self, path) -> None:
self.path = _key(path)
async def __aenter__(self):
_STATE["expect"] = self.path
return self
async def __aexit__(self, *exc):
_STATE["expect"] = None
return False
async def _attach(backend) -> str | None:
"""Attach to the running instance only, and remember the active document.
Returns the application's version, or None when nothing is running (the
backend's own ``_acad_app`` would fall back to ``Dispatch`` and launch
AutoCAD - a smoke must never do that)."""
import win32com.client
import config
from backends import com_backend as com
def _sync():
try:
app = win32com.client.GetActiveObject(config.settings.cad_progid)
except Exception:
return None
com._COM_STATE["app"] = app
_STATE["start"] = app.ActiveDocument if int(app.Documents.Count) else None
return str(app.Version)
return await backend._run(_sync)
async def _open_paths(backend) -> set[str]:
"""Full paths of the open documents that are the smoke's own (COM thread)."""
def _sync():
from backends import com_backend as com
app = com._acad_app()
paths = (str(app.Documents.Item(i).FullName) for i in range(int(app.Documents.Count)))
return {_key(p) for p in paths if _ours(p)}
return await backend._run(_sync)
async def _open(backend, path) -> str:
"""``drawing_open`` of one of the smoke's own files; returns its key."""
if not _ours(path):
raise NotOurDocument(f"refused: {Path(path).name} is not in the smoke's own folder")
result = await backend.drawing_open(str(path))
key = _key(result.get("path") or path)
if key not in _OPENED:
_OPENED.append(key)
return key
async def _close(backend, key: str) -> None:
"""Close one of the smoke's own documents, unsaved (``Close(False)``)."""
if not _ours(key):
raise NotOurDocument("refused: not one of the smoke's own documents")
if key in await _open_paths(backend):
await backend.document_close(key, discard=True)
if key in _OPENED:
_OPENED.remove(key)
async def _close_leftovers(backend) -> None:
for key in list(_OPENED):
try:
await _close(backend, key)
except Exception as exc: # best effort on the way out
print(f"could not close {Path(key).name}: {type(exc).__name__}", file=sys.stderr)
if key in _OPENED:
_OPENED.remove(key)
async def _reactivate_start(backend) -> None:
"""Hand the seat back as it was: the starting document active again."""
def _sync():
start = _STATE["start"]
if start is None:
return
try:
start.Activate()
except Exception: # it may have been closed by its owner meanwhile
pass
try:
await backend._run(_sync)
except Exception as exc:
print(f"could not re-activate the starting document: {type(exc).__name__}", file=sys.stderr)
def _model(snapshot):
"""The snapshot with its model-space records only (AutoCAD adds a paper-space
viewport when it initialises a layout; the takeoffs read model space)."""
return dataclasses.replace(
snapshot, records=tuple(r for r in snapshot.records if r.space == "Model")
)
def _types(snapshot) -> dict[str, int]:
return dict(sorted(Counter(r.type for r in _model(snapshot).records).items()))
def _check_takeoffs(truth, pid, layout) -> dict:
from engineering.understand.network import build_network
from engineering.understand.scale import scale_check
from engineering.understand.takeoff import cable_rows, pipe_rows
scale = scale_check(pid, layout)
services = tuple(sorted({run["service"] for run in truth["pipe_runs"]}))
pipe = pipe_rows(build_network(pid), pid, layout, scale=scale, services=services)
found = {(r["service"], tuple(sorted(r["tags"]))): r for r in pipe["runs"]}
pipe_mismatch = []
for run in truth["pipe_runs"]:
got = found.get((run["service"], tuple(run["tags"])))
want = run["layout_length_mm"]
have = None if got is None else got.get("layout_m")
agrees = got is not None and got["status"] == run["status"]
if want is None:
agrees = agrees and have is None
else:
agrees = agrees and have is not None and abs(float(have) * 1000.0 - want) <= 1e-6
if not agrees:
pipe_mismatch.append({"run": run["id"], "want": want, "have": have})
rows = {row["tag"]: row for row in cable_rows(pid, layout)["rows"]}
cable_mismatch = [
want["tag"]
for want in truth["cable_rows"]
if rows.get(want["tag"], {}).get("cable_m") != want["roundup_m"]
or rows.get(want["tag"], {}).get("panel") != want["panel"]
]
return {
"scale_verdict": scale["verdict"],
"within_10pct": scale.get("within_10pct"),
"runs_checked": len(truth["pipe_runs"]),
"pipe_mismatch": pipe_mismatch,
"loads_checked": len(truth["cable_rows"]),
"cable_mismatch": cable_mismatch,
}
async def _smoke(backend, folder: Path) -> dict:
from engineering.understand.diff import diff_snapshots
from engineering.understand.snapshot import read_snapshot, take_snapshot
from tests.fixtures.plant_pair import build_plant_pair
truth = build_plant_pair(folder)
pid_key = await _open(backend, truth["pid"])
async with _Current(pid_key):
started = time.perf_counter()
live_pid = await take_snapshot(backend)
export_s = time.perf_counter() - started
dwg = folder / "pid.dwg"
await backend.drawing_save_as(str(dwg), "dwg")
# SaveAs renamed the document: it is now pid.dwg, and still the smoke's own.
dwg_key = _key(dwg)
_OPENED.append(dwg_key)
for key in (pid_key, dwg_key):
await _close(backend, key)
dwg_pid = await take_snapshot(backend, str(dwg))
layout_key = await _open(backend, truth["layout"])
async with _Current(layout_key):
live_layout = await take_snapshot(backend)
takeoffs = _check_takeoffs(truth, _model(live_pid), _model(live_layout))
tag = next(
r
for r in live_layout.records
if r.type == "TEXT" and r.layer == "TAGS" and r.space == "Model"
)
async with _Current(layout_key):
await backend.entity_move(tag.handle, 10.0, 0.0)
after = await take_snapshot(backend)
diff = diff_snapshots(_model(live_layout), _model(after))
await _close(backend, layout_key)
file_pid, file_layout = read_snapshot(truth["pid"]), read_snapshot(truth["layout"])
return {
"export_s": round(export_s, 3),
"live_pid_source": live_pid.source,
"dwg_pid_source": dwg_pid.source,
"model_space": {
"pid_file": _types(file_pid),
"pid_live": _types(live_pid),
"pid_dwg": _types(dwg_pid),
"layout_file": _types(file_layout),
"layout_live": _types(live_layout),
},
"paper_space_layouts": sorted({r.space for r in live_layout.records} - {"Model"}),
"takeoffs": takeoffs,
"diff": {
"moved": tag.handle,
"added": len(diff["added"]),
"removed": len(diff["removed"]),
"changed": len(diff["changed"]),
"moved_is_the_change": f'"{tag.handle}"' in json.dumps(diff["changed"]),
},
}
async def main(keep_document: bool = False) -> int:
try:
from backends.com_backend import ComBackend
except ImportError as exc:
print(f"pywin32 not available: {exc}")
return 2
backend = ComBackend()
try:
await backend.connect()
except Exception as exc:
print(f"no live AutoCAD reachable: {exc}")
return 2
_patient(backend)
folder = Path(tempfile.mkdtemp(prefix="acadmcp_understand_smoke_"))
_STATE["folder"] = str(folder)
try:
version = await _attach(backend)
if version is None:
print("no live AutoCAD reachable: no running instance to attach to")
return 2
report = {"autocad": version, **await _smoke(backend, folder)}
print(json.dumps(report, indent=2, ensure_ascii=False))
spaces = report["model_space"]
ok = (
spaces["pid_live"] == spaces["pid_file"] == spaces["pid_dwg"]
and spaces["layout_live"] == spaces["layout_file"]
and report["takeoffs"]["scale_verdict"] == "schematic"
and not report["takeoffs"]["pipe_mismatch"]
and not report["takeoffs"]["cable_mismatch"]
and report["diff"]["changed"] == 1
and report["diff"]["added"] == report["diff"]["removed"] == 0
and report["diff"]["moved_is_the_change"]
)
return 0 if ok else 1
finally:
_STATE["expect"] = None
if not keep_document:
await _close_leftovers(backend)
await _reactivate_start(backend)
_STATE["start"] = None
await backend.disconnect()
if __name__ == "__main__":
raise SystemExit(asyncio.run(main(keep_document="--keep" in sys.argv[1:])))
+1 -1
View File
@@ -49,7 +49,7 @@ def test_correctness_suite_has_the_settings_checks():
): ):
assert CHECKS[name][1] == "Settings" assert CHECKS[name][1] == "Settings"
# v4 left the suite at 32; tracks B + G grew it to 39 (tests/test_benchmark_v5.py). # v4 left the suite at 32; tracks B + G grew it to 39 (tests/test_benchmark_v5.py).
assert len(CHECKS) == 43 assert len(CHECKS) == 48
@pytest.mark.asyncio # asyncio_mode is strict; the four tests above are sync @pytest.mark.asyncio # asyncio_mode is strict; the four tests above are sync
+1 -1
View File
@@ -48,7 +48,7 @@ def test_the_suite_carries_the_mechanical_and_sheet_checks():
assert CHECKS[name][1] == "Mechanical" assert CHECKS[name][1] == "Mechanical"
assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet" assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet"
assert CHECKS["bom_balloon_link"][1] == "Sheet" assert CHECKS["bom_balloon_link"][1] == "Sheet"
assert len(CHECKS) == 43 assert len(CHECKS) == 48
@pytest.mark.asyncio @pytest.mark.asyncio
+1 -1
View File
@@ -52,7 +52,7 @@ def test_the_suite_carries_the_four_architectural_checks():
"arch_rooms_detect_foreign", "arch_rooms_detect_foreign",
): ):
assert CHECKS[name][1] == "Architecture" assert CHECKS[name][1] == "Architecture"
assert len(CHECKS) == 43 assert len(CHECKS) == 48
@pytest.mark.asyncio @pytest.mark.asyncio
+136
View File
@@ -0,0 +1,136 @@
"""The v7 task matrix: v6 plus what track H added.
``takeoff_roundtrip`` and ``understand_foreign`` are track H's evidence, both
gated on the synthetic plant pair (tests/fixtures/plant_pair.py) whose truth
is computed from the positions it placed. Each gate is shown to fail on a
broken input - a gate that cannot fail is not evidence. The five correctness
checks run here once too, so a regression fails the suite and not only the
A/B report.
"""
from __future__ import annotations
import ezdxf
import pytest
from benchmarks.tasks_v6 import TASKS_V6
from benchmarks.tasks_v7 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V7, TASKS_V7, task_by_id
from tests.fixtures import plant_pair
UNDERSTANDING_CHECKS = (
"scale_check_detects_schematic",
"pipe_takeoff_rmst_exact",
"cable_takeoff_roundup",
"diff_detects_known_edits",
"topology_known_defects",
)
def test_v7_extends_v6_without_touching_it():
assert TASKS_V7[: len(TASKS_V6)] == TASKS_V6
assert [t.task_id for t in NEW_TASKS_V7] == ["takeoff_roundtrip", "understand_foreign"]
assert len(TASKS_V7) == 21
assert MATRICES["v6"] == TASKS_V6 and MATRICES["v7"] == TASKS_V7
assert DEFAULT_MATRIX == "v7"
assert task_by_id("takeoff_roundtrip").category == "plant"
assert task_by_id("understand_foreign").category == "understand"
def test_the_adapter_implements_every_v7_task():
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
for task in TASKS_V7:
assert hasattr(AutoCADMCPProAdapter, f"_task_{task.task_id}"), task.task_id
def test_the_suite_carries_the_five_understanding_checks():
from benchmarks.correctness_suite import CHECKS
for name in UNDERSTANDING_CHECKS:
assert CHECKS[name][1] == "Understanding"
assert len(CHECKS) == 48
async def _run(task_id, tmp_path):
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
adapter = AutoCADMCPProAdapter(backend="ezdxf")
await adapter.setup(tmp_path)
try:
await adapter._reset()
return await getattr(adapter, f"_task_{task_id}")()
finally:
await adapter.cleanup()
def _broken(edit):
"""build_plant_pair, then ``edit(truth)`` on the files it wrote."""
real = plant_pair.build_plant_pair
def build(directory):
truth = real(directory)
edit(truth)
return truth
return build
@pytest.mark.asyncio
async def test_takeoff_roundtrip_passes_headlessly(tmp_path):
passed, metrics, _artifacts = await _run("takeoff_roundtrip", tmp_path)
assert passed, metrics
assert metrics["scale_verdict"] == "schematic"
assert metrics["runs_checked"] == 9 and metrics["loads_checked"] == 4
@pytest.mark.asyncio
async def test_takeoff_roundtrip_fails_loudly_without_the_wiring_callouts(tmp_path, monkeypatch):
"""Every callout gone from the layout: no load has a panel any more, so no
cable length can be measured - and the gate names all four loads, while the
pipe half, untouched, still agrees."""
def drop_callouts(truth):
doc = ezdxf.readfile(truth["layout"])
msp = doc.modelspace()
for entity in list(msp.query("MULTILEADER")):
msp.delete_entity(entity)
doc.saveas(truth["layout"])
monkeypatch.setattr(plant_pair, "build_plant_pair", _broken(drop_callouts))
passed, metrics, _artifacts = await _run("takeoff_roundtrip", tmp_path)
assert passed is False
assert sorted(row["tag"] for row in metrics["cable_mismatch"]) == ["M11", "M12", "T102", "T202"]
assert metrics["pipe_mismatch"] == []
@pytest.mark.asyncio
async def test_understand_foreign_passes_headlessly(tmp_path):
passed, metrics, _artifacts = await _run("understand_foreign", tmp_path)
assert passed, metrics
assert metrics["clusters"] == 2 and metrics["misread_layers"] == {}
@pytest.mark.asyncio
async def test_understand_foreign_fails_loudly_when_the_units_tell_the_truth(tmp_path, monkeypatch):
"""The same layout declaring millimetres: there is no disagreement left to
flag, so the units gate fails - and only that gate."""
def honest_units(truth):
doc = ezdxf.readfile(truth["layout"])
doc.header["$INSUNITS"] = 4
doc.saveas(truth["layout"])
monkeypatch.setattr(plant_pair, "build_plant_pair", _broken(honest_units))
passed, metrics, _artifacts = await _run("understand_foreign", tmp_path)
assert passed is False
assert metrics["units_flagged"] is False
assert metrics["outlier_found"] and metrics["copies_separated"]
@pytest.mark.asyncio
@pytest.mark.parametrize("name", UNDERSTANDING_CHECKS)
async def test_each_understanding_check_passes(name):
from benchmarks.correctness_suite import CHECKS
check, _category = CHECKS[name]
assert await check() is True
+69
View File
@@ -0,0 +1,69 @@
"""scripts/field_test_takeoff.py never writes into the repository and never prints a quantity.
The script exists for client drawings, which never enter this repository
(spec §14). It is exercised here on the synthetic plant pair only - the field
test itself is never run in CI.
"""
from __future__ import annotations
import importlib.util
import json
from pathlib import Path
from tests.fixtures.plant_pair import build_plant_pair
ROOT = Path(__file__).resolve().parents[1]
def _script():
spec = importlib.util.spec_from_file_location(
"field_test_takeoff", ROOT / "scripts" / "field_test_takeoff.py"
)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
def test_an_output_folder_inside_the_repository_is_refused_before_anything_runs(tmp_path, capsys):
truth = build_plant_pair(tmp_path)
target = ROOT / "field_test_output_must_not_exist"
code = _script().main(
["--pid", truth["pid"], "--layout", truth["layout"], "--out", str(target)]
)
assert code == 2
assert not target.exists()
assert "refused" in capsys.readouterr().out
def test_a_missing_drawing_is_refused(tmp_path, capsys):
code = _script().main(
["--pid", str(tmp_path / "none.dxf"), "--layout", str(tmp_path / "none.dxf")]
)
assert code == 2
assert "does not exist" in capsys.readouterr().out
def test_the_report_lands_next_to_the_pid_and_the_console_carries_no_quantity(tmp_path, capsys):
truth = build_plant_pair(tmp_path)
code = _script().main(["--pid", truth["pid"], "--layout", truth["layout"]])
assert code == 0
summary = json.loads(capsys.readouterr().out)
assert summary["scale_verdict"] == "schematic"
assert Path(summary["report"]).parent == tmp_path
assert all(Path(path).parent == tmp_path for path in summary["workbooks"])
printed = json.dumps({k: v for k, v in summary.items() if k not in ("report", "workbooks")})
for run in truth["pipe_runs"]:
if run["layout_length_mm"] is not None:
assert f"{run['layout_length_mm']:g}" not in printed, run["id"]
for row in truth["cable_rows"]:
assert f"{row['length_mm']:g}" not in printed, row["tag"]
report = json.loads(Path(summary["report"]).read_text(encoding="utf-8"))
assert report["scale"]["verdict"] == "schematic"
def test_show_prints_the_rows_on_request(tmp_path, capsys):
truth = build_plant_pair(tmp_path)
code = _script().main(["--pid", truth["pid"], "--layout", truth["layout"], "--show"])
assert code == 0
assert '"cable"' in capsys.readouterr().out
+27
View File
@@ -190,3 +190,30 @@ def test_the_synthetic_pid_is_schematic_against_its_layout(tmp_path):
assert len(result["matched"]) >= 3 assert len(result["matched"]) >= 3
# The layout's two plan copies: the check must pick one of them. # The layout's two plan copies: the check must pick one of them.
assert result["scope"]["cluster_b"] is not None assert result["scope"]["cluster_b"] is not None
def test_a_wiring_callout_is_not_a_second_place_for_its_panel():
"""``Wiring to CP-2`` names the panel it points at; it is not written at the
panel. Read as a tag it would make every panel with a callout ambiguous
(and ``Wiring to CP1`` a phantom tag ``CP1``), so the callout takes no part
- the same rule the line-network reader (`network.tag_occurrences`) keeps."""
plan = {**PLAN, "CP-2": (3000.0, 15000.0)}
callouts = [
text("W1", "Wiring to CP-2", 12000.0, 0.0),
text("W2", "Wiring to CP1", 0.0, 9000.0),
]
result = scale_check(snapshot(tags(plan, "A")), snapshot(tags(plan, "B") + callouts))
assert result["ambiguous"] == []
assert "CP-2" in {row["tag"] for row in result["matched"]}
assert result["only_b"] == []
assert result["pair_count"] == math.comb(len(plan), 2)
def test_the_synthetic_pair_reproduces_the_generators_own_statistics(tmp_path):
"""The generator computes its truth from the positions it placed: 11 shared
tags, 55 pairs, 21.8 % within +/-10 % of the median. The check must see the
same tags - a wiring callout counted as a panel would drop to 45."""
truth = build_plant_pair(tmp_path)
result = scale_check(read_snapshot(truth["pid"]), read_snapshot(truth["layout"]))
assert result["pair_count"] == truth["scale"]["overall"]["pairs"] == 55
assert result["within_10pct"] == pytest.approx(truth["scale"]["overall"]["within_10pct"])