bench: matrix v7 adds takeoff_roundtrip and understand_foreign; five understanding checks; the live smoke and the field-test script
The evidence for track H, in the shape tracks A, E, B + G and F used.
- benchmarks/tasks_v7.py: 21 tasks, DEFAULT_MATRIX = "v7"; v6 to v2 stay
addressable. Both new tasks read the synthetic plant pair and nothing
else. takeoff_roundtrip: the P&ID called schematic, every routable run's
layout length equal to the rectilinear MST of its tags, sizes and services
as drawn (a return drawn on the supply layer included), the cable metres
rounded up after the allowance, an unstated power left empty.
understand_foreign: inches over millimetres flagged, the stray entity found
by handle, two plan copies separated, five layers classified. A test breaks
each input and watches its gate fail by name.
- Five correctness checks, every number worked out by hand in its docstring;
the published A/B report is re-run over 48 checks: 26 / 48 -> 48 / 48,
zero regressed.
- scripts/smoke_understand_com.py reads the pair through the live engine -
snapshots, the DWG route, both takeoffs and a diff - and closes every
document it opens; the CHANGELOG records what it returned.
scripts/field_test_takeoff.py runs the takeoffs on a user's own files,
refuses to write inside the repository and prints no quantity by default.
- The README token table is remeasured; check_doc_numbers derives the task
count from the v7 matrix; CLAUDE.md, both READMEs, the CHANGELOG and the
spec status line say what shipped.
One reader defect found on the way: describe.tag_occurrences read a wiring
callout ("Wiring to CP-2") as a place its panel is written, so the scale
check called CP-2 ambiguous and read "Wiring to CP1" as a phantom tag CP1 -
10 tags and 45 pairs (13.3 % within 10 %) where the generator placed 11 and
55 (21.8 %). A callout is no longer a tag occurrence, the rule
network.tag_occurrences already kept; two tests failed before the fix.
The smoke works only in documents it opened from its own temporary folder,
known by full path; it attaches with GetActiveObject only (the backend's
Dispatch fallback would launch AutoCAD), checks in the same COM-thread call
that the active document is its own before each snapshot, the DWG save and
the move, closes its documents with Close(False) and re-activates the
document that was active when it started. No new ActiveX member: Documents,
Item, FullName, ActiveDocument, Activate and Close are the ones the backend
already calls. Measured on AutoCAD 2026 (25.1s): exit 0, model-space types
equal file / live / DWG, export 0.104 s, no document of the run left open.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
835bc4b65e
commit
ba41428c88
+149
@@ -39,6 +39,16 @@ new `arch` pack, three `arch_*` critique focuses, and English or Turkish
|
|||||||
labels. No new contract member. Spec:
|
labels. No new contract member. Spec:
|
||||||
`docs/superpowers/specs/2026-09-23-v1.6-arch-design.md`.
|
`docs/superpowers/specs/2026-09-23-v1.6-arch-design.md`.
|
||||||
|
|
||||||
|
Track H of 1.6: understanding drawings somebody else made. Six tools in two
|
||||||
|
new sections — understanding and QA (26: a one-call report, a scale check
|
||||||
|
between a P&ID and its layout, a revision diff, topology checks) and plant
|
||||||
|
takeoffs (27: pipe and cable quantities whose topology comes from the P&ID
|
||||||
|
and whose lengths come from the layout) — one DXF snapshot per drawing (a
|
||||||
|
live document through `Document.Export`, never renamed), a vocabulary in five
|
||||||
|
languages, three `topo_*` critique focuses, a new `plant` pack and the
|
||||||
|
`office` extra. Read-only by default; nothing invented. No new contract
|
||||||
|
member. Spec: `docs/superpowers/specs/2026-09-24-v1.6-understand-design.md`.
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
- **Styles — 10 tools, pack `core`.** `dimstyle_list/create/modify/set_current`,
|
- **Styles — 10 tools, pack `core`.** `dimstyle_list/create/modify/set_current`,
|
||||||
@@ -139,6 +149,33 @@ labels. No new contract member. Spec:
|
|||||||
2026 (`25.1s (LMS Tech)`): the example plan drawn in one transaction, both
|
2026 (`25.1s (LMS Tech)`): the example plan drawn in one transaction, both
|
||||||
rooms read back within 0.1 % of their labels and of the hand-computed net
|
rooms read back within 0.1 % of their labels and of the hand-computed net
|
||||||
floor, and `drawing_critique(focus=None)` returned 0 issues (record below).
|
floor, and `drawing_critique(focus=None)` returned 0 issues (record below).
|
||||||
|
- **Understanding foreign drawings — 4 tools, pack `core`; plant takeoffs — 2
|
||||||
|
tools, pack `plant`.** `drawing_understand`, `drawing_scale_check`,
|
||||||
|
`drawing_diff`, `drawing_topology_check`, `pipe_takeoff`, `cable_takeoff`.
|
||||||
|
Every one reads a snapshot of frozen WCS records
|
||||||
|
(`engineering/understand/snapshot.py`); labels are read by one parser
|
||||||
|
(`%%c`, `\U+2205`, the U+2300 sign, `ø` and MTEXT format runs decoded,
|
||||||
|
Cyrillic look-alike tags folded to Latin) and layer names and texts by one
|
||||||
|
EN / TR / RU / NL / DE vocabulary. The takeoff workbook is written as CSV
|
||||||
|
always and as XLSX with the new `office` extra (`openpyxl`, also in `full`).
|
||||||
|
Lean gains `drawing_understand` and reaches 65.
|
||||||
|
- **Benchmark evidence for track H.** `benchmarks/tasks_v7.py` (21 tasks, the
|
||||||
|
default matrix; v6 to v2 stay addressable) adds `takeoff_roundtrip` and
|
||||||
|
`understand_foreign`, both on the synthetic plant pair of
|
||||||
|
`tests/fixtures/plant_pair.py`, whose truth is computed from the positions
|
||||||
|
it placed; each gate is shown to fail on a broken input. The correctness
|
||||||
|
suite grows to 48 with `scale_check_detects_schematic`,
|
||||||
|
`pipe_takeoff_rmst_exact`, `cable_takeoff_roundup`,
|
||||||
|
`diff_detects_known_edits` and `topology_known_defects`. A/B against
|
||||||
|
`v1.5.1`: **26 / 48 → 48 / 48, twenty-two `miss → pass`, zero regressed**.
|
||||||
|
- **`scripts/field_test_takeoff.py`** runs both takeoffs on a user's own P&ID
|
||||||
|
and layout and writes the report next to them; it refuses an output folder
|
||||||
|
inside the repository and prints no quantity unless `--show` is given.
|
||||||
|
- **Live COM smoke.** `scripts/smoke_understand_com.py` executed once on
|
||||||
|
AutoCAD 2026 (`25.1s (LMS Tech)`): the live snapshots matched
|
||||||
|
their files type for type in model space, the DWG route read the same
|
||||||
|
P&ID, both takeoffs matched the truth and the diff reported exactly the
|
||||||
|
one entity moved (record below).
|
||||||
|
|
||||||
- **P&ID — 9 tools, pack `pid`.**
|
- **P&ID — 9 tools, pack `pid`.**
|
||||||
- `pid_symbol_list` / `pid_symbol_insert`: a 43-symbol ISO 10628-2 /
|
- `pid_symbol_list` / `pid_symbol_insert`: a 43-symbol ISO 10628-2 /
|
||||||
@@ -318,8 +355,40 @@ labels. No new contract member. Spec:
|
|||||||
`KeyError: 'start'` recorded above as a busy-refused point read (not
|
`KeyError: 'start'` recorded above as a busy-refused point read (not
|
||||||
re-measured).
|
re-measured).
|
||||||
|
|
||||||
|
- **Track H, found after the groups landed.** Six defects in the readers,
|
||||||
|
each with a test that failed before its fix:
|
||||||
|
- A truncated DXF (an interrupted save, a partial copy) made ezdxf raise
|
||||||
|
`StopIteration`, which `asyncio.to_thread` cannot hand back to an
|
||||||
|
awaiting caller, so every track H tool reading one would have hung. The
|
||||||
|
snapshot now refuses it as "not a readable DXF file".
|
||||||
|
- `take_snapshot` of a DWG the operator already had open opened it again
|
||||||
|
read-only and closed "it" unsaved - possibly their copy, with their
|
||||||
|
unsaved work. An open document is now looked up first, exported as it
|
||||||
|
stands and left open; only a document the snapshot opened is closed.
|
||||||
|
- `drawing_diff`: a LINE drawn back to front and moved was reported as
|
||||||
|
reshaped; an INSERT carrying one attribute tag twice could lose an edit
|
||||||
|
of the first value (a repeated tag is now compared as the list of its
|
||||||
|
values); the cluster list is capped like the other lists
|
||||||
|
(`truncated["clusters"]`, the markup still clouds every cluster); and a
|
||||||
|
whole-drawing move keeps only the 32 nearest candidates per entity.
|
||||||
|
- A line number was read as an equipment tag: `100-P-001` yielded the pump
|
||||||
|
`P-001`. A token that is the tail of a word beginning with a digit is no
|
||||||
|
longer a tag.
|
||||||
|
- The pipe network merged ends and dropped stubs within `tol = 1.0`
|
||||||
|
drawing units - a metre on a metre P&ID. The tolerance now defaults to
|
||||||
|
1 mm in the P&ID's inferred unit, and a dropped stub is counted and said.
|
||||||
|
- A wiring callout was counted as a place its panel is written:
|
||||||
|
`Wiring to CP-2` made `CP-2` ambiguous in `drawing_scale_check` and
|
||||||
|
`drawing_understand`, and `Wiring to CP1` a phantom tag `CP1`. On the
|
||||||
|
synthetic pair the check matched 10 tags and 45 pairs (13.3 % within
|
||||||
|
±10 %) where the generator placed 11 and 55 (21.8 %). A callout is no
|
||||||
|
longer a tag occurrence - the rule the line-network reader already kept.
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
|
- **`MAX_DXF_BYTES` defaults to 512 MB** (was 50 MB): a real plant layout read
|
||||||
|
for track H was a 188 MB DXF. The snapshot's refusal names the file's size,
|
||||||
|
the limit and the variable; `0` still disables the check.
|
||||||
- Three tool groups appear in `system_about` and `docs/tool-inventory.json`:
|
- Three tool groups appear in `system_about` and `docs/tool-inventory.json`:
|
||||||
`styles` (10), `page_setup` (6), `environment` (22) — 204 tools in 23 groups;
|
`styles` (10), `page_setup` (6), `environment` (22) — 204 tools in 23 groups;
|
||||||
`LEAN_TOOL_NAMES` 55; capability keys `dwgprops`, `dwt_write`,
|
`LEAN_TOOL_NAMES` 55; capability keys `dwgprops`, `dwt_write`,
|
||||||
@@ -949,6 +1018,86 @@ The first run exited 1: the first room label was refused because the room
|
|||||||
reader found 0 wall segments on `A-WALL-E-N` — the `entity_list` narrowing
|
reader found 0 wall segments on `A-WALL-E-N` — the `entity_list` narrowing
|
||||||
listed under **Fixed**. The run above is the one after the fix.
|
listed under **Fixed**. The run above is the one after the fix.
|
||||||
|
|
||||||
|
### Live COM smoke — understanding (track H)
|
||||||
|
|
||||||
|
`scripts/smoke_understand_com.py`, AutoCAD 2026, Windows 11, on the synthetic
|
||||||
|
plant pair only — `take_snapshot` of `pid.dxf` opened live, the same P&ID
|
||||||
|
saved as DWG and read back through the read-only DWG route, `take_snapshot`
|
||||||
|
of `layout.dxf`, `pipe_rows` / `cable_rows` on the live snapshots against the
|
||||||
|
pair's truth, then one tag moved 10 mm and `diff_snapshots` of the layout
|
||||||
|
before and after, all through the COM engine, exit 0, every document closed:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"autocad": "25.1s (LMS Tech)",
|
||||||
|
"export_s": 0.104,
|
||||||
|
"live_pid_source": "live:pid.dxf",
|
||||||
|
"dwg_pid_source": "live:pid.dwg",
|
||||||
|
"model_space": {
|
||||||
|
"pid_file": {
|
||||||
|
"ARC": 1,
|
||||||
|
"INSERT": 15,
|
||||||
|
"LINE": 8,
|
||||||
|
"LWPOLYLINE": 8,
|
||||||
|
"MTEXT": 8,
|
||||||
|
"TEXT": 31
|
||||||
|
},
|
||||||
|
"pid_live": {
|
||||||
|
"ARC": 1,
|
||||||
|
"INSERT": 15,
|
||||||
|
"LINE": 8,
|
||||||
|
"LWPOLYLINE": 8,
|
||||||
|
"MTEXT": 8,
|
||||||
|
"TEXT": 31
|
||||||
|
},
|
||||||
|
"pid_dwg": {
|
||||||
|
"ARC": 1,
|
||||||
|
"INSERT": 15,
|
||||||
|
"LINE": 8,
|
||||||
|
"LWPOLYLINE": 8,
|
||||||
|
"MTEXT": 8,
|
||||||
|
"TEXT": 31
|
||||||
|
},
|
||||||
|
"layout_file": {
|
||||||
|
"CIRCLE": 18,
|
||||||
|
"LINE": 25,
|
||||||
|
"LWPOLYLINE": 4,
|
||||||
|
"MULTILEADER": 8,
|
||||||
|
"TEXT": 26
|
||||||
|
},
|
||||||
|
"layout_live": {
|
||||||
|
"CIRCLE": 18,
|
||||||
|
"LINE": 25,
|
||||||
|
"LWPOLYLINE": 4,
|
||||||
|
"MULTILEADER": 8,
|
||||||
|
"TEXT": 26
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"paper_space_layouts": [],
|
||||||
|
"takeoffs": {
|
||||||
|
"scale_verdict": "schematic",
|
||||||
|
"within_10pct": 0.21818181818181817,
|
||||||
|
"runs_checked": 9,
|
||||||
|
"pipe_mismatch": [],
|
||||||
|
"loads_checked": 4,
|
||||||
|
"cable_mismatch": []
|
||||||
|
},
|
||||||
|
"diff": {
|
||||||
|
"moved": "44",
|
||||||
|
"added": 0,
|
||||||
|
"removed": 0,
|
||||||
|
"changed": 1,
|
||||||
|
"moved_is_the_change": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The smoke works only in documents it brought in from its own temporary
|
||||||
|
folder, known by full path; each live snapshot, the DWG save and the move
|
||||||
|
first check, in the same COM-thread call, that the active document is the
|
||||||
|
smoke's own, and on the way out it re-activates the document that was active
|
||||||
|
when it started.
|
||||||
|
|
||||||
## [1.5.1] — 2026-08-06
|
## [1.5.1] — 2026-08-06
|
||||||
|
|
||||||
A patch for three defects found *after* 1.5.0 went to PyPI, two of them by the
|
A patch for three defects found *after* 1.5.0 went to PyPI, two of them by the
|
||||||
|
|||||||
@@ -33,6 +33,9 @@ pip install -e ".[com]"
|
|||||||
# With PDF/screenshot support via matplotlib
|
# With PDF/screenshot support via matplotlib
|
||||||
pip install -e ".[pdf]"
|
pip install -e ".[pdf]"
|
||||||
|
|
||||||
|
# XLSX workbooks (takeoffs, data_extract) via openpyxl
|
||||||
|
pip install -e ".[office]"
|
||||||
|
|
||||||
# Everything
|
# Everything
|
||||||
pip install -e ".[full]"
|
pip install -e ".[full]"
|
||||||
```
|
```
|
||||||
@@ -97,10 +100,12 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
|||||||
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
|
23. Mechanical annotation (5 tools, pack `mech`): `surface_texture` (ISO 21920-1, ISO 1302's older Ra form accepted), `weld_symbol` (ISO 2553), `centre_marks` (ISO 128-23), `section_line` (ISO 128-40 — it returns the plane `mech_view_add(kind="section")` consumes, so the label and the view cannot disagree), `hatch_material` (the ISO 128-50 map; an unknown material is refused with the list, never hatched as steel).
|
||||||
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
|
24. Sheet and delivery (11 tools, pack `core`): `sheet_frame` (ISO 5457 A4-A0 — 20 mm filing margin, 10 mm elsewhere on A3, zone grid, centring and trimming marks), `titleblock_apply` (ISO 7200 for every size; `titleblock_apply_iso_a3` is now a thin alias), `revision_add`, `bom_extract` (never modifies the drawing), `bom_table` (ISO 7573), `balloon_add` (ISO 6433, linked by XDATA so a rerun renumbers instead of duplicating), `data_extract` (CSV always; XLSX refused with `xlsx_write` when `openpyxl` is absent), `xref_attach`, `xref_manage`, `image_attach`, `drawing_export_dwg` (COM `SaveAs` natively; headless only with the ODA File Converter, otherwise `dwg_write` refuses and names the install route). The COM paths of SECTIONS 21-24 were executed live once by `scripts/smoke_mech_com.py` (AutoCAD 2026): part, section, dimensions, an ISO 4014 block with its XDATA, the parts list read off the drawing, a linked balloon, an xref attached and detached, a real `.dwg`, and `drawing_critique(focus=None)` clean.
|
||||||
25. Architecture (12 tools, pack `arch`): `arch_wall`, `arch_opening`, `arch_stair`, `arch_room`, `arch_rooms_detect`, `arch_schedule`, `arch_grid`, `arch_symbol`, `arch_catalogue_list`, `arch_catalogue_insert`, `arch_dimension_chains`, `arch_plan_from_spec`. One plan model — a wall network (axis polyline, thickness, justification, material) with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, cut faces hatched by material (`engineering/arch/materials.py`: ISO 128-50 general hatching for brick, the JIS A 0150 RC / stone / wood symbols, AutoCAD's AR-CONC / AR-SAND, AAC as a declared convention), and anything it cannot place listed in `omitted: [{element, reason}]`, never dropped. Rooms are *computed*: a label's area is the planar face its point lies in, and `arch_rooms_detect` reads rooms from any plan (confidence 1.0 for our walls, 0.6 for plain lines) without modifying it. Schedules are a read of the `ACADMCP_ARCH` records, drawn as real TABLEs. Labels and schedules speak English, or Turkish with `lang="tr"` (decimal comma). The COM paths were executed live once by `scripts/smoke_arch_com.py` (AutoCAD 2026).
|
25. Architecture (12 tools, pack `arch`): `arch_wall`, `arch_opening`, `arch_stair`, `arch_room`, `arch_rooms_detect`, `arch_schedule`, `arch_grid`, `arch_symbol`, `arch_catalogue_list`, `arch_catalogue_insert`, `arch_dimension_chains`, `arch_plan_from_spec`. One plan model — a wall network (axis polyline, thickness, justification, material) with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, cut faces hatched by material (`engineering/arch/materials.py`: ISO 128-50 general hatching for brick, the JIS A 0150 RC / stone / wood symbols, AutoCAD's AR-CONC / AR-SAND, AAC as a declared convention), and anything it cannot place listed in `omitted: [{element, reason}]`, never dropped. Rooms are *computed*: a label's area is the planar face its point lies in, and `arch_rooms_detect` reads rooms from any plan (confidence 1.0 for our walls, 0.6 for plain lines) without modifying it. Schedules are a read of the `ACADMCP_ARCH` records, drawn as real TABLEs. Labels and schedules speak English, or Turkish with `lang="tr"` (decimal comma). The COM paths were executed live once by `scripts/smoke_arch_com.py` (AutoCAD 2026).
|
||||||
|
26. Understanding & QA (4 tools, pack `core`): `drawing_understand` (one read-only report of a drawing somebody else made — declared vs inferred units with the evidence, declared vs robust extents and the outlier handles behind them, clusters (sheets, plan copies), every layer's likely discipline and service from an EN / TR / RU / NL / DE vocabulary with a confidence and the deciding keyword, equipment tags, rooms, text languages, blocks, layouts, title-block attributes), `drawing_scale_check` (a drawing against a reference: tags unique within a cluster are matched — a tag seen twice is `ambiguous`, never picked — pairs at least 2 m apart, the median ratio, IQR and the share within ±10 %: `to_scale` at 80 % or more, with its factor; `schematic` at 50 % or less; `partly` between), `drawing_diff` (two revisions matched by signature, then handle, then type + layer + nearest position; `markup=True` draws revision clouds on the newer one — its only write) and `drawing_topology_check` (dangling ends, near misses, interior crossings; the three `topo_*` critique focuses run the same engine under `focus=None` on layers no semantic focus covers). Every one reads one snapshot per drawing (`engineering/understand/snapshot.py`: `ezdxf` directly, or on a live seat `Document.Export(<tmp>/snap, "DXF", ...)`, which never renames the document and carries model space, paper space and block definitions — measured).
|
||||||
|
27. Plant takeoffs (2 tools, pack `plant`): `pipe_takeoff` (topology from the P&ID — segments merged at their ends, split at T-junctions, continued through inline fittings, a double-drawn segment counted once; service from the layer overridden by supply / return words; sizes kept as drawn — `Ø51`, `SMS51`, `DN20` are never equated — direct, by continuity or `unassigned`; lengths from the layout: the rectilinear MST of the tags a run connects, labelled an estimate, the P&ID length kept beside it as a comparison; statuses A / B / C) and `cable_takeoff` (one row per load: power from the P&ID's electrical texts, never invented; the panel from the layout's `Wiring to CPn` callouts; the Manhattan length plus the allowance rounded up to the metre; a section only from a caller's `section_rules`). Both write a workbook (Metraj / Özet / Kontrol / Metodoloji, `lang` tr / en / ru, XLSX with the `office` extra, CSV always) and never the drawing. `scripts/field_test_takeoff.py` runs both on a user's own pair and writes next to it, never into this repository. The COM paths were executed live once by `scripts/smoke_understand_com.py` (AutoCAD 2026).
|
||||||
|
|
||||||
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 64 curated drafting tools (54 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
|
**Tool profiles:** `TOOL_PROFILE=lean|full` (default `full`) controls the advertised surface — `lean` = 65 curated drafting tools (55 with `TOOL_PACKS=core`, which drops the three `pid_*`, the three `mech_*` and the four `arch_*` lean members, and keeps `drawing_understand`; the five settings essentials — `drawing_apply_standard`, `dimstyle_set_current`, `textstyle_set_current`, `page_setup_apply`, `batch_plot` — are core-pack tools and stay; including `cad_batch`, since a client with a tight tool cap is exactly the client paying most per turn). Applied in the lifespan; reported by `system_about`. The `core` profile was removed in v1.5.0 (the discovery layer replaced it); `TOOL_PROFILE=core` falls back to `full` with a warning.
|
||||||
|
|
||||||
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools; SECTION 18/19/24 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
|
**Tool packs:** `TOOL_PACKS=all|core,pid,settings,mech,arch,plant` (default `all`) advertises only the vertical packs a client uses; `core` is always on and is everything no other pack claims; `settings` is the 22 SECTION 20 environment tools, `pid` the nine P&ID tools, `mech` the 14 SECTION 21-23 mechanical tools, `arch` the 12 SECTION 25 architectural tools, `plant` the two SECTION 27 takeoffs (track D's Plant 3D readers join it); SECTION 18/19/24/26 are core; unknown names are ignored with a warning; a `lean` profile intersects with the enabled packs. `system_about` reports `tool_packs`.
|
||||||
|
|
||||||
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
|
**Tool discovery:** `DISCOVERY_MODE=off|search` (default `off`). `search` replaces the advertised catalog with `search_tools` + `call_tool`, ranking hits over an AutoCAD command / synonym corpus (`discovery/aliases.py`) layered on fastmcp's BM25 index.
|
||||||
|
|
||||||
@@ -112,7 +117,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
|||||||
|
|
||||||
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
|
**Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so.
|
||||||
|
|
||||||
**Benchmark matrix:** `benchmarks/tasks_v6.py` is the current set (19 tasks; `--matrix v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (43 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
|
**Benchmark matrix:** `benchmarks/tasks_v7.py` is the current set (21 tasks; `--matrix v6` / `v5` / `v4` / `v3` / `v2` reproduce the earlier sets). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks, the v4 `pid_roundtrip` / `page_setup_truth`, the v5 `mech_assembly` (a flange-coupling sheet gated on a zero-issue `drawing_critique(focus=None)` and a finalize score of at least 90) or the v6 `arch_roundtrip` (a two-room plan drawn by `arch_plan_from_spec`, its rooms read back by `arch_rooms_detect` within 0.1 % of the label and of the hand-computed net floor, the schedules listing the drawn tags, gated like `mech_assembly`) or the v7 `takeoff_roundtrip` / `understand_foreign` (the synthetic plant pair of `tests/fixtures/plant_pair.py` read without its truth: the P&ID called schematic, every routable run's layout length equal to the rectilinear MST of its tags, the cable metres rounded up after the allowance; units, the outlier, both plan copies and the service layers found by one `describe`) — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (48 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.5.1` (the published report is `benchmarks/results/published/ab-v1.5.1-vs-v1.6.0-dev.json`; `scripts/check_doc_numbers.py` names it in `AB_REPORT`).
|
||||||
|
|
||||||
### Adding a New Tool
|
### Adding a New Tool
|
||||||
|
|
||||||
@@ -139,7 +144,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
|||||||
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
|
- **Engineering layer scaffold**: `drawing_new` auto-bootstraps standard linetypes (CENTER, HIDDEN, PHANTOM) and engineering layers (GEOMETRY, DIM, CENTER, HIDDEN, PHANTOM, HATCH, TEXT, TITLEBLOCK). Pass `bootstrap=False` to opt out.
|
||||||
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
|
- **Generic contracts added in 1.6:** `block_define(name, entities, attdefs)` (typed primitives + ATTDEFs, both engines; `overwrite` replaces contents so INSERTs keep the name) and `entity_get/set_xdata` (typed by group code; 255-char strings, 16 KB per entity, enforced before writing).
|
||||||
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
|
- **P&ID reader rules:** `pid_graph` never modifies the drawing, reads geometry from the drawing (an INSERT's real rotation/scale) rather than the catalogue, and reports `source`/`confidence` per node (catalog 1.0, xdata 0.95, heuristic 0.6, inferred 0.3) — never raised by agreement between heuristics. Read `stats.confidence_min` before trusting a foreign drawing.
|
||||||
- **`TOOL_PACKS=all|core,pid,settings,mech,arch`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
|
- **`TOOL_PACKS=all|core,pid,settings,mech,arch,plant`** advertises only the packs a client uses; `core` is always on; `system_about` reports `tool_packs`.
|
||||||
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
|
- **Layer states are ours, not AutoCAD's.** `layer_state_*` stores a portable snapshot in an `ACADMCP_LAYERSTATES` XRECORD; it survives save/reopen on both engines and travels with the file, and it is not listed in AutoCAD's Layer States Manager. Said in the tool docstring, in `system_capabilities.layer_states` (`mode: "xrecord"`, reason `portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager`) and in the README.
|
||||||
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
|
- **Page-setup evidence is the PDF.** `batch_plot` reports `mediabox_mm` parsed from the file's `/MediaBox` (`engineering/standards/pdfinfo.py`); a page setup is verified by what a printer would get, not by the setter's return value. A style or setting write reports `changed` — re-setting a value to itself is not a change.
|
||||||
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
|
- **Nothing interprets coordinates in UCS.** `ucs_set` / `ucs_restore` store and activate a UCS for the operator's benefit; every tool input and output stays WCS (the repository rule above).
|
||||||
@@ -153,6 +158,11 @@ Tools are organized into sections (counts are indicative — `system_about` is a
|
|||||||
- **Plan text follows the plot scale**: `scale` is a denominator (50 = 1:50) and a paper height `h` is drawn at `h × scale` drawing units — no annotative objects. `lang="tr"` switches words and the decimal separator; any other language is refused with the list.
|
- **Plan text follows the plot scale**: `scale` is a denominator (50 = 1:50) and a paper height `h` is drawn at `h × scale` drawing units — no annotative objects. `lang="tr"` switches words and the decimal separator; any other language is refused with the list.
|
||||||
- **Three `arch_*` critique focuses** (`arch_room_unlabelled`, `arch_wall_gap` — two wall ends within 50 mm that do not join, `arch_opening_clash`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_ARCH` record.
|
- **Three `arch_*` critique focuses** (`arch_room_unlabelled`, `arch_wall_gap` — two wall ends within 50 mm that do not join, `arch_opening_clash`) run under `focus=None` and inside `drawing_finalize`; they return nothing on a drawing with no `ACADMCP_ARCH` record.
|
||||||
- **Nominal, not standard**: the furniture and sanitary blocks carry nominal catalogue dimensions and say so; the stair tool reports the Blondel relation (2R + G within 600–650 mm) as a design rule, never as a building code; the `arch` layer colours are a repository convention (ISO 13567 fixes names, not colours).
|
- **Nominal, not standard**: the furniture and sanitary blocks carry nominal catalogue dimensions and say so; the stair tool reports the Blondel relation (2R + G within 600–650 mm) as a design rule, never as a building code; the `arch` layer colours are a repository convention (ISO 13567 fixes names, not colours).
|
||||||
|
- **Read-only by default (track H)**: `drawing_understand`, `drawing_scale_check`, `drawing_topology_check` and both takeoffs never modify the drawing; `drawing_diff` writes only with `markup=True`, and a takeoff writes a workbook file, never the drawing.
|
||||||
|
- **Lengths come from the inferred unit**: a reader measures in the unit the geometry implies (text heights, wall and door sizes, overall size) and reports metres; a disagreement with `INSUNITS` is a warning with the numbers — a layout declaring inches over millimetre geometry is measured in millimetres, never ×25.4.
|
||||||
|
- **Scale check before a P&ID length**: a P&ID is a length source only when `drawing_scale_check` calls it `to_scale`; otherwise a run's length is the orthogonal estimate on the layout and the drawn length stays a comparison column (`length_source="pid"` on a schematic P&ID is refused unless `force=True`, quoting the statistics).
|
||||||
|
- **Nothing is invented (track H)**: a power the P&ID does not state stays empty and becomes an open item, an unreadable size stays `unassigned`, and no cable-section table is authored — a section comes only from the caller's `section_rules`.
|
||||||
|
- **Client drawings never enter the repository**: every track-H fixture is the synthetic plant pair (`tests/fixtures/plant_pair.py`), whose truth is computed from the positions it placed; `scripts/field_test_takeoff.py` refuses an output folder inside the repository and prints no quantity unless `--show` is given.
|
||||||
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
|
- **Production drawings**: For real engineering output, use the `engineering/` package primitives via the `gear_*` / `keyway_*` / `titleblock_*` MCP tools — do NOT hand-draw teeth/keyways/sections with raw `entity_create_*` calls. Always end with `drawing_finalize` for the 8-step validator.
|
||||||
|
|
||||||
### Premium Drawing Rules
|
### Premium Drawing Rules
|
||||||
|
|||||||
@@ -27,24 +27,24 @@ Live through COM on Windows, or headless through ezdxf anywhere — one typed co
|
|||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
> **v1.5 release snapshot:** 247 tools · 8 resources · 5 prompt templates · 4942 collected tests.
|
> **v1.5 release snapshot:** 247 tools · 8 resources · 5 prompt templates · 4960 collected tests.
|
||||||
> 247 is the **registered** count; a default install advertises 242 over `tools/list`,
|
> 247 is the **registered** count; a default install advertises 242 over `tools/list`,
|
||||||
> because `ENABLE_3D` is unset. `system_about` is the runtime authority.
|
> because `ENABLE_3D` is unset. `system_about` is the runtime authority.
|
||||||
|
|
||||||
## Why this exists
|
## Why this exists
|
||||||
|
|
||||||
**A big MCP server is expensive to be connected to.** The full catalog costs a client **78,772 tokens** before it has asked for anything. Discovery mode replaces it with two tools and costs **356**.
|
**A big MCP server is expensive to be connected to.** The full catalog costs a client **83,758 tokens** before it has asked for anything. Discovery mode replaces it with two tools and costs **356**.
|
||||||
|
|
||||||
**A drafter searches for `FILLET`, not `entity_fillet`.** Those command names appeared in no tool name or description — `df = 0` against a stock index, not badly ranked but *absent*. The fix was data: an authored corpus of **206 AutoCAD command names and 1439 synonym phrases** covering all 247 tools. A test refuses to let a tool exist without one.
|
**A drafter searches for `FILLET`, not `entity_fillet`.** Those command names appeared in no tool name or description — `df = 0` against a stock index, not badly ranked but *absent*. The fix was data: an authored corpus of **206 AutoCAD command names and 1439 synonym phrases** covering all 247 tools. A test refuses to let a tool exist without one.
|
||||||
|
|
||||||
| Advertised surface | Tools seen | Idle cost |
|
| Advertised surface | Tools seen | Idle cost |
|
||||||
|---|---:|---:|
|
|---|---:|---:|
|
||||||
| `TOOL_PROFILE=full` (default) | 236 | 78,772 tokens |
|
| `TOOL_PROFILE=full` (default) | 242 | 83,758 tokens |
|
||||||
| `TOOL_PACKS=core,settings` (full profile) | 201 | 60,926 tokens |
|
| `TOOL_PACKS=core,settings` (full profile) | 205 | 63,890 tokens |
|
||||||
| `TOOL_PACKS=core,mech` (full profile) | 193 | 61,872 tokens |
|
| `TOOL_PACKS=core,mech` (full profile) | 197 | 64,836 tokens |
|
||||||
| `TOOL_PACKS=core,arch` (full profile) | 191 | 60,568 tokens |
|
| `TOOL_PACKS=core,arch` (full profile) | 195 | 63,532 tokens |
|
||||||
| `TOOL_PACKS=core` (full profile) | 179 | 54,158 tokens |
|
| `TOOL_PACKS=core` (full profile) | 183 | 57,122 tokens |
|
||||||
| `TOOL_PROFILE=lean` | 64 | 21,939 tokens |
|
| `TOOL_PROFILE=lean` | 65 | 22,661 tokens |
|
||||||
| `DISCOVERY_MODE=search` | 2 | **356 tokens** |
|
| `DISCOVERY_MODE=search` | 2 | **356 tokens** |
|
||||||
|
|
||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
@@ -106,6 +106,7 @@ Claude Desktop, Cursor, or any stdio MCP host. For HTTP: `autocad-mcp --transpor
|
|||||||
| Mechanical parts | one part model (segments or outline + typed features) and one view engine: front / side / top / section / detail with hidden lines, ISO 128-50 cut faces and ISO 129 dimensions; ISO 4014/4017/4032/7089/4762 fasteners, ISO 15 bearings, DIN 471/472, DIN 509, DIN 332 and ISO 3601-2 as real blocks with attributes; ISO 21920-1 surface texture, ISO 2553 welds, ISO 128-40 section lines |
|
| Mechanical parts | one part model (segments or outline + typed features) and one view engine: front / side / top / section / detail with hidden lines, ISO 128-50 cut faces and ISO 129 dimensions; ISO 4014/4017/4032/7089/4762 fasteners, ISO 15 bearings, DIN 471/472, DIN 509, DIN 332 and ISO 3601-2 as real blocks with attributes; ISO 21920-1 surface texture, ISO 2553 welds, ISO 128-40 section lines |
|
||||||
| Sheet & delivery | ISO 5457 frames A4-A0 with zones and trim marks, ISO 7200 title blocks for every size, revision blocks with clouds, ISO 7573 parts lists with ISO 6433 balloons linked by XDATA, CSV/XLSX extraction, xrefs and images on both engines, real `.dwg` on a live seat |
|
| Sheet & delivery | ISO 5457 frames A4-A0 with zones and trim marks, ISO 7200 title blocks for every size, revision blocks with clouds, ISO 7573 parts lists with ISO 6433 balloons linked by XDATA, CSV/XLSX extraction, xrefs and images on both engines, real `.dwg` on a live seat |
|
||||||
| Architecture | one plan model — walls with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, poché by material; rooms measured from the faces the walls enclose (never typed) and read back from any plan with a confidence; door / window / room schedules as real TABLEs; a furniture and sanitary catalogue at nominal sizes; structural grid, north arrow, section / level / elevation marks; exterior dimension chains; English or Turkish labels (`lang="tr"`, decimal comma) |
|
| Architecture | one plan model — walls with hosted doors and windows — and one engine: L / T / X junctions resolved, openings cut, poché by material; rooms measured from the faces the walls enclose (never typed) and read back from any plan with a confidence; door / window / room schedules as real TABLEs; a furniture and sanitary catalogue at nominal sizes; structural grid, north arrow, section / level / elevation marks; exterior dimension chains; English or Turkish labels (`lang="tr"`, decimal comma) |
|
||||||
|
| Understanding & takeoffs | `drawing_understand` reads a drawing somebody else made in one call — declared vs inferred units, robust extents and the outlier behind them, plan copies, each layer's discipline and service in five languages, tags, rooms; `drawing_scale_check` says whether a P&ID is to scale against its layout; `pipe_takeoff` / `cable_takeoff` take the topology from the P&ID and the lengths from the layout (rectilinear MST, statuses A/B/C, power never invented) into a Turkish / English / Russian workbook; `drawing_diff` between revisions; `drawing_topology_check` for dangling ends, near misses and crossings |
|
||||||
| P&ID | catalogue blocks with ports and tags (ISO 10628-2 / ISA-5.1), port-to-port lines with ISA-5.1 classes and line numbers, `pid_graph` reads any P&ID back with confidence, instrument index / line list / equipment list, `pid_from_spec` one-call sheets |
|
| P&ID | catalogue blocks with ports and tags (ISO 10628-2 / ISA-5.1), port-to-port lines with ISA-5.1 classes and line numbers, `pid_graph` reads any P&ID back with confidence, instrument index / line list / equipment list, `pid_from_spec` one-call sheets |
|
||||||
| Styles & standards | ISO-25 / ANSI dimension styles, ISOCP / ROMANS text styles, ISO / ANSI leader styles from authored presets; `drawing_apply_standard("iso")` sets all of it plus units and the `mech` layers in one call; `changed` reports only what moved |
|
| Styles & standards | ISO-25 / ANSI dimension styles, ISOCP / ROMANS text styles, ISO / ANSI leader styles from authored presets; `drawing_apply_standard("iso")` sets all of it plus units and the `mech` layers in one call; `changed` reports only what moved |
|
||||||
| Discovery | `search_tools` ranked over an AutoCAD command and synonym corpus — `FILLET`, `BPOLY`, `QSELECT`, `WBLOCK`, `OVERKILL`, `CHSPACE` each rank **#1** of the 242-tool advertised catalog |
|
| Discovery | `search_tools` ranked over an AutoCAD command and synonym corpus — `FILLET`, `BPOLY`, `QSELECT`, `WBLOCK`, `OVERKILL`, `CHSPACE` each rank **#1** of the 242-tool advertised catalog |
|
||||||
@@ -162,23 +163,23 @@ Self-measurement, produced by scripts in [`benchmarks/`](https://github.com/U-C4
|
|||||||
|
|
||||||
### Correctness — every release re-proves itself
|
### Correctness — every release re-proves itself
|
||||||
|
|
||||||
43 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.
|
48 deterministic headless checks against the previous tag and the current tree, each in its own subprocess so a hard crash counts as a miss rather than killing the run.
|
||||||
|
|
||||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||||
|---|---:|---:|---:|---:|
|
|---|---:|---:|---:|---:|
|
||||||
| v1.5.1 *(baseline)* | 26 / 43 | 60.5 % | — | — |
|
| v1.5.1 *(baseline)* | 26 / 48 | 54.2 % | — | — |
|
||||||
| **v1.6.0-dev** *(this branch)* | **43 / 43** | **100 %** | 17 | **0** |
|
| **v1.6.0-dev** *(this branch)* | **48 / 48** | **100 %** | 22 | **0** |
|
||||||
|
|
||||||
The seventeen fixed rows are new capability (`miss → pass`). Track F adds four: `arch_junction_l_t_x` (an L corner mitres at (4100, −100) / (3900, 100), a T stem stops on the near face at (2950, 100) / (3050, 100), an X cuts all four faces, and the wall areas are 1 400 000, 1 490 000 and 2 360 000 mm² — computed by hand), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls measures 20.00 m² in its label and its record, not the 21.84 m² its axes enclose), `arch_opening_cuts_wall` (both faces are interrupted across a 900 door and two jambs close it) and `arch_rooms_detect_foreign` (a plan of plain lines on a `WALLS` layer yields 20 and 15 m² at confidence 0.6, and nothing is written). Tracks B and G add seven: `mech_part_roundtrip` (a part read back out of its own `ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a 60 x ⌀40 sleeve with a ⌀20 bore cuts 1200 mm² exactly, measured out of the drawing by `analysis_measure_entity`), `mech_iso286_on_dimension` (40 H7 is +0.025 / 0 and survives the layout), `mech_thread_unrepresented_is_caught` (the ISO 6410 focus fires on a thread drawn as a plain circle — a gate that never fires is not a gate), `std_part_iso4014_m12` (s = 18, k = 7.5, read by value), `sheet_frame_iso5457_a3` (the frame runs 20,10 to 410,287 inside the 420 × 297 sheet) and `bom_balloon_link` (two identical bolts are one row of quantity 2, and its balloon carries the row's item number). The other six are the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.
|
The twenty-two fixed rows are new capability (`miss → pass`). Track H adds five: `scale_check_detects_schematic` (the synthetic P&ID reads **schematic** against its layout, and a uniform 2× copy of six tags reads **to scale** with factor 2), `pipe_takeoff_rmst_exact` (all eight routable runs of the pair measure exactly the rectilinear MST of their tags), `cable_takeoff_roundup` (9, 17, 19 and 10 m after the 20 % allowance, and the heater's unstated power stays empty), `diff_detects_known_edits` (a move, a text change, an attribute change, an add and a delete — and nothing else) and `topology_known_defects` (a 5 mm near miss, one interior crossing, and a clean T that is neither). Track F adds four: `arch_junction_l_t_x` (an L corner mitres at (4100, −100) / (3900, 100), a T stem stops on the near face at (2950, 100) / (3050, 100), an X cuts all four faces, and the wall areas are 1 400 000, 1 490 000 and 2 360 000 mm² — computed by hand), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls measures 20.00 m² in its label and its record, not the 21.84 m² its axes enclose), `arch_opening_cuts_wall` (both faces are interrupted across a 900 door and two jambs close it) and `arch_rooms_detect_foreign` (a plan of plain lines on a `WALLS` layer yields 20 and 15 m² at confidence 0.6, and nothing is written). Tracks B and G add seven: `mech_part_roundtrip` (a part read back out of its own `ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a 60 x ⌀40 sleeve with a ⌀20 bore cuts 1200 mm² exactly, measured out of the drawing by `analysis_measure_entity`), `mech_iso286_on_dimension` (40 H7 is +0.025 / 0 and survives the layout), `mech_thread_unrepresented_is_caught` (the ISO 6410 focus fires on a thread drawn as a plain circle — a gate that never fires is not a gate), `std_part_iso4014_m12` (s = 18, k = 7.5, read by value), `sheet_frame_iso5457_a3` (the frame runs 20,10 to 410,287 inside the 420 × 297 sheet) and `bom_balloon_link` (two identical bolts are one row of quantity 2, and its balloon carries the row's item number). The other six are the three P&ID checks of track A and, from track E, `settings_dimstyle_iso25_values` (the ISO-25 preset lands in the DIMSTYLE table with ISO 129-1's numbers), `settings_layer_state_roundtrip` (save → change → restore puts the layer table back, and the state survives save/reopen because it is an XRECORD in the file) and `settings_pdf_mediabox_a3` (the plotted PDF's own `/MediaBox` reads 420 × 297). The 26 checks v1.5.1 was gated on all still pass. Against the older `v1.4.0` baseline the same 26 reported **21 / 26 → 26 / 26, five fixed, zero regressed** — two of those were repaired defects (`fail → pass`), the diameter and radius callouts, which measured the leader as geometry and dimensioned a 40 mm bore as 60 at default settings.
|
||||||
|
|
||||||
### The task matrix — tasks that can fail
|
### The task matrix — tasks that can fail
|
||||||
|
|
||||||
An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds four more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness, one where a whole mechanical sheet has to come out clean, and one where a floor plan's rooms are read back off the drawing.
|
An earlier matrix scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five were added in 1.5 because they *can* fail, and three did while being written; 1.6 adds six more — one the P&ID reader can fail on its own, one where the PDF file, not the setter, is the witness, one where a whole mechanical sheet has to come out clean, one where a floor plan's rooms are read back off the drawing, and two where a foreign plant's P&ID and layout are read: the takeoffs against lengths computed in advance, and the one-call report against the defects planted in it.
|
||||||
|
|
||||||
| Task | Verified against |
|
| Task | Verified against |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `tool_discovery` | six AutoCAD command names, each ranking #1 |
|
| `tool_discovery` | six AutoCAD command names, each ranking #1 |
|
||||||
| `token_budget` | 78,772 → 356 tokens, against a ceiling fixed in advance |
|
| `token_budget` | 83,758 → 356 tokens, against a ceiling fixed in advance |
|
||||||
| `hatch_islands` | 300 filled with the island, 400 ignoring it |
|
| `hatch_islands` | 300 filled with the island, 400 ignoring it |
|
||||||
| `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 |
|
| `selection_filter` | window 1, crossing 2, bounding box 3, polygon 1 |
|
||||||
| `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives |
|
| `measure_from_handle` | 139.2699 against the 100.0 a vertex shoelace gives |
|
||||||
@@ -186,6 +187,8 @@ An earlier matrix scored this server 10/10, which carried no information: every
|
|||||||
| `page_setup_truth` | ANSI B then ISO A3 landscape applied to Layout1, each plotted through `batch_plot` and read back from its PDF's `/MediaBox`: 432 × 279 then 420 × 297 mm, not the setter's return value (a fresh layout is already A3, so the B sheet is what a no-op setter cannot fake) |
|
| `page_setup_truth` | ANSI B then ISO A3 landscape applied to Layout1, each plotted through `batch_plot` and read back from its PDF's `/MediaBox`: 432 × 279 then 420 × 297 mm, not the setter's return value (a fresh layout is already A3, so the B sheet is what a no-op setter cannot fake) |
|
||||||
| `mech_assembly` | a flange-coupling A3 sheet — hub with a full section, flange with its bolt-circle end view, two ISO 4014 bolts with ISO 4032 nuts, an ISO 7573 parts list read off the drawing, linked balloons, ISO 5457 frame, ISO 7200 title block — gated on `drawing_critique(focus=None)` returning **zero** issues and a finalize score of at least **90**; a test drops one balloon and watches the gate fail |
|
| `mech_assembly` | a flange-coupling A3 sheet — hub with a full section, flange with its bolt-circle end view, two ISO 4014 bolts with ISO 4032 nuts, an ISO 7573 parts list read off the drawing, linked balloons, ISO 5457 frame, ISO 7200 title block — gated on `drawing_critique(focus=None)` returning **zero** issues and a finalize score of at least **90**; a test drops one balloon and watches the gate fail |
|
||||||
| `arch_roundtrip` | a two-room plan — an entrance door, an interior door, two windows, a stair — drawn by `arch_plan_from_spec` and read back by `arch_rooms_detect`, which never sees the spec: each room within 0.1 % of its label and of the net floor computed by hand (4825 × 5750 and 3825 × 5750 mm), the door and window schedules listing D1, D2, W1, W2, `drawing_critique(focus=None)` at **zero** and a finalize score of at least **90**; a test removes one room label and watches the gate fail |
|
| `arch_roundtrip` | a two-room plan — an entrance door, an interior door, two windows, a stair — drawn by `arch_plan_from_spec` and read back by `arch_rooms_detect`, which never sees the spec: each room within 0.1 % of its label and of the net floor computed by hand (4825 × 5750 and 3825 × 5750 mm), the door and window schedules listing D1, D2, W1, W2, `drawing_critique(focus=None)` at **zero** and a finalize score of at least **90**; a test removes one room label and watches the gate fail |
|
||||||
|
| `takeoff_roundtrip` | the synthetic plant pair — a P&ID stretched 1.3× in one room and rearranged in the other, Russian supply / return words, a return drawn on the supply layer, a reducer, a segment drawn twice — read by the scale check (**schematic**: 21.8 % of 55 pairs within ±10 %), `pipe_takeoff` (every routable run's layout length equal to the rectilinear MST of its tags, sizes and services as drawn, the run to a tag the layout lacks status C) and `cable_takeoff` (9 / 17 / 19 / 10 m after the 20 % allowance, the heater's power left empty); a test removes the wiring callouts and watches the cable half fail by name |
|
||||||
|
| `understand_foreign` | the pair's layout — inches declared over millimetres, a stray line 5,000 km out, two plan copies — comes back as millimetres with an `INSUNITS` warning, the stray line's handle and two clusters; the P&ID's `P_product piping` / `P_cipsupplyline` / `P_cipreturnline` / `P_ijswater` / `E_power` classify as product / CIP supply / CIP return / ice water / electrical; a test makes the layout declare millimetres and watches the units gate fail |
|
||||||
|
|
||||||
### Headless performance
|
### Headless performance
|
||||||
|
|
||||||
|
|||||||
+15
-4
@@ -127,6 +127,17 @@ result for it.
|
|||||||
|
|
||||||
The competitor reports carry no result for it.
|
The competitor reports carry no result for it.
|
||||||
|
|
||||||
|
### Matrix v7 (v1.6) — understanding foreign drawings
|
||||||
|
|
||||||
|
`tasks_v7.py` keeps the v6 nineteen unchanged and adds track H's evidence, both on the synthetic plant pair of `tests/fixtures/plant_pair.py` (invented names; its truth is computed from the positions it placed, never typed):
|
||||||
|
|
||||||
|
| Task | Category | Verified against |
|
||||||
|
|---|---|---|
|
||||||
|
| `takeoff_roundtrip` | plant | `scale_check(pid, layout)` must say **schematic** (room 1 is a 1.3× stretch, room 2 is rearranged: 21.8 % of the 55 tag pairs within ±10 % of the median ratio); `pipe_rows(..., length_source="auto")` must report all nine runs with the truth's status, each routable run's layout length equal to the rectilinear MST of its tags to a micron (PR1 5500, PR2 13200, CS1 20700 mm, …), its size and service as drawn — the CIP return drawn on the supply layer included — and both sizes across the reducer; `cable_rows` must give M11 / T102 / M12 / T202 their panels, 6900 / 14100 / 15800 / 8000 mm and 9 / 17 / 19 / 10 m after the 20 % allowance, the heater's power empty. `tests/test_benchmark_v7.py` deletes the wiring callouts and the gate names all four loads |
|
||||||
|
| `understand_foreign` | understand | `describe(layout)` must infer millimetres and warn about `INSUNITS` (the file says inches), name the stray line's handle in its extents and report two clusters (the plan is drawn twice, 100 m apart); `describe(pid)` must classify `P_product piping`, `P_cipsupplyline`, `P_cipreturnline`, `P_ijswater` and `E_power` as product, CIP supply, CIP return, ice water and electrical. `tests/test_benchmark_v7.py` makes the layout declare millimetres and the units gate fails alone |
|
||||||
|
|
||||||
|
The competitor reports carry no result for them.
|
||||||
|
|
||||||
## Live competitor lane (v1.4)
|
## Live competitor lane (v1.4)
|
||||||
|
|
||||||
Two competitor adapters now execute the exact same task matrix, black-box over
|
Two competitor adapters now execute the exact same task matrix, black-box over
|
||||||
@@ -244,15 +255,15 @@ python benchmarks/compare_versions.py --json results.json
|
|||||||
|
|
||||||
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
|
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
|
||||||
|
|
||||||
43 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
48 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||||
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
|
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
|
||||||
|
|
||||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||||
|---------|----------------|-----------|-------|-----------|
|
|---------|----------------|-----------|-------|-----------|
|
||||||
| **v1.5.1** (baseline) | 26 / 43 | 60.5 % | — | — |
|
| **v1.5.1** (baseline) | 26 / 48 | 54.2 % | — | — |
|
||||||
| **v1.6.0-dev** (this branch) | 43 / 43 | 100 % | 17 | **0** |
|
| **v1.6.0-dev** (this branch) | 48 / 48 | 100 % | 22 | **0** |
|
||||||
|
|
||||||
The seventeen are all `miss → pass` — v1.5.1 has none of the methods. Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven
|
The twenty-two are all `miss → pass` — v1.5.1 has none of the methods. Five are track H: `scale_check_detects_schematic` (the synthetic P&ID is schematic against its layout; a uniform 2× copy is to scale with factor 2), `pipe_takeoff_rmst_exact` (eight routable runs, each exactly the rectilinear MST of its tags), `cable_takeoff_roundup` (9 / 17 / 19 / 10 m, the unstated power left empty), `diff_detects_known_edits` (five known edits and nothing else) and `topology_known_defects` (one near miss, one crossing, a clean T). Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven
|
||||||
are tracks B and G: `mech_part_roundtrip` (a part read back from its own
|
are tracks B and G: `mech_part_roundtrip` (a part read back from its own
|
||||||
`ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a
|
`ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a
|
||||||
sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces,
|
sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces,
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
"""Reference adapter for this repository, covering the v6 task matrix.
|
"""Reference adapter for this repository, covering the v7 task matrix.
|
||||||
|
|
||||||
Every task verifies against a value worked out independently of the code under
|
Every task verifies against a value worked out independently of the code under
|
||||||
test — a closed-form area, a count of entities placed on purpose, a token
|
test — a closed-form area, a count of entities placed on purpose, a token
|
||||||
@@ -12,7 +12,7 @@ import math
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||||
from benchmarks.tasks_v6 import TaskSpec
|
from benchmarks.tasks_v7 import TaskSpec
|
||||||
from engineering.delivery import deliver_drawing
|
from engineering.delivery import deliver_drawing
|
||||||
from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
|
from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
|
||||||
from engineering.refiner import refine_drawing
|
from engineering.refiner import refine_drawing
|
||||||
@@ -779,6 +779,147 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
|
|||||||
[str(sheet)],
|
[str(sheet)],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
async def _task_takeoff_roundtrip(self):
|
||||||
|
"""Pipe and cable quantities off the synthetic plant pair, against its truth.
|
||||||
|
|
||||||
|
Track H's evidence (spec §15). The pair is written fresh into the
|
||||||
|
artifact folder by ``tests/fixtures/plant_pair.py``, whose truth is
|
||||||
|
computed from the positions it placed. Then only the two DXFs are read:
|
||||||
|
|
||||||
|
* ``scale_check(pid, layout)`` must say ``schematic`` - room 1 is the
|
||||||
|
layout stretched 1.3x, room 2 is rearranged, and 21.8 % of the 55 tag
|
||||||
|
pairs lie within ±10 % of the median ratio;
|
||||||
|
* ``pipe_rows(..., length_source="auto")`` must report every run of the
|
||||||
|
P&ID (the ice-water utility included, because ``services`` names it)
|
||||||
|
with the truth's status; a routable run's layout length must equal the
|
||||||
|
rectilinear MST of its tags exactly, its sizes (both across the
|
||||||
|
reducer) and its service must be the drawn ones (the CIP return drawn
|
||||||
|
on the supply layer included); the run ending on T105, which the
|
||||||
|
layout lacks, is status C with no layout length;
|
||||||
|
* ``cable_rows(pid, layout)`` must give every load its panel from the
|
||||||
|
layout's wiring callouts, its stated power (the heater's stays empty),
|
||||||
|
the Manhattan length to the panel and the metres rounded up after the
|
||||||
|
20 % allowance.
|
||||||
|
|
||||||
|
Any one slipping fails the task, and the metrics name what slipped.
|
||||||
|
"""
|
||||||
|
from engineering.understand.network import build_network
|
||||||
|
from engineering.understand.scale import scale_check
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
from engineering.understand.takeoff import cable_rows, pipe_rows
|
||||||
|
from tests.fixtures.plant_pair import build_plant_pair
|
||||||
|
|
||||||
|
truth = build_plant_pair(self.artifact_dir / "plant_pair")
|
||||||
|
pid = read_snapshot(truth["pid"])
|
||||||
|
layout = read_snapshot(truth["layout"])
|
||||||
|
scale = scale_check(pid, layout)
|
||||||
|
services = tuple(sorted({run["service"] for run in truth["pipe_runs"]}))
|
||||||
|
pipe = pipe_rows(
|
||||||
|
build_network(pid), pid, layout, scale=scale, services=services, length_source="auto"
|
||||||
|
)
|
||||||
|
found = {(run["service"], tuple(sorted(run["tags"]))): run for run in pipe["runs"]}
|
||||||
|
pipe_mismatch = []
|
||||||
|
for run in truth["pipe_runs"]:
|
||||||
|
got = found.get((run["service"], tuple(run["tags"])))
|
||||||
|
if got is None:
|
||||||
|
pipe_mismatch.append({"run": run["id"], "reason": "no run with these tags"})
|
||||||
|
continue
|
||||||
|
reasons = []
|
||||||
|
if got["status"] != run["status"]:
|
||||||
|
reasons.append(f"status {got['status']} != {run['status']}")
|
||||||
|
want, have = run["layout_length_mm"], got.get("layout_m")
|
||||||
|
if want is None and have is not None:
|
||||||
|
reasons.append(f"a layout length ({have} m) for an unroutable run")
|
||||||
|
elif want is not None and (have is None or abs(float(have) * 1000.0 - want) > 1e-6):
|
||||||
|
reasons.append(f"layout length {have} m != {want} mm")
|
||||||
|
if set(got["diameters"]) != set(run["diameters"]):
|
||||||
|
reasons.append(f"sizes {sorted(got['diameters'])} != {sorted(run['diameters'])}")
|
||||||
|
if reasons:
|
||||||
|
pipe_mismatch.append({"run": run["id"], "reason": "; ".join(reasons)})
|
||||||
|
|
||||||
|
cable = {row["tag"]: row for row in cable_rows(pid, layout)["rows"]}
|
||||||
|
cable_mismatch = []
|
||||||
|
for want in truth["cable_rows"]:
|
||||||
|
got = cable.get(want["tag"])
|
||||||
|
if got is None:
|
||||||
|
cable_mismatch.append({"tag": want["tag"], "wrong": ["missing"]})
|
||||||
|
continue
|
||||||
|
wrong = [key for key in ("panel", "kw") if got.get(key) != want[key]]
|
||||||
|
if got.get("cable_m") != want["roundup_m"]:
|
||||||
|
wrong.append("cable_m")
|
||||||
|
length = got.get("length_m")
|
||||||
|
if length is None or abs(float(length) * 1000.0 - want["length_mm"]) > 1e-6:
|
||||||
|
wrong.append("length_m")
|
||||||
|
if wrong:
|
||||||
|
cable_mismatch.append({"tag": want["tag"], "wrong": wrong})
|
||||||
|
|
||||||
|
passed = scale["verdict"] == truth["scale_verdict"] and not pipe_mismatch
|
||||||
|
passed = passed and not cable_mismatch
|
||||||
|
return (
|
||||||
|
passed,
|
||||||
|
{
|
||||||
|
"scale_verdict": scale["verdict"],
|
||||||
|
"within_10pct": scale.get("within_10pct"),
|
||||||
|
"length_source": pipe.get("length_source"),
|
||||||
|
"runs_checked": len(truth["pipe_runs"]),
|
||||||
|
"pipe_mismatch": pipe_mismatch,
|
||||||
|
"loads_checked": len(truth["cable_rows"]),
|
||||||
|
"cable_mismatch": cable_mismatch,
|
||||||
|
},
|
||||||
|
[truth["pid"], truth["layout"]],
|
||||||
|
)
|
||||||
|
|
||||||
|
async def _task_understand_foreign(self):
|
||||||
|
"""One call on a foreign layout reports what is wrong with it, by evidence.
|
||||||
|
|
||||||
|
Track H's evidence (spec §5, §15), on the synthetic plant pair: the
|
||||||
|
layout declares inches (``$INSUNITS = 1``) over millimetre geometry, a
|
||||||
|
stray LINE 5,000 km away stretches its declared extents, and the plan is
|
||||||
|
drawn twice, 100 m apart. ``describe`` of the layout must infer
|
||||||
|
millimetres, report that they disagree with the declared unit and warn
|
||||||
|
about ``INSUNITS``, name the stray entity's handle in its extents, and
|
||||||
|
report exactly two clusters; ``describe`` of the P&ID must classify its
|
||||||
|
four service layers and the electrical layer as the generator names them
|
||||||
|
(English and Dutch names). A warning or an extents report is searched as
|
||||||
|
JSON, so the gate holds whatever shape the evidence takes.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
from engineering.understand.describe import describe
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
from tests.fixtures.plant_pair import build_plant_pair
|
||||||
|
|
||||||
|
truth = build_plant_pair(self.artifact_dir / "plant_pair")
|
||||||
|
layout = describe(read_snapshot(truth["layout"]))
|
||||||
|
pid = describe(read_snapshot(truth["pid"]))
|
||||||
|
units = layout["units"]
|
||||||
|
units_flagged = (
|
||||||
|
str(units.get("inferred")) == truth["layout_true_unit"]
|
||||||
|
and units.get("agree") is False
|
||||||
|
and "INSUNITS" in json.dumps(layout["warnings"], ensure_ascii=False)
|
||||||
|
)
|
||||||
|
outlier_found = f'"{truth["outlier_handle"]}"' in json.dumps(layout["extents"])
|
||||||
|
copies_separated = len(layout["clusters"]) == truth["cluster_count"]
|
||||||
|
read = {row["name"]: row["service"] for row in pid["layers"]}
|
||||||
|
misread = {
|
||||||
|
name: read.get(name)
|
||||||
|
for name, service in truth["layer_services"].items()
|
||||||
|
if read.get(name) != service
|
||||||
|
}
|
||||||
|
passed = units_flagged and outlier_found and copies_separated and not misread
|
||||||
|
return (
|
||||||
|
passed,
|
||||||
|
{
|
||||||
|
"units": units,
|
||||||
|
"units_flagged": units_flagged,
|
||||||
|
"outlier_found": outlier_found,
|
||||||
|
"clusters": len(layout["clusters"]),
|
||||||
|
"copies_separated": copies_separated,
|
||||||
|
"misread_layers": misread,
|
||||||
|
},
|
||||||
|
[truth["layout"], truth["pid"]],
|
||||||
|
)
|
||||||
|
|
||||||
async def _task_auditable_delivery(self):
|
async def _task_auditable_delivery(self):
|
||||||
await ensure_engineering_layers(self.backend)
|
await ensure_engineering_layers(self.backend)
|
||||||
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
|
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
|
||||||
|
|||||||
@@ -818,6 +818,214 @@ async def arch_rooms_detect_foreign():
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Understanding foreign drawings (track H) ───────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _plant_pair() -> dict:
|
||||||
|
"""The synthetic plant pair, written to a fresh temporary folder."""
|
||||||
|
from tests.fixtures.plant_pair import build_plant_pair
|
||||||
|
|
||||||
|
return build_plant_pair(tempfile.mkdtemp(prefix="acadmcp_plant_"))
|
||||||
|
|
||||||
|
|
||||||
|
def _quoted(handle: str) -> str:
|
||||||
|
"""A handle as it appears in a JSON dump - quoted, so '2A' never matches '12A3'."""
|
||||||
|
return f'"{handle}"'
|
||||||
|
|
||||||
|
|
||||||
|
async def scale_check_detects_schematic():
|
||||||
|
"""The synthetic P&ID is schematic against its layout; a uniform 2x copy is not.
|
||||||
|
|
||||||
|
The pair: room 1 of the P&ID is the layout stretched 1.3x, room 2 is
|
||||||
|
rearranged, and of the 55 pairs of the 11 shared tags 21.8 % lie within
|
||||||
|
±10 % of the median ratio - at most 50 % is ``schematic`` (spec §6).
|
||||||
|
The control: six tags at (0, 0), (4000, 0), (8000, 0), (0, 3000),
|
||||||
|
(4000, 3000), (8000, 6000) and the same tags at twice those coordinates -
|
||||||
|
the nearest pair is 3000 mm apart, every ratio is exactly 2.0, so the
|
||||||
|
doubled drawing is ``to_scale`` against the first with factor 2.
|
||||||
|
"""
|
||||||
|
import ezdxf
|
||||||
|
|
||||||
|
from engineering.understand.scale import scale_check
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
|
||||||
|
truth = _plant_pair()
|
||||||
|
pair = scale_check(read_snapshot(truth["pid"]), read_snapshot(truth["layout"]))
|
||||||
|
folder = tempfile.mkdtemp(prefix="acadmcp_scale_")
|
||||||
|
positions = [(0, 0), (4000, 0), (8000, 0), (0, 3000), (4000, 3000), (8000, 6000)]
|
||||||
|
paths = []
|
||||||
|
for factor in (1.0, 2.0):
|
||||||
|
doc = ezdxf.new("R2018")
|
||||||
|
msp = doc.modelspace()
|
||||||
|
for number, (x, y) in enumerate(positions, start=1):
|
||||||
|
msp.add_circle((x * factor, y * factor), 500.0)
|
||||||
|
msp.add_text(f"T{number}", dxfattribs={"insert": (x * factor, y * factor)})
|
||||||
|
path = os.path.join(folder, f"x{factor:g}.dxf")
|
||||||
|
doc.saveas(path)
|
||||||
|
paths.append(path)
|
||||||
|
control = scale_check(read_snapshot(paths[1]), read_snapshot(paths[0]))
|
||||||
|
return (
|
||||||
|
pair["verdict"] == "schematic"
|
||||||
|
and control["verdict"] == "to_scale"
|
||||||
|
and abs(float(control["factor"]) - 2.0) < 1e-9
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def pipe_takeoff_rmst_exact():
|
||||||
|
"""Every routable pipe run's layout length is the rectilinear MST of its tags.
|
||||||
|
|
||||||
|
Measured on the synthetic pair with ``length_source="layout"``. The truth
|
||||||
|
is the generator's own Manhattan / Prim arithmetic over the tag positions it
|
||||||
|
placed, hand-checked in tests/test_plant_pair.py - PR1 5500 (5000 + 500),
|
||||||
|
PR2 13200 (the three distances 6000 / 7200 / 13200, spanning tree 6000 +
|
||||||
|
7200), CS1 20700 (20500 + 200). All eight routable runs must agree to a
|
||||||
|
micron; the run ending on T105, which the layout lacks, is not routable.
|
||||||
|
"""
|
||||||
|
from engineering.understand.network import build_network
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
from engineering.understand.takeoff import pipe_rows
|
||||||
|
|
||||||
|
truth = _plant_pair()
|
||||||
|
pid, layout = read_snapshot(truth["pid"]), read_snapshot(truth["layout"])
|
||||||
|
services = tuple(sorted({run["service"] for run in truth["pipe_runs"]}))
|
||||||
|
result = pipe_rows(build_network(pid), pid, layout, services=services, length_source="layout")
|
||||||
|
found = {(r["service"], tuple(sorted(r["tags"]))): r for r in result["runs"]}
|
||||||
|
routable = [run for run in truth["pipe_runs"] if run["layout_length_mm"] is not None]
|
||||||
|
for run in routable:
|
||||||
|
got = found.get((run["service"], tuple(run["tags"])))
|
||||||
|
if got is None or got.get("layout_m") is None:
|
||||||
|
return False
|
||||||
|
if abs(float(got["layout_m"]) * 1000.0 - run["layout_length_mm"]) > 1e-6:
|
||||||
|
return False
|
||||||
|
return len(routable) == 8
|
||||||
|
|
||||||
|
|
||||||
|
async def cable_takeoff_roundup():
|
||||||
|
"""Cable rows: the panel from the callout, Manhattan length, metres rounded up.
|
||||||
|
|
||||||
|
The truth rows of the synthetic pair, by hand: M11 (5500, 8200) to CP-1
|
||||||
|
(1200, 10800) is 6900 mm, 6.9 m x 1.2 = 8.28 -> 9; T102 (8000, 3500) to CP-1
|
||||||
|
is 14100 mm -> 16.92 -> 17; M12 (26300, 8400) to CP-2 (35000, 1300) is
|
||||||
|
15800 mm -> 18.96 -> 19; T202 (28700, 3000) to CP-2 is 8000 mm -> 9.6 -> 10.
|
||||||
|
The heater on T202 states no power, and its row must keep an empty one.
|
||||||
|
"""
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
from engineering.understand.takeoff import cable_rows
|
||||||
|
|
||||||
|
truth = _plant_pair()
|
||||||
|
rows = {
|
||||||
|
row["tag"]: row
|
||||||
|
for row in cable_rows(read_snapshot(truth["pid"]), read_snapshot(truth["layout"]))["rows"]
|
||||||
|
}
|
||||||
|
expected = {"M11": ("CP-1", 9), "T102": ("CP-1", 17), "M12": ("CP-2", 19), "T202": ("CP-2", 10)}
|
||||||
|
for tag, (panel, metres) in expected.items():
|
||||||
|
row = rows.get(tag)
|
||||||
|
if row is None or row.get("panel") != panel or row.get("cable_m") != metres:
|
||||||
|
return False
|
||||||
|
return rows["T202"].get("kw") is None and rows["M11"].get("kw") == 5.5
|
||||||
|
|
||||||
|
|
||||||
|
async def diff_detects_known_edits():
|
||||||
|
"""Two revisions differing by exactly five edits - and the diff says so.
|
||||||
|
|
||||||
|
A move (a LINE shifted 10 mm), a text change (PUMP A -> PUMP B), an
|
||||||
|
attribute change (TAG T101 -> T102), an add (a new LINE) and a delete (a
|
||||||
|
LINE removed); a CIRCLE stays untouched. The edits are made on a copy read
|
||||||
|
back from the first file, so every surviving entity keeps its handle. The
|
||||||
|
diff must list the added line as added, the removed line as removed, the
|
||||||
|
three edited entities as changed - and nothing else, the circle included.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
import ezdxf
|
||||||
|
|
||||||
|
from engineering.understand.diff import diff_snapshots
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
|
||||||
|
folder = tempfile.mkdtemp(prefix="acadmcp_diff_")
|
||||||
|
old_path, new_path = os.path.join(folder, "rev_a.dxf"), os.path.join(folder, "rev_b.dxf")
|
||||||
|
doc = ezdxf.new("R2018")
|
||||||
|
msp = doc.modelspace()
|
||||||
|
moved = msp.add_line((0, 0), (100, 0), dxfattribs={"layer": "PIPE"})
|
||||||
|
text = msp.add_text("PUMP A", dxfattribs={"insert": (0, 50), "height": 2.5})
|
||||||
|
block = doc.blocks.new("TAGGED")
|
||||||
|
block.add_circle((0, 0), 10)
|
||||||
|
block.add_attdef("TAG", (0, 0), dxfattribs={"height": 2.5})
|
||||||
|
insert = msp.add_blockref("TAGGED", (200, 0))
|
||||||
|
insert.add_auto_attribs({"TAG": "T101"})
|
||||||
|
gone = msp.add_line((0, 100), (100, 100), dxfattribs={"layer": "PIPE"})
|
||||||
|
kept = msp.add_circle((300, 300), 20)
|
||||||
|
doc.saveas(old_path)
|
||||||
|
|
||||||
|
revision = ezdxf.readfile(old_path)
|
||||||
|
revision.entitydb[moved.dxf.handle].translate(10, 0, 0)
|
||||||
|
revision.entitydb[text.dxf.handle].dxf.text = "PUMP B"
|
||||||
|
revision.entitydb[insert.dxf.handle].get_attrib("TAG").dxf.text = "T102"
|
||||||
|
revision.modelspace().delete_entity(revision.entitydb[gone.dxf.handle])
|
||||||
|
added = revision.modelspace().add_line((500, 500), (600, 500), dxfattribs={"layer": "PIPE"})
|
||||||
|
revision.saveas(new_path)
|
||||||
|
|
||||||
|
diff = diff_snapshots(read_snapshot(old_path), read_snapshot(new_path))
|
||||||
|
changed = json.dumps(diff["changed"])
|
||||||
|
dumped = changed + json.dumps(diff["added"]) + json.dumps(diff["removed"])
|
||||||
|
return (
|
||||||
|
len(diff["added"]) == 1
|
||||||
|
and len(diff["removed"]) == 1
|
||||||
|
and len(diff["changed"]) == 3
|
||||||
|
and _quoted(added.dxf.handle) in json.dumps(diff["added"])
|
||||||
|
and _quoted(gone.dxf.handle) in json.dumps(diff["removed"])
|
||||||
|
and all(_quoted(e.dxf.handle) in changed for e in (moved, text, insert))
|
||||||
|
and _quoted(kept.dxf.handle) not in dumped
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def topology_known_defects():
|
||||||
|
"""Dangling ends, a near miss and a crossing, placed on purpose - and nothing else.
|
||||||
|
|
||||||
|
On layer PIPE: a closed 1000 x 1000 square (no free end, no defect); a LINE
|
||||||
|
from (500, -500) that stops at (500, -5), 5 mm short of the square's bottom
|
||||||
|
edge (a near miss at gap 10, and a dangling end); two LINEs crossing at
|
||||||
|
(2500, 500), away from all four of their ends (one interior crossing); and a
|
||||||
|
clean T - a LINE ending on the middle of another at (4500, 0), which is a
|
||||||
|
junction, not a crossing and not a near miss.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
import ezdxf
|
||||||
|
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
from engineering.understand.topology import topology_findings
|
||||||
|
|
||||||
|
doc = ezdxf.new("R2018")
|
||||||
|
msp = doc.modelspace()
|
||||||
|
pipe = {"layer": "PIPE"}
|
||||||
|
ring = msp.add_lwpolyline(
|
||||||
|
[(0, 0), (1000, 0), (1000, 1000), (0, 1000)], close=True, dxfattribs=pipe
|
||||||
|
)
|
||||||
|
short = msp.add_line((500, -500), (500, -5), dxfattribs=pipe)
|
||||||
|
cross_a = msp.add_line((2000, 0), (3000, 1000), dxfattribs=pipe)
|
||||||
|
cross_b = msp.add_line((2000, 1000), (3000, 0), dxfattribs=pipe)
|
||||||
|
bar = msp.add_line((4000, 0), (5000, 0), dxfattribs=pipe)
|
||||||
|
stem = msp.add_line((4500, 0), (4500, 500), dxfattribs=pipe)
|
||||||
|
path = os.path.join(tempfile.mkdtemp(prefix="acadmcp_topo_"), "topology.dxf")
|
||||||
|
doc.saveas(path)
|
||||||
|
|
||||||
|
found = topology_findings(read_snapshot(path), layers=["PIPE"], gap=10.0)
|
||||||
|
near = json.dumps(found["near_miss"])
|
||||||
|
crossing = json.dumps(found["crossing"])
|
||||||
|
dangling = json.dumps(found["dangling"])
|
||||||
|
return (
|
||||||
|
_quoted(short.dxf.handle) in near
|
||||||
|
and _quoted(short.dxf.handle) in dangling
|
||||||
|
and _quoted(ring.dxf.handle) not in dangling
|
||||||
|
and _quoted(ring.dxf.handle) not in crossing
|
||||||
|
and len(found["crossing"]) == 1
|
||||||
|
and _quoted(cross_a.dxf.handle) in crossing
|
||||||
|
and _quoted(cross_b.dxf.handle) in crossing
|
||||||
|
and all(_quoted(e.dxf.handle) not in crossing + near for e in (bar, stem))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
CHECKS = {
|
CHECKS = {
|
||||||
"core_line_length": (core_line_length, "Core"),
|
"core_line_length": (core_line_length, "Core"),
|
||||||
"core_circle_radius": (core_circle_radius, "Core"),
|
"core_circle_radius": (core_circle_radius, "Core"),
|
||||||
@@ -862,6 +1070,11 @@ CHECKS = {
|
|||||||
"arch_room_area_net": (arch_room_area_net, "Architecture"),
|
"arch_room_area_net": (arch_room_area_net, "Architecture"),
|
||||||
"arch_opening_cuts_wall": (arch_opening_cuts_wall, "Architecture"),
|
"arch_opening_cuts_wall": (arch_opening_cuts_wall, "Architecture"),
|
||||||
"arch_rooms_detect_foreign": (arch_rooms_detect_foreign, "Architecture"),
|
"arch_rooms_detect_foreign": (arch_rooms_detect_foreign, "Architecture"),
|
||||||
|
"scale_check_detects_schematic": (scale_check_detects_schematic, "Understanding"),
|
||||||
|
"pipe_takeoff_rmst_exact": (pipe_takeoff_rmst_exact, "Understanding"),
|
||||||
|
"cable_takeoff_roundup": (cable_takeoff_roundup, "Understanding"),
|
||||||
|
"diff_detects_known_edits": (diff_detects_known_edits, "Understanding"),
|
||||||
|
"topology_known_defects": (topology_known_defects, "Understanding"),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ import json
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from benchmarks.tasks_v6 import TASKS_V6
|
from benchmarks.tasks_v7 import TASKS_V7
|
||||||
|
|
||||||
ROOT = Path(__file__).resolve().parents[1]
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
|
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
|
||||||
@@ -66,8 +66,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
|||||||
|
|
||||||
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
|
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
|
||||||
|
|
||||||
task_ids = [task.task_id for task in TASKS_V6]
|
task_ids = [task.task_id for task in TASKS_V7]
|
||||||
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V6]
|
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V7]
|
||||||
servers = [report["adapter"] for report in reports]
|
servers = [report["adapter"] for report in reports]
|
||||||
status_by_server = {
|
status_by_server = {
|
||||||
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
|
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
{
|
{
|
||||||
"summary": {
|
"summary": {
|
||||||
"baseline_ref": "v1.5.1",
|
"baseline_ref": "v1.5.1",
|
||||||
"total": 43,
|
"total": 48,
|
||||||
"old_pass": 26,
|
"old_pass": 26,
|
||||||
"new_pass": 43,
|
"new_pass": 48,
|
||||||
"old_pct": 60.5,
|
"old_pct": 54.2,
|
||||||
"new_pct": 100.0,
|
"new_pct": 100.0,
|
||||||
"fixed": 17,
|
"fixed": 22,
|
||||||
"regressed": 0
|
"regressed": 0
|
||||||
},
|
},
|
||||||
"rows": [
|
"rows": [
|
||||||
@@ -267,6 +267,36 @@
|
|||||||
"category": "Architecture",
|
"category": "Architecture",
|
||||||
"old": "miss",
|
"old": "miss",
|
||||||
"new": "pass"
|
"new": "pass"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "scale_check_detects_schematic",
|
||||||
|
"category": "Understanding",
|
||||||
|
"old": "miss",
|
||||||
|
"new": "pass"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "pipe_takeoff_rmst_exact",
|
||||||
|
"category": "Understanding",
|
||||||
|
"old": "miss",
|
||||||
|
"new": "pass"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "cable_takeoff_roundup",
|
||||||
|
"category": "Understanding",
|
||||||
|
"old": "miss",
|
||||||
|
"new": "pass"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "diff_detects_known_edits",
|
||||||
|
"category": "Understanding",
|
||||||
|
"old": "miss",
|
||||||
|
"new": "pass"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"check": "topology_known_defects",
|
||||||
|
"category": "Understanding",
|
||||||
|
"old": "miss",
|
||||||
|
"new": "pass"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
@@ -16,7 +16,7 @@ from pathlib import Path
|
|||||||
|
|
||||||
from backends.capability import UnsupportedCapabilityError
|
from backends.capability import UnsupportedCapabilityError
|
||||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||||
from benchmarks.tasks_v6 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
|
from benchmarks.tasks_v7 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
|
||||||
|
|
||||||
|
|
||||||
def _load_registry() -> dict[str, dict]:
|
def _load_registry() -> dict[str, dict]:
|
||||||
|
|||||||
@@ -0,0 +1,71 @@
|
|||||||
|
"""The v7 task matrix: v6 plus what track H added.
|
||||||
|
|
||||||
|
Track H teaches the server to read drawings somebody else made, so its two
|
||||||
|
tasks are gated on a drawing pair whose answers are known in advance - the
|
||||||
|
synthetic plant pair of ``tests/fixtures/plant_pair.py``, which reproduces
|
||||||
|
every field finding of the design spec with invented names:
|
||||||
|
|
||||||
|
* ``takeoff_roundtrip`` - the scale check calls the P&ID ``schematic``; every
|
||||||
|
routable pipe run's layout length equals the rectilinear minimum spanning
|
||||||
|
tree of its tags exactly; its size, service and status match the generator's
|
||||||
|
truth; the cable rows equal the Manhattan lengths from each load to its panel
|
||||||
|
with the allowance rounded up to the metre.
|
||||||
|
* ``understand_foreign`` - one ``describe`` of the layout flags inches declared
|
||||||
|
over millimetre geometry, finds the stray entity by handle and separates the
|
||||||
|
two plan copies; one ``describe`` of the P&ID classifies its English and
|
||||||
|
Dutch service layers.
|
||||||
|
|
||||||
|
The published competitor reports were never asked these questions; the chart
|
||||||
|
shows ``not_run`` for them, not zero.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from benchmarks.tasks_v2 import TASKS_V2
|
||||||
|
from benchmarks.tasks_v3 import TASKS_V3
|
||||||
|
from benchmarks.tasks_v4 import TASKS_V4
|
||||||
|
from benchmarks.tasks_v5 import TASKS_V5
|
||||||
|
from benchmarks.tasks_v6 import TASKS_V6, TaskSpec
|
||||||
|
|
||||||
|
#: Added in v1.6.0 by track H.
|
||||||
|
NEW_TASKS_V7: tuple[TaskSpec, ...] = (
|
||||||
|
TaskSpec(
|
||||||
|
"takeoff_roundtrip",
|
||||||
|
"plant",
|
||||||
|
"Take pipe and cable quantities off a schematic P&ID and its layout: the P&ID "
|
||||||
|
"called schematic, every run's layout length equal to the rectilinear MST of "
|
||||||
|
"its tags, sizes and services as drawn, cable metres rounded up after the "
|
||||||
|
"allowance",
|
||||||
|
1.0,
|
||||||
|
),
|
||||||
|
TaskSpec(
|
||||||
|
"understand_foreign",
|
||||||
|
"understand",
|
||||||
|
"Read a foreign layout in one call: inches declared over millimetres flagged, "
|
||||||
|
"the stray entity found, both plan copies separated, the service layers of its "
|
||||||
|
"P&ID classified",
|
||||||
|
1.0,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
TASKS_V7: tuple[TaskSpec, ...] = TASKS_V6 + NEW_TASKS_V7
|
||||||
|
|
||||||
|
#: Runner-selectable task sets. v2 to v6 stay addressable so an earlier report
|
||||||
|
#: can be reproduced exactly rather than only described.
|
||||||
|
MATRICES: dict[str, tuple[TaskSpec, ...]] = {
|
||||||
|
"v2": TASKS_V2,
|
||||||
|
"v3": TASKS_V3,
|
||||||
|
"v4": TASKS_V4,
|
||||||
|
"v5": TASKS_V5,
|
||||||
|
"v6": TASKS_V6,
|
||||||
|
"v7": TASKS_V7,
|
||||||
|
}
|
||||||
|
|
||||||
|
DEFAULT_MATRIX = "v7"
|
||||||
|
|
||||||
|
|
||||||
|
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
||||||
|
for task in MATRICES[matrix]:
|
||||||
|
if task.task_id == task_id:
|
||||||
|
return task
|
||||||
|
raise KeyError(f"{task_id} is not in matrix {matrix}")
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
# v1.6 Track H — understanding foreign drawings, takeoffs, diff and topology
|
# v1.6 Track H — understanding foreign drawings, takeoffs, diff and topology
|
||||||
|
|
||||||
**Date:** 2026-09-24 · **Status:** design approved in conversation ·
|
**Date:** 2026-09-24 · **Status:** shipped (v1.6.0-dev, track H) ·
|
||||||
**Parent:** `2026-09-15-v1.6-roadmap-design.md` (track H; track D stays separate
|
**Parent:** `2026-09-15-v1.6-roadmap-design.md` (track H; track D stays separate
|
||||||
and follows) · **Baseline:** main after track F (`0abbe55`, 241 tools)
|
and follows) · **Baseline:** main after track F (`0abbe55`, 241 tools)
|
||||||
|
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ from collections.abc import Sequence
|
|||||||
from engineering.arch.faces import Face, face_containing, planar_faces
|
from engineering.arch.faces import Face, face_containing, planar_faces
|
||||||
from engineering.pid.graph import ATTRIBUTE_TAGS, KEYWORDS
|
from engineering.pid.graph import ATTRIBUTE_TAGS, KEYWORDS
|
||||||
from engineering.understand.clusters import find_clusters
|
from engineering.understand.clusters import find_clusters
|
||||||
from engineering.understand.labels import parse_tag
|
from engineering.understand.labels import parse_tag, wiring_target
|
||||||
from engineering.understand.snapshot import EntityRecord, Snapshot, length_of
|
from engineering.understand.snapshot import EntityRecord, Snapshot, length_of
|
||||||
from engineering.understand.units import infer_units, robust_extents, wall_segments
|
from engineering.understand.units import infer_units, robust_extents, wall_segments
|
||||||
from engineering.understand.vocab import classify_layer, room_label
|
from engineering.understand.vocab import classify_layer, room_label
|
||||||
@@ -113,6 +113,9 @@ def tag_occurrences(snap: Snapshot, *, prep: dict | None = None) -> list[dict]:
|
|||||||
|
|
||||||
A tag is read with `labels.parse_tag` from a TEXT / MTEXT / MULTILEADER,
|
A tag is read with `labels.parse_tag` from a TEXT / MTEXT / MULTILEADER,
|
||||||
or from an INSERT attribute named like a tag (`pid.graph.ATTRIBUTE_TAGS`).
|
or from an INSERT attribute named like a tag (`pid.graph.ATTRIBUTE_TAGS`).
|
||||||
|
A wiring callout (`labels.wiring_target`: ``Wiring to CP-2``) is not an
|
||||||
|
occurrence of the panel it names - it stands at the load, not the panel -
|
||||||
|
the rule `network.tag_occurrences` keeps too.
|
||||||
Two readings of one tag in one space closer than
|
Two readings of one tag in one space closer than
|
||||||
:data:`TAG_MERGE_FRACTION` of the diagonal are one occurrence (the
|
:data:`TAG_MERGE_FRACTION` of the diagonal are one occurrence (the
|
||||||
attribute's position and confidence win). Rows carry ``tag``, ``handle``,
|
attribute's position and confidence win). Rows carry ``tag``, ``handle``,
|
||||||
@@ -128,6 +131,8 @@ def tag_occurrences(snap: Snapshot, *, prep: dict | None = None) -> list[dict]:
|
|||||||
continue
|
continue
|
||||||
readings = []
|
readings = []
|
||||||
if rec.type in _TEXT_TYPES and rec.text:
|
if rec.type in _TEXT_TYPES and rec.text:
|
||||||
|
if wiring_target(rec.text):
|
||||||
|
continue # names the panel it points at; it is not written there
|
||||||
readings.append((parse_tag(rec.text), "text", TEXT_TAG_CONFIDENCE))
|
readings.append((parse_tag(rec.text), "text", TEXT_TAG_CONFIDENCE))
|
||||||
elif rec.type == "INSERT":
|
elif rec.type == "INSERT":
|
||||||
for name, value in rec.attribs:
|
for name, value in rec.attribs:
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ def derived() -> dict[str, object]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
from benchmarks import correctness_suite
|
from benchmarks import correctness_suite
|
||||||
from benchmarks.tasks_v6 import TASKS_V6
|
from benchmarks.tasks_v7 import TASKS_V7
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"collected_tests": _collected_tests(),
|
"collected_tests": _collected_tests(),
|
||||||
@@ -94,7 +94,7 @@ def derived() -> dict[str, object]:
|
|||||||
"ab_fixed": ab["fixed"],
|
"ab_fixed": ab["fixed"],
|
||||||
"ab_regressed": ab["regressed"],
|
"ab_regressed": ab["regressed"],
|
||||||
"suite_checks": len(correctness_suite.CHECKS),
|
"suite_checks": len(correctness_suite.CHECKS),
|
||||||
"matrix_tasks": len(TASKS_V6),
|
"matrix_tasks": len(TASKS_V7),
|
||||||
"perf_ms": perf,
|
"perf_ms": perf,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
"""Run both takeoffs on a real P&ID and layout, and write the report next to them.
|
||||||
|
|
||||||
|
LOCAL ONLY. This is the field test of track H (spec §15): the user points it at
|
||||||
|
a client's P&ID and layout, compares what comes out with the hand-made
|
||||||
|
workbooks, and keeps the result on their own disk. Nothing it reads or writes
|
||||||
|
may enter this repository (spec §14), so:
|
||||||
|
|
||||||
|
* the report is written next to the P&ID (or into ``--out``) and **the script
|
||||||
|
refuses any output folder inside this repository**, before reading a byte;
|
||||||
|
* the console gets file names, row counts and the scale verdict - **never a
|
||||||
|
quantity**, unless ``--show`` is given on the user's own terminal;
|
||||||
|
* it is never run in CI and never by a test on a real drawing; the test next
|
||||||
|
to it (tests/test_field_test_takeoff.py) runs it on the synthetic plant pair.
|
||||||
|
|
||||||
|
What it writes: ``pipe_takeoff.xlsx`` / ``cable_takeoff.xlsx`` (CSV beside
|
||||||
|
them always, XLSX when the ``office`` extra is installed) through the same
|
||||||
|
``write_workbook`` the tools use, and ``field_test_report.json`` with the scale
|
||||||
|
check and both results in full.
|
||||||
|
|
||||||
|
uv run --frozen python scripts/field_test_takeoff.py --pid D:/job/pid.dxf --layout D:/job/layout.dxf
|
||||||
|
uv run --frozen python scripts/field_test_takeoff.py --pid ... --layout ... --lang en --show
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
|
||||||
|
#: Exit code for a refused run (output inside the repository, a missing file).
|
||||||
|
REFUSED = 2
|
||||||
|
|
||||||
|
|
||||||
|
def _inside_repository(folder: Path) -> bool:
|
||||||
|
resolved = folder.resolve()
|
||||||
|
return resolved == ROOT or ROOT in resolved.parents
|
||||||
|
|
||||||
|
|
||||||
|
def _parse(argv) -> argparse.Namespace:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Pipe and cable takeoffs on a real P&ID + layout (local only)."
|
||||||
|
)
|
||||||
|
parser.add_argument("--pid", required=True, help="the P&ID, a .dxf")
|
||||||
|
parser.add_argument("--layout", required=True, help="the layout, a .dxf")
|
||||||
|
parser.add_argument("--out", default=None, help="report folder (default: the P&ID's folder)")
|
||||||
|
parser.add_argument("--lang", default="tr", choices=("tr", "en", "ru"))
|
||||||
|
parser.add_argument("--allowance", type=float, default=0.20)
|
||||||
|
parser.add_argument(
|
||||||
|
"--show", action="store_true", help="also print the totals (quantities) to the console"
|
||||||
|
)
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None) -> int:
|
||||||
|
args = _parse(argv)
|
||||||
|
pid_path, layout_path = Path(args.pid), Path(args.layout)
|
||||||
|
out = Path(args.out) if args.out else pid_path.resolve().parent
|
||||||
|
if _inside_repository(out):
|
||||||
|
print(
|
||||||
|
f"refused: {out} is inside the repository ({ROOT}); client drawings and "
|
||||||
|
"anything derived from them never enter it. Pass --out outside it."
|
||||||
|
)
|
||||||
|
return REFUSED
|
||||||
|
for path in (pid_path, layout_path):
|
||||||
|
if not path.is_file():
|
||||||
|
print(f"refused: {path} does not exist")
|
||||||
|
return REFUSED
|
||||||
|
|
||||||
|
from engineering.understand.network import build_network
|
||||||
|
from engineering.understand.report import write_workbook
|
||||||
|
from engineering.understand.scale import scale_check
|
||||||
|
from engineering.understand.snapshot import read_snapshot
|
||||||
|
from engineering.understand.takeoff import cable_rows, pipe_rows
|
||||||
|
|
||||||
|
out.mkdir(parents=True, exist_ok=True)
|
||||||
|
pid = read_snapshot(str(pid_path))
|
||||||
|
layout = read_snapshot(str(layout_path))
|
||||||
|
scale = scale_check(pid, layout)
|
||||||
|
pipe = pipe_rows(build_network(pid), pid, layout, scale=scale, allowance=args.allowance)
|
||||||
|
cable = cable_rows(pid, layout, allowance=args.allowance)
|
||||||
|
written = [
|
||||||
|
write_workbook(pipe, kind="pipe", lang=args.lang, path=str(out / "pipe_takeoff.xlsx")),
|
||||||
|
write_workbook(cable, kind="cable", lang=args.lang, path=str(out / "cable_takeoff.xlsx")),
|
||||||
|
]
|
||||||
|
report = out / "field_test_report.json"
|
||||||
|
report.write_text(
|
||||||
|
json.dumps(
|
||||||
|
{"scale": scale, "pipe": pipe, "cable": cable, "written": written},
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
default=str,
|
||||||
|
),
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"scale_verdict": scale["verdict"],
|
||||||
|
"pipe_rows": len(pipe["rows"]),
|
||||||
|
"cable_rows": len(cable["rows"]),
|
||||||
|
"report": str(report),
|
||||||
|
"workbooks": [
|
||||||
|
path
|
||||||
|
for item in written
|
||||||
|
for path in ([item["xlsx"]] if item["xlsx"] else []) + list(item["csv"])
|
||||||
|
],
|
||||||
|
},
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if args.show:
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{"pipe": pipe["rows"], "cable": cable["rows"]},
|
||||||
|
indent=2,
|
||||||
|
ensure_ascii=False,
|
||||||
|
default=str,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,386 @@
|
|||||||
|
"""Exercise track H once against the live AutoCAD on this machine.
|
||||||
|
|
||||||
|
Builds the synthetic plant pair (tests/fixtures/plant_pair.py) in a temporary
|
||||||
|
folder and reads it through the live engine - never an operator's drawing:
|
||||||
|
|
||||||
|
open pid.dxf -> take_snapshot(backend) (Document.Export, "DXF")
|
||||||
|
-> drawing_save_as(pid.dwg), close it
|
||||||
|
-> take_snapshot(backend, pid.dwg) (opened read-only, exported, closed)
|
||||||
|
open layout.dxf -> take_snapshot(backend)
|
||||||
|
pipe_rows / cable_rows on the two live snapshots, against the pair's truth
|
||||||
|
entity_move one tag text by 10 mm -> take_snapshot -> diff_snapshots
|
||||||
|
against the snapshot taken before the move
|
||||||
|
|
||||||
|
and prints what came back as JSON. Exit codes:
|
||||||
|
|
||||||
|
0 every live snapshot holds the same model-space entities (by type) as the
|
||||||
|
file it came from, the takeoffs match the truth, and the diff reports
|
||||||
|
exactly the one moved entity
|
||||||
|
1 it ran, but something disagreed (the JSON says what)
|
||||||
|
2 no live CAD application reachable (pywin32 missing, or no running
|
||||||
|
instance to attach to) -- the COM paths then remain fake-tested only
|
||||||
|
|
||||||
|
**The operator's drawings are never touched.** AutoCAD may have other
|
||||||
|
documents open, with unsaved work in them, while this runs. So:
|
||||||
|
|
||||||
|
* the smoke only ever attaches to a running instance (``GetActiveObject``);
|
||||||
|
it never creates one, because ``Dispatch`` would launch AutoCAD;
|
||||||
|
* every document it works in is one it brought in itself from the files it
|
||||||
|
just wrote into its own temporary folder, and it knows them by full path -
|
||||||
|
never by a name another document could share;
|
||||||
|
* every call that reads or writes *the current document* (a live snapshot, the
|
||||||
|
DWG save, the move) first checks, inside the same COM-thread call, that the
|
||||||
|
active document is the smoke's own - and refuses otherwise;
|
||||||
|
* on the way out (also when a step fails) it closes its own documents with
|
||||||
|
``Close(False)`` and re-activates the document that was active when it
|
||||||
|
started; unless ``--keep`` is given, then its documents stay open;
|
||||||
|
* nothing it prints names a document other than its own.
|
||||||
|
|
||||||
|
AUTOCAD_MCP_BACKEND=com uv run --frozen python scripts/smoke_understand_com.py
|
||||||
|
uv run --frozen python scripts/smoke_understand_com.py --keep # leave them open
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import dataclasses
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from collections import Counter
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
|
||||||
|
#: ``RPC_E_CALL_REJECTED``: AutoCAD's message filter refused the call because
|
||||||
|
#: it was busy (a document still initialising after ``Documents.Open``). A
|
||||||
|
#: refused call was not executed.
|
||||||
|
RPC_E_CALL_REJECTED = -2147418111
|
||||||
|
|
||||||
|
#: How long a refused call is retried before the smoke gives up on it.
|
||||||
|
REJECTED_PATIENCE_S = 90.0
|
||||||
|
|
||||||
|
#: Full paths (normalised) of the documents this run opened and has not closed.
|
||||||
|
_OPENED: list[str] = []
|
||||||
|
|
||||||
|
#: The run's state shared with the COM thread: the folder its files live in,
|
||||||
|
#: the document that must be active for the next current-document call (None:
|
||||||
|
#: the call does not touch the current document), and the document that was
|
||||||
|
#: active when the run started (a COM object, used on the COM thread only).
|
||||||
|
_STATE: dict = {"folder": None, "expect": None, "start": None}
|
||||||
|
|
||||||
|
|
||||||
|
class NotOurDocument(RuntimeError):
|
||||||
|
"""The active document is not the one the smoke is about to read or write."""
|
||||||
|
|
||||||
|
|
||||||
|
def _key(path) -> str:
|
||||||
|
return str(Path(str(path)).resolve()).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _ours(path) -> bool:
|
||||||
|
folder = _STATE["folder"]
|
||||||
|
return bool(path) and folder is not None and Path(_key(path)).parent == Path(_key(folder))
|
||||||
|
|
||||||
|
|
||||||
|
def _check_active() -> None:
|
||||||
|
"""COM thread: refuse unless the active document is the expected own one."""
|
||||||
|
expected = _STATE["expect"]
|
||||||
|
if expected is None:
|
||||||
|
return
|
||||||
|
from backends import com_backend as com
|
||||||
|
|
||||||
|
app = com._acad_app()
|
||||||
|
active = str(app.ActiveDocument.FullName) if int(app.Documents.Count) else ""
|
||||||
|
if not active or _key(active) != expected:
|
||||||
|
# Never name the other document: it may be an operator's drawing.
|
||||||
|
raise NotOurDocument(
|
||||||
|
f"refused: the active document is not the smoke's own {Path(expected).name}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _patient(backend) -> None:
|
||||||
|
"""Retry any call AutoCAD refused with ``RPC_E_CALL_REJECTED``, and run the
|
||||||
|
active-document check in the same COM-thread call as the call it guards."""
|
||||||
|
run = backend._run
|
||||||
|
|
||||||
|
async def patient(func, *args, **kwargs):
|
||||||
|
def guarded(*a, **kw):
|
||||||
|
_check_active()
|
||||||
|
return func(*a, **kw)
|
||||||
|
|
||||||
|
deadline = time.monotonic() + REJECTED_PATIENCE_S
|
||||||
|
pause = 0.5
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
return await run(guarded, *args, **kwargs)
|
||||||
|
except RuntimeError as exc:
|
||||||
|
cause = exc.__cause__
|
||||||
|
hr = cause.args[0] if cause is not None and cause.args else None
|
||||||
|
if hr != RPC_E_CALL_REJECTED or time.monotonic() + pause > deadline:
|
||||||
|
raise
|
||||||
|
await asyncio.sleep(pause)
|
||||||
|
pause = min(pause * 2.0, 5.0)
|
||||||
|
|
||||||
|
backend._run = patient
|
||||||
|
|
||||||
|
|
||||||
|
class _Current:
|
||||||
|
"""``async with _Current(path):`` - calls inside it must find ``path`` active."""
|
||||||
|
|
||||||
|
def __init__(self, path) -> None:
|
||||||
|
self.path = _key(path)
|
||||||
|
|
||||||
|
async def __aenter__(self):
|
||||||
|
_STATE["expect"] = self.path
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def __aexit__(self, *exc):
|
||||||
|
_STATE["expect"] = None
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
async def _attach(backend) -> str | None:
|
||||||
|
"""Attach to the running instance only, and remember the active document.
|
||||||
|
|
||||||
|
Returns the application's version, or None when nothing is running (the
|
||||||
|
backend's own ``_acad_app`` would fall back to ``Dispatch`` and launch
|
||||||
|
AutoCAD - a smoke must never do that)."""
|
||||||
|
import win32com.client
|
||||||
|
|
||||||
|
import config
|
||||||
|
from backends import com_backend as com
|
||||||
|
|
||||||
|
def _sync():
|
||||||
|
try:
|
||||||
|
app = win32com.client.GetActiveObject(config.settings.cad_progid)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
com._COM_STATE["app"] = app
|
||||||
|
_STATE["start"] = app.ActiveDocument if int(app.Documents.Count) else None
|
||||||
|
return str(app.Version)
|
||||||
|
|
||||||
|
return await backend._run(_sync)
|
||||||
|
|
||||||
|
|
||||||
|
async def _open_paths(backend) -> set[str]:
|
||||||
|
"""Full paths of the open documents that are the smoke's own (COM thread)."""
|
||||||
|
|
||||||
|
def _sync():
|
||||||
|
from backends import com_backend as com
|
||||||
|
|
||||||
|
app = com._acad_app()
|
||||||
|
paths = (str(app.Documents.Item(i).FullName) for i in range(int(app.Documents.Count)))
|
||||||
|
return {_key(p) for p in paths if _ours(p)}
|
||||||
|
|
||||||
|
return await backend._run(_sync)
|
||||||
|
|
||||||
|
|
||||||
|
async def _open(backend, path) -> str:
|
||||||
|
"""``drawing_open`` of one of the smoke's own files; returns its key."""
|
||||||
|
if not _ours(path):
|
||||||
|
raise NotOurDocument(f"refused: {Path(path).name} is not in the smoke's own folder")
|
||||||
|
result = await backend.drawing_open(str(path))
|
||||||
|
key = _key(result.get("path") or path)
|
||||||
|
if key not in _OPENED:
|
||||||
|
_OPENED.append(key)
|
||||||
|
return key
|
||||||
|
|
||||||
|
|
||||||
|
async def _close(backend, key: str) -> None:
|
||||||
|
"""Close one of the smoke's own documents, unsaved (``Close(False)``)."""
|
||||||
|
if not _ours(key):
|
||||||
|
raise NotOurDocument("refused: not one of the smoke's own documents")
|
||||||
|
if key in await _open_paths(backend):
|
||||||
|
await backend.document_close(key, discard=True)
|
||||||
|
if key in _OPENED:
|
||||||
|
_OPENED.remove(key)
|
||||||
|
|
||||||
|
|
||||||
|
async def _close_leftovers(backend) -> None:
|
||||||
|
for key in list(_OPENED):
|
||||||
|
try:
|
||||||
|
await _close(backend, key)
|
||||||
|
except Exception as exc: # best effort on the way out
|
||||||
|
print(f"could not close {Path(key).name}: {type(exc).__name__}", file=sys.stderr)
|
||||||
|
if key in _OPENED:
|
||||||
|
_OPENED.remove(key)
|
||||||
|
|
||||||
|
|
||||||
|
async def _reactivate_start(backend) -> None:
|
||||||
|
"""Hand the seat back as it was: the starting document active again."""
|
||||||
|
|
||||||
|
def _sync():
|
||||||
|
start = _STATE["start"]
|
||||||
|
if start is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
start.Activate()
|
||||||
|
except Exception: # it may have been closed by its owner meanwhile
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
await backend._run(_sync)
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"could not re-activate the starting document: {type(exc).__name__}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def _model(snapshot):
|
||||||
|
"""The snapshot with its model-space records only (AutoCAD adds a paper-space
|
||||||
|
viewport when it initialises a layout; the takeoffs read model space)."""
|
||||||
|
return dataclasses.replace(
|
||||||
|
snapshot, records=tuple(r for r in snapshot.records if r.space == "Model")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _types(snapshot) -> dict[str, int]:
|
||||||
|
return dict(sorted(Counter(r.type for r in _model(snapshot).records).items()))
|
||||||
|
|
||||||
|
|
||||||
|
def _check_takeoffs(truth, pid, layout) -> dict:
|
||||||
|
from engineering.understand.network import build_network
|
||||||
|
from engineering.understand.scale import scale_check
|
||||||
|
from engineering.understand.takeoff import cable_rows, pipe_rows
|
||||||
|
|
||||||
|
scale = scale_check(pid, layout)
|
||||||
|
services = tuple(sorted({run["service"] for run in truth["pipe_runs"]}))
|
||||||
|
pipe = pipe_rows(build_network(pid), pid, layout, scale=scale, services=services)
|
||||||
|
found = {(r["service"], tuple(sorted(r["tags"]))): r for r in pipe["runs"]}
|
||||||
|
pipe_mismatch = []
|
||||||
|
for run in truth["pipe_runs"]:
|
||||||
|
got = found.get((run["service"], tuple(run["tags"])))
|
||||||
|
want = run["layout_length_mm"]
|
||||||
|
have = None if got is None else got.get("layout_m")
|
||||||
|
agrees = got is not None and got["status"] == run["status"]
|
||||||
|
if want is None:
|
||||||
|
agrees = agrees and have is None
|
||||||
|
else:
|
||||||
|
agrees = agrees and have is not None and abs(float(have) * 1000.0 - want) <= 1e-6
|
||||||
|
if not agrees:
|
||||||
|
pipe_mismatch.append({"run": run["id"], "want": want, "have": have})
|
||||||
|
rows = {row["tag"]: row for row in cable_rows(pid, layout)["rows"]}
|
||||||
|
cable_mismatch = [
|
||||||
|
want["tag"]
|
||||||
|
for want in truth["cable_rows"]
|
||||||
|
if rows.get(want["tag"], {}).get("cable_m") != want["roundup_m"]
|
||||||
|
or rows.get(want["tag"], {}).get("panel") != want["panel"]
|
||||||
|
]
|
||||||
|
return {
|
||||||
|
"scale_verdict": scale["verdict"],
|
||||||
|
"within_10pct": scale.get("within_10pct"),
|
||||||
|
"runs_checked": len(truth["pipe_runs"]),
|
||||||
|
"pipe_mismatch": pipe_mismatch,
|
||||||
|
"loads_checked": len(truth["cable_rows"]),
|
||||||
|
"cable_mismatch": cable_mismatch,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def _smoke(backend, folder: Path) -> dict:
|
||||||
|
from engineering.understand.diff import diff_snapshots
|
||||||
|
from engineering.understand.snapshot import read_snapshot, take_snapshot
|
||||||
|
from tests.fixtures.plant_pair import build_plant_pair
|
||||||
|
|
||||||
|
truth = build_plant_pair(folder)
|
||||||
|
|
||||||
|
pid_key = await _open(backend, truth["pid"])
|
||||||
|
async with _Current(pid_key):
|
||||||
|
started = time.perf_counter()
|
||||||
|
live_pid = await take_snapshot(backend)
|
||||||
|
export_s = time.perf_counter() - started
|
||||||
|
dwg = folder / "pid.dwg"
|
||||||
|
await backend.drawing_save_as(str(dwg), "dwg")
|
||||||
|
# SaveAs renamed the document: it is now pid.dwg, and still the smoke's own.
|
||||||
|
dwg_key = _key(dwg)
|
||||||
|
_OPENED.append(dwg_key)
|
||||||
|
for key in (pid_key, dwg_key):
|
||||||
|
await _close(backend, key)
|
||||||
|
dwg_pid = await take_snapshot(backend, str(dwg))
|
||||||
|
|
||||||
|
layout_key = await _open(backend, truth["layout"])
|
||||||
|
async with _Current(layout_key):
|
||||||
|
live_layout = await take_snapshot(backend)
|
||||||
|
|
||||||
|
takeoffs = _check_takeoffs(truth, _model(live_pid), _model(live_layout))
|
||||||
|
|
||||||
|
tag = next(
|
||||||
|
r
|
||||||
|
for r in live_layout.records
|
||||||
|
if r.type == "TEXT" and r.layer == "TAGS" and r.space == "Model"
|
||||||
|
)
|
||||||
|
async with _Current(layout_key):
|
||||||
|
await backend.entity_move(tag.handle, 10.0, 0.0)
|
||||||
|
after = await take_snapshot(backend)
|
||||||
|
diff = diff_snapshots(_model(live_layout), _model(after))
|
||||||
|
await _close(backend, layout_key)
|
||||||
|
|
||||||
|
file_pid, file_layout = read_snapshot(truth["pid"]), read_snapshot(truth["layout"])
|
||||||
|
return {
|
||||||
|
"export_s": round(export_s, 3),
|
||||||
|
"live_pid_source": live_pid.source,
|
||||||
|
"dwg_pid_source": dwg_pid.source,
|
||||||
|
"model_space": {
|
||||||
|
"pid_file": _types(file_pid),
|
||||||
|
"pid_live": _types(live_pid),
|
||||||
|
"pid_dwg": _types(dwg_pid),
|
||||||
|
"layout_file": _types(file_layout),
|
||||||
|
"layout_live": _types(live_layout),
|
||||||
|
},
|
||||||
|
"paper_space_layouts": sorted({r.space for r in live_layout.records} - {"Model"}),
|
||||||
|
"takeoffs": takeoffs,
|
||||||
|
"diff": {
|
||||||
|
"moved": tag.handle,
|
||||||
|
"added": len(diff["added"]),
|
||||||
|
"removed": len(diff["removed"]),
|
||||||
|
"changed": len(diff["changed"]),
|
||||||
|
"moved_is_the_change": f'"{tag.handle}"' in json.dumps(diff["changed"]),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
async def main(keep_document: bool = False) -> int:
|
||||||
|
try:
|
||||||
|
from backends.com_backend import ComBackend
|
||||||
|
except ImportError as exc:
|
||||||
|
print(f"pywin32 not available: {exc}")
|
||||||
|
return 2
|
||||||
|
backend = ComBackend()
|
||||||
|
try:
|
||||||
|
await backend.connect()
|
||||||
|
except Exception as exc:
|
||||||
|
print(f"no live AutoCAD reachable: {exc}")
|
||||||
|
return 2
|
||||||
|
_patient(backend)
|
||||||
|
folder = Path(tempfile.mkdtemp(prefix="acadmcp_understand_smoke_"))
|
||||||
|
_STATE["folder"] = str(folder)
|
||||||
|
try:
|
||||||
|
version = await _attach(backend)
|
||||||
|
if version is None:
|
||||||
|
print("no live AutoCAD reachable: no running instance to attach to")
|
||||||
|
return 2
|
||||||
|
report = {"autocad": version, **await _smoke(backend, folder)}
|
||||||
|
print(json.dumps(report, indent=2, ensure_ascii=False))
|
||||||
|
spaces = report["model_space"]
|
||||||
|
ok = (
|
||||||
|
spaces["pid_live"] == spaces["pid_file"] == spaces["pid_dwg"]
|
||||||
|
and spaces["layout_live"] == spaces["layout_file"]
|
||||||
|
and report["takeoffs"]["scale_verdict"] == "schematic"
|
||||||
|
and not report["takeoffs"]["pipe_mismatch"]
|
||||||
|
and not report["takeoffs"]["cable_mismatch"]
|
||||||
|
and report["diff"]["changed"] == 1
|
||||||
|
and report["diff"]["added"] == report["diff"]["removed"] == 0
|
||||||
|
and report["diff"]["moved_is_the_change"]
|
||||||
|
)
|
||||||
|
return 0 if ok else 1
|
||||||
|
finally:
|
||||||
|
_STATE["expect"] = None
|
||||||
|
if not keep_document:
|
||||||
|
await _close_leftovers(backend)
|
||||||
|
await _reactivate_start(backend)
|
||||||
|
_STATE["start"] = None
|
||||||
|
await backend.disconnect()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(asyncio.run(main(keep_document="--keep" in sys.argv[1:])))
|
||||||
@@ -49,7 +49,7 @@ def test_correctness_suite_has_the_settings_checks():
|
|||||||
):
|
):
|
||||||
assert CHECKS[name][1] == "Settings"
|
assert CHECKS[name][1] == "Settings"
|
||||||
# v4 left the suite at 32; tracks B + G grew it to 39 (tests/test_benchmark_v5.py).
|
# v4 left the suite at 32; tracks B + G grew it to 39 (tests/test_benchmark_v5.py).
|
||||||
assert len(CHECKS) == 43
|
assert len(CHECKS) == 48
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio # asyncio_mode is strict; the four tests above are sync
|
@pytest.mark.asyncio # asyncio_mode is strict; the four tests above are sync
|
||||||
|
|||||||
@@ -48,7 +48,7 @@ def test_the_suite_carries_the_mechanical_and_sheet_checks():
|
|||||||
assert CHECKS[name][1] == "Mechanical"
|
assert CHECKS[name][1] == "Mechanical"
|
||||||
assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet"
|
assert CHECKS["sheet_frame_iso5457_a3"][1] == "Sheet"
|
||||||
assert CHECKS["bom_balloon_link"][1] == "Sheet"
|
assert CHECKS["bom_balloon_link"][1] == "Sheet"
|
||||||
assert len(CHECKS) == 43
|
assert len(CHECKS) == 48
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
|
|||||||
@@ -52,7 +52,7 @@ def test_the_suite_carries_the_four_architectural_checks():
|
|||||||
"arch_rooms_detect_foreign",
|
"arch_rooms_detect_foreign",
|
||||||
):
|
):
|
||||||
assert CHECKS[name][1] == "Architecture"
|
assert CHECKS[name][1] == "Architecture"
|
||||||
assert len(CHECKS) == 43
|
assert len(CHECKS) == 48
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
|
|||||||
@@ -0,0 +1,136 @@
|
|||||||
|
"""The v7 task matrix: v6 plus what track H added.
|
||||||
|
|
||||||
|
``takeoff_roundtrip`` and ``understand_foreign`` are track H's evidence, both
|
||||||
|
gated on the synthetic plant pair (tests/fixtures/plant_pair.py) whose truth
|
||||||
|
is computed from the positions it placed. Each gate is shown to fail on a
|
||||||
|
broken input - a gate that cannot fail is not evidence. The five correctness
|
||||||
|
checks run here once too, so a regression fails the suite and not only the
|
||||||
|
A/B report.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ezdxf
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from benchmarks.tasks_v6 import TASKS_V6
|
||||||
|
from benchmarks.tasks_v7 import DEFAULT_MATRIX, MATRICES, NEW_TASKS_V7, TASKS_V7, task_by_id
|
||||||
|
from tests.fixtures import plant_pair
|
||||||
|
|
||||||
|
UNDERSTANDING_CHECKS = (
|
||||||
|
"scale_check_detects_schematic",
|
||||||
|
"pipe_takeoff_rmst_exact",
|
||||||
|
"cable_takeoff_roundup",
|
||||||
|
"diff_detects_known_edits",
|
||||||
|
"topology_known_defects",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_v7_extends_v6_without_touching_it():
|
||||||
|
assert TASKS_V7[: len(TASKS_V6)] == TASKS_V6
|
||||||
|
assert [t.task_id for t in NEW_TASKS_V7] == ["takeoff_roundtrip", "understand_foreign"]
|
||||||
|
assert len(TASKS_V7) == 21
|
||||||
|
assert MATRICES["v6"] == TASKS_V6 and MATRICES["v7"] == TASKS_V7
|
||||||
|
assert DEFAULT_MATRIX == "v7"
|
||||||
|
assert task_by_id("takeoff_roundtrip").category == "plant"
|
||||||
|
assert task_by_id("understand_foreign").category == "understand"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_adapter_implements_every_v7_task():
|
||||||
|
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
||||||
|
|
||||||
|
for task in TASKS_V7:
|
||||||
|
assert hasattr(AutoCADMCPProAdapter, f"_task_{task.task_id}"), task.task_id
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_suite_carries_the_five_understanding_checks():
|
||||||
|
from benchmarks.correctness_suite import CHECKS
|
||||||
|
|
||||||
|
for name in UNDERSTANDING_CHECKS:
|
||||||
|
assert CHECKS[name][1] == "Understanding"
|
||||||
|
assert len(CHECKS) == 48
|
||||||
|
|
||||||
|
|
||||||
|
async def _run(task_id, tmp_path):
|
||||||
|
from benchmarks.adapters.autocad_mcp_pro import AutoCADMCPProAdapter
|
||||||
|
|
||||||
|
adapter = AutoCADMCPProAdapter(backend="ezdxf")
|
||||||
|
await adapter.setup(tmp_path)
|
||||||
|
try:
|
||||||
|
await adapter._reset()
|
||||||
|
return await getattr(adapter, f"_task_{task_id}")()
|
||||||
|
finally:
|
||||||
|
await adapter.cleanup()
|
||||||
|
|
||||||
|
|
||||||
|
def _broken(edit):
|
||||||
|
"""build_plant_pair, then ``edit(truth)`` on the files it wrote."""
|
||||||
|
real = plant_pair.build_plant_pair
|
||||||
|
|
||||||
|
def build(directory):
|
||||||
|
truth = real(directory)
|
||||||
|
edit(truth)
|
||||||
|
return truth
|
||||||
|
|
||||||
|
return build
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_takeoff_roundtrip_passes_headlessly(tmp_path):
|
||||||
|
passed, metrics, _artifacts = await _run("takeoff_roundtrip", tmp_path)
|
||||||
|
assert passed, metrics
|
||||||
|
assert metrics["scale_verdict"] == "schematic"
|
||||||
|
assert metrics["runs_checked"] == 9 and metrics["loads_checked"] == 4
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_takeoff_roundtrip_fails_loudly_without_the_wiring_callouts(tmp_path, monkeypatch):
|
||||||
|
"""Every callout gone from the layout: no load has a panel any more, so no
|
||||||
|
cable length can be measured - and the gate names all four loads, while the
|
||||||
|
pipe half, untouched, still agrees."""
|
||||||
|
|
||||||
|
def drop_callouts(truth):
|
||||||
|
doc = ezdxf.readfile(truth["layout"])
|
||||||
|
msp = doc.modelspace()
|
||||||
|
for entity in list(msp.query("MULTILEADER")):
|
||||||
|
msp.delete_entity(entity)
|
||||||
|
doc.saveas(truth["layout"])
|
||||||
|
|
||||||
|
monkeypatch.setattr(plant_pair, "build_plant_pair", _broken(drop_callouts))
|
||||||
|
passed, metrics, _artifacts = await _run("takeoff_roundtrip", tmp_path)
|
||||||
|
assert passed is False
|
||||||
|
assert sorted(row["tag"] for row in metrics["cable_mismatch"]) == ["M11", "M12", "T102", "T202"]
|
||||||
|
assert metrics["pipe_mismatch"] == []
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_understand_foreign_passes_headlessly(tmp_path):
|
||||||
|
passed, metrics, _artifacts = await _run("understand_foreign", tmp_path)
|
||||||
|
assert passed, metrics
|
||||||
|
assert metrics["clusters"] == 2 and metrics["misread_layers"] == {}
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_understand_foreign_fails_loudly_when_the_units_tell_the_truth(tmp_path, monkeypatch):
|
||||||
|
"""The same layout declaring millimetres: there is no disagreement left to
|
||||||
|
flag, so the units gate fails - and only that gate."""
|
||||||
|
|
||||||
|
def honest_units(truth):
|
||||||
|
doc = ezdxf.readfile(truth["layout"])
|
||||||
|
doc.header["$INSUNITS"] = 4
|
||||||
|
doc.saveas(truth["layout"])
|
||||||
|
|
||||||
|
monkeypatch.setattr(plant_pair, "build_plant_pair", _broken(honest_units))
|
||||||
|
passed, metrics, _artifacts = await _run("understand_foreign", tmp_path)
|
||||||
|
assert passed is False
|
||||||
|
assert metrics["units_flagged"] is False
|
||||||
|
assert metrics["outlier_found"] and metrics["copies_separated"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
@pytest.mark.parametrize("name", UNDERSTANDING_CHECKS)
|
||||||
|
async def test_each_understanding_check_passes(name):
|
||||||
|
from benchmarks.correctness_suite import CHECKS
|
||||||
|
|
||||||
|
check, _category = CHECKS[name]
|
||||||
|
assert await check() is True
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
"""scripts/field_test_takeoff.py never writes into the repository and never prints a quantity.
|
||||||
|
|
||||||
|
The script exists for client drawings, which never enter this repository
|
||||||
|
(spec §14). It is exercised here on the synthetic plant pair only - the field
|
||||||
|
test itself is never run in CI.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from tests.fixtures.plant_pair import build_plant_pair
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def _script():
|
||||||
|
spec = importlib.util.spec_from_file_location(
|
||||||
|
"field_test_takeoff", ROOT / "scripts" / "field_test_takeoff.py"
|
||||||
|
)
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_output_folder_inside_the_repository_is_refused_before_anything_runs(tmp_path, capsys):
|
||||||
|
truth = build_plant_pair(tmp_path)
|
||||||
|
target = ROOT / "field_test_output_must_not_exist"
|
||||||
|
code = _script().main(
|
||||||
|
["--pid", truth["pid"], "--layout", truth["layout"], "--out", str(target)]
|
||||||
|
)
|
||||||
|
assert code == 2
|
||||||
|
assert not target.exists()
|
||||||
|
assert "refused" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_missing_drawing_is_refused(tmp_path, capsys):
|
||||||
|
code = _script().main(
|
||||||
|
["--pid", str(tmp_path / "none.dxf"), "--layout", str(tmp_path / "none.dxf")]
|
||||||
|
)
|
||||||
|
assert code == 2
|
||||||
|
assert "does not exist" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_report_lands_next_to_the_pid_and_the_console_carries_no_quantity(tmp_path, capsys):
|
||||||
|
truth = build_plant_pair(tmp_path)
|
||||||
|
code = _script().main(["--pid", truth["pid"], "--layout", truth["layout"]])
|
||||||
|
assert code == 0
|
||||||
|
summary = json.loads(capsys.readouterr().out)
|
||||||
|
assert summary["scale_verdict"] == "schematic"
|
||||||
|
assert Path(summary["report"]).parent == tmp_path
|
||||||
|
assert all(Path(path).parent == tmp_path for path in summary["workbooks"])
|
||||||
|
printed = json.dumps({k: v for k, v in summary.items() if k not in ("report", "workbooks")})
|
||||||
|
for run in truth["pipe_runs"]:
|
||||||
|
if run["layout_length_mm"] is not None:
|
||||||
|
assert f"{run['layout_length_mm']:g}" not in printed, run["id"]
|
||||||
|
for row in truth["cable_rows"]:
|
||||||
|
assert f"{row['length_mm']:g}" not in printed, row["tag"]
|
||||||
|
report = json.loads(Path(summary["report"]).read_text(encoding="utf-8"))
|
||||||
|
assert report["scale"]["verdict"] == "schematic"
|
||||||
|
|
||||||
|
|
||||||
|
def test_show_prints_the_rows_on_request(tmp_path, capsys):
|
||||||
|
truth = build_plant_pair(tmp_path)
|
||||||
|
code = _script().main(["--pid", truth["pid"], "--layout", truth["layout"], "--show"])
|
||||||
|
assert code == 0
|
||||||
|
assert '"cable"' in capsys.readouterr().out
|
||||||
@@ -190,3 +190,30 @@ def test_the_synthetic_pid_is_schematic_against_its_layout(tmp_path):
|
|||||||
assert len(result["matched"]) >= 3
|
assert len(result["matched"]) >= 3
|
||||||
# The layout's two plan copies: the check must pick one of them.
|
# The layout's two plan copies: the check must pick one of them.
|
||||||
assert result["scope"]["cluster_b"] is not None
|
assert result["scope"]["cluster_b"] is not None
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_wiring_callout_is_not_a_second_place_for_its_panel():
|
||||||
|
"""``Wiring to CP-2`` names the panel it points at; it is not written at the
|
||||||
|
panel. Read as a tag it would make every panel with a callout ambiguous
|
||||||
|
(and ``Wiring to CP1`` a phantom tag ``CP1``), so the callout takes no part
|
||||||
|
- the same rule the line-network reader (`network.tag_occurrences`) keeps."""
|
||||||
|
plan = {**PLAN, "CP-2": (3000.0, 15000.0)}
|
||||||
|
callouts = [
|
||||||
|
text("W1", "Wiring to CP-2", 12000.0, 0.0),
|
||||||
|
text("W2", "Wiring to CP1", 0.0, 9000.0),
|
||||||
|
]
|
||||||
|
result = scale_check(snapshot(tags(plan, "A")), snapshot(tags(plan, "B") + callouts))
|
||||||
|
assert result["ambiguous"] == []
|
||||||
|
assert "CP-2" in {row["tag"] for row in result["matched"]}
|
||||||
|
assert result["only_b"] == []
|
||||||
|
assert result["pair_count"] == math.comb(len(plan), 2)
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_synthetic_pair_reproduces_the_generators_own_statistics(tmp_path):
|
||||||
|
"""The generator computes its truth from the positions it placed: 11 shared
|
||||||
|
tags, 55 pairs, 21.8 % within +/-10 % of the median. The check must see the
|
||||||
|
same tags - a wiring callout counted as a panel would drop to 45."""
|
||||||
|
truth = build_plant_pair(tmp_path)
|
||||||
|
result = scale_check(read_snapshot(truth["pid"]), read_snapshot(truth["layout"]))
|
||||||
|
assert result["pair_count"] == truth["scale"]["overall"]["pairs"] == 55
|
||||||
|
assert result["within_10pct"] == pytest.approx(truth["scale"]["overall"]["within_10pct"])
|
||||||
|
|||||||
Reference in New Issue
Block a user