bench: matrix v6 adds arch_roundtrip; four architectural correctness checks; the live arch smoke

The evidence for track F, in the shape tracks A, E and B + G used.

- benchmarks/tasks_v6.py: 19 tasks, DEFAULT_MATRIX = "v6"; v5 to v2 stay
  addressable. arch_roundtrip draws the example plan with
  arch_plan_from_spec and reads it back without the spec: rooms_detect must
  find each labelled room within 0.1 % of its label and of the net floor
  computed by hand (4825 x 5750 and 3825 x 5750), the schedules must list
  D1, D2, W1, W2, and the plan is gated on a zero-issue critique and a
  finalize score of at least 90. A test removes one room label and the gate
  fails naming arch_room_unlabelled.
- Four correctness checks, every number worked out by hand in the docstring:
  L / T / X junction corners and wall areas, a 4 x 5 m room labelled 20.00 m2
  rather than its 21.84 m2 axis area, a door interrupting both faces, and a
  plan of plain lines read as 20 and 15 m2 at confidence 0.6. The published
  A/B report is re-run over 43 checks: 26 / 43 -> 43 / 43, zero regressed.
- scripts/smoke_arch_com.py runs the whole plan once on a live seat and
  closes every document it opens; the CHANGELOG records what it returned.
- The token suite measures a TOOL_PACKS=core,arch row; check_doc_numbers
  derives the task count from the v6 matrix; CLAUDE.md, both READMEs and
  the spec status line say what shipped.
- fix(com): the first live smoke exited 1 - the room reader found 0 wall
  segments. Measured on AutoCAD 2026 (2026-09-23): with a makepy cache,
  ModelSpace.Item(i) returns the gen_py IAcadEntity wrapper
  (type(item).__name__ == "IAcadEntity") and item.StartPoint on an
  AcDbLine raises AttributeError, so every entity_list row lost its
  type-specific properties (28 wall LINEs, 0 with start/end). _entity_info
  now re-dispatches exactly that wrapper through _com_unnarrow; after the
  fix 38 of 53 rows carry start/end and the smoke exits 0 with both rooms
  at 27 743 750 / 21 993 750 mm2 and 0 critique issues. Re-dispatch
  measured at ~1.7 ms per entity (dynamic vs gencache alike).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-23 23:35:46 +03:00
co-authored by Claude Opus 5.5
parent 6e004cc658
commit 0abbe55041
20 changed files with 952 additions and 53 deletions
+14 -4
View File
@@ -117,6 +117,16 @@ the live seat a missing `MECH` layer and a parts list that could not see a
`BLOCKREFERENCE`; each is fixed with a test. The competitor reports carry no
result for it.
### Matrix v6 (v1.6) — the architectural plan
`tasks_v6.py` keeps the v5 eighteen unchanged and adds track F's evidence:
| Task | Category | Verified against |
|---|---|---|
| `arch_roundtrip` | arch | the plan of `engineering/arch/spec.py::EXAMPLE_SPEC` — a 9 × 6 m brick ring at 250 mm split by a 100 mm AAC wall, an entrance door, an interior door, two windows, a straight stair, two labelled rooms, the exterior chains and the door / window / room schedules — drawn in one transaction by `draw_plan_from_spec`, then read back without the spec: `rooms_detect` on the wall layer must find each labelled room within 0.1 % of the area its label carries **and** of the net floor computed by hand (4825 × 5750 = 27 743 750 and 3825 × 5750 = 21 993 750 mm²), `schedule_rows` must list exactly D1, D2 and W1, W2, and the saved plan is gated like `mech_assembly`: `run_critique(focus=None)` returns **zero** issues and `combine(validator, critique)` scores at least **90**. `tests/test_benchmark_v6.py` removes one room label to prove the gate fails and names `arch_room_unlabelled` |
The competitor reports carry no result for it.
## Live competitor lane (v1.4)
Two competitor adapters now execute the exact same task matrix, black-box over
@@ -234,15 +244,15 @@ python benchmarks/compare_versions.py --json results.json
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
39 checks, ezdxf backend, one subprocess per check. Machine-readable report:
43 checks, ezdxf backend, one subprocess per check. Machine-readable report:
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
| Version | Checks passing | Pass rate | Fixed | Regressed |
|---------|----------------|-----------|-------|-----------|
| **v1.5.1** (baseline) | 26 / 39 | 66.7 % | — | — |
| **v1.6.0-dev** (this branch) | 39 / 39 | 100 % | 13 | **0** |
| **v1.5.1** (baseline) | 26 / 43 | 60.5 % | — | — |
| **v1.6.0-dev** (this branch) | 43 / 43 | 100 % | 17 | **0** |
The thirteen are all `miss → pass` — v1.5.1 has none of the methods. Seven
The seventeen are all `miss → pass` — v1.5.1 has none of the methods. Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven
are tracks B and G: `mech_part_roundtrip` (a part read back from its own
`ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a
sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces,