bench: matrix v6 adds arch_roundtrip; four architectural correctness checks; the live arch smoke
The evidence for track F, in the shape tracks A, E and B + G used. - benchmarks/tasks_v6.py: 19 tasks, DEFAULT_MATRIX = "v6"; v5 to v2 stay addressable. arch_roundtrip draws the example plan with arch_plan_from_spec and reads it back without the spec: rooms_detect must find each labelled room within 0.1 % of its label and of the net floor computed by hand (4825 x 5750 and 3825 x 5750), the schedules must list D1, D2, W1, W2, and the plan is gated on a zero-issue critique and a finalize score of at least 90. A test removes one room label and the gate fails naming arch_room_unlabelled. - Four correctness checks, every number worked out by hand in the docstring: L / T / X junction corners and wall areas, a 4 x 5 m room labelled 20.00 m2 rather than its 21.84 m2 axis area, a door interrupting both faces, and a plan of plain lines read as 20 and 15 m2 at confidence 0.6. The published A/B report is re-run over 43 checks: 26 / 43 -> 43 / 43, zero regressed. - scripts/smoke_arch_com.py runs the whole plan once on a live seat and closes every document it opens; the CHANGELOG records what it returned. - The token suite measures a TOOL_PACKS=core,arch row; check_doc_numbers derives the task count from the v6 matrix; CLAUDE.md, both READMEs and the spec status line say what shipped. - fix(com): the first live smoke exited 1 - the room reader found 0 wall segments. Measured on AutoCAD 2026 (2026-09-23): with a makepy cache, ModelSpace.Item(i) returns the gen_py IAcadEntity wrapper (type(item).__name__ == "IAcadEntity") and item.StartPoint on an AcDbLine raises AttributeError, so every entity_list row lost its type-specific properties (28 wall LINEs, 0 with start/end). _entity_info now re-dispatches exactly that wrapper through _com_unnarrow; after the fix 38 of 53 rows carry start/end and the smoke exits 0 with both rooms at 27 743 750 / 21 993 750 mm2 and 0 critique issues. Re-dispatch measured at ~1.7 ms per entity (dynamic vs gencache alike). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
6e004cc658
commit
0abbe55041
+14
-4
@@ -117,6 +117,16 @@ the live seat a missing `MECH` layer and a parts list that could not see a
|
||||
`BLOCKREFERENCE`; each is fixed with a test. The competitor reports carry no
|
||||
result for it.
|
||||
|
||||
### Matrix v6 (v1.6) — the architectural plan
|
||||
|
||||
`tasks_v6.py` keeps the v5 eighteen unchanged and adds track F's evidence:
|
||||
|
||||
| Task | Category | Verified against |
|
||||
|---|---|---|
|
||||
| `arch_roundtrip` | arch | the plan of `engineering/arch/spec.py::EXAMPLE_SPEC` — a 9 × 6 m brick ring at 250 mm split by a 100 mm AAC wall, an entrance door, an interior door, two windows, a straight stair, two labelled rooms, the exterior chains and the door / window / room schedules — drawn in one transaction by `draw_plan_from_spec`, then read back without the spec: `rooms_detect` on the wall layer must find each labelled room within 0.1 % of the area its label carries **and** of the net floor computed by hand (4825 × 5750 = 27 743 750 and 3825 × 5750 = 21 993 750 mm²), `schedule_rows` must list exactly D1, D2 and W1, W2, and the saved plan is gated like `mech_assembly`: `run_critique(focus=None)` returns **zero** issues and `combine(validator, critique)` scores at least **90**. `tests/test_benchmark_v6.py` removes one room label to prove the gate fails and names `arch_room_unlabelled` |
|
||||
|
||||
The competitor reports carry no result for it.
|
||||
|
||||
## Live competitor lane (v1.4)
|
||||
|
||||
Two competitor adapters now execute the exact same task matrix, black-box over
|
||||
@@ -234,15 +244,15 @@ python benchmarks/compare_versions.py --json results.json
|
||||
|
||||
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
|
||||
|
||||
39 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
43 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|---------|----------------|-----------|-------|-----------|
|
||||
| **v1.5.1** (baseline) | 26 / 39 | 66.7 % | — | — |
|
||||
| **v1.6.0-dev** (this branch) | 39 / 39 | 100 % | 13 | **0** |
|
||||
| **v1.5.1** (baseline) | 26 / 43 | 60.5 % | — | — |
|
||||
| **v1.6.0-dev** (this branch) | 43 / 43 | 100 % | 17 | **0** |
|
||||
|
||||
The thirteen are all `miss → pass` — v1.5.1 has none of the methods. Seven
|
||||
The seventeen are all `miss → pass` — v1.5.1 has none of the methods. Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven
|
||||
are tracks B and G: `mech_part_roundtrip` (a part read back from its own
|
||||
`ACADMCP_MECH` XDATA equals the part drawn), `mech_section_hatch_area` (a
|
||||
sleeve 60 long, ⌀40 outside, ⌀20 bore: a full section cuts two 60 × 10 faces,
|
||||
|
||||
Reference in New Issue
Block a user