Umut CANandClaude Opus 5.5
0abbe55041
The evidence for track F, in the shape tracks A, E and B + G used. - benchmarks/tasks_v6.py: 19 tasks, DEFAULT_MATRIX = "v6"; v5 to v2 stay addressable. arch_roundtrip draws the example plan with arch_plan_from_spec and reads it back without the spec: rooms_detect must find each labelled room within 0.1 % of its label and of the net floor computed by hand (4825 x 5750 and 3825 x 5750), the schedules must list D1, D2, W1, W2, and the plan is gated on a zero-issue critique and a finalize score of at least 90. A test removes one room label and the gate fails naming arch_room_unlabelled. - Four correctness checks, every number worked out by hand in the docstring: L / T / X junction corners and wall areas, a 4 x 5 m room labelled 20.00 m2 rather than its 21.84 m2 axis area, a door interrupting both faces, and a plan of plain lines read as 20 and 15 m2 at confidence 0.6. The published A/B report is re-run over 43 checks: 26 / 43 -> 43 / 43, zero regressed. - scripts/smoke_arch_com.py runs the whole plan once on a live seat and closes every document it opens; the CHANGELOG records what it returned. - The token suite measures a TOOL_PACKS=core,arch row; check_doc_numbers derives the task count from the v6 matrix; CLAUDE.md, both READMEs and the spec status line say what shipped. - fix(com): the first live smoke exited 1 - the room reader found 0 wall segments. Measured on AutoCAD 2026 (2026-09-23): with a makepy cache, ModelSpace.Item(i) returns the gen_py IAcadEntity wrapper (type(item).__name__ == "IAcadEntity") and item.StartPoint on an AcDbLine raises AttributeError, so every entity_list row lost its type-specific properties (28 wall LINEs, 0 with start/end). _entity_info now re-dispatches exactly that wrapper through _com_unnarrow; after the fix 38 of 53 rows carry start/end and the smoke exits 0 with both rooms at 27 743 750 / 21 993 750 mm2 and 0 critique issues. Re-dispatch measured at ~1.7 ms per entity (dynamic vs gencache alike). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
55 lines
1.9 KiB
Python
55 lines
1.9 KiB
Python
"""The v6 task matrix: v5 plus what track F added.
|
|
|
|
``arch_roundtrip`` (track F) is the roadmap evidence for the plan model: a
|
|
two-room plan - an entrance door, an interior door, two windows and a stair -
|
|
drawn by ``arch_plan_from_spec``, then read back by ``arch_rooms_detect``,
|
|
which never sees the spec. Each room's detected area has to agree with the area
|
|
its label carries within 0.1 %, and both with the net floor computed by hand;
|
|
the door and window schedules have to list the drawn tags; and the sheet is
|
|
gated on the same two numbers as ``mech_assembly``: ``drawing_critique
|
|
(focus=None)`` returns zero issues and the finalize score is at least 90.
|
|
|
|
The published competitor reports were never asked this question; the chart
|
|
shows ``not_run`` for them, not zero.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from benchmarks.tasks_v2 import TASKS_V2
|
|
from benchmarks.tasks_v3 import TASKS_V3
|
|
from benchmarks.tasks_v4 import TASKS_V4
|
|
from benchmarks.tasks_v5 import TASKS_V5, TaskSpec
|
|
|
|
#: Added in v1.6.0 by track F.
|
|
NEW_TASKS_V6: tuple[TaskSpec, ...] = (
|
|
TaskSpec(
|
|
"arch_roundtrip",
|
|
"arch",
|
|
"Draw a two-room plan from one spec, read its rooms back from the drawing "
|
|
"within 0.1 %, schedule its tags, with a clean critique and a finalize score "
|
|
"of at least 90",
|
|
1.0,
|
|
),
|
|
)
|
|
|
|
TASKS_V6: tuple[TaskSpec, ...] = TASKS_V5 + NEW_TASKS_V6
|
|
|
|
#: Runner-selectable task sets. v2 to v5 stay addressable so an earlier report
|
|
#: can be reproduced exactly rather than only described.
|
|
MATRICES: dict[str, tuple[TaskSpec, ...]] = {
|
|
"v2": TASKS_V2,
|
|
"v3": TASKS_V3,
|
|
"v4": TASKS_V4,
|
|
"v5": TASKS_V5,
|
|
"v6": TASKS_V6,
|
|
}
|
|
|
|
DEFAULT_MATRIX = "v6"
|
|
|
|
|
|
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
|
for task in MATRICES[matrix]:
|
|
if task.task_id == task_id:
|
|
return task
|
|
raise KeyError(f"{task_id} is not in matrix {matrix}")
|