bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks
tasks_v4.py extends v3 with one task the reader can fail on its own: the example spec is drawn through pid_from_spec's code and read back by the graph builder, which never sees the spec. The runner, the chart and the doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3" explicitly now that the default has moved. The plan's adapter check expected the two unnumbered lines to come back with no line number; draw_line builds one from the default number_format (bare sequence when size/service/spec/insulation are empty), so the task asserts the two authored numbers on the two process lines and sequence- built XDATA numbers on the other two instead. The published A/B report still covers 26 checks; Task 21 re-runs it against v1.5.1, which is what clears the remaining doc-number drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
co-authored by
Claude Opus 5
parent
e911a3a8fb
commit
8d9c444b76
@@ -0,0 +1,38 @@
|
||||
"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A).
|
||||
|
||||
``pid_roundtrip`` draws a five-item P&ID through the same code the tools call
|
||||
and reads it back: the edge count, the absence of dangling ends and the
|
||||
instrument index are each derived independently of the drawing code. The
|
||||
published competitor reports were never asked this question; the chart shows
|
||||
``not_run`` for them, not zero.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
|
||||
from benchmarks.tasks_v3 import TASKS_V3
|
||||
|
||||
#: Added in v1.6.0. The reader can fail this one on its own.
|
||||
NEW_TASKS_V4: tuple[TaskSpec, ...] = (
|
||||
TaskSpec(
|
||||
"pid_roundtrip",
|
||||
"pid",
|
||||
"Draw a P&ID from a spec and read back a matching graph, index and line list",
|
||||
1.0,
|
||||
),
|
||||
)
|
||||
|
||||
TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4
|
||||
|
||||
#: Runner-selectable task sets. v2 and v3 stay addressable so an earlier
|
||||
#: report can be reproduced exactly rather than only described.
|
||||
MATRICES: dict[str, tuple[TaskSpec, ...]] = {"v2": TASKS_V2, "v3": TASKS_V3, "v4": TASKS_V4}
|
||||
|
||||
DEFAULT_MATRIX = "v4"
|
||||
|
||||
|
||||
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
||||
for task in MATRICES[matrix]:
|
||||
if task.task_id == task_id:
|
||||
return task
|
||||
raise KeyError(f"{task_id} is not in matrix {matrix}")
|
||||
Reference in New Issue
Block a user