bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks
tasks_v4.py extends v3 with one task the reader can fail on its own: the example spec is drawn through pid_from_spec's code and read back by the graph builder, which never sees the spec. The runner, the chart and the doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3" explicitly now that the default has moved. The plan's adapter check expected the two unnumbered lines to come back with no line number; draw_line builds one from the default number_format (bare sequence when size/service/spec/insulation are empty), so the task asserts the two authored numbers on the two process lines and sequence- built XDATA numbers on the other two instead. The published A/B report still covers 26 checks; Task 21 re-runs it against v1.5.1, which is what clears the remaining doc-number drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
co-authored by
Claude Opus 5
parent
e911a3a8fb
commit
8d9c444b76
@@ -73,7 +73,7 @@ def derived() -> dict[str, object]:
|
||||
}
|
||||
|
||||
from benchmarks import correctness_suite
|
||||
from benchmarks.tasks_v3 import TASKS_V3
|
||||
from benchmarks.tasks_v4 import TASKS_V4
|
||||
|
||||
return {
|
||||
"collected_tests": _collected_tests(),
|
||||
@@ -87,7 +87,7 @@ def derived() -> dict[str, object]:
|
||||
"ab_fixed": ab["fixed"],
|
||||
"ab_regressed": ab["regressed"],
|
||||
"suite_checks": len(correctness_suite.CHECKS),
|
||||
"v3_tasks": len(TASKS_V3),
|
||||
"v4_tasks": len(TASKS_V4),
|
||||
"perf_ms": perf,
|
||||
}
|
||||
|
||||
@@ -158,7 +158,7 @@ CHECKS: list[tuple[str, str, str, str]] = [
|
||||
"ab_fixed",
|
||||
),
|
||||
("CLAUDE.md", "A/B suite size", r"correctness_suite\.py` \((\d+) checks\)", "ab_total"),
|
||||
("CLAUDE.md", "v3 task count", r"the current set \((\d+) tasks", "v3_tasks"),
|
||||
("CLAUDE.md", "v4 task count", r"the current set \((\d+) tasks", "v4_tasks"),
|
||||
]
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user