anatoly/autocad-mcp-pro
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks

Browse Source 
tasks_v4.py extends v3 with one task the reader can fail on its own: the
example spec is drawn through pid_from_spec's code and read back by the
graph builder, which never sees the spec. The runner, the chart and the
doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier
sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3"
explicitly now that the default has moved.

The plan's adapter check expected the two unnumbered lines to come back
with no line number; draw_line builds one from the default number_format
(bare sequence when size/service/spec/insulation are empty), so the task
asserts the two authored numbers on the two process lines and sequence-
built XDATA numbers on the other two instead.

The published A/B report still covers 26 checks; Task 21 re-runs it
against v1.5.1, which is what clears the remaining doc-number drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-16 13:06:52 +03:00
co-authored by Claude Opus 5
parent e911a3a8fb
commit 8d9c444b76
11 changed files with 192 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_benchmark_v3.py
+1 -1
View File
@@ -100,7 +100,7 @@ async def test_each_new_task_passes_on_the_headless_backend(task, tmp_path):
@pytest.mark.asyncio
async def test_the_full_v3_matrix_runs_green(tmp_path):
report = await run_tasks(AutoCADMCPProAdapter(), TASKS_V3, tmp_path, timeout=180.0)
report = await run_tasks(AutoCADMCPProAdapter(), TASKS_V3, tmp_path, timeout=180.0, matrix="v3")
assert report["summary"]["attempted"] == 15
assert report["summary"]["passed"] == 15, [