bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks
tasks_v4.py extends v3 with one task the reader can fail on its own: the example spec is drawn through pid_from_spec's code and read back by the graph builder, which never sees the spec. The runner, the chart and the doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3" explicitly now that the default has moved. The plan's adapter check expected the two unnumbered lines to come back with no line number; draw_line builds one from the default number_format (bare sequence when size/service/spec/insulation are empty), so the task asserts the two authored numbers on the two process lines and sequence- built XDATA numbers on the other two instead. The published A/B report still covers 26 checks; Task 21 re-runs it against v1.5.1, which is what clears the remaining doc-number drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
co-authored by
Claude Opus 5
parent
e911a3a8fb
commit
8d9c444b76
+17
-4
@@ -41,12 +41,13 @@ Unsupported tasks remain in the fixed-matrix denominator with score zero,
|
||||
preventing partial implementations from receiving an inflated score.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --list --matrix v3
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v3 --json
|
||||
python -m benchmarks.run_competitors --list --matrix v4
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v4 --json
|
||||
python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
|
||||
```
|
||||
|
||||
The release-machine ezdxf self-check is **15/15 (100.0)** on the v3 matrix.
|
||||
The release-machine ezdxf self-check is **16/16 (100.0)** on the v4 matrix
|
||||
(`--matrix v3` / `--matrix v2` reproduce the earlier sets).
|
||||
Repository stars and raw tool counts do not contribute to the score. Adapter
|
||||
registration lives in `competitors.yaml`.
|
||||
|
||||
@@ -86,6 +87,18 @@ zero we invented would be indistinguishable from a zero we measured.
|
||||
only described, and `--publish` writes the artifact-path-free file that
|
||||
`results/published/` holds.
|
||||
|
||||
### Matrix v4 (v1.6) — the P&ID round trip
|
||||
|
||||
`tasks_v4.py` keeps the v3 fifteen unchanged and adds one task the reader can
|
||||
fail on its own:
|
||||
|
||||
| Task | Category | Verified against |
|
||||
|---|---|---|
|
||||
| `pid_roundtrip` | pid | the example spec drawn through `pid_from_spec`'s code, then read back by the graph builder, which never sees the spec: 5 nodes, 4 edges, 0 dangling ends, `confidence_min` 1.0, zero critique issues, an instrument index of exactly `FIC-101` wired to `FCV-101`, and the two authored line numbers back on the two process lines while the two unnumbered lines carry sequence-built numbers from XDATA |
|
||||
|
||||
The competitor reports carry no result for it, for the same reason they carry
|
||||
none for the v3 five. `--matrix v3` reproduces the v1.5 set exactly.
|
||||
|
||||
## Live competitor lane (v1.4)
|
||||
|
||||
Two competitor adapters now execute the exact same task matrix, black-box over
|
||||
@@ -203,7 +216,7 @@ python benchmarks/compare_versions.py --json results.json
|
||||
|
||||
### Result — v1.5.1 vs v1.5.0 (release gate)
|
||||
|
||||
26 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
29 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
[`results/published/ab-v1.5.0-vs-v1.5.1.json`](results/published/ab-v1.5.0-vs-v1.5.1.json).
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|
||||
Reference in New Issue
Block a user