bench: matrix v4 adds page_setup_truth; correctness suite grows to 32 with three settings checks (A/B vs v1.5.1: 26 -> 32 / 32)

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
2026-09-22 13:07:31 +03:00
co-authored by Claude Opus 5
parent 02dbb133b5
commit c2942d3c94
8 changed files with 230 additions and 40 deletions
+19 -7
View File
@@ -1,10 +1,16 @@
"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A).
"""The v4 task matrix: v3 plus what 1.6 added.
``pid_roundtrip`` draws a five-item P&ID through the same code the tools call
and reads it back: the edge count, the absence of dangling ends and the
instrument index are each derived independently of the drawing code. The
published competitor reports were never asked this question; the chart shows
``not_run`` for them, not zero.
``pid_roundtrip`` (track A) draws a five-item P&ID through the same code the
tools call and reads it back: the edge count, the absence of dangling ends
and the instrument index are each derived independently of the drawing code.
``page_setup_truth`` (track E) applies an ISO A3 landscape page setup to a
layout, plots it through ``batch_plot`` and reads the sheet size back from
the PDF's own ``/MediaBox`` — the setup is verified by the file, not by the
setter's return value.
The published competitor reports were never asked either question; the chart
shows ``not_run`` for them, not zero.
"""
from __future__ import annotations
@@ -12,7 +18,7 @@ from __future__ import annotations
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
from benchmarks.tasks_v3 import TASKS_V3
#: Added in v1.6.0. The reader can fail this one on its own.
#: Added in v1.6.0. Each can fail on its own.
NEW_TASKS_V4: tuple[TaskSpec, ...] = (
TaskSpec(
"pid_roundtrip",
@@ -20,6 +26,12 @@ NEW_TASKS_V4: tuple[TaskSpec, ...] = (
"Draw a P&ID from a spec and read back a matching graph, index and line list",
1.0,
),
TaskSpec(
"page_setup_truth",
"pagesetup",
"Apply an ISO A3 page setup, plot it, and read the sheet size back from the PDF",
1.0,
),
)
TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4