bench: matrix v4 adds page_setup_truth; correctness suite grows to 32 with three settings checks (A/B vs v1.5.1: 26 -> 32 / 32)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
co-authored by
Claude Opus 5
parent
02dbb133b5
commit
c2942d3c94
+19
-7
@@ -1,10 +1,16 @@
|
||||
"""The v4 task matrix: v3 plus the P&ID round trip (v1.6, track A).
|
||||
"""The v4 task matrix: v3 plus what 1.6 added.
|
||||
|
||||
``pid_roundtrip`` draws a five-item P&ID through the same code the tools call
|
||||
and reads it back: the edge count, the absence of dangling ends and the
|
||||
instrument index are each derived independently of the drawing code. The
|
||||
published competitor reports were never asked this question; the chart shows
|
||||
``not_run`` for them, not zero.
|
||||
``pid_roundtrip`` (track A) draws a five-item P&ID through the same code the
|
||||
tools call and reads it back: the edge count, the absence of dangling ends
|
||||
and the instrument index are each derived independently of the drawing code.
|
||||
|
||||
``page_setup_truth`` (track E) applies an ISO A3 landscape page setup to a
|
||||
layout, plots it through ``batch_plot`` and reads the sheet size back from
|
||||
the PDF's own ``/MediaBox`` — the setup is verified by the file, not by the
|
||||
setter's return value.
|
||||
|
||||
The published competitor reports were never asked either question; the chart
|
||||
shows ``not_run`` for them, not zero.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -12,7 +18,7 @@ from __future__ import annotations
|
||||
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
|
||||
from benchmarks.tasks_v3 import TASKS_V3
|
||||
|
||||
#: Added in v1.6.0. The reader can fail this one on its own.
|
||||
#: Added in v1.6.0. Each can fail on its own.
|
||||
NEW_TASKS_V4: tuple[TaskSpec, ...] = (
|
||||
TaskSpec(
|
||||
"pid_roundtrip",
|
||||
@@ -20,6 +26,12 @@ NEW_TASKS_V4: tuple[TaskSpec, ...] = (
|
||||
"Draw a P&ID from a spec and read back a matching graph, index and line list",
|
||||
1.0,
|
||||
),
|
||||
TaskSpec(
|
||||
"page_setup_truth",
|
||||
"pagesetup",
|
||||
"Apply an ISO A3 page setup, plot it, and read the sheet size back from the PDF",
|
||||
1.0,
|
||||
),
|
||||
)
|
||||
|
||||
TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4
|
||||
|
||||
Reference in New Issue
Block a user