Umut CANandClaude Opus 5
18c7c69da6
A fresh Layout1 is already ISO A3 landscape (ezdxf R2018 default: 420 x 297, plot_rotation 0), so an A3-only /MediaBox read-back passed with page_setup_apply stubbed to a no-op: the A/B gate row and the v4 matrix task could not fail on the setter. Both now apply ANSI B (432 x 279, a size no fresh document has; spec section 9 names it) and read that PDF first, then apply ISO A3, read its PDF, and require the setter's changed.size_mm to have moved from B to A3. The stub fails at the first PDF. The moved-sheet comparison is +/-0.5 mm, the spec's tolerance, not exact: measured live on AutoCAD 2026 (scratch Documents.Add(), closed unsaved), the COM engine reports ANSI B as 431.8 x 279.4 and the DWG To PDF.pc3 file's MediaBox as 431.8 x 279.4 then 420.158 x 297.039 for A3; the task passes there with changed.size_mm [[431.8, 279.4], [420.0, 297.0]]. Headless it is [[432.0, 279.0], [420.0, 297.0]]. tests/test_benchmark_v4.py gains the mutation test the finding describes (setter stubbed -> both rows must fail; both passed on the old code) and a run of the gate row itself; the headless task test asserts the B stage. README rows and the tasks_v4 docstring say what the task now does; the README collected-tests figure follows the two new tests (3282 -> 3284). The regenerated A/B report against v1.5.1 is byte-identical (26 -> 32/32). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
53 lines
1.9 KiB
Python
53 lines
1.9 KiB
Python
"""The v4 task matrix: v3 plus what 1.6 added.
|
||
|
||
``pid_roundtrip`` (track A) draws a five-item P&ID through the same code the
|
||
tools call and reads it back: the edge count, the absence of dangling ends
|
||
and the instrument index are each derived independently of the drawing code.
|
||
|
||
``page_setup_truth`` (track E) applies an ISO A3 landscape page setup to a
|
||
layout, plots it through ``batch_plot`` and reads the sheet size back from
|
||
the PDF's own ``/MediaBox`` — the setup is verified by the file, not by the
|
||
setter's return value. A fresh Layout1 is already A3 landscape, so the task
|
||
moves the sheet to ANSI B (432 × 279) and reads that PDF first; a setter that
|
||
writes nothing fails there, not at the A3 read-back.
|
||
|
||
The published competitor reports were never asked either question; the chart
|
||
shows ``not_run`` for them, not zero.
|
||
"""
|
||
|
||
from __future__ import annotations
|
||
|
||
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
|
||
from benchmarks.tasks_v3 import TASKS_V3
|
||
|
||
#: Added in v1.6.0. Each can fail on its own.
|
||
NEW_TASKS_V4: tuple[TaskSpec, ...] = (
|
||
TaskSpec(
|
||
"pid_roundtrip",
|
||
"pid",
|
||
"Draw a P&ID from a spec and read back a matching graph, index and line list",
|
||
1.0,
|
||
),
|
||
TaskSpec(
|
||
"page_setup_truth",
|
||
"pagesetup",
|
||
"Apply an ISO A3 page setup, plot it, and read the sheet size back from the PDF",
|
||
1.0,
|
||
),
|
||
)
|
||
|
||
TASKS_V4: tuple[TaskSpec, ...] = TASKS_V3 + NEW_TASKS_V4
|
||
|
||
#: Runner-selectable task sets. v2 and v3 stay addressable so an earlier
|
||
#: report can be reproduced exactly rather than only described.
|
||
MATRICES: dict[str, tuple[TaskSpec, ...]] = {"v2": TASKS_V2, "v3": TASKS_V3, "v4": TASKS_V4}
|
||
|
||
DEFAULT_MATRIX = "v4"
|
||
|
||
|
||
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
||
for task in MATRICES[matrix]:
|
||
if task.task_id == task_id:
|
||
return task
|
||
raise KeyError(f"{task_id} is not in matrix {matrix}")
|