Umut CANandClaude Opus 5
b03a7615b1
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
95 lines
3.5 KiB
Python
95 lines
3.5 KiB
Python
"""M5 — handle grounding: connecting the picture to the handles.
|
|
|
|
A screenshot shows the model what the drawing looks like and nothing it can act
|
|
on. Every modify tool takes a handle, and there was no way to tell which of the
|
|
hex strings from `entity_list` is the circle in the top-left — so the model
|
|
either guessed or re-derived the mapping from coordinates, which is exactly what
|
|
CLAUDE.md's no-coordinate-guessing rule exists to prevent.
|
|
|
|
The overlay labels each entity with its handle at its own bounding-box centre,
|
|
so "the circle at the top-left" and "handle 2F" become the same statement.
|
|
|
|
Off by default: labels are ink on the drawing, and on a busy sheet they would
|
|
bury the geometry they are meant to explain. When there are more entities than
|
|
labels allowed, the image says so rather than labelling a subset and letting it
|
|
read as the whole picture.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import pytest
|
|
|
|
pytestmark = pytest.mark.asyncio
|
|
|
|
pytest.importorskip("matplotlib", reason="rendering needs the [pdf] extra")
|
|
|
|
|
|
def _png_size(data: bytes) -> tuple[int, int]:
|
|
"""Width/height straight out of the PNG IHDR — no Pillow needed."""
|
|
assert data[:8] == b"\x89PNG\r\n\x1a\n", "not a PNG"
|
|
return int.from_bytes(data[16:20], "big"), int.from_bytes(data[20:24], "big")
|
|
|
|
|
|
async def test_a_plain_screenshot_is_unchanged(backend):
|
|
await backend.entity_create_circle(10, 10, 5)
|
|
png = await backend.view_screenshot()
|
|
|
|
assert png is not None
|
|
assert _png_size(png)[0] > 0
|
|
|
|
|
|
async def test_the_overlay_changes_the_image(backend):
|
|
"""The labels have to actually be drawn, not merely computed."""
|
|
await backend.entity_create_circle(10, 10, 5)
|
|
await backend.entity_create_line(0, 0, 20, 0)
|
|
|
|
plain = await backend.view_screenshot()
|
|
labelled = await backend.view_screenshot(overlay_handles=True)
|
|
|
|
assert labelled is not None
|
|
assert labelled != plain, "the overlay produced a byte-identical image"
|
|
|
|
|
|
async def test_the_overlay_reports_which_handles_it_drew(backend):
|
|
circle = await backend.entity_create_circle(10, 10, 5)
|
|
line = await backend.entity_create_line(0, 0, 20, 0)
|
|
|
|
report = await backend.view_screenshot_grounded()
|
|
|
|
assert set(report["labels"]) == {circle.handle, line.handle}
|
|
assert report["labelled"] == 2
|
|
assert report["total"] == 2
|
|
assert report["truncated"] is False
|
|
assert report["png"][:8] == b"\x89PNG\r\n\x1a\n"
|
|
|
|
|
|
async def test_each_label_sits_on_its_own_entity(backend):
|
|
"""A label in the wrong place is worse than no label — it grounds a lie."""
|
|
left = await backend.entity_create_circle(-50, 0, 5)
|
|
right = await backend.entity_create_circle(50, 0, 5)
|
|
|
|
report = await backend.view_screenshot_grounded()
|
|
|
|
assert report["labels"][left.handle][0] == pytest.approx(-50, abs=1.0)
|
|
assert report["labels"][right.handle][0] == pytest.approx(50, abs=1.0)
|
|
|
|
|
|
async def test_a_crowded_drawing_says_it_only_labelled_some(backend):
|
|
for i in range(30):
|
|
await backend.entity_create_circle(i * 10, 0, 2)
|
|
|
|
report = await backend.view_screenshot_grounded(max_labels=10)
|
|
|
|
assert report["labelled"] == 10
|
|
assert report["total"] == 30
|
|
assert report["truncated"] is True, (
|
|
"labelling a subset silently would let the picture read as the whole drawing"
|
|
)
|
|
|
|
|
|
async def test_an_empty_drawing_grounds_nothing_without_failing(backend):
|
|
report = await backend.view_screenshot_grounded()
|
|
assert report["labels"] == {}
|
|
assert report["total"] == 0
|
|
assert report["png"] is not None
|