Umut CANandClaude Opus 5
b03a7615b1
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
102 lines
3.9 KiB
Python
102 lines
3.9 KiB
Python
"""T0.8-B — which engine a test talks to must not be a matter of luck.
|
|
|
|
``_make_backend`` read ``os.environ["AUTOCAD_MCP_BACKEND"]`` directly while
|
|
``config.settings.backend`` — the setting that *looks* like the control — had no
|
|
production reader at all. So ``monkeypatch.setattr(config.settings, "backend",
|
|
"ezdxf")`` was a no-op that read like a safety guard, and on a developer machine
|
|
with AutoCAD open an unguarded test would attach to the operator's live unsaved
|
|
drawing and start creating entities in it. That is not hypothetical: it happened
|
|
during this release's development.
|
|
|
|
The fix makes ``Settings.backend`` a property that reads the environment live,
|
|
and points ``_make_backend`` at it. One source of truth, and late binding is
|
|
preserved so ``monkeypatch.setenv`` — which four call sites and
|
|
``benchmarks/token_suite.py`` already use correctly — keeps working. Assigning to
|
|
``config.settings.backend`` now raises instead of quietly protecting nothing.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import sys
|
|
import types
|
|
|
|
import pytest
|
|
|
|
import config
|
|
import server
|
|
|
|
pytestmark = pytest.mark.asyncio
|
|
|
|
|
|
class _ComBackendSentinel:
|
|
"""Stands in for the real COM backend and screams if anyone builds it."""
|
|
|
|
constructed: list[str] = []
|
|
|
|
def __init__(self, *args, **kwargs):
|
|
_ComBackendSentinel.constructed.append("ComBackend() CONSTRUCTED")
|
|
raise AssertionError("a test tried to construct the live COM backend")
|
|
|
|
|
|
@pytest.fixture
|
|
def com_sentinel(monkeypatch):
|
|
"""Replace backends.com_backend with a module that cannot connect anywhere."""
|
|
_ComBackendSentinel.constructed.clear()
|
|
fake = types.ModuleType("backends.com_backend")
|
|
fake.ComBackend = _ComBackendSentinel
|
|
monkeypatch.setitem(sys.modules, "backends.com_backend", fake)
|
|
return _ComBackendSentinel
|
|
|
|
|
|
async def test_the_env_var_is_what_actually_selects_the_backend(monkeypatch, com_sentinel):
|
|
from backends.ezdxf_backend import EzdxfBackend
|
|
|
|
monkeypatch.setenv("AUTOCAD_MCP_BACKEND", "ezdxf")
|
|
backend = await server._make_backend()
|
|
|
|
assert isinstance(backend, EzdxfBackend)
|
|
assert com_sentinel.constructed == [], "the COM backend must never have been touched"
|
|
await backend.disconnect()
|
|
|
|
|
|
async def test_settings_backend_reports_what_make_backend_will_use(monkeypatch):
|
|
"""The setting must not be able to disagree with the selector."""
|
|
for value in ("ezdxf", "com", "auto"):
|
|
monkeypatch.setenv("AUTOCAD_MCP_BACKEND", value)
|
|
assert config.settings.backend == value
|
|
|
|
monkeypatch.setenv("AUTOCAD_MCP_BACKEND", " EzDxF ")
|
|
assert config.settings.backend == "ezdxf", "must normalise like the selector does"
|
|
|
|
monkeypatch.delenv("AUTOCAD_MCP_BACKEND", raising=False)
|
|
assert config.settings.backend == "auto"
|
|
|
|
|
|
async def test_assigning_to_settings_backend_fails_loudly(monkeypatch):
|
|
"""The dead monkeypatch that made two tests *look* guarded must now raise.
|
|
|
|
Silently accepting the assignment is the whole bug: the test passes, reads
|
|
as safe, and talks to live AutoCAD anyway.
|
|
"""
|
|
monkeypatch.setenv("AUTOCAD_MCP_BACKEND", "ezdxf")
|
|
with pytest.raises(AttributeError):
|
|
config.settings.backend = "com"
|
|
assert config.settings.backend == "ezdxf"
|
|
|
|
|
|
async def test_make_backend_reads_the_setting_not_the_environment_directly(monkeypatch):
|
|
"""One source of truth: patching the property must steer the selector.
|
|
|
|
If ``_make_backend`` goes back to reading ``os.environ`` itself, this fails —
|
|
which is the regression guard, because that direct read is what let the
|
|
setting and the behaviour drift apart in the first place.
|
|
"""
|
|
from backends.ezdxf_backend import EzdxfBackend
|
|
|
|
monkeypatch.setenv("AUTOCAD_MCP_BACKEND", "com")
|
|
monkeypatch.setattr(type(config.settings), "backend", property(lambda self: "ezdxf"))
|
|
|
|
backend = await server._make_backend()
|
|
assert isinstance(backend, EzdxfBackend)
|
|
await backend.disconnect()
|