release: v1.5.0 — discovery layer, honesty debt closed, Wave A
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Umut CAN
co-authored by
Claude Opus 5
Claude Opus 5
parent
d5a0ff5109
commit
b03a7615b1
@@ -14,8 +14,9 @@ import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from backends.capability import UnsupportedCapabilityError
|
||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec, task_by_id
|
||||
from benchmarks.tasks_v3 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
|
||||
|
||||
|
||||
def _load_registry() -> dict[str, dict]:
|
||||
@@ -78,6 +79,7 @@ async def run_tasks(
|
||||
artifact_dir: str | Path,
|
||||
*,
|
||||
timeout: float = 30.0,
|
||||
matrix: str = DEFAULT_MATRIX,
|
||||
) -> dict:
|
||||
destination = Path(artifact_dir).resolve()
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
@@ -90,6 +92,12 @@ async def run_tasks(
|
||||
result = await asyncio.wait_for(adapter.run_task(task), timeout=timeout)
|
||||
except TimeoutError:
|
||||
result = TaskResult(task.task_id, "timeout", 0.0, f"Exceeded {timeout}s")
|
||||
except UnsupportedCapabilityError as exc:
|
||||
# "Your server got this wrong" and "your engine cannot reach
|
||||
# this" are different findings about a competitor. The enum has
|
||||
# carried `unsupported` since v2 for exactly this, but a
|
||||
# refusal is still an exception and used to land as `fail`.
|
||||
result = TaskResult(task.task_id, "unsupported", 0.0, f"[{exc.capability}] {exc}")
|
||||
except Exception as exc:
|
||||
result = TaskResult(task.task_id, "fail", 0.0, str(exc), stderr_summary=str(exc))
|
||||
result.duration_ms = round((time.perf_counter() - started) * 1000, 2)
|
||||
@@ -108,7 +116,11 @@ async def run_tasks(
|
||||
weight_by_id[item.task_id] * item.score for item in results if item.status != "unsupported"
|
||||
)
|
||||
return {
|
||||
# The *report* schema is unchanged — same keys, same enum — so the
|
||||
# chart renderers still read v1.4's published files. `matrix` says
|
||||
# which task set ran, which is the thing that actually moved.
|
||||
"schema_version": "2.0",
|
||||
"matrix": matrix,
|
||||
"adapter": adapter.name,
|
||||
"adapter_metadata": adapter.metadata(),
|
||||
"git_sha": _git_sha(),
|
||||
@@ -137,6 +149,12 @@ async def run_tasks(
|
||||
def _parser(registry: dict[str, dict]) -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--list", action="store_true", help="List the fixed benchmark tasks")
|
||||
parser.add_argument(
|
||||
"--matrix",
|
||||
default=DEFAULT_MATRIX,
|
||||
choices=sorted(MATRICES),
|
||||
help="Task set to run. v2 stays addressable so a v1.4 report can be reproduced.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--server",
|
||||
default="autocad-mcp-pro",
|
||||
@@ -148,14 +166,35 @@ def _parser(registry: dict[str, dict]) -> argparse.ArgumentParser:
|
||||
parser.add_argument("--artifact-dir", default="benchmarks/results/latest")
|
||||
parser.add_argument("--timeout", type=float, default=30.0)
|
||||
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON")
|
||||
parser.add_argument(
|
||||
"--publish",
|
||||
type=Path,
|
||||
help=(
|
||||
"Also write the report here with artifact paths reduced to filenames. "
|
||||
"How results/published/*.json are produced — it was a manual edit before."
|
||||
),
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def _sanitize_for_publication(report: dict) -> dict:
|
||||
"""Drop absolute paths from artifacts, keep the hash that identifies them.
|
||||
|
||||
A published report should not carry the run machine's directory layout;
|
||||
the sha256 is what makes the artifact checkable, and that stays.
|
||||
"""
|
||||
published = json.loads(json.dumps(report))
|
||||
for result in published["results"]:
|
||||
for artifact in result["artifacts"]:
|
||||
artifact.pop("path", None)
|
||||
return published
|
||||
|
||||
|
||||
def main() -> None:
|
||||
registry = _load_registry()
|
||||
args = _parser(registry).parse_args()
|
||||
if args.list:
|
||||
for task in TASKS_V2:
|
||||
for task in MATRICES[args.matrix]:
|
||||
print(f"{task.task_id}\t{task.category}\t{task.description}")
|
||||
return
|
||||
|
||||
@@ -165,10 +204,27 @@ def main() -> None:
|
||||
raise SystemExit(
|
||||
f"{args.server} supports backends {supported_backends}, not {args.backend!r}"
|
||||
)
|
||||
tasks = [task_by_id(task_id) for task_id in args.task] if args.task else list(TASKS_V2)
|
||||
tasks = (
|
||||
[task_by_id(task_id, args.matrix) for task_id in args.task]
|
||||
if args.task
|
||||
else list(MATRICES[args.matrix])
|
||||
)
|
||||
adapter = _make_adapter(entry, args.backend)
|
||||
report = asyncio.run(run_tasks(adapter, tasks, args.artifact_dir, timeout=args.timeout))
|
||||
report = asyncio.run(
|
||||
run_tasks(adapter, tasks, args.artifact_dir, timeout=args.timeout, matrix=args.matrix)
|
||||
)
|
||||
rendered = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
if args.publish:
|
||||
args.publish.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.publish.write_text(
|
||||
json.dumps(
|
||||
_sanitize_for_publication(report), ensure_ascii=False, indent=2, sort_keys=True
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
print(f"Published: {args.publish}", file=sys.stderr)
|
||||
print(rendered)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user