Umut CANandClaude Opus 5
b03a7615b1
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
85 lines
3.1 KiB
Python
85 lines
3.1 KiB
Python
"""The v3 task matrix: the v2 ten, plus five the v2 ten could not see.
|
|
|
|
v2 scored this server 10/10, and that number was structurally uninformative:
|
|
every task in it exercised something the server was already built around. A
|
|
matrix that cannot fail is a matrix that measures nothing, so the five added
|
|
here were picked to cover capability the v2 set never touches — discovery,
|
|
idle token cost, hatch islands, selection semantics, and measuring an entity
|
|
that is already in the drawing.
|
|
|
|
Two of the five are renamed from the release plan, and the renames are the
|
|
point rather than bookkeeping:
|
|
|
|
``tool_discovery_bilingual`` -> ``tool_discovery``
|
|
The plan assumed a Turkish query normalizer. One was written and then
|
|
removed: this repository is English throughout, and the Turkish was an
|
|
inference from the author's chat language, not a product requirement. A
|
|
task named "bilingual" would advertise a capability that does not exist,
|
|
which is the class of claim this release spent four milestones deleting.
|
|
|
|
``measurement_massprops`` -> ``measure_from_handle``
|
|
No mass properties ship. The plan's eight-tool measurement family (F2) was
|
|
cut to one after measurement showed the repo already carried eight
|
|
``analysis_*`` tools; centroids and moments of inertia were not among the
|
|
survivors. What did ship is measuring a *handle* instead of a caller's
|
|
remembered coordinates, so that is what the task is called.
|
|
|
|
The v2 ids and weights are untouched. The published competitor reports were
|
|
run against the v2 matrix at v1.4 and are not re-scored here — those servers
|
|
have never been asked these five questions, and inventing a result for them
|
|
would be worse than the gap. The task-matrix chart renders them ``not_run``.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
|
|
|
|
#: Added in v1.5.0. Every one of these can fail on this server.
|
|
NEW_TASKS_V3: tuple[TaskSpec, ...] = (
|
|
TaskSpec(
|
|
"tool_discovery",
|
|
"discovery",
|
|
"Resolve AutoCAD command names to tools through the search layer",
|
|
1.0,
|
|
),
|
|
TaskSpec(
|
|
"token_budget",
|
|
"efficiency",
|
|
"Keep the idle catalog under a fixed token ceiling",
|
|
1.0,
|
|
),
|
|
TaskSpec(
|
|
"hatch_islands",
|
|
"hatch",
|
|
"Hatch an area with an island and measure what it fills",
|
|
1.0,
|
|
),
|
|
TaskSpec(
|
|
"selection_filter",
|
|
"selection",
|
|
"Separate window from crossing, and a polygon from its bounding box",
|
|
1.0,
|
|
),
|
|
TaskSpec(
|
|
"measure_from_handle",
|
|
"measurement",
|
|
"Measure a curved boundary from its handle, bulges included",
|
|
1.0,
|
|
),
|
|
)
|
|
|
|
TASKS_V3: tuple[TaskSpec, ...] = TASKS_V2 + NEW_TASKS_V3
|
|
|
|
#: Runner-selectable task sets. v2 stays addressable so a v2 report can be
|
|
#: reproduced exactly rather than only described.
|
|
MATRICES: dict[str, tuple[TaskSpec, ...]] = {"v2": TASKS_V2, "v3": TASKS_V3}
|
|
|
|
DEFAULT_MATRIX = "v3"
|
|
|
|
|
|
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
|
for task in MATRICES[matrix]:
|
|
if task.task_id == task_id:
|
|
return task
|
|
raise KeyError(f"Unknown benchmark task: {task_id}")
|