release: v1.5.0 — discovery layer, honesty debt closed, Wave A
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Umut CAN
co-authored by
Claude Opus 5
Claude Opus 5
parent
d5a0ff5109
commit
b03a7615b1
+124
-31
@@ -11,7 +11,7 @@ This repository publishes two deliberately separate forms of evidence:
|
||||
delivery, backend reach, engineering production, tests, and security. The
|
||||
dated data lives in [`source_review.json`](source_review.json), and the README
|
||||
graphic is generated—not hand-edited—from that file.
|
||||
2. **Fixed-task runtime benchmark.** Adapters execute the same ten tasks and
|
||||
2. **Fixed-task runtime benchmark.** Adapters execute the same task matrix and
|
||||
return `pass`, `partial`, `unsupported`, `fail`, `timeout`, or `not_run`.
|
||||
AutoCAD MCP Pro currently has the reference adapter; other projects do not
|
||||
receive runtime scores until an adapter actually runs their public interface.
|
||||
@@ -26,38 +26,80 @@ python -m benchmarks.render_chart
|
||||
The source-review chart is not presented as a shared live AutoCAD run. Review
|
||||
dates, evidence grades, project URLs, and the boundary are stored with the data.
|
||||
|
||||
## Fixed-task runtime runner v2
|
||||
## Fixed-task runtime runner
|
||||
|
||||
The v2 runner separates runtime evidence from the source-review rubric. Every
|
||||
adapter receives the same ten tasks and produces the same closed result enum:
|
||||
The runner separates runtime evidence from the source-review rubric. Every
|
||||
adapter receives the same tasks and produces the same closed result enum:
|
||||
`pass`, `partial`, `unsupported`, `fail`, `timeout`, or `not_run`. A timeout is
|
||||
reported per task. Execution is in-process, so adapters that delegate blocking
|
||||
work must provide cancellation-safe task implementations or be wrapped by an
|
||||
external process supervisor. Reports include commit SHA, Python/platform
|
||||
details, backend capability claims, durations, coverage, and hashes for returned
|
||||
artifacts. Unsupported tasks remain in the fixed-matrix denominator with score
|
||||
zero, preventing partial implementations from receiving an inflated score.
|
||||
reported per task, and a capability refusal is recorded `unsupported` rather
|
||||
than `fail` — "cannot reach this" and "got this wrong" are different findings.
|
||||
Execution is in-process, so adapters that delegate blocking work must provide
|
||||
cancellation-safe task implementations or be wrapped by an external process
|
||||
supervisor. Reports include commit SHA, Python/platform details, backend
|
||||
capability claims, durations, coverage, and hashes for returned artifacts.
|
||||
Unsupported tasks remain in the fixed-matrix denominator with score zero,
|
||||
preventing partial implementations from receiving an inflated score.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --list
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --json
|
||||
python -m benchmarks.run_competitors --task table_mleader --task auditable_delivery --json
|
||||
python -m benchmarks.run_competitors --list --matrix v3
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v3 --json
|
||||
python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
|
||||
```
|
||||
|
||||
The release-machine ezdxf self-check is **10/10 (100.0)**. Repository stars and
|
||||
raw tool counts do not contribute to the score. Adapter registration lives in
|
||||
`competitors.yaml`.
|
||||
The release-machine ezdxf self-check is **15/15 (100.0)** on the v3 matrix.
|
||||
Repository stars and raw tool counts do not contribute to the score. Adapter
|
||||
registration lives in `competitors.yaml`.
|
||||
|
||||
### Matrix v3 (v1.5.0) — five tasks that can fail
|
||||
|
||||
v2 scored this server 10/10, and that number carried no information: every task
|
||||
in it exercised something the server was designed around. `tasks_v3.py` keeps
|
||||
the v2 ten unchanged — their ids and weights key the published v1.4 reports —
|
||||
and adds five chosen because they *can* fail. Three of them did, while being
|
||||
written:
|
||||
|
||||
| Task | Category | Verified against |
|
||||
|---|---|---|
|
||||
| `tool_discovery` | discovery | six AutoCAD command names, each ranking #1 of 154 tools |
|
||||
| `token_budget` | efficiency | 40,305 → 356 tokens advertised, against a 2,000 ceiling fixed in advance |
|
||||
| `hatch_islands` | hatch | 300 filled with the island, 400 ignoring it |
|
||||
| `selection_filter` | selection | window 1, crossing 2, bounding box 3, polygon 1 |
|
||||
| `measure_from_handle` | measurement | 139.2699 against the 100.0 a vertex shoelace gives |
|
||||
|
||||
Token counts come from the offline ratio estimator and the report labels them
|
||||
`ratio/v1 (estimate)`; `token_suite.py --tokenizer anthropic` counts for real.
|
||||
|
||||
**Two renames from the release plan, and they are the point.**
|
||||
`tool_discovery_bilingual` → `tool_discovery`: a Turkish query normalizer was
|
||||
written and then removed, because this repository is English throughout and the
|
||||
Turkish was inferred from the author's chat language rather than required by the
|
||||
product. `measurement_massprops` → `measure_from_handle`: no centroid or
|
||||
moment-of-inertia tool ships.
|
||||
|
||||
**The competitor reports are not re-scored.** Those servers are pinned at v1.4
|
||||
commits and were run against the v2 matrix; they have never been asked these
|
||||
five questions. Their scores stand over the ten they ran, and the matrix chart
|
||||
leaves the five new rows blank (`not run`) instead of scoring them zero — a
|
||||
zero we invented would be indistinguishable from a zero we measured.
|
||||
|
||||
`--matrix v2` stays available so a v1.4 report can be reproduced rather than
|
||||
only described, and `--publish` writes the artifact-path-free file that
|
||||
`results/published/` holds.
|
||||
|
||||
## Live competitor lane (v1.4)
|
||||
|
||||
Two competitor adapters now execute the exact same task matrix, black-box over
|
||||
MCP stdio, against commits pinned in `competitors.yaml`:
|
||||
|
||||
| Server | Pinned | Score | Pass | Coverage |
|
||||
|---|---|---:|---:|---:|
|
||||
| autocad-mcp-pro (reference) | working tree | 100.0 | 10/10 | 100% |
|
||||
| beiming183-cloud/AutoCAD-MCP | `11f7c47e` | 50.0 | 5/10 | 50% |
|
||||
| puran-water/autocad-mcp | `95476a33` | 45.0 | 4/10 | 50% |
|
||||
| Server | Matrix | Pinned | Score | Pass | Coverage |
|
||||
|---|---|---|---:|---:|---:|
|
||||
| autocad-mcp-pro (reference) | v3 (15) | working tree | 100.0 | 15/15 | 100% |
|
||||
| beiming183-cloud/AutoCAD-MCP | v2 (10) | `11f7c47e` | 50.0 | 5/10 | 50% |
|
||||
| puran-water/autocad-mcp | v2 (10) | `95476a33` | 45.0 | 4/10 | 50% |
|
||||
|
||||
The competitor rows are the v1.4 runs, unchanged. Read the matrix column
|
||||
before the score column: the denominators differ because the five v3 tasks
|
||||
were added after those runs and have not been put to them.
|
||||
|
||||
Method and boundaries:
|
||||
|
||||
@@ -84,12 +126,16 @@ live under [`results/published/`](results/published/); the README charts are
|
||||
regenerated from them:
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
||||
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf
|
||||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf --matrix v2
|
||||
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf --matrix v2
|
||||
python -m benchmarks.render_live_chart # score bars
|
||||
python -m benchmarks.render_matrix_chart # tasks x servers status heatmap
|
||||
```
|
||||
|
||||
`--publish <path>` writes the same report with artifact paths reduced to
|
||||
filenames, which is how `results/published/` is produced; the sha256 stays,
|
||||
because that is what makes an artifact checkable.
|
||||
|
||||
## Headless performance lane
|
||||
|
||||
`perf_suite.py` measures wall time on fixed workloads through the same backend
|
||||
@@ -107,6 +153,36 @@ python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.js
|
||||
python -m benchmarks.render_perf_chart
|
||||
```
|
||||
|
||||
### v1.4.0 → v1.5.0, same interpreter
|
||||
|
||||
The published v1.4 report was recorded on CPython 3.14 and the v1.5 one on
|
||||
3.11.15, so comparing the two files directly shows an apparent 2–6× speedup
|
||||
that is almost entirely the interpreter. The honest comparison re-runs v1.4.0
|
||||
in a `git worktree` with the *current* suite file — same instrument, different
|
||||
code under test, one interpreter. Median of three runs each:
|
||||
|
||||
| Workload | v1.4.0 | v1.5.0 | Change |
|
||||
|---|---:|---:|---|
|
||||
| `create_lines_2k` | 245 ms | 315 ms | **28% slower** |
|
||||
| `roundtrip_10k` | 1,765 ms | 2,146 ms | **22% slower** |
|
||||
| `region_query_10k` | 1,090 ms | 1,039 ms | within spread |
|
||||
| `premium_pass` | 375 ms | 44 ms | faster; the v1.4 spread (279–772 ms) is too wide to quantify |
|
||||
|
||||
Attributed rather than reported: `EZDXF_CALL_TIMEOUT=0` puts v1.5.0 back to
|
||||
244.8 ms and 1,679 ms, which are v1.4.0's numbers. The entire cost is the
|
||||
per-call `asyncio.wait_for` wrapped around every headless call in this release,
|
||||
so a single hung ezdxf call can no longer wedge a server whose document lock is
|
||||
one `asyncio.Lock`. Reproduce:
|
||||
|
||||
```bash
|
||||
git worktree add /tmp/wt-v140 v1.4.0
|
||||
cp benchmarks/perf_suite.py /tmp/wt-v140/benchmarks/
|
||||
(cd /tmp/wt-v140 && PYTHONPATH=. python benchmarks/perf_suite.py --out /tmp/v140.json)
|
||||
python -m benchmarks.perf_suite --out /tmp/v150.json
|
||||
EZDXF_CALL_TIMEOUT=0 python -m benchmarks.perf_suite --out /tmp/v150-notimeout.json
|
||||
git worktree remove --force /tmp/wt-v140
|
||||
```
|
||||
|
||||
## Correctness A/B suite
|
||||
|
||||
`correctness_suite.py` is a set of deterministic, headless (ezdxf-backend) checks
|
||||
@@ -123,19 +199,36 @@ python benchmarks/compare_versions.py v1.0.0 # vs a tag/ref
|
||||
python benchmarks/compare_versions.py --json results.json
|
||||
```
|
||||
|
||||
### Result — v1.4.0 vs v1.3.0 (release gate)
|
||||
### Result — v1.5.0 vs v1.4.0 (release gate)
|
||||
|
||||
21 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
24 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
[`results/published/ab-v1.4.0-vs-v1.5.0.json`](results/published/ab-v1.4.0-vs-v1.5.0.json).
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|---------|----------------|-----------|-------|-----------|
|
||||
| **v1.4.0** (baseline) | 21 / 24 | 87.5 % | — | — |
|
||||
| **v1.5.0** (this release) | 24 / 24 | 100 % | 3 | **0** |
|
||||
|
||||
Read the three "fixed" carefully: they are **new capability, not repaired
|
||||
regressions**. v1.4.0 has no `entity_measure` and no boundary tracing, so it
|
||||
misses `hatch_area_subtracts_its_island`, `boundary_area_agrees_with_measure`
|
||||
and `two_vertex_circle_has_area` by not having the method rather than by
|
||||
getting it wrong. They are in the suite because all three broke during v1.5.0
|
||||
development and nothing outside their own unit tests would have caught it.
|
||||
|
||||
The 21 pre-existing checks are unchanged and all still pass: the discovery
|
||||
layer, `cad_batch`, the 19-module contract split, the layout/viewport family
|
||||
and Wave A landed with **zero correctness regressions**.
|
||||
|
||||
### Result — v1.4.0 vs v1.3.0
|
||||
|
||||
21 checks. Report:
|
||||
[`results/published/ab-v1.3.0-vs-v1.4.0.json`](results/published/ab-v1.3.0-vs-v1.4.0.json).
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|---------|----------------|-----------|-------|-----------|
|
||||
| **v1.3.0** (baseline) | 21 / 21 | 100 % | — | — |
|
||||
| **v1.4.0** (this release) | 21 / 21 | 100 % | 0 | **0** |
|
||||
|
||||
The v1.4.0 feature and infrastructure work (CI, packaging, competitor lane,
|
||||
tool profiles, paper space, ISO 286 fits, opt-in solids) introduced **zero
|
||||
correctness regressions**.
|
||||
| **v1.3.0** (baseline) | 21 / 21 | 100 % | — | — |
|
||||
| **v1.4.0** | 21 / 21 | 100 % | 0 | **0** |
|
||||
|
||||
### Result — v1.1.0 vs v1.0.0 (`origin/main`, commit 15fa2bc)
|
||||
|
||||
|
||||
@@ -1,15 +1,45 @@
|
||||
"""Reference v2 adapter for this repository."""
|
||||
"""Reference adapter for this repository, covering the v3 task matrix.
|
||||
|
||||
Every task verifies against a value worked out independently of the code under
|
||||
test — a closed-form area, a count of entities placed on purpose, a token
|
||||
ceiling fixed before the measurement. A task that only checks the server agrees
|
||||
with itself is a task that cannot fail.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||
from benchmarks.tasks_v2 import TaskSpec
|
||||
from benchmarks.tasks_v3 import TaskSpec
|
||||
from engineering.delivery import deliver_drawing
|
||||
from engineering.layers import ensure_engineering_layers, ensure_standard_linetypes
|
||||
from engineering.refiner import refine_drawing
|
||||
|
||||
#: AutoCAD command names a drafter types, none of which appeared in any tool
|
||||
#: name or description before v1.5.0 — measured `df = 0` against the stock BM25
|
||||
#: index, so no amount of tuning could have surfaced them.
|
||||
#: Each command is answered by a *set*, because more than one tool can be a
|
||||
#: correct answer -- QSELECT is both "filter by properties" and "select
|
||||
#: similar", and picking a winner between them would test the tie-break rather
|
||||
#: than the discovery.
|
||||
DISCOVERY_QUERIES: tuple[tuple[str, frozenset[str]], ...] = (
|
||||
("FILLET", frozenset({"entity_fillet"})),
|
||||
("BPOLY", frozenset({"boundary_trace"})),
|
||||
("QSELECT", frozenset({"selection_filter", "entity_select_smart"})),
|
||||
("WBLOCK", frozenset({"block_create_from_entities"})),
|
||||
("OVERKILL", frozenset({"drawing_refine"})),
|
||||
("CHSPACE", frozenset({"entity_change_space"})),
|
||||
)
|
||||
|
||||
#: How far down the ranking still counts as found. A drafter reads the top few.
|
||||
DISCOVERY_TOP_N = 3
|
||||
|
||||
#: The advertised catalog must stay under this in discovery mode. Fixed here
|
||||
#: rather than derived from the measurement, so the measurement can miss it.
|
||||
TOKEN_CEILING = 2_000
|
||||
|
||||
|
||||
class AutoCADMCPProAdapter(BenchmarkAdapter):
|
||||
name = "autocad-mcp-pro"
|
||||
@@ -138,6 +168,202 @@ class AutoCADMCPProAdapter(BenchmarkAdapter):
|
||||
await reopened.disconnect()
|
||||
return count == 1, {"entity_count": count}, [str(path)]
|
||||
|
||||
# ── v3 (v1.5.0) ─────────────────────────────────────────────────────────
|
||||
|
||||
async def _task_tool_discovery(self):
|
||||
"""Can a drafter find the tool by typing the command they know?
|
||||
|
||||
Ranked through the real transform, not a stub: the corpus is what makes
|
||||
these commands findable at all, and testing anything else would test
|
||||
the fixture.
|
||||
"""
|
||||
import server
|
||||
from discovery.transform import CadSearchTransform
|
||||
|
||||
tools = list(await server.mcp._list_tools())
|
||||
transform = CadSearchTransform()
|
||||
|
||||
ranks: dict[str, int | None] = {}
|
||||
for command, accepted in DISCOVERY_QUERIES:
|
||||
hits = [
|
||||
tool.name for tool in transform.rank(tools, command, limit=DISCOVERY_TOP_N).hits
|
||||
]
|
||||
found = next((index for index, name in enumerate(hits) if name in accepted), None)
|
||||
ranks[command] = None if found is None else found + 1
|
||||
|
||||
passed = all(rank is not None for rank in ranks.values())
|
||||
return passed, {"ranks": ranks, "top_n": DISCOVERY_TOP_N}, []
|
||||
|
||||
async def _task_token_budget(self):
|
||||
"""What the client pays before it has asked for anything.
|
||||
|
||||
The ceiling is a constant, so a regression that doubles the catalog
|
||||
fails this rather than quietly reporting a bigger number.
|
||||
"""
|
||||
import server
|
||||
from benchmarks.token_suite import (
|
||||
FORMAT_JSON_SCHEMA,
|
||||
_client,
|
||||
_wire_payload,
|
||||
build_counter,
|
||||
)
|
||||
|
||||
async def _advertised(mode: str) -> tuple[int, int]:
|
||||
"""Tokens on the wire for `tools/list`, and how many tools that is.
|
||||
|
||||
Measured through a real client rather than the registry: the
|
||||
registry is what the transform hides, so counting it would make
|
||||
discovery mode look free.
|
||||
"""
|
||||
server._apply_discovery_mode(mode)
|
||||
async with _client() as client:
|
||||
tools = await client.list_tools()
|
||||
return counter.count(_wire_payload(tools), FORMAT_JSON_SCHEMA), len(tools)
|
||||
|
||||
counter = build_counter("ratio")
|
||||
try:
|
||||
full, catalog_count = await _advertised("off")
|
||||
discovery, advertised_count = await _advertised("search")
|
||||
finally:
|
||||
server._apply_discovery_mode("off")
|
||||
|
||||
passed = discovery < TOKEN_CEILING and discovery < full
|
||||
return (
|
||||
passed,
|
||||
{
|
||||
"catalog_tokens": full,
|
||||
"discovery_tokens": discovery,
|
||||
"ceiling": TOKEN_CEILING,
|
||||
"reduction_x": round(full / discovery, 1) if discovery else None,
|
||||
"catalog_tools": catalog_count,
|
||||
"advertised_tools": advertised_count,
|
||||
# Named, because "40,305 tokens" reads like a count and is not
|
||||
# one. `benchmarks/token_suite.py --tokenizer anthropic` counts
|
||||
# for real; this lane has to stay offline and deterministic.
|
||||
"tokenizer": f"{counter.name} (estimate)" if counter.estimated else counter.name,
|
||||
},
|
||||
[],
|
||||
)
|
||||
|
||||
async def _task_hatch_islands(self):
|
||||
"""A section view is mostly holes, so the island is the measurement.
|
||||
|
||||
20x20 outer square with a 10x10 island: the filled area is 300, and a
|
||||
server that records the island without subtracting it reports 400 while
|
||||
looking entirely correct on screen.
|
||||
"""
|
||||
await ensure_engineering_layers(self.backend)
|
||||
hatch = await self.backend.entity_create_hatch(
|
||||
"ANSI31", [[0, 0], [20, 0], [20, 20], [0, 20]], layer="HATCH"
|
||||
)
|
||||
island = [
|
||||
{"type": "line", "start": [5, 5], "end": [15, 5]},
|
||||
{"type": "line", "start": [15, 5], "end": [15, 15]},
|
||||
{"type": "line", "start": [15, 15], "end": [5, 15]},
|
||||
{"type": "line", "start": [5, 15], "end": [5, 5]},
|
||||
]
|
||||
added = await self.backend.hatch_add_boundary(hatch.handle, island)
|
||||
measured = await self.backend.entity_measure(hatch.handle)
|
||||
ignored = await self.backend.hatch_edit(hatch.handle, style="ignore")
|
||||
over_island = await self.backend.entity_measure(hatch.handle)
|
||||
|
||||
passed = (
|
||||
added["path_count"] == 2
|
||||
and abs(measured["area"] - 300.0) < 1e-6
|
||||
and measured["hatch_style"] == "outer" # what entity_create_hatch writes
|
||||
and ignored["changed"] == ["style"]
|
||||
and abs(over_island["area"] - 400.0) < 1e-6
|
||||
)
|
||||
return (
|
||||
passed,
|
||||
{
|
||||
"path_count": added["path_count"],
|
||||
"filled_area": measured["area"],
|
||||
"area_ignoring_islands": over_island["area"],
|
||||
"expected_filled": 300.0,
|
||||
"hatch_style": measured["hatch_style"],
|
||||
},
|
||||
[],
|
||||
)
|
||||
|
||||
async def _task_selection_filter(self):
|
||||
"""Window is not crossing, and a polygon is not its bounding box.
|
||||
|
||||
Three circles placed so each distinction has exactly one right answer:
|
||||
one wholly inside, one straddling the edge, one out in the corner that
|
||||
the bounding box catches and the triangle does not.
|
||||
"""
|
||||
await ensure_engineering_layers(self.backend)
|
||||
await self.backend.entity_create_circle(5, 5, 1, layer="GEOMETRY")
|
||||
await self.backend.entity_create_circle(10, 5, 3, layer="GEOMETRY")
|
||||
await self.backend.entity_create_circle(18, 18, 0.5, layer="GEOMETRY")
|
||||
await self.backend.entity_create_circle(40, 40, 1, layer="TEXT")
|
||||
|
||||
window = await self.backend.selection_window(0, 0, 10, 10, mode="window")
|
||||
crossing = await self.backend.selection_window(0, 0, 10, 10, mode="crossing")
|
||||
bbox = await self.backend.selection_window(0, 0, 20, 20, mode="window")
|
||||
triangle = await self.backend.selection_polygon([[0, 0], [20, 0], [0, 20]], mode="window")
|
||||
by_layer = await self.backend.selection_filter(entity_type="CIRCLE", layer="TEXT")
|
||||
|
||||
passed = (
|
||||
window["count"] == 1
|
||||
and crossing["count"] == 2
|
||||
and bbox["count"] == 3
|
||||
and triangle["count"] == 1
|
||||
and by_layer["count"] == 1
|
||||
and by_layer["filtered_by"] == ["entity_type", "layer"]
|
||||
)
|
||||
return (
|
||||
passed,
|
||||
{
|
||||
"window": window["count"],
|
||||
"crossing": crossing["count"],
|
||||
"bounding_box": bbox["count"],
|
||||
"polygon": triangle["count"],
|
||||
"by_layer": by_layer["count"],
|
||||
},
|
||||
[],
|
||||
)
|
||||
|
||||
async def _task_measure_from_handle(self):
|
||||
"""The 28.2% that reading coordinates back loses.
|
||||
|
||||
A 10x10 square whose top edge is a semicircle encloses 139.2699. Chain
|
||||
the loose edges into a boundary, then measure that boundary by handle;
|
||||
shoelacing the four corners the model would have remembered gives 100.
|
||||
"""
|
||||
await ensure_engineering_layers(self.backend)
|
||||
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
|
||||
await self.backend.entity_create_line(10, 0, 10, 10, layer="GEOMETRY")
|
||||
await self.backend.entity_create_line(0, 10, 0, 0, layer="GEOMETRY")
|
||||
await self.backend.entity_create_arc(5, 10, 5, 0, 180, layer="GEOMETRY")
|
||||
|
||||
loop = await self.backend.boundary_trace(5.0, 5.0)
|
||||
measured = await self.backend.entity_measure(loop["handle"])
|
||||
naive = await self.backend.analysis_measure_area([[0, 0], [10, 0], [10, 10], [0, 10]])
|
||||
|
||||
true_area = 100.0 + math.pi * 25.0 / 2.0
|
||||
lost = (true_area - naive) / true_area
|
||||
passed = (
|
||||
loop["ok"] is True
|
||||
and abs(measured["area"] - true_area) < 1e-5
|
||||
and abs(loop["area"] - measured["area"]) < 1e-9
|
||||
and measured["method"] == "analytic_bulge"
|
||||
and measured["exact"] is True
|
||||
and 0.28 < lost < 0.29
|
||||
)
|
||||
return (
|
||||
passed,
|
||||
{
|
||||
"measured_area": measured["area"],
|
||||
"true_area": round(true_area, 6),
|
||||
"vertex_shoelace_area": naive,
|
||||
"fraction_lost_by_reading_points_back": round(lost, 4),
|
||||
"boundary_agrees": abs(loop["area"] - measured["area"]) < 1e-9,
|
||||
},
|
||||
[],
|
||||
)
|
||||
|
||||
async def _task_auditable_delivery(self):
|
||||
await ensure_engineering_layers(self.backend)
|
||||
await self.backend.entity_create_line(0, 0, 10, 0, layer="GEOMETRY")
|
||||
|
||||
@@ -223,6 +223,52 @@ async def center_linetype_applied():
|
||||
return info.linetype.upper() == "CENTER"
|
||||
|
||||
|
||||
# ── Measurement (v1.5.0) ───────────────────────────────────────────────────────
|
||||
#
|
||||
# These three are NEW capability, not repaired regressions: v1.4.0 has no
|
||||
# `entity_measure` and no boundary tracing, so it misses them by not having the
|
||||
# method rather than by getting it wrong. They are here because all three broke
|
||||
# during v1.5.0 development and nothing outside their own unit tests would have
|
||||
# noticed.
|
||||
|
||||
|
||||
async def hatch_area_subtracts_its_island():
|
||||
"""A section view is mostly holes; counting them is a 33% error here."""
|
||||
b = await _b()
|
||||
h = await b.entity_create_hatch("ANSI31", [[0, 0], [20, 0], [20, 20], [0, 20]])
|
||||
await b.hatch_add_boundary(
|
||||
h.handle,
|
||||
[
|
||||
{"type": "line", "start": [5, 5], "end": [15, 5]},
|
||||
{"type": "line", "start": [15, 5], "end": [15, 15]},
|
||||
{"type": "line", "start": [15, 15], "end": [5, 15]},
|
||||
{"type": "line", "start": [5, 15], "end": [5, 5]},
|
||||
],
|
||||
)
|
||||
return abs((await b.entity_measure(h.handle))["area"] - 300.0) < 1e-6
|
||||
|
||||
|
||||
async def boundary_area_agrees_with_measure():
|
||||
"""Two tools, one shape, one number."""
|
||||
b = await _b()
|
||||
await b.entity_create_line(0, 0, 10, 0)
|
||||
await b.entity_create_line(10, 0, 10, 10)
|
||||
await b.entity_create_line(0, 10, 0, 0)
|
||||
await b.entity_create_arc(5, 10, 5, 0, 180)
|
||||
loop = await b.boundary_trace(5.0, 5.0)
|
||||
measured = await b.entity_measure(loop["handle"])
|
||||
true_area = 100.0 + math.pi * 25.0 / 2.0
|
||||
return abs(loop["area"] - measured["area"]) < 1e-9 and abs(measured["area"] - true_area) < 1e-5
|
||||
|
||||
|
||||
async def two_vertex_circle_has_area():
|
||||
"""DXF's most compact circle. The shoelace over its two points is 0."""
|
||||
from engineering.measure import polygon_area_perimeter
|
||||
|
||||
area, _ = polygon_area_perimeter([(-10.0, 0.0, 1.0), (10.0, 0.0, 1.0)], closed=True)
|
||||
return abs(area - math.pi * 100.0) < 1e-9
|
||||
|
||||
|
||||
CHECKS = {
|
||||
"core_line_length": (core_line_length, "Core"),
|
||||
"core_circle_radius": (core_circle_radius, "Core"),
|
||||
@@ -245,6 +291,9 @@ CHECKS = {
|
||||
"construction_left_iso_caught": (construction_left_iso_caught, "Quality gate"),
|
||||
"gear_no_self_overlap": (gear_no_self_overlap, "Engineering"),
|
||||
"center_linetype_applied": (center_linetype_applied, "Entities"),
|
||||
"hatch_area_subtracts_its_island": (hatch_area_subtracts_its_island, "Measurement"),
|
||||
"boundary_area_agrees_with_measure": (boundary_area_agrees_with_measure, "Measurement"),
|
||||
"two_vertex_circle_has_area": (two_vertex_circle_has_area, "Measurement"),
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from benchmarks.tasks_v2 import TASKS_V2
|
||||
from benchmarks.tasks_v3 import TASKS_V3
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
|
||||
@@ -66,8 +66,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||
|
||||
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
|
||||
|
||||
task_ids = [task.task_id for task in TASKS_V2]
|
||||
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V2]
|
||||
task_ids = [task.task_id for task in TASKS_V3]
|
||||
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V3]
|
||||
servers = [report["adapter"] for report in reports]
|
||||
status_by_server = {
|
||||
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
|
||||
@@ -132,12 +132,17 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||
Patch(facecolor=STATUS_COLORS["partial"], label="partial"),
|
||||
Patch(facecolor=STATUS_COLORS["fail"], label="fail"),
|
||||
Patch(facecolor=STATUS_COLORS["unsupported"], label="unsupported (no equivalent)"),
|
||||
# Real rows now, not a theoretical state: the competitor reports were
|
||||
# pinned and run against the v2 matrix, so the five v3 tasks were
|
||||
# never put to them. An invented zero would look identical to a
|
||||
# measured one, so they stay blank and say why.
|
||||
Patch(facecolor=STATUS_COLORS["not_run"], edgecolor="#30363d", label="not run"),
|
||||
]
|
||||
ax.legend(
|
||||
handles=legend_handles,
|
||||
loc="upper left",
|
||||
bbox_to_anchor=(0, -0.02),
|
||||
ncol=4,
|
||||
ncol=5,
|
||||
frameon=False,
|
||||
fontsize=9,
|
||||
labelcolor="#e6edf3",
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
{
|
||||
"summary": {
|
||||
"baseline_ref": "v1.4.0",
|
||||
"total": 24,
|
||||
"old_pass": 21,
|
||||
"new_pass": 24,
|
||||
"old_pct": 87.5,
|
||||
"new_pct": 100.0,
|
||||
"fixed": 3,
|
||||
"regressed": 0
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"check": "core_line_length",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_circle_radius",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_layer_create",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_polyline_closed",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_linear_dim",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_save_dxf_roundtrip",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "dim_aligned_no_error",
|
||||
"category": "Dimensions",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "dim_angular_no_error",
|
||||
"category": "Dimensions",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "array_polar_360_distinct",
|
||||
"category": "Modify",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "point_intersection_line_line",
|
||||
"category": "Geometry",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "point_tangent_external",
|
||||
"category": "Geometry",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arc_has_length",
|
||||
"category": "Query",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arc_select_by_length",
|
||||
"category": "Query",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "ezdxf_bounding_box",
|
||||
"category": "Query",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mtext_rotation_roundtrip",
|
||||
"category": "Entities",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "screenshot_png",
|
||||
"category": "Render",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "dim_overlap_critique_fires",
|
||||
"category": "Quality gate",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "iso13567_dim_layer",
|
||||
"category": "Quality gate",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "construction_left_iso_caught",
|
||||
"category": "Quality gate",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "gear_no_self_overlap",
|
||||
"category": "Engineering",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "center_linetype_applied",
|
||||
"category": "Entities",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "hatch_area_subtracts_its_island",
|
||||
"category": "Measurement",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "boundary_area_agrees_with_measure",
|
||||
"category": "Measurement",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "two_vertex_circle_has_area",
|
||||
"category": "Measurement",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -5,6 +5,21 @@
|
||||
"capabilities": {
|
||||
"backend": "ezdxf",
|
||||
"features": {
|
||||
"audit_detail": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"boundary_trace": {
|
||||
"mode": "computed",
|
||||
"reason": "straight_edges_split_at_intersections;curved_edges_not_split",
|
||||
"supported": true
|
||||
},
|
||||
"chspace": {
|
||||
"mode": "computed",
|
||||
"reason": "top_view_untwisted_viewports_only;dimensions_refused_unless_frozen;acis_proxy_table_refused;viewport_clipping_reported_not_applied",
|
||||
"supported": true
|
||||
},
|
||||
"delivery": {
|
||||
"mode": "shared",
|
||||
"reason": null,
|
||||
@@ -15,26 +30,61 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"dwg": {
|
||||
"mode": null,
|
||||
"reason": "no_dwg_writer_requires_live_autocad",
|
||||
"supported": false
|
||||
},
|
||||
"dxf": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"handle_overlay": {
|
||||
"mode": "rendered",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"hatch_edge_paths": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"lisp": {
|
||||
"mode": null,
|
||||
"reason": "live_com_only",
|
||||
"supported": false
|
||||
},
|
||||
"measure_area_acis": {
|
||||
"mode": null,
|
||||
"reason": "acis_evaluation_requires_live_autocad",
|
||||
"supported": false
|
||||
},
|
||||
"mleader": {
|
||||
"mode": "composite",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"paper_space": {
|
||||
"mtext_background_color": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"ocs_normalized": {
|
||||
"mode": "wcs",
|
||||
"reason": "reported_xy_is_wcs;z_component_not_reported;text_rotation_stays_in_entity_frame;hatch_boundaries_not_reported",
|
||||
"supported": true
|
||||
},
|
||||
"ocs_tilted_plane": {
|
||||
"mode": null,
|
||||
"reason": "planned_1.4",
|
||||
"reason": "2d_xy_cannot_address_a_tilted_plane",
|
||||
"supported": false
|
||||
},
|
||||
"paper_space": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"pdf": {
|
||||
"mode": "rendered",
|
||||
"reason": null,
|
||||
@@ -55,9 +105,14 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"revcloud": {
|
||||
"mode": "native",
|
||||
"reason": "ezdxf_revcloud_helper",
|
||||
"supported": true
|
||||
},
|
||||
"solid_3d": {
|
||||
"mode": null,
|
||||
"reason": "planned_1.5",
|
||||
"reason": "acis_generation_requires_live_autocad",
|
||||
"supported": false
|
||||
},
|
||||
"table": {
|
||||
@@ -69,6 +124,21 @@
|
||||
"mode": "snapshot",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"undo_history": {
|
||||
"mode": null,
|
||||
"reason": "set_EZDXF_UNDO_DEPTH_to_enable",
|
||||
"supported": false
|
||||
},
|
||||
"viewport_render": {
|
||||
"mode": "rendered",
|
||||
"reason": "viewport_borders_not_drawn",
|
||||
"supported": true
|
||||
},
|
||||
"wipeout": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -76,15 +146,16 @@
|
||||
"environment": {
|
||||
"implementation": "CPython",
|
||||
"machine": "AMD64",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"python": "3.14.0"
|
||||
"platform": "Windows-10-10.0.26200-SP0",
|
||||
"python": "3.11.15"
|
||||
},
|
||||
"generated_at": "2026-07-23T11:39:04.277175+00:00",
|
||||
"git_sha": "4a2e9d658f051cb756f92cb8fae945c64b8cc7d4",
|
||||
"generated_at": "2026-08-06T06:27:59.080169+00:00",
|
||||
"git_sha": "d5a0ff5109ddbd37d88371c10a1ed087201de4b5",
|
||||
"matrix": "v3",
|
||||
"results": [
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 8.14,
|
||||
"duration_ms": 2.49,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"line_length": 5.0
|
||||
@@ -97,7 +168,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 3.39,
|
||||
"duration_ms": 1.41,
|
||||
"message": "",
|
||||
"metrics": {},
|
||||
"score": 100.0,
|
||||
@@ -108,7 +179,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 8.11,
|
||||
"duration_ms": 3.31,
|
||||
"message": "",
|
||||
"metrics": {},
|
||||
"score": 100.0,
|
||||
@@ -119,7 +190,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 348.74,
|
||||
"duration_ms": 17.55,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"entity_type": "DIMENSION"
|
||||
@@ -132,7 +203,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 15.74,
|
||||
"duration_ms": 3.7,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"representations": [
|
||||
@@ -148,7 +219,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 28.95,
|
||||
"duration_ms": 29.09,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"entity_count": 0
|
||||
@@ -161,7 +232,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 2.5,
|
||||
"duration_ms": 1.2,
|
||||
"message": "",
|
||||
"metrics": {},
|
||||
"score": 100.0,
|
||||
@@ -172,7 +243,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 29.19,
|
||||
"duration_ms": 8.42,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"final_score": 100.0,
|
||||
@@ -215,11 +286,11 @@
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "roundtrip.dxf",
|
||||
"sha256": "6413204a5baec19a5fbf551418eb56e30b1488ca5bb755e6a8709ac8e8ab0b22",
|
||||
"size_bytes": 18442
|
||||
"sha256": "d11dfce5eddaf849075c62854a8526efaaa3bbd779d374e4c287de77f361caf7",
|
||||
"size_bytes": 18440
|
||||
}
|
||||
],
|
||||
"duration_ms": 16.66,
|
||||
"duration_ms": 32.29,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"entity_count": 1
|
||||
@@ -234,11 +305,11 @@
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "manifest.json",
|
||||
"sha256": "38c2ee819b49b9c66b2311b952e856eff9e3a7565194cce2edea60a3211d233b",
|
||||
"size_bytes": 3581
|
||||
"sha256": "a74b2e96dbe7ea373717f26a8d26e58fe079a61370eb5d646f04a09f1b7cca2f",
|
||||
"size_bytes": 5662
|
||||
}
|
||||
],
|
||||
"duration_ms": 4380.8,
|
||||
"duration_ms": 305.81,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"score": {
|
||||
@@ -255,16 +326,107 @@
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "auditable_delivery"
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 1149.77,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"ranks": {
|
||||
"BPOLY": 1,
|
||||
"CHSPACE": 1,
|
||||
"FILLET": 1,
|
||||
"OVERKILL": 1,
|
||||
"QSELECT": 1,
|
||||
"WBLOCK": 1
|
||||
},
|
||||
"top_n": 3
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "tool_discovery"
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 53.97,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"advertised_tools": 2,
|
||||
"catalog_tokens": 40305,
|
||||
"catalog_tools": 149,
|
||||
"ceiling": 2000,
|
||||
"discovery_tokens": 356,
|
||||
"reduction_x": 113.2,
|
||||
"tokenizer": "ratio/v1 (estimate)"
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "token_budget"
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 10.6,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"area_ignoring_islands": 400.0,
|
||||
"expected_filled": 300.0,
|
||||
"filled_area": 300.0,
|
||||
"hatch_style": "outer",
|
||||
"path_count": 2
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "hatch_islands"
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 8.0,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"bounding_box": 3,
|
||||
"by_layer": 1,
|
||||
"crossing": 2,
|
||||
"polygon": 1,
|
||||
"window": 1
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "selection_filter"
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 5.36,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"boundary_agrees": true,
|
||||
"fraction_lost_by_reading_points_back": 0.282,
|
||||
"measured_area": 139.269908,
|
||||
"true_area": 139.269908,
|
||||
"vertex_shoelace_area": 100.0
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "measure_from_handle"
|
||||
}
|
||||
],
|
||||
"schema_version": "2.0",
|
||||
"summary": {
|
||||
"attempted": 10,
|
||||
"attempted": 15,
|
||||
"coverage_percent": 100.0,
|
||||
"passed": 10,
|
||||
"passed": 15,
|
||||
"score": 100.0,
|
||||
"supported": 10,
|
||||
"supported": 15,
|
||||
"unsupported": 0
|
||||
},
|
||||
"timeout_seconds": 30.0
|
||||
}
|
||||
"timeout_seconds": 180.0
|
||||
}
|
||||
|
||||
@@ -3,8 +3,8 @@
|
||||
"environment": {
|
||||
"implementation": "CPython",
|
||||
"machine": "AMD64",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"python": "3.14.0"
|
||||
"platform": "Windows-10-10.0.26200-SP0",
|
||||
"python": "3.11.15"
|
||||
},
|
||||
"schema_version": "perf/1.0",
|
||||
"workloads": [
|
||||
@@ -14,22 +14,22 @@
|
||||
"entity_count": 2000
|
||||
},
|
||||
"metric": "entities_per_second",
|
||||
"metric_value": 3148.0,
|
||||
"wall_ms": 635.3,
|
||||
"metric_value": 6343.1,
|
||||
"wall_ms": 315.3,
|
||||
"workload_id": "create_lines_2k"
|
||||
},
|
||||
{
|
||||
"description": "Create 10,000 lines, export DXF, reopen and verify",
|
||||
"detail": {
|
||||
"build_ms": 10884.9,
|
||||
"build_ms": 1574.1,
|
||||
"entity_count": 10000,
|
||||
"export_ms": 1439.2,
|
||||
"file_bytes": 1369662,
|
||||
"reopen_ms": 1607.5
|
||||
"export_ms": 261.8,
|
||||
"file_bytes": 1369663,
|
||||
"reopen_ms": 320.2
|
||||
},
|
||||
"metric": "entities_per_second",
|
||||
"metric_value": 717.8,
|
||||
"wall_ms": 13931.6,
|
||||
"metric_value": 4637.9,
|
||||
"wall_ms": 2156.1,
|
||||
"workload_id": "roundtrip_10k"
|
||||
},
|
||||
{
|
||||
@@ -39,7 +39,7 @@
|
||||
},
|
||||
"metric": "entities_matched",
|
||||
"metric_value": 2500.0,
|
||||
"wall_ms": 3434.2,
|
||||
"wall_ms": 890.8,
|
||||
"workload_id": "region_query_10k"
|
||||
},
|
||||
{
|
||||
@@ -49,7 +49,7 @@
|
||||
},
|
||||
"metric": "critique_issues",
|
||||
"metric_value": 0.0,
|
||||
"wall_ms": 315.9,
|
||||
"wall_ms": 176.5,
|
||||
"workload_id": "premium_pass"
|
||||
}
|
||||
]
|
||||
|
||||
@@ -14,8 +14,9 @@ import time
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
from backends.capability import UnsupportedCapabilityError
|
||||
from benchmarks.adapters.base import BenchmarkAdapter, TaskResult
|
||||
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec, task_by_id
|
||||
from benchmarks.tasks_v3 import DEFAULT_MATRIX, MATRICES, TaskSpec, task_by_id
|
||||
|
||||
|
||||
def _load_registry() -> dict[str, dict]:
|
||||
@@ -78,6 +79,7 @@ async def run_tasks(
|
||||
artifact_dir: str | Path,
|
||||
*,
|
||||
timeout: float = 30.0,
|
||||
matrix: str = DEFAULT_MATRIX,
|
||||
) -> dict:
|
||||
destination = Path(artifact_dir).resolve()
|
||||
destination.mkdir(parents=True, exist_ok=True)
|
||||
@@ -90,6 +92,12 @@ async def run_tasks(
|
||||
result = await asyncio.wait_for(adapter.run_task(task), timeout=timeout)
|
||||
except TimeoutError:
|
||||
result = TaskResult(task.task_id, "timeout", 0.0, f"Exceeded {timeout}s")
|
||||
except UnsupportedCapabilityError as exc:
|
||||
# "Your server got this wrong" and "your engine cannot reach
|
||||
# this" are different findings about a competitor. The enum has
|
||||
# carried `unsupported` since v2 for exactly this, but a
|
||||
# refusal is still an exception and used to land as `fail`.
|
||||
result = TaskResult(task.task_id, "unsupported", 0.0, f"[{exc.capability}] {exc}")
|
||||
except Exception as exc:
|
||||
result = TaskResult(task.task_id, "fail", 0.0, str(exc), stderr_summary=str(exc))
|
||||
result.duration_ms = round((time.perf_counter() - started) * 1000, 2)
|
||||
@@ -108,7 +116,11 @@ async def run_tasks(
|
||||
weight_by_id[item.task_id] * item.score for item in results if item.status != "unsupported"
|
||||
)
|
||||
return {
|
||||
# The *report* schema is unchanged — same keys, same enum — so the
|
||||
# chart renderers still read v1.4's published files. `matrix` says
|
||||
# which task set ran, which is the thing that actually moved.
|
||||
"schema_version": "2.0",
|
||||
"matrix": matrix,
|
||||
"adapter": adapter.name,
|
||||
"adapter_metadata": adapter.metadata(),
|
||||
"git_sha": _git_sha(),
|
||||
@@ -137,6 +149,12 @@ async def run_tasks(
|
||||
def _parser(registry: dict[str, dict]) -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--list", action="store_true", help="List the fixed benchmark tasks")
|
||||
parser.add_argument(
|
||||
"--matrix",
|
||||
default=DEFAULT_MATRIX,
|
||||
choices=sorted(MATRICES),
|
||||
help="Task set to run. v2 stays addressable so a v1.4 report can be reproduced.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--server",
|
||||
default="autocad-mcp-pro",
|
||||
@@ -148,14 +166,35 @@ def _parser(registry: dict[str, dict]) -> argparse.ArgumentParser:
|
||||
parser.add_argument("--artifact-dir", default="benchmarks/results/latest")
|
||||
parser.add_argument("--timeout", type=float, default=30.0)
|
||||
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON")
|
||||
parser.add_argument(
|
||||
"--publish",
|
||||
type=Path,
|
||||
help=(
|
||||
"Also write the report here with artifact paths reduced to filenames. "
|
||||
"How results/published/*.json are produced — it was a manual edit before."
|
||||
),
|
||||
)
|
||||
return parser
|
||||
|
||||
|
||||
def _sanitize_for_publication(report: dict) -> dict:
|
||||
"""Drop absolute paths from artifacts, keep the hash that identifies them.
|
||||
|
||||
A published report should not carry the run machine's directory layout;
|
||||
the sha256 is what makes the artifact checkable, and that stays.
|
||||
"""
|
||||
published = json.loads(json.dumps(report))
|
||||
for result in published["results"]:
|
||||
for artifact in result["artifacts"]:
|
||||
artifact.pop("path", None)
|
||||
return published
|
||||
|
||||
|
||||
def main() -> None:
|
||||
registry = _load_registry()
|
||||
args = _parser(registry).parse_args()
|
||||
if args.list:
|
||||
for task in TASKS_V2:
|
||||
for task in MATRICES[args.matrix]:
|
||||
print(f"{task.task_id}\t{task.category}\t{task.description}")
|
||||
return
|
||||
|
||||
@@ -165,10 +204,27 @@ def main() -> None:
|
||||
raise SystemExit(
|
||||
f"{args.server} supports backends {supported_backends}, not {args.backend!r}"
|
||||
)
|
||||
tasks = [task_by_id(task_id) for task_id in args.task] if args.task else list(TASKS_V2)
|
||||
tasks = (
|
||||
[task_by_id(task_id, args.matrix) for task_id in args.task]
|
||||
if args.task
|
||||
else list(MATRICES[args.matrix])
|
||||
)
|
||||
adapter = _make_adapter(entry, args.backend)
|
||||
report = asyncio.run(run_tasks(adapter, tasks, args.artifact_dir, timeout=args.timeout))
|
||||
report = asyncio.run(
|
||||
run_tasks(adapter, tasks, args.artifact_dir, timeout=args.timeout, matrix=args.matrix)
|
||||
)
|
||||
rendered = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
if args.publish:
|
||||
args.publish.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.publish.write_text(
|
||||
json.dumps(
|
||||
_sanitize_for_publication(report), ensure_ascii=False, indent=2, sort_keys=True
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
newline="\n",
|
||||
)
|
||||
print(f"Published: {args.publish}", file=sys.stderr)
|
||||
print(rendered)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
"""The v3 task matrix: the v2 ten, plus five the v2 ten could not see.
|
||||
|
||||
v2 scored this server 10/10, and that number was structurally uninformative:
|
||||
every task in it exercised something the server was already built around. A
|
||||
matrix that cannot fail is a matrix that measures nothing, so the five added
|
||||
here were picked to cover capability the v2 set never touches — discovery,
|
||||
idle token cost, hatch islands, selection semantics, and measuring an entity
|
||||
that is already in the drawing.
|
||||
|
||||
Two of the five are renamed from the release plan, and the renames are the
|
||||
point rather than bookkeeping:
|
||||
|
||||
``tool_discovery_bilingual`` -> ``tool_discovery``
|
||||
The plan assumed a Turkish query normalizer. One was written and then
|
||||
removed: this repository is English throughout, and the Turkish was an
|
||||
inference from the author's chat language, not a product requirement. A
|
||||
task named "bilingual" would advertise a capability that does not exist,
|
||||
which is the class of claim this release spent four milestones deleting.
|
||||
|
||||
``measurement_massprops`` -> ``measure_from_handle``
|
||||
No mass properties ship. The plan's eight-tool measurement family (F2) was
|
||||
cut to one after measurement showed the repo already carried eight
|
||||
``analysis_*`` tools; centroids and moments of inertia were not among the
|
||||
survivors. What did ship is measuring a *handle* instead of a caller's
|
||||
remembered coordinates, so that is what the task is called.
|
||||
|
||||
The v2 ids and weights are untouched. The published competitor reports were
|
||||
run against the v2 matrix at v1.4 and are not re-scored here — those servers
|
||||
have never been asked these five questions, and inventing a result for them
|
||||
would be worse than the gap. The task-matrix chart renders them ``not_run``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from benchmarks.tasks_v2 import TASKS_V2, TaskSpec
|
||||
|
||||
#: Added in v1.5.0. Every one of these can fail on this server.
|
||||
NEW_TASKS_V3: tuple[TaskSpec, ...] = (
|
||||
TaskSpec(
|
||||
"tool_discovery",
|
||||
"discovery",
|
||||
"Resolve AutoCAD command names to tools through the search layer",
|
||||
1.0,
|
||||
),
|
||||
TaskSpec(
|
||||
"token_budget",
|
||||
"efficiency",
|
||||
"Keep the idle catalog under a fixed token ceiling",
|
||||
1.0,
|
||||
),
|
||||
TaskSpec(
|
||||
"hatch_islands",
|
||||
"hatch",
|
||||
"Hatch an area with an island and measure what it fills",
|
||||
1.0,
|
||||
),
|
||||
TaskSpec(
|
||||
"selection_filter",
|
||||
"selection",
|
||||
"Separate window from crossing, and a polygon from its bounding box",
|
||||
1.0,
|
||||
),
|
||||
TaskSpec(
|
||||
"measure_from_handle",
|
||||
"measurement",
|
||||
"Measure a curved boundary from its handle, bulges included",
|
||||
1.0,
|
||||
),
|
||||
)
|
||||
|
||||
TASKS_V3: tuple[TaskSpec, ...] = TASKS_V2 + NEW_TASKS_V3
|
||||
|
||||
#: Runner-selectable task sets. v2 stays addressable so a v2 report can be
|
||||
#: reproduced exactly rather than only described.
|
||||
MATRICES: dict[str, tuple[TaskSpec, ...]] = {"v2": TASKS_V2, "v3": TASKS_V3}
|
||||
|
||||
DEFAULT_MATRIX = "v3"
|
||||
|
||||
|
||||
def task_by_id(task_id: str, matrix: str = DEFAULT_MATRIX) -> TaskSpec:
|
||||
for task in MATRICES[matrix]:
|
||||
if task.task_id == task_id:
|
||||
return task
|
||||
raise KeyError(f"Unknown benchmark task: {task_id}")
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user