Umut CANandClaude Opus 5
b03a7615b1
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
281 lines
14 KiB
Markdown
281 lines
14 KiB
Markdown
# Benchmarks
|
||
|
||
|
||
|
||
## Two evidence layers
|
||
|
||
This repository publishes two deliberately separate forms of evidence:
|
||
|
||
1. **Source-reviewed capability benchmark.** Nine named AutoCAD MCP projects are
|
||
scored with a fixed 100-point rubric covering CAD breadth, correctness and
|
||
delivery, backend reach, engineering production, tests, and security. The
|
||
dated data lives in [`source_review.json`](source_review.json), and the README
|
||
graphic is generated—not hand-edited—from that file.
|
||
2. **Fixed-task runtime benchmark.** Adapters execute the same task matrix and
|
||
return `pass`, `partial`, `unsupported`, `fail`, `timeout`, or `not_run`.
|
||
AutoCAD MCP Pro currently has the reference adapter; other projects do not
|
||
receive runtime scores until an adapter actually runs their public interface.
|
||
|
||
Regenerate the source-review chart:
|
||
|
||
```bash
|
||
pip install -e ".[pdf]"
|
||
python -m benchmarks.render_chart
|
||
```
|
||
|
||
The source-review chart is not presented as a shared live AutoCAD run. Review
|
||
dates, evidence grades, project URLs, and the boundary are stored with the data.
|
||
|
||
## Fixed-task runtime runner
|
||
|
||
The runner separates runtime evidence from the source-review rubric. Every
|
||
adapter receives the same tasks and produces the same closed result enum:
|
||
`pass`, `partial`, `unsupported`, `fail`, `timeout`, or `not_run`. A timeout is
|
||
reported per task, and a capability refusal is recorded `unsupported` rather
|
||
than `fail` — "cannot reach this" and "got this wrong" are different findings.
|
||
Execution is in-process, so adapters that delegate blocking work must provide
|
||
cancellation-safe task implementations or be wrapped by an external process
|
||
supervisor. Reports include commit SHA, Python/platform details, backend
|
||
capability claims, durations, coverage, and hashes for returned artifacts.
|
||
Unsupported tasks remain in the fixed-matrix denominator with score zero,
|
||
preventing partial implementations from receiving an inflated score.
|
||
|
||
```bash
|
||
python -m benchmarks.run_competitors --list --matrix v3
|
||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v3 --json
|
||
python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
|
||
```
|
||
|
||
The release-machine ezdxf self-check is **15/15 (100.0)** on the v3 matrix.
|
||
Repository stars and raw tool counts do not contribute to the score. Adapter
|
||
registration lives in `competitors.yaml`.
|
||
|
||
### Matrix v3 (v1.5.0) — five tasks that can fail
|
||
|
||
v2 scored this server 10/10, and that number carried no information: every task
|
||
in it exercised something the server was designed around. `tasks_v3.py` keeps
|
||
the v2 ten unchanged — their ids and weights key the published v1.4 reports —
|
||
and adds five chosen because they *can* fail. Three of them did, while being
|
||
written:
|
||
|
||
| Task | Category | Verified against |
|
||
|---|---|---|
|
||
| `tool_discovery` | discovery | six AutoCAD command names, each ranking #1 of 154 tools |
|
||
| `token_budget` | efficiency | 40,305 → 356 tokens advertised, against a 2,000 ceiling fixed in advance |
|
||
| `hatch_islands` | hatch | 300 filled with the island, 400 ignoring it |
|
||
| `selection_filter` | selection | window 1, crossing 2, bounding box 3, polygon 1 |
|
||
| `measure_from_handle` | measurement | 139.2699 against the 100.0 a vertex shoelace gives |
|
||
|
||
Token counts come from the offline ratio estimator and the report labels them
|
||
`ratio/v1 (estimate)`; `token_suite.py --tokenizer anthropic` counts for real.
|
||
|
||
**Two renames from the release plan, and they are the point.**
|
||
`tool_discovery_bilingual` → `tool_discovery`: a Turkish query normalizer was
|
||
written and then removed, because this repository is English throughout and the
|
||
Turkish was inferred from the author's chat language rather than required by the
|
||
product. `measurement_massprops` → `measure_from_handle`: no centroid or
|
||
moment-of-inertia tool ships.
|
||
|
||
**The competitor reports are not re-scored.** Those servers are pinned at v1.4
|
||
commits and were run against the v2 matrix; they have never been asked these
|
||
five questions. Their scores stand over the ten they ran, and the matrix chart
|
||
leaves the five new rows blank (`not run`) instead of scoring them zero — a
|
||
zero we invented would be indistinguishable from a zero we measured.
|
||
|
||
`--matrix v2` stays available so a v1.4 report can be reproduced rather than
|
||
only described, and `--publish` writes the artifact-path-free file that
|
||
`results/published/` holds.
|
||
|
||
## Live competitor lane (v1.4)
|
||
|
||
Two competitor adapters now execute the exact same task matrix, black-box over
|
||
MCP stdio, against commits pinned in `competitors.yaml`:
|
||
|
||
| Server | Matrix | Pinned | Score | Pass | Coverage |
|
||
|---|---|---|---:|---:|---:|
|
||
| autocad-mcp-pro (reference) | v3 (15) | working tree | 100.0 | 15/15 | 100% |
|
||
| beiming183-cloud/AutoCAD-MCP | v2 (10) | `11f7c47e` | 50.0 | 5/10 | 50% |
|
||
| puran-water/autocad-mcp | v2 (10) | `95476a33` | 45.0 | 4/10 | 50% |
|
||
|
||
The competitor rows are the v1.4 runs, unchanged. Read the matrix column
|
||
before the score column: the denominators differ because the five v3 tasks
|
||
were added after those runs and have not been put to them.
|
||
|
||
Method and boundaries:
|
||
|
||
- `benchmarks/competitors_env.py` clones the pinned SHA into
|
||
`benchmarks/.competitors/<id>/` (gitignored), builds an isolated venv, and
|
||
installs the competitor from its own `pyproject.toml`.
|
||
- `benchmarks/adapters/mcp_stdio.py` drives the competitor exactly like an MCP
|
||
host (fastmcp `Client` over stdio). No competitor code is imported.
|
||
- Task playbooks call each server's **own documented tool contract**, read from
|
||
the pinned source (consolidated `operation` + payload tools; beiming's
|
||
`doc_id`/`expected_revision` discipline is honored via
|
||
`transaction(operation="context")`).
|
||
- **Verification never trusts the competitor's response**: every geometry claim
|
||
must survive `save`/`save_as_dxf` and re-opening the DXF with ezdxf inside
|
||
the harness.
|
||
- Tasks with no documented equivalent are reported `unsupported` with the
|
||
reason string; they stay in the fixed-matrix denominator at score zero.
|
||
- This lane is headless (ezdxf backends only). File-IPC / live-AutoCAD lanes
|
||
(including COM-only servers such as best-cad-mcp or daobataotie/CAD-MCP)
|
||
require a local AutoCAD session and are out of CI scope by design.
|
||
|
||
Published machine-readable reports (artifact paths sanitized to filenames)
|
||
live under [`results/published/`](results/published/); the README charts are
|
||
regenerated from them:
|
||
|
||
```bash
|
||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf --matrix v2
|
||
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf --matrix v2
|
||
python -m benchmarks.render_live_chart # score bars
|
||
python -m benchmarks.render_matrix_chart # tasks x servers status heatmap
|
||
```
|
||
|
||
`--publish <path>` writes the same report with artifact paths reduced to
|
||
filenames, which is how `results/published/` is produced; the sha256 stays,
|
||
because that is what makes an artifact checkable.
|
||
|
||
## Headless performance lane
|
||
|
||
`perf_suite.py` measures wall time on fixed workloads through the same backend
|
||
methods the MCP tools call (server-side overhead included): 2,000 individual
|
||
line creates, a 10,000-entity build → DXF export → reopen roundtrip, a region
|
||
query over 10,000 entities, and a full premium quality pass (ISO layers,
|
||
geometry, dimensions, complete critique). The report records the machine
|
||
fingerprint; numbers move with hardware, workload definitions do not.
|
||
**Self-measurement only** — competitor servers would pay an extra stdio
|
||
serialization cost that in-process runs do not, so no cross-server timing
|
||
claims are made.
|
||
|
||
```bash
|
||
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||
python -m benchmarks.render_perf_chart
|
||
```
|
||
|
||
### v1.4.0 → v1.5.0, same interpreter
|
||
|
||
The published v1.4 report was recorded on CPython 3.14 and the v1.5 one on
|
||
3.11.15, so comparing the two files directly shows an apparent 2–6× speedup
|
||
that is almost entirely the interpreter. The honest comparison re-runs v1.4.0
|
||
in a `git worktree` with the *current* suite file — same instrument, different
|
||
code under test, one interpreter. Median of three runs each:
|
||
|
||
| Workload | v1.4.0 | v1.5.0 | Change |
|
||
|---|---:|---:|---|
|
||
| `create_lines_2k` | 245 ms | 315 ms | **28% slower** |
|
||
| `roundtrip_10k` | 1,765 ms | 2,146 ms | **22% slower** |
|
||
| `region_query_10k` | 1,090 ms | 1,039 ms | within spread |
|
||
| `premium_pass` | 375 ms | 44 ms | faster; the v1.4 spread (279–772 ms) is too wide to quantify |
|
||
|
||
Attributed rather than reported: `EZDXF_CALL_TIMEOUT=0` puts v1.5.0 back to
|
||
244.8 ms and 1,679 ms, which are v1.4.0's numbers. The entire cost is the
|
||
per-call `asyncio.wait_for` wrapped around every headless call in this release,
|
||
so a single hung ezdxf call can no longer wedge a server whose document lock is
|
||
one `asyncio.Lock`. Reproduce:
|
||
|
||
```bash
|
||
git worktree add /tmp/wt-v140 v1.4.0
|
||
cp benchmarks/perf_suite.py /tmp/wt-v140/benchmarks/
|
||
(cd /tmp/wt-v140 && PYTHONPATH=. python benchmarks/perf_suite.py --out /tmp/v140.json)
|
||
python -m benchmarks.perf_suite --out /tmp/v150.json
|
||
EZDXF_CALL_TIMEOUT=0 python -m benchmarks.perf_suite --out /tmp/v150-notimeout.json
|
||
git worktree remove --force /tmp/wt-v140
|
||
```
|
||
|
||
## Correctness A/B suite
|
||
|
||
`correctness_suite.py` is a set of deterministic, headless (ezdxf-backend) checks
|
||
that exercise real drawing operations. `compare_versions.py` runs the **same**
|
||
suite against an older git ref and the current checkout, each check in its own
|
||
subprocess so a hard crash (e.g. a matplotlib-in-thread `SIGSEGV`) is recorded as
|
||
a miss rather than taking down the run.
|
||
|
||
Reproduce:
|
||
|
||
```bash
|
||
python benchmarks/compare_versions.py # current tree vs origin/main
|
||
python benchmarks/compare_versions.py v1.0.0 # vs a tag/ref
|
||
python benchmarks/compare_versions.py --json results.json
|
||
```
|
||
|
||
### Result — v1.5.0 vs v1.4.0 (release gate)
|
||
|
||
24 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||
[`results/published/ab-v1.4.0-vs-v1.5.0.json`](results/published/ab-v1.4.0-vs-v1.5.0.json).
|
||
|
||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||
|---------|----------------|-----------|-------|-----------|
|
||
| **v1.4.0** (baseline) | 21 / 24 | 87.5 % | — | — |
|
||
| **v1.5.0** (this release) | 24 / 24 | 100 % | 3 | **0** |
|
||
|
||
Read the three "fixed" carefully: they are **new capability, not repaired
|
||
regressions**. v1.4.0 has no `entity_measure` and no boundary tracing, so it
|
||
misses `hatch_area_subtracts_its_island`, `boundary_area_agrees_with_measure`
|
||
and `two_vertex_circle_has_area` by not having the method rather than by
|
||
getting it wrong. They are in the suite because all three broke during v1.5.0
|
||
development and nothing outside their own unit tests would have caught it.
|
||
|
||
The 21 pre-existing checks are unchanged and all still pass: the discovery
|
||
layer, `cad_batch`, the 19-module contract split, the layout/viewport family
|
||
and Wave A landed with **zero correctness regressions**.
|
||
|
||
### Result — v1.4.0 vs v1.3.0
|
||
|
||
21 checks. Report:
|
||
[`results/published/ab-v1.3.0-vs-v1.4.0.json`](results/published/ab-v1.3.0-vs-v1.4.0.json).
|
||
|
||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||
|---------|----------------|-----------|-------|-----------|
|
||
| **v1.3.0** (baseline) | 21 / 21 | 100 % | — | — |
|
||
| **v1.4.0** | 21 / 21 | 100 % | 0 | **0** |
|
||
|
||
### Result — v1.1.0 vs v1.0.0 (`origin/main`, commit 15fa2bc)
|
||
|
||
21 checks, ezdxf backend, CPython 3.14, one subprocess per check.
|
||
|
||
| Version | Checks passing | Pass rate |
|
||
|---------|----------------|-----------|
|
||
| **v1.0.0** (public release) | 8 / 21 | **38.1 %** |
|
||
| **v1.1.0** (this release) | 21 / 21 | **100 %** |
|
||
|
||
**13 defects fixed, 0 regressions** — a 2.6× higher correctness pass-rate (+61.9 pts).
|
||
|
||
Fixed (failed on v1.0.0, pass on v1.1.0):
|
||
|
||
| Check | Category | What it proves |
|
||
|-------|----------|----------------|
|
||
| `dim_aligned_no_error` | Dimensions | aligned dim no longer raises `TypeError` |
|
||
| `dim_angular_no_error` | Dimensions | angular dim no longer raises `TypeError` |
|
||
| `array_polar_360_distinct` | Modify | full-circle array no longer duplicates the original |
|
||
| `point_intersection_line_line` | Geometry | deterministic line/line intersection exists & is correct |
|
||
| `point_tangent_external` | Geometry | external-point tangent is perpendicular & on-circle |
|
||
| `arc_has_length` | Query | ARC carries a `length` property |
|
||
| `arc_select_by_length` | Query | `entity_select_smart` length_range selects arcs |
|
||
| `ezdxf_bounding_box` | Query | `bounding_box` populated (COM/ezdxf parity) |
|
||
| `mtext_rotation_roundtrip` | Entities | MTEXT honors caller rotation |
|
||
| `screenshot_png` | Render | headless render returns a valid PNG (no GUI-thread crash) |
|
||
| `dim_overlap_critique_fires` | Quality gate | `dim_overlap` critique is live (was a no-op) |
|
||
| `iso13567_dim_layer` | Quality gate | `dimension_auto` lands dims on the active set's layer |
|
||
| `gear_no_self_overlap` | Engineering | gear outline never dips inside the root circle (z ≥ 42) |
|
||
|
||
Passing on **both** versions (6 core ops + 2 others) keep the suite honest — it is
|
||
not a cherry-picked list of failures; the baseline genuinely does basic CAD work.
|
||
|
||
### Caveats / honesty notes
|
||
|
||
- The baseline is `origin/main` (the public v1.0.0). All four audited fix sprints
|
||
land between it and v1.1.0 — see `docs/analysis/`.
|
||
- `screenshot_png` on the baseline is host-dependent: it is recorded as a miss
|
||
here because the default matplotlib backend is a GUI backend and the render runs
|
||
off the main thread. On a host where matplotlib defaults to `Agg`, the baseline
|
||
may not crash. v1.1.0 forces `Agg` and is deterministic.
|
||
- `center_linetype_applied` passes on both because it only checks the linetype
|
||
**attribute** string (set on both); the v1.0.0 "renders as Continuous because the
|
||
linetype was never loaded" defect is not observable through `EntityInfo`.
|
||
- COM-backend behaviour is not covered here (no live AutoCAD); see the mocked-COM
|
||
suites in `tests/`.
|
||
|
||
Model-by-model tool-call success rates remain future work; scale/throughput
|
||
numbers live in the headless performance lane above.
|