docs(bench): nine-server public AutoCAD MCP leaderboard as the README hero visual
source_review.json extended with three projects reviewed 2026-07-23 (LokmenoWer/best-cad-mcp B/74, daobataotie/CAD-MCP B/35, zh19980811/Easy-MCP-AutoCad C/30) and our entry updated to the released v1.4.0 (95: CI matrix, PyPI packaging, A/B gate, paper space, ISO 286, opt-in solids). Chart regenerated via render_chart; README benchmark section now opens with the leaderboard + 9-row table, source-review lane merged into it. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
co-authored by
Cursor
parent
47a90dbc54
commit
af4813e7b6
@@ -36,12 +36,42 @@ Extras: `[com]` live AutoCAD (pywin32 + Pillow), `[pdf]` PDF/screenshot
|
||||
rendering (matplotlib), `[full]` everything. Working from a checkout is the
|
||||
same commands with `pip install -e ".[full]"` and `python server.py`.
|
||||
|
||||
## Benchmarks — three lanes, all reproducible
|
||||
## Benchmarks
|
||||
|
||||
Every number below is generated by a script in [`benchmarks/`](benchmarks/)
|
||||
and rendered to SVG with Python (matplotlib). No hand-edited charts, no
|
||||
adjectives.
|
||||
|
||||
### The leaderboard — nine public AutoCAD MCP servers, one rubric
|
||||
|
||||

|
||||
|
||||
| # | Project | Score | Evidence |
|
||||
|--:|---|---:|:---:|
|
||||
| 1 | **[U-C4N/Autocad-MCP](https://github.com/U-C4N/Autocad-MCP)** (this repo, v1.4.0) | **95** | A |
|
||||
| 2 | [varavista/autocad-mcp](https://github.com/varavista/autocad-mcp) | 84 | A |
|
||||
| 3 | [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | 83 | A |
|
||||
| 4 | [LokmenoWer/best-cad-mcp](https://github.com/LokmenoWer/best-cad-mcp) | 74 | B |
|
||||
| 5 | [AnCode666/multiCAD-mcp](https://github.com/AnCode666/multiCAD-mcp) | 73 | A |
|
||||
| 6 | [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | 70 | A |
|
||||
| 7 | [NCO-1986/AutoCAD_mcp](https://github.com/NCO-1986/AutoCAD_mcp) | 68 | B |
|
||||
| 8 | [daobataotie/CAD-MCP](https://github.com/daobataotie/CAD-MCP) | 35 | B |
|
||||
| 9 | [zh19980811/Easy-MCP-AutoCad](https://github.com/zh19980811/Easy-MCP-AutoCad) | 30 | C |
|
||||
|
||||
Fixed public 100-point rubric — CAD breadth (25), correctness and delivery
|
||||
(20), backends/platforms (15), engineering production (15), tests and
|
||||
maintenance (15), security and operations (10) — applied to the public source
|
||||
and documentation of each project
|
||||
([`benchmarks/source_review.json`](benchmarks/source_review.json), dated).
|
||||
Stars and raw tool counts score nothing. Evidence grade **A** = source plus
|
||||
locally executed evidence, **B** = deep source review without a full live run,
|
||||
**C** = documented tool surface only. This is a source review, not a shared
|
||||
live run — the lanes below are the runtime evidence.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.render_chart
|
||||
```
|
||||
|
||||
### 1 · Live-run scores — same harness, pinned competitors
|
||||
|
||||

|
||||
@@ -129,18 +159,7 @@ For contrast, the same suite caught the v1.0.0 → v1.1.0 jump:
|
||||
python benchmarks/compare_versions.py v1.3.0 --json ab.json
|
||||
```
|
||||
|
||||
### 5 · Source-reviewed capability rubric (context)
|
||||
|
||||

|
||||
|
||||
Six named AutoCAD MCP projects scored against a fixed public 100-point rubric
|
||||
(CAD breadth, correctness and delivery, backend reach, engineering production,
|
||||
tests, security), generated from
|
||||
[`benchmarks/source_review.json`](benchmarks/source_review.json). This is a
|
||||
dated source review — evidence grade **A** means source plus executable local
|
||||
evidence was reviewed; **B** means the live AutoCAD path was not fully
|
||||
executed. It is not a shared live run; lanes 1-3 above are the runtime
|
||||
evidence. Full rubric, caveats and the version A/B suite:
|
||||
Full rubric, caveats, method and boundaries for every lane:
|
||||
[`benchmarks/README.md`](benchmarks/README.md).
|
||||
|
||||
## What ships in v1.4
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
# Benchmarks
|
||||
|
||||

|
||||
|
||||
## Two evidence layers
|
||||
|
||||
This repository publishes two deliberately separate forms of evidence:
|
||||
|
||||
1. **Source-reviewed capability benchmark.** Six named AutoCAD MCP projects are
|
||||
1. **Source-reviewed capability benchmark.** Nine named AutoCAD MCP projects are
|
||||
scored with a fixed 100-point rubric covering CAD breadth, correctness and
|
||||
delivery, backend reach, engineering production, tests, and security. The
|
||||
dated data lives in [`source_review.json`](source_review.json), and the README
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"reviewed_at": "2026-07-17",
|
||||
"reviewed_at": "2026-07-23",
|
||||
"title": "AutoCAD MCP source-reviewed capability benchmark",
|
||||
"evidence_boundary": "source_review_not_shared_live_run",
|
||||
"methodology": "Public source, documented behavior, and locally recorded test evidence scored with a fixed 100-point rubric. Repository stars and raw tool counts do not score points.",
|
||||
@@ -16,10 +16,10 @@
|
||||
{
|
||||
"repository": "U-C4N/Autocad-MCP",
|
||||
"url": "https://github.com/U-C4N/Autocad-MCP",
|
||||
"reviewed_version": "1.3.0 working tree",
|
||||
"score": 93,
|
||||
"reviewed_version": "1.4.0 (PyPI release)",
|
||||
"score": 95,
|
||||
"evidence_grade": "A",
|
||||
"note": "Dual COM/ezdxf backends, standards-aware quality gates, deterministic geometry, closed-loop refinement, and auditable delivery."
|
||||
"note": "Dual COM/ezdxf backends, paper space, ISO 129/286/1101 annotation, closed-loop quality gates, auditable delivery; v1.4 adds CI matrix incl. Windows, PyPI packaging, and a version A/B correctness gate (21/21, 0 regressions)."
|
||||
},
|
||||
{
|
||||
"repository": "varavista/autocad-mcp",
|
||||
@@ -60,6 +60,30 @@
|
||||
"score": 68,
|
||||
"evidence_grade": "B",
|
||||
"note": ".NET main-thread integration with a broad live AutoCAD operation surface."
|
||||
},
|
||||
{
|
||||
"repository": "LokmenoWer/best-cad-mcp",
|
||||
"url": "https://github.com/LokmenoWer/best-cad-mcp",
|
||||
"reviewed_version": "v1.5.0 (2026-07-23 snapshot)",
|
||||
"score": 74,
|
||||
"evidence_grade": "B",
|
||||
"note": "Very broad live-AutoCAD operation surface with CAD-IR understanding, visual grounding, and guarded CADPlan dry-run/rollback; COM-only (no headless lane), no standards-based generators or toleranced annotation."
|
||||
},
|
||||
{
|
||||
"repository": "daobataotie/CAD-MCP",
|
||||
"url": "https://github.com/daobataotie/CAD-MCP",
|
||||
"reviewed_version": "2026-07-23 snapshot",
|
||||
"score": 35,
|
||||
"evidence_grade": "B",
|
||||
"note": "Multi-CAD reach (AutoCAD/GstarCAD/ZWCAD) over Windows COM with a small basic drawing surface; no verification, transactions, tests, or delivery evidence documented."
|
||||
},
|
||||
{
|
||||
"repository": "zh19980811/Easy-MCP-AutoCad",
|
||||
"url": "https://github.com/zh19980811/Easy-MCP-AutoCad",
|
||||
"reviewed_version": "2026-07-23 snapshot",
|
||||
"score": 30,
|
||||
"evidence_grade": "C",
|
||||
"note": "Basic COM drawing tool set with SQLite persistence; reviewed from the documented tool surface only, execution path not exercised locally."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
|
Before Width: | Height: | Size: 70 KiB After Width: | Height: | Size: 80 KiB |
Reference in New Issue
Block a user