diff --git a/README.md b/README.md index 11ae7f3..642944f 100644 --- a/README.md +++ b/README.md @@ -36,12 +36,42 @@ Extras: `[com]` live AutoCAD (pywin32 + Pillow), `[pdf]` PDF/screenshot rendering (matplotlib), `[full]` everything. Working from a checkout is the same commands with `pip install -e ".[full]"` and `python server.py`. -## Benchmarks — three lanes, all reproducible +## Benchmarks Every number below is generated by a script in [`benchmarks/`](benchmarks/) and rendered to SVG with Python (matplotlib). No hand-edited charts, no adjectives. +### The leaderboard — nine public AutoCAD MCP servers, one rubric + +![Public AutoCAD MCP capability leaderboard](docs/assets/autocad-mcp-benchmark.svg) + +| # | Project | Score | Evidence | +|--:|---|---:|:---:| +| 1 | **[U-C4N/Autocad-MCP](https://github.com/U-C4N/Autocad-MCP)** (this repo, v1.4.0) | **95** | A | +| 2 | [varavista/autocad-mcp](https://github.com/varavista/autocad-mcp) | 84 | A | +| 3 | [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | 83 | A | +| 4 | [LokmenoWer/best-cad-mcp](https://github.com/LokmenoWer/best-cad-mcp) | 74 | B | +| 5 | [AnCode666/multiCAD-mcp](https://github.com/AnCode666/multiCAD-mcp) | 73 | A | +| 6 | [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | 70 | A | +| 7 | [NCO-1986/AutoCAD_mcp](https://github.com/NCO-1986/AutoCAD_mcp) | 68 | B | +| 8 | [daobataotie/CAD-MCP](https://github.com/daobataotie/CAD-MCP) | 35 | B | +| 9 | [zh19980811/Easy-MCP-AutoCad](https://github.com/zh19980811/Easy-MCP-AutoCad) | 30 | C | + +Fixed public 100-point rubric — CAD breadth (25), correctness and delivery +(20), backends/platforms (15), engineering production (15), tests and +maintenance (15), security and operations (10) — applied to the public source +and documentation of each project +([`benchmarks/source_review.json`](benchmarks/source_review.json), dated). +Stars and raw tool counts score nothing. Evidence grade **A** = source plus +locally executed evidence, **B** = deep source review without a full live run, +**C** = documented tool surface only. This is a source review, not a shared +live run — the lanes below are the runtime evidence. + +```bash +python -m benchmarks.render_chart +``` + ### 1 · Live-run scores — same harness, pinned competitors ![Live benchmark](docs/assets/autocad-mcp-livebench.svg) @@ -129,18 +159,7 @@ For contrast, the same suite caught the v1.0.0 → v1.1.0 jump: python benchmarks/compare_versions.py v1.3.0 --json ab.json ``` -### 5 · Source-reviewed capability rubric (context) - -![Source-reviewed AutoCAD MCP capability benchmark](docs/assets/autocad-mcp-benchmark.svg) - -Six named AutoCAD MCP projects scored against a fixed public 100-point rubric -(CAD breadth, correctness and delivery, backend reach, engineering production, -tests, security), generated from -[`benchmarks/source_review.json`](benchmarks/source_review.json). This is a -dated source review — evidence grade **A** means source plus executable local -evidence was reviewed; **B** means the live AutoCAD path was not fully -executed. It is not a shared live run; lanes 1-3 above are the runtime -evidence. Full rubric, caveats and the version A/B suite: +Full rubric, caveats, method and boundaries for every lane: [`benchmarks/README.md`](benchmarks/README.md). ## What ships in v1.4 diff --git a/benchmarks/README.md b/benchmarks/README.md index b0ae3bd..5df4130 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,10 +1,12 @@ # Benchmarks +![Public AutoCAD MCP capability leaderboard](../docs/assets/autocad-mcp-benchmark.svg) + ## Two evidence layers This repository publishes two deliberately separate forms of evidence: -1. **Source-reviewed capability benchmark.** Six named AutoCAD MCP projects are +1. **Source-reviewed capability benchmark.** Nine named AutoCAD MCP projects are scored with a fixed 100-point rubric covering CAD breadth, correctness and delivery, backend reach, engineering production, tests, and security. The dated data lives in [`source_review.json`](source_review.json), and the README diff --git a/benchmarks/source_review.json b/benchmarks/source_review.json index d5446b3..7e2951a 100644 --- a/benchmarks/source_review.json +++ b/benchmarks/source_review.json @@ -1,6 +1,6 @@ { "schema_version": "1.0", - "reviewed_at": "2026-07-17", + "reviewed_at": "2026-07-23", "title": "AutoCAD MCP source-reviewed capability benchmark", "evidence_boundary": "source_review_not_shared_live_run", "methodology": "Public source, documented behavior, and locally recorded test evidence scored with a fixed 100-point rubric. Repository stars and raw tool counts do not score points.", @@ -16,10 +16,10 @@ { "repository": "U-C4N/Autocad-MCP", "url": "https://github.com/U-C4N/Autocad-MCP", - "reviewed_version": "1.3.0 working tree", - "score": 93, + "reviewed_version": "1.4.0 (PyPI release)", + "score": 95, "evidence_grade": "A", - "note": "Dual COM/ezdxf backends, standards-aware quality gates, deterministic geometry, closed-loop refinement, and auditable delivery." + "note": "Dual COM/ezdxf backends, paper space, ISO 129/286/1101 annotation, closed-loop quality gates, auditable delivery; v1.4 adds CI matrix incl. Windows, PyPI packaging, and a version A/B correctness gate (21/21, 0 regressions)." }, { "repository": "varavista/autocad-mcp", @@ -60,6 +60,30 @@ "score": 68, "evidence_grade": "B", "note": ".NET main-thread integration with a broad live AutoCAD operation surface." + }, + { + "repository": "LokmenoWer/best-cad-mcp", + "url": "https://github.com/LokmenoWer/best-cad-mcp", + "reviewed_version": "v1.5.0 (2026-07-23 snapshot)", + "score": 74, + "evidence_grade": "B", + "note": "Very broad live-AutoCAD operation surface with CAD-IR understanding, visual grounding, and guarded CADPlan dry-run/rollback; COM-only (no headless lane), no standards-based generators or toleranced annotation." + }, + { + "repository": "daobataotie/CAD-MCP", + "url": "https://github.com/daobataotie/CAD-MCP", + "reviewed_version": "2026-07-23 snapshot", + "score": 35, + "evidence_grade": "B", + "note": "Multi-CAD reach (AutoCAD/GstarCAD/ZWCAD) over Windows COM with a small basic drawing surface; no verification, transactions, tests, or delivery evidence documented." + }, + { + "repository": "zh19980811/Easy-MCP-AutoCad", + "url": "https://github.com/zh19980811/Easy-MCP-AutoCad", + "reviewed_version": "2026-07-23 snapshot", + "score": 30, + "evidence_grade": "C", + "note": "Basic COM drawing tool set with SQLite persistence; reviewed from the documented tool surface only, execution path not exercised locally." } ] } diff --git a/docs/assets/autocad-mcp-benchmark.svg b/docs/assets/autocad-mcp-benchmark.svg index a60f51e..0f6f7c7 100644 --- a/docs/assets/autocad-mcp-benchmark.svg +++ b/docs/assets/autocad-mcp-benchmark.svg @@ -30,8 +30,8 @@ z @@ -41,7 +41,7 @@ z +" clip-path="url(#p5a75d4577b)" style="fill: none; stroke: #d0d7de; stroke-opacity: 0.8; stroke-width: 0.8; stroke-linecap: square"/> @@ -85,18 +85,18 @@ z - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + - + - - + + + - + @@ -1030,11 +1161,67 @@ z - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - @@ -1137,11 +1287,11 @@ z - - - + + + - + - - + + + - + +" clip-path="url(#p5a75d4577b)" style="fill: #6e7781"/> - +" clip-path="url(#p5a75d4577b)" style="fill: #6e7781"/> - +" clip-path="url(#p5a75d4577b)" style="fill: #6e7781"/> - +" clip-path="url(#p5a75d4577b)" style="fill: #6e7781"/> - +" clip-path="url(#p5a75d4577b)" style="fill: #6e7781"/> - + + + + + + + + + +" clip-path="url(#p5a75d4577b)" style="fill: #0969da"/> - - + + - - + + @@ -1411,9 +1585,178 @@ z - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - @@ -1535,9 +1838,9 @@ z - + - + - - + - - - - + @@ -1644,22 +1892,9 @@ z - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -1691,9 +1952,9 @@ z - - - + + + - + @@ -1736,7 +1997,7 @@ z - + @@ -1899,7 +2160,7 @@ z - + @@ -2375,8 +2636,8 @@ z - - + +