feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only - benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time) - benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports) - render_live_chart now tolerates other lanes sharing results/published/ - README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap - published perf report: benchmarks/results/published/perf-ezdxf.json Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Umutcan Edizsalan
co-authored by
Cursor
Cursor
parent
67f2c3b575
commit
21cbc62df5
@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Headless performance lane** (`benchmarks/perf_suite.py` +
|
||||||
|
`render_perf_chart.py`): fixed wall-time workloads (2k individual creates,
|
||||||
|
10k build/export/reopen roundtrip, 10k region query, full premium quality
|
||||||
|
pass) measured through the same backend methods the MCP tools call;
|
||||||
|
machine fingerprint recorded, self-measurement only.
|
||||||
|
- **Task-matrix chart** (`benchmarks/render_matrix_chart.py`): tasks × servers
|
||||||
|
status heatmap rendered from the published live-run reports, so coverage
|
||||||
|
gaps are visible per capability instead of one folded score.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- README rebuilt: PyPI-first install, a four-lane benchmark section with
|
||||||
|
Python-generated visuals (scores, task matrix, performance, source review),
|
||||||
|
v1.4 feature table, updated client/config/roadmap sections.
|
||||||
|
- `render_live_chart` now skips non-task-matrix reports so multiple lanes can
|
||||||
|
share `benchmarks/results/published/`.
|
||||||
|
|
||||||
_Next (1.5): screenshot overlay + handle grounding v1, ezdxf redo (forward
|
_Next (1.5): screenshot overlay + handle grounding v1, ezdxf redo (forward
|
||||||
snapshots), COM `block_create_from_entities`, ISO 286 transition/interference
|
snapshots), COM `block_create_from_entities`, ISO 286 transition/interference
|
||||||
hole letters (delta rule), titleblock on paper-space layouts._
|
hole letters (delta rule), titleblock on paper-space layouts._
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
# AutoCAD MCP Pro
|
# AutoCAD MCP Pro
|
||||||
|
|
||||||
Production-grade AutoCAD automation for AI agents—live through COM on Windows,
|
Production-grade AutoCAD automation for AI agents — live through COM on
|
||||||
or headless through ezdxf on any platform.
|
Windows, or headless through ezdxf on any platform.
|
||||||
|
|
||||||
[](https://github.com/U-C4N/Autocad-MCP/actions/workflows/ci.yml)
|
[](https://github.com/U-C4N/Autocad-MCP/actions/workflows/ci.yml)
|
||||||
[](https://pypi.org/project/autocad-mcp-pro/)
|
[](https://pypi.org/project/autocad-mcp-pro/)
|
||||||
@@ -11,84 +11,38 @@ or headless through ezdxf on any platform.
|
|||||||
[](https://github.com/U-C4N/Autocad-MCP/stargazers)
|
[](https://github.com/U-C4N/Autocad-MCP/stargazers)
|
||||||
|
|
||||||
One typed MCP contract controls two execution engines. Build and edit drawings,
|
One typed MCP contract controls two execution engines. Build and edit drawings,
|
||||||
query exact geometry, apply engineering standards, refine quality issues inside
|
query exact geometry, apply engineering standards (ISO 128/129/286/1101),
|
||||||
transactions, and deliver hashed artifacts with validation evidence.
|
refine quality issues inside transactions, and deliver hashed artifacts with
|
||||||
|
validation evidence.
|
||||||
|
|
||||||
> **v1.4 release snapshot:** 131 tools · 6 resources · 5 prompt templates ·
|
> **v1.4 release snapshot:** 131 tools · 6 resources · 5 prompt templates ·
|
||||||
> 474 collected tests. Runtime discovery through `system_about` is authoritative.
|
> 474 collected tests. Runtime discovery through `system_about` is authoritative.
|
||||||
|
|
||||||
## Start in 60 seconds
|
## Install in 10 seconds
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://github.com/U-C4N/Autocad-MCP.git
|
pip install autocad-mcp-pro # or: uvx autocad-mcp-pro
|
||||||
cd Autocad-MCP
|
autocad-mcp # stdio MCP server, backend auto-selected
|
||||||
python -m venv .venv
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Activate the environment and start the stdio server:
|
Headless (no AutoCAD required, any OS) or live AutoCAD on Windows:
|
||||||
|
|
||||||
```powershell
|
|
||||||
# Windows
|
|
||||||
.venv\Scripts\Activate.ps1
|
|
||||||
pip install -e .
|
|
||||||
python server.py
|
|
||||||
```
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# macOS / Linux
|
AUTOCAD_MCP_BACKEND=ezdxf autocad-mcp # portable DXF engine
|
||||||
source .venv/bin/activate
|
AUTOCAD_MCP_BACKEND=com autocad-mcp # live AutoCAD (pip install "autocad-mcp-pro[com]")
|
||||||
pip install -e .
|
|
||||||
python server.py
|
|
||||||
```
|
```
|
||||||
|
|
||||||
That starts the headless-capable server with automatic backend selection. For
|
Extras: `[com]` live AutoCAD (pywin32 + Pillow), `[pdf]` PDF/screenshot
|
||||||
live AutoCAD control on Windows:
|
rendering (matplotlib), `[full]` everything. Working from a checkout is the
|
||||||
|
same commands with `pip install -e ".[full]"` and `python server.py`.
|
||||||
|
|
||||||
```powershell
|
## Benchmarks — three lanes, all reproducible
|
||||||
pip install -e ".[com]"
|
|
||||||
$env:AUTOCAD_MCP_BACKEND = "com"
|
|
||||||
python server.py
|
|
||||||
```
|
|
||||||
|
|
||||||
No AutoCAD installation is needed for the ezdxf backend:
|
Every number below is generated by a script in [`benchmarks/`](benchmarks/)
|
||||||
|
and rendered to SVG with Python (matplotlib). No hand-edited charts, no
|
||||||
|
adjectives.
|
||||||
|
|
||||||
```powershell
|
### 1 · Live-run scores — same harness, pinned competitors
|
||||||
$env:AUTOCAD_MCP_BACKEND = "ezdxf"
|
|
||||||
python server.py
|
|
||||||
```
|
|
||||||
|
|
||||||
## Evidence, not adjectives
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
The graphic is generated with Matplotlib from
|
|
||||||
[`benchmarks/source_review.json`](benchmarks/source_review.json). A fixed,
|
|
||||||
public 100-point rubric covers functional CAD breadth, correctness and delivery,
|
|
||||||
backend reach, engineering production, tests, and security.
|
|
||||||
|
|
||||||
| Project | Score | Evidence |
|
|
||||||
|---|---:|:---:|
|
|
||||||
| **[U-C4N/Autocad-MCP](https://github.com/U-C4N/Autocad-MCP)** | **93** | A |
|
|
||||||
| [varavista/autocad-mcp](https://github.com/varavista/autocad-mcp) | 84 | A |
|
|
||||||
| [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | 83 | A |
|
|
||||||
| [AnCode666/multiCAD-mcp](https://github.com/AnCode666/multiCAD-mcp) | 73 | A |
|
|
||||||
| [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | 70 | A |
|
|
||||||
| [NCO-1986/AutoCAD_mcp](https://github.com/NCO-1986/AutoCAD_mcp) | 68 | B |
|
|
||||||
|
|
||||||
This is a dated source review—not a claim that every project completed a shared
|
|
||||||
live AutoCAD run. Evidence grade **A** means source plus executable local
|
|
||||||
evidence was reviewed; **B** means source and tool surface were reviewed but the
|
|
||||||
live AutoCAD path was not fully executed.
|
|
||||||
|
|
||||||
Reproduce the chart and run the ten-task reference adapter:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
pip install -e ".[pdf]"
|
|
||||||
python -m benchmarks.render_chart
|
|
||||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --json
|
|
||||||
```
|
|
||||||
|
|
||||||
### Live-run lane — same harness, pinned competitors
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
@@ -98,13 +52,13 @@ python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --
|
|||||||
| [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | `11f7c47` | 50.0 | 5/10 | 50% |
|
| [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | `11f7c47` | 50.0 | 5/10 | 50% |
|
||||||
| [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | `95476a3` | 45.0 | 4/10 | 50% |
|
| [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | `95476a3` | 45.0 | 4/10 | 50% |
|
||||||
|
|
||||||
Same ten fixed tasks, same runner, headless ezdxf lane. Competitors are driven
|
Ten fixed tasks, one runner, headless ezdxf lane. Competitors are driven
|
||||||
**black-box over MCP stdio** using their own documented tool contracts at the
|
**black-box over MCP stdio** using their own documented tool contracts at the
|
||||||
pinned commit, and every geometry claim is verified by re-opening the exported
|
pinned commit, and every geometry claim is verified by re-opening the exported
|
||||||
DXF with ezdxf in the harness — no competitor self-reporting. `unsupported`
|
DXF with ezdxf inside the harness — no self-reporting. `unsupported` scores
|
||||||
scores zero in the fixed matrix; per-task statuses with exact reasons are
|
zero in the fixed matrix; per-task reasons are committed under
|
||||||
committed under [`benchmarks/results/published/`](benchmarks/results/published/).
|
[`benchmarks/results/published/`](benchmarks/results/published/). File-IPC
|
||||||
File-IPC (live AutoCAD) lanes run locally, not in CI.
|
(live AutoCAD) lanes run locally, not in CI.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
||||||
@@ -112,39 +66,97 @@ python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend e
|
|||||||
python -m benchmarks.render_live_chart
|
python -m benchmarks.render_live_chart
|
||||||
```
|
```
|
||||||
|
|
||||||
The complete rubric, caveats, version A/B suite, and reproduction commands live
|
### 2 · Task matrix — where the score comes from
|
||||||
in [`benchmarks/`](benchmarks/README.md).
|
|
||||||
|
|
||||||
## Why this architecture holds up
|
|
||||||
|
|
||||||
### One contract, two engines
|
The single score hides *which* capabilities exist, so the matrix shows every
|
||||||
|
task against every server: green = verified pass, amber = partial, gray = the
|
||||||
|
server documents no equivalent (planning, refinement, TABLE/MLEADER, hashed
|
||||||
|
delivery), red = attempted but failed verification.
|
||||||
|
|
||||||
The COM backend controls a live AutoCAD session on Windows through a
|
```bash
|
||||||
single-threaded STA executor. The ezdxf backend creates, edits, renders, and
|
python -m benchmarks.render_matrix_chart
|
||||||
validates DXF drawings without AutoCAD. Both return the same typed entities,
|
```
|
||||||
layers, blocks, drawing metadata, and capability format.
|
|
||||||
|
|
||||||
### Geometry agents do not have to guess
|
### 3 · Headless performance — scale evidence
|
||||||
|
|
||||||
Tools such as `point_from_snap`, `point_intersection`, and `point_tangent`
|
|
||||||
resolve exact construction coordinates from drawing entities. Handle-preserving
|
|
||||||
edit tools and smart selection keep agents attached to real geometry instead of
|
|
||||||
reconstructing state from prose.
|
|
||||||
|
|
||||||
### Quality is a loop
|
| Workload | Wall time | Throughput / result |
|
||||||
|
|---|---:|---|
|
||||||
|
| Create 2,000 lines (individual calls) | 0.64 s | ~3,100 entities/s |
|
||||||
|
| 10,000 lines: build + DXF export + reopen | 13.9 s | ~700 entities/s end-to-end |
|
||||||
|
| Region query over 10,000 entities | 3.4 s | 2,500 matched |
|
||||||
|
| Premium pass (layers + part + dims + full critique) | 0.32 s | 0 issues |
|
||||||
|
|
||||||
`drawing_preflight` checks whether an engineering request is ready to draw.
|
Workloads call the same backend methods the MCP tools call, so server-side
|
||||||
`drawing_critique` reports standards and geometry issues. `drawing_refine`
|
overhead is included. Self-measurement only — competitor servers would pay an
|
||||||
repairs the supported subset inside isolated transactions and rolls back rounds
|
extra stdio serialization cost that in-process runs do not, so no cross-server
|
||||||
that reduce quality. `drawing_finalize` combines structural validation,
|
timing claims are made. Numbers move with hardware; the machine fingerprint is
|
||||||
standards critique, a scalar score, and invalidity ratio.
|
recorded in the report.
|
||||||
|
|
||||||
### Delivery carries evidence
|
```bash
|
||||||
|
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||||||
|
python -m benchmarks.render_perf_chart
|
||||||
|
```
|
||||||
|
|
||||||
`drawing_deliver` creates the requested artifacts, computes SHA-256 hashes,
|
### 4 · Source-reviewed capability rubric (context)
|
||||||
reopens the canonical DXF, compares entity/layer/type counts and bounds, and
|
|
||||||
writes `manifest.json` plus `validation.json`. A successful tool call is not
|
|
||||||
treated as proof that the drawing survived export.
|
|
||||||
|
Six named AutoCAD MCP projects scored against a fixed public 100-point rubric
|
||||||
|
(CAD breadth, correctness and delivery, backend reach, engineering production,
|
||||||
|
tests, security), generated from
|
||||||
|
[`benchmarks/source_review.json`](benchmarks/source_review.json). This is a
|
||||||
|
dated source review — evidence grade **A** means source plus executable local
|
||||||
|
evidence was reviewed; **B** means the live AutoCAD path was not fully
|
||||||
|
executed. It is not a shared live run; lanes 1-3 above are the runtime
|
||||||
|
evidence. Full rubric, caveats and the version A/B suite:
|
||||||
|
[`benchmarks/README.md`](benchmarks/README.md).
|
||||||
|
|
||||||
|
## What ships in v1.4
|
||||||
|
|
||||||
|
| Area | Tools |
|
||||||
|
|---|---|
|
||||||
|
| Drawing lifecycle | create, open, save, export DXF/PDF, audit, purge, undo/redo |
|
||||||
|
| Geometry | lines, arcs, polylines, splines, hatches, trim/extend/fillet/chamfer, handle-preserving edits |
|
||||||
|
| Annotation | ISO 129 toleranced dimensions, **ISO 286 fits** (`fit="H7"`), TABLE, MLEADER, GD&T frames + datums (ISO 1101) |
|
||||||
|
| Engineering generators | involute gears (front + section A-A), DIN 6885 keyed bores, ISO A3 titleblock |
|
||||||
|
| **Paper space** *(new)* | `layout_list/create/set_current`, scaled `viewport_create`, `drawing_export_pdf(layout=...)` |
|
||||||
|
| **3D solids** *(new, opt-in)* | `solid_box/cylinder/extrude/revolve/boolean` on live AutoCAD (`ENABLE_3D=true`) |
|
||||||
|
| Quality loop | `drawing_preflight` → `drawing_plan` → `drawing_critique` → `drawing_refine` → `drawing_finalize` (0-100 score) |
|
||||||
|
| Delivery | `drawing_deliver`: DXF/PDF/PNG + SHA-256 manifest + reopen-parity checks |
|
||||||
|
| Discovery | **tool profiles** *(new)*: `TOOL_PROFILE=lean` (~46 tools) / `core` / `full` for clients with tight tool caps |
|
||||||
|
|
||||||
|
The tool surface evolves — ask `system_about` for the live grouped inventory
|
||||||
|
and `system_capabilities` for per-backend support modes instead of copying
|
||||||
|
lists from this README.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
A[MCP host / AI agent] --> B[FastMCP server]
|
||||||
|
B --> C[Error - audit - timing - logging middleware]
|
||||||
|
C --> D[Typed tool contract + tool profiles]
|
||||||
|
D --> E[COM backend]
|
||||||
|
D --> F[ezdxf backend]
|
||||||
|
E --> G[Live AutoCAD]
|
||||||
|
F --> H[Headless DXF]
|
||||||
|
E --> I[Engineering quality + delivery]
|
||||||
|
F --> I
|
||||||
|
```
|
||||||
|
|
||||||
|
- `server.py` — FastMCP surface, lifespan, middleware, resources, prompts.
|
||||||
|
- `backends/base.py` — shared typed contract + capability model.
|
||||||
|
- `backends/com_backend.py` — single-STA-thread COM executor with per-call
|
||||||
|
timeouts.
|
||||||
|
- `backends/ezdxf_backend.py` — `asyncio.to_thread`-wrapped DXF engine with
|
||||||
|
snapshot transactions.
|
||||||
|
- `engineering/` — standards-aware planning, generators, critique, scoring,
|
||||||
|
fits, refinement, validation, delivery.
|
||||||
|
- `security.py` — path validation and command/AutoLISP sanitization.
|
||||||
|
|
||||||
## Backend capabilities
|
## Backend capabilities
|
||||||
|
|
||||||
@@ -153,210 +165,135 @@ treated as proof that the drawing survived export.
|
|||||||
| Live AutoCAD document control | ✓ | — |
|
| Live AutoCAD document control | ✓ | — |
|
||||||
| Headless DXF creation and editing | — | ✓ |
|
| Headless DXF creation and editing | — | ✓ |
|
||||||
| Cross-platform execution | — | ✓ |
|
| Cross-platform execution | — | ✓ |
|
||||||
| Deterministic entity/query contract | ✓ | ✓ |
|
|
||||||
| Transactions and rollback | ✓ | ✓ |
|
| Transactions and rollback | ✓ | ✓ |
|
||||||
|
| Paper-space layouts + viewports | ✓ | ✓ |
|
||||||
|
| Viewport model-content rendering | ✓ | — |
|
||||||
| TABLE and MLEADER semantics | Native | Portable composite |
|
| TABLE and MLEADER semantics | Native | Portable composite |
|
||||||
|
| 3D solids (opt-in) | ✓ | — (no headless ACIS) |
|
||||||
| Screenshots | AutoCAD window | Matplotlib render |
|
| Screenshots | AutoCAD window | Matplotlib render |
|
||||||
| Raw AutoCAD commands / AutoLISP | Opt-in | — |
|
| Raw AutoCAD commands / AutoLISP | Opt-in | — |
|
||||||
|
|
||||||
Use `system_capabilities` at runtime instead of assuming backend support. The
|
`system_capabilities` returns this machine-readably (`native`, `composite`,
|
||||||
machine-readable response distinguishes `native`, `composite`, `emulated`, and
|
`rendered`, `snapshot`, `shared`, `unsupported`) — use it at runtime instead of
|
||||||
`unsupported` behavior where the engines differ.
|
assuming.
|
||||||
|
|
||||||
## A production drawing workflow
|
## A production drawing workflow
|
||||||
|
|
||||||
The server exposes low-level CAD primitives, but the intended engineering path
|
|
||||||
uses deterministic orchestration and validation:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
drawing_preflight → drawing_plan → drawing_apply_iso_layers
|
drawing_preflight → drawing_plan → drawing_apply_iso_layers
|
||||||
→ deterministic create/edit tools → dimension_auto
|
→ deterministic create/edit tools → dimension_auto (+ fit="H7" callouts)
|
||||||
→ drawing_critique → drawing_refine
|
→ drawing_critique → drawing_refine
|
||||||
→ drawing_finalize → drawing_deliver
|
→ layout_create + viewport_create → drawing_finalize → drawing_deliver
|
||||||
```
|
```
|
||||||
|
|
||||||
Representative capability groups include:
|
|
||||||
|
|
||||||
| Area | Examples |
|
|
||||||
|---|---|
|
|
||||||
| Drawing lifecycle | create, open, save, export, audit, purge, undo/redo |
|
|
||||||
| Geometry | lines, arcs, polylines, hatches, splines, trim/extend/fillet/chamfer |
|
|
||||||
| Annotation | dimensions, ISO 129 tolerances, TABLE, MLEADER, text editing |
|
|
||||||
| Engineering | gears, keyed bores, title blocks, ISO 1101 GD&T and datums |
|
|
||||||
| Query | entity/layer/block inspection, selection, bounds, distance and area |
|
|
||||||
| Quality | preflight, critique, score, transactional refinement and finalization |
|
|
||||||
| Delivery | PDF/PNG/DXF output, hashes, manifests and reopen-parity checks |
|
|
||||||
|
|
||||||
The tool surface evolves. Ask `system_about` for the live grouped inventory
|
|
||||||
instead of building integrations around a copied list from this README.
|
|
||||||
|
|
||||||
## Installation profiles
|
|
||||||
|
|
||||||
| Profile | Command | Use case |
|
|
||||||
|---|---|---|
|
|
||||||
| Core | `pip install -e .` | Headless ezdxf workflows |
|
|
||||||
| COM | `pip install -e ".[com]"` | Live AutoCAD on Windows |
|
|
||||||
| PDF | `pip install -e ".[pdf]"` | PDF export and headless screenshots |
|
|
||||||
| Full | `pip install -e ".[full]"` | Both backends and all render features |
|
|
||||||
|
|
||||||
Requirements:
|
|
||||||
|
|
||||||
- Python 3.11+
|
|
||||||
- [FastMCP](https://github.com/jlowin/fastmcp) 3.0+
|
|
||||||
- [ezdxf](https://github.com/mozman/ezdxf) 1.3+
|
|
||||||
- AutoCAD, `pywin32`, and Pillow only for live COM control
|
|
||||||
- Matplotlib only for PDF/headless rendering and benchmark visualization
|
|
||||||
|
|
||||||
## Connect an MCP client
|
## Connect an MCP client
|
||||||
|
|
||||||
### Claude Desktop
|
### Claude Desktop / Cursor / any stdio host
|
||||||
|
|
||||||
Add a server entry to `claude_desktop_config.json` and use the absolute path to
|
|
||||||
your checkout:
|
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"mcpServers": {
|
"mcpServers": {
|
||||||
"autocad": {
|
"autocad": {
|
||||||
"command": "python",
|
"command": "autocad-mcp",
|
||||||
"args": ["C:\\path\\to\\Autocad-MCP\\server.py"],
|
|
||||||
"env": {
|
"env": {
|
||||||
"AUTOCAD_MCP_BACKEND": "auto",
|
"AUTOCAD_MCP_BACKEND": "auto",
|
||||||
"ALLOWED_PATHS": "C:\\Users\\you\\Documents\\AutoCAD"
|
"ALLOWED_PATHS": "C:\\Users\\you\\Documents\\AutoCAD",
|
||||||
|
"TOOL_PROFILE": "full"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Cursor, Cline, Continue, Goose, and other stdio-capable MCP hosts use the same
|
After `pip install autocad-mcp-pro` the `autocad-mcp` command is on PATH; from
|
||||||
`command` plus `args` shape. The model vendor is not part of the server
|
a checkout use `"command": "python"`, `"args": ["path/to/server.py"]`. The
|
||||||
contract.
|
model vendor is not part of the server contract.
|
||||||
|
|
||||||
### Local HTTP
|
### Local HTTP
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
fastmcp run server.py:mcp --transport http --port 8000
|
autocad-mcp --transport http --port 8000
|
||||||
```
|
```
|
||||||
|
|
||||||
Loopback is the default. A non-loopback bind is refused unless remote HTTP is
|
Loopback only by default; a non-loopback bind is refused unless remote HTTP is
|
||||||
explicitly enabled and bearer-token authentication is configured.
|
explicitly enabled **and** a bearer token is configured.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
Set variables in your shell, process manager, or MCP client configuration.
|
|
||||||
[`.env.example`](.env.example) is a reference; the server reads process
|
|
||||||
environment variables directly.
|
|
||||||
|
|
||||||
| Variable | Default | Purpose |
|
| Variable | Default | Purpose |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `AUTOCAD_MCP_BACKEND` | `auto` | Select `auto`, `com`, or `ezdxf` |
|
| `AUTOCAD_MCP_BACKEND` | `auto` | `auto`, `com`, or `ezdxf` |
|
||||||
|
| `TOOL_PROFILE` | `full` | `lean` (~46 curated tools), `core` (hides escape hatches), `full` |
|
||||||
|
| `ENABLE_3D` | `false` | Expose the opt-in `solid_*` tools (COM backend) |
|
||||||
| `LOG_LEVEL` | `INFO` | Python logging level |
|
| `LOG_LEVEL` | `INFO` | Python logging level |
|
||||||
| `ALLOWED_PATHS` | empty | Comma-separated absolute paths the server may access |
|
| `ALLOWED_PATHS` | empty | Comma-separated absolute paths the server may access |
|
||||||
| `MAX_UNDO_STACK` | `5` | Maximum retained undo snapshots |
|
| `MAX_UNDO_STACK` | `5` | Maximum retained undo snapshots |
|
||||||
| `MAX_DXF_BYTES` | `52428800` | Reject larger DXF input; `0` disables the check |
|
| `MAX_DXF_BYTES` | `52428800` | Reject larger DXF input; `0` disables |
|
||||||
| `MAX_LIST_LIMIT` | `5000` | Bound list and selection response sizes |
|
| `MAX_LIST_LIMIT` | `5000` | Bound list/selection response sizes |
|
||||||
| `COM_CALL_TIMEOUT` | `60` | Per-call live AutoCAD timeout in seconds |
|
| `COM_CALL_TIMEOUT` | `60` | Per-call live AutoCAD timeout (seconds) |
|
||||||
| `DANGEROUS_COMMANDS_ENABLED` | `false` | Allow blocked commands/LISP and report unsafe mode |
|
| `DANGEROUS_COMMANDS_ENABLED` | `false` | Allow blocked commands/LISP; reported as unsafe mode |
|
||||||
| `ALLOW_REMOTE_HTTP` | `false` | Permit a non-loopback HTTP bind |
|
| `ALLOW_REMOTE_HTTP` | `false` | Permit a non-loopback HTTP bind |
|
||||||
| `MCP_AUTH_TOKEN` | empty | Bearer token required for remote HTTP |
|
| `MCP_AUTH_TOKEN` | empty | Bearer token required for remote HTTP |
|
||||||
|
|
||||||
Backend selection:
|
|
||||||
|
|
||||||
| Value | Behavior |
|
|
||||||
|---|---|
|
|
||||||
| `auto` | Try COM on Windows, then fall back to ezdxf |
|
|
||||||
| `com` | Require a reachable live AutoCAD instance |
|
|
||||||
| `ezdxf` | Force portable, headless DXF operation |
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart LR
|
|
||||||
A[MCP host / AI agent] --> B[FastMCP server]
|
|
||||||
B --> C[Error · audit · timing · logging middleware]
|
|
||||||
C --> D[Typed tool contract]
|
|
||||||
D --> E[COM backend]
|
|
||||||
D --> F[ezdxf backend]
|
|
||||||
E --> G[Live AutoCAD]
|
|
||||||
F --> H[Headless DXF]
|
|
||||||
E --> I[Engineering quality + delivery]
|
|
||||||
F --> I
|
|
||||||
```
|
|
||||||
|
|
||||||
- `server.py` owns the FastMCP surface, lifespan, middleware, resources, and
|
|
||||||
prompt templates.
|
|
||||||
- `backends/base.py` defines the shared contract and capability model.
|
|
||||||
- `backends/com_backend.py` serializes COM calls through one STA thread with
|
|
||||||
per-call timeouts.
|
|
||||||
- `backends/ezdxf_backend.py` wraps synchronous DXF work with
|
|
||||||
`asyncio.to_thread` and snapshot-based transactions.
|
|
||||||
- `engineering/` contains standards-aware planning, drawing, critique, scoring,
|
|
||||||
refinement, validation, and delivery.
|
|
||||||
- `security.py` centralizes path validation and command/LISP sanitization.
|
|
||||||
|
|
||||||
## Security model
|
## Security model
|
||||||
|
|
||||||
The server treats tool input as untrusted:
|
Tool input is treated as untrusted:
|
||||||
|
|
||||||
- Every file path passes through traversal and allowed-root validation.
|
- every path passes traversal + allowed-root validation;
|
||||||
- Dangerous AutoCAD commands and AutoLISP channels are blocked by default.
|
- dangerous AutoCAD commands and AutoLISP channels are blocked by default,
|
||||||
- Remote HTTP requires explicit opt-in and a bearer token.
|
with regression tests for known bypass patterns;
|
||||||
- DXF size, query-result size, undo history, and COM call duration are bounded.
|
- remote HTTP requires explicit opt-in plus a bearer token;
|
||||||
- Audit middleware records tool timing and failures.
|
- DXF size, response sizes, undo history and COM call duration are bounded;
|
||||||
- `system_status` and `system_about` expose `unsafe_mode` when dangerous
|
- audit middleware records tool timing and failures;
|
||||||
|
- `system_status` / `system_about` expose `unsafe_mode` when dangerous
|
||||||
operations are enabled.
|
operations are enabled.
|
||||||
|
|
||||||
`DANGEROUS_COMMANDS_ENABLED=true` is a trust decision, not an installation
|
Please report vulnerabilities privately via [GitHub](https://github.com/U-C4N)
|
||||||
shortcut. Keep it disabled for agents or clients you do not fully control.
|
rather than public issues.
|
||||||
|
|
||||||
Please avoid filing public issues for vulnerabilities. Contact the maintainer
|
|
||||||
through [GitHub](https://github.com/U-C4N) first.
|
|
||||||
|
|
||||||
## Development
|
## Development
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install -e ".[full]"
|
pip install -e ".[full]"
|
||||||
pip install pytest pytest-asyncio pytest-cov ruff build
|
pip install pytest pytest-asyncio pytest-cov ruff build
|
||||||
python -m pytest
|
python -m pytest # 474 tests
|
||||||
python -m ruff check .
|
python -m ruff check . && python -m ruff format --check .
|
||||||
python -m ruff format --check .
|
|
||||||
python -m build
|
python -m build
|
||||||
```
|
```
|
||||||
|
|
||||||
Focused benchmark checks:
|
CI runs the same gates on Linux (3.11/3.12), Windows (mocked-COM suite),
|
||||||
|
plus package, Docker and MCP-registry-schema jobs. Releases are tag-driven:
|
||||||
```bash
|
`git tag vX.Y.Z && git push origin vX.Y.Z` builds, publishes to PyPI and cuts
|
||||||
python -m pytest tests/test_benchmark_chart.py tests/test_benchmark_v2.py -q
|
the GitHub Release. See
|
||||||
python -m benchmarks.render_chart
|
[`docs/RELEASE-DISTRIBUTION.md`](docs/RELEASE-DISTRIBUTION.md).
|
||||||
```
|
|
||||||
|
|
||||||
Repository map:
|
Repository map:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
server.py FastMCP surface and orchestration
|
server.py FastMCP surface and orchestration
|
||||||
config.py Environment-driven settings
|
config.py Environment-driven settings
|
||||||
security.py Path, command, and AutoLISP guards
|
security.py Path, command, and AutoLISP guards
|
||||||
version.py Canonical package version
|
version.py Canonical package version
|
||||||
backends/ COM and ezdxf implementations
|
backends/ COM and ezdxf implementations
|
||||||
engineering/ Standards, critique, refinement, delivery
|
engineering/ Standards, generators, critique, fits, delivery
|
||||||
benchmarks/ Reproducible data, runners, and reports
|
benchmarks/ Runners, adapters, published reports, chart renderers
|
||||||
tests/ Headless, mocked-COM, contract, and security tests
|
tests/ Headless, mocked-COM, contract, and security tests
|
||||||
```
|
```
|
||||||
|
|
||||||
## Roadmap
|
## Roadmap (1.5)
|
||||||
|
|
||||||
The next priorities are capability-aware tool discovery, paper-space/layout
|
Screenshot overlay + handle grounding, ezdxf redo (forward snapshots), COM
|
||||||
depth, ISO 286 fits, Windows CI coverage, opt-in COM 3D solids, and real
|
`block_create_from_entities`, ISO 286 transition/interference hole letters
|
||||||
cross-project runtime adapters. Features are published when their contracts and
|
(delta rule), titleblock on paper-space layouts. Features are published when
|
||||||
limitations are testable—not when they make a longer checklist.
|
their contracts and limitations are testable — not when they make a longer
|
||||||
|
checklist.
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
|
|
||||||
Pull requests are welcome, especially for reproducible competitor adapters,
|
Pull requests are welcome — especially reproducible competitor adapters,
|
||||||
mocked/live COM coverage, engineering standards, and backend parity. For a
|
mocked/live COM coverage, engineering standards, and backend parity. For a
|
||||||
large change, open an issue first so the tool contract and evidence plan can be
|
large change, open an issue first so the tool contract and evidence plan can
|
||||||
agreed before implementation.
|
be agreed before implementation.
|
||||||
|
|
||||||
## Author
|
## Author
|
||||||
|
|
||||||
|
|||||||
+20
-2
@@ -78,13 +78,31 @@ Method and boundaries:
|
|||||||
require a local AutoCAD session and are out of CI scope by design.
|
require a local AutoCAD session and are out of CI scope by design.
|
||||||
|
|
||||||
Published machine-readable reports (artifact paths sanitized to filenames)
|
Published machine-readable reports (artifact paths sanitized to filenames)
|
||||||
live under [`results/published/`](results/published/); the README chart is
|
live under [`results/published/`](results/published/); the README charts are
|
||||||
regenerated from them:
|
regenerated from them:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
||||||
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf
|
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf
|
||||||
python -m benchmarks.render_live_chart
|
python -m benchmarks.render_live_chart # score bars
|
||||||
|
python -m benchmarks.render_matrix_chart # tasks x servers status heatmap
|
||||||
|
```
|
||||||
|
|
||||||
|
## Headless performance lane
|
||||||
|
|
||||||
|
`perf_suite.py` measures wall time on fixed workloads through the same backend
|
||||||
|
methods the MCP tools call (server-side overhead included): 2,000 individual
|
||||||
|
line creates, a 10,000-entity build → DXF export → reopen roundtrip, a region
|
||||||
|
query over 10,000 entities, and a full premium quality pass (ISO layers,
|
||||||
|
geometry, dimensions, complete critique). The report records the machine
|
||||||
|
fingerprint; numbers move with hardware, workload definitions do not.
|
||||||
|
**Self-measurement only** — competitor servers would pay an extra stdio
|
||||||
|
serialization cost that in-process runs do not, so no cross-server timing
|
||||||
|
claims are made.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||||||
|
python -m benchmarks.render_perf_chart
|
||||||
```
|
```
|
||||||
|
|
||||||
## Correctness A/B suite
|
## Correctness A/B suite
|
||||||
|
|||||||
@@ -0,0 +1,210 @@
|
|||||||
|
"""Headless performance suite (v3 lane): wall-time on real drawing workloads.
|
||||||
|
|
||||||
|
Complements the correctness/coverage lanes with *scale* evidence: how long the
|
||||||
|
ezdxf engine takes to create, roundtrip, query and quality-check real geometry.
|
||||||
|
Workloads drive the backend exactly like the MCP tools do (same async methods),
|
||||||
|
so the numbers include the server-side call overhead, not just raw ezdxf.
|
||||||
|
|
||||||
|
Honesty boundaries:
|
||||||
|
|
||||||
|
- Self-measurement only — no competitor timing claims (their servers would pay
|
||||||
|
an extra stdio serialization cost ours does not pay in-process).
|
||||||
|
- Wall-clock on the machine recorded in the report; absolute numbers move with
|
||||||
|
hardware, the workload definitions do not.
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
python -m benchmarks.perf_suite # human summary
|
||||||
|
python -m benchmarks.perf_suite --json # machine-readable report
|
||||||
|
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import platform
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from dataclasses import asdict, dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from backends.ezdxf_backend import EzdxfBackend
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class WorkloadResult:
|
||||||
|
workload_id: str
|
||||||
|
description: str
|
||||||
|
wall_ms: float
|
||||||
|
metric: str
|
||||||
|
metric_value: float
|
||||||
|
detail: dict
|
||||||
|
|
||||||
|
|
||||||
|
async def _timed(coro) -> tuple[float, object]:
|
||||||
|
started = time.perf_counter()
|
||||||
|
result = await coro
|
||||||
|
return (time.perf_counter() - started) * 1000.0, result
|
||||||
|
|
||||||
|
|
||||||
|
async def workload_create_lines_2k(backend: EzdxfBackend) -> WorkloadResult:
|
||||||
|
"""2,000 individual entity_create_line calls (per-call tool overhead included)."""
|
||||||
|
await backend.drawing_new()
|
||||||
|
started = time.perf_counter()
|
||||||
|
for index in range(2_000):
|
||||||
|
y = float(index % 100)
|
||||||
|
await backend.entity_create_line(0.0, y, 100.0, y + 1.0)
|
||||||
|
wall_ms = (time.perf_counter() - started) * 1000.0
|
||||||
|
count = (await backend.drawing_info()).entity_count
|
||||||
|
return WorkloadResult(
|
||||||
|
"create_lines_2k",
|
||||||
|
"Create 2,000 lines through individual backend calls",
|
||||||
|
round(wall_ms, 1),
|
||||||
|
"entities_per_second",
|
||||||
|
round(2_000 / (wall_ms / 1000.0), 1),
|
||||||
|
{"entity_count": count},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def workload_roundtrip_10k(backend: EzdxfBackend, scratch: Path) -> WorkloadResult:
|
||||||
|
"""Build a 10,000-entity drawing, export DXF, reopen, read info."""
|
||||||
|
await backend.drawing_new()
|
||||||
|
build_started = time.perf_counter()
|
||||||
|
for index in range(10_000):
|
||||||
|
x = float(index % 500)
|
||||||
|
y = float(index // 500)
|
||||||
|
await backend.entity_create_line(x, y, x + 0.8, y + 0.8)
|
||||||
|
build_ms = (time.perf_counter() - build_started) * 1000.0
|
||||||
|
|
||||||
|
target = scratch / "perf_roundtrip_10k.dxf"
|
||||||
|
export_ms, _ = await _timed(backend.drawing_export_dxf(str(target)))
|
||||||
|
|
||||||
|
reopened = EzdxfBackend()
|
||||||
|
await reopened.connect()
|
||||||
|
try:
|
||||||
|
reopen_ms, _ = await _timed(reopened.drawing_open(str(target)))
|
||||||
|
count = (await reopened.drawing_info()).entity_count
|
||||||
|
finally:
|
||||||
|
await reopened.disconnect()
|
||||||
|
|
||||||
|
total = build_ms + export_ms + reopen_ms
|
||||||
|
return WorkloadResult(
|
||||||
|
"roundtrip_10k",
|
||||||
|
"Create 10,000 lines, export DXF, reopen and verify",
|
||||||
|
round(total, 1),
|
||||||
|
"entities_per_second",
|
||||||
|
round(10_000 / (total / 1000.0), 1),
|
||||||
|
{
|
||||||
|
"build_ms": round(build_ms, 1),
|
||||||
|
"export_ms": round(export_ms, 1),
|
||||||
|
"reopen_ms": round(reopen_ms, 1),
|
||||||
|
"file_bytes": target.stat().st_size,
|
||||||
|
"entity_count": count,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def workload_region_query_10k(backend: EzdxfBackend) -> WorkloadResult:
|
||||||
|
"""Rectangular region query against the 10,000-entity drawing."""
|
||||||
|
# backend still holds the 10k drawing from the roundtrip workload builder;
|
||||||
|
# rebuild deterministically to stay order-independent.
|
||||||
|
await backend.drawing_new()
|
||||||
|
for index in range(10_000):
|
||||||
|
x = float(index % 500)
|
||||||
|
y = float(index // 500)
|
||||||
|
await backend.entity_create_line(x, y, x + 0.8, y + 0.8)
|
||||||
|
wall_ms, found = await _timed(backend.analysis_entities_in_region(0.0, 0.0, 250.0, 10.0))
|
||||||
|
return WorkloadResult(
|
||||||
|
"region_query_10k",
|
||||||
|
"Crossing-selection region query over 10,000 entities",
|
||||||
|
round(wall_ms, 1),
|
||||||
|
"entities_matched",
|
||||||
|
float(len(found)),
|
||||||
|
{"entity_count": 10_000},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def workload_premium_pass(backend: EzdxfBackend) -> WorkloadResult:
|
||||||
|
"""Small end-to-end quality loop: layers, geometry, dimensions, critique."""
|
||||||
|
started = time.perf_counter()
|
||||||
|
await backend.drawing_new()
|
||||||
|
await backend.drawing_apply_iso_layers("mech")
|
||||||
|
plate = [
|
||||||
|
(0.0, 0.0, 120.0, 0.0),
|
||||||
|
(120.0, 0.0, 120.0, 80.0),
|
||||||
|
(120.0, 80.0, 0.0, 80.0),
|
||||||
|
(0.0, 80.0, 0.0, 0.0),
|
||||||
|
]
|
||||||
|
for x1, y1, x2, y2 in plate:
|
||||||
|
await backend.entity_create_line(x1, y1, x2, y2, layer="GEOMETRY")
|
||||||
|
await backend.entity_create_circle(60.0, 40.0, 12.0, layer="GEOMETRY")
|
||||||
|
await backend.dimension_linear(0, 0, 120, 0, 60, -15, layer="DIM")
|
||||||
|
await backend.dimension_linear(0, 0, 0, 80, -15, 40, 90, layer="DIM")
|
||||||
|
issues = await backend.drawing_critique(None)
|
||||||
|
wall_ms = (time.perf_counter() - started) * 1000.0
|
||||||
|
return WorkloadResult(
|
||||||
|
"premium_pass",
|
||||||
|
"Plate with hole: ISO layers, geometry, dimensions and full critique",
|
||||||
|
round(wall_ms, 1),
|
||||||
|
"critique_issues",
|
||||||
|
float(len(issues)),
|
||||||
|
{"entities": (await backend.drawing_info()).entity_count},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def run_suite(scratch: Path) -> dict:
|
||||||
|
backend = EzdxfBackend()
|
||||||
|
await backend.connect()
|
||||||
|
try:
|
||||||
|
results = [
|
||||||
|
await workload_create_lines_2k(backend),
|
||||||
|
await workload_roundtrip_10k(backend, scratch),
|
||||||
|
await workload_region_query_10k(backend),
|
||||||
|
await workload_premium_pass(backend),
|
||||||
|
]
|
||||||
|
finally:
|
||||||
|
await backend.disconnect()
|
||||||
|
return {
|
||||||
|
"schema_version": "perf/1.0",
|
||||||
|
"backend": "ezdxf",
|
||||||
|
"environment": {
|
||||||
|
"python": sys.version.split()[0],
|
||||||
|
"implementation": platform.python_implementation(),
|
||||||
|
"platform": platform.platform(),
|
||||||
|
"machine": platform.machine(),
|
||||||
|
},
|
||||||
|
"workloads": [asdict(item) for item in results],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON")
|
||||||
|
parser.add_argument("--out", type=Path, default=None, help="Also write the report to a file")
|
||||||
|
parser.add_argument("--scratch", type=Path, default=Path("benchmarks/results/perf-scratch"))
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = build_parser().parse_args()
|
||||||
|
args.scratch.mkdir(parents=True, exist_ok=True)
|
||||||
|
report = asyncio.run(run_suite(args.scratch))
|
||||||
|
rendered = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)
|
||||||
|
if args.out is not None:
|
||||||
|
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
args.out.write_text(rendered + "\n", encoding="utf-8")
|
||||||
|
if args.json:
|
||||||
|
print(rendered)
|
||||||
|
else:
|
||||||
|
for item in report["workloads"]:
|
||||||
|
print(
|
||||||
|
f"{item['workload_id']:>18} {item['wall_ms']:>10.1f} ms "
|
||||||
|
f"{item['metric']}={item['metric_value']}"
|
||||||
|
)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -23,7 +23,7 @@ def load_reports(input_dir: Path) -> list[dict[str, Any]]:
|
|||||||
for path in sorted(input_dir.glob("*.json")):
|
for path in sorted(input_dir.glob("*.json")):
|
||||||
data = json.loads(path.read_text(encoding="utf-8"))
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
if data.get("schema_version") != "2.0":
|
if data.get("schema_version") != "2.0":
|
||||||
raise ValueError(f"{path.name}: unsupported schema_version")
|
continue # other lanes (e.g. perf/1.0) share the published folder
|
||||||
summary = data.get("summary") or {}
|
summary = data.get("summary") or {}
|
||||||
if "score" not in summary or "coverage_percent" not in summary:
|
if "score" not in summary or "coverage_percent" not in summary:
|
||||||
raise ValueError(f"{path.name}: summary is missing score/coverage")
|
raise ValueError(f"{path.name}: summary is missing score/coverage")
|
||||||
|
|||||||
@@ -0,0 +1,174 @@
|
|||||||
|
"""Render the live-run task matrix (tasks x servers) as a deterministic SVG.
|
||||||
|
|
||||||
|
Input: the published per-server reports under ``benchmarks/results/published/``
|
||||||
|
(the same files the score chart uses). Output: a status heatmap — every task
|
||||||
|
against every server — so coverage gaps are visible per capability instead of
|
||||||
|
being folded into one number.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from benchmarks.tasks_v2 import TASKS_V2
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
|
||||||
|
DEFAULT_OUTPUT = ROOT / "docs" / "assets" / "autocad-mcp-taskmatrix.svg"
|
||||||
|
|
||||||
|
STATUS_COLORS = {
|
||||||
|
"pass": "#2da44e",
|
||||||
|
"partial": "#d4a72c",
|
||||||
|
"fail": "#cf222e",
|
||||||
|
"timeout": "#a40e26",
|
||||||
|
"unsupported": "#d0d7de",
|
||||||
|
"not_run": "#f6f8fa",
|
||||||
|
}
|
||||||
|
|
||||||
|
STATUS_GLYPHS = {
|
||||||
|
"pass": "P",
|
||||||
|
"partial": "~",
|
||||||
|
"fail": "F",
|
||||||
|
"timeout": "T",
|
||||||
|
"unsupported": "-",
|
||||||
|
"not_run": "?",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_reports(input_dir: Path) -> list[dict[str, Any]]:
|
||||||
|
reports: list[dict[str, Any]] = []
|
||||||
|
for path in sorted(input_dir.glob("*.json")):
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
if data.get("schema_version") != "2.0":
|
||||||
|
continue # perf reports and other lanes live in the same folder
|
||||||
|
reports.append(data)
|
||||||
|
if not reports:
|
||||||
|
raise SystemExit(f"no task-matrix reports found in {input_dir}")
|
||||||
|
# our server first, then alphabetical
|
||||||
|
reports.sort(key=lambda item: (item["adapter"] != "autocad-mcp-pro", item["adapter"]))
|
||||||
|
return reports
|
||||||
|
|
||||||
|
|
||||||
|
def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||||
|
try:
|
||||||
|
import matplotlib
|
||||||
|
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
from matplotlib.patches import Patch, Rectangle
|
||||||
|
except ImportError as exc:
|
||||||
|
raise SystemExit(
|
||||||
|
'Matplotlib is required. Install it with: pip install -e ".[pdf]"'
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
|
||||||
|
|
||||||
|
task_ids = [task.task_id for task in TASKS_V2]
|
||||||
|
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V2]
|
||||||
|
servers = [report["adapter"] for report in reports]
|
||||||
|
status_by_server = {
|
||||||
|
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
|
||||||
|
for report in reports
|
||||||
|
}
|
||||||
|
|
||||||
|
fig, ax = plt.subplots(
|
||||||
|
figsize=(3.4 + 2.4 * len(servers), 1.8 + 0.52 * len(task_ids)),
|
||||||
|
constrained_layout=True,
|
||||||
|
)
|
||||||
|
fig.patch.set_facecolor("#f6f8fa")
|
||||||
|
ax.set_facecolor("#f6f8fa")
|
||||||
|
|
||||||
|
for row, task_id in enumerate(task_ids):
|
||||||
|
for col, server in enumerate(servers):
|
||||||
|
status = status_by_server[server].get(task_id, "not_run")
|
||||||
|
color = STATUS_COLORS.get(status, "#f6f8fa")
|
||||||
|
ax.add_patch(
|
||||||
|
Rectangle(
|
||||||
|
(col + 0.06, row + 0.08),
|
||||||
|
0.88,
|
||||||
|
0.84,
|
||||||
|
facecolor=color,
|
||||||
|
edgecolor="#d0d7de",
|
||||||
|
linewidth=0.8,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
glyph_color = "#ffffff" if status in ("pass", "fail", "timeout") else "#24292f"
|
||||||
|
ax.text(
|
||||||
|
col + 0.5,
|
||||||
|
row + 0.5,
|
||||||
|
STATUS_GLYPHS.get(status, "?"),
|
||||||
|
ha="center",
|
||||||
|
va="center",
|
||||||
|
fontsize=11,
|
||||||
|
fontweight="bold",
|
||||||
|
color=glyph_color,
|
||||||
|
)
|
||||||
|
|
||||||
|
ax.set_xlim(0, len(servers))
|
||||||
|
ax.set_ylim(len(task_ids), 0)
|
||||||
|
ax.set_xticks([index + 0.5 for index in range(len(servers))])
|
||||||
|
ax.set_xticklabels(servers, color="#24292f", fontsize=10, fontweight="bold")
|
||||||
|
ax.xaxis.set_ticks_position("top")
|
||||||
|
ax.set_yticks([index + 0.5 for index in range(len(task_ids))])
|
||||||
|
ax.set_yticklabels(task_labels, color="#24292f", fontsize=9)
|
||||||
|
ax.tick_params(length=0)
|
||||||
|
for spine in ax.spines.values():
|
||||||
|
spine.set_visible(False)
|
||||||
|
|
||||||
|
ax.set_title(
|
||||||
|
"Task matrix — live headless lane",
|
||||||
|
loc="left",
|
||||||
|
fontsize=17,
|
||||||
|
fontweight="bold",
|
||||||
|
color="#24292f",
|
||||||
|
pad=34,
|
||||||
|
)
|
||||||
|
|
||||||
|
legend_handles = [
|
||||||
|
Patch(facecolor=STATUS_COLORS["pass"], label="pass"),
|
||||||
|
Patch(facecolor=STATUS_COLORS["partial"], label="partial"),
|
||||||
|
Patch(facecolor=STATUS_COLORS["fail"], label="fail"),
|
||||||
|
Patch(facecolor=STATUS_COLORS["unsupported"], label="unsupported (no equivalent)"),
|
||||||
|
]
|
||||||
|
ax.legend(
|
||||||
|
handles=legend_handles,
|
||||||
|
loc="upper left",
|
||||||
|
bbox_to_anchor=(0, -0.02),
|
||||||
|
ncol=4,
|
||||||
|
frameon=False,
|
||||||
|
fontsize=9,
|
||||||
|
)
|
||||||
|
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
fig.savefig(
|
||||||
|
output,
|
||||||
|
format="svg",
|
||||||
|
facecolor=fig.get_facecolor(),
|
||||||
|
metadata={"Creator": "AutoCAD MCP Pro task-matrix renderer", "Date": None},
|
||||||
|
)
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
svg = output.read_text(encoding="utf-8")
|
||||||
|
normalized = "\n".join(line.rstrip() for line in svg.splitlines()) + "\n"
|
||||||
|
output.write_text(normalized, encoding="utf-8", newline="\n")
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--input-dir", type=Path, default=DEFAULT_INPUT_DIR)
|
||||||
|
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = build_parser().parse_args()
|
||||||
|
render_chart(load_reports(args.input_dir), args.output)
|
||||||
|
print(f"Rendered task-matrix chart: {args.output}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,146 @@
|
|||||||
|
"""Render the headless performance suite as a deterministic SVG.
|
||||||
|
|
||||||
|
Input: the report produced by ``benchmarks.perf_suite --out ...``
|
||||||
|
(default: ``benchmarks/results/published/perf-ezdxf.json``).
|
||||||
|
Output: a GitHub-readable log-scale wall-time chart per workload.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
DEFAULT_INPUT = ROOT / "benchmarks" / "results" / "published" / "perf-ezdxf.json"
|
||||||
|
DEFAULT_OUTPUT = ROOT / "docs" / "assets" / "autocad-mcp-perf.svg"
|
||||||
|
|
||||||
|
_LABELS = {
|
||||||
|
"create_lines_2k": "Create 2,000 lines (individual calls)",
|
||||||
|
"roundtrip_10k": "10,000 lines: build + DXF export + reopen",
|
||||||
|
"region_query_10k": "Region query over 10,000 entities",
|
||||||
|
"premium_pass": "Premium pass: layers + part + dims + full critique",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_report(path: Path) -> dict[str, Any]:
|
||||||
|
data = json.loads(path.read_text(encoding="utf-8"))
|
||||||
|
if data.get("schema_version") != "perf/1.0":
|
||||||
|
raise ValueError("unsupported perf report schema_version")
|
||||||
|
if not data.get("workloads"):
|
||||||
|
raise ValueError("perf report contains no workloads")
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
def _annotation(item: dict[str, Any]) -> str:
|
||||||
|
wall_s = item["wall_ms"] / 1000.0
|
||||||
|
if item["metric"] == "entities_per_second":
|
||||||
|
return f"{wall_s:.2f} s · {item['metric_value']:,.0f} entities/s"
|
||||||
|
if item["metric"] == "entities_matched":
|
||||||
|
return f"{wall_s:.2f} s · {item['metric_value']:,.0f} matched"
|
||||||
|
if item["metric"] == "critique_issues":
|
||||||
|
return f"{wall_s:.2f} s · {item['metric_value']:.0f} issues"
|
||||||
|
return f"{wall_s:.2f} s"
|
||||||
|
|
||||||
|
|
||||||
|
def render_chart(data: dict[str, Any], output: Path) -> None:
|
||||||
|
try:
|
||||||
|
import matplotlib
|
||||||
|
|
||||||
|
matplotlib.use("Agg")
|
||||||
|
import matplotlib.pyplot as plt
|
||||||
|
except ImportError as exc:
|
||||||
|
raise SystemExit(
|
||||||
|
'Matplotlib is required. Install it with: pip install -e ".[pdf]"'
|
||||||
|
) from exc
|
||||||
|
|
||||||
|
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-perf-v1"
|
||||||
|
workloads = sorted(data["workloads"], key=lambda item: item["wall_ms"])
|
||||||
|
labels = [_LABELS.get(item["workload_id"], item["workload_id"]) for item in workloads]
|
||||||
|
values = [item["wall_ms"] for item in workloads]
|
||||||
|
|
||||||
|
fig, ax = plt.subplots(figsize=(12, 4.6), constrained_layout=True)
|
||||||
|
fig.patch.set_facecolor("#f6f8fa")
|
||||||
|
ax.set_facecolor("#f6f8fa")
|
||||||
|
bars = ax.barh(labels, values, color="#0969da", height=0.58)
|
||||||
|
|
||||||
|
ax.set_xscale("log")
|
||||||
|
ax.set_xlabel("Wall time in ms (log scale) — lower is better", color="#24292f", labelpad=10)
|
||||||
|
ax.set_title(
|
||||||
|
"Headless performance — ezdxf engine",
|
||||||
|
loc="left",
|
||||||
|
fontsize=18,
|
||||||
|
fontweight="bold",
|
||||||
|
color="#24292f",
|
||||||
|
pad=28,
|
||||||
|
)
|
||||||
|
env = data.get("environment", {})
|
||||||
|
ax.text(
|
||||||
|
0,
|
||||||
|
1.02,
|
||||||
|
f"Fixed workloads via the same backend methods the MCP tools call · "
|
||||||
|
f"Python {env.get('python', '?')} · {env.get('platform', '?')}",
|
||||||
|
transform=ax.transAxes,
|
||||||
|
color="#57606a",
|
||||||
|
fontsize=9,
|
||||||
|
)
|
||||||
|
|
||||||
|
for bar, item in zip(bars, workloads, strict=True):
|
||||||
|
ax.text(
|
||||||
|
bar.get_width() * 1.12,
|
||||||
|
bar.get_y() + bar.get_height() / 2,
|
||||||
|
_annotation(item),
|
||||||
|
va="center",
|
||||||
|
color="#24292f",
|
||||||
|
fontweight="bold",
|
||||||
|
fontsize=9,
|
||||||
|
)
|
||||||
|
|
||||||
|
ax.set_xlim(right=max(values) * 8)
|
||||||
|
ax.text(
|
||||||
|
0,
|
||||||
|
-0.24,
|
||||||
|
"Self-measurement only: competitor servers would pay extra stdio serialization cost in-process runs do not. "
|
||||||
|
"Reproduce: python -m benchmarks.perf_suite",
|
||||||
|
transform=ax.transAxes,
|
||||||
|
color="#57606a",
|
||||||
|
fontsize=9,
|
||||||
|
)
|
||||||
|
ax.grid(axis="x", color="#d0d7de", linewidth=0.8, alpha=0.8)
|
||||||
|
ax.set_axisbelow(True)
|
||||||
|
ax.tick_params(axis="x", colors="#57606a")
|
||||||
|
ax.tick_params(axis="y", colors="#24292f", length=0, pad=8)
|
||||||
|
for spine in ax.spines.values():
|
||||||
|
spine.set_visible(False)
|
||||||
|
|
||||||
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
fig.savefig(
|
||||||
|
output,
|
||||||
|
format="svg",
|
||||||
|
facecolor=fig.get_facecolor(),
|
||||||
|
metadata={"Creator": "AutoCAD MCP Pro perf renderer", "Date": None},
|
||||||
|
)
|
||||||
|
plt.close(fig)
|
||||||
|
|
||||||
|
svg = output.read_text(encoding="utf-8")
|
||||||
|
normalized = "\n".join(line.rstrip() for line in svg.splitlines()) + "\n"
|
||||||
|
output.write_text(normalized, encoding="utf-8", newline="\n")
|
||||||
|
|
||||||
|
|
||||||
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--input", type=Path, default=DEFAULT_INPUT)
|
||||||
|
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = build_parser().parse_args()
|
||||||
|
render_chart(load_report(args.input), args.output)
|
||||||
|
print(f"Rendered perf chart: {args.output}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
{
|
||||||
|
"backend": "ezdxf",
|
||||||
|
"environment": {
|
||||||
|
"implementation": "CPython",
|
||||||
|
"machine": "AMD64",
|
||||||
|
"platform": "Windows-11-10.0.26200-SP0",
|
||||||
|
"python": "3.14.0"
|
||||||
|
},
|
||||||
|
"schema_version": "perf/1.0",
|
||||||
|
"workloads": [
|
||||||
|
{
|
||||||
|
"description": "Create 2,000 lines through individual backend calls",
|
||||||
|
"detail": {
|
||||||
|
"entity_count": 2000
|
||||||
|
},
|
||||||
|
"metric": "entities_per_second",
|
||||||
|
"metric_value": 3148.0,
|
||||||
|
"wall_ms": 635.3,
|
||||||
|
"workload_id": "create_lines_2k"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"description": "Create 10,000 lines, export DXF, reopen and verify",
|
||||||
|
"detail": {
|
||||||
|
"build_ms": 10884.9,
|
||||||
|
"entity_count": 10000,
|
||||||
|
"export_ms": 1439.2,
|
||||||
|
"file_bytes": 1369662,
|
||||||
|
"reopen_ms": 1607.5
|
||||||
|
},
|
||||||
|
"metric": "entities_per_second",
|
||||||
|
"metric_value": 717.8,
|
||||||
|
"wall_ms": 13931.6,
|
||||||
|
"workload_id": "roundtrip_10k"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"description": "Crossing-selection region query over 10,000 entities",
|
||||||
|
"detail": {
|
||||||
|
"entity_count": 10000
|
||||||
|
},
|
||||||
|
"metric": "entities_matched",
|
||||||
|
"metric_value": 2500.0,
|
||||||
|
"wall_ms": 3434.2,
|
||||||
|
"workload_id": "region_query_10k"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"description": "Plate with hole: ISO layers, geometry, dimensions and full critique",
|
||||||
|
"detail": {
|
||||||
|
"entities": 7
|
||||||
|
},
|
||||||
|
"metric": "critique_issues",
|
||||||
|
"metric_value": 0.0,
|
||||||
|
"wall_ms": 315.9,
|
||||||
|
"workload_id": "premium_pass"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 87 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 69 KiB |
Reference in New Issue
Block a user