feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only - benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time) - benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports) - render_live_chart now tolerates other lanes sharing results/published/ - README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap - published perf report: benchmarks/results/published/perf-ezdxf.json Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Umutcan Edizsalan
co-authored by
Cursor
Cursor
parent
67f2c3b575
commit
21cbc62df5
@@ -7,6 +7,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- **Headless performance lane** (`benchmarks/perf_suite.py` +
|
||||
`render_perf_chart.py`): fixed wall-time workloads (2k individual creates,
|
||||
10k build/export/reopen roundtrip, 10k region query, full premium quality
|
||||
pass) measured through the same backend methods the MCP tools call;
|
||||
machine fingerprint recorded, self-measurement only.
|
||||
- **Task-matrix chart** (`benchmarks/render_matrix_chart.py`): tasks × servers
|
||||
status heatmap rendered from the published live-run reports, so coverage
|
||||
gaps are visible per capability instead of one folded score.
|
||||
|
||||
### Changed
|
||||
|
||||
- README rebuilt: PyPI-first install, a four-lane benchmark section with
|
||||
Python-generated visuals (scores, task matrix, performance, source review),
|
||||
v1.4 feature table, updated client/config/roadmap sections.
|
||||
- `render_live_chart` now skips non-task-matrix reports so multiple lanes can
|
||||
share `benchmarks/results/published/`.
|
||||
|
||||
_Next (1.5): screenshot overlay + handle grounding v1, ezdxf redo (forward
|
||||
snapshots), COM `block_create_from_entities`, ISO 286 transition/interference
|
||||
hole letters (delta rule), titleblock on paper-space layouts._
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# AutoCAD MCP Pro
|
||||
|
||||
Production-grade AutoCAD automation for AI agents—live through COM on Windows,
|
||||
or headless through ezdxf on any platform.
|
||||
Production-grade AutoCAD automation for AI agents — live through COM on
|
||||
Windows, or headless through ezdxf on any platform.
|
||||
|
||||
[](https://github.com/U-C4N/Autocad-MCP/actions/workflows/ci.yml)
|
||||
[](https://pypi.org/project/autocad-mcp-pro/)
|
||||
@@ -11,84 +11,38 @@ or headless through ezdxf on any platform.
|
||||
[](https://github.com/U-C4N/Autocad-MCP/stargazers)
|
||||
|
||||
One typed MCP contract controls two execution engines. Build and edit drawings,
|
||||
query exact geometry, apply engineering standards, refine quality issues inside
|
||||
transactions, and deliver hashed artifacts with validation evidence.
|
||||
query exact geometry, apply engineering standards (ISO 128/129/286/1101),
|
||||
refine quality issues inside transactions, and deliver hashed artifacts with
|
||||
validation evidence.
|
||||
|
||||
> **v1.4 release snapshot:** 131 tools · 6 resources · 5 prompt templates ·
|
||||
> 474 collected tests. Runtime discovery through `system_about` is authoritative.
|
||||
|
||||
## Start in 60 seconds
|
||||
## Install in 10 seconds
|
||||
|
||||
```bash
|
||||
git clone https://github.com/U-C4N/Autocad-MCP.git
|
||||
cd Autocad-MCP
|
||||
python -m venv .venv
|
||||
pip install autocad-mcp-pro # or: uvx autocad-mcp-pro
|
||||
autocad-mcp # stdio MCP server, backend auto-selected
|
||||
```
|
||||
|
||||
Activate the environment and start the stdio server:
|
||||
|
||||
```powershell
|
||||
# Windows
|
||||
.venv\Scripts\Activate.ps1
|
||||
pip install -e .
|
||||
python server.py
|
||||
```
|
||||
Headless (no AutoCAD required, any OS) or live AutoCAD on Windows:
|
||||
|
||||
```bash
|
||||
# macOS / Linux
|
||||
source .venv/bin/activate
|
||||
pip install -e .
|
||||
python server.py
|
||||
AUTOCAD_MCP_BACKEND=ezdxf autocad-mcp # portable DXF engine
|
||||
AUTOCAD_MCP_BACKEND=com autocad-mcp # live AutoCAD (pip install "autocad-mcp-pro[com]")
|
||||
```
|
||||
|
||||
That starts the headless-capable server with automatic backend selection. For
|
||||
live AutoCAD control on Windows:
|
||||
Extras: `[com]` live AutoCAD (pywin32 + Pillow), `[pdf]` PDF/screenshot
|
||||
rendering (matplotlib), `[full]` everything. Working from a checkout is the
|
||||
same commands with `pip install -e ".[full]"` and `python server.py`.
|
||||
|
||||
```powershell
|
||||
pip install -e ".[com]"
|
||||
$env:AUTOCAD_MCP_BACKEND = "com"
|
||||
python server.py
|
||||
```
|
||||
## Benchmarks — three lanes, all reproducible
|
||||
|
||||
No AutoCAD installation is needed for the ezdxf backend:
|
||||
Every number below is generated by a script in [`benchmarks/`](benchmarks/)
|
||||
and rendered to SVG with Python (matplotlib). No hand-edited charts, no
|
||||
adjectives.
|
||||
|
||||
```powershell
|
||||
$env:AUTOCAD_MCP_BACKEND = "ezdxf"
|
||||
python server.py
|
||||
```
|
||||
|
||||
## Evidence, not adjectives
|
||||
|
||||
|
||||
|
||||
The graphic is generated with Matplotlib from
|
||||
[`benchmarks/source_review.json`](benchmarks/source_review.json). A fixed,
|
||||
public 100-point rubric covers functional CAD breadth, correctness and delivery,
|
||||
backend reach, engineering production, tests, and security.
|
||||
|
||||
| Project | Score | Evidence |
|
||||
|---|---:|:---:|
|
||||
| **[U-C4N/Autocad-MCP](https://github.com/U-C4N/Autocad-MCP)** | **93** | A |
|
||||
| [varavista/autocad-mcp](https://github.com/varavista/autocad-mcp) | 84 | A |
|
||||
| [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | 83 | A |
|
||||
| [AnCode666/multiCAD-mcp](https://github.com/AnCode666/multiCAD-mcp) | 73 | A |
|
||||
| [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | 70 | A |
|
||||
| [NCO-1986/AutoCAD_mcp](https://github.com/NCO-1986/AutoCAD_mcp) | 68 | B |
|
||||
|
||||
This is a dated source review—not a claim that every project completed a shared
|
||||
live AutoCAD run. Evidence grade **A** means source plus executable local
|
||||
evidence was reviewed; **B** means source and tool surface were reviewed but the
|
||||
live AutoCAD path was not fully executed.
|
||||
|
||||
Reproduce the chart and run the ten-task reference adapter:
|
||||
|
||||
```bash
|
||||
pip install -e ".[pdf]"
|
||||
python -m benchmarks.render_chart
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --json
|
||||
```
|
||||
|
||||
### Live-run lane — same harness, pinned competitors
|
||||
### 1 · Live-run scores — same harness, pinned competitors
|
||||
|
||||
|
||||
|
||||
@@ -98,13 +52,13 @@ python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --
|
||||
| [beiming183-cloud/AutoCAD-MCP](https://github.com/beiming183-cloud/AutoCAD-MCP) | `11f7c47` | 50.0 | 5/10 | 50% |
|
||||
| [puran-water/autocad-mcp](https://github.com/puran-water/autocad-mcp) | `95476a3` | 45.0 | 4/10 | 50% |
|
||||
|
||||
Same ten fixed tasks, same runner, headless ezdxf lane. Competitors are driven
|
||||
Ten fixed tasks, one runner, headless ezdxf lane. Competitors are driven
|
||||
**black-box over MCP stdio** using their own documented tool contracts at the
|
||||
pinned commit, and every geometry claim is verified by re-opening the exported
|
||||
DXF with ezdxf in the harness — no competitor self-reporting. `unsupported`
|
||||
scores zero in the fixed matrix; per-task statuses with exact reasons are
|
||||
committed under [`benchmarks/results/published/`](benchmarks/results/published/).
|
||||
File-IPC (live AutoCAD) lanes run locally, not in CI.
|
||||
DXF with ezdxf inside the harness — no self-reporting. `unsupported` scores
|
||||
zero in the fixed matrix; per-task reasons are committed under
|
||||
[`benchmarks/results/published/`](benchmarks/results/published/). File-IPC
|
||||
(live AutoCAD) lanes run locally, not in CI.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
||||
@@ -112,39 +66,97 @@ python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend e
|
||||
python -m benchmarks.render_live_chart
|
||||
```
|
||||
|
||||
The complete rubric, caveats, version A/B suite, and reproduction commands live
|
||||
in [`benchmarks/`](benchmarks/README.md).
|
||||
### 2 · Task matrix — where the score comes from
|
||||
|
||||
## Why this architecture holds up
|
||||
|
||||
|
||||
### One contract, two engines
|
||||
The single score hides *which* capabilities exist, so the matrix shows every
|
||||
task against every server: green = verified pass, amber = partial, gray = the
|
||||
server documents no equivalent (planning, refinement, TABLE/MLEADER, hashed
|
||||
delivery), red = attempted but failed verification.
|
||||
|
||||
The COM backend controls a live AutoCAD session on Windows through a
|
||||
single-threaded STA executor. The ezdxf backend creates, edits, renders, and
|
||||
validates DXF drawings without AutoCAD. Both return the same typed entities,
|
||||
layers, blocks, drawing metadata, and capability format.
|
||||
```bash
|
||||
python -m benchmarks.render_matrix_chart
|
||||
```
|
||||
|
||||
### Geometry agents do not have to guess
|
||||
### 3 · Headless performance — scale evidence
|
||||
|
||||
Tools such as `point_from_snap`, `point_intersection`, and `point_tangent`
|
||||
resolve exact construction coordinates from drawing entities. Handle-preserving
|
||||
edit tools and smart selection keep agents attached to real geometry instead of
|
||||
reconstructing state from prose.
|
||||
|
||||
|
||||
### Quality is a loop
|
||||
| Workload | Wall time | Throughput / result |
|
||||
|---|---:|---|
|
||||
| Create 2,000 lines (individual calls) | 0.64 s | ~3,100 entities/s |
|
||||
| 10,000 lines: build + DXF export + reopen | 13.9 s | ~700 entities/s end-to-end |
|
||||
| Region query over 10,000 entities | 3.4 s | 2,500 matched |
|
||||
| Premium pass (layers + part + dims + full critique) | 0.32 s | 0 issues |
|
||||
|
||||
`drawing_preflight` checks whether an engineering request is ready to draw.
|
||||
`drawing_critique` reports standards and geometry issues. `drawing_refine`
|
||||
repairs the supported subset inside isolated transactions and rolls back rounds
|
||||
that reduce quality. `drawing_finalize` combines structural validation,
|
||||
standards critique, a scalar score, and invalidity ratio.
|
||||
Workloads call the same backend methods the MCP tools call, so server-side
|
||||
overhead is included. Self-measurement only — competitor servers would pay an
|
||||
extra stdio serialization cost that in-process runs do not, so no cross-server
|
||||
timing claims are made. Numbers move with hardware; the machine fingerprint is
|
||||
recorded in the report.
|
||||
|
||||
### Delivery carries evidence
|
||||
```bash
|
||||
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||||
python -m benchmarks.render_perf_chart
|
||||
```
|
||||
|
||||
`drawing_deliver` creates the requested artifacts, computes SHA-256 hashes,
|
||||
reopens the canonical DXF, compares entity/layer/type counts and bounds, and
|
||||
writes `manifest.json` plus `validation.json`. A successful tool call is not
|
||||
treated as proof that the drawing survived export.
|
||||
### 4 · Source-reviewed capability rubric (context)
|
||||
|
||||
|
||||
|
||||
Six named AutoCAD MCP projects scored against a fixed public 100-point rubric
|
||||
(CAD breadth, correctness and delivery, backend reach, engineering production,
|
||||
tests, security), generated from
|
||||
[`benchmarks/source_review.json`](benchmarks/source_review.json). This is a
|
||||
dated source review — evidence grade **A** means source plus executable local
|
||||
evidence was reviewed; **B** means the live AutoCAD path was not fully
|
||||
executed. It is not a shared live run; lanes 1-3 above are the runtime
|
||||
evidence. Full rubric, caveats and the version A/B suite:
|
||||
[`benchmarks/README.md`](benchmarks/README.md).
|
||||
|
||||
## What ships in v1.4
|
||||
|
||||
| Area | Tools |
|
||||
|---|---|
|
||||
| Drawing lifecycle | create, open, save, export DXF/PDF, audit, purge, undo/redo |
|
||||
| Geometry | lines, arcs, polylines, splines, hatches, trim/extend/fillet/chamfer, handle-preserving edits |
|
||||
| Annotation | ISO 129 toleranced dimensions, **ISO 286 fits** (`fit="H7"`), TABLE, MLEADER, GD&T frames + datums (ISO 1101) |
|
||||
| Engineering generators | involute gears (front + section A-A), DIN 6885 keyed bores, ISO A3 titleblock |
|
||||
| **Paper space** *(new)* | `layout_list/create/set_current`, scaled `viewport_create`, `drawing_export_pdf(layout=...)` |
|
||||
| **3D solids** *(new, opt-in)* | `solid_box/cylinder/extrude/revolve/boolean` on live AutoCAD (`ENABLE_3D=true`) |
|
||||
| Quality loop | `drawing_preflight` → `drawing_plan` → `drawing_critique` → `drawing_refine` → `drawing_finalize` (0-100 score) |
|
||||
| Delivery | `drawing_deliver`: DXF/PDF/PNG + SHA-256 manifest + reopen-parity checks |
|
||||
| Discovery | **tool profiles** *(new)*: `TOOL_PROFILE=lean` (~46 tools) / `core` / `full` for clients with tight tool caps |
|
||||
|
||||
The tool surface evolves — ask `system_about` for the live grouped inventory
|
||||
and `system_capabilities` for per-backend support modes instead of copying
|
||||
lists from this README.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[MCP host / AI agent] --> B[FastMCP server]
|
||||
B --> C[Error - audit - timing - logging middleware]
|
||||
C --> D[Typed tool contract + tool profiles]
|
||||
D --> E[COM backend]
|
||||
D --> F[ezdxf backend]
|
||||
E --> G[Live AutoCAD]
|
||||
F --> H[Headless DXF]
|
||||
E --> I[Engineering quality + delivery]
|
||||
F --> I
|
||||
```
|
||||
|
||||
- `server.py` — FastMCP surface, lifespan, middleware, resources, prompts.
|
||||
- `backends/base.py` — shared typed contract + capability model.
|
||||
- `backends/com_backend.py` — single-STA-thread COM executor with per-call
|
||||
timeouts.
|
||||
- `backends/ezdxf_backend.py` — `asyncio.to_thread`-wrapped DXF engine with
|
||||
snapshot transactions.
|
||||
- `engineering/` — standards-aware planning, generators, critique, scoring,
|
||||
fits, refinement, validation, delivery.
|
||||
- `security.py` — path validation and command/AutoLISP sanitization.
|
||||
|
||||
## Backend capabilities
|
||||
|
||||
@@ -153,210 +165,135 @@ treated as proof that the drawing survived export.
|
||||
| Live AutoCAD document control | ✓ | — |
|
||||
| Headless DXF creation and editing | — | ✓ |
|
||||
| Cross-platform execution | — | ✓ |
|
||||
| Deterministic entity/query contract | ✓ | ✓ |
|
||||
| Transactions and rollback | ✓ | ✓ |
|
||||
| Paper-space layouts + viewports | ✓ | ✓ |
|
||||
| Viewport model-content rendering | ✓ | — |
|
||||
| TABLE and MLEADER semantics | Native | Portable composite |
|
||||
| 3D solids (opt-in) | ✓ | — (no headless ACIS) |
|
||||
| Screenshots | AutoCAD window | Matplotlib render |
|
||||
| Raw AutoCAD commands / AutoLISP | Opt-in | — |
|
||||
|
||||
Use `system_capabilities` at runtime instead of assuming backend support. The
|
||||
machine-readable response distinguishes `native`, `composite`, `emulated`, and
|
||||
`unsupported` behavior where the engines differ.
|
||||
`system_capabilities` returns this machine-readably (`native`, `composite`,
|
||||
`rendered`, `snapshot`, `shared`, `unsupported`) — use it at runtime instead of
|
||||
assuming.
|
||||
|
||||
## A production drawing workflow
|
||||
|
||||
The server exposes low-level CAD primitives, but the intended engineering path
|
||||
uses deterministic orchestration and validation:
|
||||
|
||||
```text
|
||||
drawing_preflight → drawing_plan → drawing_apply_iso_layers
|
||||
→ deterministic create/edit tools → dimension_auto
|
||||
→ deterministic create/edit tools → dimension_auto (+ fit="H7" callouts)
|
||||
→ drawing_critique → drawing_refine
|
||||
→ drawing_finalize → drawing_deliver
|
||||
→ layout_create + viewport_create → drawing_finalize → drawing_deliver
|
||||
```
|
||||
|
||||
Representative capability groups include:
|
||||
|
||||
| Area | Examples |
|
||||
|---|---|
|
||||
| Drawing lifecycle | create, open, save, export, audit, purge, undo/redo |
|
||||
| Geometry | lines, arcs, polylines, hatches, splines, trim/extend/fillet/chamfer |
|
||||
| Annotation | dimensions, ISO 129 tolerances, TABLE, MLEADER, text editing |
|
||||
| Engineering | gears, keyed bores, title blocks, ISO 1101 GD&T and datums |
|
||||
| Query | entity/layer/block inspection, selection, bounds, distance and area |
|
||||
| Quality | preflight, critique, score, transactional refinement and finalization |
|
||||
| Delivery | PDF/PNG/DXF output, hashes, manifests and reopen-parity checks |
|
||||
|
||||
The tool surface evolves. Ask `system_about` for the live grouped inventory
|
||||
instead of building integrations around a copied list from this README.
|
||||
|
||||
## Installation profiles
|
||||
|
||||
| Profile | Command | Use case |
|
||||
|---|---|---|
|
||||
| Core | `pip install -e .` | Headless ezdxf workflows |
|
||||
| COM | `pip install -e ".[com]"` | Live AutoCAD on Windows |
|
||||
| PDF | `pip install -e ".[pdf]"` | PDF export and headless screenshots |
|
||||
| Full | `pip install -e ".[full]"` | Both backends and all render features |
|
||||
|
||||
Requirements:
|
||||
|
||||
- Python 3.11+
|
||||
- [FastMCP](https://github.com/jlowin/fastmcp) 3.0+
|
||||
- [ezdxf](https://github.com/mozman/ezdxf) 1.3+
|
||||
- AutoCAD, `pywin32`, and Pillow only for live COM control
|
||||
- Matplotlib only for PDF/headless rendering and benchmark visualization
|
||||
|
||||
## Connect an MCP client
|
||||
|
||||
### Claude Desktop
|
||||
|
||||
Add a server entry to `claude_desktop_config.json` and use the absolute path to
|
||||
your checkout:
|
||||
### Claude Desktop / Cursor / any stdio host
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"autocad": {
|
||||
"command": "python",
|
||||
"args": ["C:\\path\\to\\Autocad-MCP\\server.py"],
|
||||
"command": "autocad-mcp",
|
||||
"env": {
|
||||
"AUTOCAD_MCP_BACKEND": "auto",
|
||||
"ALLOWED_PATHS": "C:\\Users\\you\\Documents\\AutoCAD"
|
||||
"ALLOWED_PATHS": "C:\\Users\\you\\Documents\\AutoCAD",
|
||||
"TOOL_PROFILE": "full"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Cursor, Cline, Continue, Goose, and other stdio-capable MCP hosts use the same
|
||||
`command` plus `args` shape. The model vendor is not part of the server
|
||||
contract.
|
||||
After `pip install autocad-mcp-pro` the `autocad-mcp` command is on PATH; from
|
||||
a checkout use `"command": "python"`, `"args": ["path/to/server.py"]`. The
|
||||
model vendor is not part of the server contract.
|
||||
|
||||
### Local HTTP
|
||||
|
||||
```bash
|
||||
fastmcp run server.py:mcp --transport http --port 8000
|
||||
autocad-mcp --transport http --port 8000
|
||||
```
|
||||
|
||||
Loopback is the default. A non-loopback bind is refused unless remote HTTP is
|
||||
explicitly enabled and bearer-token authentication is configured.
|
||||
Loopback only by default; a non-loopback bind is refused unless remote HTTP is
|
||||
explicitly enabled **and** a bearer token is configured.
|
||||
|
||||
## Configuration
|
||||
|
||||
Set variables in your shell, process manager, or MCP client configuration.
|
||||
[`.env.example`](.env.example) is a reference; the server reads process
|
||||
environment variables directly.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `AUTOCAD_MCP_BACKEND` | `auto` | Select `auto`, `com`, or `ezdxf` |
|
||||
| `AUTOCAD_MCP_BACKEND` | `auto` | `auto`, `com`, or `ezdxf` |
|
||||
| `TOOL_PROFILE` | `full` | `lean` (~46 curated tools), `core` (hides escape hatches), `full` |
|
||||
| `ENABLE_3D` | `false` | Expose the opt-in `solid_*` tools (COM backend) |
|
||||
| `LOG_LEVEL` | `INFO` | Python logging level |
|
||||
| `ALLOWED_PATHS` | empty | Comma-separated absolute paths the server may access |
|
||||
| `MAX_UNDO_STACK` | `5` | Maximum retained undo snapshots |
|
||||
| `MAX_DXF_BYTES` | `52428800` | Reject larger DXF input; `0` disables the check |
|
||||
| `MAX_LIST_LIMIT` | `5000` | Bound list and selection response sizes |
|
||||
| `COM_CALL_TIMEOUT` | `60` | Per-call live AutoCAD timeout in seconds |
|
||||
| `DANGEROUS_COMMANDS_ENABLED` | `false` | Allow blocked commands/LISP and report unsafe mode |
|
||||
| `MAX_DXF_BYTES` | `52428800` | Reject larger DXF input; `0` disables |
|
||||
| `MAX_LIST_LIMIT` | `5000` | Bound list/selection response sizes |
|
||||
| `COM_CALL_TIMEOUT` | `60` | Per-call live AutoCAD timeout (seconds) |
|
||||
| `DANGEROUS_COMMANDS_ENABLED` | `false` | Allow blocked commands/LISP; reported as unsafe mode |
|
||||
| `ALLOW_REMOTE_HTTP` | `false` | Permit a non-loopback HTTP bind |
|
||||
| `MCP_AUTH_TOKEN` | empty | Bearer token required for remote HTTP |
|
||||
|
||||
Backend selection:
|
||||
|
||||
| Value | Behavior |
|
||||
|---|---|
|
||||
| `auto` | Try COM on Windows, then fall back to ezdxf |
|
||||
| `com` | Require a reachable live AutoCAD instance |
|
||||
| `ezdxf` | Force portable, headless DXF operation |
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[MCP host / AI agent] --> B[FastMCP server]
|
||||
B --> C[Error · audit · timing · logging middleware]
|
||||
C --> D[Typed tool contract]
|
||||
D --> E[COM backend]
|
||||
D --> F[ezdxf backend]
|
||||
E --> G[Live AutoCAD]
|
||||
F --> H[Headless DXF]
|
||||
E --> I[Engineering quality + delivery]
|
||||
F --> I
|
||||
```
|
||||
|
||||
- `server.py` owns the FastMCP surface, lifespan, middleware, resources, and
|
||||
prompt templates.
|
||||
- `backends/base.py` defines the shared contract and capability model.
|
||||
- `backends/com_backend.py` serializes COM calls through one STA thread with
|
||||
per-call timeouts.
|
||||
- `backends/ezdxf_backend.py` wraps synchronous DXF work with
|
||||
`asyncio.to_thread` and snapshot-based transactions.
|
||||
- `engineering/` contains standards-aware planning, drawing, critique, scoring,
|
||||
refinement, validation, and delivery.
|
||||
- `security.py` centralizes path validation and command/LISP sanitization.
|
||||
|
||||
## Security model
|
||||
|
||||
The server treats tool input as untrusted:
|
||||
Tool input is treated as untrusted:
|
||||
|
||||
- Every file path passes through traversal and allowed-root validation.
|
||||
- Dangerous AutoCAD commands and AutoLISP channels are blocked by default.
|
||||
- Remote HTTP requires explicit opt-in and a bearer token.
|
||||
- DXF size, query-result size, undo history, and COM call duration are bounded.
|
||||
- Audit middleware records tool timing and failures.
|
||||
- `system_status` and `system_about` expose `unsafe_mode` when dangerous
|
||||
- every path passes traversal + allowed-root validation;
|
||||
- dangerous AutoCAD commands and AutoLISP channels are blocked by default,
|
||||
with regression tests for known bypass patterns;
|
||||
- remote HTTP requires explicit opt-in plus a bearer token;
|
||||
- DXF size, response sizes, undo history and COM call duration are bounded;
|
||||
- audit middleware records tool timing and failures;
|
||||
- `system_status` / `system_about` expose `unsafe_mode` when dangerous
|
||||
operations are enabled.
|
||||
|
||||
`DANGEROUS_COMMANDS_ENABLED=true` is a trust decision, not an installation
|
||||
shortcut. Keep it disabled for agents or clients you do not fully control.
|
||||
|
||||
Please avoid filing public issues for vulnerabilities. Contact the maintainer
|
||||
through [GitHub](https://github.com/U-C4N) first.
|
||||
Please report vulnerabilities privately via [GitHub](https://github.com/U-C4N)
|
||||
rather than public issues.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
pip install -e ".[full]"
|
||||
pip install pytest pytest-asyncio pytest-cov ruff build
|
||||
python -m pytest
|
||||
python -m ruff check .
|
||||
python -m ruff format --check .
|
||||
python -m pytest # 474 tests
|
||||
python -m ruff check . && python -m ruff format --check .
|
||||
python -m build
|
||||
```
|
||||
|
||||
Focused benchmark checks:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/test_benchmark_chart.py tests/test_benchmark_v2.py -q
|
||||
python -m benchmarks.render_chart
|
||||
```
|
||||
CI runs the same gates on Linux (3.11/3.12), Windows (mocked-COM suite),
|
||||
plus package, Docker and MCP-registry-schema jobs. Releases are tag-driven:
|
||||
`git tag vX.Y.Z && git push origin vX.Y.Z` builds, publishes to PyPI and cuts
|
||||
the GitHub Release. See
|
||||
[`docs/RELEASE-DISTRIBUTION.md`](docs/RELEASE-DISTRIBUTION.md).
|
||||
|
||||
Repository map:
|
||||
|
||||
```text
|
||||
server.py FastMCP surface and orchestration
|
||||
config.py Environment-driven settings
|
||||
security.py Path, command, and AutoLISP guards
|
||||
version.py Canonical package version
|
||||
backends/ COM and ezdxf implementations
|
||||
engineering/ Standards, critique, refinement, delivery
|
||||
benchmarks/ Reproducible data, runners, and reports
|
||||
tests/ Headless, mocked-COM, contract, and security tests
|
||||
server.py FastMCP surface and orchestration
|
||||
config.py Environment-driven settings
|
||||
security.py Path, command, and AutoLISP guards
|
||||
version.py Canonical package version
|
||||
backends/ COM and ezdxf implementations
|
||||
engineering/ Standards, generators, critique, fits, delivery
|
||||
benchmarks/ Runners, adapters, published reports, chart renderers
|
||||
tests/ Headless, mocked-COM, contract, and security tests
|
||||
```
|
||||
|
||||
## Roadmap
|
||||
## Roadmap (1.5)
|
||||
|
||||
The next priorities are capability-aware tool discovery, paper-space/layout
|
||||
depth, ISO 286 fits, Windows CI coverage, opt-in COM 3D solids, and real
|
||||
cross-project runtime adapters. Features are published when their contracts and
|
||||
limitations are testable—not when they make a longer checklist.
|
||||
Screenshot overlay + handle grounding, ezdxf redo (forward snapshots), COM
|
||||
`block_create_from_entities`, ISO 286 transition/interference hole letters
|
||||
(delta rule), titleblock on paper-space layouts. Features are published when
|
||||
their contracts and limitations are testable — not when they make a longer
|
||||
checklist.
|
||||
|
||||
## Contributing
|
||||
|
||||
Pull requests are welcome, especially for reproducible competitor adapters,
|
||||
Pull requests are welcome — especially reproducible competitor adapters,
|
||||
mocked/live COM coverage, engineering standards, and backend parity. For a
|
||||
large change, open an issue first so the tool contract and evidence plan can be
|
||||
agreed before implementation.
|
||||
large change, open an issue first so the tool contract and evidence plan can
|
||||
be agreed before implementation.
|
||||
|
||||
## Author
|
||||
|
||||
|
||||
+20
-2
@@ -78,13 +78,31 @@ Method and boundaries:
|
||||
require a local AutoCAD session and are out of CI scope by design.
|
||||
|
||||
Published machine-readable reports (artifact paths sanitized to filenames)
|
||||
live under [`results/published/`](results/published/); the README chart is
|
||||
live under [`results/published/`](results/published/); the README charts are
|
||||
regenerated from them:
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
|
||||
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf
|
||||
python -m benchmarks.render_live_chart
|
||||
python -m benchmarks.render_live_chart # score bars
|
||||
python -m benchmarks.render_matrix_chart # tasks x servers status heatmap
|
||||
```
|
||||
|
||||
## Headless performance lane
|
||||
|
||||
`perf_suite.py` measures wall time on fixed workloads through the same backend
|
||||
methods the MCP tools call (server-side overhead included): 2,000 individual
|
||||
line creates, a 10,000-entity build → DXF export → reopen roundtrip, a region
|
||||
query over 10,000 entities, and a full premium quality pass (ISO layers,
|
||||
geometry, dimensions, complete critique). The report records the machine
|
||||
fingerprint; numbers move with hardware, workload definitions do not.
|
||||
**Self-measurement only** — competitor servers would pay an extra stdio
|
||||
serialization cost that in-process runs do not, so no cross-server timing
|
||||
claims are made.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||||
python -m benchmarks.render_perf_chart
|
||||
```
|
||||
|
||||
## Correctness A/B suite
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
"""Headless performance suite (v3 lane): wall-time on real drawing workloads.
|
||||
|
||||
Complements the correctness/coverage lanes with *scale* evidence: how long the
|
||||
ezdxf engine takes to create, roundtrip, query and quality-check real geometry.
|
||||
Workloads drive the backend exactly like the MCP tools do (same async methods),
|
||||
so the numbers include the server-side call overhead, not just raw ezdxf.
|
||||
|
||||
Honesty boundaries:
|
||||
|
||||
- Self-measurement only — no competitor timing claims (their servers would pay
|
||||
an extra stdio serialization cost ours does not pay in-process).
|
||||
- Wall-clock on the machine recorded in the report; absolute numbers move with
|
||||
hardware, the workload definitions do not.
|
||||
|
||||
Run:
|
||||
|
||||
python -m benchmarks.perf_suite # human summary
|
||||
python -m benchmarks.perf_suite --json # machine-readable report
|
||||
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import platform
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from backends.ezdxf_backend import EzdxfBackend
|
||||
|
||||
|
||||
@dataclass
|
||||
class WorkloadResult:
|
||||
workload_id: str
|
||||
description: str
|
||||
wall_ms: float
|
||||
metric: str
|
||||
metric_value: float
|
||||
detail: dict
|
||||
|
||||
|
||||
async def _timed(coro) -> tuple[float, object]:
|
||||
started = time.perf_counter()
|
||||
result = await coro
|
||||
return (time.perf_counter() - started) * 1000.0, result
|
||||
|
||||
|
||||
async def workload_create_lines_2k(backend: EzdxfBackend) -> WorkloadResult:
|
||||
"""2,000 individual entity_create_line calls (per-call tool overhead included)."""
|
||||
await backend.drawing_new()
|
||||
started = time.perf_counter()
|
||||
for index in range(2_000):
|
||||
y = float(index % 100)
|
||||
await backend.entity_create_line(0.0, y, 100.0, y + 1.0)
|
||||
wall_ms = (time.perf_counter() - started) * 1000.0
|
||||
count = (await backend.drawing_info()).entity_count
|
||||
return WorkloadResult(
|
||||
"create_lines_2k",
|
||||
"Create 2,000 lines through individual backend calls",
|
||||
round(wall_ms, 1),
|
||||
"entities_per_second",
|
||||
round(2_000 / (wall_ms / 1000.0), 1),
|
||||
{"entity_count": count},
|
||||
)
|
||||
|
||||
|
||||
async def workload_roundtrip_10k(backend: EzdxfBackend, scratch: Path) -> WorkloadResult:
|
||||
"""Build a 10,000-entity drawing, export DXF, reopen, read info."""
|
||||
await backend.drawing_new()
|
||||
build_started = time.perf_counter()
|
||||
for index in range(10_000):
|
||||
x = float(index % 500)
|
||||
y = float(index // 500)
|
||||
await backend.entity_create_line(x, y, x + 0.8, y + 0.8)
|
||||
build_ms = (time.perf_counter() - build_started) * 1000.0
|
||||
|
||||
target = scratch / "perf_roundtrip_10k.dxf"
|
||||
export_ms, _ = await _timed(backend.drawing_export_dxf(str(target)))
|
||||
|
||||
reopened = EzdxfBackend()
|
||||
await reopened.connect()
|
||||
try:
|
||||
reopen_ms, _ = await _timed(reopened.drawing_open(str(target)))
|
||||
count = (await reopened.drawing_info()).entity_count
|
||||
finally:
|
||||
await reopened.disconnect()
|
||||
|
||||
total = build_ms + export_ms + reopen_ms
|
||||
return WorkloadResult(
|
||||
"roundtrip_10k",
|
||||
"Create 10,000 lines, export DXF, reopen and verify",
|
||||
round(total, 1),
|
||||
"entities_per_second",
|
||||
round(10_000 / (total / 1000.0), 1),
|
||||
{
|
||||
"build_ms": round(build_ms, 1),
|
||||
"export_ms": round(export_ms, 1),
|
||||
"reopen_ms": round(reopen_ms, 1),
|
||||
"file_bytes": target.stat().st_size,
|
||||
"entity_count": count,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
async def workload_region_query_10k(backend: EzdxfBackend) -> WorkloadResult:
|
||||
"""Rectangular region query against the 10,000-entity drawing."""
|
||||
# backend still holds the 10k drawing from the roundtrip workload builder;
|
||||
# rebuild deterministically to stay order-independent.
|
||||
await backend.drawing_new()
|
||||
for index in range(10_000):
|
||||
x = float(index % 500)
|
||||
y = float(index // 500)
|
||||
await backend.entity_create_line(x, y, x + 0.8, y + 0.8)
|
||||
wall_ms, found = await _timed(backend.analysis_entities_in_region(0.0, 0.0, 250.0, 10.0))
|
||||
return WorkloadResult(
|
||||
"region_query_10k",
|
||||
"Crossing-selection region query over 10,000 entities",
|
||||
round(wall_ms, 1),
|
||||
"entities_matched",
|
||||
float(len(found)),
|
||||
{"entity_count": 10_000},
|
||||
)
|
||||
|
||||
|
||||
async def workload_premium_pass(backend: EzdxfBackend) -> WorkloadResult:
|
||||
"""Small end-to-end quality loop: layers, geometry, dimensions, critique."""
|
||||
started = time.perf_counter()
|
||||
await backend.drawing_new()
|
||||
await backend.drawing_apply_iso_layers("mech")
|
||||
plate = [
|
||||
(0.0, 0.0, 120.0, 0.0),
|
||||
(120.0, 0.0, 120.0, 80.0),
|
||||
(120.0, 80.0, 0.0, 80.0),
|
||||
(0.0, 80.0, 0.0, 0.0),
|
||||
]
|
||||
for x1, y1, x2, y2 in plate:
|
||||
await backend.entity_create_line(x1, y1, x2, y2, layer="GEOMETRY")
|
||||
await backend.entity_create_circle(60.0, 40.0, 12.0, layer="GEOMETRY")
|
||||
await backend.dimension_linear(0, 0, 120, 0, 60, -15, layer="DIM")
|
||||
await backend.dimension_linear(0, 0, 0, 80, -15, 40, 90, layer="DIM")
|
||||
issues = await backend.drawing_critique(None)
|
||||
wall_ms = (time.perf_counter() - started) * 1000.0
|
||||
return WorkloadResult(
|
||||
"premium_pass",
|
||||
"Plate with hole: ISO layers, geometry, dimensions and full critique",
|
||||
round(wall_ms, 1),
|
||||
"critique_issues",
|
||||
float(len(issues)),
|
||||
{"entities": (await backend.drawing_info()).entity_count},
|
||||
)
|
||||
|
||||
|
||||
async def run_suite(scratch: Path) -> dict:
|
||||
backend = EzdxfBackend()
|
||||
await backend.connect()
|
||||
try:
|
||||
results = [
|
||||
await workload_create_lines_2k(backend),
|
||||
await workload_roundtrip_10k(backend, scratch),
|
||||
await workload_region_query_10k(backend),
|
||||
await workload_premium_pass(backend),
|
||||
]
|
||||
finally:
|
||||
await backend.disconnect()
|
||||
return {
|
||||
"schema_version": "perf/1.0",
|
||||
"backend": "ezdxf",
|
||||
"environment": {
|
||||
"python": sys.version.split()[0],
|
||||
"implementation": platform.python_implementation(),
|
||||
"platform": platform.platform(),
|
||||
"machine": platform.machine(),
|
||||
},
|
||||
"workloads": [asdict(item) for item in results],
|
||||
}
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON")
|
||||
parser.add_argument("--out", type=Path, default=None, help="Also write the report to a file")
|
||||
parser.add_argument("--scratch", type=Path, default=Path("benchmarks/results/perf-scratch"))
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = build_parser().parse_args()
|
||||
args.scratch.mkdir(parents=True, exist_ok=True)
|
||||
report = asyncio.run(run_suite(args.scratch))
|
||||
rendered = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
if args.out is not None:
|
||||
args.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.out.write_text(rendered + "\n", encoding="utf-8")
|
||||
if args.json:
|
||||
print(rendered)
|
||||
else:
|
||||
for item in report["workloads"]:
|
||||
print(
|
||||
f"{item['workload_id']:>18} {item['wall_ms']:>10.1f} ms "
|
||||
f"{item['metric']}={item['metric_value']}"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -23,7 +23,7 @@ def load_reports(input_dir: Path) -> list[dict[str, Any]]:
|
||||
for path in sorted(input_dir.glob("*.json")):
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if data.get("schema_version") != "2.0":
|
||||
raise ValueError(f"{path.name}: unsupported schema_version")
|
||||
continue # other lanes (e.g. perf/1.0) share the published folder
|
||||
summary = data.get("summary") or {}
|
||||
if "score" not in summary or "coverage_percent" not in summary:
|
||||
raise ValueError(f"{path.name}: summary is missing score/coverage")
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Render the live-run task matrix (tasks x servers) as a deterministic SVG.
|
||||
|
||||
Input: the published per-server reports under ``benchmarks/results/published/``
|
||||
(the same files the score chart uses). Output: a status heatmap — every task
|
||||
against every server — so coverage gaps are visible per capability instead of
|
||||
being folded into one number.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from benchmarks.tasks_v2 import TASKS_V2
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
|
||||
DEFAULT_OUTPUT = ROOT / "docs" / "assets" / "autocad-mcp-taskmatrix.svg"
|
||||
|
||||
STATUS_COLORS = {
|
||||
"pass": "#2da44e",
|
||||
"partial": "#d4a72c",
|
||||
"fail": "#cf222e",
|
||||
"timeout": "#a40e26",
|
||||
"unsupported": "#d0d7de",
|
||||
"not_run": "#f6f8fa",
|
||||
}
|
||||
|
||||
STATUS_GLYPHS = {
|
||||
"pass": "P",
|
||||
"partial": "~",
|
||||
"fail": "F",
|
||||
"timeout": "T",
|
||||
"unsupported": "-",
|
||||
"not_run": "?",
|
||||
}
|
||||
|
||||
|
||||
def load_reports(input_dir: Path) -> list[dict[str, Any]]:
|
||||
reports: list[dict[str, Any]] = []
|
||||
for path in sorted(input_dir.glob("*.json")):
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if data.get("schema_version") != "2.0":
|
||||
continue # perf reports and other lanes live in the same folder
|
||||
reports.append(data)
|
||||
if not reports:
|
||||
raise SystemExit(f"no task-matrix reports found in {input_dir}")
|
||||
# our server first, then alphabetical
|
||||
reports.sort(key=lambda item: (item["adapter"] != "autocad-mcp-pro", item["adapter"]))
|
||||
return reports
|
||||
|
||||
|
||||
def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||
try:
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
from matplotlib.patches import Patch, Rectangle
|
||||
except ImportError as exc:
|
||||
raise SystemExit(
|
||||
'Matplotlib is required. Install it with: pip install -e ".[pdf]"'
|
||||
) from exc
|
||||
|
||||
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
|
||||
|
||||
task_ids = [task.task_id for task in TASKS_V2]
|
||||
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V2]
|
||||
servers = [report["adapter"] for report in reports]
|
||||
status_by_server = {
|
||||
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
|
||||
for report in reports
|
||||
}
|
||||
|
||||
fig, ax = plt.subplots(
|
||||
figsize=(3.4 + 2.4 * len(servers), 1.8 + 0.52 * len(task_ids)),
|
||||
constrained_layout=True,
|
||||
)
|
||||
fig.patch.set_facecolor("#f6f8fa")
|
||||
ax.set_facecolor("#f6f8fa")
|
||||
|
||||
for row, task_id in enumerate(task_ids):
|
||||
for col, server in enumerate(servers):
|
||||
status = status_by_server[server].get(task_id, "not_run")
|
||||
color = STATUS_COLORS.get(status, "#f6f8fa")
|
||||
ax.add_patch(
|
||||
Rectangle(
|
||||
(col + 0.06, row + 0.08),
|
||||
0.88,
|
||||
0.84,
|
||||
facecolor=color,
|
||||
edgecolor="#d0d7de",
|
||||
linewidth=0.8,
|
||||
)
|
||||
)
|
||||
glyph_color = "#ffffff" if status in ("pass", "fail", "timeout") else "#24292f"
|
||||
ax.text(
|
||||
col + 0.5,
|
||||
row + 0.5,
|
||||
STATUS_GLYPHS.get(status, "?"),
|
||||
ha="center",
|
||||
va="center",
|
||||
fontsize=11,
|
||||
fontweight="bold",
|
||||
color=glyph_color,
|
||||
)
|
||||
|
||||
ax.set_xlim(0, len(servers))
|
||||
ax.set_ylim(len(task_ids), 0)
|
||||
ax.set_xticks([index + 0.5 for index in range(len(servers))])
|
||||
ax.set_xticklabels(servers, color="#24292f", fontsize=10, fontweight="bold")
|
||||
ax.xaxis.set_ticks_position("top")
|
||||
ax.set_yticks([index + 0.5 for index in range(len(task_ids))])
|
||||
ax.set_yticklabels(task_labels, color="#24292f", fontsize=9)
|
||||
ax.tick_params(length=0)
|
||||
for spine in ax.spines.values():
|
||||
spine.set_visible(False)
|
||||
|
||||
ax.set_title(
|
||||
"Task matrix — live headless lane",
|
||||
loc="left",
|
||||
fontsize=17,
|
||||
fontweight="bold",
|
||||
color="#24292f",
|
||||
pad=34,
|
||||
)
|
||||
|
||||
legend_handles = [
|
||||
Patch(facecolor=STATUS_COLORS["pass"], label="pass"),
|
||||
Patch(facecolor=STATUS_COLORS["partial"], label="partial"),
|
||||
Patch(facecolor=STATUS_COLORS["fail"], label="fail"),
|
||||
Patch(facecolor=STATUS_COLORS["unsupported"], label="unsupported (no equivalent)"),
|
||||
]
|
||||
ax.legend(
|
||||
handles=legend_handles,
|
||||
loc="upper left",
|
||||
bbox_to_anchor=(0, -0.02),
|
||||
ncol=4,
|
||||
frameon=False,
|
||||
fontsize=9,
|
||||
)
|
||||
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
fig.savefig(
|
||||
output,
|
||||
format="svg",
|
||||
facecolor=fig.get_facecolor(),
|
||||
metadata={"Creator": "AutoCAD MCP Pro task-matrix renderer", "Date": None},
|
||||
)
|
||||
plt.close(fig)
|
||||
|
||||
svg = output.read_text(encoding="utf-8")
|
||||
normalized = "\n".join(line.rstrip() for line in svg.splitlines()) + "\n"
|
||||
output.write_text(normalized, encoding="utf-8", newline="\n")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--input-dir", type=Path, default=DEFAULT_INPUT_DIR)
|
||||
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = build_parser().parse_args()
|
||||
render_chart(load_reports(args.input_dir), args.output)
|
||||
print(f"Rendered task-matrix chart: {args.output}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,146 @@
|
||||
"""Render the headless performance suite as a deterministic SVG.
|
||||
|
||||
Input: the report produced by ``benchmarks.perf_suite --out ...``
|
||||
(default: ``benchmarks/results/published/perf-ezdxf.json``).
|
||||
Output: a GitHub-readable log-scale wall-time chart per workload.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DEFAULT_INPUT = ROOT / "benchmarks" / "results" / "published" / "perf-ezdxf.json"
|
||||
DEFAULT_OUTPUT = ROOT / "docs" / "assets" / "autocad-mcp-perf.svg"
|
||||
|
||||
_LABELS = {
|
||||
"create_lines_2k": "Create 2,000 lines (individual calls)",
|
||||
"roundtrip_10k": "10,000 lines: build + DXF export + reopen",
|
||||
"region_query_10k": "Region query over 10,000 entities",
|
||||
"premium_pass": "Premium pass: layers + part + dims + full critique",
|
||||
}
|
||||
|
||||
|
||||
def load_report(path: Path) -> dict[str, Any]:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
if data.get("schema_version") != "perf/1.0":
|
||||
raise ValueError("unsupported perf report schema_version")
|
||||
if not data.get("workloads"):
|
||||
raise ValueError("perf report contains no workloads")
|
||||
return data
|
||||
|
||||
|
||||
def _annotation(item: dict[str, Any]) -> str:
|
||||
wall_s = item["wall_ms"] / 1000.0
|
||||
if item["metric"] == "entities_per_second":
|
||||
return f"{wall_s:.2f} s · {item['metric_value']:,.0f} entities/s"
|
||||
if item["metric"] == "entities_matched":
|
||||
return f"{wall_s:.2f} s · {item['metric_value']:,.0f} matched"
|
||||
if item["metric"] == "critique_issues":
|
||||
return f"{wall_s:.2f} s · {item['metric_value']:.0f} issues"
|
||||
return f"{wall_s:.2f} s"
|
||||
|
||||
|
||||
def render_chart(data: dict[str, Any], output: Path) -> None:
|
||||
try:
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
except ImportError as exc:
|
||||
raise SystemExit(
|
||||
'Matplotlib is required. Install it with: pip install -e ".[pdf]"'
|
||||
) from exc
|
||||
|
||||
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-perf-v1"
|
||||
workloads = sorted(data["workloads"], key=lambda item: item["wall_ms"])
|
||||
labels = [_LABELS.get(item["workload_id"], item["workload_id"]) for item in workloads]
|
||||
values = [item["wall_ms"] for item in workloads]
|
||||
|
||||
fig, ax = plt.subplots(figsize=(12, 4.6), constrained_layout=True)
|
||||
fig.patch.set_facecolor("#f6f8fa")
|
||||
ax.set_facecolor("#f6f8fa")
|
||||
bars = ax.barh(labels, values, color="#0969da", height=0.58)
|
||||
|
||||
ax.set_xscale("log")
|
||||
ax.set_xlabel("Wall time in ms (log scale) — lower is better", color="#24292f", labelpad=10)
|
||||
ax.set_title(
|
||||
"Headless performance — ezdxf engine",
|
||||
loc="left",
|
||||
fontsize=18,
|
||||
fontweight="bold",
|
||||
color="#24292f",
|
||||
pad=28,
|
||||
)
|
||||
env = data.get("environment", {})
|
||||
ax.text(
|
||||
0,
|
||||
1.02,
|
||||
f"Fixed workloads via the same backend methods the MCP tools call · "
|
||||
f"Python {env.get('python', '?')} · {env.get('platform', '?')}",
|
||||
transform=ax.transAxes,
|
||||
color="#57606a",
|
||||
fontsize=9,
|
||||
)
|
||||
|
||||
for bar, item in zip(bars, workloads, strict=True):
|
||||
ax.text(
|
||||
bar.get_width() * 1.12,
|
||||
bar.get_y() + bar.get_height() / 2,
|
||||
_annotation(item),
|
||||
va="center",
|
||||
color="#24292f",
|
||||
fontweight="bold",
|
||||
fontsize=9,
|
||||
)
|
||||
|
||||
ax.set_xlim(right=max(values) * 8)
|
||||
ax.text(
|
||||
0,
|
||||
-0.24,
|
||||
"Self-measurement only: competitor servers would pay extra stdio serialization cost in-process runs do not. "
|
||||
"Reproduce: python -m benchmarks.perf_suite",
|
||||
transform=ax.transAxes,
|
||||
color="#57606a",
|
||||
fontsize=9,
|
||||
)
|
||||
ax.grid(axis="x", color="#d0d7de", linewidth=0.8, alpha=0.8)
|
||||
ax.set_axisbelow(True)
|
||||
ax.tick_params(axis="x", colors="#57606a")
|
||||
ax.tick_params(axis="y", colors="#24292f", length=0, pad=8)
|
||||
for spine in ax.spines.values():
|
||||
spine.set_visible(False)
|
||||
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
fig.savefig(
|
||||
output,
|
||||
format="svg",
|
||||
facecolor=fig.get_facecolor(),
|
||||
metadata={"Creator": "AutoCAD MCP Pro perf renderer", "Date": None},
|
||||
)
|
||||
plt.close(fig)
|
||||
|
||||
svg = output.read_text(encoding="utf-8")
|
||||
normalized = "\n".join(line.rstrip() for line in svg.splitlines()) + "\n"
|
||||
output.write_text(normalized, encoding="utf-8", newline="\n")
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--input", type=Path, default=DEFAULT_INPUT)
|
||||
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
|
||||
return parser
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = build_parser().parse_args()
|
||||
render_chart(load_report(args.input), args.output)
|
||||
print(f"Rendered perf chart: {args.output}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"backend": "ezdxf",
|
||||
"environment": {
|
||||
"implementation": "CPython",
|
||||
"machine": "AMD64",
|
||||
"platform": "Windows-11-10.0.26200-SP0",
|
||||
"python": "3.14.0"
|
||||
},
|
||||
"schema_version": "perf/1.0",
|
||||
"workloads": [
|
||||
{
|
||||
"description": "Create 2,000 lines through individual backend calls",
|
||||
"detail": {
|
||||
"entity_count": 2000
|
||||
},
|
||||
"metric": "entities_per_second",
|
||||
"metric_value": 3148.0,
|
||||
"wall_ms": 635.3,
|
||||
"workload_id": "create_lines_2k"
|
||||
},
|
||||
{
|
||||
"description": "Create 10,000 lines, export DXF, reopen and verify",
|
||||
"detail": {
|
||||
"build_ms": 10884.9,
|
||||
"entity_count": 10000,
|
||||
"export_ms": 1439.2,
|
||||
"file_bytes": 1369662,
|
||||
"reopen_ms": 1607.5
|
||||
},
|
||||
"metric": "entities_per_second",
|
||||
"metric_value": 717.8,
|
||||
"wall_ms": 13931.6,
|
||||
"workload_id": "roundtrip_10k"
|
||||
},
|
||||
{
|
||||
"description": "Crossing-selection region query over 10,000 entities",
|
||||
"detail": {
|
||||
"entity_count": 10000
|
||||
},
|
||||
"metric": "entities_matched",
|
||||
"metric_value": 2500.0,
|
||||
"wall_ms": 3434.2,
|
||||
"workload_id": "region_query_10k"
|
||||
},
|
||||
{
|
||||
"description": "Plate with hole: ISO layers, geometry, dimensions and full critique",
|
||||
"detail": {
|
||||
"entities": 7
|
||||
},
|
||||
"metric": "critique_issues",
|
||||
"metric_value": 0.0,
|
||||
"wall_ms": 315.9,
|
||||
"workload_id": "premium_pass"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 87 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 69 KiB |
Reference in New Issue
Block a user