anatoly/autocad-mcp-pro
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes

Browse Source 
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only
- benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time)
- benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports)
- render_live_chart now tolerates other lanes sharing results/published/
- README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap
- published perf report: benchmarks/results/published/perf-ezdxf.json

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Umutcan Edizsalan
2026-07-23 15:53:04 +03:00
co-authored by Cursor
parent 67f2c3b575
commit 21cbc62df5
11 changed files with 8047 additions and 231 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/README.md
+20 -2
View File
@@ -78,13 +78,31 @@ Method and boundaries:
require a local AutoCAD session and are out of CI scope by design.
Published machine-readable reports (artifact paths sanitized to filenames)
live under [`results/published/`](results/published/); the README chart is
live under [`results/published/`](results/published/); the README charts are
regenerated from them:
```bash
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf
python -m benchmarks.render_live_chart
python -m benchmarks.render_live_chart # score bars
python -m benchmarks.render_matrix_chart # tasks x servers status heatmap
```
## Headless performance lane
`perf_suite.py` measures wall time on fixed workloads through the same backend
methods the MCP tools call (server-side overhead included): 2,000 individual
line creates, a 10,000-entity build → DXF export → reopen roundtrip, a region
query over 10,000 entities, and a full premium quality pass (ISO layers,
geometry, dimensions, complete critique). The report records the machine
fingerprint; numbers move with hardware, workload definitions do not.
**Self-measurement only** — competitor servers would pay an extra stdio
serialization cost that in-process runs do not, so no cross-server timing
claims are made.
```bash
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
python -m benchmarks.render_perf_chart
```
## Correctness A/B suite
benchmarks/perf_suite.py
+210
View File
@@ -0,0 +1,210 @@
"""Headless performance suite (v3 lane): wall-time on real drawing workloads.
Complements the correctness/coverage lanes with *scale* evidence: how long the
ezdxf engine takes to create, roundtrip, query and quality-check real geometry.
Workloads drive the backend exactly like the MCP tools do (same async methods),
so the numbers include the server-side call overhead, not just raw ezdxf.
Honesty boundaries:
- Self-measurement only — no competitor timing claims (their servers would pay
an extra stdio serialization cost ours does not pay in-process).
- Wall-clock on the machine recorded in the report; absolute numbers move with
hardware, the workload definitions do not.
Run:
python -m benchmarks.perf_suite # human summary
python -m benchmarks.perf_suite --json # machine-readable report
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
"""
from __future__ import annotations
import argparse
import asyncio
import json
import platform
import sys
import time
from dataclasses import asdict, dataclass
from pathlib import Path
from backends.ezdxf_backend import EzdxfBackend
@dataclass
class WorkloadResult:
workload_id: str
description: str
wall_ms: float
metric: str
metric_value: float
detail: dict
async def _timed(coro) -> tuple[float, object]:
started = time.perf_counter()
result = await coro
return (time.perf_counter() - started) * 1000.0, result
async def workload_create_lines_2k(backend: EzdxfBackend) -> WorkloadResult:
"""2,000 individual entity_create_line calls (per-call tool overhead included)."""
await backend.drawing_new()
started = time.perf_counter()
for index in range(2_000):
y = float(index % 100)
await backend.entity_create_line(0.0, y, 100.0, y + 1.0)
wall_ms = (time.perf_counter() - started) * 1000.0
count = (await backend.drawing_info()).entity_count
return WorkloadResult(
"create_lines_2k",
"Create 2,000 lines through individual backend calls",
round(wall_ms, 1),
"entities_per_second",
round(2_000 / (wall_ms / 1000.0), 1),
{"entity_count": count},
)
async def workload_roundtrip_10k(backend: EzdxfBackend, scratch: Path) -> WorkloadResult:
"""Build a 10,000-entity drawing, export DXF, reopen, read info."""
await backend.drawing_new()
build_started = time.perf_counter()
for index in range(10_000):
x = float(index % 500)
y = float(index // 500)
await backend.entity_create_line(x, y, x + 0.8, y + 0.8)
build_ms = (time.perf_counter() - build_started) * 1000.0
target = scratch / "perf_roundtrip_10k.dxf"
export_ms, _ = await _timed(backend.drawing_export_dxf(str(target)))
reopened = EzdxfBackend()
await reopened.connect()
try:
reopen_ms, _ = await _timed(reopened.drawing_open(str(target)))
count = (await reopened.drawing_info()).entity_count
finally:
await reopened.disconnect()
total = build_ms + export_ms + reopen_ms
return WorkloadResult(
"roundtrip_10k",
"Create 10,000 lines, export DXF, reopen and verify",
round(total, 1),
"entities_per_second",
round(10_000 / (total / 1000.0), 1),
{
"build_ms": round(build_ms, 1),
"export_ms": round(export_ms, 1),
"reopen_ms": round(reopen_ms, 1),
"file_bytes": target.stat().st_size,
"entity_count": count,
},
)
async def workload_region_query_10k(backend: EzdxfBackend) -> WorkloadResult:
"""Rectangular region query against the 10,000-entity drawing."""
# backend still holds the 10k drawing from the roundtrip workload builder;
# rebuild deterministically to stay order-independent.
await backend.drawing_new()
for index in range(10_000):
x = float(index % 500)
y = float(index // 500)
await backend.entity_create_line(x, y, x + 0.8, y + 0.8)
wall_ms, found = await _timed(backend.analysis_entities_in_region(0.0, 0.0, 250.0, 10.0))
return WorkloadResult(
"region_query_10k",
"Crossing-selection region query over 10,000 entities",
round(wall_ms, 1),
"entities_matched",
float(len(found)),
{"entity_count": 10_000},
)
async def workload_premium_pass(backend: EzdxfBackend) -> WorkloadResult:
"""Small end-to-end quality loop: layers, geometry, dimensions, critique."""
started = time.perf_counter()
await backend.drawing_new()
await backend.drawing_apply_iso_layers("mech")
plate = [
(0.0, 0.0, 120.0, 0.0),
(120.0, 0.0, 120.0, 80.0),
(120.0, 80.0, 0.0, 80.0),
(0.0, 80.0, 0.0, 0.0),
]
for x1, y1, x2, y2 in plate:
await backend.entity_create_line(x1, y1, x2, y2, layer="GEOMETRY")
await backend.entity_create_circle(60.0, 40.0, 12.0, layer="GEOMETRY")
await backend.dimension_linear(0, 0, 120, 0, 60, -15, layer="DIM")
await backend.dimension_linear(0, 0, 0, 80, -15, 40, 90, layer="DIM")
issues = await backend.drawing_critique(None)
wall_ms = (time.perf_counter() - started) * 1000.0
return WorkloadResult(
"premium_pass",
"Plate with hole: ISO layers, geometry, dimensions and full critique",
round(wall_ms, 1),
"critique_issues",
float(len(issues)),
{"entities": (await backend.drawing_info()).entity_count},
)
async def run_suite(scratch: Path) -> dict:
backend = EzdxfBackend()
await backend.connect()
try:
results = [
await workload_create_lines_2k(backend),
await workload_roundtrip_10k(backend, scratch),
await workload_region_query_10k(backend),
await workload_premium_pass(backend),
]
finally:
await backend.disconnect()
return {
"schema_version": "perf/1.0",
"backend": "ezdxf",
"environment": {
"python": sys.version.split()[0],
"implementation": platform.python_implementation(),
"platform": platform.platform(),
"machine": platform.machine(),
},
"workloads": [asdict(item) for item in results],
}
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--json", action="store_true", help="Print machine-readable JSON")
parser.add_argument("--out", type=Path, default=None, help="Also write the report to a file")
parser.add_argument("--scratch", type=Path, default=Path("benchmarks/results/perf-scratch"))
return parser
def main() -> int:
args = build_parser().parse_args()
args.scratch.mkdir(parents=True, exist_ok=True)
report = asyncio.run(run_suite(args.scratch))
rendered = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)
if args.out is not None:
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(rendered + "\n", encoding="utf-8")
if args.json:
print(rendered)
else:
for item in report["workloads"]:
print(
f"{item['workload_id']:>18} {item['wall_ms']:>10.1f} ms "
f"{item['metric']}={item['metric_value']}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
benchmarks/render_live_chart.py
+1 -1
View File
@@ -23,7 +23,7 @@ def load_reports(input_dir: Path) -> list[dict[str, Any]]:
for path in sorted(input_dir.glob("*.json")):
data = json.loads(path.read_text(encoding="utf-8"))
if data.get("schema_version") != "2.0":
raise ValueError(f"{path.name}: unsupported schema_version")
continue # other lanes (e.g. perf/1.0) share the published folder
summary = data.get("summary") or {}
if "score" not in summary or "coverage_percent" not in summary:
raise ValueError(f"{path.name}: summary is missing score/coverage")
benchmarks/render_matrix_chart.py
+174
View File
@@ -0,0 +1,174 @@
"""Render the live-run task matrix (tasks x servers) as a deterministic SVG.
Input: the published per-server reports under ``benchmarks/results/published/``
(the same files the score chart uses). Output: a status heatmap — every task
against every server — so coverage gaps are visible per capability instead of
being folded into one number.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
from benchmarks.tasks_v2 import TASKS_V2
ROOT = Path(__file__).resolve().parents[1]
DEFAULT_INPUT_DIR = ROOT / "benchmarks" / "results" / "published"
DEFAULT_OUTPUT = ROOT / "docs" / "assets" / "autocad-mcp-taskmatrix.svg"
STATUS_COLORS = {
"pass": "#2da44e",
"partial": "#d4a72c",
"fail": "#cf222e",
"timeout": "#a40e26",
"unsupported": "#d0d7de",
"not_run": "#f6f8fa",
}
STATUS_GLYPHS = {
"pass": "P",
"partial": "~",
"fail": "F",
"timeout": "T",
"unsupported": "-",
"not_run": "?",
}
def load_reports(input_dir: Path) -> list[dict[str, Any]]:
reports: list[dict[str, Any]] = []
for path in sorted(input_dir.glob("*.json")):
data = json.loads(path.read_text(encoding="utf-8"))
if data.get("schema_version") != "2.0":
continue # perf reports and other lanes live in the same folder
reports.append(data)
if not reports:
raise SystemExit(f"no task-matrix reports found in {input_dir}")
# our server first, then alphabetical
reports.sort(key=lambda item: (item["adapter"] != "autocad-mcp-pro", item["adapter"]))
return reports
def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
try:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
from matplotlib.patches import Patch, Rectangle
except ImportError as exc:
raise SystemExit(
'Matplotlib is required. Install it with: pip install -e ".[pdf]"'
) from exc
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-taskmatrix-v1"
task_ids = [task.task_id for task in TASKS_V2]
task_labels = [f"{task.task_id} ({task.category})" for task in TASKS_V2]
servers = [report["adapter"] for report in reports]
status_by_server = {
report["adapter"]: {item["task_id"]: item["status"] for item in report["results"]}
for report in reports
}
fig, ax = plt.subplots(
figsize=(3.4 + 2.4 * len(servers), 1.8 + 0.52 * len(task_ids)),
constrained_layout=True,
)
fig.patch.set_facecolor("#f6f8fa")
ax.set_facecolor("#f6f8fa")
for row, task_id in enumerate(task_ids):
for col, server in enumerate(servers):
status = status_by_server[server].get(task_id, "not_run")
color = STATUS_COLORS.get(status, "#f6f8fa")
ax.add_patch(
Rectangle(
(col + 0.06, row + 0.08),
0.88,
0.84,
facecolor=color,
edgecolor="#d0d7de",
linewidth=0.8,
)
)
glyph_color = "#ffffff" if status in ("pass", "fail", "timeout") else "#24292f"
ax.text(
col + 0.5,
row + 0.5,
STATUS_GLYPHS.get(status, "?"),
ha="center",
va="center",
fontsize=11,
fontweight="bold",
color=glyph_color,
)
ax.set_xlim(0, len(servers))
ax.set_ylim(len(task_ids), 0)
ax.set_xticks([index + 0.5 for index in range(len(servers))])
ax.set_xticklabels(servers, color="#24292f", fontsize=10, fontweight="bold")
ax.xaxis.set_ticks_position("top")
ax.set_yticks([index + 0.5 for index in range(len(task_ids))])
ax.set_yticklabels(task_labels, color="#24292f", fontsize=9)
ax.tick_params(length=0)
for spine in ax.spines.values():
spine.set_visible(False)
ax.set_title(
"Task matrix — live headless lane",
loc="left",
fontsize=17,
fontweight="bold",
color="#24292f",
pad=34,
)
legend_handles = [
Patch(facecolor=STATUS_COLORS["pass"], label="pass"),
Patch(facecolor=STATUS_COLORS["partial"], label="partial"),
Patch(facecolor=STATUS_COLORS["fail"], label="fail"),
Patch(facecolor=STATUS_COLORS["unsupported"], label="unsupported (no equivalent)"),
]
ax.legend(
handles=legend_handles,
loc="upper left",
bbox_to_anchor=(0, -0.02),
ncol=4,
frameon=False,
fontsize=9,
)
output.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(
output,
format="svg",
facecolor=fig.get_facecolor(),
metadata={"Creator": "AutoCAD MCP Pro task-matrix renderer", "Date": None},
)
plt.close(fig)
svg = output.read_text(encoding="utf-8")
normalized = "\n".join(line.rstrip() for line in svg.splitlines()) + "\n"
output.write_text(normalized, encoding="utf-8", newline="\n")
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--input-dir", type=Path, default=DEFAULT_INPUT_DIR)
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
return parser
def main() -> int:
args = build_parser().parse_args()
render_chart(load_reports(args.input_dir), args.output)
print(f"Rendered task-matrix chart: {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
benchmarks/render_perf_chart.py
+146
View File
@@ -0,0 +1,146 @@
"""Render the headless performance suite as a deterministic SVG.
Input: the report produced by ``benchmarks.perf_suite --out ...``
(default: ``benchmarks/results/published/perf-ezdxf.json``).
Output: a GitHub-readable log-scale wall-time chart per workload.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
from typing import Any
ROOT = Path(__file__).resolve().parents[1]
DEFAULT_INPUT = ROOT / "benchmarks" / "results" / "published" / "perf-ezdxf.json"
DEFAULT_OUTPUT = ROOT / "docs" / "assets" / "autocad-mcp-perf.svg"
_LABELS = {
"create_lines_2k": "Create 2,000 lines (individual calls)",
"roundtrip_10k": "10,000 lines: build + DXF export + reopen",
"region_query_10k": "Region query over 10,000 entities",
"premium_pass": "Premium pass: layers + part + dims + full critique",
}
def load_report(path: Path) -> dict[str, Any]:
data = json.loads(path.read_text(encoding="utf-8"))
if data.get("schema_version") != "perf/1.0":
raise ValueError("unsupported perf report schema_version")
if not data.get("workloads"):
raise ValueError("perf report contains no workloads")
return data
def _annotation(item: dict[str, Any]) -> str:
wall_s = item["wall_ms"] / 1000.0
if item["metric"] == "entities_per_second":
return f"{wall_s:.2f} s · {item['metric_value']:,.0f} entities/s"
if item["metric"] == "entities_matched":
return f"{wall_s:.2f} s · {item['metric_value']:,.0f} matched"
if item["metric"] == "critique_issues":
return f"{wall_s:.2f} s · {item['metric_value']:.0f} issues"
return f"{wall_s:.2f} s"
def render_chart(data: dict[str, Any], output: Path) -> None:
try:
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
except ImportError as exc:
raise SystemExit(
'Matplotlib is required. Install it with: pip install -e ".[pdf]"'
) from exc
matplotlib.rcParams["svg.hashsalt"] = "autocad-mcp-perf-v1"
workloads = sorted(data["workloads"], key=lambda item: item["wall_ms"])
labels = [_LABELS.get(item["workload_id"], item["workload_id"]) for item in workloads]
values = [item["wall_ms"] for item in workloads]
fig, ax = plt.subplots(figsize=(12, 4.6), constrained_layout=True)
fig.patch.set_facecolor("#f6f8fa")
ax.set_facecolor("#f6f8fa")
bars = ax.barh(labels, values, color="#0969da", height=0.58)
ax.set_xscale("log")
ax.set_xlabel("Wall time in ms (log scale) — lower is better", color="#24292f", labelpad=10)
ax.set_title(
"Headless performance — ezdxf engine",
loc="left",
fontsize=18,
fontweight="bold",
color="#24292f",
pad=28,
)
env = data.get("environment", {})
ax.text(
0,
1.02,
f"Fixed workloads via the same backend methods the MCP tools call · "
f"Python {env.get('python', '?')} · {env.get('platform', '?')}",
transform=ax.transAxes,
color="#57606a",
fontsize=9,
)
for bar, item in zip(bars, workloads, strict=True):
ax.text(
bar.get_width() * 1.12,
bar.get_y() + bar.get_height() / 2,
_annotation(item),
va="center",
color="#24292f",
fontweight="bold",
fontsize=9,
)
ax.set_xlim(right=max(values) * 8)
ax.text(
0,
-0.24,
"Self-measurement only: competitor servers would pay extra stdio serialization cost in-process runs do not. "
"Reproduce: python -m benchmarks.perf_suite",
transform=ax.transAxes,
color="#57606a",
fontsize=9,
)
ax.grid(axis="x", color="#d0d7de", linewidth=0.8, alpha=0.8)
ax.set_axisbelow(True)
ax.tick_params(axis="x", colors="#57606a")
ax.tick_params(axis="y", colors="#24292f", length=0, pad=8)
for spine in ax.spines.values():
spine.set_visible(False)
output.parent.mkdir(parents=True, exist_ok=True)
fig.savefig(
output,
format="svg",
facecolor=fig.get_facecolor(),
metadata={"Creator": "AutoCAD MCP Pro perf renderer", "Date": None},
)
plt.close(fig)
svg = output.read_text(encoding="utf-8")
normalized = "\n".join(line.rstrip() for line in svg.splitlines()) + "\n"
output.write_text(normalized, encoding="utf-8", newline="\n")
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--input", type=Path, default=DEFAULT_INPUT)
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
return parser
def main() -> int:
args = build_parser().parse_args()
render_chart(load_report(args.input), args.output)
print(f"Rendered perf chart: {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
benchmarks/results/published/perf-ezdxf.json
+56
View File
@@ -0,0 +1,56 @@
{
"backend": "ezdxf",
"environment": {
"implementation": "CPython",
"machine": "AMD64",
"platform": "Windows-11-10.0.26200-SP0",
"python": "3.14.0"
},
"schema_version": "perf/1.0",
"workloads": [
{
"description": "Create 2,000 lines through individual backend calls",
"detail": {
"entity_count": 2000
},
"metric": "entities_per_second",
"metric_value": 3148.0,
"wall_ms": 635.3,
"workload_id": "create_lines_2k"
},
{
"description": "Create 10,000 lines, export DXF, reopen and verify",
"detail": {
"build_ms": 10884.9,
"entity_count": 10000,
"export_ms": 1439.2,
"file_bytes": 1369662,
"reopen_ms": 1607.5
},
"metric": "entities_per_second",
"metric_value": 717.8,
"wall_ms": 13931.6,
"workload_id": "roundtrip_10k"
},
{
"description": "Crossing-selection region query over 10,000 entities",
"detail": {
"entity_count": 10000
},
"metric": "entities_matched",
"metric_value": 2500.0,
"wall_ms": 3434.2,
"workload_id": "region_query_10k"
},
{
"description": "Plate with hole: ISO layers, geometry, dimensions and full critique",
"detail": {
"entities": 7
},
"metric": "critique_issues",
"metric_value": 0.0,
"wall_ms": 315.9,
"workload_id": "premium_pass"
}
]
}