anatoly/autocad-mcp-pro
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes

Browse Source 
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only
- benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time)
- benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports)
- render_live_chart now tolerates other lanes sharing results/published/
- README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap
- published perf report: benchmarks/results/published/perf-ezdxf.json

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Umutcan Edizsalan
2026-07-23 15:53:04 +03:00
co-authored by Cursor
parent 67f2c3b575
commit 21cbc62df5
11 changed files with 8047 additions and 231 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/README.md
+20 -2
View File
@@ -78,13 +78,31 @@ Method and boundaries:
require a local AutoCAD session and are out of CI scope by design.
Published machine-readable reports (artifact paths sanitized to filenames)
live under [`results/published/`](results/published/); the README chart is
live under [`results/published/`](results/published/); the README charts are
regenerated from them:
```bash
python -m benchmarks.run_competitors --server puran-water-autocad-mcp --backend ezdxf
python -m benchmarks.run_competitors --server beiming183-autocad-mcp --backend ezdxf
python -m benchmarks.render_live_chart
python -m benchmarks.render_live_chart # score bars
python -m benchmarks.render_matrix_chart # tasks x servers status heatmap
```
## Headless performance lane
`perf_suite.py` measures wall time on fixed workloads through the same backend
methods the MCP tools call (server-side overhead included): 2,000 individual
line creates, a 10,000-entity build → DXF export → reopen roundtrip, a region
query over 10,000 entities, and a full premium quality pass (ISO layers,
geometry, dimensions, complete critique). The report records the machine
fingerprint; numbers move with hardware, workload definitions do not.
**Self-measurement only** — competitor servers would pay an extra stdio
serialization cost that in-process runs do not, so no cross-server timing
claims are made.
```bash
python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.json
python -m benchmarks.render_perf_chart
```
## Correctness A/B suite