Umutcan Edizsalan and Cursor
21cbc62df5
feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes
...
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only
- benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time)
- benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports)
- render_live_chart now tolerates other lanes sharing results/published/
- README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap
- published perf report: benchmarks/results/published/perf-ezdxf.json
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-23 15:53:04 +03:00
Umutcan Edizsalan and Cursor
6d7c39e012
feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results
...
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf)
- competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor
- adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons
- run_competitors.py: registry-driven --server from competitors.yaml
- published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg
- benchmark.yml: weekly + manual dispatch matrix
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-23 15:15:27 +03:00
Umutcan Edizsalan and Cursor
9ec6c4a989
style: apply ruff format across the repository
...
ruff format was never run repo-wide; CI now enforces 'ruff format --check', so normalize every module once. No behavior changes.
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-23 15:14:51 +03:00
Umutcan Edizsalan
4a2e9d658f
release: prepare v1.3.0
2026-07-17 15:48:43 +03:00
Umutcan Edizsalan
1d47408157
fix: align benchmark docs with verification
2026-07-17 15:16:56 +03:00
Umutcan Edizsalan
b092c1a539
docs: explain benchmark evidence boundaries
2026-07-17 15:08:46 +03:00
Umutcan Edizsalan
426701346b
feat: add reproducible benchmark visualization
2026-07-17 15:08:15 +03:00
Umutcan Edizsalan
5583730a79
docs(benchmarks): reproducible v1.0.0 -> v1.1.0 correctness A/B suite
...
- benchmarks/correctness_suite.py: 21 deterministic headless (ezdxf) checks
across core ops, dimensions, arrays, geometry, query/parity, render, the
quality gate, and engineering.
- benchmarks/compare_versions.py: one-command A/B runner — checks out an older
ref into a throwaway worktree and runs the same suite against both, each check
in its own subprocess so a crash is a miss, not a run-killer.
- benchmarks/README.md: methodology, results table, honesty caveats.
- README: Benchmarks section now reports the measured result —
v1.0.0 8/21 (38.1%) -> v1.1.0 21/21 (100%); 13 fixed, 0 regressions.
2026-06-19 12:31:03 +03:00
