anatoly/autocad-mcp-pro
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
36 Commits 2 Branches 5 Tags
47a90dbc540af4e82df05a74ea1e2925947960ac
Commit Graph
6 Commits
Author SHA1 Message Date
Umutcan EdizsalanandCursor 47a90dbc54 docs(bench): add the v1.3.0 -> v1.4.0 correctness A/B gate to the README benchmark lanes 
Ran benchmarks/compare_versions.py against the v1.3.0 tag: 21/21 on both versions, 0 fixed / 0 regressed - the v1.4.0 feature+infra work introduced no correctness regressions. Main README gains lane 4 (version A/B, with the historical v1.0.0 -> v1.1.0 contrast); benchmarks/README records the release-gate result and drops the stale 'performance coming' stub.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-23 16:00:58 +03:00
Umutcan EdizsalanandCursor 21cbc62df5 feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes 
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only
- benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time)
- benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports)
- render_live_chart now tolerates other lanes sharing results/published/
- README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap
- published perf report: benchmarks/results/published/perf-ezdxf.json

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-23 15:53:04 +03:00
Umutcan EdizsalanandCursor 6d7c39e012 feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results 
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf)
- competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor
- adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons
- run_competitors.py: registry-driven --server from competitors.yaml
- published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg
- benchmark.yml: weekly + manual dispatch matrix

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-07-23 15:15:27 +03:00
Umutcan Edizsalan 4a2e9d658f release: prepare v1.3.0 2026-07-17 15:48:43 +03:00
Umutcan Edizsalan b092c1a539 docs: explain benchmark evidence boundaries 2026-07-17 15:08:46 +03:00
Umutcan Edizsalan 5583730a79 docs(benchmarks): reproducible v1.0.0 -> v1.1.0 correctness A/B suite 
- benchmarks/correctness_suite.py: 21 deterministic headless (ezdxf) checks
  across core ops, dimensions, arrays, geometry, query/parity, render, the
  quality gate, and engineering.
- benchmarks/compare_versions.py: one-command A/B runner — checks out an older
  ref into a throwaway worktree and runs the same suite against both, each check
  in its own subprocess so a crash is a miss, not a run-killer.
- benchmarks/README.md: methodology, results table, honesty caveats.
- README: Benchmarks section now reports the measured result —
  v1.0.0 8/21 (38.1%) -> v1.1.0 21/21 (100%); 13 fixed, 0 regressions.
 2026-06-19 12:31:03 +03:00