Umutcan Edizsalan and Cursor
bae0750d4c
docs(bench): dark leaderboard styling for all benchmark charts
...
Leaderboard restyled to the LLM-benchmark look: dark background, rank-numbered labels, per-rank color palette, 'NN / 100 - grade' annotations. The live-run, task-matrix and performance charts share the same dark theme for a consistent README identity. Charts regenerated deterministically from their data files.
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-23 16:14:04 +03:00
Umutcan Edizsalan and Cursor
21cbc62df5
feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes
...
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only
- benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time)
- benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports)
- render_live_chart now tolerates other lanes sharing results/published/
- README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap
- published perf report: benchmarks/results/published/perf-ezdxf.json
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-23 15:53:04 +03:00
Umutcan Edizsalan and Cursor
6d7c39e012
feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results
...
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf)
- competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor
- adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons
- run_competitors.py: registry-driven --server from competitors.yaml
- published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg
- benchmark.yml: weekly + manual dispatch matrix
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-07-23 15:15:27 +03:00
