Commit Graph
7 Commits
Author SHA1 Message Date
Umutcan EdizsalanandCursor 6d7c39e012 feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf)
- competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor
- adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons
- run_competitors.py: registry-driven --server from competitors.yaml
- published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg
- benchmark.yml: weekly + manual dispatch matrix

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-23 15:15:27 +03:00
Umutcan EdizsalanandCursor 9ec6c4a989 style: apply ruff format across the repository
ruff format was never run repo-wide; CI now enforces 'ruff format --check', so normalize every module once. No behavior changes.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-23 15:14:51 +03:00
Umutcan Edizsalan 4a2e9d658f release: prepare v1.3.0 2026-07-17 15:48:43 +03:00
Umutcan Edizsalan 1d47408157 fix: align benchmark docs with verification 2026-07-17 15:16:56 +03:00
Umutcan Edizsalan b092c1a539 docs: explain benchmark evidence boundaries 2026-07-17 15:08:46 +03:00
Umutcan Edizsalan 426701346b feat: add reproducible benchmark visualization 2026-07-17 15:08:15 +03:00
Umutcan Edizsalan 5583730a79 docs(benchmarks): reproducible v1.0.0 -> v1.1.0 correctness A/B suite
- benchmarks/correctness_suite.py: 21 deterministic headless (ezdxf) checks
  across core ops, dimensions, arrays, geometry, query/parity, render, the
  quality gate, and engineering.
- benchmarks/compare_versions.py: one-command A/B runner — checks out an older
  ref into a throwaway worktree and runs the same suite against both, each check
  in its own subprocess so a crash is a miss, not a run-killer.
- benchmarks/README.md: methodology, results table, honesty caveats.
- README: Benchmarks section now reports the measured result —
  v1.0.0 8/21 (38.1%) -> v1.1.0 21/21 (100%); 13 fixed, 0 regressions.
2026-06-19 12:31:03 +03:00