feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results

Browse Source

- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf)
- competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor
- adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons
- run_competitors.py: registry-driven --server from competitors.yaml
- published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg
- benchmark.yml: weekly + manual dispatch matrix

Co-authored-by: Cursor <cursoragent@cursor.com>

...

This commit is contained in:

Umutcan Edizsalan

2026-07-23 15:15:27 +03:00

co-authored by Cursor

parent 2f56a73093

commit 6d7c39e012

13 changed files with 4082 additions and 16 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

docs/assets/autocad-mcp-livebench.svg

+2213

File diff suppressed because it is too large Load Diff

Side by Side After Width:  |  Height:  |  Size: 69 KiB