feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf) - competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor - adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons - run_competitors.py: registry-driven --server from competitors.yaml - published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg - benchmark.yml: weekly + manual dispatch matrix Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
co-authored by
Cursor
parent
2f56a73093
commit
6d7c39e012
@@ -0,0 +1,43 @@
|
||||
name: Benchmark
|
||||
|
||||
# Headless competitor benchmark (v2 fixed-task matrix). Manual or weekly:
|
||||
# clones the pinned competitor SHAs, runs the black-box MCP stdio lane and
|
||||
# uploads the machine-readable reports as artifacts. Results committed to
|
||||
# benchmarks/results/ are produced from these runs.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "17 5 * * 1"
|
||||
|
||||
jobs:
|
||||
headless-matrix:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
server:
|
||||
- autocad-mcp-pro
|
||||
- puran-water-autocad-mcp
|
||||
- beiming183-autocad-mcp
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
- name: Install
|
||||
run: pip install -e ".[pdf]"
|
||||
- name: Run benchmark (ezdxf lane)
|
||||
run: |
|
||||
python -m benchmarks.run_competitors \
|
||||
--server "${{ matrix.server }}" \
|
||||
--backend ezdxf \
|
||||
--artifact-dir "benchmarks/results/ci/${{ matrix.server }}" \
|
||||
> "report-${{ matrix.server }}.json"
|
||||
- name: Upload report
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: benchmark-${{ matrix.server }}
|
||||
path: |
|
||||
report-${{ matrix.server }}.json
|
||||
benchmarks/results/ci/${{ matrix.server }}
|
||||
Reference in New Issue
Block a user