Umutcan EdizsalanandCursor
6d7c39e012
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf) - competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor - adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons - run_competitors.py: registry-driven --server from competitors.yaml - published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg - benchmark.yml: weekly + manual dispatch matrix Co-authored-by: Cursor <cursoragent@cursor.com>
44 lines
1.3 KiB
YAML
44 lines
1.3 KiB
YAML
name: Benchmark
|
|
|
|
# Headless competitor benchmark (v2 fixed-task matrix). Manual or weekly:
|
|
# clones the pinned competitor SHAs, runs the black-box MCP stdio lane and
|
|
# uploads the machine-readable reports as artifacts. Results committed to
|
|
# benchmarks/results/ are produced from these runs.
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
schedule:
|
|
- cron: "17 5 * * 1"
|
|
|
|
jobs:
|
|
headless-matrix:
|
|
runs-on: ubuntu-latest
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
server:
|
|
- autocad-mcp-pro
|
|
- puran-water-autocad-mcp
|
|
- beiming183-autocad-mcp
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: actions/setup-python@v5
|
|
with:
|
|
python-version: "3.12"
|
|
- name: Install
|
|
run: pip install -e ".[pdf]"
|
|
- name: Run benchmark (ezdxf lane)
|
|
run: |
|
|
python -m benchmarks.run_competitors \
|
|
--server "${{ matrix.server }}" \
|
|
--backend ezdxf \
|
|
--artifact-dir "benchmarks/results/ci/${{ matrix.server }}" \
|
|
> "report-${{ matrix.server }}.json"
|
|
- name: Upload report
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: benchmark-${{ matrix.server }}
|
|
path: |
|
|
report-${{ matrix.server }}.json
|
|
benchmarks/results/ci/${{ matrix.server }}
|