Umutcan EdizsalanandCursor
6d7c39e012
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf) - competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor - adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons - run_competitors.py: registry-driven --server from competitors.yaml - published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg - benchmark.yml: weekly + manual dispatch matrix Co-authored-by: Cursor <cursoragent@cursor.com>
27 lines
995 B
YAML
27 lines
995 B
YAML
{
|
|
"schema_version": "2.0",
|
|
"competitors": [
|
|
{
|
|
"id": "autocad-mcp-pro",
|
|
"adapter": "benchmarks.adapters.autocad_mcp_pro:AutoCADMCPProAdapter",
|
|
"backends": ["ezdxf", "com"]
|
|
},
|
|
{
|
|
"id": "puran-water-autocad-mcp",
|
|
"adapter": "benchmarks.adapters.puran_water:PuranWaterAdapter",
|
|
"backends": ["ezdxf"],
|
|
"repo": "https://github.com/puran-water/autocad-mcp",
|
|
"pinned_sha": "95476a33a1c246308326eb4709d6379ef2efdbc1",
|
|
"notes": "Black-box MCP stdio lane; File-IPC (live AutoCAD) features run in the local COM lane only."
|
|
},
|
|
{
|
|
"id": "beiming183-autocad-mcp",
|
|
"adapter": "benchmarks.adapters.beiming_autocad:BeimingAutocadAdapter",
|
|
"backends": ["ezdxf"],
|
|
"repo": "https://github.com/beiming183-cloud/AutoCAD-MCP",
|
|
"pinned_sha": "11f7c47e5038796a20451b38b23032e625b5aa26",
|
|
"notes": "Black-box MCP stdio lane; deliver/undo/redo are documented File-IPC only and reported unsupported."
|
|
}
|
|
]
|
|
}
|