feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf) - competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor - adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons - run_competitors.py: registry-driven --server from competitors.yaml - published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg - benchmark.yml: weekly + manual dispatch matrix Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
co-authored by
Cursor
parent
2f56a73093
commit
6d7c39e012
@@ -5,6 +5,22 @@
|
||||
"id": "autocad-mcp-pro",
|
||||
"adapter": "benchmarks.adapters.autocad_mcp_pro:AutoCADMCPProAdapter",
|
||||
"backends": ["ezdxf", "com"]
|
||||
},
|
||||
{
|
||||
"id": "puran-water-autocad-mcp",
|
||||
"adapter": "benchmarks.adapters.puran_water:PuranWaterAdapter",
|
||||
"backends": ["ezdxf"],
|
||||
"repo": "https://github.com/puran-water/autocad-mcp",
|
||||
"pinned_sha": "95476a33a1c246308326eb4709d6379ef2efdbc1",
|
||||
"notes": "Black-box MCP stdio lane; File-IPC (live AutoCAD) features run in the local COM lane only."
|
||||
},
|
||||
{
|
||||
"id": "beiming183-autocad-mcp",
|
||||
"adapter": "benchmarks.adapters.beiming_autocad:BeimingAutocadAdapter",
|
||||
"backends": ["ezdxf"],
|
||||
"repo": "https://github.com/beiming183-cloud/AutoCAD-MCP",
|
||||
"pinned_sha": "11f7c47e5038796a20451b38b23032e625b5aa26",
|
||||
"notes": "Black-box MCP stdio lane; deliver/undo/redo are documented File-IPC only and reported unsupported."
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user