docs: explain benchmark evidence boundaries

This commit is contained in:
Umutcan Edizsalan
2026-07-17 15:08:46 +03:00
parent 426701346b
commit b092c1a539
+43
View File
@@ -1,5 +1,48 @@
# Benchmarks # Benchmarks
## Two evidence layers
This repository publishes two deliberately separate forms of evidence:
1. **Source-reviewed capability benchmark.** Six named AutoCAD MCP projects are
scored with a fixed 100-point rubric covering CAD breadth, correctness and
delivery, backend reach, engineering production, tests, and security. The
dated data lives in [`source_review.json`](source_review.json), and the README
graphic is generated—not hand-edited—from that file.
2. **Fixed-task runtime benchmark.** Adapters execute the same ten tasks and
return `pass`, `partial`, `unsupported`, `fail`, `timeout`, or `not_run`.
AutoCAD MCP Pro currently has the reference adapter; other projects do not
receive runtime scores until an adapter actually runs their public interface.
Regenerate the source-review chart:
```bash
pip install -e ".[pdf]"
python -m benchmarks.render_chart
```
The source-review chart is not presented as a shared live AutoCAD run. Review
dates, evidence grades, project URLs, and the boundary are stored with the data.
## Fixed-task runtime runner v2
The v2 runner separates runtime evidence from the source-review rubric. Every
adapter receives the same ten tasks and produces the same closed result enum:
`pass`, `partial`, `unsupported`, `fail`, `timeout`, or `not_run`. A timeout is
isolated to its task. Reports include commit SHA, Python/platform details,
backend capability claims, durations, and hashes for returned artifacts.
```bash
python -m benchmarks.run_competitors --list
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --json
python -m benchmarks.run_competitors --task table_mleader --task auditable_delivery --json
```
The release-machine v1.3 ezdxf self-check is **10/10 (100.0)**. It is not a
competitor ranking. Cross-project scores become comparable only when another
adapter runs this exact task matrix; repository stars and raw tool counts do not
contribute to the score. Adapter registration lives in `competitors.yaml`.
## Correctness A/B suite ## Correctness A/B suite
`correctness_suite.py` is a set of deterministic, headless (ezdxf-backend) checks `correctness_suite.py` is a set of deterministic, headless (ezdxf-backend) checks