docs(bench): add the v1.3.0 -> v1.4.0 correctness A/B gate to the README benchmark lanes

Ran benchmarks/compare_versions.py against the v1.3.0 tag: 21/21 on both versions, 0 fixed / 0 regressed - the v1.4.0 feature+infra work introduced no correctness regressions. Main README gains lane 4 (version A/B, with the historical v1.0.0 -> v1.1.0 contrast); benchmarks/README records the release-gate result and drops the stale 'performance coming' stub.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Umutcan Edizsalan
2026-07-23 16:00:58 +03:00
co-authored by Cursor
parent d3e5fe4542
commit 47a90dbc54
3 changed files with 185 additions and 6 deletions
+29 -1
View File
@@ -101,7 +101,35 @@ python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.js
python -m benchmarks.render_perf_chart
```
### 4 · Source-reviewed capability rubric (context)
### 4 · Version A/B — every release re-proves correctness
`compare_versions.py` runs the same 21 deterministic, headless correctness
checks against the previous release tag and the current tree — each check in
its own subprocess, so a hard crash counts as a miss instead of killing the
run.
**v1.4.0 release gate** (baseline `v1.3.0`, report:
[`ab-v1.3.0-vs-v1.4.0.json`](benchmarks/results/published/ab-v1.3.0-vs-v1.4.0.json)):
| Version | Checks passing | Pass rate | Fixed | Regressed |
|---|---:|---:|---:|---:|
| v1.3.0 (baseline) | 21 / 21 | 100 % | — | — |
| **v1.4.0** (this release) | **21 / 21** | **100 %** | 0 | **0** |
v1.4.0 added CI, packaging, benchmarks, tool profiles, paper space, ISO 286
fits and opt-in 3D solids **without breaking a single correctness check**.
For contrast, the same suite caught the v1.0.0 → v1.1.0 jump:
| Version | Checks passing | Pass rate |
|---|---:|---:|
| v1.0.0 (first public release) | 8 / 21 | 38.1 % |
| v1.1.0 | 21 / 21 | 100 % |
```bash
python benchmarks/compare_versions.py v1.3.0 --json ab.json
```
### 5 · Source-reviewed capability rubric (context)
![Source-reviewed AutoCAD MCP capability benchmark](docs/assets/autocad-mcp-benchmark.svg)