docs(benchmarks): reproducible v1.0.0 -> v1.1.0 correctness A/B suite

- benchmarks/correctness_suite.py: 21 deterministic headless (ezdxf) checks
  across core ops, dimensions, arrays, geometry, query/parity, render, the
  quality gate, and engineering.
- benchmarks/compare_versions.py: one-command A/B runner — checks out an older
  ref into a throwaway worktree and runs the same suite against both, each check
  in its own subprocess so a crash is a miss, not a run-killer.
- benchmarks/README.md: methodology, results table, honesty caveats.
- README: Benchmarks section now reports the measured result —
  v1.0.0 8/21 (38.1%) -> v1.1.0 21/21 (100%); 13 fixed, 0 regressions.
This commit is contained in:
Umutcan Edizsalan
2026-06-19 12:31:03 +03:00
parent b797d4c9f1
commit 5583730a79
4 changed files with 441 additions and 1 deletions
+27 -1
View File
@@ -299,7 +299,33 @@ If you find a security issue, please open a private contact rather than a public
## Benchmarks
A reproducible benchmark suite is in active development. Targets:
### Correctness — v1.1.0 vs v1.0.0
A reproducible A/B suite ([`benchmarks/`](benchmarks/)) runs the **same** 21
deterministic, headless (ezdxf) checks against the public v1.0.0 release
(`origin/main`) and v1.1.0 — each check in its own subprocess, so a hard crash is
recorded as a miss instead of taking down the run.
| Version | Checks passing | Pass rate |
|---------|----------------|-----------|
| **v1.0.0** (public release) | 8 / 21 | **38.1 %** |
| **v1.1.0** (this release) | 21 / 21 | **100 %** |
**13 defects fixed · 0 regressions · 2.6× higher correctness pass-rate (+61.9 pts).**
Fixes span dimensions (aligned/angular no longer raise), full-circle arrays, deterministic
geometry (`point_intersection`/`point_tangent`), ARC selection, property parity, MTEXT
rotation, the headless screenshot crash, the now-live `dim_overlap` critique, ISO-13567
layer routing, and gear-outline geometry. Six core operations pass on **both** versions, so
the suite is not a cherry-picked failure list — see [`benchmarks/README.md`](benchmarks/README.md)
for the full table, methodology, and honesty caveats.
```bash
python benchmarks/compare_versions.py # reproduce: current tree vs origin/main
```
### Performance (throughput) — coming
A scale/throughput suite is in active development. Targets:
- **Drawing scale** — 1k / 10k / 100k entity workloads
- **Workload classes** — bulk creation, mass-modify, region select, render-and-screenshot, end-to-end design pass