Files

T

Umutcan Edizsalan 5583730a79 docs(benchmarks): reproducible v1.0.0 -> v1.1.0 correctness A/B suite

- benchmarks/correctness_suite.py: 21 deterministic headless (ezdxf) checks
  across core ops, dimensions, arrays, geometry, query/parity, render, the
  quality gate, and engineering.
- benchmarks/compare_versions.py: one-command A/B runner — checks out an older
  ref into a throwaway worktree and runs the same suite against both, each check
  in its own subprocess so a crash is a miss, not a run-killer.
- benchmarks/README.md: methodology, results table, honesty caveats.
- README: Benchmarks section now reports the measured result —
  v1.0.0 8/21 (38.1%) -> v1.1.0 21/21 (100%); 13 fixed, 0 regressions.

2026-06-19 12:31:03 +03:00

3.4 KiB

Raw Blame History

Benchmarks

Correctness A/B suite

correctness_suite.py is a set of deterministic, headless (ezdxf-backend) checks that exercise real drawing operations. compare_versions.py runs the same suite against an older git ref and the current checkout, each check in its own subprocess so a hard crash (e.g. a matplotlib-in-thread SIGSEGV) is recorded as a miss rather than taking down the run.

Reproduce:

python benchmarks/compare_versions.py            # current tree vs origin/main
python benchmarks/compare_versions.py v1.0.0     # vs a tag/ref
python benchmarks/compare_versions.py --json results.json

Result — v1.1.0 vs v1.0.0 (`origin/main`, commit 15fa2bc)

21 checks, ezdxf backend, CPython 3.14, one subprocess per check.

Version	Checks passing	Pass rate
v1.0.0 (public release)	8 / 21	38.1 %
v1.1.0 (this release)	21 / 21	100 %

13 defects fixed, 0 regressions — a 2.6× higher correctness pass-rate (+61.9 pts).

Fixed (failed on v1.0.0, pass on v1.1.0):

Check	Category	What it proves
`dim_aligned_no_error`	Dimensions	aligned dim no longer raises `TypeError`
`dim_angular_no_error`	Dimensions	angular dim no longer raises `TypeError`
`array_polar_360_distinct`	Modify	full-circle array no longer duplicates the original
`point_intersection_line_line`	Geometry	deterministic line/line intersection exists & is correct
`point_tangent_external`	Geometry	external-point tangent is perpendicular & on-circle
`arc_has_length`	Query	ARC carries a `length` property
`arc_select_by_length`	Query	`entity_select_smart` length_range selects arcs
`ezdxf_bounding_box`	Query	`bounding_box` populated (COM/ezdxf parity)
`mtext_rotation_roundtrip`	Entities	MTEXT honors caller rotation
`screenshot_png`	Render	headless render returns a valid PNG (no GUI-thread crash)
`dim_overlap_critique_fires`	Quality gate	`dim_overlap` critique is live (was a no-op)
`iso13567_dim_layer`	Quality gate	`dimension_auto` lands dims on the active set's layer
`gear_no_self_overlap`	Engineering	gear outline never dips inside the root circle (z ≥ 42)

Passing on both versions (6 core ops + 2 others) keep the suite honest — it is not a cherry-picked list of failures; the baseline genuinely does basic CAD work.

Caveats / honesty notes

The baseline is origin/main (the public v1.0.0). All four audited fix sprints land between it and v1.1.0 — see docs/analysis/.
screenshot_png on the baseline is host-dependent: it is recorded as a miss here because the default matplotlib backend is a GUI backend and the render runs off the main thread. On a host where matplotlib defaults to Agg, the baseline may not crash. v1.1.0 forces Agg and is deterministic.
center_linetype_applied passes on both because it only checks the linetype attribute string (set on both); the v1.0.0 "renders as Continuous because the linetype was never loaded" defect is not observable through EntityInfo.
COM-backend behaviour is not covered here (no live AutoCAD); see the mocked-COM suites in tests/.

Performance (throughput) — coming

Scale / throughput numbers (1k–100k entities, render timings, model-by-model tool-call success rate) are tracked in the main README and will be published here with a hardware footprint.

3.4 KiB Raw Blame History Unescape Escape