diff --git a/CLAUDE.md b/CLAUDE.md index c0923a8..b2d688e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,7 +101,7 @@ Tools are organized into sections (counts are indicative — `system_about` is a **Measuring:** never read vertices back and shoelace them — that loses 28.2% of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry, and its payload states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. A HATCH reports the area it *fills* (outer loops minus islands) plus `hatch_style`; on a curved hatch edge `flatten_tolerance` is not the accuracy knob, because ezdxf hands boundaries over as cubic Beziers whose ~0.028% circle error is already baked in. `boundary_trace` / `boundary_from_entities` report the same number `analysis_measure_entity` gives for the polyline they just drew — if those ever diverge, one of them is approximating and neither says so. -**Benchmark matrix:** `benchmarks/tasks_v3.py` is the current set (15 tasks; `--matrix v2` reproduces the v1.4 ten). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (24 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.4.0`. +**Benchmark matrix:** `benchmarks/tasks_v3.py` is the current set (15 tasks; `--matrix v2` reproduces the v1.4 ten). The published competitor reports are pinned v1.4 runs against v2 and are **never** back-filled for the five v3 tasks — the chart shows them `not run`, not zero. `benchmarks/correctness_suite.py` (26 checks) is the release A/B gate: `python benchmarks/compare_versions.py v1.4.0`. ### Adding a New Tool diff --git a/README.md b/README.md index 5c407b6..8d22849 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ AUTOCAD_MCP_BACKEND=com autocad-mcp # live AutoCAD (pip install "autocad-mc | *(none)* | `fastmcp`, `ezdxf`, `pydantic` | Headless DXF on any OS — **no rendering** | | `[com]` | `pywin32`, `Pillow` | Live AutoCAD control + window capture | | `[pdf]` | `matplotlib` | PDF export and headless PNG rendering, any platform | -| `[full]` | everything above | Development and CI | +| `[full]` | everything above | Live AutoCAD and rendering in one install | > [!NOTE] > The bare install **cannot draw pixels**. `ezdxf.addons.drawing.frontend` does an unconditional top-level `import PIL.Image`, so Pillow is required by *every* ezdxf render path — PNG screenshot and PDF export alike — not just by the COM window capture it is listed under. Add `[pdf]` (or `[full]`) if you want images on Linux or macOS. @@ -106,13 +106,13 @@ Works with Claude Desktop, Cursor, or any stdio MCP host. For HTTP: `autocad-mcp Engineering generatorsinvolute gears (front + section A-A), DIN 6885 keyed bores, ISO A3 title block Discovery ✨search_tools ranked over an AutoCAD command and synonym corpus — FILLET, BPOLY, QSELECT, WBLOCK, OVERKILL, CHSPACE each rank #1 of the 149-tool advertised catalog Batching ✨cad_batch runs a whole step list in one round trip; fields= projects the response on 11 result-heavy tools (a closed, gated list) -Paper space ✨tab lifecycle, viewports and drawing_export_pdf(layout=...) on both engines; entity_change_space (CHSPACE) headless only -Selection ✨window vs crossing stated back to the caller; a polygon tested against its own shape, not its bounding box -Boundaries ✨boundary_trace (BOUNDARY/BPOLY) chains loose edges into one closed polyline, arcs kept as bulges — headless engine only +Paper space ✨8 of the 12 layout tools are new — layout_delete/rename/copy, viewport_list/set_scale/lock/delete, entity_change_space (CHSPACE, headless only); layout listing, creation and activation, viewport_create and drawing_export_pdf(layout=...) shipped in v1.4.0, which wrongly declared headless viewport rendering impossible +Selection ✨window vs crossing stated back to the caller; the polygon is tested as a shape, not as its bounding box — the entity side is still its bounding box, on both engines +Boundaries ✨boundary_trace (BOUNDARY/BPOLY) chains loose edges into one closed polyline; ARC edges keep their exact bulges, while a bulged polyline segment or any other curved edge is chained as its chord — headless engine only Measurement ✨analysis_measure_entity measures what is in the drawing, by handle Hatch depth ✨gradients, in-place edits and island styles on both; typed edge boundaries headless only Annotation objects ✨WIPEOUT, REVCLOUD, MTEXT background masks — headless only, ActiveX exposes no member — plus text find/replace on both -3D solidssolid_box/cylinder/extrude/revolve/boolean on live AutoCAD (ENABLE_3D=true) +3D solidssolid_box/cylinder/extrude/revolve/boolean on live AutoCAD (ENABLE_3D=true) — unverified against a live seat: they shipped in v1.4.0, and the v1.5.0 live run covered only what that release added Quality loopdrawing_preflight → drawing_plan → drawing_critique → drawing_refine → drawing_finalize (0–100 score) Deliverydrawing_deliver: DXF/PDF/PNG + SHA-256 manifest + reopen-parity checks @@ -123,7 +123,9 @@ Alongside the tools, **6 MCP resources** cost nothing in the tool budget and are ### Two rules worth knowing before you draw -**Every coordinate in and out of a tool is WCS, on both engines** — with one deliberate exception. DXF stores CIRCLE / ARC / LWPOLYLINE / TEXT / INSERT geometry in a per-entity frame, and before v1.5.0 the word `extrusion` appeared nowhere in this codebase, so a mirrored or rotated entity reported coordinates that were quietly wrong. An entity in a plane tilted out of WCS XY now reports `plane_normal` and omits the fields xy cannot express, rather than answering with a number that looks fine. The exception is **TEXT `rotation`**, which stays in the entity frame on purpose: a mirrored TEXT is mirror-imaged, and no single scalar angle expresses that. +**Coordinates come out in WCS, and the exceptions are declared rather than hidden.** DXF stores CIRCLE / ARC / LWPOLYLINE / TEXT / INSERT geometry in a per-entity frame, and before v1.5.0 the word `extrusion` appeared nowhere in this codebase, so a mirrored or rotated entity reported coordinates that were quietly wrong. An entity in a plane tilted out of WCS XY now reports `plane_normal` and omits the fields xy cannot express, rather than answering with a number that looks fine. + +Three things stay in the entity frame on purpose, and `system_capabilities`' `ocs_normalized` reason string is the live list: **TEXT `rotation`** (a mirrored TEXT is mirror-imaged, and no single scalar angle expresses that), **z components**, and **HATCH boundary geometry**. On the live backend that same reason string also records `2d_polyline_coordinates_not_verified_against_live_autocad` — read it at runtime rather than trusting this paragraph. **Never read vertices back and shoelace them.** That loses **28.2%** of the area on a semicircular edge, silently. `analysis_measure_entity(handle)` reads the real geometry and states its own accuracy: `exact`, `flatten_tolerance`, `assumed_closed`, `self_intersecting`. @@ -137,7 +139,7 @@ There are **27 capability keys**. `system_capabilities` returns the live map, an |---|:---:|:---:| | Live document control | ✅ | — | | Cross-platform, no AutoCAD | — | ✅ | -| Transactions and rollback | ✅ | ✅ | +| Transactions and rollback | ✅ *undo mark; rollback sends `_UNDO B` and AutoCAD never confirms it landed — best-effort, not atomic* | ✅ *full DXF snapshot, restored synchronously* | | Paper-space layouts + viewports | ✅ | ✅ | | Viewport model-content rendering | ✅ | ✅ ᵐ *(no viewport borders)* | | Selection window / crossing / polygon | ✅ | ✅ | @@ -152,18 +154,18 @@ There are **27 capability keys**. `system_capabilities` returns the live map, an | CHSPACE (`entity_change_space`) | — *unverified against a live seat* | ✅ ᶜ | | Undo history | ✅ | opt-in — `EZDXF_UNDO_DEPTH` defaults to `0` | | TABLE and MLEADER | Native | Portable composite | -| Raw commands / AutoLISP | Opt-in | — | +| Raw commands / AutoLISP | ✅ *on by default, denylist-sanitized — `DANGEROUS_COMMANDS_ENABLED=true` removes the denylist* | — *no live command line* | | Screenshots and PDF export | AutoCAD window capture | Matplotlib render ᵐ | ᵐ Needs matplotlib — the `[pdf]` or `[full]` extra. Without it `png`, `pdf`, `viewport_render` and `handle_overlay` all report unsupported on the headless engine.
ᶜ The headless CHSPACE carries four named restrictions in its own capability reason: top-view untwisted viewports only, dimensions refused unless frozen, ACIS/proxy/table refused, and viewport clipping reported rather than applied. > [!IMPORTANT] -> Every COM path added in v1.5.0 was executed against a **live AutoCAD 2026** before release. That run found four defects no amount of reading would have caught — the largest being that `AcadPViewport` has no `ViewCenter` member, so the line setting it had been silently swallowed by a bare `except` since v1.4. +> The COM paths added in v1.5.0 were executed against a **live AutoCAD 2026** before release — two documented exceptions aside: the `AcDbLWPolyline` OCS→WCS translation in `entity_get` and the COM half of `block_create_from_entities` ship unit-tested against a fake ActiveX object, not hardware-verified. The live run was 25 calls: 18 succeeded, 1 refused a specific argument, and 6 refused with the capability keys they declare. That run found four defects, three of which no amount of reading would have caught (the fourth was `analysis_list_properties` calling a helper that does not exist on `ComBackend`) — the largest being that `AcadPViewport` has no `ViewCenter` member, so the line setting it had been silently swallowed by a bare `except` since v1.4. ## Evidence -Every number below is produced by a script in [`benchmarks/`](https://github.com/U-C4N/Autocad-MCP/blob/main/benchmarks/) and rendered to SVG by Python. No hand-edited charts. +Every chart below is rendered to SVG by Python from committed data — none is hand-edited. The numbers are not all script output, and the two exceptions are marked where they appear: **section 4** quotes timings collected by hand across separate runs, and **section 5** is a hand-assigned source review whose nine scores are literals in [`source_review.json`](https://github.com/U-C4N/Autocad-MCP/blob/main/benchmarks/source_review.json) that the renderer validates and draws but does not compute. ### 1 · Live-run scores @@ -178,13 +180,13 @@ Every number below is produced by a script in [`benchmarks/`](https://github.com > [!WARNING] > **Read the matrix column before the score column.** The five v3 tasks are new in v1.5.0; the competitor runs are from 2026-07-23 against pinned competitor commits, and were never put to them. Their scores are over the ten tasks they actually ran, and the five new rows render blank rather than as zeros — an invented zero is indistinguishable from a measured one. These bars do not share a denominator, which is why each one carries its matrix. -Competitors are driven **black-box over MCP stdio** using their own documented tool contracts at the pinned commit, and every geometry claim is verified by re-opening the exported DXF with ezdxf inside the harness. No self-reporting. A capability refusal is recorded `unsupported`, not `fail` — "cannot reach this" and "got this wrong" are different findings. +Competitors are driven **black-box over MCP stdio** against the tool contracts read from their pinned source, at **50% coverage on both**: five of the ten tasks per competitor are recorded `unsupported` — the operation is absent from that checkout, or refused by it — and those rows score zero against a denominator of ten, so 50.0 and 45.0 are coverage-weighted, not ten measured attempts. Over the tasks actually driven, beiming scores 100.0 and puran-water 90.0. Where a task does run, every geometry claim the competitor makes is re-derived by re-opening the exported DXF with ezdxf inside the harness — their own tool output is never the source of truth. This repo's row is not run that way: it runs in-process against the backend, and 13 of its 15 tasks assert on backend return values rather than on a re-opened DXF (only `dxf_roundtrip` and `auditable_delivery` write a file at all). What is independent there is the expected value — a closed-form area, entities placed on purpose, a token ceiling fixed before the measurement — not the observation. A capability refusal is recorded `unsupported`, not `fail` — "cannot reach this" and "got this wrong" are different findings. That classification is written into the adapter from the competitor's pinned source; a refusal that only appears at runtime becomes `unsupported` where the adapter recognises it (beiming's `E_UNSUPPORTED` on `transaction begin` is the one that did) and lands as `fail` otherwise. ### 2 · Task matrix — where the score comes from Task matrix heatmap: 15 tasks against three servers -**v2 scored this repo 10/10, and that number carried no information**: every task in it exercised something the server was built around. v3 adds five tasks chosen because they *can* fail — and three of them did, while being written. +**v2 scored this repo 10/10, and that number carried no information**: every task in it exercised something the server was built around. v3 adds five tasks chosen because they *can* fail — and two of them did, while being written: `hatch_islands` (a HATCH could not be measured at all) and `measure_from_handle` (the boundary tools and the measurement tool disagreed about the same shape). A third measurement defect fixed in the same release — a circle stored as two bulged vertices measured 0.0 — is carried by `two_vertex_circle_has_area` in the correctness suite, not by a benchmark task. | v3 task | Verified against | |---|---| @@ -218,15 +220,17 @@ The 21 pre-existing checks are unchanged and all still pass: the discovery layer | Workload | Wall time | Result | |---|---:|---| -| Create 2,000 lines (individual calls) | 0.32 s | ~6,300 entities/s | -| 10,000 lines: build + DXF export + reopen | 2.16 s | ~4,600 entities/s end-to-end | -| Region query over 10,000 entities | 0.89 s | 2,500 matched | -| Premium pass (layers + part + dims + full critique) | 0.18 s | 0 issues | +| Create 2,000 lines (individual calls) | 0.35 s | ~5,716 entities/s | +| 10,000 lines: build + DXF export + reopen | 2.18 s | ~4,591 entities/s end-to-end | +| Region query over 10,000 entities | 2.47 s | 2,500 matched | +| Premium pass (layers + part + dims + full critique) | 0.26 s | 0 issues | -Workloads call the same backend methods the MCP tools call, so server-side overhead is included. **Self-measurement only** — competitor servers pay an extra stdio serialization cost that in-process runs do not, so no cross-server timing claims are made. +One run of each workload — `perf_suite` executes each exactly once per invocation, and these are the figures in the committed [`perf-ezdxf.json`](https://github.com/U-C4N/Autocad-MCP/blob/main/benchmarks/results/published/perf-ezdxf.json). Run-to-run spread on this machine is wide enough to matter: treat them as an order of magnitude, not a stopwatch. + +Workloads call the same backend methods the MCP tools call, so the backend's own per-call overhead — worker-thread hand-off, document lock, the `EZDXF_CALL_TIMEOUT` guard — is included. The FastMCP layer above it is **not**: `perf_suite.py` never builds the server, so tool dispatch, argument validation, the five middleware and result serialization all sit outside the timer, and in-process they cost roughly an order of magnitude more than the backend call itself. Read these as backend-engine numbers, not per-tool-call latency. **Self-measurement only** — competitor servers pay an extra stdio serialization cost that in-process runs do not, so no cross-server timing claims are made. > [!CAUTION] -> These are **not** an improvement over v1.4.0. Measured against v1.4.0 on the same interpreter, this release is **28% slower** at creating entities — see [what this release is bad at](#what-this-release-is-bad-at) for the number and its full attribution. +> These are **not** an improvement over v1.4.0, and they are not the engine's floor. The per-call timeout guard costs about **21%** of entity-creation throughput — see [what this release is bad at](#what-this-release-is-bad-at) for the paired measurement and why the guard is kept. ### 5 · Source-review leaderboard @@ -234,7 +238,7 @@ Workloads call the same backend methods the MCP tools call, so server-side overh Nine public AutoCAD MCP projects against a fixed 100-point rubric — CAD breadth (25), correctness and delivery (20), backends/platforms (15), engineering production (15), tests and maintenance (15), security and operations (10). Stars and raw tool counts score nothing. Data and dates in [`benchmarks/source_review.json`](https://github.com/U-C4N/Autocad-MCP/blob/main/benchmarks/source_review.json). -**This table is not re-scored per release.** This repository's row is still the one written against **v1.4.0**. Raising our own number for v1.5.0 would mean re-reviewing eight other projects the same day to keep it fair, and that has not been done — so the score stays where the evidence is. +**This table is not re-scored per release** — no row is, including the other eight. This repository's row is still the one written against **v1.4.0**. Raising our own number for v1.5.0 would mean re-reviewing eight other projects the same day to keep it fair, and that has not been done — so the score stays where the evidence is. Method, caveats and boundaries for every lane: [`benchmarks/README.md`](https://github.com/U-C4N/Autocad-MCP/blob/main/benchmarks/README.md). @@ -242,17 +246,39 @@ Method, caveats and boundaries for every lane: [`benchmarks/README.md`](https:// A page that only lists strengths is a page that has not been measured. -**Entity creation is 28% slower than v1.4.0.** Measured on one interpreter, median of three runs: 2,000 lines went 245 ms → 315 ms, and the 10,000-line roundtrip 1,765 ms → 2,146 ms. Fully attributed — setting `EZDXF_CALL_TIMEOUT=0` returns v1.5.0 to **244.8 ms** and **1,679 ms**, which are v1.4.0's numbers exactly. The cost is the per-call `asyncio.wait_for` this release wrapped around every headless call so that one hung call can no longer wedge a server whose document lock is a single `asyncio.Lock`. A deliberate trade, with a documented knob, on the 1.6 roadmap to win back. +**Entity creation is ~21% slower than it would be without the call timeout.** +The timeout is what stops one hung ezdxf call wedging a server whose document +lock is a single `asyncio.Lock`, and it is not free. Measured on one +interpreter, three runs of each configuration with the median taken by hand +(`perf_suite` runs each workload once per invocation — it cannot produce a +median itself): -> The v1.4 README printed *slower* numbers than the table above — 0.64 s and 13.9 s for the same two workloads. That looks like a 2–6× win and is not one: the old report was recorded on CPython 3.14 and this one on 3.11.15. Interpreter, not code. Comparing the two published files directly is the mistake this paragraph exists to stop. +| Workload | `EZDXF_CALL_TIMEOUT=0` | default (120 s) | Cost | +|---|---:|---:|---:| +| 2,000 lines | 243.7 ms | 294.3 ms | +20.8% | +| 10,000-line roundtrip | 1,709.6 ms | 2,049.2 ms | +19.9% | -**Wave A shipped 13 tools of a planned 39.** Every cut is backed by a measurement. The REGION and 2D-boolean family went because `add_region()` produces a REGION with **zero ACIS bytes**, and the `greiner_hormann` substitute loses **28.2%** of the area on a square with one semicircular edge — shipping it would have re-created the exact class of silent error this release spent four milestones deleting. Most of the planned measurement family was already covered by the eight `analysis_*` tools that existed. +v1.5.0 paid **28.6%** on the first workload; v1.5.1 rebuilt the wrapper around a +single `asyncio.timeout_at` instead of two `asyncio.wait_for`s — `wait_for` +wraps its awaitable in a Task, so the old shape created two extra Tasks and two +timer handles on *every* call. That recovered about a third of the overhead and +fixed a latent double-release, but it did not close the gap: the remaining cost +is the Task the abandon-on-timeout design genuinely needs. The knob is +documented rather than hidden, and the item stays open. + +> These figures are hand-collected across separate single-shot runs and have no +> published artifact; `benchmarks/results/published/perf-ezdxf.json` records one +> default-configuration run only. + +> The README on `main` through the v1.4 era printed *slower* numbers than the table above — 0.64 s and 13.9 s for the same two workloads. That looks like a 2–6× win and is not one: the old report was recorded on CPython 3.14 and this one on 3.11.15. Almost entirely the interpreter, not code — run on one interpreter, this release is *slower* than v1.4.0 on both workloads, not faster. Comparing the two published files directly is the mistake this paragraph exists to stop. + +**Wave A shipped 13 tools of a planned ~39.** Every cut is backed by a measurement. The REGION and 2D-boolean family went because `add_region()` produces a REGION with **zero ACIS bytes**, and the substitute, `greiner_hormann`, takes and returns flat vertex lists, so a bulged edge goes in as its chord: **28.2%** of the area on a square with one semicircular edge — shipping it would have re-created the exact class of silent error this release's honesty pass exists to delete. Of the planned measurement family only `analysis_measure_entity` shipped — measuring a drawn entity by handle, which neither backend could do — and the centroid and moment-of-inertia tools were cut with nothing replacing them. Of the eight `analysis_*` tools that already existed, the two tagged `measure` both measured points the caller typed in. **`system_run_command` / `system_run_lisp` are a guardrail, not a security boundary** — the rejection message clients receive says so in those words. A 36-verb command denylist and a 24-pattern AutoLISP denylist refuse the obviously destructive cases, but AutoCAD accepts hundreds of commands and any loaded ARX/LISP adds more, so the list cannot be complete, and `DANGEROUS_COMMANDS_ENABLED=true` switches both off entirely. **Path validation is per-tool, and unscoped until you scope it.** Nine tools call `validate_path`; nothing in the middleware enforces it server-wide. With `ALLOWED_PATHS` empty — the default — the only positive bound is a ten-entry system-directory denylist that does not include `C:/Users`, `/home`, `/root` or `/var`. **Set `ALLOWED_PATHS`.** It is the setting that actually scopes the filesystem. -**Non-AutoCAD ProgIDs are unverified.** `CAD_PROGID` changes which COM application the backend attaches to, and nothing beyond the connection has been tested against BricsCAD, ZWCAD or GstarCAD. They are listed in `.env.example` commented out and marked UNVERIFIED. +**Non-AutoCAD ProgIDs are unverified.** `CAD_PROGID` changes which COM application the backend attaches to — the configured ProgID is used at both `GetActiveObject` and `Dispatch`, and an unrecognised one fails loudly rather than falling back to AutoCAD. Nothing has been tested against BricsCAD, ZWCAD or GstarCAD, the connection included: no running seat of any of them was ever attached to. They are listed in `.env.example` commented out and marked UNVERIFIED. ## Configuration diff --git a/benchmarks/README.md b/benchmarks/README.md index 5a7ded9..d5343bb 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -60,7 +60,7 @@ written: | Task | Category | Verified against | |---|---|---| -| `tool_discovery` | discovery | six AutoCAD command names, each ranking #1 of 154 tools | +| `tool_discovery` | discovery | six AutoCAD command names, each found in the top 3 of the 154-tool registry the harness ranks — all six came back #1 in the published run | | `token_budget` | efficiency | 40,305 → 356 tokens advertised, against a 2,000 ceiling fixed in advance | | `hatch_islands` | hatch | 300 filled with the island, 400 ignoring it | | `selection_filter` | selection | window 1, crossing 2, bounding box 3, polygon 1 | @@ -153,33 +153,36 @@ python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.js python -m benchmarks.render_perf_chart ``` -### v1.4.0 → v1.5.0, same interpreter +### What the call-timeout guard costs -The published v1.4 report was recorded on CPython 3.14 and the v1.5 one on -3.11.15, so comparing the two files directly shows an apparent 2–6× speedup -that is almost entirely the interpreter. The honest comparison re-runs v1.4.0 -in a `git worktree` with the *current* suite file — same instrument, different -code under test, one interpreter. Median of three runs each: +The published v1.4 report was recorded on CPython 3.14 and the later ones on +3.11.15, so comparing those two files directly shows an apparent 2–6× speedup +that is almost entirely the interpreter. The comparison that means something is +the guard against no guard, on one interpreter, back to back. Three runs of +each configuration, median taken by hand — `perf_suite` executes each workload +once per invocation and cannot produce a median itself: -| Workload | v1.4.0 | v1.5.0 | Change | -|---|---:|---:|---| -| `create_lines_2k` | 245 ms | 315 ms | **28% slower** | -| `roundtrip_10k` | 1,765 ms | 2,146 ms | **22% slower** | -| `region_query_10k` | 1,090 ms | 1,039 ms | within spread | -| `premium_pass` | 375 ms | 44 ms | faster; the v1.4 spread (279–772 ms) is too wide to quantify | +| Workload | `EZDXF_CALL_TIMEOUT=0` | default (120 s) | Cost | +|---|---:|---:|---:| +| `create_lines_2k` | 243.7 ms | 294.3 ms | **+20.8%** | +| `roundtrip_10k` | 1,709.6 ms | 2,049.2 ms | **+19.9%** | -Attributed rather than reported: `EZDXF_CALL_TIMEOUT=0` puts v1.5.0 back to -244.8 ms and 1,679 ms, which are v1.4.0's numbers. The entire cost is the -per-call `asyncio.wait_for` wrapped around every headless call in this release, -so a single hung ezdxf call can no longer wedge a server whose document lock is -one `asyncio.Lock`. Reproduce: +v1.5.0 paid 28.6% on the first workload; v1.5.1 rebuilt the wrapper around a +single `asyncio.timeout_at` instead of two `asyncio.wait_for`s — `wait_for` +wraps its awaitable in a Task, so the old shape created two extra Tasks and two +timer handles per call — which recovered about a third of the overhead. The +rest is the Task the abandon-on-timeout design genuinely needs. The guard is +what stops one hung ezdxf call wedging a server whose document lock is a single +`asyncio.Lock`, so it stays; the knob is documented rather than hidden. + +These figures are hand-collected and have **no published artifact** — +`perf-ezdxf.json` records one default-configuration run only. Reproduce: ```bash -git worktree add /tmp/wt-v140 v1.4.0 -cp benchmarks/perf_suite.py /tmp/wt-v140/benchmarks/ -(cd /tmp/wt-v140 && PYTHONPATH=. python benchmarks/perf_suite.py --out /tmp/v140.json) -python -m benchmarks.perf_suite --out /tmp/v150.json -EZDXF_CALL_TIMEOUT=0 python -m benchmarks.perf_suite --out /tmp/v150-notimeout.json +for i in 1 2 3; do + python -m benchmarks.perf_suite --out /tmp/on-$i.json + EZDXF_CALL_TIMEOUT=0 python -m benchmarks.perf_suite --out /tmp/off-$i.json +done git worktree remove --force /tmp/wt-v140 ``` @@ -201,20 +204,28 @@ python benchmarks/compare_versions.py --json results.json ### Result — v1.5.0 vs v1.4.0 (release gate) -24 checks, ezdxf backend, one subprocess per check. Machine-readable report: +26 checks, ezdxf backend, one subprocess per check. Machine-readable report: [`results/published/ab-v1.4.0-vs-v1.5.0.json`](results/published/ab-v1.4.0-vs-v1.5.0.json). | Version | Checks passing | Pass rate | Fixed | Regressed | |---------|----------------|-----------|-------|-----------| -| **v1.4.0** (baseline) | 21 / 24 | 87.5 % | — | — | -| **v1.5.0** (this release) | 24 / 24 | 100 % | 3 | **0** | +| **v1.4.0** (baseline) | 21 / 26 | 80.8 % | — | — | +| **v1.5.0** (this release) | 26 / 26 | 100 % | 5 | **0** | -Read the three "fixed" carefully: they are **new capability, not repaired -regressions**. v1.4.0 has no `entity_measure` and no boundary tracing, so it -misses `hatch_area_subtracts_its_island`, `boundary_area_agrees_with_measure` -and `two_vertex_circle_has_area` by not having the method rather than by -getting it wrong. They are in the suite because all three broke during v1.5.0 -development and nothing outside their own unit tests would have caught it. +The pass rate is against the 26-check v1.5.0 suite; v1.4.0 passed 21/21 of the +checks that existed when it shipped. The five splits into two kinds, and the +difference is the whole reason this lane distinguishes `miss` from `fail`: + +* **Three are new capability.** v1.4.0 has no `entity_measure` and no boundary + tracing, so it misses `hatch_area_subtracts_its_island`, + `boundary_area_agrees_with_measure` and `two_vertex_circle_has_area` by not + having the method (`miss → pass`). They are in the suite because all three + broke during v1.5.0 development and nothing outside their own unit tests + would have caught it. +* **Two are repaired defects.** v1.4.0 *has* `dimension_diameter` and + `dimension_radius` and gets them wrong (`fail → pass`): `leader_length`, a + text placement, was being measured as geometry, so at default settings every + diameter callout came out 2 × `leader_length` too large. The 21 pre-existing checks are unchanged and all still pass: the discovery layer, `cad_batch`, the 19-module contract split, the layout/viewport family diff --git a/benchmarks/results/published/perf-ezdxf.json b/benchmarks/results/published/perf-ezdxf.json index 171a6f2..28f1732 100644 --- a/benchmarks/results/published/perf-ezdxf.json +++ b/benchmarks/results/published/perf-ezdxf.json @@ -14,22 +14,22 @@ "entity_count": 2000 }, "metric": "entities_per_second", - "metric_value": 6343.1, - "wall_ms": 315.3, + "metric_value": 5715.8, + "wall_ms": 349.9, "workload_id": "create_lines_2k" }, { "description": "Create 10,000 lines, export DXF, reopen and verify", "detail": { - "build_ms": 1574.1, + "build_ms": 1633.1, "entity_count": 10000, - "export_ms": 261.8, + "export_ms": 273.0, "file_bytes": 1369663, - "reopen_ms": 320.2 + "reopen_ms": 272.1 }, "metric": "entities_per_second", - "metric_value": 4637.9, - "wall_ms": 2156.1, + "metric_value": 4590.8, + "wall_ms": 2178.3, "workload_id": "roundtrip_10k" }, { @@ -39,7 +39,7 @@ }, "metric": "entities_matched", "metric_value": 2500.0, - "wall_ms": 890.8, + "wall_ms": 2466.9, "workload_id": "region_query_10k" }, { @@ -49,7 +49,7 @@ }, "metric": "critique_issues", "metric_value": 0.0, - "wall_ms": 176.5, + "wall_ms": 255.7, "workload_id": "premium_pass" } ] diff --git a/docs/assets/autocad-mcp-perf.svg b/docs/assets/autocad-mcp-perf.svg index 6b91d0a..30033ab 100644 --- a/docs/assets/autocad-mcp-perf.svg +++ b/docs/assets/autocad-mcp-perf.svg @@ -39,8 +39,8 @@ z - @@ -50,12 +50,12 @@ L 0 3.5 " style="stroke: #8b949e; stroke-width: 0.8"/> - + - + - - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - + @@ -708,7 +701,7 @@ z - + @@ -955,7 +948,7 @@ z - + @@ -1066,81 +1059,10 @@ z - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - @@ -1554,8 +1547,8 @@ z - - + + - - @@ -1741,8 +1733,8 @@ z - - + + @@ -1761,8 +1753,8 @@ z - - + + - - - + - + @@ -1938,11 +1908,11 @@ z - + - - - + + + @@ -1957,9 +1927,67 @@ z - - + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - + + @@ -2176,38 +2213,6 @@ z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -