db5b4d3ef8f8b0911e71af9cbb3944639f8ee41b
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Umut CANandClaude Opus 5.5
|
ba41428c88 |
bench: matrix v7 adds takeoff_roundtrip and understand_foreign; five understanding checks; the live smoke and the field-test script
The evidence for track H, in the shape tracks A, E, B + G and F used.
- benchmarks/tasks_v7.py: 21 tasks, DEFAULT_MATRIX = "v7"; v6 to v2 stay
addressable. Both new tasks read the synthetic plant pair and nothing
else. takeoff_roundtrip: the P&ID called schematic, every routable run's
layout length equal to the rectilinear MST of its tags, sizes and services
as drawn (a return drawn on the supply layer included), the cable metres
rounded up after the allowance, an unstated power left empty.
understand_foreign: inches over millimetres flagged, the stray entity found
by handle, two plan copies separated, five layers classified. A test breaks
each input and watches its gate fail by name.
- Five correctness checks, every number worked out by hand in its docstring;
the published A/B report is re-run over 48 checks: 26 / 48 -> 48 / 48,
zero regressed.
- scripts/smoke_understand_com.py reads the pair through the live engine -
snapshots, the DWG route, both takeoffs and a diff - and closes every
document it opens; the CHANGELOG records what it returned.
scripts/field_test_takeoff.py runs the takeoffs on a user's own files,
refuses to write inside the repository and prints no quantity by default.
- The README token table is remeasured; check_doc_numbers derives the task
count from the v7 matrix; CLAUDE.md, both READMEs, the CHANGELOG and the
spec status line say what shipped.
One reader defect found on the way: describe.tag_occurrences read a wiring
callout ("Wiring to CP-2") as a place its panel is written, so the scale
check called CP-2 ambiguous and read "Wiring to CP1" as a phantom tag CP1 -
10 tags and 45 pairs (13.3 % within 10 %) where the generator placed 11 and
55 (21.8 %). A callout is no longer a tag occurrence, the rule
network.tag_occurrences already kept; two tests failed before the fix.
The smoke works only in documents it opened from its own temporary folder,
known by full path; it attaches with GetActiveObject only (the backend's
Dispatch fallback would launch AutoCAD), checks in the same COM-thread call
that the active document is its own before each snapshot, the DWG save and
the move, closes its documents with Close(False) and re-activates the
document that was active when it started. No new ActiveX member: Documents,
Item, FullName, ActiveDocument, Activate and Close are the ones the backend
already calls. Measured on AutoCAD 2026 (25.1s): exit 0, model-space types
equal file / live / DWG, export 0.104 s, no document of the run left open.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
|
||
|
Umut CANandClaude Opus 5.5
|
0abbe55041 |
bench: matrix v6 adds arch_roundtrip; four architectural correctness checks; the live arch smoke
The evidence for track F, in the shape tracks A, E and B + G used. - benchmarks/tasks_v6.py: 19 tasks, DEFAULT_MATRIX = "v6"; v5 to v2 stay addressable. arch_roundtrip draws the example plan with arch_plan_from_spec and reads it back without the spec: rooms_detect must find each labelled room within 0.1 % of its label and of the net floor computed by hand (4825 x 5750 and 3825 x 5750), the schedules must list D1, D2, W1, W2, and the plan is gated on a zero-issue critique and a finalize score of at least 90. A test removes one room label and the gate fails naming arch_room_unlabelled. - Four correctness checks, every number worked out by hand in the docstring: L / T / X junction corners and wall areas, a 4 x 5 m room labelled 20.00 m2 rather than its 21.84 m2 axis area, a door interrupting both faces, and a plan of plain lines read as 20 and 15 m2 at confidence 0.6. The published A/B report is re-run over 43 checks: 26 / 43 -> 43 / 43, zero regressed. - scripts/smoke_arch_com.py runs the whole plan once on a live seat and closes every document it opens; the CHANGELOG records what it returned. - The token suite measures a TOOL_PACKS=core,arch row; check_doc_numbers derives the task count from the v6 matrix; CLAUDE.md, both READMEs and the spec status line say what shipped. - fix(com): the first live smoke exited 1 - the room reader found 0 wall segments. Measured on AutoCAD 2026 (2026-09-23): with a makepy cache, ModelSpace.Item(i) returns the gen_py IAcadEntity wrapper (type(item).__name__ == "IAcadEntity") and item.StartPoint on an AcDbLine raises AttributeError, so every entity_list row lost its type-specific properties (28 wall LINEs, 0 with start/end). _entity_info now re-dispatches exactly that wrapper through _com_unnarrow; after the fix 38 of 53 rows carry start/end and the smoke exits 0 with both rooms at 27 743 750 / 21 993 750 mm2 and 0 critique issues. Re-dispatch measured at ~1.7 ms per entity (dynamic vs gencache alike). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd |
||
|
Umut CANandClaude Opus 5.5
|
898e775f93 |
bench: matrix v5 adds mech_assembly; the correctness suite grows with the mechanical and sheet checks
The evidence for tracks B and G, in the shape tracks A and E used.
- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
addressable so an earlier report reproduces exactly rather than being only
described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
built from the part model and the sheet standard, through the code the
tools call (apply_titleblock, draw_part/add_view/dimension_part,
insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
two numbers that cannot be negotiated: drawing_critique(focus=None) returns
zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
the v5 matrix). The adapter returns the focuses that fired, so a
regression names itself, and a test drops one balloon to prove the gate
fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
(25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
the drawing, a linked balloon, an xref attached / listed / detached, a
31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
the first call after Documents.Add is refused) and tracks its documents by
the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
code:
* ezdxf dimension_diameter put the text beyond the SECOND chord point;
ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
front-view diameter stood on its own chain dimension. ezdxf now follows
ActiveX.
* draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
every target layer before the first entity.
* bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
(ObjectName AcDbBlockReference) - the live parts list was empty.
* COM _entity_info reported no dimension text_position: ModelSpace.Item
narrows to IAcadEntity, which has no TextPosition (measured
AttributeError); read through _com_unnarrow (TextPosition measured on
AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
diameter collided with its own bore.
* pid graph raised KeyError 'start' mid-critique on a LINE whose point read
AutoCAD refused while busy; such a line is left out.
* the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
the task count from the v5 matrix, the token table is remeasured with a
TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
collected-tests figure is retaken (3990).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
|
||
|
Umut CANandClaude Opus 5
|
18c7c69da6 |
fix(task-25): page_setup_truth and settings_pdf_mediabox_a3 move the sheet to ANSI B before proving A3
A fresh Layout1 is already ISO A3 landscape (ezdxf R2018 default: 420 x 297, plot_rotation 0), so an A3-only /MediaBox read-back passed with page_setup_apply stubbed to a no-op: the A/B gate row and the v4 matrix task could not fail on the setter. Both now apply ANSI B (432 x 279, a size no fresh document has; spec section 9 names it) and read that PDF first, then apply ISO A3, read its PDF, and require the setter's changed.size_mm to have moved from B to A3. The stub fails at the first PDF. The moved-sheet comparison is +/-0.5 mm, the spec's tolerance, not exact: measured live on AutoCAD 2026 (scratch Documents.Add(), closed unsaved), the COM engine reports ANSI B as 431.8 x 279.4 and the DWG To PDF.pc3 file's MediaBox as 431.8 x 279.4 then 420.158 x 297.039 for A3; the task passes there with changed.size_mm [[431.8, 279.4], [420.0, 297.0]]. Headless it is [[432.0, 279.0], [420.0, 297.0]]. tests/test_benchmark_v4.py gains the mutation test the finding describes (setter stubbed -> both rows must fail; both passed on the old code) and a run of the gate row itself; the headless task test asserts the B stage. README rows and the tasks_v4 docstring say what the task now does; the README collected-tests figure follows the two new tests (3282 -> 3284). The regenerated A/B report against v1.5.1 is byte-identical (26 -> 32/32). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd |
||
|
Umut CANandClaude Opus 5
|
c2942d3c94 |
bench: matrix v4 adds page_setup_truth; correctness suite grows to 32 with three settings checks (A/B vs v1.5.1: 26 -> 32 / 32)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd |
||
|
Umut CANandClaude Opus 5
|
2be5e4180d |
docs(pid): track A shipped — CLAUDE.md, README, changelog with the live COM smoke record
scripts/smoke_pid_com.py runs the whole track against the live AutoCAD in a new document. ComBackend.connect() is lazy by design, so the script reads reachability from system_status()["connected"] rather than from connect() as the plan sketched, and closes its document without saving unless --keep is passed. On this machine (AutoCAD 2026, "25.1s (LMS Tech)") it exits 0: 5 nodes, 4 edges, 0 dangling, confidence_min 1.0, 0 critique issues, 2 XDATA chunks — identical to the headless engine on the same spec. The first attempt exited 2 on a stale win32com gen_py cache (a package folder holding only __pycache__); an environment fault, no backend change needed. A/B against v1.5.1 published as ab-v1.5.1-vs-v1.6.0-dev.json: 26/29 -> 29/29, three miss -> pass, zero regressed. check_doc_numbers.py now names the report and the two row labels in one place (AB_REPORT, AB_*_LABEL), and README.md / benchmarks/README.md carry the new rows; the v1.5.0 and v1.4.0 tables stay as history. README: P&ID row, the showcase and catalogue images, the token table re-measured (161 / 44,930 by default, 152 / 41,420 with TOOL_PACKS=core, lean 50 / 14,007), every stale count reconciled (154 -> 166 registered, 149 -> 161 advertised, corpus 158/672 -> 162/731 by the same count, 19 -> 20 groups, 6 -> 8 resources, six-task matrix with pid_roundtrip), a "Known limitations of the P&ID track" subsection, and the snapshot line at this machine's 2423 collected tests (CI has not run on this tree; noted in the changelog with the pre-existing 1369 vs 1363 drift). CLAUDE.md: section 17, sections 5 and 7 counts, TOOL_PACKS, the two 1.6 contracts and the reader rules. Spec status -> implemented. The plan and spec markdown were reformatted by ruff (fenced Python only) because `ruff format --check .` was already red on them at HEAD and the gate requires the whole tree clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd |
||
|
Umut CANandClaude Opus 5
|
8d9c444b76 |
bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks
tasks_v4.py extends v3 with one task the reader can fail on its own: the example spec is drawn through pid_from_spec's code and read back by the graph builder, which never sees the spec. The runner, the chart and the doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3" explicitly now that the default has moved. The plan's adapter check expected the two unnumbered lines to come back with no line number; draw_line builds one from the default number_format (bare sequence when size/service/spec/insulation are empty), so the task asserts the two authored numbers on the two process lines and sequence- built XDATA numbers on the other two instead. The published A/B report still covers 26 checks; Task 21 re-runs it against v1.5.1, which is what clears the remaining doc-number drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd |
||
|
Umut CANandClaude Opus 5
|
ca1c79fd01 |
fix: seven defects an adversarial audit found, and the gates that missed them
Release / test (full, windows-latest, 3.11) (push) Canceled after 0s
Release / test (pdf, ubuntu-latest, 3.11) (push) Canceled after 0s
Release / test (pdf, ubuntu-latest, 3.12) (push) Canceled after 0s
Release / build (push) Canceled after 0s
Release / publish-pypi (push) Canceled after 0s
Release / github-release (push) Canceled after 0s
Every one of these was reported ok/clean to the caller while being wrong.
Silently wrong numbers
- selection_filter(min_area) read `entity.dxf.get("area", 0.0)` for SOLID,
TRACE and HATCH, none of which carry that attribute, so every hatch
measured 0.0. A 1600 mm^2 hatch was excluded by min_area=1.0 while
`filtered_by` asserted the filter had run, and COM (which reads ActiveX
.Area) returned a different selection for the same call. It now answers
from the measurement `entity_measure` uses, so the two cannot disagree.
- Every headless dimension rendered through ezdxf's bare `Standard` style:
1.0 mm text with a comma decimal marker, against ISO 3098's 2.5 mm minimum
and ISO 129's point -- while the document header itself said $DIMTXT 2.5.
`ezdxf.new(setup=True)` is not the fix; measured, it renders a 100 mm
dimension as "10000". The attributes are now stored explicitly.
- Saving writes `dimrnd = 0.0` onto the Standard style, and ezdxf applies
`xround(value, dimrnd)` whenever the attribute is present at all --
`xround(33.333, 0.0)` is 33. So on any reopened file, including one this
server saved, every dimension was rounded to a whole unit before DIMDEC
formatted it: a 12.75 mm feature dimensioned 13. AutoCAD means DIMRND 0 as
*no* rounding; the sentinel is now discarded at every load site, and a real
0.5 mm rounding is left alone.
- The DIM* header fold gave out-of-range values teeth they did not have when
they died in the header: dim_decimals=-1 returned ok and then killed the
next dimension with `ValueError: Format specifier missing precision` from
inside ezdxf, and 40 printed forty digits of float noise onto the sheet.
Ranges are now AutoCAD's own (DIMDEC 0-8, DIMZIN 0-15, lengths positive).
Security
- security.py recorded the policy as "sanitize_command and sanitize_lisp each
have exactly ONE caller ... the policy is scoped to the CHANNEL". The
premise was false. `_ensure_linetype_loaded` and `ComBackend.linetype_load`
build `_-LINETYPE _LOAD {name} {file}` and hand it to SendCommand, which
treats a newline and ';' as segment separators -- so a `linetype=` argument
carrying either injected further AutoCAD commands, and the 36-verb denylist
never saw that channel. Guarded with the DXF symbol-name rule (the
injection characters fall out of it rather than being enumerated) plus
validate_path for the file argument, which was also the one path-taking
tool reaching the filesystem unchecked. The policy comment is corrected.
Gates that could not see
- drawing_critique returned zero issues on a quarantined document: all seven
checks wrapped their backend call in a bare `except Exception`. CLAUDE.md
rule 7 makes that call the gate an agent is told to trust. Only the
quarantine now escapes; an ordinary broken check still degrades.
`_check_dimension_legibility` also read `_doc` directly from the event loop
thread, missing the gate and performing exactly the unsynchronised access
the quarantine exists to prevent. `construction_clear` answered
"deleted: 0" -- there was nothing to clear -- for the same reason.
- The iso128 focus checked layer lineweights only, so a sheet lettered at
1.0 mm critiqued clean. It now reads the effective dimension height and
separator, header-then-style, in the order the renderer resolves.
- check_doc_numbers.py derived five values it never checked, and those were
the ones that had drifted. The collected-test count (README said 1274, the
tree had 1369) and the A/B pass-rate and Fixed columns are now gated.
- benchmarks/README.md's v1.4.0 table was impossible on its face -- 21
passing plus 5 fixed is 26, not 24 -- and contradicted the report it
linked, because `ab-v1.4.0-vs-v1.5.0.json` was regenerated from a tree that
already carried the dimension fixes and so never measured the v1.5.0 tag.
Replaced with a real run: ab-v1.4.0-vs-v1.5.1.json, 21/26 -> 26/26.
Release gate
- release.yml ran one ubuntu/3.11 lane, so a Windows-only defect could reach
PyPI with no Windows job having executed. That is not hypothetical: v1.5.0
shipped `except pywintypes.com_error` raising NameError wherever pywin32 is
absent. It now runs the same three lanes as ci.yml, Windows included.
1368 passed, 1 xfailed. ruff clean. Tool inventory and all 16 gated document
numbers match the tree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Umut CANandClaude Opus 5
|
6b9860e68e |
docs: apply 27 corrections from the adversarial triage, and rework the perf lane
The triage over the 43 unaddressed verdicts returned 4 code and 35 doc items. The code ones shipped in e24b46d; these are the documents. Twenty-one applied mechanically after a validation pass confirmed each locator still matched exactly once and each replacement was supported by the tree. Ten were held back because a re-verify pass found them mutually exclusive -- three patches rewriting the same benchmarks/README block, three rewriting the same OCS paragraph -- and applying them in sequence would have corrupted the files. Those were resolved by hand into one coherent edit each. One was rejected. The corrections worth naming: - **The A/B table contradicted itself in three files.** benchmarks/README.md and CLAUDE.md said 24 checks / 21-24 / 87.5% / "fixed 3" while the suite has 26 and the published JSON says 21/26, 80.8%, five fixed. All three now agree, and the five are split into the three new-capability `miss -> pass` and the two repaired-defect `fail -> pass`, which is the distinction the lane exists to make. - **"Every number below is produced by a script"** was false twice over: section 5's nine scores are literals a human assigned, and section 4 now quotes hand-collected timings. The claim is narrowed to the charts, which really are all rendered from committed data, and both exceptions are named. - **The OCS paragraph over-quantified.** TEXT rotation was given as the single exception; z components and HATCH boundaries are also entity-frame, and the live backend records `2d_polyline_coordinates_not_verified_against_live_autocad` in the same reason string. It now points at `ocs_normalized` as the live list. - **Transactions were shown as bare parity.** The server itself grades them differently -- `FeatureCapability(True, "undo_mark")` on COM against `(True, "snapshot")` headless -- and COM rollback sends `_UNDO B` without confirming it landed. The row says so. - **"Raw commands: Opt-in"** named a switch that does not exist. Both tools are registered unconditionally; `DANGEROUS_COMMANDS_ENABLED` only disables the sanitizer. - **"Wave A shipped 13 of a planned 39"** hardened an approximation: the only source writes `~39`. - **CAD_PROGID** said nothing beyond the connection was tested against BricsCAD/ZWCAD/GstarCAD, implying the connection was. It was not. Perf lane reworked against a fresh paired measurement. The old text claimed "28% slower than v1.4.0, median of three runs" and that `EZDXF_CALL_TIMEOUT=0` restored v1.4.0's numbers "exactly". Three problems: `perf_suite` runs each workload once per invocation and cannot produce a median, "exactly" was refuted by the figures in its own sentence, and the number moved when the timeout wrapper was rebuilt. It now reports the guard's cost -- +20.8% and +19.9%, three paired runs, median taken by hand -- says the figures have no published artifact, and says the wrapper rework recovered about a third of the v1.5.0 overhead rather than closing it. perf-ezdxf.json and the perf chart regenerated against the shipped code, and the README table rebuilt from that artifact so the two cannot drift. 1267 passed, 1 xfailed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Umut CANandClaude Opus 5
|
b03a7615b1 |
release: v1.5.0 — discovery layer, honesty debt closed, Wave A
Tool count 131 -> 154; 474 -> 1242 collected tests. The release exists for two measured reasons. A 149-tool catalog costs a client ~40,300 tokens before it has asked for anything, and a drafter searching the way drafters think -- FILLET, BPOLY, QSELECT, WBLOCK -- scored df=0 against the stock index, because those words were simply not in the data. Discovery and batching - DISCOVERY_MODE=search replaces the catalog with search_tools + call_tool, ranked over an AutoCAD command and synonym corpus (discovery/aliases.py) on top of fastmcp's BM25 index. Measured: 40,305 -> 356 tokens advertised. - cad_batch runs a step list in one round trip; fields= projects result-heavy responses. The `core` profile is gone -- discovery replaced it. Honesty debt (Tier 0) - OCS/ECS: `extrusion` appeared nowhere in this codebase, so mirrored and rotated geometry reported coordinates that were quietly wrong. backends/ocs.py translates at the backend boundary; every tool coordinate is WCS on both engines, and a tilted plane refuses rather than answering with a number that looks fine. - drawing_save_as(fmt="dwg") wrote DXF bytes to a .dwg path. drawing_audit never repaired. analysis_measure_area shoelaced points the caller typed in. - ezdxf calls gained a per-call timeout, so one hung call can no longer wedge a server whose document lock is a single asyncio.Lock. Architecture - The 1854-line backend ABC is 19 contract modules; @capability(key, reason=) makes optional methods declarative and a gate fails on undeclared holes. - 8 layout/viewport tools, CHSPACE, CAD_PROGID escape hatch. - Wave A: selection filters, boundary tracing, hatch depth, annotation objects, measurement -- 13 tools of a planned 39. The REGION and 2D-boolean family was cut on evidence: add_region() yields zero ACIS bytes and the greiner_hormann substitute loses 28.2% of the area on a square with one semicircular edge. Verified against a live AutoCAD 2026, which found four COM defects no amount of reading would have caught -- AcadPViewport has no ViewCenter, and the line that set it had been swallowed by a bare except since v1.4. Benchmark v3 (15 tasks) and what it caught - v2 scored this server 10/10, which carried no information: every task in it exercised something the server was built around. Five tasks were added because they can fail, and three did while being written -- a hatch could not be measured though the refusal listed HATCH as measurable, the boundary tools and the measure tool disagreed about the same shape (139.2177 vs 139.2699), and a circle stored as two bulged vertices measured 0.0. - Competitor reports are NOT re-scored. They are pinned v1.4 runs against v2 and were never asked these five questions; the chart shows those rows `not run`. - Correctness A/B: 21/24 -> 24/24, zero regressions. Performance, stated plainly The published v1.4 perf report was recorded on CPython 3.14 and the new one on 3.11.15, so the apparent 2-6x speedup is the interpreter, not this release. Re-measured on one interpreter, v1.5.0 is 28% slower creating entities and 22% slower on the 10k roundtrip. Fully attributed: EZDXF_CALL_TIMEOUT=0 restores v1.4.0's exact numbers. That is the deadlock fix being paid for, the knob is documented, and winning it back is on the 1.6 roadmap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Umutcan EdizsalanandCursor
|
af4813e7b6 |
docs(bench): nine-server public AutoCAD MCP leaderboard as the README hero visual
source_review.json extended with three projects reviewed 2026-07-23 (LokmenoWer/best-cad-mcp B/74, daobataotie/CAD-MCP B/35, zh19980811/Easy-MCP-AutoCad C/30) and our entry updated to the released v1.4.0 (95: CI matrix, PyPI packaging, A/B gate, paper space, ISO 286, opt-in solids). Chart regenerated via render_chart; README benchmark section now opens with the leaderboard + 9-row table, source-review lane merged into it. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Umutcan EdizsalanandCursor
|
47a90dbc54 |
docs(bench): add the v1.3.0 -> v1.4.0 correctness A/B gate to the README benchmark lanes
Ran benchmarks/compare_versions.py against the v1.3.0 tag: 21/21 on both versions, 0 fixed / 0 regressed - the v1.4.0 feature+infra work introduced no correctness regressions. Main README gains lane 4 (version A/B, with the historical v1.0.0 -> v1.1.0 contrast); benchmarks/README records the release-gate result and drops the stale 'performance coming' stub. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Umutcan EdizsalanandCursor
|
21cbc62df5 |
feat(bench): performance lane + task-matrix chart; docs: README rebuilt around four evidence lanes
- benchmarks/perf_suite.py: fixed headless workloads (2k creates, 10k build/export/reopen, 10k region query, premium quality pass) timed through the same backend methods the tools call; machine fingerprint recorded; self-measurement only - benchmarks/render_perf_chart.py -> docs/assets/autocad-mcp-perf.svg (log-scale wall time) - benchmarks/render_matrix_chart.py -> docs/assets/autocad-mcp-taskmatrix.svg (tasks x servers status heatmap from published reports) - render_live_chart now tolerates other lanes sharing results/published/ - README: PyPI-first install, four-lane benchmark section with Python-generated visuals, v1.4 feature table, updated client/config/roadmap - published perf report: benchmarks/results/published/perf-ezdxf.json Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Umutcan EdizsalanandCursor
|
6d7c39e012 |
feat(bench): live competitor lane - black-box MCP stdio adapters, pinned checkouts, published results
- adapters/mcp_stdio.py: drives competitors over MCP stdio like a real host; payload-level error detection, fastmcp output-wrapper normalization, cold-start warmup, artifact-based verification (harness re-opens every exported DXF with ezdxf) - competitors_env.py: reproducible pinned-SHA checkout + isolated venv per competitor - adapters for puran-water/autocad-mcp (95476a3) and beiming183-cloud/AutoCAD-MCP (11f7c47) mapped to their own documented tool contracts; unmapped tasks report 'unsupported' with reasons - run_competitors.py: registry-driven --server from competitors.yaml - published reports (pro 100.0 / beiming 50.0 / puran 45.0) + render_live_chart.py -> docs/assets/autocad-mcp-livebench.svg - benchmark.yml: weekly + manual dispatch matrix Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Umutcan Edizsalan
|
4a2e9d658f | release: prepare v1.3.0 | ||
|
Umutcan Edizsalan
|
b092c1a539 | docs: explain benchmark evidence boundaries | ||
|
Umutcan Edizsalan
|
5583730a79 |
docs(benchmarks): reproducible v1.0.0 -> v1.1.0 correctness A/B suite
- benchmarks/correctness_suite.py: 21 deterministic headless (ezdxf) checks across core ops, dimensions, arrays, geometry, query/parity, render, the quality gate, and engineering. - benchmarks/compare_versions.py: one-command A/B runner — checks out an older ref into a throwaway worktree and runs the same suite against both, each check in its own subprocess so a crash is a miss, not a run-killer. - benchmarks/README.md: methodology, results table, honesty caveats. - README: Benchmarks section now reports the measured result — v1.0.0 8/21 (38.1%) -> v1.1.0 21/21 (100%); 13 fixed, 0 regressions. |