Commit Graph
4 Commits
Author SHA1 Message Date
Umut CANandClaude Opus 5.5 898e775f93 bench: matrix v5 adds mech_assembly; the correctness suite grows with the mechanical and sheet checks
The evidence for tracks B and G, in the shape tracks A and E used.

- benchmarks/tasks_v5.py: 18 tasks, DEFAULT_MATRIX = "v5"; v4, v3 and v2 stay
  addressable so an earlier report reproduces exactly rather than being only
  described. mech_assembly is roadmap criterion 3 - a flange-coupling sheet
  built from the part model and the sheet standard, through the code the
  tools call (apply_titleblock, draw_part/add_view/dimension_part,
  insert_std_part, extract_records -> draw_bom_table, add_balloon), gated on
  two numbers that cannot be negotiated: drawing_critique(focus=None) returns
  zero issues and the finalize score is at least 90 (it scores 100, 18/18 on
  the v5 matrix). The adapter returns the focuses that fired, so a
  regression names itself, and a test drops one balloon to prove the gate
  fails and names mech_bom_balloon_mismatch.
- Seven correctness checks (32 -> 39). mech_section_hatch_area is the one
  that cannot be faked: a sleeve of length 60, outside diameter 40 and bore
  20 cuts two faces totalling 1200 mm^2 exactly, read back out of the drawing
  through entity_measure. sheet_frame_iso5457_a3 pins the A3 frame (20,10 to
  410,287 inside the 420 x 297 trimmed sheet); mech_iso286_on_dimension pins
  40 H7 at +0.025/0 through the layout; std_part_iso4014_m12 pins s = 18 and
  k = 7.5 by value; mech_thread_unrepresented_is_caught is the seventh: a
  quality gate that never fires is not a gate.
- scripts/smoke_mech_com.py executed every new COM path once on AutoCAD 2026
  (25.1s): frame, title block, a sectioned shaft with 12 dimensions, an ISO
  4014 block with 1 ACADMCP_MECH chunk, a weld symbol, the parts list read off
  the drawing, a linked balloon, an xref attached / listed / detached, a
  31687-byte .dwg (AC1032) through ActiveX SaveAs, critique 0 issues, exit 0,
  every scratch document closed. It retries RPC_E_CALL_REJECTED (measured:
  the first call after Documents.Add is refused) and tracks its documents by
  the document list, since SaveAs renames them.
- Defects the evidence found, each fixed with a test that fails on the old
  code:
  * ezdxf dimension_diameter put the text beyond the SECOND chord point;
    ActiveX AddDimDiametric puts it beyond the first (measured live: a d50
    from (x,170) to (x,220), leader 10, text at (x,160)). Headlessly every
    front-view diameter stood on its own chain dimension. ezdxf now follows
    ActiveX.
  * draw_part's anchor goes on MECH; ActiveX refuses entity.Layer on a
    missing layer (measured -0x7ffdfff7 'Key not found'). draw.py ensures
    every target layer before the first entity.
  * bom.extract_records filtered on INSERT; ActiveX names it BLOCKREFERENCE
    (ObjectName AcDbBlockReference) - the live parts list was empty.
  * COM _entity_info reported no dimension text_position: ModelSpace.Item
    narrows to IAcadEntity, which has no TextPosition (measured
    AttributeError); read through _com_unnarrow (TextPosition measured on
    AcDbDiametricDimension, WCS). dim_overlap compared bbox centres and a
    diameter collided with its own bore.
  * pid graph raised KeyError 'start' mid-critique on a LINE whose point read
    AutoCAD refused while busy; such a line is left out.
  * the validator read "ISO 4014 - M12x60" as fake dimension text.
- The published A/B report is re-run over the grown suite rather than
  re-typed (26/39 -> 39/39, 13 fixed, 0 regressed), check_doc_numbers derives
  the task count from the v5 matrix, the token table is remeasured with a
  TOOL_PACKS=core,mech row (193 tools / 61,673 tokens), and the README's
  collected-tests figure is retaken (3990).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
2026-09-23 13:30:02 +03:00
Umut CANandClaude Opus 5 2be5e4180d docs(pid): track A shipped — CLAUDE.md, README, changelog with the live COM smoke record
scripts/smoke_pid_com.py runs the whole track against the live AutoCAD in
a new document. ComBackend.connect() is lazy by design, so the script reads
reachability from system_status()["connected"] rather than from connect()
as the plan sketched, and closes its document without saving unless --keep
is passed. On this machine (AutoCAD 2026, "25.1s (LMS Tech)") it exits 0:
5 nodes, 4 edges, 0 dangling, confidence_min 1.0, 0 critique issues,
2 XDATA chunks — identical to the headless engine on the same spec. The
first attempt exited 2 on a stale win32com gen_py cache (a package folder
holding only __pycache__); an environment fault, no backend change needed.

A/B against v1.5.1 published as ab-v1.5.1-vs-v1.6.0-dev.json: 26/29 ->
29/29, three miss -> pass, zero regressed. check_doc_numbers.py now names
the report and the two row labels in one place (AB_REPORT, AB_*_LABEL),
and README.md / benchmarks/README.md carry the new rows; the v1.5.0 and
v1.4.0 tables stay as history.

README: P&ID row, the showcase and catalogue images, the token table
re-measured (161 / 44,930 by default, 152 / 41,420 with TOOL_PACKS=core,
lean 50 / 14,007), every stale count reconciled (154 -> 166 registered,
149 -> 161 advertised, corpus 158/672 -> 162/731 by the same count, 19 ->
20 groups, 6 -> 8 resources, six-task matrix with pid_roundtrip), a
"Known limitations of the P&ID track" subsection, and the snapshot line at
this machine's 2423 collected tests (CI has not run on this tree; noted in
the changelog with the pre-existing 1369 vs 1363 drift).

CLAUDE.md: section 17, sections 5 and 7 counts, TOOL_PACKS, the two 1.6
contracts and the reader rules. Spec status -> implemented. The plan and
spec markdown were reformatted by ruff (fenced Python only) because
`ruff format --check .` was already red on them at HEAD and the gate
requires the whole tree clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
2026-09-16 13:34:06 +03:00
Umut CANandClaude Opus 5 8d9c444b76 bench: matrix v4 adds pid_roundtrip; correctness suite grows to 29 with three P&ID checks
tasks_v4.py extends v3 with one task the reader can fail on its own: the
example spec is drawn through pid_from_spec's code and read back by the
graph builder, which never sees the spec. The runner, the chart and the
doc-number gate now default to v4; --matrix v3 / v2 reproduce the earlier
sets, and tests/test_benchmark_v3.py labels its full-matrix run "v3"
explicitly now that the default has moved.

The plan's adapter check expected the two unnumbered lines to come back
with no line number; draw_line builds one from the default number_format
(bare sequence when size/service/spec/insulation are empty), so the task
asserts the two authored numbers on the two process lines and sequence-
built XDATA numbers on the other two instead.

The published A/B report still covers 26 checks; Task 21 re-runs it
against v1.5.1, which is what clears the remaining doc-number drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
2026-09-16 13:06:52 +03:00
Umut CANandClaude Opus 5 ca1c79fd01 fix: seven defects an adversarial audit found, and the gates that missed them
Release / test (full, windows-latest, 3.11) (push) Canceled after 0s
Release / test (pdf, ubuntu-latest, 3.11) (push) Canceled after 0s
Release / test (pdf, ubuntu-latest, 3.12) (push) Canceled after 0s
Release / build (push) Canceled after 0s
Release / publish-pypi (push) Canceled after 0s
Release / github-release (push) Canceled after 0s
Every one of these was reported ok/clean to the caller while being wrong.

Silently wrong numbers
- selection_filter(min_area) read `entity.dxf.get("area", 0.0)` for SOLID,
  TRACE and HATCH, none of which carry that attribute, so every hatch
  measured 0.0. A 1600 mm^2 hatch was excluded by min_area=1.0 while
  `filtered_by` asserted the filter had run, and COM (which reads ActiveX
  .Area) returned a different selection for the same call. It now answers
  from the measurement `entity_measure` uses, so the two cannot disagree.
- Every headless dimension rendered through ezdxf's bare `Standard` style:
  1.0 mm text with a comma decimal marker, against ISO 3098's 2.5 mm minimum
  and ISO 129's point -- while the document header itself said $DIMTXT 2.5.
  `ezdxf.new(setup=True)` is not the fix; measured, it renders a 100 mm
  dimension as "10000". The attributes are now stored explicitly.
- Saving writes `dimrnd = 0.0` onto the Standard style, and ezdxf applies
  `xround(value, dimrnd)` whenever the attribute is present at all --
  `xround(33.333, 0.0)` is 33. So on any reopened file, including one this
  server saved, every dimension was rounded to a whole unit before DIMDEC
  formatted it: a 12.75 mm feature dimensioned 13. AutoCAD means DIMRND 0 as
  *no* rounding; the sentinel is now discarded at every load site, and a real
  0.5 mm rounding is left alone.
- The DIM* header fold gave out-of-range values teeth they did not have when
  they died in the header: dim_decimals=-1 returned ok and then killed the
  next dimension with `ValueError: Format specifier missing precision` from
  inside ezdxf, and 40 printed forty digits of float noise onto the sheet.
  Ranges are now AutoCAD's own (DIMDEC 0-8, DIMZIN 0-15, lengths positive).

Security
- security.py recorded the policy as "sanitize_command and sanitize_lisp each
  have exactly ONE caller ... the policy is scoped to the CHANNEL". The
  premise was false. `_ensure_linetype_loaded` and `ComBackend.linetype_load`
  build `_-LINETYPE _LOAD {name} {file}` and hand it to SendCommand, which
  treats a newline and ';' as segment separators -- so a `linetype=` argument
  carrying either injected further AutoCAD commands, and the 36-verb denylist
  never saw that channel. Guarded with the DXF symbol-name rule (the
  injection characters fall out of it rather than being enumerated) plus
  validate_path for the file argument, which was also the one path-taking
  tool reaching the filesystem unchecked. The policy comment is corrected.

Gates that could not see
- drawing_critique returned zero issues on a quarantined document: all seven
  checks wrapped their backend call in a bare `except Exception`. CLAUDE.md
  rule 7 makes that call the gate an agent is told to trust. Only the
  quarantine now escapes; an ordinary broken check still degrades.
  `_check_dimension_legibility` also read `_doc` directly from the event loop
  thread, missing the gate and performing exactly the unsynchronised access
  the quarantine exists to prevent. `construction_clear` answered
  "deleted: 0" -- there was nothing to clear -- for the same reason.
- The iso128 focus checked layer lineweights only, so a sheet lettered at
  1.0 mm critiqued clean. It now reads the effective dimension height and
  separator, header-then-style, in the order the renderer resolves.
- check_doc_numbers.py derived five values it never checked, and those were
  the ones that had drifted. The collected-test count (README said 1274, the
  tree had 1369) and the A/B pass-rate and Fixed columns are now gated.
- benchmarks/README.md's v1.4.0 table was impossible on its face -- 21
  passing plus 5 fixed is 26, not 24 -- and contradicted the report it
  linked, because `ab-v1.4.0-vs-v1.5.0.json` was regenerated from a tree that
  already carried the dimension fixes and so never measured the v1.5.0 tag.
  Replaced with a real run: ab-v1.4.0-vs-v1.5.1.json, 21/26 -> 26/26.

Release gate
- release.yml ran one ubuntu/3.11 lane, so a Windows-only defect could reach
  PyPI with no Windows job having executed. That is not hypothetical: v1.5.0
  shipped `except pywintypes.com_error` raising NameError wherever pywin32 is
  absent. It now runs the same three lanes as ci.yml, Windows included.

1368 passed, 1 xfailed. ruff clean. Tool inventory and all 16 gated document
numbers match the tree.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-06 23:07:50 +03:00