release: 1.6.0
The v1.6 roadmap in five tracks: 154 -> 247 registered tools in 28 groups. Evidence re-measured for the release: - correctness A/B against v1.5.1: 26/48 -> 48/48, 22 fixed, 0 regressed (ab-v1.5.1-vs-v1.6.0.json, replacing the -dev report); - the reference adapter on matrix v7: 21/21 (autocad-mcp-pro.json, which was the v3 run of 1.5); the published-report test follows it to v7; - the 100-point rubric re-scored per category, every withheld point named: 93 (the competitor rows are their July snapshots, not re-reviewed); - the perf lane as medians of five alternating runs, beside a v1.4.0 comparison on one interpreter (perf-v1.4.0-vs-v1.6.0.json). Two roadmap criteria are not green, and the changelog, the README and the roadmap say so: criterion 5 is withdrawn (track D, Plant 3D, dropped) and criterion 7 is not met (creation 1.25x / 1.38x v1.4.0's time, still 1.17x / 1.20x with the timeout off). Carried to 1.7. README rewritten for 1.6: what is new, the mechanical and architectural sheets, the rubric and task-matrix charts, the measured limits (including the four feature tables that ship empty), and the 1.7 roadmap. server.json describes the 1.6 surface and declares TOOL_PACKS. CLAUDE.md corrects the ISO 286 letter set and the empty feature tables. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
This commit is contained in:
Umut CAN
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
7e7fe11a71
commit
17bf3a39ce
+74
-32
@@ -41,13 +41,14 @@ Unsupported tasks remain in the fixed-matrix denominator with score zero,
|
||||
preventing partial implementations from receiving an inflated score.
|
||||
|
||||
```bash
|
||||
python -m benchmarks.run_competitors --list --matrix v5
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v5 --json
|
||||
python -m benchmarks.run_competitors --list --matrix v7
|
||||
python -m benchmarks.run_competitors --server autocad-mcp-pro --backend ezdxf --matrix v7 --json
|
||||
python -m benchmarks.run_competitors --task table_mleader --task hatch_islands --json
|
||||
```
|
||||
|
||||
The release-machine ezdxf self-check is **18/18 (100.0)** on the v5 matrix
|
||||
(`--matrix v4` / `--matrix v3` / `--matrix v2` reproduce the earlier sets).
|
||||
The release-machine ezdxf self-check is **21/21 (100.0)** on the v7 matrix
|
||||
(`--matrix v6` … `--matrix v2` reproduce the earlier sets); the published
|
||||
report is [`results/published/autocad-mcp-pro.json`](results/published/autocad-mcp-pro.json).
|
||||
Repository stars and raw tool counts do not contribute to the score. Adapter
|
||||
registration lives in `competitors.yaml`.
|
||||
|
||||
@@ -145,13 +146,13 @@ MCP stdio, against commits pinned in `competitors.yaml`:
|
||||
|
||||
| Server | Matrix | Pinned | Score | Pass | Coverage |
|
||||
|---|---|---|---:|---:|---:|
|
||||
| autocad-mcp-pro (reference) | v3 (15) | working tree | 100.0 | 15/15 | 100% |
|
||||
| autocad-mcp-pro (reference) | v7 (21) | 1.6.0 tree | 100.0 | 21/21 | 100% |
|
||||
| beiming183-cloud/AutoCAD-MCP | v2 (10) | `11f7c47e` | 50.0 | 5/10 | 50% |
|
||||
| puran-water/autocad-mcp | v2 (10) | `95476a33` | 45.0 | 4/10 | 50% |
|
||||
|
||||
The competitor rows are the v1.4 runs, unchanged. Read the matrix column
|
||||
before the score column: the denominators differ because the five v3 tasks
|
||||
were added after those runs and have not been put to them.
|
||||
before the score column: the denominators differ because the eleven tasks
|
||||
added since v2 came after those runs and have not been put to them.
|
||||
|
||||
Method and boundaries:
|
||||
|
||||
@@ -205,38 +206,79 @@ python -m benchmarks.perf_suite --out benchmarks/results/published/perf-ezdxf.js
|
||||
python -m benchmarks.render_perf_chart
|
||||
```
|
||||
|
||||
### What the call-timeout guard costs
|
||||
### v1.4.0 against v1.6.0, and what the call-timeout guard costs
|
||||
|
||||
The published v1.4 report was recorded on CPython 3.14 and the later ones on
|
||||
3.11.15, so comparing those two files directly shows an apparent 2–6× speedup
|
||||
that is almost entirely the interpreter. The comparison that means something is
|
||||
the guard against no guard, on one interpreter, back to back. Three runs of
|
||||
each configuration, median taken by hand — `perf_suite` executes each workload
|
||||
once per invocation and cannot produce a median itself:
|
||||
3.11.15, so comparing those files directly shows an apparent speedup that is
|
||||
almost entirely the interpreter. The comparison that means something runs one
|
||||
interpreter and one ezdxf over each tree's own code: 1.6.0's `perf_suite.py`
|
||||
(v1.4.0 had none) against a v1.4.0 worktree, against 1.6.0, and against 1.6.0
|
||||
with the guard off — five rounds, the three configurations alternating within
|
||||
a round. Medians, from
|
||||
[`results/published/perf-v1.4.0-vs-v1.6.0.json`](results/published/perf-v1.4.0-vs-v1.6.0.json),
|
||||
which keeps every run:
|
||||
|
||||
| Workload | `EZDXF_CALL_TIMEOUT=0` | default (120 s) | Cost |
|
||||
| Workload | v1.4.0 | v1.6.0 | v1.6.0, `EZDXF_CALL_TIMEOUT=0` |
|
||||
|---|---:|---:|---:|
|
||||
| `create_lines_2k` | 243.7 ms | 294.3 ms | **+20.8%** |
|
||||
| `roundtrip_10k` | 1,709.6 ms | 2,049.2 ms | **+19.9%** |
|
||||
| `create_lines_2k` | 244.5 ms | 306.1 ms (1.25×) | 284.9 ms (1.17×) |
|
||||
| `roundtrip_10k` | 1,847.9 ms | 2,553.5 ms (1.38×) | 2,216.4 ms (1.20×) |
|
||||
| `region_query_10k` | 665.2 ms | 712.2 ms (1.07×) | 783.9 ms (1.18×) |
|
||||
| `premium_pass` | 150.6 ms | 65.3 ms (0.43×) | 69.9 ms (0.46×) |
|
||||
|
||||
v1.5.0 paid 28.6% on the first workload; v1.5.1 rebuilt the wrapper around a
|
||||
single `asyncio.timeout_at` instead of two `asyncio.wait_for`s — `wait_for`
|
||||
wraps its awaitable in a Task, so the old shape created two extra Tasks and two
|
||||
timer handles per call — which recovered about a third of the overhead. The
|
||||
rest is the Task the abandon-on-timeout design genuinely needs. The guard is
|
||||
what stops one hung ezdxf call wedging a server whose document lock is a single
|
||||
`asyncio.Lock`, so it stays; the knob is documented rather than hidden.
|
||||
The guard explains part of the creation cost, not all of it. v1.5.1 rebuilt
|
||||
the wrapper around one `asyncio.timeout_at` instead of two `asyncio.wait_for`s
|
||||
(`wait_for` wraps its awaitable in a Task, so the old shape created two extra
|
||||
Tasks and two timer handles per call), which recovered about a third of its
|
||||
overhead; the rest is the Task the abandon-on-timeout design needs. With the
|
||||
guard off, creation is still 1.17× and 1.20× v1.4.0's time. A cProfile of the
|
||||
event-loop thread over the 2,000 creates counts **491,071** function calls on
|
||||
1.6.0 against **319,038** on v1.4.0: per-call work added since 1.4 that 1.6
|
||||
has not removed. The region-query row, whose guard-off median is the slower
|
||||
one, is this machine's noise floor (about ±10 %).
|
||||
|
||||
These figures are hand-collected and have **no published artifact** —
|
||||
`perf-ezdxf.json` records one default-configuration run only. Reproduce:
|
||||
Roadmap criterion 7 — headless creation throughput back to v1.4.0's numbers —
|
||||
is therefore **not met** and moves to 1.7. Earlier versions of this page said
|
||||
that `EZDXF_CALL_TIMEOUT=0` returned creation to v1.4.0's numbers; measured
|
||||
this way, it does not. The guard stays: it is what stops one hung ezdxf call
|
||||
wedging a server whose document lock is a single `asyncio.Lock`.
|
||||
|
||||
`perf-ezdxf.json`, the chart's input, is the median of the five
|
||||
default-configuration runs, with every run kept in `runs_ms`.
|
||||
|
||||
Reproduce (the v1.4.0 tree needs 1.6.0's `benchmarks/perf_suite.py` copied in):
|
||||
|
||||
```bash
|
||||
for i in 1 2 3; do
|
||||
python -m benchmarks.perf_suite --out /tmp/on-$i.json
|
||||
EZDXF_CALL_TIMEOUT=0 python -m benchmarks.perf_suite --out /tmp/off-$i.json
|
||||
git worktree add ../v140 v1.4.0 && cp benchmarks/perf_suite.py ../v140/benchmarks/
|
||||
for i in 1 2 3 4 5; do
|
||||
python -m benchmarks.perf_suite --json > main-$i.json
|
||||
EZDXF_CALL_TIMEOUT=0 python -m benchmarks.perf_suite --json > off-$i.json
|
||||
(cd ../v140 && python -m benchmarks.perf_suite --json) > v140-$i.json
|
||||
done
|
||||
```
|
||||
|
||||
The profile, run from each tree's root:
|
||||
|
||||
```python
|
||||
import asyncio, cProfile, pstats
|
||||
from backends.ezdxf_backend import EzdxfBackend
|
||||
|
||||
|
||||
async def work():
|
||||
b = EzdxfBackend()
|
||||
await b.connect()
|
||||
await b.drawing_new()
|
||||
for i in range(2000):
|
||||
await b.entity_create_line(0, i, 100, i)
|
||||
await b.disconnect()
|
||||
|
||||
|
||||
pr = cProfile.Profile()
|
||||
pr.enable()
|
||||
asyncio.run(work())
|
||||
pr.disable()
|
||||
print(pstats.Stats(pr).total_calls)
|
||||
```
|
||||
|
||||
## Correctness A/B suite
|
||||
|
||||
`correctness_suite.py` is a set of deterministic, headless (ezdxf-backend) checks
|
||||
@@ -253,15 +295,15 @@ python benchmarks/compare_versions.py v1.0.0 # vs a tag/ref
|
||||
python benchmarks/compare_versions.py --json results.json
|
||||
```
|
||||
|
||||
### Result — this branch (1.6.0-dev) vs v1.5.1 (release gate)
|
||||
### Result — v1.6.0 vs v1.5.1 (release gate)
|
||||
|
||||
48 checks, ezdxf backend, one subprocess per check. Machine-readable report:
|
||||
[`results/published/ab-v1.5.1-vs-v1.6.0-dev.json`](results/published/ab-v1.5.1-vs-v1.6.0-dev.json).
|
||||
[`results/published/ab-v1.5.1-vs-v1.6.0.json`](results/published/ab-v1.5.1-vs-v1.6.0.json).
|
||||
|
||||
| Version | Checks passing | Pass rate | Fixed | Regressed |
|
||||
|---------|----------------|-----------|-------|-----------|
|
||||
| **v1.5.1** (baseline) | 26 / 48 | 54.2 % | — | — |
|
||||
| **v1.6.0-dev** (this branch) | 48 / 48 | 100 % | 22 | **0** |
|
||||
| **v1.6.0** (this release) | 48 / 48 | 100 % | 22 | **0** |
|
||||
|
||||
The twenty-two are all `miss → pass` — v1.5.1 has none of the methods. Five are track H: `scale_check_detects_schematic` (the synthetic P&ID is schematic against its layout; a uniform 2× copy is to scale with factor 2), `pipe_takeoff_rmst_exact` (eight routable runs, each exactly the rectilinear MST of its tags), `cable_takeoff_roundup` (9 / 17 / 19 / 10 m, the unstated power left empty), `diff_detects_known_edits` (five known edits and nothing else) and `topology_known_defects` (one near miss, one crossing, a clean T). Four are track F: `arch_junction_l_t_x` (L, T and X junction outlines against hand-computed corners and wall areas), `arch_room_area_net` (a 4 × 5 m room between 200 mm walls is labelled 20.00 m², not its 21.84 m² axis area), `arch_opening_cuts_wall` (both faces interrupted across a door, two jambs) and `arch_rooms_detect_foreign` (plain lines on a WALLS layer read as 20 and 15 m² at confidence 0.6, nothing written). Seven
|
||||
are tracks B and G: `mech_part_roundtrip` (a part read back from its own
|
||||
@@ -286,7 +328,7 @@ through the lister; `settings_layer_state_roundtrip`: save → change → restor
|
||||
puts the layer table back and the state survives save/reopen as an XRECORD in
|
||||
the file; `settings_pdf_mediabox_a3`: the plotted PDF's own `/MediaBox` reads
|
||||
420 × 297 mm). Every one of the 26 checks it was released on still passes, so
|
||||
the four tracks added capability without moving a number it had already earned.
|
||||
the five tracks added capability without moving a number it had already earned.
|
||||
|
||||
### Result — v1.5.1 vs v1.5.0
|
||||
|
||||
|
||||
@@ -95,7 +95,7 @@ def render_chart(data: dict[str, Any], output: Path) -> None:
|
||||
ax.set_title(
|
||||
"Public AutoCAD MCP leaderboard — Source-reviewed capability benchmark",
|
||||
loc="left",
|
||||
fontsize=17,
|
||||
fontsize=15,
|
||||
fontweight="bold",
|
||||
color="#e6edf3",
|
||||
pad=28,
|
||||
|
||||
@@ -103,8 +103,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||
ax.text(
|
||||
0,
|
||||
-0.22,
|
||||
"Unsupported tasks score 0 within a server's own matrix. The five v3 tasks were "
|
||||
"added after the competitor runs and were never put to them; per-task statuses are "
|
||||
"Unsupported tasks score 0 within a server's own matrix. The tasks added after v2 "
|
||||
"were never put to the pinned competitor runs; per-task statuses are "
|
||||
"in benchmarks/results/published/.",
|
||||
transform=ax.transAxes,
|
||||
color="#8b949e",
|
||||
|
||||
@@ -133,8 +133,8 @@ def render_chart(reports: list[dict[str, Any]], output: Path) -> None:
|
||||
Patch(facecolor=STATUS_COLORS["fail"], label="fail"),
|
||||
Patch(facecolor=STATUS_COLORS["unsupported"], label="unsupported (no equivalent)"),
|
||||
# Real rows now, not a theoretical state: the competitor reports were
|
||||
# pinned and run against the v2 matrix, so the five v3 tasks were
|
||||
# never put to them. An invented zero would look identical to a
|
||||
# pinned and run against the v2 matrix, so every task added since v2
|
||||
# was never put to them. An invented zero would look identical to a
|
||||
# measured one, so they stay blank and say why.
|
||||
Patch(facecolor=STATUS_COLORS["not_run"], edgecolor="#30363d", label="not run"),
|
||||
]
|
||||
|
||||
@@ -0,0 +1,302 @@
|
||||
{
|
||||
"summary": {
|
||||
"baseline_ref": "v1.5.1",
|
||||
"total": 48,
|
||||
"old_pass": 26,
|
||||
"new_pass": 48,
|
||||
"old_pct": 54.2,
|
||||
"new_pct": 100.0,
|
||||
"fixed": 22,
|
||||
"regressed": 0
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"check": "core_line_length",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_circle_radius",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_layer_create",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_polyline_closed",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_linear_dim",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "core_save_dxf_roundtrip",
|
||||
"category": "Core",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "dim_aligned_no_error",
|
||||
"category": "Dimensions",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "dim_angular_no_error",
|
||||
"category": "Dimensions",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "array_polar_360_distinct",
|
||||
"category": "Modify",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "point_intersection_line_line",
|
||||
"category": "Geometry",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "point_tangent_external",
|
||||
"category": "Geometry",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arc_has_length",
|
||||
"category": "Query",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arc_select_by_length",
|
||||
"category": "Query",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "ezdxf_bounding_box",
|
||||
"category": "Query",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mtext_rotation_roundtrip",
|
||||
"category": "Entities",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "screenshot_png",
|
||||
"category": "Render",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "dim_overlap_critique_fires",
|
||||
"category": "Quality gate",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "iso13567_dim_layer",
|
||||
"category": "Quality gate",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "construction_left_iso_caught",
|
||||
"category": "Quality gate",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "gear_no_self_overlap",
|
||||
"category": "Engineering",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "center_linetype_applied",
|
||||
"category": "Entities",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "hatch_area_subtracts_its_island",
|
||||
"category": "Measurement",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "boundary_area_agrees_with_measure",
|
||||
"category": "Measurement",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "two_vertex_circle_has_area",
|
||||
"category": "Measurement",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "diameter_dim_measures_the_diameter",
|
||||
"category": "Dimensions",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "radius_dim_ignores_the_leader_length",
|
||||
"category": "Dimensions",
|
||||
"old": "pass",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "pid_block_define_attdef_roundtrip",
|
||||
"category": "P&ID",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "pid_tag_parse_fic",
|
||||
"category": "P&ID",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "pid_graph_edge_count",
|
||||
"category": "P&ID",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "settings_dimstyle_iso25_values",
|
||||
"category": "Settings",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "settings_layer_state_roundtrip",
|
||||
"category": "Settings",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "settings_pdf_mediabox_a3",
|
||||
"category": "Settings",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_part_roundtrip",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_section_hatch_area",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_iso286_on_dimension",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "mech_thread_unrepresented_is_caught",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "std_part_iso4014_m12",
|
||||
"category": "Mechanical",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "sheet_frame_iso5457_a3",
|
||||
"category": "Sheet",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "bom_balloon_link",
|
||||
"category": "Sheet",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arch_junction_l_t_x",
|
||||
"category": "Architecture",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arch_room_area_net",
|
||||
"category": "Architecture",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arch_opening_cuts_wall",
|
||||
"category": "Architecture",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "arch_rooms_detect_foreign",
|
||||
"category": "Architecture",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "scale_check_detects_schematic",
|
||||
"category": "Understanding",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "pipe_takeoff_rmst_exact",
|
||||
"category": "Understanding",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "cable_takeoff_roundup",
|
||||
"category": "Understanding",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "diff_detects_known_edits",
|
||||
"category": "Understanding",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
},
|
||||
{
|
||||
"check": "topology_known_defects",
|
||||
"category": "Understanding",
|
||||
"old": "miss",
|
||||
"new": "pass"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -25,6 +25,11 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"documents": {
|
||||
"mode": "registry",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"drawing_2d": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
@@ -35,11 +40,31 @@
|
||||
"reason": "no_dwg_writer_requires_live_autocad",
|
||||
"supported": false
|
||||
},
|
||||
"dwg_write": {
|
||||
"mode": null,
|
||||
"reason": "requires_oda_file_converter:https://www.opendesign.com/guestfiles/oda_file_converter",
|
||||
"supported": false
|
||||
},
|
||||
"dwgprops": {
|
||||
"mode": null,
|
||||
"reason": "summary_fields_need_live_autocad;custom_properties_need_r2004_or_newer",
|
||||
"supported": false
|
||||
},
|
||||
"dwt_write": {
|
||||
"mode": null,
|
||||
"reason": "dwt_is_a_dwg_container_requires_live_autocad",
|
||||
"supported": false
|
||||
},
|
||||
"dxf": {
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"explode_opaque_members": {
|
||||
"mode": null,
|
||||
"reason": "ezdxf_cannot_transform:OLE2FRAME,VIEWPORT,ACAD_PROXY_ENTITY_without_proxy_graphic;refused_before_writing",
|
||||
"supported": false
|
||||
},
|
||||
"handle_overlay": {
|
||||
"mode": "rendered",
|
||||
"reason": null,
|
||||
@@ -50,11 +75,26 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"interactive_prompt": {
|
||||
"mode": null,
|
||||
"reason": "no_operator_to_prompt_headlessly",
|
||||
"supported": false
|
||||
},
|
||||
"layer_states": {
|
||||
"mode": "xrecord",
|
||||
"reason": "portable_acadmcp_xrecord;not_listed_in_autocad_layer_states_manager",
|
||||
"supported": true
|
||||
},
|
||||
"lisp": {
|
||||
"mode": null,
|
||||
"reason": "live_com_only",
|
||||
"supported": false
|
||||
},
|
||||
"live_application": {
|
||||
"mode": null,
|
||||
"reason": "headless_engine_has_no_application_to_launch_or_attach",
|
||||
"supported": false
|
||||
},
|
||||
"measure_area_acis": {
|
||||
"mode": null,
|
||||
"reason": "acis_evaluation_requires_live_autocad",
|
||||
@@ -70,6 +110,11 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"named_views": {
|
||||
"mode": "vport_active",
|
||||
"reason": "restore_sets_the_view_the_file_opens_on;no_live_display",
|
||||
"supported": true
|
||||
},
|
||||
"ocs_normalized": {
|
||||
"mode": "wcs",
|
||||
"reason": "reported_xy_is_wcs;z_component_not_reported;text_rotation_stays_in_entity_frame;hatch_boundaries_not_reported",
|
||||
@@ -95,6 +140,11 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"preferences": {
|
||||
"mode": null,
|
||||
"reason": "preferences_live_in_the_running_application_not_in_a_file",
|
||||
"supported": false
|
||||
},
|
||||
"preflight": {
|
||||
"mode": "shared",
|
||||
"reason": null,
|
||||
@@ -105,6 +155,11 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"registry_sysvar": {
|
||||
"mode": null,
|
||||
"reason": "registry_saved_variables_have_no_home_in_a_file",
|
||||
"supported": false
|
||||
},
|
||||
"revcloud": {
|
||||
"mode": "native",
|
||||
"reason": "ezdxf_revcloud_helper",
|
||||
@@ -125,6 +180,11 @@
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"ucs": {
|
||||
"mode": "stored_not_interpreted",
|
||||
"reason": "tool_coordinates_stay_wcs",
|
||||
"supported": true
|
||||
},
|
||||
"undo_history": {
|
||||
"mode": null,
|
||||
"reason": "set_EZDXF_UNDO_DEPTH_to_enable",
|
||||
@@ -139,6 +199,16 @@
|
||||
"mode": "native",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"xlsx_write": {
|
||||
"mode": "openpyxl",
|
||||
"reason": null,
|
||||
"supported": true
|
||||
},
|
||||
"xref_live": {
|
||||
"mode": null,
|
||||
"reason": "reload_and_bind_need_a_live_seat;list_detach_path_are_headless",
|
||||
"supported": false
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -149,13 +219,13 @@
|
||||
"platform": "Windows-10-10.0.26200-SP0",
|
||||
"python": "3.11.15"
|
||||
},
|
||||
"generated_at": "2026-08-06T06:27:59.080169+00:00",
|
||||
"git_sha": "d5a0ff5109ddbd37d88371c10a1ed087201de4b5",
|
||||
"matrix": "v3",
|
||||
"generated_at": "2026-09-24T20:58:06.608758+00:00",
|
||||
"git_sha": "554daff8d08c7d891974e55db046e7c70c15e185",
|
||||
"matrix": "v7",
|
||||
"results": [
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 2.49,
|
||||
"duration_ms": 2.71,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"line_length": 5.0
|
||||
@@ -168,7 +238,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 1.41,
|
||||
"duration_ms": 1.92,
|
||||
"message": "",
|
||||
"metrics": {},
|
||||
"score": 100.0,
|
||||
@@ -179,7 +249,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 3.31,
|
||||
"duration_ms": 3.38,
|
||||
"message": "",
|
||||
"metrics": {},
|
||||
"score": 100.0,
|
||||
@@ -190,7 +260,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 17.55,
|
||||
"duration_ms": 24.95,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"entity_type": "DIMENSION"
|
||||
@@ -203,7 +273,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 3.7,
|
||||
"duration_ms": 3.64,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"representations": [
|
||||
@@ -219,7 +289,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 29.09,
|
||||
"duration_ms": 8.37,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"entity_count": 0
|
||||
@@ -232,7 +302,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 1.2,
|
||||
"duration_ms": 0.85,
|
||||
"message": "",
|
||||
"metrics": {},
|
||||
"score": 100.0,
|
||||
@@ -243,7 +313,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 8.42,
|
||||
"duration_ms": 17.45,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"final_score": 100.0,
|
||||
@@ -286,11 +356,11 @@
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "roundtrip.dxf",
|
||||
"sha256": "d11dfce5eddaf849075c62854a8526efaaa3bbd779d374e4c287de77f361caf7",
|
||||
"size_bytes": 18440
|
||||
"sha256": "28a18050aea00646f9ff7a5fdf8fe3b8783cd4cb77d71108b3834c746e7259c5",
|
||||
"size_bytes": 18446
|
||||
}
|
||||
],
|
||||
"duration_ms": 32.29,
|
||||
"duration_ms": 7.21,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"entity_count": 1
|
||||
@@ -305,11 +375,11 @@
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "manifest.json",
|
||||
"sha256": "a74b2e96dbe7ea373717f26a8d26e58fe079a61370eb5d646f04a09f1b7cca2f",
|
||||
"size_bytes": 5662
|
||||
"sha256": "7178055a030b533e83c4a914466c7474350e7728657cd624c9ebc01c753d9e13",
|
||||
"size_bytes": 7929
|
||||
}
|
||||
],
|
||||
"duration_ms": 305.81,
|
||||
"duration_ms": 274.87,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"score": {
|
||||
@@ -329,7 +399,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 1149.77,
|
||||
"duration_ms": 699.92,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"ranks": {
|
||||
@@ -350,15 +420,15 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 53.97,
|
||||
"duration_ms": 86.79,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"advertised_tools": 2,
|
||||
"catalog_tokens": 40305,
|
||||
"catalog_tools": 149,
|
||||
"catalog_tokens": 83921,
|
||||
"catalog_tools": 242,
|
||||
"ceiling": 2000,
|
||||
"discovery_tokens": 356,
|
||||
"reduction_x": 113.2,
|
||||
"reduction_x": 235.7,
|
||||
"tokenizer": "ratio/v1 (estimate)"
|
||||
},
|
||||
"score": 100.0,
|
||||
@@ -369,7 +439,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 10.6,
|
||||
"duration_ms": 7.47,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"area_ignoring_islands": 400.0,
|
||||
@@ -386,7 +456,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 8.0,
|
||||
"duration_ms": 26.1,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"bounding_box": 3,
|
||||
@@ -403,7 +473,7 @@
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 5.36,
|
||||
"duration_ms": 5.83,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"boundary_agrees": true,
|
||||
@@ -417,16 +487,330 @@
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "measure_from_handle"
|
||||
},
|
||||
{
|
||||
"artifacts": [],
|
||||
"duration_ms": 58.19,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"authored_line_numbers_kept": true,
|
||||
"confidence_min": 1.0,
|
||||
"critique_issues": 0,
|
||||
"dangling": 0,
|
||||
"edges": 4,
|
||||
"instrument_index_rows": 1,
|
||||
"line_list_rows": 4,
|
||||
"nodes": 5,
|
||||
"unnumbered_lines_sequenced": true
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "pid_roundtrip"
|
||||
},
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "untitled-Layout1-ansi_b.pdf",
|
||||
"sha256": "af3c19d4d8b14fbc116ab807bb6feee9fc900fc114e57cd6feb658a6c0dabffb",
|
||||
"size_bytes": 1287
|
||||
},
|
||||
{
|
||||
"filename": "untitled-Layout1-iso_a3.pdf",
|
||||
"sha256": "8d1d1ed505903b6f692ed6e91e9f52577c2c78db3bc2942b949ea4a3d73c9b01",
|
||||
"size_bytes": 1287
|
||||
}
|
||||
],
|
||||
"duration_ms": 65.46,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"mediabox_mm": [
|
||||
420.0,
|
||||
297.0
|
||||
],
|
||||
"mediabox_mm_before": [
|
||||
432.0,
|
||||
279.0
|
||||
],
|
||||
"orientation": "landscape",
|
||||
"paper": "ISO_A3",
|
||||
"pdf_bytes": 1287,
|
||||
"plot_style_known": true,
|
||||
"size_mm_changed": [
|
||||
[
|
||||
432.0,
|
||||
279.0
|
||||
],
|
||||
[
|
||||
420.0,
|
||||
297.0
|
||||
]
|
||||
],
|
||||
"size_mm_listed": [
|
||||
420.0,
|
||||
297.0
|
||||
]
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "page_setup_truth"
|
||||
},
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "mech_assembly.dxf",
|
||||
"sha256": "407dcbed3eb17be7f9d72dda2c1951e8f97fa3fb6090713440404a3f1497d1b2",
|
||||
"size_bytes": 101158
|
||||
}
|
||||
],
|
||||
"duration_ms": 487.19,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"critique_focuses": [],
|
||||
"critique_issues": 0,
|
||||
"fasteners": 4,
|
||||
"grade": "A",
|
||||
"invalidity_ratio": 0.0,
|
||||
"parts_list_rows": 2,
|
||||
"score": 100.0,
|
||||
"validator_findings": []
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "mech_assembly"
|
||||
},
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "arch_roundtrip.dxf",
|
||||
"sha256": "ebec378ef28670e091e651b4130a6eac5a0d2fb480d60f9335b37de28ea713c7",
|
||||
"size_bytes": 82829
|
||||
}
|
||||
],
|
||||
"duration_ms": 982.07,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"confidence_min": 1.0,
|
||||
"critique_focuses": [],
|
||||
"critique_issues": 0,
|
||||
"door_tags": [
|
||||
"D1",
|
||||
"D2"
|
||||
],
|
||||
"faces_detected": 2,
|
||||
"grade": "A",
|
||||
"invalidity_ratio": 0.0,
|
||||
"omitted": [],
|
||||
"rooms": {
|
||||
"01": {
|
||||
"agrees": true,
|
||||
"detected_mm2": 27743750.0,
|
||||
"hand_mm2": 27743750.0,
|
||||
"label_mm2": 27743750.0
|
||||
},
|
||||
"02": {
|
||||
"agrees": true,
|
||||
"detected_mm2": 21993750.0,
|
||||
"hand_mm2": 21993750.0,
|
||||
"label_mm2": 21993750.0
|
||||
}
|
||||
},
|
||||
"score": 100.0,
|
||||
"validator_findings": [],
|
||||
"window_tags": [
|
||||
"W1",
|
||||
"W2"
|
||||
]
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "arch_roundtrip"
|
||||
},
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "pid.dxf",
|
||||
"sha256": "94dbdc026878927560a6c81e2283a7d2a26fbe6a398475d415c6383728c46672",
|
||||
"size_bytes": 33795
|
||||
},
|
||||
{
|
||||
"filename": "layout.dxf",
|
||||
"sha256": "37963e237f27f9f495b30d7c0b3bf7f4eff54ff4fec9665f34ab50ddc3f3c7a7",
|
||||
"size_bytes": 40503
|
||||
}
|
||||
],
|
||||
"duration_ms": 614.2,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"cable_mismatch": [],
|
||||
"length_source": "layout",
|
||||
"loads_checked": 4,
|
||||
"pipe_mismatch": [],
|
||||
"runs_checked": 9,
|
||||
"scale_verdict": "schematic",
|
||||
"within_10pct": 0.21818181818181817
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "takeoff_roundtrip"
|
||||
},
|
||||
{
|
||||
"artifacts": [
|
||||
{
|
||||
"filename": "layout.dxf",
|
||||
"sha256": "ff728965a5698cf9ddf9d9a01cfa244102d183065a149488d4d29dbcee404e85",
|
||||
"size_bytes": 40503
|
||||
},
|
||||
{
|
||||
"filename": "pid.dxf",
|
||||
"sha256": "de4f1ab4102d220efa7167b192b8fdaf9d57cbf300241e945c16967770401cd8",
|
||||
"size_bytes": 33795
|
||||
}
|
||||
],
|
||||
"duration_ms": 311.97,
|
||||
"message": "",
|
||||
"metrics": {
|
||||
"clusters": 2,
|
||||
"copies_separated": true,
|
||||
"misread_layers": {},
|
||||
"outlier_found": true,
|
||||
"units": {
|
||||
"agree": false,
|
||||
"confidence": 0.76,
|
||||
"declared": {
|
||||
"code": 1,
|
||||
"label": "Inches",
|
||||
"name": "in"
|
||||
},
|
||||
"evidence": [
|
||||
{
|
||||
"as": {
|
||||
"cm": 2.5,
|
||||
"ft": 76.2,
|
||||
"in": 6.35,
|
||||
"m": 250.0,
|
||||
"mm": 0.25
|
||||
},
|
||||
"band": [
|
||||
0.0018,
|
||||
4.0
|
||||
],
|
||||
"band_unit": "m",
|
||||
"count": 26,
|
||||
"informative": true,
|
||||
"kind": "text_height",
|
||||
"plausible": [
|
||||
"mm",
|
||||
"cm"
|
||||
],
|
||||
"value": 250.0,
|
||||
"weight": 0.5
|
||||
},
|
||||
{
|
||||
"as": {
|
||||
"cm": 2.0,
|
||||
"ft": 60.96,
|
||||
"in": 5.08,
|
||||
"m": 200.0,
|
||||
"mm": 0.2
|
||||
},
|
||||
"band": [
|
||||
0.05,
|
||||
1.0
|
||||
],
|
||||
"band_unit": "m",
|
||||
"count": 24,
|
||||
"informative": true,
|
||||
"kind": "wall_thickness",
|
||||
"plausible": [
|
||||
"mm"
|
||||
],
|
||||
"value": 200.0,
|
||||
"weight": 1.0
|
||||
},
|
||||
{
|
||||
"as": {
|
||||
"cm": 21480.0,
|
||||
"ft": 19955572.992000002,
|
||||
"in": 138580.368,
|
||||
"m": 214800000.0,
|
||||
"mm": 214.79999999999998
|
||||
},
|
||||
"band": [
|
||||
1.0,
|
||||
5000.0
|
||||
],
|
||||
"band_unit": "m2",
|
||||
"count": 4,
|
||||
"informative": true,
|
||||
"kind": "room_area",
|
||||
"plausible": [
|
||||
"mm"
|
||||
],
|
||||
"value": 214800000.0,
|
||||
"weight": 1.0
|
||||
},
|
||||
{
|
||||
"as": {
|
||||
"cm": 1364.0,
|
||||
"ft": 41574.72,
|
||||
"in": 3464.56,
|
||||
"m": 136400.0,
|
||||
"mm": 136.4
|
||||
},
|
||||
"band": [
|
||||
0.01,
|
||||
2000.0
|
||||
],
|
||||
"band_unit": "m",
|
||||
"count": 80,
|
||||
"informative": true,
|
||||
"kind": "overall_size",
|
||||
"plausible": [
|
||||
"mm",
|
||||
"cm"
|
||||
],
|
||||
"value": 136400.0,
|
||||
"weight": 0.25
|
||||
}
|
||||
],
|
||||
"inferred": "mm",
|
||||
"mm_per_unit": 1.0,
|
||||
"scores": {
|
||||
"cm": 0.75,
|
||||
"ft": 0,
|
||||
"in": 0.5,
|
||||
"m": 0,
|
||||
"mm": 2.75
|
||||
},
|
||||
"warning": "INSUNITS declares Inches (1) but the geometry reads as mm: text_height 250 = 0.25 m as mm, 6.35 m as in; wall_thickness 200 = 0.2 m as mm, 5.08 m as in; room_area 2.148e+08 = 214.8 m2 as mm, 1.386e+05 m2 as in; overall_size 136400 = 136.4 m as mm, 3465 m as in. Lengths are measured in mm; the header is not changed."
|
||||
},
|
||||
"units_flagged": true
|
||||
},
|
||||
"score": 100.0,
|
||||
"status": "pass",
|
||||
"stderr_summary": "",
|
||||
"stdout_summary": "",
|
||||
"task_id": "understand_foreign"
|
||||
}
|
||||
],
|
||||
"schema_version": "2.0",
|
||||
"summary": {
|
||||
"attempted": 15,
|
||||
"attempted": 21,
|
||||
"coverage_percent": 100.0,
|
||||
"passed": 15,
|
||||
"passed": 21,
|
||||
"score": 100.0,
|
||||
"supported": 15,
|
||||
"supported": 21,
|
||||
"unsupported": 0
|
||||
},
|
||||
"timeout_seconds": 180.0
|
||||
"timeout_seconds": 30.0
|
||||
}
|
||||
|
||||
@@ -14,23 +14,37 @@
|
||||
"entity_count": 2000
|
||||
},
|
||||
"metric": "entities_per_second",
|
||||
"metric_value": 5715.8,
|
||||
"wall_ms": 349.9,
|
||||
"workload_id": "create_lines_2k"
|
||||
"wall_ms": 306.1,
|
||||
"workload_id": "create_lines_2k",
|
||||
"runs_ms": [
|
||||
287.6,
|
||||
373.6,
|
||||
298.2,
|
||||
425.1,
|
||||
306.1
|
||||
],
|
||||
"metric_value": 6533.8
|
||||
},
|
||||
{
|
||||
"description": "Create 10,000 lines, export DXF, reopen and verify",
|
||||
"detail": {
|
||||
"build_ms": 1633.1,
|
||||
"build_ms": 2149.4,
|
||||
"entity_count": 10000,
|
||||
"export_ms": 273.0,
|
||||
"file_bytes": 1369663,
|
||||
"reopen_ms": 272.1
|
||||
"export_ms": 516.6,
|
||||
"file_bytes": 1369662,
|
||||
"reopen_ms": 535.2
|
||||
},
|
||||
"metric": "entities_per_second",
|
||||
"metric_value": 4590.8,
|
||||
"wall_ms": 2178.3,
|
||||
"workload_id": "roundtrip_10k"
|
||||
"wall_ms": 2553.5,
|
||||
"workload_id": "roundtrip_10k",
|
||||
"runs_ms": [
|
||||
3201.3,
|
||||
2550.0,
|
||||
3118.3,
|
||||
2553.5,
|
||||
2289.4
|
||||
],
|
||||
"metric_value": 3916.2
|
||||
},
|
||||
{
|
||||
"description": "Crossing-selection region query over 10,000 entities",
|
||||
@@ -39,8 +53,15 @@
|
||||
},
|
||||
"metric": "entities_matched",
|
||||
"metric_value": 2500.0,
|
||||
"wall_ms": 2466.9,
|
||||
"workload_id": "region_query_10k"
|
||||
"wall_ms": 712.2,
|
||||
"workload_id": "region_query_10k",
|
||||
"runs_ms": [
|
||||
712.2,
|
||||
806.7,
|
||||
295.1,
|
||||
836.4,
|
||||
300.6
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "Plate with hole: ISO layers, geometry, dimensions and full critique",
|
||||
@@ -49,8 +70,16 @@
|
||||
},
|
||||
"metric": "critique_issues",
|
||||
"metric_value": 0.0,
|
||||
"wall_ms": 255.7,
|
||||
"workload_id": "premium_pass"
|
||||
"wall_ms": 65.3,
|
||||
"workload_id": "premium_pass",
|
||||
"runs_ms": [
|
||||
76.3,
|
||||
65.3,
|
||||
36.9,
|
||||
83.8,
|
||||
41.3
|
||||
]
|
||||
}
|
||||
]
|
||||
],
|
||||
"aggregation": "median of 5 default-configuration runs, one machine, one interpreter"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,142 @@
|
||||
{
|
||||
"schema_version": "perf-ab/1.0",
|
||||
"method": "benchmarks/perf_suite.py of 1.6.0 run against each tree's own code with one CPython 3.11.15 and one ezdxf, five rounds, the three configurations alternating within a round; medians of wall_ms. v1.4.0 had no perf suite; the 1.6.0 script drives the same backend methods.",
|
||||
"environment": {
|
||||
"implementation": "CPython",
|
||||
"machine": "AMD64",
|
||||
"platform": "Windows-10-10.0.26200-SP0",
|
||||
"python": "3.11.15"
|
||||
},
|
||||
"medians_ms": {
|
||||
"v1.4.0": {
|
||||
"create_lines_2k": 244.5,
|
||||
"roundtrip_10k": 1847.9,
|
||||
"region_query_10k": 665.2,
|
||||
"premium_pass": 150.6
|
||||
},
|
||||
"v1.6.0": {
|
||||
"create_lines_2k": 306.1,
|
||||
"roundtrip_10k": 2553.5,
|
||||
"region_query_10k": 712.2,
|
||||
"premium_pass": 65.3
|
||||
},
|
||||
"v1.6.0 EZDXF_CALL_TIMEOUT=0": {
|
||||
"create_lines_2k": 284.9,
|
||||
"roundtrip_10k": 2216.4,
|
||||
"region_query_10k": 783.9,
|
||||
"premium_pass": 69.9
|
||||
}
|
||||
},
|
||||
"ratios_to_v1_4_0": {
|
||||
"v1.6.0": {
|
||||
"create_lines_2k": 1.25,
|
||||
"roundtrip_10k": 1.38,
|
||||
"region_query_10k": 1.07,
|
||||
"premium_pass": 0.43
|
||||
},
|
||||
"v1.6.0 EZDXF_CALL_TIMEOUT=0": {
|
||||
"create_lines_2k": 1.17,
|
||||
"roundtrip_10k": 1.2,
|
||||
"region_query_10k": 1.18,
|
||||
"premium_pass": 0.46
|
||||
}
|
||||
},
|
||||
"runs": {
|
||||
"v1.4.0": [
|
||||
{
|
||||
"create_lines_2k": 244.5,
|
||||
"roundtrip_10k": 1936.8,
|
||||
"region_query_10k": 670.3,
|
||||
"premium_pass": 160.3
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 221.9,
|
||||
"roundtrip_10k": 1592.5,
|
||||
"region_query_10k": 638.2,
|
||||
"premium_pass": 150.6
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 236.7,
|
||||
"roundtrip_10k": 1706.6,
|
||||
"region_query_10k": 927.2,
|
||||
"premium_pass": 140.2
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 261.0,
|
||||
"roundtrip_10k": 1979.2,
|
||||
"region_query_10k": 665.2,
|
||||
"premium_pass": 191.4
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 308.4,
|
||||
"roundtrip_10k": 1847.9,
|
||||
"region_query_10k": 296.5,
|
||||
"premium_pass": 70.3
|
||||
}
|
||||
],
|
||||
"v1.6.0": [
|
||||
{
|
||||
"create_lines_2k": 287.6,
|
||||
"roundtrip_10k": 3201.3,
|
||||
"region_query_10k": 712.2,
|
||||
"premium_pass": 76.3
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 373.6,
|
||||
"roundtrip_10k": 2550.0,
|
||||
"region_query_10k": 806.7,
|
||||
"premium_pass": 65.3
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 298.2,
|
||||
"roundtrip_10k": 3118.3,
|
||||
"region_query_10k": 295.1,
|
||||
"premium_pass": 36.9
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 425.1,
|
||||
"roundtrip_10k": 2553.5,
|
||||
"region_query_10k": 836.4,
|
||||
"premium_pass": 83.8
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 306.1,
|
||||
"roundtrip_10k": 2289.4,
|
||||
"region_query_10k": 300.6,
|
||||
"premium_pass": 41.3
|
||||
}
|
||||
],
|
||||
"v1.6.0 EZDXF_CALL_TIMEOUT=0": [
|
||||
{
|
||||
"create_lines_2k": 285.3,
|
||||
"roundtrip_10k": 1817.2,
|
||||
"region_query_10k": 926.5,
|
||||
"premium_pass": 61.4
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 256.1,
|
||||
"roundtrip_10k": 2520.9,
|
||||
"region_query_10k": 750.3,
|
||||
"premium_pass": 80.2
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 260.2,
|
||||
"roundtrip_10k": 1714.1,
|
||||
"region_query_10k": 783.9,
|
||||
"premium_pass": 76.7
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 351.7,
|
||||
"roundtrip_10k": 3057.1,
|
||||
"region_query_10k": 940.8,
|
||||
"premium_pass": 69.9
|
||||
},
|
||||
{
|
||||
"create_lines_2k": 284.9,
|
||||
"roundtrip_10k": 2216.4,
|
||||
"region_query_10k": 311.2,
|
||||
"premium_pass": 41.7
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -1,25 +1,88 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"reviewed_at": "2026-07-23",
|
||||
"reviewed_at": "2026-09-24",
|
||||
"title": "AutoCAD MCP source-reviewed capability benchmark",
|
||||
"evidence_boundary": "source_review_not_shared_live_run",
|
||||
"methodology": "Public source, documented behavior, and locally recorded test evidence scored with a fixed 100-point rubric. Repository stars and raw tool counts do not score points.",
|
||||
"methodology": "Public source, documented behavior, and locally recorded test evidence scored with a fixed 100-point rubric. Repository stars and raw tool counts do not score points. The U-C4N row was re-scored for 1.6.0 on 2026-09-24 with per-category evidence; competitor rows are their July snapshots and were not re-reviewed.",
|
||||
"rubric": [
|
||||
{"category": "Functional CAD coverage", "weight": 25},
|
||||
{"category": "Correctness and delivery", "weight": 20},
|
||||
{"category": "Backends and platforms", "weight": 15},
|
||||
{"category": "Engineering production", "weight": 15},
|
||||
{"category": "Tests and maintenance", "weight": 15},
|
||||
{"category": "Security and operations", "weight": 10}
|
||||
{
|
||||
"category": "Functional CAD coverage",
|
||||
"weight": 25
|
||||
},
|
||||
{
|
||||
"category": "Correctness and delivery",
|
||||
"weight": 20
|
||||
},
|
||||
{
|
||||
"category": "Backends and platforms",
|
||||
"weight": 15
|
||||
},
|
||||
{
|
||||
"category": "Engineering production",
|
||||
"weight": 15
|
||||
},
|
||||
{
|
||||
"category": "Tests and maintenance",
|
||||
"weight": 15
|
||||
},
|
||||
{
|
||||
"category": "Security and operations",
|
||||
"weight": 10
|
||||
}
|
||||
],
|
||||
"projects": [
|
||||
{
|
||||
"repository": "U-C4N/Autocad-MCP",
|
||||
"url": "https://github.com/U-C4N/Autocad-MCP",
|
||||
"reviewed_version": "1.4.0 (PyPI release)",
|
||||
"score": 95,
|
||||
"reviewed_version": "1.6.0 (release)",
|
||||
"score": 93,
|
||||
"evidence_grade": "A",
|
||||
"note": "Dual COM/ezdxf backends, paper space, ISO 129/286/1101 annotation, closed-loop quality gates, auditable delivery; v1.4 adds CI matrix incl. Windows, PyPI packaging, and a version A/B correctness gate (21/21, 0 regressions)."
|
||||
"note": "Scored per category, each with its evidence (roadmap criterion 8). 1.4.0 received a single overall 95; this review is stricter, and every point withheld is named. New in 1.5-1.6: P&ID, environment, mechanical, sheet, architecture, understanding and takeoffs; A/B 48/48 with 0 regressions; live COM smokes on every track.",
|
||||
"category_scores": [
|
||||
{
|
||||
"category": "Functional CAD coverage",
|
||||
"weight": 25,
|
||||
"score": 24,
|
||||
"evidence": "247 tools: drafting, ISO 129/286/1101 annotation, paper space, styles and page setup, environment, P&ID (ISO 10628-2 / ISA-5.1, read back with confidence), a mechanical part model with standard parts, ISO 5457/7200/7573/6433 sheets, an architectural plan engine, drawing understanding, diff, topology and plant takeoffs. Not covered: electrical schematics, image-to-CAD tracing, Plant 3D, obstacle-avoiding routing, headless REGION/3DSOLID."
|
||||
},
|
||||
{
|
||||
"category": "Correctness and delivery",
|
||||
"weight": 20,
|
||||
"score": 19,
|
||||
"evidence": "A/B gate against v1.5.1: 48/48 checks, 0 regressions (ab-v1.5.1-vs-v1.6.0.json). Matrix v7: 21 tasks whose gates can fail (zero critique issues and a finalize score of at least 90, PDF /MediaBox, rooms within 0.1 %, takeoff lengths exact). drawing_deliver writes a manifest and checks reopen parity. A field test on a real P&ID pair drove nine fixes. Minus: a headless DIMENSION does not re-render, and a takeoff on a real P&ID still needs the engineer's scope decisions."
|
||||
},
|
||||
{
|
||||
"category": "Backends and platforms",
|
||||
"weight": 15,
|
||||
"score": 14,
|
||||
"evidence": "COM and ezdxf behind one typed contract, with capability-map parity enforced by a test. Every 1.6 track's COM path ran live on AutoCAD 2026. CI runs Linux 3.11/3.12 and Windows. Minus: BricsCAD, ZWCAD and GstarCAD are unverified."
|
||||
},
|
||||
{
|
||||
"category": "Engineering production",
|
||||
"weight": 15,
|
||||
"score": 14,
|
||||
"evidence": "Transcribed standards tables (ISO 4014/4017/4032/7089/4762, ISO 15, ISO 286, ISO 5457, ISO 7200, ISO 7573) and one part model and view engine. 18 domain critique focuses (P&ID 6, mechanical 6, architecture 3, topology 3) feed drawing_finalize's 0-100 score. Minus: four DIN/ISO feature tables ship empty until verified rows exist."
|
||||
},
|
||||
{
|
||||
"category": "Tests and maintenance",
|
||||
"weight": 15,
|
||||
"score": 14,
|
||||
"evidence": "About 4,990 tests on a locked dependency graph. The release gate re-runs three OS/Python lanes; a document-number gate and the per-release A/B suite also run. The perf lane is published (perf-v1.4.0-vs-v1.6.0.json). Minus: entity creation is 1.25x (2,000 lines) and 1.38x (10,000-line roundtrip) slower than v1.4.0, so roadmap criterion 7 is not met."
|
||||
},
|
||||
{
|
||||
"category": "Security and operations",
|
||||
"weight": 10,
|
||||
"score": 8,
|
||||
"evidence": "Per-tool path validation, a loopback-only HTTP bind unless a bearer token is set, the audit middleware, and a field-test script that refuses to write inside the repository. Minus: ALLOWED_PATHS is empty, and so unscoped, by default; command and LISP filtering is a denylist, which is a guardrail rather than a security boundary."
|
||||
}
|
||||
],
|
||||
"history": [
|
||||
{
|
||||
"version": "1.4.0",
|
||||
"score": 95,
|
||||
"reviewed_at": "2026-07-23"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"repository": "varavista/autocad-mcp",
|
||||
@@ -85,5 +148,6 @@
|
||||
"evidence_grade": "C",
|
||||
"note": "Basic COM drawing tool set with SQLite persistence; reviewed from the documented tool surface only, execution path not exercised locally."
|
||||
}
|
||||
]
|
||||
],
|
||||
"competitors_reviewed_at": "2026-07-23"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user