diff --git a/docs/superpowers/specs/2026-09-24-v1.6-understand-design.md b/docs/superpowers/specs/2026-09-24-v1.6-understand-design.md new file mode 100644 index 0000000..706ee98 --- /dev/null +++ b/docs/superpowers/specs/2026-09-24-v1.6-understand-design.md @@ -0,0 +1,305 @@ +# v1.6 Track H — understanding foreign drawings, takeoffs, diff and topology + +**Date:** 2026-09-24 · **Status:** design approved in conversation · +**Parent:** `2026-09-15-v1.6-roadmap-design.md` (track H; track D stays separate +and follows) · **Baseline:** main after track F (`0abbe55`, 241 tools) + +## 1. Summary + +Every track so far taught the server to *draw*. This one teaches it to *read* a +drawing somebody else made and turn it into a deliverable — the job a plant +engineer does by hand when a client sends a P&ID and a layout and asks how much +pipe and cable the job needs. + +The design is grounded in a real client project from the user's archive, read +on 2026-09-24 (the files are confidential and never enter this repository; see +§14): a process plant with a 35,590-entity P&ID (96 layers, labels in +Russian, layer names in English and Dutch, 296 block names, most anonymous), +a 22,399-entity layout, a hand-made pipe takeoff and a hand-made cable +takeoff. Four findings shape everything below: + +1. **Today's reader cannot read it.** `pid_graph` found 441 nodes on that P&ID — + 427 of them `unknown_block` at confidence 0.3 — and 663 edges, every one of + line class `unknown`, in 73 s. Its confidences were honest; its answer was + not useful. It cannot tell a CIP supply line from a product line, read a + diameter label, or recognise a Russian service word. +2. **The P&ID is not to scale.** Matching equipment tags between the P&ID and + the layout, distances between the same pairs of tanks differ by factors from + 0.1 to 5.6; inside one room the P&ID is a consistent ~1.28× stretch of the + plan, inside another it is rearranged outright. A pipe length measured on + that P&ID is a length of drawn line, not of pipe — and the hand-made pipe + takeoff was measured exactly that way, while the hand-made cable takeoff of + the same project states in its own method sheet that the P&ID is unscaled. + No tool warned anybody. This track makes that check a tool. +3. **The layout lies about its units.** `INSUNITS` says inches; the geometry is + millimetres (a 5.9 km wall total, a 150–280 m building). One stray entity + drags the extents to −6,450 km, and two copies of the plan sit side by side + in model space. +4. **Real drawings are big.** The two DXFs are 55 MB and 188 MB; `drawing_open` + refuses anything over 50 MB, and reading 35,000 entities one COM call at a + time takes minutes where a DXF snapshot takes half a second. + +## 2. Scope + +**In:** six tools (§12), three `topo_*` critique focuses, the `office` extra +(`openpyxl`), a DXF-snapshot reader for the live engine, the raised size limit, +synthetic fixtures that reproduce the field findings, two benchmark tasks, +correctness checks and a local field-test script. + +**Out of 1.6 (recorded):** + +- Plant 3D `.dcf` reading (track D, next). +- Cable cross-section *standards*: no IEC 60364-5-52 table is authored. A + section is chosen only from a rule table the caller supplies (the shop's own + rule); without one, no section is proposed. Voltage-drop calculation is out. +- Routing pipes around walls and equipment. A layout length is an orthogonal + estimate between connected equipment (§9) — the method the user's cable + takeoff already uses — and is labelled an estimate. +- OCR of raster underlays and text inside PDF-imported hatch geometry. + +## 3. Architecture + +``` +engineering/understand/ + __init__.py + snapshot.py one bulk read of a drawing: ezdxf directly, or a DXF snapshot of + the live document through Document.Export (§4) + vocab.py multilingual keyword tables (EN / TR / RU / NL / DE): disciplines, + services, equipment kinds, room words, supply/return words + labels.py parsers: diameters (Ø51, Ø25,4, SMS51, DN20, 20x27, 1,5"), + equipment tags, electrical data (kW, phases, voltage, +N, VFD) + units.py declared vs inferred units, robust extents, outliers + clusters.py spatial clusters of model-space content (sheets, plan copies) + describe.py drawing_understand: the report assembled from the above + scale.py drawing_scale_check: tag matching and ratio statistics + network.py line networks: junction merge, T-split, fittings bridged, runs, + service and diameter per edge, equipment attachment + route.py orthogonal lengths: Manhattan distance, rectilinear MST + takeoff.py pipe_takeoff and cable_takeoff rows, statuses, totals + report.py the takeoff workbook (Metraj / Özet / Kontrol / Metodoloji), CSV + diff.py entity signatures, matching, the change report + topology.py dangling endpoints, near misses, interior crossings + critique.py the three topo_* focuses + draw.py the async layer: snapshot, markup (revision clouds), exports +scripts/field_test_takeoff.py local only: runs the takeoffs on the user's files +``` + +Everything except `snapshot.py` and `draw.py` is pure: entity records in, +results out. The entity record is a small frozen dataclass (type, layer, +handle, geometry, text, block name, attributes) filled once per drawing, so a +35,000-entity drawing is read once and every analysis runs in memory. + +## 4. Reading a drawing (the snapshot) + +- **Headless:** `ezdxf.readfile` on the path, or the document already open. +- **Live:** `Document.Export(, "DXF", )` writes a DXF copy **without renaming the document** — measured on + AutoCAD 2026 on 2026-09-24: the document kept its name, an unsaved document + exported too, and ezdxf read the result. The extension argument must be + `"DXF"`; `".dxf"` fails with *Invalid argument* (measured). The snapshot goes + to a temporary folder and is deleted afterwards. Whether paper-space content + and xrefs reach the snapshot is measured by the plan's first live task and + stated in the tool docstrings. +- `MAX_DXF_BYTES` default rises from 50 MB to 512 MB, and the refusal names the + size and the variable (a real layout here was 188 MB). + +Every tool in this track is read-only unless it says otherwise (only +`drawing_diff(markup=True)` and the takeoff export write anything, and the +export writes a file, never the drawing). + +## 5. `drawing_understand` + +One call, one report, every claim with its evidence and a confidence: + +- **Units:** declared `INSUNITS` versus the unit the geometry implies (text + heights, typical wall / door sizes, dimension values, overall size). A + disagreement is a warning with the numbers, never a silent correction. +- **Extents:** declared versus robust (quantile) extents; the outlier entities + that stretch them, by handle. +- **Clusters:** separate bodies of content in model space — sheets, plan copies, + details — each with its box, its entity count and its most telling labels. +- **Layers:** every layer with its entity count and drawn length, and the + discipline / service it most likely carries (piping · product, CIP supply, + CIP return, glycol, steam, condensate, water, air, drain; electrical; walls; + equipment; annotation), from the multilingual vocabulary and the texts found + on it; confidence 0.9 for a direct keyword, lower for weaker evidence. +- **Equipment tags**, **rooms** (label words in five languages — *room*, *oda*, + *mahal*, *помещение*, *ruimte* — plus faces of wall layers through track F's + finder), **text languages**, **blocks** (named / anonymous, kinds by name), + **layouts** and **title-block attributes**. + +## 6. `drawing_scale_check` + +Compares a drawing with a reference drawing of the same plant. Equipment tags +found once in each are matched; for every pair of matched tags at least 2 m +apart the ratio of their distances is taken. The report gives the matched tags, +the median ratio, the interquartile range, the share of pairs within ±10 % of +the median, the same statistics per room or cluster, and a verdict: +`to_scale` (≥ 80 % of pairs within ±10 %; the factor is reported, so a drawing +at a different but uniform scale is recognised), `schematic` (≤ 50 %), or +`partly` in between. Tag labels are not equipment centres; the ±10 % band and +the 2 m floor are there to absorb that, and the report says so. + +## 7. Line networks (the shared engine) + +Segments (LINE, LWPOLYLINE, POLYLINE, ARC) on the chosen layers become a graph: +endpoints merged within a tolerance, segments split where another ends on them +(T-junctions), and a segment that ends at an inline fitting (an INSERT whose +box it touches — reducer, valve, coupling) continued through it. A connected +component is a **run**. + +- **Service** of an edge: the layer's vocabulary class, overridden by supply / + return words within a search distance (`ПОДАЧА`, `ВОЗВРАТ`, `BESLEME`, + `DÖNÜŞ`, `SUPPLY`, `RETURN`, `AANVOER`, `RETOUR`), carried along the run. Layer + alone is not trusted to split CIP supply from return — the field P&ID proved + why. +- **Diameter** of an edge: the diameter label nearest to it within the search + distance (*direct*), else carried along the run from a labelled edge until a + reducer or a conflicting label (*continuity*), else *unassigned*. Labels are + kept exactly as drawn: `Ø51`, `SMS51`, `DN20`, `Ø25,4` and `Ø25` are different + sizes unless the drawing equates them. +- **Equipment attachment:** a run end inside (or within tolerance of) the box of + the geometry a tag labels is attached to that tag. + +## 8. `pipe_takeoff` + +Rows by room × service × diameter, in the layout of the user's own takeoff +workbook: + +- **Lengths.** `length_source="auto"` (default) runs the scale check between + the P&ID and the layout. If the P&ID is `to_scale`, runs are measured on it. + If not, the length of each run is estimated on the layout: the positions of + the tags it connects, joined by a **rectilinear minimum spanning tree** + (Manhattan distance; a two-ended run is |Δx| + |Δy|), plus + `vertical_allowance` per connection (default 0, stated in the report). The + schematic length drawn on the P&ID is kept alongside as a comparison column, + never as the answer. `"pid"` and `"layout"` choose a source and the report + says which was chosen; `"pid"` on a P&ID the scale check calls `schematic` is + refused unless `force=True`. With no layout given, `auto` measures on the P&ID + and every row carries `scale_verified: false`, stated on the method sheet. + An orthogonal route runs along x first, then y — deterministic, and stated. +- **Status:** **A** — every end on a tag found on the layout and the diameter + direct; **B** — diameter by continuity; **C** — not routable (an end not on + equipment, a tag missing from the layout): reported on the *Kontrol* sheet + with its schematic length only, counted in the total and flagged. +- **Rooms:** a routed length is apportioned to the layout rooms its orthogonal + route crosses (room polygons from the layout's wall faces and room labels); + without room polygons, to the room label nearest the run. +- **Scope:** `services` defaults to product + CIP supply + CIP return (the + user's takeoff); utilities are included only on request. +- **Output:** a workbook — *Metraj* (rows with direct and continuity lengths + split, status, allowance, allowance-included), *Özet* (by service, by + diameter, by room), *Kontrol* (unassigned diameters and unroutable runs with + the reason), *Metodoloji* (generated: sources, the scale-check result, the + rules applied, the tolerances) — with Turkish, English or Russian headers + (`lang`), written as XLSX when `openpyxl` is installed (the new `office` + extra) and as CSV always. `allowance` defaults to 0.20 and is its own column, + never mixed into the net length. + +## 9. `cable_takeoff` + +One row per electrical load, the user's cable method made repeatable: + +- **Power and supply** from the P&ID's electrical texts near each tag (`P = + 18,5 кВт`, `3 ф. 400 В + N + PE`, `VFD`, in Russian, Turkish and English + spellings). A load with no stated power keeps an empty power cell and an open + item — **power is never invented**. +- **Panel** from the layout's electrical callouts (`Wiring to CP1` and its + variants) or the P&ID; panel names normalised (`CP 1` → `CP-1`). +- **Length:** Manhattan distance on the layout from the tag to its panel, plus + the allowance, rounded up to the metre (`ROUNDUP(length × (1 + allowance))`, + as the user's sheet does). With no layout given, lengths stay empty and each + row gets an open item — the P&ID is not a length source for cables. +- **Room totals** of known power, with the rule for panel packages versus their + child motors stated in the method sheet so no load is counted twice. +- **Section:** only from a caller-supplied `section_rules` table; otherwise + blank. +- Same workbook shape and languages as §8, with an *Open items* list. + +## 10. `drawing_diff` + +Two revisions of one drawing (paths, or the current document against a path). +Entity signatures (type, layer, geometry rounded to a tolerance; text content +and position; block name, insertion, rotation, scale and attributes) are +matched exactly first, then by handle, then by type + layer + nearest position +within a move tolerance; what is left is added or removed. The report gives +counts by layer and type, the changed entities with what changed (moved by, text +from → to, attribute from → to, radius from → to), block-definition changes and +a capped list per category. `markup=True` draws revision clouds around the +clusters of change on the newer drawing with track G's revision tools — the +only write in the tool, and only on request. + +## 11. `drawing_topology_check` and the `topo_*` focuses + +Dangling endpoints, near misses (two ends, or an end and a segment, within a +gap tolerance but not joined) and interior crossings (two segments crossing away +from their ends), on the layers given or, by default, on the network layers +`drawing_understand` classifies with confidence ≥ 0.9. The three critique +focuses (`topo_dangling_endpoint`, `topo_near_miss`, `topo_interior_crossing`) +run the same engine under `focus=None`, **excluding layers a semantic focus +already covers** (P&ID line layers → `pid_*`, architectural walls → `arch_*`), +so one defect is never scored twice; on a drawing made entirely by this server +they are silent. + +## 12. Tool surface, packs, lean + +| Tool | Pack | Writes? | +|---|---|---| +| `drawing_understand` | core (lean) | no | +| `drawing_scale_check` | core | no | +| `drawing_diff` | core | only `markup=True` | +| `drawing_topology_check` | core | no | +| `pipe_takeoff` | `plant` | a workbook file | +| `cable_takeoff` | `plant` | a workbook file | + +**+6 → 247 tools.** New pack `plant` (track D's readers join it later). Lean +gains `drawing_understand` (→ 65). Aliases in five languages (metraj, metre +cetveli, boru listesi, kablo listesi, спецификация, ведомость, takeoff, BOQ, +fark, karşılaştır, revizyon farkı, compare, DWGCOMPARE, birim, ölçek, scale +check). + +## 13. Refusals + +Before any work: a missing or unreadable file, a DXF over the size limit (with +the variable named), `lang` outside tr/en/ru, a non-positive tolerance or +allowance outside 0–1, `length_source="pid"` with a P&ID the scale check calls +schematic *without* `force=True` (the refusal quotes the statistics), an XLSX +export without `openpyxl` (the `xlsx_write` capability, naming the extra; CSV +still written). + +## 14. Confidentiality + +Client drawings, spreadsheets and anything derived from them that identifies +the client or its business — file names, client and site names, takeoff +quantities — never enter the repository, its tests, its +benchmark results or its commit messages. The fixtures are **synthetic** +(§15). `scripts/field_test_takeoff.py` runs the tools on files the user points +it at and writes its report next to them, outside the repository; its output is +never committed. + +## 15. Evidence + +- **Synthetic plant pair** (`tests/fixtures/plant_pair.py`): a generator that + builds a schematic P&ID DXF and a layout DXF of one small plant with known + answers — equipment tags present in both, the P&ID deliberately stretched + 1.3× in one room and rearranged in another, service layers with English / + Dutch names, Russian supply / return words, `Ø` / `SMS` / `DN` / `a x b` + labels, a reducer, an inline valve, electrical texts in Russian, `Wiring to + CPn` callouts, a layout with `INSUNITS` set to inches over millimetre + geometry, one far outlier entity and two plan copies. +- **Benchmark tasks** (matrix v7, 21 tasks): `takeoff_roundtrip` — the scale + check calls the synthetic P&ID `schematic`, every pipe run's layout length + equals the hand-computed rectilinear MST exactly, diameters and services match + the generator's truth, and the cable rows equal the generator's Manhattan + lengths with the allowance rounded up; `understand_foreign` — units flagged, + the outlier found, both plan copies separated, the service layers classified. +- **Correctness checks** (+5): `scale_check_detects_schematic`, + `pipe_takeoff_rmst_exact`, `cable_takeoff_roundup`, `diff_detects_known_edits` + (a move, a text change, an attribute change, an add and a delete, and nothing + else), `topology_known_defects`. +- **Field test** (local, not committed): `pipe_takeoff` and `cable_takeoff` on + the user's real pair, compared with their hand-made workbooks; the difference + is expected where the hand-made pipe takeoff measured the unscaled P&ID, and + the report explains each row. +- **Live smoke** `scripts/smoke_understand_com.py`: the snapshot, the takeoffs + and the diff on AutoCAD 2026, every document closed.