docs: track H (understanding foreign drawings) design spec

Read a drawing somebody else made and turn it into a deliverable: a one-call drawing report (units that lie, stray extents, plan copies, multilingual layer meaning), a scale check between a P&ID and its layout, pipe and cable takeoffs whose topology comes from the P&ID and whose lengths come from the layout, a revision diff by geometric signature, and topology checks. Grounded in a real client project read locally (never committed): its P&ID proved to be unscaled against its layout, which no tool had flagged. Six tools (+ pack plant), three topo_* focuses, the office extra, a live DXF snapshot through Document.Export (measured: the document is not renamed), and synthetic fixtures that reproduce the field findings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FKFStjEFzpGyLq5bMpWxTd
2026-09-24 09:32:31 +03:00
parent 0abbe55041
commit 18995bf854
1 changed files with 305 additions and 0 deletions
@@ -0,0 +1,305 @@
 # v1.6 Track H — understanding foreign drawings, takeoffs, diff and topology
 **Date:** 2026-09-24 · **Status:** design approved in conversation ·
 **Parent:** `2026-09-15-v1.6-roadmap-design.md` (track H; track D stays separate
 and follows) · **Baseline:** main after track F (`0abbe55`, 241 tools)
 ## 1. Summary
 Every track so far taught the server to *draw*. This one teaches it to *read* a
 drawing somebody else made and turn it into a deliverable — the job a plant
 engineer does by hand when a client sends a P&ID and a layout and asks how much
 pipe and cable the job needs.
 The design is grounded in a real client project from the user's archive, read
 on 2026-09-24 (the files are confidential and never enter this repository; see
 §14): a process plant with a 35,590-entity P&ID (96 layers, labels in
 Russian, layer names in English and Dutch, 296 block names, most anonymous),
 a 22,399-entity layout, a hand-made pipe takeoff and a hand-made cable
 takeoff. Four findings shape everything below:
 1. **Today's reader cannot read it.** `pid_graph` found 441 nodes on that P&ID —
   427 of them `unknown_block` at confidence 0.3 — and 663 edges, every one of
   line class `unknown`, in 73 s. Its confidences were honest; its answer was
   not useful. It cannot tell a CIP supply line from a product line, read a
   diameter label, or recognise a Russian service word.
 2. **The P&ID is not to scale.** Matching equipment tags between the P&ID and
   the layout, distances between the same pairs of tanks differ by factors from
   0.1 to 5.6; inside one room the P&ID is a consistent ~1.28× stretch of the
   plan, inside another it is rearranged outright. A pipe length measured on
   that P&ID is a length of drawn line, not of pipe — and the hand-made pipe
   takeoff was measured exactly that way, while the hand-made cable takeoff of
   the same project states in its own method sheet that the P&ID is unscaled.
   No tool warned anybody. This track makes that check a tool.
 3. **The layout lies about its units.** `INSUNITS` says inches; the geometry is
   millimetres (a 5.9 km wall total, a 150–280 m building). One stray entity
   drags the extents to −6,450 km, and two copies of the plan sit side by side
   in model space.
 4. **Real drawings are big.** The two DXFs are 55 MB and 188 MB; `drawing_open`
   refuses anything over 50 MB, and reading 35,000 entities one COM call at a
   time takes minutes where a DXF snapshot takes half a second.
 ## 2. Scope
 **In:** six tools (§12), three `topo_*` critique focuses, the `office` extra
 (`openpyxl`), a DXF-snapshot reader for the live engine, the raised size limit,
 synthetic fixtures that reproduce the field findings, two benchmark tasks,
 correctness checks and a local field-test script.
 **Out of 1.6 (recorded):**
 - Plant 3D `.dcf` reading (track D, next).
 - Cable cross-section *standards*: no IEC 60364-5-52 table is authored. A
  section is chosen only from a rule table the caller supplies (the shop's own
  rule); without one, no section is proposed. Voltage-drop calculation is out.
 - Routing pipes around walls and equipment. A layout length is an orthogonal
  estimate between connected equipment (§9) — the method the user's cable
  takeoff already uses — and is labelled an estimate.
 - OCR of raster underlays and text inside PDF-imported hatch geometry.
 ## 3. Architecture
 ```
 engineering/understand/
  __init__.py
  snapshot.py     one bulk read of a drawing: ezdxf directly, or a DXF snapshot of
                  the live document through Document.Export (§4)
  vocab.py        multilingual keyword tables (EN / TR / RU / NL / DE): disciplines,
                  services, equipment kinds, room words, supply/return words
  labels.py       parsers: diameters (Ø51, Ø25,4, SMS51, DN20, 20x27, 1,5"),
                  equipment tags, electrical data (kW, phases, voltage, +N, VFD)
  units.py        declared vs inferred units, robust extents, outliers
  clusters.py     spatial clusters of model-space content (sheets, plan copies)
  describe.py     drawing_understand: the report assembled from the above
  scale.py        drawing_scale_check: tag matching and ratio statistics
  network.py      line networks: junction merge, T-split, fittings bridged, runs,
                  service and diameter per edge, equipment attachment
  route.py        orthogonal lengths: Manhattan distance, rectilinear MST
  takeoff.py      pipe_takeoff and cable_takeoff rows, statuses, totals
  report.py       the takeoff workbook (Metraj / Özet / Kontrol / Metodoloji), CSV
  diff.py         entity signatures, matching, the change report
  topology.py     dangling endpoints, near misses, interior crossings
  critique.py     the three topo_* focuses
  draw.py         the async layer: snapshot, markup (revision clouds), exports
 scripts/field_test_takeoff.py   local only: runs the takeoffs on the user's files
 ```
 Everything except `snapshot.py` and `draw.py` is pure: entity records in,
 results out. The entity record is a small frozen dataclass (type, layer,
 handle, geometry, text, block name, attributes) filled once per drawing, so a
 35,000-entity drawing is read once and every analysis runs in memory.
 ## 4. Reading a drawing (the snapshot)
 - **Headless:** `ezdxf.readfile` on the path, or the document already open.
 - **Live:** `Document.Export(<path without extension>, "DXF", <selection set of
  all>)` writes a DXF copy **without renaming the document** — measured on
  AutoCAD 2026 on 2026-09-24: the document kept its name, an unsaved document
  exported too, and ezdxf read the result. The extension argument must be
  `"DXF"`; `".dxf"` fails with *Invalid argument* (measured). The snapshot goes
  to a temporary folder and is deleted afterwards. Whether paper-space content
  and xrefs reach the snapshot is measured by the plan's first live task and
  stated in the tool docstrings.
 - `MAX_DXF_BYTES` default rises from 50 MB to 512 MB, and the refusal names the
  size and the variable (a real layout here was 188 MB).
 Every tool in this track is read-only unless it says otherwise (only
 `drawing_diff(markup=True)` and the takeoff export write anything, and the
 export writes a file, never the drawing).
 ## 5. `drawing_understand`
 One call, one report, every claim with its evidence and a confidence:
 - **Units:** declared `INSUNITS` versus the unit the geometry implies (text
  heights, typical wall / door sizes, dimension values, overall size). A
  disagreement is a warning with the numbers, never a silent correction.
 - **Extents:** declared versus robust (quantile) extents; the outlier entities
  that stretch them, by handle.
 - **Clusters:** separate bodies of content in model space — sheets, plan copies,
  details — each with its box, its entity count and its most telling labels.
 - **Layers:** every layer with its entity count and drawn length, and the
  discipline / service it most likely carries (piping · product, CIP supply,
  CIP return, glycol, steam, condensate, water, air, drain; electrical; walls;
  equipment; annotation), from the multilingual vocabulary and the texts found
  on it; confidence 0.9 for a direct keyword, lower for weaker evidence.
 - **Equipment tags**, **rooms** (label words in five languages — *room*, *oda*,
  *mahal*, *помещение*, *ruimte* — plus faces of wall layers through track F's
  finder), **text languages**, **blocks** (named / anonymous, kinds by name),
  **layouts** and **title-block attributes**.
 ## 6. `drawing_scale_check`
 Compares a drawing with a reference drawing of the same plant. Equipment tags
 found once in each are matched; for every pair of matched tags at least 2 m
 apart the ratio of their distances is taken. The report gives the matched tags,
 the median ratio, the interquartile range, the share of pairs within ±10 % of
 the median, the same statistics per room or cluster, and a verdict:
 `to_scale` (≥ 80 % of pairs within ±10 %; the factor is reported, so a drawing
 at a different but uniform scale is recognised), `schematic` (≤ 50 %), or
 `partly` in between. Tag labels are not equipment centres; the ±10 % band and
 the 2 m floor are there to absorb that, and the report says so.
 ## 7. Line networks (the shared engine)
 Segments (LINE, LWPOLYLINE, POLYLINE, ARC) on the chosen layers become a graph:
 endpoints merged within a tolerance, segments split where another ends on them
 (T-junctions), and a segment that ends at an inline fitting (an INSERT whose
 box it touches — reducer, valve, coupling) continued through it. A connected
 component is a **run**.
 - **Service** of an edge: the layer's vocabulary class, overridden by supply /
  return words within a search distance (`ПОДАЧА`, `ВОЗВРАТ`, `BESLEME`,
  `DÖNÜŞ`, `SUPPLY`, `RETURN`, `AANVOER`, `RETOUR`), carried along the run. Layer
  alone is not trusted to split CIP supply from return — the field P&ID proved
  why.
 - **Diameter** of an edge: the diameter label nearest to it within the search
  distance (*direct*), else carried along the run from a labelled edge until a
  reducer or a conflicting label (*continuity*), else *unassigned*. Labels are
  kept exactly as drawn: `Ø51`, `SMS51`, `DN20`, `Ø25,4` and `Ø25` are different
  sizes unless the drawing equates them.
 - **Equipment attachment:** a run end inside (or within tolerance of) the box of
  the geometry a tag labels is attached to that tag.
 ## 8. `pipe_takeoff`
 Rows by room × service × diameter, in the layout of the user's own takeoff
 workbook:
 - **Lengths.** `length_source="auto"` (default) runs the scale check between
  the P&ID and the layout. If the P&ID is `to_scale`, runs are measured on it.
  If not, the length of each run is estimated on the layout: the positions of
  the tags it connects, joined by a **rectilinear minimum spanning tree**
  (Manhattan distance; a two-ended run is |Δx| + |Δy|), plus
  `vertical_allowance` per connection (default 0, stated in the report). The
  schematic length drawn on the P&ID is kept alongside as a comparison column,
  never as the answer. `"pid"` and `"layout"` choose a source and the report
  says which was chosen; `"pid"` on a P&ID the scale check calls `schematic` is
  refused unless `force=True`. With no layout given, `auto` measures on the P&ID
  and every row carries `scale_verified: false`, stated on the method sheet.
  An orthogonal route runs along x first, then y — deterministic, and stated.
 - **Status:** **A** — every end on a tag found on the layout and the diameter
  direct; **B** — diameter by continuity; **C** — not routable (an end not on
  equipment, a tag missing from the layout): reported on the *Kontrol* sheet
  with its schematic length only, counted in the total and flagged.
 - **Rooms:** a routed length is apportioned to the layout rooms its orthogonal
  route crosses (room polygons from the layout's wall faces and room labels);
  without room polygons, to the room label nearest the run.
 - **Scope:** `services` defaults to product + CIP supply + CIP return (the
  user's takeoff); utilities are included only on request.
 - **Output:** a workbook — *Metraj* (rows with direct and continuity lengths
  split, status, allowance, allowance-included), *Özet* (by service, by
  diameter, by room), *Kontrol* (unassigned diameters and unroutable runs with
  the reason), *Metodoloji* (generated: sources, the scale-check result, the
  rules applied, the tolerances) — with Turkish, English or Russian headers
  (`lang`), written as XLSX when `openpyxl` is installed (the new `office`
  extra) and as CSV always. `allowance` defaults to 0.20 and is its own column,
  never mixed into the net length.
 ## 9. `cable_takeoff`
 One row per electrical load, the user's cable method made repeatable:
 - **Power and supply** from the P&ID's electrical texts near each tag (`P =
  18,5 кВт`, `3 ф. 400 В + N + PE`, `VFD`, in Russian, Turkish and English
  spellings). A load with no stated power keeps an empty power cell and an open
  item — **power is never invented**.
 - **Panel** from the layout's electrical callouts (`Wiring to CP1` and its
  variants) or the P&ID; panel names normalised (`CP 1` → `CP-1`).
 - **Length:** Manhattan distance on the layout from the tag to its panel, plus
  the allowance, rounded up to the metre (`ROUNDUP(length × (1 + allowance))`,
  as the user's sheet does). With no layout given, lengths stay empty and each
  row gets an open item — the P&ID is not a length source for cables.
 - **Room totals** of known power, with the rule for panel packages versus their
  child motors stated in the method sheet so no load is counted twice.
 - **Section:** only from a caller-supplied `section_rules` table; otherwise
  blank.
 - Same workbook shape and languages as §8, with an *Open items* list.
 ## 10. `drawing_diff`
 Two revisions of one drawing (paths, or the current document against a path).
 Entity signatures (type, layer, geometry rounded to a tolerance; text content
 and position; block name, insertion, rotation, scale and attributes) are
 matched exactly first, then by handle, then by type + layer + nearest position
 within a move tolerance; what is left is added or removed. The report gives
 counts by layer and type, the changed entities with what changed (moved by, text
 from → to, attribute from → to, radius from → to), block-definition changes and
 a capped list per category. `markup=True` draws revision clouds around the
 clusters of change on the newer drawing with track G's revision tools — the
 only write in the tool, and only on request.
 ## 11. `drawing_topology_check` and the `topo_*` focuses
 Dangling endpoints, near misses (two ends, or an end and a segment, within a
 gap tolerance but not joined) and interior crossings (two segments crossing away
 from their ends), on the layers given or, by default, on the network layers
 `drawing_understand` classifies with confidence ≥ 0.9. The three critique
 focuses (`topo_dangling_endpoint`, `topo_near_miss`, `topo_interior_crossing`)
 run the same engine under `focus=None`, **excluding layers a semantic focus
 already covers** (P&ID line layers → `pid_*`, architectural walls → `arch_*`),
 so one defect is never scored twice; on a drawing made entirely by this server
 they are silent.
 ## 12. Tool surface, packs, lean
 | Tool | Pack | Writes? |
 |---|---|---|
 | `drawing_understand` | core (lean) | no |
 | `drawing_scale_check` | core | no |
 | `drawing_diff` | core | only `markup=True` |
 | `drawing_topology_check` | core | no |
 | `pipe_takeoff` | `plant` | a workbook file |
 | `cable_takeoff` | `plant` | a workbook file |
 **+6 → 247 tools.** New pack `plant` (track D's readers join it later). Lean
 gains `drawing_understand` (→ 65). Aliases in five languages (metraj, metre
 cetveli, boru listesi, kablo listesi, спецификация, ведомость, takeoff, BOQ,
 fark, karşılaştır, revizyon farkı, compare, DWGCOMPARE, birim, ölçek, scale
 check).
 ## 13. Refusals
 Before any work: a missing or unreadable file, a DXF over the size limit (with
 the variable named), `lang` outside tr/en/ru, a non-positive tolerance or
 allowance outside 0–1, `length_source="pid"` with a P&ID the scale check calls
 schematic *without* `force=True` (the refusal quotes the statistics), an XLSX
 export without `openpyxl` (the `xlsx_write` capability, naming the extra; CSV
 still written).
 ## 14. Confidentiality
 Client drawings, spreadsheets and anything derived from them that identifies
 the client or its business — file names, client and site names, takeoff
 quantities — never enter the repository, its tests, its
 benchmark results or its commit messages. The fixtures are **synthetic**
 (§15). `scripts/field_test_takeoff.py` runs the tools on files the user points
 it at and writes its report next to them, outside the repository; its output is
 never committed.
 ## 15. Evidence
 - **Synthetic plant pair** (`tests/fixtures/plant_pair.py`): a generator that
  builds a schematic P&ID DXF and a layout DXF of one small plant with known
  answers — equipment tags present in both, the P&ID deliberately stretched
  1.3× in one room and rearranged in another, service layers with English /
  Dutch names, Russian supply / return words, `Ø` / `SMS` / `DN` / `a x b`
  labels, a reducer, an inline valve, electrical texts in Russian, `Wiring to
  CPn` callouts, a layout with `INSUNITS` set to inches over millimetre
  geometry, one far outlier entity and two plan copies.
 - **Benchmark tasks** (matrix v7, 21 tasks): `takeoff_roundtrip` — the scale
  check calls the synthetic P&ID `schematic`, every pipe run's layout length
  equals the hand-computed rectilinear MST exactly, diameters and services match
  the generator's truth, and the cable rows equal the generator's Manhattan
  lengths with the allowance rounded up; `understand_foreign` — units flagged,
  the outlier found, both plan copies separated, the service layers classified.
 - **Correctness checks** (+5): `scale_check_detects_schematic`,
  `pipe_takeoff_rmst_exact`, `cable_takeoff_roundup`, `diff_detects_known_edits`
  (a move, a text change, an attribute change, an add and a delete, and nothing
  else), `topology_known_defects`.
 - **Field test** (local, not committed): `pipe_takeoff` and `cable_takeoff` on
  the user's real pair, compared with their hand-made workbooks; the difference
  is expected where the hand-made pipe takeoff measured the unscaled P&ID, and
  the report explains each row.
 - **Live smoke** `scripts/smoke_understand_com.py`: the snapshot, the takeoffs
  and the diff on AutoCAD 2026, every document closed.