diff --git a/.agents/phases/todo/163_pdf_strip_additive_selfheal/00_phase.md b/.agents/phases/todo/163_pdf_strip_additive_selfheal/00_phase.md new file mode 100644 index 0000000..fec0bed --- /dev/null +++ b/.agents/phases/todo/163_pdf_strip_additive_selfheal/00_phase.md @@ -0,0 +1,73 @@ +# Phase 163 — PDF page strip: additive text+pages view, write-first atomic copies, deterministic fast-path re-gate (self-heal) + +**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, the document modal for `Letumdei_ Heirs to the Imperium/Letumdei_ Heirs to the Imperium.pdf` (doc id `b72f5515-dc40-4167-98ff-179a6503e6e5`). Every page rendered "Page N unavailable." and the document was unreadable in the viewer. **Pre-fix evidence (measured live, 2026-10-07):** the payload carries 36 `page_urls` (so `page_count = 36`); `GET /api/documents//pages/1…36` **all 404**; the content is genuine extracted text (114 389 chars, 36 `---PAGE-N---` markers, exactly ONE `[Page 36 illustration: …]` block — a map page under `pdf_pictorial_min_chars`); the same server serves the 10/3-indexed IKEA manual's pages 200 (the pages dir was NOT wiped — only this doc's copies are gone). Two stacked defects: + +1. **Eligibility consequence (the "shouldn't be a pictorial pdf" complaint).** The phase-140 gate is per-page (`len(text.strip()) < pdf_pictorial_min_chars`, default 100); ONE sparse page in a text-heavy 36-page book set `pictorial_described = True` → the doc joined the strip class → the phase-148 viewer renders the strip **INSTEAD OF** the text content and the Summary panel. The owner's call (2026-10-07): **additive strip** — a usable-layer (pictorial-class) doc shows its TEXT (plus the Summary panel, parity with any text doc) with a **Text/Pages toggle** for the strip; the text is never hidden. The fallback class (unusable text layer — the phase-133 screenshots ARE the content) keeps today's strip-only view, DOM byte-identical. +2. **Lost copies, stale `page_count`, no self-heal.** `_persist_pdf_pages` **deletes the old copies from disk first**, and the filesystem work is NOT transactional with the DB session: any later rollback of the file's transaction (the per-file `EmbeddingError` handler, a killed sync) — or an external wipe of `pdf_page_dir` — leaves the row claiming `page_count = 36` over ZERO files. The phase-147 fast path ("the copies persist") never re-renders, so the strip 404s forever. The owner's call: **write-before-delete + fast-path re-gate** — the changed path writes the new copies (atomically, per file) BEFORE any deletion; the unchanged paths deterministically re-gate the strip (no LLM/vision call) — re-stamp the class, re-render missing copies, or clear a strip that no longer qualifies. + +**Story:** n/a (owner-directed bug fix; extends `148_pdf_pages_in_sources`, `147_sync_changed_only`, `140_pictorial_pdf_pages`, `133_pdf_screenshot_fallback`). + +## Objective +A pictorial-class PDF is a **readable text document with the pages one toggle away** (Text default; Pages shows the phase-148 strip); a fallback-class PDF keeps the strip-only view byte-identically; and a strip never dies silently — the next sync **re-gates and re-renders** missing copies (or clears a strip that no longer qualifies) with zero LLM calls. The live `Letumdei` doc is repaired by the next production Sync (no manual script): additive mode stamped, 36 copies re-rendered, the book's text back in the viewer. + +## Dependencies +- `148_pdf_pages_in_sources` (complete) — `page_count`, `pdf_page_dir`, `-p{N}.png`, the `/pages/{n}` route, the viewer's `docPagesStrip`, and the persist/regate seams this phase changes. +- `147_sync_changed_only` (complete) — the `file_hash` fast path + shared `_unchanged_doc` (the re-gate hook site) whose "the copies persist" assumption this phase replaces. +- `140_pictorial_pdf_pages` (complete) — the deterministic `pictorial_pages` gate + the kill switch `settings.pdf_pictorial` (the re-gate mirrors this gate exactly). +- `133_pdf_screenshot_fallback` (complete) — `is_usable_pdf_text` (the fallback-class signal the re-gate mirrors). + +## Design (shared by all tasks — the executor reads this, not the chat) + +**D1 — the strip-mode wire signal (task 01, LOCKED A2/A3).** New nullable `documents.page_strip_mode` VARCHAR(16) (migration **0027**, tested downgrade, A13): `"primary"` = fallback class (strip replaces the text — today's behavior), `"additive"` = pictorial class (text + toggle). Set exactly when `page_count` is set (cleared with it). `DocContent` gains `page_strip_mode: str | None = None` with the `image_url` omission rule: **ABSENT (never null) when `page_count` is NULL** — every non-qualifying doc's payload stays byte-identical to pre-phase. For a qualifying doc the key is present; a **NULL-mode pre-phase row serializes ABSENT** and the viewer treats absent as `"primary"` (existing rows keep today's DOM until the re-gate stamps them). The `/pages/{n}` route is untouched. + +**D2 — write-first atomic persist (task 02, LOCKED A4).** `_persist_pdf_pages` reorders: **render → write every page (atomic per file: write to `.png.tmp`, `os.replace`) → only then delete stale copies** outside the written set (the shrunk-PDF tail; the current blanket `_delete_page_copies` BEFORE the render is removed — a rollback or crash can no longer leave the row over zero files). The partial-write failure path unlinks BOTH the new final files and the `.tmp` files before re-raising (no orphans; the existing fail-soft contract — caller sets `page_count` NULL + warning — unchanged). The function also stamps `doc.page_strip_mode`: `"primary"` when `fallback_fired`, `"additive"` when `pictorial_described` (the caller passes the CLASS, not just the boolean — `eligible=fallback_fired or pictorial_described` is replaced by `strip_class="fallback"|"pictorial"|None`); the ineligible arm clears mode with `page_count`. **The eligibility RULE itself is unchanged** (the phase-148 lock `fallback_fired or pictorial_described` stands — only its VIEW consequence changes, via D1/D4). + +**D3 — the deterministic fast-path re-gate (task 03, LOCKED A5).** A new `_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths — inside `_unchanged_doc` (an optional `full_path` kwarg, defaulting to `Path(doc.full_path)`), so it fires on the `file_hash` fast path AND the content-digest-hit path — guarded by `doc.page_count is not None` (only PDF rows ever carry it; every other file's fast path keeps its zero-extra-read contract). It mirrors the changed-path gate **deterministically from the current file bytes + current settings — ZERO vision/LLM calls** (`render_pdf_pages` is CPU-only, A21-safe): +- `not is_usable_pdf_text(extract_pdf_text(full_path))` → class `fallback` (mode `primary`) — same doc as today; +- elif `settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=settings.pdf_pictorial_min_chars)` is non-empty → class `pictorial` (mode `additive`) — **candidates are exactly the existing gate's** (text-carrying pages under `min_chars`; no-text pages are NOT candidates, mirroring `pictorial_pages` — the re-gate adds no new semantics); +- else → **repair arm**: the strip no longer qualifies (e.g. the owner lowered `pdf_pictorial_min_chars`) → `page_count = None`, `page_strip_mode = None`, `_delete_page_copies(doc, page_dir)`. + +For a re-gated-eligible doc, **verify the copies**: any of `page_dir/-p1..p{page_count}.png` missing → re-render ALL pages (`render_pdf_pages`), atomic write-first (D2's writer), and set `page_count` to the fresh render's length (a drift names the doc in a warning; bytes unchanged makes drift pathological). A fresh render producing NO pages (unopenable despite the just-computed hash — pathological) keeps the row as-is with a warning (fail-soft, never clears on a transient). One greppable INFO line whenever re-gate ACTS (never per-unchanged-doc): `import: PDF strip re-gate source=… path=… — healed N pages | cleared stale strip | mode ->`. The re-gate mutates the row INSIDE the unchanged path's transaction (the commit already owned by `_unchanged_doc`'s caller flow); a copy-verify/render exception is caught, warned, and never fails the sync. + +**D4 — the additive viewer (task 04, LOCKED A1/A6).** `renderDocument` (the ONE shared core — modal AND `/document.html`): the strip branch now keys on the mode — +- `page_strip_mode` absent or `"primary"` (with `page_urls`): the EXACT current DOM (`.doc-pages` strip, no toggle, no Summary panel) — byte-identical, pinned; +- `"additive"` (with `page_urls`): the text doc renders **exactly as a non-qualifying text doc** (Summary panel + content, the pre-148 branch reused), PLUS a `Text`/`Pages` toggle above the content: `
` with two `