Compare commits

...
3 Commits
Author SHA1 Message Date
ducoterra 6ba46e735a phase: 163_pdf_strip_additive_selfheal
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Successful in 21s
All gates verified. Final report:

**Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)**

- Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs).
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%)
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation
- `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed
- `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins)
- Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅
- **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**.
- Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit.
- **Next pending phase:** `164_chat_error_honesty`
2026-10-07 13:57:30 -04:00
ducoterra 74eb2e265c docs(phases): add phase 164 (honest pre-stream failure line, shared error-detail serializer, history caps for full-length answers)
Captures the 2026-10-07 live '[object Object]' finding: a >32k-char
answer rides back as history, the 422's list-of-objects detail hits the
unguarded body.detail assignment, and the zero-frame settle lies about
a dropped connection. Caps raise as a chain (HistoryTurn + ChatMessage
mirror per phase-83 SEC-05) with a bounded anonymous total; the budget
trim stays locked and untouched.

LLM-Generated: true
2026-10-07 09:57:40 -04:00
ducoterra 855fa27a30 docs(phases): add phase 163 (additive PDF strip view, write-first atomic copies, fast-path self-heal)
Captures the 2026-10-07 live findings on Letumdei_ Heirs to the Imperium:
a single sparse map page flipped a text PDF into the strip-only viewer,
and the strip's page copies were lost from disk while page_count stayed
stamped (no self-heal path). Owner decisions recorded: additive
text+Pages toggle for the pictorial class, write-before-delete atomic
persist, deterministic zero-LLM fast-path re-gate.

LLM-Generated: true
2026-10-07 09:46:37 -04:00
57 changed files with 5154 additions and 212 deletions
@@ -0,0 +1,73 @@
# Phase 163 — PDF page strip: additive text+pages view, write-first atomic copies, deterministic fast-path re-gate (self-heal)
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, the document modal for `Letumdei_ Heirs to the Imperium/Letumdei_ Heirs to the Imperium.pdf` (doc id `b72f5515-dc40-4167-98ff-179a6503e6e5`). Every page rendered "Page N unavailable." and the document was unreadable in the viewer. **Pre-fix evidence (measured live, 2026-10-07):** the payload carries 36 `page_urls` (so `page_count = 36`); `GET /api/documents/<id>/pages/1…36` **all 404**; the content is genuine extracted text (114 389 chars, 36 `---PAGE-N---` markers, exactly ONE `[Page 36 illustration: …]` block — a map page under `pdf_pictorial_min_chars`); the same server serves the 10/3-indexed IKEA manual's pages 200 (the pages dir was NOT wiped — only this doc's copies are gone). Two stacked defects:
1. **Eligibility consequence (the "shouldn't be a pictorial pdf" complaint).** The phase-140 gate is per-page (`len(text.strip()) < pdf_pictorial_min_chars`, default 100); ONE sparse page in a text-heavy 36-page book set `pictorial_described = True` → the doc joined the strip class → the phase-148 viewer renders the strip **INSTEAD OF** the text content and the Summary panel. The owner's call (2026-10-07): **additive strip** — a usable-layer (pictorial-class) doc shows its TEXT (plus the Summary panel, parity with any text doc) with a **Text/Pages toggle** for the strip; the text is never hidden. The fallback class (unusable text layer — the phase-133 screenshots ARE the content) keeps today's strip-only view, DOM byte-identical.
2. **Lost copies, stale `page_count`, no self-heal.** `_persist_pdf_pages` **deletes the old copies from disk first**, and the filesystem work is NOT transactional with the DB session: any later rollback of the file's transaction (the per-file `EmbeddingError` handler, a killed sync) — or an external wipe of `pdf_page_dir` — leaves the row claiming `page_count = 36` over ZERO files. The phase-147 fast path ("the copies persist") never re-renders, so the strip 404s forever. The owner's call: **write-before-delete + fast-path re-gate** — the changed path writes the new copies (atomically, per file) BEFORE any deletion; the unchanged paths deterministically re-gate the strip (no LLM/vision call) — re-stamp the class, re-render missing copies, or clear a strip that no longer qualifies.
**Story:** n/a (owner-directed bug fix; extends `148_pdf_pages_in_sources`, `147_sync_changed_only`, `140_pictorial_pdf_pages`, `133_pdf_screenshot_fallback`).
## Objective
A pictorial-class PDF is a **readable text document with the pages one toggle away** (Text default; Pages shows the phase-148 strip); a fallback-class PDF keeps the strip-only view byte-identically; and a strip never dies silently — the next sync **re-gates and re-renders** missing copies (or clears a strip that no longer qualifies) with zero LLM calls. The live `Letumdei` doc is repaired by the next production Sync (no manual script): additive mode stamped, 36 copies re-rendered, the book's text back in the viewer.
## Dependencies
- `148_pdf_pages_in_sources` (complete) — `page_count`, `pdf_page_dir`, `<doc-id>-p{N}.png`, the `/pages/{n}` route, the viewer's `docPagesStrip`, and the persist/regate seams this phase changes.
- `147_sync_changed_only` (complete) — the `file_hash` fast path + shared `_unchanged_doc` (the re-gate hook site) whose "the copies persist" assumption this phase replaces.
- `140_pictorial_pdf_pages` (complete) — the deterministic `pictorial_pages` gate + the kill switch `settings.pdf_pictorial` (the re-gate mirrors this gate exactly).
- `133_pdf_screenshot_fallback` (complete) — `is_usable_pdf_text` (the fallback-class signal the re-gate mirrors).
## Design (shared by all tasks — the executor reads this, not the chat)
**D1 — the strip-mode wire signal (task 01, LOCKED A2/A3).** New nullable `documents.page_strip_mode` VARCHAR(16) (migration **0027**, tested downgrade, A13): `"primary"` = fallback class (strip replaces the text — today's behavior), `"additive"` = pictorial class (text + toggle). Set exactly when `page_count` is set (cleared with it). `DocContent` gains `page_strip_mode: str | None = None` with the `image_url` omission rule: **ABSENT (never null) when `page_count` is NULL** — every non-qualifying doc's payload stays byte-identical to pre-phase. For a qualifying doc the key is present; a **NULL-mode pre-phase row serializes ABSENT** and the viewer treats absent as `"primary"` (existing rows keep today's DOM until the re-gate stamps them). The `/pages/{n}` route is untouched.
**D2 — write-first atomic persist (task 02, LOCKED A4).** `_persist_pdf_pages` reorders: **render → write every page (atomic per file: write to `<name>.png.tmp`, `os.replace`) → only then delete stale copies** outside the written set (the shrunk-PDF tail; the current blanket `_delete_page_copies` BEFORE the render is removed — a rollback or crash can no longer leave the row over zero files). The partial-write failure path unlinks BOTH the new final files and the `.tmp` files before re-raising (no orphans; the existing fail-soft contract — caller sets `page_count` NULL + warning — unchanged). The function also stamps `doc.page_strip_mode`: `"primary"` when `fallback_fired`, `"additive"` when `pictorial_described` (the caller passes the CLASS, not just the boolean — `eligible=fallback_fired or pictorial_described` is replaced by `strip_class="fallback"|"pictorial"|None`); the ineligible arm clears mode with `page_count`. **The eligibility RULE itself is unchanged** (the phase-148 lock `fallback_fired or pictorial_described` stands — only its VIEW consequence changes, via D1/D4).
**D3 — the deterministic fast-path re-gate (task 03, LOCKED A5).** A new `_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths — inside `_unchanged_doc` (an optional `full_path` kwarg, defaulting to `Path(doc.full_path)`), so it fires on the `file_hash` fast path AND the content-digest-hit path — guarded by `doc.page_count is not None` (only PDF rows ever carry it; every other file's fast path keeps its zero-extra-read contract). It mirrors the changed-path gate **deterministically from the current file bytes + current settings — ZERO vision/LLM calls** (`render_pdf_pages` is CPU-only, A21-safe):
- `not is_usable_pdf_text(extract_pdf_text(full_path))` → class `fallback` (mode `primary`) — same doc as today;
- elif `settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=settings.pdf_pictorial_min_chars)` is non-empty → class `pictorial` (mode `additive`) — **candidates are exactly the existing gate's** (text-carrying pages under `min_chars`; no-text pages are NOT candidates, mirroring `pictorial_pages` — the re-gate adds no new semantics);
- else → **repair arm**: the strip no longer qualifies (e.g. the owner lowered `pdf_pictorial_min_chars`) → `page_count = None`, `page_strip_mode = None`, `_delete_page_copies(doc, page_dir)`.
For a re-gated-eligible doc, **verify the copies**: any of `page_dir/<doc.id>-p1..p{page_count}.png` missing → re-render ALL pages (`render_pdf_pages`), atomic write-first (D2's writer), and set `page_count` to the fresh render's length (a drift names the doc in a warning; bytes unchanged makes drift pathological). A fresh render producing NO pages (unopenable despite the just-computed hash — pathological) keeps the row as-is with a warning (fail-soft, never clears on a transient). One greppable INFO line whenever re-gate ACTS (never per-unchanged-doc): `import: PDF strip re-gate source=… path=… — healed N pages | cleared stale strip | mode <old>-><new>`. The re-gate mutates the row INSIDE the unchanged path's transaction (the commit already owned by `_unchanged_doc`'s caller flow); a copy-verify/render exception is caught, warned, and never fails the sync.
**D4 — the additive viewer (task 04, LOCKED A1/A6).** `renderDocument` (the ONE shared core — modal AND `/document.html`): the strip branch now keys on the mode —
- `page_strip_mode` absent or `"primary"` (with `page_urls`): the EXACT current DOM (`.doc-pages` strip, no toggle, no Summary panel) — byte-identical, pinned;
- `"additive"` (with `page_urls`): the text doc renders **exactly as a non-qualifying text doc** (Summary panel + content, the pre-148 branch reused), PLUS a `Text`/`Pages` toggle above the content: `<div class="doc-view-toggle" role="group" aria-label="Document view">` with two `<button type="button" aria-pressed>`; **default `Text`** (LOCKED A6 — no sparse-ratio default flip); switching to `Pages` appends/reveals the `docPagesStrip` (imgs `loading="lazy"` — hidden-until-shown never fetches) and hides the text block; switching back reverses. State is per-render (no persistence). createElement + properties/textContent only (the never-build-HTML rule); styles.css gains `.doc-view-toggle` per the house button idiom (focus-visible ring, ≥4.5:1, pressed state is text + style, never color alone — B5).
- no `page_urls`: byte-identical today's text doc (untouched).
**D5 — the live repair (task 05, records; no code).** With D3 shipped, the next production Sync takes the `Letumdei` doc's unchanged fast path → re-gate: usable text layer + ≥1 sparse candidate (page 36) → mode `additive` stamped → all 36 copies missing → re-rendered → the viewer shows the book's text with the Pages toggle. The IKEA-class doc (bytes unchanged, copies present, mode NULL pre-phase) gets mode `additive` stamped with zero render work. The E2E fixture reproduces the exact live shape (one illustration block on the last page, copies deleted out from under a stamped `page_count`).
**Invariants (all tasks).** The changed-path eligibility rule (`fallback_fired or pictorial_described`), every vision pass, the `/pages/{n}` route, the fallback-class strip DOM, the non-qualifying payloads, all SSE frame shapes, the mock-LLM markers, and the prune (`_delete_page_copies` at prune time) are byte-identical. `PLAN.md` untouched (the §7 viewer line describing the strip is folded at the next redo; AGENTS.md's viewer bullets get the additive note in the final pass).
## Tasks
1. `01_schema_wire_strip_mode.md` — the `page_strip_mode` column + migration 0027 + the `DocContent` field + the payload builder + unit pins (D1).
2. `02_persist_write_first.md` — `_persist_pdf_pages` write-before-delete + atomic replace + the `strip_class` stamp (D2).
3. `03_fast_path_regate.md` — `_regate_pdf_strip` on the unchanged paths: mirror gate, heal, repair, log line (D3).
4. `04_viewer_additive_toggle.md` — `renderDocument` mode branch + Text/Pages toggle + styles + frontend unit pins (D4).
5. `05_e2e_selfheal_pins.md` — the dedicated Playwright suite (additive view, primary byte-identity, heal, repair arm), the final gates, the docs pass (D5 + gates).
## Testing & Quality
- Unit: `tests/unit/test_pdf_page_copies.py` (D2: write-first order, atomic replace, partial-write cleanup, the class stamp; D1: column defaults), `tests/unit/test_importer_unchanged_fast_path.py` (D3: every re-gate arm incl. kill-switch, heal, repair, fail-softs, the zero-extra-read guard for non-strip docs), `tests/unit/test_docs_api.py` or the phase-148 payload-pin file (D1: the key present/absent rule, the byte-identity of non-qualifying payloads), a frontend source-contract file per the house style (D4: the mode branch, the toggle semantics, the never-build-HTML rule, the primary-DOM byte-identity).
- E2E: `tests/e2e/test_pdf_strip_additive_selfheal.py`, **in isolation** (`uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov`); `test_pdf_pages_in_sources.py` / `test_pictorial_pdf.py` / `test_garbage_pdf.py` stay green (the re-cut pins included).
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] A pictorial-class PDF opens in the viewer showing its text + Summary panel with a Text/Pages toggle (Text pressed by default); Pages shows the strip and every page image loads (200); a fallback-class PDF's strip-only DOM is byte-identical to pre-phase (no toggle).
- [ ] The changed path writes copies before deleting any (measured by the unit order pin); a mid-write failure leaves the prior copies intact and no orphans.
- [ ] Deleting a strip's copies on disk + a re-sync (unchanged bytes) restores them and stamps the mode (the E2E heal test); a stale `page_count` on a now-ineligible doc is cleared on re-sync with the copies removed (the E2E repair test); zero vision/LLM calls on the fast path (asserted via the mock).
- [ ] Non-qualifying-doc payloads and the primary-class DOM are byte-identical; the full `uv run pytest` suite is green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass (the viewer description gains the additive-mode sentence; `PLAN.md` untouched per the invariants).
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate. Post-deploy (owner runtime, recorded here): the next production Sync self-heals `Letumdei` per D5 — no manual script.
## Locked decisions
- **A1 — additive strip**: usable-layer (pictorial-class) PDFs show text + Summary + a Text/Pages toggle; the text is NEVER hidden for them; fallback-class keeps the strip-only view byte-identical (owner-confirmed 2026-10-07 — supersedes phase-148's "strip instead of the text content" for the pictorial class only).
- **A2 — the wire signal**: `page_strip_mode: "primary"|"additive"` ABSENT (never null) when `page_count` is NULL; absent ⇒ viewer treats `"primary"`; non-qualifying payloads byte-identical (owner-accepted default, 2026-10-07).
- **A3 — the column**: `documents.page_strip_mode` VARCHAR(16) nullable, migration 0027 with a tested downgrade (A13); stamped/cleared exactly with `page_count`.
- **A4 — write-first atomic copies**: render → atomic per-file write (tmp + `os.replace`) → stale-tail delete; a rollback can never leave the row over zero files; the fail-soft contract unchanged (owner-confirmed 2026-10-07).
- **A5 — the deterministic re-gate**: unchanged paths mirror the changed-path gate exactly (`is_usable_pdf_text` / `pictorial_pages` + the `pdf_pictorial` kill switch, current settings), re-render missing copies (CPU-only, no vision), or clear a stale strip; only rows with `page_count` pay the extra PDF parse; fail-soft on any re-gate exception (owner-confirmed 2026-10-07).
- **A6 — default view = Text** for additive docs (no sparse-ratio default flip; the illustration descriptions already live in the text via phase 140) (owner-accepted default, 2026-10-07).
- **A7 — the changed-path eligibility RULE is unchanged** (`fallback_fired or pictorial_described`, phase-148 lock intact); this phase changes only the persist ordering, the recovery, and the view consequence.
## Commit
```bash
git add app/ frontend/ tests/ alembic/ .agents/phases/ && git commit --no-gpg-sign -m "fix(pdf): additive text+pages viewer for pictorial PDFs, write-first atomic page copies, and a deterministic fast-path re-gate that self-heals lost strip images"
```
@@ -0,0 +1,26 @@
# Task 01 — The `page_strip_mode` column + wire signal (phase-163 D1, LOCKED A2/A3)
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D1 — the live modal showed the strip replacing the text of a genuinely textual book; the viewer needs an explicit class signal (fallback class = strip primary; pictorial class = additive) that never disturbs non-qualifying payloads.
**Story:** n/a
## Objective
`documents.page_strip_mode` (VARCHAR(16), nullable, migration 0027 with a tested downgrade) exists and `DocContent` carries `page_strip_mode` with the `image_url` omission rule — ABSENT (never null) when `page_count` is NULL, present for qualifying docs — so the frontend (task 04) can branch on the class, and every non-qualifying doc's payload stays byte-identical to pre-phase.
## Work
1. `app/models.py` — `Document.page_strip_mode: Mapped[str | None]` (VARCHAR(16), nullable, default NULL), docstring right after `page_count` (phase 148's column): the strip CLASS — `"primary"` = the phase-133 fallback class (the strip IS the view — the screenshots are the content), `"additive"` = the phase-140 pictorial class (the text view + the strip behind a toggle); NULL = no strip. Stamped/cleared exactly with `page_count` (tasks 02/03 own the writes; this task only adds the column).
2. `alembic/versions/0027_documents_page_strip_mode.py` — add the column; downgrade drops it. Follow `0026_documents_page_count.py` exactly (revision naming, `nullable=True`, no index). `uv run alembic upgrade head && uv run alembic downgrade -1 && uv run alembic upgrade head` round-trips.
3. `app/schemas.py` — `DocContent` gains `page_strip_mode: str | None = None` beside `page_urls`, docstring: the phase-163 strip class; ABSENT (never null — the `image_url` omission rule, `exclude_none`-style serialization already in place for `image_url` — verify how `DocContent` drops `image_url=None` and apply the SAME mechanism) when there is no strip; `"primary"`/`"additive"` otherwise. A pre-phase row (column NULL while `page_count` is set) serializes ABSENT — the viewer treats absent as `"primary"` (task 04).
4. `app/api/docs.py` — the content payload builder (the `page_urls=(…)` block, ~line 250): pass `page_strip_mode=doc.page_strip_mode` (the None-drops-the-key serialization handles the absent rule). The `/documents/{doc_id}/pages/{n}` route is NOT touched.
5. Unit tests — extend the phase-148 payload-pin file (find where `page_urls` serialization is pinned — `tests/unit/test_docs_api.py` or `test_pdf_page_copies.py`):
- a qualifying doc with mode set: the serialized payload carries `"page_strip_mode": "primary"` / `"additive"` next to `page_urls`;
- a doc with `page_count` set and mode NULL (pre-phase row): the key is ABSENT (not null) and `page_urls` still present — the exact live `Letumdei` state;
- a non-qualifying doc (`page_count` NULL): NO `page_strip_mode` and NO `page_urls` keys — pin the payload byte-identity against the current expected dict;
- `page_strip_mode` never appears alone: with `page_count` NULL it is absent even if the column somehow holds a value (the builder keys the whole strip block off `page_count` — if the current builder does NOT, make it so).
## Testing & Quality
- Unit: the cases above; the existing `page_urls` pins stay green unmodified except where a test asserts an exact payload dict (then re-cut in this task). Coverage **>90%** on `app/`.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies 0027; downgrade removes the column; upgrade again is clean.
- [ ] The three payload shapes (mode set / mode NULL with strip / no strip) serialize exactly per D1, pinned by unit tests.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,32 @@
# Task 02 — Write-first atomic page-copy persist + the strip-class stamp (phase-163 D2, LOCKED A4/A7)
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D2 — the live loss: the changed path deletes the old copies BEFORE rendering, outside the DB transaction, so any later per-file rollback (the `EmbeddingError` handler in the import loop) restores the stale `page_count` over ZERO files on disk (filesystem ops are not transactional). External wipes (`pdf_page_dir` cleaned) land the same state; the heal is task 03.
**Story:** n/a
## Objective
`_persist_pdf_pages` never removes a live copy before the replacement bytes are durably written: render → atomic per-file write (tmp + `os.replace`) → stale-tail delete, and the function stamps `page_strip_mode` from a `strip_class` keyword instead of the bare `eligible` boolean — the eligibility RULE (`fallback_fired or pictorial_described`, phase-148 lock) is unchanged at the call site.
## Work
1. `app/rag/importer.py` — `_persist_pdf_pages(doc, *, full_path, page_dir, strip_class, source, rel)` (replaces the `eligible: bool` keyword):
- `strip_class` is `"fallback" | "pictorial" | None` (the caller passes the CLASS — see 2).
- `strip_class is None`: clear BOTH `doc.page_count = None` and `doc.page_strip_mode = None`, THEN `_delete_page_copies(doc, page_dir)` (the deletion-first order is correct HERE — there is no replacement set to protect; note the helper reads `doc.page_count`, so call it BEFORE nulling or pass the old count — check the current order and keep the deletion actually deleting: today the helper runs FIRST while the count is still set; preserve that behavior).
- eligible: `render_pdf_pages(full_path)` → the no-pages guard unchanged (`page_count`/mode NULL, warning) → `page_dir.mkdir` → **write loop FIRST**: every non-empty page to `<doc-id>-p{n}.png.tmp` then `os.replace` onto the final name (atomic per file — a reader never sees a torn PNG); collect the written names → **then** delete the stale set: `for n in range(1, doc.page_count + 1)` beyond/present-but-not-written → unlink (the shrunk-PDF tail; reuse `_delete_page_copies` semantics by deleting the old extent files that are NOT in the written set — a file about to be overwritten is not deleted at all).
- the failure path: on any exception in the render/write, unlink this call's `.tmp` AND final-name files it wrote (track both), then re-raise (the caller's fail-soft handler — `page_count` NULL + warning — is extended to also null `page_strip_mode`). The OLD copies are untouched by a failed call (they were never deleted up-front — the crash/rollback orphan state is structurally impossible now).
- success tail: `doc.page_count = len(pages)` AND `doc.page_strip_mode = "primary"` if `strip_class == "fallback"` else `"additive"`.
- docstring: the write-first rationale (the 2026-10-07 live loss: delete-first + a later rollback = the row over zero files) + the per-file atomicity note.
2. `app/rag/importer.py` — the `_index_pdf_file` call site (~line 1671): `strip_class = "fallback" if fallback_fired else ("pictorial" if pictorial_described else None)` passed as the keyword; the `eligible=fallback_fired or pictorial_described` expression is GONE (the class carries the same truth table — pin it in the docstring: class non-None ⇔ eligible, the phase-148 rule verbatim). The `except Exception` fail-soft handler nulls mode too.
3. `tests/unit/test_pdf_page_copies.py` — extend (the file's existing fixture pattern):
- **the order pin**: patch `_delete_page_copies`-class file ops / use a real tmp dir — after a successful re-render of a CHANGED doc, the new files exist and the stale tail (old count > new count) is gone; the overwritten pages keep the NEW bytes;
- **the failure pin**: a render that raises mid-write (the 2nd of 3 pages) → the FIRST render's old copies still on disk (the rollback-orphan state is unrepresentable), no `.tmp`/partial orphans from this call, the caller's fail-soft fires (`page_count`/mode NULL + the warning);
- **the atomic pin**: the writer uses tmp + `os.replace` (spy or a failing-replace test — a reader opening the final path never sees truncated bytes; assert the tmp name pattern is used and `os.replace` is called);
- **the class stamp**: fallback → `page_strip_mode == "primary"`; pictorial → `"additive"`; ineligible → both NULL and old copies deleted;
- the existing phase-148 pins (gap = per-page `b""` leaves no file and counts in `page_count`; the shrunk tail) stay green, re-cut only where the deletion order changes expectations.
## Testing & Quality
- Unit: the pins above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] A successful changed-path render: new copies durable BEFORE any old copy is removed (measured by the order pin); the shrunk tail still cannot leak.
- [ ] A mid-write failure leaves the prior copies intact and zero orphans (files on disk exactly the prior set), `page_count`/`page_strip_mode` NULL, sync continues (the fail-soft unchanged).
- [ ] `strip_class` carries the exact phase-148 eligibility truth table; no call site passes a bare `eligible` anymore.
- [ ] `uv run pytest tests/unit/test_pdf_page_copies.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,37 @@
# Task 03 — The deterministic fast-path re-gate: heal & repair (phase-163 D3, LOCKED A5)
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D3 — with the copies lost (external wipe, or the pre-phase-163 delete-first window), the phase-147 fast path assumes "the copies persist" and NEVER re-renders: the live `Letumdei` row (36 `page_urls`, zero files, unchanged bytes on every sync since) is the proof. The unchanged paths must re-gate the strip deterministically — no LLM/vision calls — heal missing copies, and clear strips that no longer qualify.
**Story:** n/a
## Objective
`_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths (the `file_hash` fast path and the content-digest-hit path), guarded by `doc.page_count is not None`: it mirrors the changed-path gate from the current bytes + settings, re-stamps `page_strip_mode`, re-renders when any copy is missing, or clears a stale strip — one greppable INFO line only when it ACTS, fail-soft on every exception.
## Work
1. `app/rag/importer.py` — new `_regate_pdf_strip(session, doc, *, full_path, llm, summary) -> None`:
- **mirror the changed-path gate exactly** (the truth table in the phase overview D3):
`text = extract_pdf_text(full_path)`; `is_usable = is_usable_pdf_text(text)`;
- `not is_usable` → class `fallback` (mode `"primary"`) — the phase-133 signal verbatim;
- elif `llm.settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=llm.settings.pdf_pictorial_min_chars)` non-empty → class `pictorial` (mode `"additive"`) — **the existing gate verbatim** (text-carrying pages under `min_chars` only; no-text pages are NOT candidates — `pictorial_pages`'s own contract; the re-gate adds no new eligibility semantics);
- else → **repair arm**: `_delete_page_copies(doc, page_dir)` (while `page_count` is still set — the helper reads it), then `doc.page_count = None`, `doc.page_strip_mode = None`; INFO `import: PDF strip re-gate source=… path=… — cleared stale strip`.
- **mode re-stamp** (eligible arms): a mode change (NULL→stamped, or the owner's kill-switch/class flip) writes `doc.page_strip_mode` and, when it changes an already-set mode, INFO `… mode <old>-><new>`; a NULL→stamp on existing rows is silent-beneficial (fold into the same line only when work follows).
- **copy verify + heal** (eligible arms): `page_dir = Path(llm.settings.pdf_page_dir).expanduser()`; missing = any of `page_dir/f"{doc.id}-p{n}.png"` for `n in 1..doc.page_count` absent. Any missing → `render_pdf_pages(full_path)`: the no-pages result (unopenable — pathological right after the fast path hashed the file) keeps the row untouched + WARNING `import: PDF strip re-gate source=… path=… — re-render produced no pages (state kept)`; else WRITE EVERY page via task-02's atomic writer (tmp + `os.replace`) — reuse it (extract the shared writer helper from task 02 rather than duplicating), then delete the stale tail beyond the new count; `doc.page_count = len(pages)` — a drift from the old count adds a WARNING naming both counts; INFO `… healed N pages`.
- **fail-soft**: the whole body in `try/except Exception` → `logger.warning` (the re-gate never fails a sync; the row keeps its pre-re-gate state — but note the repair arm's DB writes and the heal's row writes are in the caller's transaction, so an exception AFTER them rolls back naturally with the file anyway).
- docstring: the 2026-10-07 live state as the rationale; the zero-LLM-call property (render is CPU — A21-safe); the guard means every non-strip file's fast path keeps its zero-extra-read contract.
2. `app/rag/importer.py` — `_unchanged_doc(...)`: gains the optional keyword `full_path: Path | None = None` (defaults to `Path(doc.full_path)`); at its tail, `if doc.page_count is not None and str(doc.path).lower().endswith(".pdf"): _regate_pdf_strip(session, doc, full_path=…, llm=llm, summary=summary)`. Call sites: the `_index_file` fast path passes the live `full_path`; the branches' content-digest-hit calls pass their own `full_path` (every call site already has it). Docstring: the phase-147 "the copies persist" paragraph is replaced — "unchanged files re-gate the PDF strip deterministically (phase 163)".
3. `tests/unit/test_importer_unchanged_fast_path.py` — extend (the file's fast-path fixture pattern; the mock embedder asserts ZERO chat/vision calls across every case):
- **heal**: a stamped strip doc (`page_count=3`, mode NULL or set), copies deleted out from under it, unchanged bytes → sync → the 3 files exist (PNG bytes), mode stamped `"additive"` (usable layer + sparse candidate), INFO `healed 3 pages`, `page_count` unchanged (no drift);
- **fallback heal**: an unusable-layer doc → mode `"primary"`, copies restored;
- **no-op**: all copies present, mode set → zero file writes, zero INFO lines (pin "INFO never fires per unchanged doc" — assert the log records contain no re-gate line);
- **repair**: a row with `page_count` set (copies present) on a PDF whose gate now says ineligible (clean text, zero sparse pages) → cleared (both columns NULL, copies deleted), INFO `cleared stale strip`;
- **kill switch**: `pdf_pictorial=False` + pictorial-class doc → repair arm clears the strip (the changed-path semantics mirrored);
- **fail-soft**: `render_pdf_pages` patched to raise → sync completes, row keeps its prior state, WARNING, no copies touched;
- **guard**: a non-PDF unchanged file and a PDF with `page_count` NULL → `extract_pdf_text` NEVER called (spy) — the zero-extra-read contract intact.
## Testing & Quality
- Unit: the cases above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] Unchanged re-sync restores deleted copies (files + row + route-serveable) and stamps the mode — the live-loss state cannot persist past one sync (unit-proven; the E2E pins it in task 05).
- [ ] A stale strip on a now-ineligible doc (incl. via the kill switch) is cleared with its copies; the re-gate fires ZERO LLM/vision calls and no INFO when there is nothing to do.
- [ ] Non-strip fast paths perform zero PDF parsing (spy-pinned); every re-gate exception is fail-soft.
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py tests/unit/test_pdf_page_copies.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,35 @@
# Task 04 — The additive viewer: text + Summary + the Text/Pages toggle (phase-163 D4, LOCKED A1/A6)
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D4 — the live complaint: "Notice how all the pages are unavailable… it shouldn't be a pictorial pdf." One short map page flipped a 114 KB-text book into the phase-148 strip-only view (the strip renders INSTEAD of the text content AND the Summary panel). The owner's call: additive for the pictorial class — the text is never hidden; the strip is a toggle away.
**Story:** n/a
## Objective
`renderDocument` (the ONE shared core — the Sources modal AND `/document.html`) branches on `doc.page_strip_mode`: absent/`"primary"` keeps the current strip-only DOM byte-identically; `"additive"` renders the doc exactly as a text doc (Summary panel + content) plus a `Text`/`Pages` toggle (default `Text`) revealing the existing `docPagesStrip`.
## Work
1. `frontend/assets/document.js` — the `renderDocument` strip branch (~line 374, `const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;`):
- `pageUrls` truthy AND `doc.page_strip_mode === "additive"`: fall through to the EXISTING text-doc rendering (the pre-148 branch — Summary panel + content, reused unchanged), THEN append the view toggle + the strip container:
- `<div class="doc-view-toggle" role="group" aria-label="Document view">` + two `<button type="button" class="doc-view-btn" aria-pressed>` — labels `Text` / `Pages`; initial state `Text` pressed (`"true"`), the strip NOT in the content flow;
- `Pages` click: `aria-pressed` flips; the text block + Summary panel hide (a class toggle, e.g. a `.doc-views[data-view="pages"]` wrapper — CSS-driven, never re-render), the `docPagesStrip(doc, pageUrls)` (the EXISTING builder, unmodified — lazy imgs never fetch while hidden) is appended once and shown;
- `Text` click reverses (the strip node stays mounted — no re-fetch on flip-flop);
- structure: wrap the text view + the strip in one `.doc-views` container appended to `contentEl` (both surfaces get it via the shared core; the wrapper exists ONLY on additive docs — every other doc's DOM is untouched).
- `pageUrls` truthy AND mode absent or `"primary"`: the CURRENT branch verbatim (strip only, no Summary panel, no toggle) — byte-identical DOM, pinned;
- falsy `pageUrls`: untouched.
- the never-build-HTML rule: createElement + properties/textContent only (the labels are fixed strings — no doc-derived text inside the toggle); the `strip_mode` read is `doc.page_strip_mode === "additive"` (anything else ⇒ primary — the tolerant-reading house rule).
- the module docstring: the phase-163 paragraph (the additive class contract, LOCKED A1/A6) replacing the phase-148 "INSTEAD of the text content AND the .doc-summary panel" sentence's scope (it now reads: primary class only).
2. `frontend/assets/styles.css` — `.doc-view-toggle` (house button idiom beside the existing segmented/pill controls — check `.format-badge`/the modal action buttons for the closest idiom): the two buttons adjacent, pressed state via `aria-pressed="true"` styling (weight/background pair — **text + style, never color alone**, B5), `:focus-visible` 3px ring, ≥4.5:1 text contrast on both states, ≥44px touch targets on mobile (§7.1); `.doc-views[data-view="pages"]` hides the text block + Summary panel and shows the strip; default (`data-view="text"`) hides the strip container entirely.
3. Unit (frontend source-contract — the house pattern; extend the file that already pins the phase-148 strip DOM — locate it via `docPagesStrip` in `tests/unit/`, likely `test_pdf_page_copies.py` / a viewer-contract file):
- the additive branch builds the `.doc-views` wrapper + the toggle with the exact labels/roles/`aria-pressed` initial state (source-regex or the file's established DOM-assertion style);
- the primary/absent branch contains NO `doc-view-toggle` string and the strip-only structure is byte-identical to the pre-phase pins;
- the toggle handler mutates only `aria-pressed` + the `data-view` attribute (no innerHTML anywhere in the new code path — the never-build-HTML pin);
- `loading = "lazy"` remains on the strip imgs (the existing pin re-runs green).
## Testing & Quality
- Unit: the contract pins above; coverage **>90%** on `app/` (no `app/` change in this task — the gate must simply stay green).
- The phase's E2E (task 05) exercises the real DOM; `test_pdf_pages_in_sources.py` (the phase-148 E2E) is re-cut ONLY where a modal-DOM assertion meets the additive fixture (the fixture's mode stamp decides; prefer fixture docs that keep primary semantics for the existing pins).
## Completion Criteria
- [ ] An additive doc shows Summary + text with `Text` pressed by default; one click shows the strip (every page img loads); clicking back shows the text with zero refetch (the strip node persists — E2E-pinned in task 05).
- [ ] A primary/absent-mode doc's viewer DOM is byte-identical to the pre-phase structure (unit pin green); a text doc and every non-PDF are untouched.
- [ ] The toggle passes the WCAG basics: labeled group, `aria-pressed` state as text + style, focus-visible ring (checked against §7.2; the E2E asserts the attributes).
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,33 @@
# Task 05 — The dedicated E2E: additive view, primary byte-identity, heal & repair; the final gates (phase-163 D5)
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D5 + the AGENTS.md per-phase E2E rule (A16).
**Story:** n/a
**Pre-fix evidence (measured on the live server, 2026-10-07 — recorded here per house practice):** `Letumdei_ Heirs to the Imperium.pdf` — payload carries 36 `page_urls`; `GET /api/documents/b72f5515-dc40-4167-98ff-179a6503e6e5/pages/1,2,18,36` → **404/404/404/404** (page 37 → 404, the out-of-range arm); the content is 114 389 chars of genuine extracted text with 36 `---PAGE-N---` markers and exactly ONE `[Page 36 illustration: …]` block; the same server serves `besta-frame-white__AA-1272068-11-100.pdf` page 1 → **200** (the dir is intact; only the 10/1-indexed doc's copies were lost). The viewer showed "Page N unavailable." on every page with the text hidden.
## Objective
One Playwright suite proves the whole phase end to end — the additive view with the toggle, the fallback class's byte-identical strip-only DOM, the disk-level heal, and the repair arm — and the phase's final pass runs every gate plus the full documentation consistency review.
## Work
1. `tests/e2e/test_pdf_strip_additive_selfheal.py` — new suite (run in isolation; read `tests/e2e/test_pdf_pages_in_sources.py` and `tests/e2e/test_pictorial_pdf.py` first and reuse their synthetic-PDF builder + fixture-KB pattern; deterministic mock LLM, the default):
- **fixture**: source `strip_fixture` with (a) `manual.pdf` — usable text layer, 3 text pages + 1 sparse `<100-char` map-style page (the pictorial class — the Letumdei shape in miniature); (b) `scanned.pdf` — the garbage/unusable-layer fixture (the phase-133 pattern) → fallback class; (c) `plain.pdf` — clean text, zero sparse pages → no strip.
- **assert 1 (additive view)**: index → the modal for `manual.pdf`: mode stamped `additive` (via the content API), the text + Summary panel visible by default, the `Text`/`Pages` toggle present with `Text` `aria-pressed="true"`; click `Pages` → the strip renders, EVERY page `<img>` has `naturalWidth > 0` (routes serve — no "unavailable" note anywhere); click `Text` → text visible, strip hidden, the strip node still mounted (no refetch: assert the img count does not reset).
- **assert 2 (primary byte-identity)**: the modal for `scanned.pdf`: the strip-only DOM (no `.doc-view-toggle`, no Summary panel) and the "Page N" figures match the phase-148 structure (the existing suite's DOM assertions adapted); `plain.pdf` renders the plain text doc with no strip artifacts.
- **assert 3 (heal)**: for `manual.pdf`, delete ALL `pdf_page_dir/<id>-p*.png` (the fixture's page dir) + confirm a page route 404s → run a sync (unchanged bytes → the fast path) → the files exist, the page route 200s, the payload still carries `page_urls` + `additive`, the viewer's Pages view loads every img (the live-loss state, healed by the next sync — the exact D5 promise).
- **assert 4 (repair)**: direct-DB seed — set `page_count` (and stale copies on disk) on `plain.pdf`'s row (the stale-eligible state no current index produces) → sync → `page_count`/`page_strip_mode` NULL (read through the fixture session), the stale copies deleted, the modal shows the plain text (the strip gone).
- **assert 5 (no vision)**: the whole suite's syncs ran against the deterministic mock — no chat/vision endpoint hits beyond the initial index's own passes (the heal burns zero LLM calls; assert via the mock's call counter or the suite's request log, the house pattern).
2. Final pass (the phase pipeline runs this after the tasks): full-suite + gates:
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**;
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` green **in isolation**; `tests/e2e/test_pdf_pages_in_sources.py`, `test_pictorial_pdf.py`, `test_garbage_pdf.py`, `test_document_viewer*.py`, `test_image_documents.py` green;
- `uv run ruff check . && uv run pyright` clean; the alembic 0027 up/down/up round-trip clean.
3. Documentation & consistency (mandatory): review `README.md` and `AGENTS.md` IN FULL against the post-phase app — the viewer description gains the additive-mode sentence (text + Text/Pages toggle for pictorial-class PDFs; the fallback class keeps the strip), the importer notes (write-first copies, the fast-path re-gate + its INFO line) — and fix EVERY inconsistency found, including pre-existing ones not caused by this phase. `PLAN.md` is NOT touched (its §7 strip line folds at the next redo — recorded in the phase overview invariants).
## Testing & Quality
- This task IS the E2E + gates task; its own new-code coverage rides the suite (page route + fixture helpers are test-side; `app/` coverage stays **>90%**).
## Completion Criteria
- [ ] All five E2E asserts green in isolation; the four existing viewer/PDF suites green unmodified (except explicitly re-cut pins, each named in the commit body).
- [ ] The heal assert proves: copies deleted externally → one unchanged-byte sync → files restored, routes serve, viewer complete (the live `Letumdei` failure mode is impossible after this phase).
- [ ] `uv run pytest` fully green; coverage >90%; ruff + pyright clean; migration round-trip clean.
- [ ] `README.md` + `AGENTS.md` reviewed in full, every inconsistency fixed (the additive-viewer sentence included); the phase dir moves to `complete/` and the single `--no-gpg-sign` commit (see the overview) is made by the pipeline gate.
- [ ] Owner follow-up recorded (not code): after deploy, the next production Sync self-heals `Letumdei` (D5) — verify once via the Sources modal (text visible, Pages loads all 36) and `grep` the server log for the one `import: PDF strip re-gate … healed 36 pages` line.
@@ -0,0 +1,66 @@
# Phase 164 — Chat failure honesty: the shared error-detail serializer, a distinct pre-stream failure line, and caps that accept a full-length answer
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, right after the live answer cap was raised to `BOR_LLM_MAX_OUTPUT_TOKENS=131072`: when an answer hit the length limit, the next turn's bubble read *"My answer didn't make it — the connection dropped. Use Retry to ask again. Failed [object Object]."* The failure chain (traced in code, 2026-10-07): a long answer (> 32 000 chars) lives in `bor.chat.v1` and rides back to the server as `history` on the next `POST /api/chat`; `HistoryTurn.text` (`max_length=32_000`) rejects it with a **422 whose `detail` is a LIST OF OBJECTS** (`[{loc, msg, type}, …]`); the chat's pre-stream error path (`frontend/assets/app.js` ~3956: `if (body.detail) detail = body.detail`) has **no type guard** — `new Error(<array>)` stringifies to `"[object Object]"`, which the zero-frame settle shows as the fixed `FAILED_TURN_TEXT` ("the connection dropped" — untrue: the server WAS reachable and answered 422) plus the garbage detail. This is the SECOND occurrence of the same class: `HistoryTurn`'s own docstring records the 4 000 → 32 000 raise for an identical 422 (phase-42 E2E); the 32 000 ceiling broke again the moment outputs could exceed it — **at the DEFAULT `max_output_tokens=32_768` (~130k chars), not only at 131072**.
**Story:** n/a (owner-directed bug fix; extends the phase-120 failed-turn contract, the phase-114 hint channel, the phase-83 SEC-05 boundary caps, the phase-74 history budgets).
## Objective
A failed turn states **what actually happened**: pre-stream HTTP rejections get their own honest line (not "the connection dropped") with the validation message serialized verbatim — never `[object Object]` — and a full-length answer (up to a 131 072-token cap ≈ 524k chars) flows through the whole chain: accepted as `history` on the next turn, and saveable/restorable as part of a saved chat. The anonymous saved-chat surface keeps a bounded worst case (the phase-83 SEC-05 property survives the raise).
## Dependencies
- `83_chat_save_payload_limits` (complete) — the SEC-05 caps this phase raises TOGETHER (`ChatMessage.text`/`thinking` mirror `HistoryTurn`; the mirror invariant must stay true).
- `74`-line history budgets (complete; `history_to_messages`, `history_max_turns=40` / `history_max_chars=24_000`) — the LOCKED budget trim (owner 2026-09-08: overflow turns dropped WHOLE, never cut) — **untouched**; it is the real prompt-side gate and stays so.
- `120_failed_turn_persistence` (complete; the `FAILED_TURN_TEXT` / `finalizeFailedTurn` / `.failed-note` contract) and `114` hint channel (`err.hint` → banner) — the bubble/note mechanics this phase extends.
## Design (shared by all tasks — the executor reads this, not the chat)
**D1 — the caps chain up, together (task 01, LOCKED A1).** `app/schemas.py`:
- `HistoryTurn.text` / `HistoryTurn.thinking`: `max_length=600_000` (rationale IN the docstring: a full-cap answer at the owner's `BOR_LLM_MAX_OUTPUT_TOKENS` up to 131 072 ≈ 524 300 chars at ~4 chars/token — 600 000 is that with margin; the budget trim (locked A3, 2026-09-08) remains the real prompt gate — a turn over `history_max_chars` is dropped WHOLE, so a long answer trims itself out of later prompts; this phase fixes the 422, NOT the trim (its locked semantics are a separate owner decision, recorded in the overview));
- `ChatRequest.history` list: `max_length=100` → `500` (the same "DoS sanity ceiling only" docstring, now also noting the client sends the whole `bor.chat.v1` minus the current question);
- `ChatMessage.text` / `ChatMessage.thinking`: `max_length=600_000` (MIRRORS `HistoryTurn` — the phase-83 invariant "a saved chat can never legitimately carry more than the chat path accepts" must stay TRUE: raise both or neither);
- `SavedChatCreate.messages` / `SavedChatUpdate.messages`: `max_length=200` → `500` (symmetry with the history list) PLUS a new `model_validator` on both: **total chars across all messages (`len(text) + len(thinking or "")`) ≤ 8_000_000** — this bounds the ANONYMOUS SEC-05 surface at the same order as the audited pre-phase worst case (200 × 64k ≈ 12.8 MB) while fitting ~15 full-cap answers; the validator error surfaces as the standard 422 (`ValueError` message names the budget);
- `ChatRequest.message`: `max_length=4000` UNCHANGED (the composer textarea carries the cap client-side — pin it).
**D2 — the shared `errorDetailText` serializer (task 02, LOCKED A2).** `frontend/assets/app.js` exports `errorDetailText(value, fallback)` (exported const/function, the `TURN_TIMEOUT_MS` unit-pin idiom): a non-empty **string** → verbatim; an **array** → per-entry `entry.msg` when the entry is an object else `String(entry)`, empties dropped, `"; "`-joined, all-empty → `fallback`; an **object** → `String(entry.msg)` when present, else `fallback`; nullish / empty / anything else → `fallback`. NEVER `JSON.stringify` a raw detail (the FastAPI 422 entries carry the FULL `input` — the whole conversation — which must never reach the DOM; only `msg` and plain strings serialize). Applied at the TWO unguarded chat paths: the pre-stream `!res.ok` handler (~3956 — `detail = errorDetailText(body.detail, \`Brain's API answered with HTTP ${res.status}.\`)`) and the SSE `error`-frame throw (~4353 — `new Error(errorDetailText(ev.detail, "Something went wrong on my side."))`, the `err.hint` line untouched). The seven EXISTING guarded copies (`app.js` tuning form, `tuning.js`, `theme.js`, `document.js`, `git-sources.js`, `doc-edit.js`) are behaviorally correct and stay byte-identical (no opportunistic refactor — the new helper matches their semantics for the shared shapes, generalized to arrays-of-strings and multi-entry joins).
**D3 — the distinct pre-stream failure line (task 03, LOCKED A3/A4).** `app.js`:
- `const FAILED_REQUEST_TEXT = "I couldn't take that — my API rejected the request. Use Retry to ask again.";` (sibling of `FAILED_TURN_TEXT`, its docstring: the PRE-STREAM HTTP-rejection settle — the server answered, the request did not pass; the real detail rides the `.failed-note` + persisted `error` key exactly as `FAILED_TURN_TEXT` does);
- the pre-stream `!res.ok` throw carries `err.bubbleText = FAILED_REQUEST_TEXT` and `err.hint = "The app answered — it rejected the request before the model ran. Nothing was lost but this send."` (the phase-114 banner-hint channel);
- `runTurn`'s catch passes `bubbleText: err?.bubbleText` into `finalizeFailedTurn` (new options key); the zero-frame settle uses `bubbleText || FAILED_TURN_TEXT` — the persisted record text stays "what the user saw is what is stored";
- UNCHANGED settles: a transport-level failure (the `fetch` itself rejects — `TypeError`) has no `bubbleText` → keeps `FAILED_TURN_TEXT` (that one IS the connection); the SSE `error`-frame zero-frame path (a model-side failure) keeps `FAILED_TURN_TEXT`; the stop / abort paths are untouched. A partial-stream settle (`wrap` + `acc`) never shows a bubble text at all — unchanged.
**D4 — verification (task 04).** Dedicated Playwright suite `tests/e2e/test_chat_error_honesty.py` (isolation; deterministic mock LLM) with (a) a `page.route` stub of `POST /api/chat` returning the EXACT FastAPI 422 shape for a long-history turn → assert the bubble reads `FAILED_REQUEST_TEXT`, the `.failed-note` detail contains the pydantic `msg` verbatim ("String should have at most 600000 characters" — never `[object Object]`, never `input` text), the banner hint shows, Retry + Edit land, composer re-enables (§7.4); (b) a 500 stub with a non-JSON body → the `HTTP 500` default string line (the serializer's fallback path); (c) a real-server acceptance: send a turn whose history carries a ~525k-char prior brain answer → the request is ACCEPTED (no 422) and the turn settles normally with the mock (the budget trim may drop the giant turn WHOLE from the prompt — the locked A3 behavior; the test asserts acceptance + settle, not prompt content); (d) a stubbed SSE `error` frame carrying an ARRAY detail → the note serializes (client tolerance). Server-side acceptance/422 matrix for every raised cap + the total-chars validator rides integration pins in task 01 (the `test_chats_api.py` pattern).
**Invariants (all tasks).** The budget trim (`history_to_messages`, budgets, whole-drop, contiguous window) is byte-identical (locked A3 2026-09-08). `ChatRequest.message` 4000. The `FAILED_TURN_TEXT` wording, the phase-120 persistence keys (`failed`, `error` ≤500), Retry/Edit pairing, the SSE frame shapes, and the seven guarded frontend detail-readers are byte-identical. `PLAN.md` untouched.
## Tasks
1. `01_caps_long_answer_chain.md` — the `schemas.py` raise + the total-chars validator + docstrings + the unit/integration acceptance-rejection matrix (D1).
2. `02_error_detail_serializer.md` — the exported `errorDetailText` + the two chat call sites + unit pins (D2).
3. `03_prestream_failure_copy.md` — `FAILED_REQUEST_TEXT` + the `bubbleText` threading + the banner hint + the failed-turn pins (D3).
4. `04_e2e_error_honesty.md` — the dedicated E2E (D4) + the final gates + the docs consistency pass (incl. the proxy body-size deploy note).
## Testing & Quality
- Unit: `tests/unit/test_schemas.py` (or the file pinning `HistoryTurn`/`ChatMessage` bounds — locate via `32_000` in `tests/`): every cap's boundary (600_000 accept / +1 reject; 500 vs 501 turns; the total-8M validator: accept ~15 full-cap messages, reject beyond), the 422 `detail` shapes; `tests/unit/test_frontend_error_detail.py` (new; the `test_frontend_*` house pattern): all serializer shapes incl. the exact live FastAPI 422 body; `tests/unit/test_frontend_failed_turn.py` extended: the `bubbleText` threading (pre-stream vs transport vs SSE-frame settles), the never-`innerHTML` rule.
- Integration: `tests/integration/test_chats_api.py` extended — a saved chat carrying ONE 525k-char answer saves 201 (the phase-83 surface stays bounded by the total validator), over-total rejects 422.
- E2E: `tests/e2e/test_chat_error_honesty.py` green **in isolation**; the phase-120 failed-turn E2E stays green (the `FAILED_TURN_TEXT` path unchanged for transport failures).
- Coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] A pre-stream 422 shows the distinct honest line + the validation `msg` verbatim in the note — `[object Object]` is structurally impossible on the chat path (unit + E2E pinned; no raw `input` reaches the DOM).
- [ ] A history turn with a 525k-char brain answer is ACCEPTED (200 turn settles); >600k rejects 422; the saved-chat chain accepts the same answer inside a 201 and the anonymous surface stays bounded (total-8M validator pinned).
- [ ] Transport failures and SSE error frames keep `FAILED_TURN_TEXT` verbatim; stop/abort paths byte-identical; §7.4 failure feedback re-enables every control.
- [ ] The budget trim, `message` 4000, the seven guarded detail-readers, and all SSE shapes are byte-identical; full `uv run pytest` green; coverage >90%; ruff + pyright clean.
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass — includes the deploy note: the proxy `client_max_body_size` must accommodate the now-legal large chat POSTs (recommend ≥ 16 MB) — and every other inconsistency fixed.
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
## Locked decisions
- **A1 — the caps chain together**: `HistoryTurn.text/thinking` AND `ChatMessage.text/thinking` → 600_000 (≈ 131 072 × 4 chars with margin); history list → 500; saved-messages list → 500 with a total-chars ≤ 8_000_000 model validator (the SEC-05 bounded-anonymous property preserved at the audited order); `message` 4000 unchanged; the phase-83 mirror invariant explicitly honored (owner-confirmed 2026-10-07 — "static generous caps", with the SEC-05 coupling added by this phase's audit of the chain — raise ONE side would break saves).
- **A2 — `errorDetailText`**: the shared serializer's exact semantics (string verbatim; array entries `msg`-else-String joined `"; "`; object `msg`-else-fallback; never `JSON.stringify`; never the `input`), exported from `app.js`, applied at the two unguarded chat paths ONLY; the seven guarded copies untouched (owner-confirmed 2026-10-07).
- **A3 — `FAILED_REQUEST_TEXT`**: distinct pre-stream HTTP-rejection bubble text verbatim; the persisted record is what the user saw (the phase-120 contract extended, not changed); transport failures keep `FAILED_TURN_TEXT` (owner-confirmed 2026-10-07 — "distinct pre-stream line").
- **A4 — the budget trim is NOT touched**: the locked 2026-09-08 whole-drop contiguous-window semantics stand; the consequence — a >24k-char answer is dropped whole from later prompts (follow-ups WORK with a trimmed window, without that answer's context) — is recorded here; making long answers useful in later prompts (bigger `BOR_HISTORY_MAX_CHARS` is pure env, or a smarter cut) is an owner call outside this phase.
- **A5 — PLAN.md untouched** (the caps rationale lives in the schema docstrings; folded at the next redo).
## Commit
```bash
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): honest pre-stream failure line, a shared error-detail serializer, and history caps that accept a full-length answer"
```
@@ -0,0 +1,30 @@
# Task 01 — The caps chain up: 600k text, 500 turns, a bounded anonymous total (phase-164 D1, LOCKED A1/A4)
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D1 — the live 422: a >32k-char answer rode back as `history` and rejected the NEXT turn. `HistoryTurn`'s docstring already records this failure class happening once (4 000 → 32 000, phase-42 E2E); the ceiling broke again the moment outputs could exceed 32k — at the DEFAULT `max_output_tokens=32_768` too.
**Story:** n/a
## Objective
Every schema cap in the long-answer chain (`HistoryTurn` → `ChatRequest.history` → `ChatMessage` → saved-chat lists) accepts a full-cap answer (~524k chars at `max_output_tokens=131072`), the phase-83 mirror invariant between the chat path and the saved-chat path stays TRUE, and the anonymous saved-chat worst case stays bounded by a new total-chars validator.
## Work
1. `app/schemas.py` — `HistoryTurn`: `text`/`thinking` `max_length=32_000`/`32000` → **600_000** both; docstring: the cap accepts a full-length answer (`BOR_LLM_MAX_OUTPUT_TOKENS` up to 131 072 ≈ 524 300 chars at ~4 chars/token, plus margin — the 2026-10-07 live 422: the 32 000 ceiling rejected the FOLLOW-UP turn of a long answer, the same class as the recorded 4 000 → 32 000 raise); the real gate stays the locked budget trim (`history_to_messages` — overflow turns dropped WHOLE; a long answer trims itself out of later prompts, never a 422).
2. `app/schemas.py` — `ChatRequest.history`: `max_length=100` → **500**; the docstring sentence updates: still "a DoS sanity ceiling only — the config budgets do the real trimming (A3)", plus: the client sends the whole `bor.chat.v1` minus the current question, so the ceiling must clear long conversations.
3. `app/schemas.py` — `ChatMessage.text`/`thinking`: → **600_000** (docstring: the phase-83 SEC-05 mirror RAISED with `HistoryTurn` — the invariant "a saved chat can never legitimately carry more than the chat path accepts" survives the raise; raise ONE side and every done-time auto-save of a long answer 422s into the phase-113 quiet-failure).
4. `app/schemas.py` — `SavedChatCreate.messages` / `SavedChatUpdate.messages`: `max_length=200` → **500** AND a `model_validator(mode="after")` on each: total chars `sum(len(m.text) + len(m.thinking or "") for m in messages) <= 8_000_000` else `raise ValueError("saved chat exceeds the 8,000,000-character budget")` — docstring: bounds the ANONYMOUS SEC-05 worst case at the audited order (the pre-phase 200×64k ≈ 12.8 MB) while fitting ~15 full-cap answers; the 422 is the framework's standard validation response (the phase-83 A2 boundary-only style — NO route changes).
5. `ChatRequest.message`: **unchanged** `max_length=4000` (the composer textarea caps client-side — `app.js` ~1122 comment; pin it stays).
6. Unit — extend the file pinning these bounds (locate via `32_000` under `tests/unit/`):
- `HistoryTurn.text` 600_000 accept / 600_001 reject; same for `thinking`;
- `history` 500 accept / 501 reject; `message` 4000 boundary re-pinned unchanged;
- `ChatMessage.text/thinking` 600_000 accept / +1 reject;
- messages 500 accept; the total validator: ~15 × 524k messages REJECT with the budget-naming `ValueError` surfaced in the 422 `detail`, a saved chat with ONE 525k answer + normal turns ACCEPTS;
- the 422 `detail` SHAPE of a cap rejection is the pydantic list-of-objects (the serializer's task-02 input — pinned here as the server-side half of the contract).
7. Integration — `tests/integration/test_chats_api.py`: `POST /api/chats` with a one-525k-answer conversation → 201 (round-trips through JSONB — the DB is JSONB, no column changes needed — verify); over-total → 422; the existing phase-83 boundary matrix green (re-cut only the pins that assert the literal `32_000`/`200` values, naming each in the task's completion).
## Testing & Quality
- Unit + integration above; coverage **>90%** on `app/`.
## Completion Criteria
- [ ] The full-cap-answer conversation flows: accepted as `/api/chat` history (integration-level) AND saveable as a saved chat (201), with the 8M total cap the only new anonymous bound.
- [ ] `ChatMessage` mirrors `HistoryTurn` exactly (the phase-83 invariant re-stated in both docstrings, verified by identical-boundary tests).
- [ ] The budget-trim suites (`history_to_messages`) are untouched and green — D1 changes what ARRIVES, never what the trim keeps.
- [ ] `uv run pytest tests/unit tests/integration -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,35 @@
# Task 02 — The shared `errorDetailText` serializer for the two chat paths (phase-164 D2, LOCKED A2)
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D2 — the live "[object Object]": `frontend/assets/app.js` ~3956 does `if (body.detail) detail = body.detail` on a FastAPI 422 whose `detail` is a LIST OF OBJECTS; `new Error(<array>)` → message `"[object Object]"`. The SSE `error`-frame path (~4353 `new Error(ev.detail || …)`) is the second unguarded reader. The guarded idiom ALREADY exists in seven copies (`tuning.js`, `theme.js`, `document.js`, `git-sources.js`, `doc-edit.js`, `app.js`'s tuning form) — this task generalizes it ONCE and applies it ONLY where it is missing.
**Story:** n/a
## Objective
`app.js` exports `errorDetailText(value, fallback)` with exact, total semantics (every input shape yields a human string or the fallback — `[object Object]` and raw `input` leakage are structurally impossible), wired at the pre-stream HTTP-error handler and the SSE `error`-frame throw; the seven existing guarded copies stay byte-identical.
## Work
1. `frontend/assets/app.js` — new exported function (the `THINKING_NEAR_BOTTOM_PX` / `TURN_TIMEOUT_MS` export-for-unit-pin idiom; place beside the other error constants):
```
errorDetailText(value, fallback) → string
```
- non-empty **string** → the string verbatim;
- **array** → per entry: `String(entry.msg)` when the entry is an object with a non-empty `msg`, else (non-object entry) `String(entry)`; entries with neither drop; join with `"; "`; empty result → `fallback`;
- **object** → `String(value.msg)` when a non-empty `msg` exists, else `fallback`;
- nullish / empty string / any other type → `fallback`;
- NEVER `JSON.stringify` anything, NEVER touch `input`/`loc`/`ctx` (the FastAPI entries carry the full rejected input — the whole conversation — which must never reach the DOM). Docstring names the live 2026-10-07 `[object Object]` failure and the `input`-leak rule.
2. `frontend/assets/app.js` — the pre-stream handler (~3956, the `!res.ok || !res.body` block): replace the unguarded assignment with `const text = errorDetailText(body.detail, \`Brain's API answered with HTTP ${res.status}.\`); throw new Error(text);` (the `try`-around-`res.json` and the non-JSON catch keep their current shape; the default per-status string remains the fallback). The `bubbleText`/hint threading is task 03 — do NOT add it here.
3. `frontend/assets/app.js` — the SSE `error` frame (~4353): `const err = new Error(errorDetailText(ev.detail, "Something went wrong on my side."));` — the `err.hint = ev.hint;` line byte-identical. A string `detail` (every server-built frame, `ChatErrorEvent`) serializes VERBATIM — pin that the user-visible strings for "Question too long", "dropped the connection" etc. are unchanged.
4. `tests/unit/test_frontend_error_detail.py` (new — read `tests/unit/test_frontend_failed_turn.py` first and reuse its harness/assertion style):
- the EXACT live 422 body (`{"detail":[{"type":"string_too_long","loc":["body","history",5,"text"],"msg":"String should have at most 600000 characters","input":"<huge>"}]}`) → the `msg` verbatim, and the result contains NO `input` text and no `"[object Object]"`;
- multi-entry array → the `msg`s joined `"; "`; mixed entries (object + plain string) → joined; all-empty array → fallback;
- object with `msg` → verbatim; object without → fallback;
- string → verbatim (incl. the server's fixed copy — pinned); `""`/null/undefined/number → fallback;
- source-contract pins: the two call sites now read through `errorDetailText`; `JSON.stringify` appears NOWHERE in the new helper; the seven guarded copies are byte-identical (hash their regions or assert the untouched lines).
## Testing & Quality
- Unit: the shape matrix above (the harness the `test_frontend_*` files use); no `app/` change — coverage gate stays green.
## Completion Criteria
- [ ] Every serialized shape yields a human string or the named fallback — `[object Object]` is unrepresentable at the two chat call sites (unit-proven).
- [ ] A 422 `detail` never puts request `input` text into the DOM (pinned); server string frames render byte-identical copy.
- [ ] The seven existing guarded readers are untouched; the `err.hint` channel is untouched.
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,28 @@
# Task 03 — The distinct pre-stream failure line: `FAILED_REQUEST_TEXT` (phase-164 D3, LOCKED A3)
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D3 — the live bubble said *"My answer didn't make it — the connection dropped"* for a **422**: the server was reachable and answered deliberately; the copy lied, and the owner-confirmed remedy is a distinct honest line for pre-stream HTTP rejections.
**Story:** n/a
## Objective
A pre-stream HTTP rejection (the `!res.ok` settle) bubbles `FAILED_REQUEST_TEXT` — "I couldn't take that — my API rejected the request. Use Retry to ask again." — with the serialized detail on the `.failed-note` and the phase-114 hint on the banner; transport failures (the `fetch` itself rejects) and SSE `error` frames keep `FAILED_TURN_TEXT` byte-identically; the phase-120 persistence contract ("what the user saw is what is stored") extends to the new text.
## Work
1. `frontend/assets/app.js` — new constant beside `FAILED_TURN_TEXT`: `const FAILED_REQUEST_TEXT = "I couldn't take that — my API rejected the request. Use Retry to ask again.";` docstring: the PRE-STREAM HTTP-rejection settle (the server ANSWERED with a 4xx/5xx — the request died before the model; the 2026-10-07 live 422-after-a-long-answer is the finding); the real detail rides the in-bubble `.failed-note` + the persisted `error` key exactly as `FAILED_TURN_TEXT`'s does; NOT for transport failures (no answer happened at all — `FAILED_TURN_TEXT` is honest there).
2. `frontend/assets/app.js` — the pre-stream throw (task 02's site): attach `err.bubbleText = FAILED_REQUEST_TEXT;` and `err.hint = "The app answered — it rejected the request before the model ran. Nothing was lost but this send.";` to the thrown `Error` (the phase-114 `hint` channel already flows from `err.hint` to `showErrorBanner` via the catch's `setUiState` opts — verify, do not duplicate).
3. `frontend/assets/app.js` — `finalizeFailedTurn(detail, { acc, thinking, tools, segments, wrap, leavePartialIndex, bubbleText })`: the ZERO-frame branch (`else` — no `wrap`) renders `bubbleText || FAILED_TURN_TEXT` (the `addMessage` text AND the `rememberBrainTurn` record text — same value, the stored-what-you-saw contract) and the note/persistence/Retry/Edit machinery is otherwise untouched (partial-stream settles never receive a `bubbleText` by construction — a pre-stream throw has no `wrap`; still, the partial branch must ignore the key: read it ONLY in the zero-frame branch). Docstring update (the `bubbleText` key, its caller-only meaning).
4. `frontend/assets/app.js` — `runTurn`'s catch: pass `bubbleText: err?.bubbleText` into the `finalizeFailedTurn` options (the ONE line-level change; the detail/hint/banner sequencing after the call is untouched).
5. `tests/unit/test_frontend_failed_turn.py` (extend; the file's harness):
- a thrown `Error` with `bubbleText` → the zero-frame bubble + persisted record text are `FAILED_REQUEST_TEXT`, note = the detail, `failed: true` + the `error` key unchanged in shape (≤500 cap intact);
- a thrown `Error` WITHOUT `bubbleText` (the transport-failure shape) → `FAILED_TURN_TEXT` byte-identical (the existing pins re-run green);
- the SSE-frame throw (task 02's site) sets NO `bubbleText` → model-side failures keep `FAILED_TURN_TEXT`;
- the banner: `err.hint` reaches the opts the same way for both throw kinds (the phase-114 pin re-runs);
- `FAILED_REQUEST_TEXT`'s exact string pinned ONCE (the copy is owner-locked); the never-`innerHTML` rule holds.
## Testing & Quality
- Unit: the pins above; the phase-120 suite green WITHOUT modification (the default path byte-identical); coverage gate unchanged (no `app/` code).
## Completion Criteria
- [ ] Pre-stream HTTP error → bubble `FAILED_REQUEST_TEXT`, note carries the serialized detail (task 02's serializer), hint on the banner, Retry + Edit land, composer re-enables (§7.4).
- [ ] Transport failures and SSE error frames are byte-identical to the phase-120 behavior (`FAILED_TURN_TEXT`, same persistence).
- [ ] The persisted `bor.chat.v1` record for a new-line failure stores the new text + `failed: true` + the capped `error` — a refresh restores it identically (the same property the existing suite pins for `FAILED_TURN_TEXT`).
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 04 — The dedicated E2E: error honesty + long-answer acceptance; the final gates (phase-164 D4)
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D4 + the AGENTS.md per-phase E2E rule (A16).
**Story:** n/a
**Pre-fix evidence (2026-10-07, recorded per house practice):** owner-live instance after raising `BOR_LLM_MAX_OUTPUT_TOKENS=131072` — the post-long-answer turn's bubble read *"My answer didn't make it — the connection dropped. Use Retry to ask again. Failed [object Object]."*; code trace: the 422's list-of-objects `detail` hit the unguarded `body.detail` assignment (`app.js` ~3956) → `new Error(<array>)` → `"[object Object]"` (the example was not preserved live — the cap raise removed the repro; the 32 000 ceiling repros it at DEFAULT settings with any >32k-char answer). `HistoryTurn`'s own docstring records the identical failure class once already (4 000 ceiling, phase-42 E2E).
## Objective
One Playwright suite proves the phase end to end against the REAL client code paths — the distinct pre-stream line with the serialized validation message (never `[object Object]`, never `input` leakage), the fallback and tolerance paths, and the real-server acceptance of a full-length-answer history — and the final pass runs every gate plus the full documentation consistency review.
## Work
1. `tests/e2e/test_chat_error_honesty.py` (new; isolation; deterministic mock LLM; read `tests/e2e/test_frontend_failed_turn*.py`-adjacent suites and any existing `page.route` usage first and reuse the established stubbing approach — if no route-stub precedent exists, intercept `POST /api/chat` with Playwright `page.route` fulfilling the fetch; the assertions must drive the real `app.js` send path):
- **assert 1 (the live bug)**: stub `POST /api/chat` → 422 with the EXACT FastAPI shape for a long-history rejection (`{"detail":[{"type":"string_too_long","loc":["body","history",5,"text"],"msg":"String should have at most 600000 characters","input":"x".repeat(1000)}]}`) → send a question: the bubble text is `FAILED_REQUEST_TEXT` verbatim; the `.failed-note` detail contains the `msg` substring; NO `[object Object]` anywhere in the DOM; the rejected `input` text is NOT in the DOM; the banner shows the hint; Retry + Edit are present; the composer is re-enabled (§7.4).
- **assert 2 (fallback)**: stub → 500 with a NON-JSON body → the note/line settle on the `HTTP 500` default string, `FAILED_REQUEST_TEXT` bubble (still pre-stream), no crash.
- **assert 3 (acceptance — the D1 fix)**: seed `localStorage` `bor.chat.v1` with a conversation whose LAST brain record carries a ~525k-char answer (the seeded-restore pattern the saved-chat suites use), reload, send a short question → the request is NOT 422 (assert the response status 200 via the route/network log) and the turn settles normally with the mock answer (the locked budget trim may drop the giant turn WHOLE from the prompt — asserting settle, never prompt content, per A4).
- **assert 4 (SSE-frame tolerance)**: stub → a valid SSE stream whose `error` frame carries an ARRAY `detail` → the note serializes the `msg` (client tolerance; server-built string frames are pinned in task 02 units).
2. Final pass (the phase pipeline runs this after the tasks): full-suite + gates:
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**;
- `uv run pytest tests/e2e/test_chat_error_honesty.py -v --no-cov` green **in isolation**; the phase-120/114 failed-turn + banner suites and `test_chats_api.py` green (re-cut pins named in the commit body);
- `uv run ruff check . && uv run pyright` clean.
3. Documentation & consistency (mandatory): review `README.md` and `AGENTS.md` IN FULL against the post-phase app — add the **deploy note**: *the proxy in front of the app must accept chat POSTs sized to the new caps (recommend `client_max_body_size ≥ 16m` — a follow-up turn after a full-cap answer posts ~0.5 MB+, a saved chat up to the 8 MB budget); a proxy with a smaller limit turns a legal request into a 413* — review the error-handling descriptions for accuracy (the failed-turn copy is documented where phase 120's bubbles are), and fix EVERY inconsistency found, including pre-existing ones not caused by this phase. `PLAN.md` untouched (A5).
## Testing & Quality
- This task IS the E2E + gates task; its fixtures are test-side; `app/` coverage stays **>90%**.
## Completion Criteria
- [ ] All four asserts green in isolation; assert 1 proves the exact live message renders honestly end to end (bubble + note + hint + controls).
- [ ] The seeded 525k-answer conversation sends, is accepted, and settles — the live "every follow-up dies after a long answer" failure mode is impossible after this phase.
- [ ] `uv run pytest` fully green; coverage >90%; ruff + pyright clean; no existing suite modified except re-cut pins named in the commit body.
- [ ] `README.md` + `AGENTS.md` reviewed in full, the proxy body-size deploy note added, every inconsistency fixed; the phase dir moves to `complete/` and the single `--no-gpg-sign` commit (overview) is made by the pipeline gate.
@@ -0,0 +1,13 @@
All gates verified. Final report:
**Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)**
- Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs).
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%)
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation
- `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed
- `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins)
- Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅
- **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**.
- Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit.
- **Next pending phase:** `164_chat_error_honesty`
@@ -0,0 +1,124 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
.................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 277 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 136 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 473 0 100%
--------------------------------------------------
TOTAL 5869 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,21 @@
All green. Final state:
**Implemented (task 01 — phase 163 D1, LOCKED A2/A3)**
- `app/models.py`: `Document.page_strip_mode` (VARCHAR(16), nullable, no server default, unindexed) right after `page_count`, docstringed as the strip CLASS (`primary` = fallback class, `additive` = pictorial class, NULL = no strip).
- `alembic/versions/0027_documents_page_strip_mode.py`: additive add/drop migration mirroring 0026; `upgrade head → downgrade -1 → upgrade head` round-trips clean (DB now at `0027 (head)`).
- `app/schemas.py`: `DocContent.page_strip_mode` + the `image_url`-style omission in the existing `model_serializer` (ABSENT, never `null`).
- `app/api/docs.py`: content payload passes `page_strip_mode=doc.page_strip_mode if doc.page_count is not None else None` — the whole strip block keys off `page_count`, so a stray mode never travels alone; `/pages/{n}` untouched; endpoint docstring + README "Which kind of strip it is (phase 163)" section updated. No AGENTS.md change (its viewer bullets are the final pass's).
- Tests: `tests/integration/test_migration_0027.py` (new, house pattern: contract + pre-phase `page_count=36`/mode-NULL backfill + ORM round-trip both classes + tested downgrade + round-trip), `tests/unit/test_models.py` (column contract + ORM defaults), `tests/unit/test_document_viewer.py` (model omission rule; endpoint: stamped primary/additive, pre-phase mode-NULL row → key absent with `page_urls` kept, stray mode + `page_count` NULL → neither key, exact non-qualifying key-set), `tests/unit/test_pdf_page_copies.py` (importer leaves the class unstamped until task 02), `tests/integration/test_docs_api.py` (`_seed_page_doc` gains `page_strip_mode`; real-DB class pins + never-alone + exact key set).
**Gates**
- `uv run pytest` → **3772 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **3772 passed, app TOTAL 99%** (`app/schemas.py` 100%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- `uv run alembic upgrade head && downgrade -1 && upgrade head` → clean, `current` = `0027 (head)`.
- E2E regression (isolation, `--no-cov`): `test_pdf_pages_in_sources.py` 4 passed, `test_pictorial_pdf.py`+`test_garbage_pdf.py` 6 passed, `test_smoke.py` 3 passed.
**Notes / decisions**
- Non-qualifying payloads stay byte-identical (both strip keys dropped when `page_count` is NULL); pre-phase rows serialize the class ABSENT and the viewer will read that as `primary` (task 04) — pinned, no data backfill.
- Column carries no CHECK constraint (purely additive per A13); the class values are enforced by the writers (tasks 02/03).
- No live-infrastructure changes needed for this task.
**Next pending task:** `163_pdf_strip_additive_selfheal/02_persist_write_first.md`.
@@ -0,0 +1,124 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
............................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 277 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 136 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 764 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 473 0 100%
--------------------------------------------------
TOTAL 5792 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,13 @@
**Task 02 complete — `_persist_pdf_pages` write-first atomic persist + `strip_class` stamp (D2, A4/A7).**
- `app/rag/importer.py`: `_persist_pdf_pages` rewritten — render → per-file atomic write (`<name>.png.tmp` + `os.replace`, never a direct final-path write) → stale-tail delete LAST (only old-extent files not rewritten; a file about to be overwritten is never deleted); failure path unlinks this call's `.tmp`s and files it **created** (an overwritten copy keeps standing — deleting it would re-open the live-loss hole); `strip_class="fallback"|"pictorial"|None` replaces `eligible`, stamping `page_strip_mode` `primary`/`additive` exactly with `page_count`; the None arm and no-pages guard delete-then-null (helper reads the count); module + function docstrings carry the write-first rationale and the verbatim phase-148 truth table.
- Call site (`_index_pdf_file`): `strip_class = "fallback" if fallback_fired else ("pictorial" if pictorial_described else None)`; fail-soft handler nulls `page_strip_mode` too; comments updated.
- `tests/unit/test_pdf_page_copies.py`: re-cut task-01 pin → additive stamp; fallback→`primary`, ineligible/qualifier-loss/no-pages→NULL pins; new pins — order (event log: all `os.replace`s strictly before the first stale `unlink`, new bytes win), mid-write failure (prior 10-file set intact, zero `.tmp` orphans, count/mode NULL, doc indexed), atomic tmp+replace writer (every write to `.png.tmp`, `dst`/`src` pairing), best-effort stale-tail OSError branch; writer-exception test re-cut for tmp semantics.
Gates: `uv run pytest --cov=app --cov-report=term-missing` → **3776 passed, app coverage 99%** (importer 99%; remaining misses pre-existing); `uv run ruff check . && uv run pyright` → clean; E2E `test_pdf_pages_in_sources` (4), `test_pictorial_pdf` (3), `test_garbage_pdf` (3) all pass in isolation.
Docs: no further doc impact — README's phase-163 strip-class section (task 01) already covers the user-visible contract; deletion ordering is internal.
Notable decision: the partial-write cleanup keeps an **overwritten** final (keeps only newly-created files) — strictly satisfies "a failed call never leaves more files than before, and never removes a live copy" (LOCKED A4).
Next pending task: `163_pdf_strip_additive_selfheal/03_fast_path_regate.md`.
@@ -0,0 +1,124 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 80%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 277 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 136 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 788 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 473 0 100%
--------------------------------------------------
TOTAL 5816 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,10 @@
**Task 03 complete — all green.**
- **`app/rag/importer.py`**: new `_regate_pdf_strip(session, doc, *, full_path, llm, summary)` — mirrors the changed-path gate verbatim from current bytes + settings (phase-133 `is_usable_pdf_text` → phase-140 `pictorial_pages` + `pdf_pictorial` kill switch), **zero LLM/vision calls**; heals any missing `<id>-p{N}.png` (full re-render), stamps `page_strip_mode` (silent NULL→stamp, `mode old->new` on a set mode), clears a strip that no longer qualifies + deletes its copies, one greppable INFO only when it **acts**, whole body fail-soft (`re-gate failed` / `re-render produced no pages (state kept)` / `page_count drift a->b` warnings).
- `_unchanged_doc` gained `full_path: Path | None = None`; the **guarded** re-gate (`page_count` set + `.pdf`) runs at **both** exits (incl. the D1 manual-date return) and commits what it wrote; all 4 call sites (fast path, text/PDF/image content-digest hits) pass their `full_path`. Docstrings: the phase-147 "the copies persist" assumption is replaced (module doc + `_unchanged_doc` step 7 + `_index_pdf_file`).
- De-duplicated task 02's writer: extracted `_write_page_copies` (tmp + `os.replace`, orphan cleanup) and `_delete_stale_page_tail`; `_persist_pdf_pages` and the heal now share them (behavior unchanged).
- **Tests**: 11 new pins in `tests/unit/test_importer_unchanged_fast_path.py` — heal (pictorial, live NULL-mode), fallback heal (`primary`), healthy no-op (zero writes, zero re-gate log), mode re-stamp, repair arm, kill-switch clear, render-raise fail-soft, empty-render keeps row, count drift, content-digest-hit path, non-strip zero-parse guard (spy).
- `uv run pytest tests/unit/test_importer_unchanged_fast_path.py tests/unit/test_pdf_page_copies.py --no-cov` → exit 0 (24 + 15 tests); `uv run pytest --cov=app` → **3787 passed**, `app/rag/importer.py` **99%** (only pre-existing misses: 883, 3506-3507, 3793), TOTAL 99%; `uv run ruff check .` → clean; `uv run pyright` → **0 errors**; PDF E2E files pass in isolation (`test_pdf_pages_in_sources`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_sync_changed_only`, exit 0).
- **Docs**: README — new "The strip repairs itself (phase 163)" viewer bullet, the "Updating the documents" fast-path qualifier, and the `BOR_PDF_PAGE_DIR` env-row note (dedup avoided). **AGENTS.md: no change** (per the phase overview its viewer bullet lands in the phase's final pass).
- **Notable**: (1) running all of `tests/e2e` in one process fails on a **pre-existing** design artifact — `tests/e2e/test_extensionless_import.py:70` mutates `os.environ["BOR_IMPORT_EXTENSIONS"]` at import, so other suites' 13-doc seed drops to 5; unaffected by this task, and the project's gate is per-file isolation (flagging for the owner/final pass). (2) Restored `.agents/screenshots/doc_modal_scrollbar_1280.png` (a tracked artifact my E2E runs overwrote) from HEAD so the phase commit stays clean.
- **Next pending task**: `163_pdf_strip_additive_selfheal/04_viewer_additive_toggle.md`.
@@ -0,0 +1,124 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 38%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 96%]
........................................................................ [ 98%]
........................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 277 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 136 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 473 0 100%
--------------------------------------------------
TOTAL 5869 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,18 @@
**Task 04 complete — report**
**Implemented**
- `frontend/assets/document.js`: `renderDocument` now branches on `doc.page_strip_mode` — absent/`"primary"` keeps the phase-148 strip-only branch verbatim (byte-identical); `"additive"` renders the text doc (Summary panel + content) inside a `.doc-views` wrapper with a `Text`/`Pages` toggle (`role=group`, `aria-pressed`, Text default per LOCKED A6); strip mounts once, hidden via CSS `data-view` swap (lazy imgs never fetch while hidden, no refetch on flip-flop); never-build-HTML rule kept (module still has exactly one `innerHTML`); module docstring gained the phase-163 paragraph and re-scoped the phase-148 "INSTEAD of" sentence to the primary class.
- `frontend/assets/styles.css`: `.doc-view-toggle` ghost-pill pair (house `.auth-link`/`.share-chat-btn` idiom, chat-column cap, ≥44px targets, global 3px focus ring, pressed = brand fill + weight 700 — text + style, never color alone) + `.doc-views[data-view]` show/hide rules.
- New `tests/unit/test_viewer_additive_toggle.py`: 9 pins (source-contract + node behavioral runs of the real core: additive shape, primary/absent byte-identity, text-doc untouched, flip semantics with strip-node persistence, lazy imgs, XSS, CSS contract).
- Re-cut stale/newly-affected pins: `test_summary_edit_ui.py` (slot mount), `test_wide_column_css.py` (7th reading-column selector), E2E additive-view contracts in `test_pdf_pages_in_sources.py` and `test_pictorial_pdf.py` (text-first default, Pages flip reveals loaded strip, description rides the shown text).
**Gates**
- `uv run pytest` → **3796 passed**, 0 failed; coverage `app/` **99%** (>90%).
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
- E2E in isolation: `test_pdf_pages_in_sources.py` 4✓, `test_pictorial_pdf.py` 3✓, `test_garbage_pdf.py` 3✓, plus `test_pdf_support.py` 6✓, `test_document_viewer.py` 8✓, `test_summary_in_viewer.py` 4✓.
**Docs**: README — phase-148 viewer bullet scope-fixed to "primary class only" (the phase-163 bullets from tasks 01–03 already describe the toggle); AGENTS.md — no doc impact (viewer bullet note is reserved for the phase final pass).
**Decisions**: additive E2E witnesses flipped from "descriptions not displayed" to "descriptions ride the shown text" (intended consequence of LOCKED A1/A6); toggle lives in a `.doc-views` wrapper only on additive docs — all other DOM untouched.
**Next pending task**: `163_pdf_strip_additive_selfheal/05_e2e_selfheal_pins.md`.
@@ -0,0 +1,124 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
.................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 277 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 136 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 473 0 100%
--------------------------------------------------
TOTAL 5869 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,16 @@
**Task 05 complete — report**
**Implemented**
- New `tests/e2e/test_pdf_strip_additive_selfheal.py` (5 tests, phase-163 A16 suite): fixture `strip_fixture` = `manual.pdf` (3 dense pages + 1 sparse map page → additive), `scanned.pdf` (XML-garbage layer → primary), `plain.pdf` (no strip); asserts 1 additive view (payload stamp, Text default, Pages flip loads all 4 imgs, flip-back keeps the mounted strip node — dataset-probed, zero refetch); 2 primary/plain DOM byte-identity incl. the key-omission payload pins; 3 heal (wipe copies → route 404 → one unchanged-byte sync → files restored, route 200, viewer complete); 4 repair (direct-DB stale `page_count`/mode + fake copies → cleared + deleted); 5 zero-LLM (mock `/v1/e2e/captured`: index burned both vision passes, heal/repair captured zero content calls — the sync prelude's fixed `ping` health probe is the one documented exemption).
- Docs pass: README verified in full (viewer + importer + env-table claims cross-checked against code; defaults script-audited vs `Settings` — no drift); AGENTS.md rule 5 gained the phase-163 viewer-class note.
**Gates** (all run this session, DB up)
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation.
- `test_pdf_pages_in_sources / test_pictorial_pdf / test_garbage_pdf / test_document_viewer / test_document_viewer_context / test_image_documents` → all exit 0, unmodified.
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 passed**, coverage **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors; alembic **0027 downgrade→upgrade round-trip clean**.
**Docs**: README — no further edits needed (phase bullets from tasks 01–03 verified accurate); AGENTS.md — viewer-class note added.
**Decisions**: heal/repair zero-call pins exempt only `check_models`' `ping` (every sync pays it, predates the phase); anything else fails the pin. Pre-existing `.agents/screenshots/doc_modal_scrollbar_1280.png` modification (earlier E2E run) left for the pipeline commit.
**Next pending task**: `164_chat_error_honesty/01_caps_long_answer_chain.md`.
@@ -0,0 +1,124 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 7%]
........................................................................ [ 9%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
.................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 277 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 196 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 266 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 136 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 473 0 100%
--------------------------------------------------
TOTAL 5869 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
Binary file not shown.

Before

Width:  |  Height:  |  Size: 42 KiB

After

Width:  |  Height:  |  Size: 43 KiB

+5 -1
View File
@@ -21,7 +21,11 @@
panels — `max-height: min(70vh, 64rem)` — with a sticky header row; panels — `max-height: min(70vh, 64rem)` — with a sticky header row;
a clipped action label shows an ellipsis hint (`.cell-clip`) with a clipped action label shows an ellipsis hint (`.cell-clip`) with
the full text kept reachable via the control's aria-label the full text kept reachable via the control's aria-label
(phase 156)) and (phase 156); the document viewer's page strip is CLASS-scoped —
the primary class shows the strip in place of the text, the
additive class never hides the text: Text/Pages toggle group
(`aria-pressed`, Text default) with the strip behind it
(phase 163)) and
meets WCAG 2.1 AA basics (semantic meets WCAG 2.1 AA basics (semantic
landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the
stream). stream).
+38 -5
View File
@@ -239,9 +239,11 @@ idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
streamed) and checked against the last index's stamp *before* any streamed) and checked against the last index's stamp *before* any
extraction or vision work — an unchanged file costs one hash + one row extraction or vision work — an unchanged file costs one hash + one row
lookup (no re-extract, no vision, no re-embed), while a changed file lookup (no re-extract, no vision, no re-embed), while a changed file
re-indexes fully. A refresh after a normal editing session takes re-indexes fully. The one exception is a PDF row that still claims a
seconds — pictorial PDFs included (their expensive first-index vision page strip: it is re-gated from its current bytes (no vision, no LLM —
burns only once per file version): see "The strip repairs itself" below). A refresh after a normal editing
session takes seconds — pictorial PDFs included (their expensive
first-index vision burns only once per file version):
```bash ```bash
# After editing/adding/removing notes: # After editing/adding/removing notes:
@@ -844,11 +846,42 @@ uploaded archives):
shows the PAGES: a vertical strip of lazy page images with shows the PAGES: a vertical strip of lazy page images with
"Page N" captions, in place of the text content and the Summary "Page N" captions, in place of the text content and the Summary
panel (the per-page descriptions stay in the DB for Q&A — the panel (the per-page descriptions stay in the DB for Q&A — the
embeddings and summary are untouched, just not displayed). A lost embeddings and summary are untouched, just not displayed). Since
phase 163 this strip-only shape is the PRIMARY class only — the
pictorial class keeps its text behind a Text/Pages toggle, per the
next bullet. A lost
copy shows a "Page N unavailable" note for that page only. Text- copy shows a "Page N unavailable" note for that page only. Text-
heavy PDFs (usable layer, zero pictorial candidates — and the whole heavy PDFs (usable layer, zero pictorial candidates — and the whole
sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view, sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view,
byte-identical to pre-phase. byte-identical to pre-phase.
- **Which kind of strip it is (phase 163).** The two qualifying classes
are not the same problem: the screenshot class has no usable text to
show, while a book with one sparse page (an illustration map page in a
36-page text) does. The document row therefore records the strip's
CLASS beside the page count (`documents.page_strip_mode` —
`primary` = the screenshots ARE the content, so the strip is the view;
`additive` = a usable text layer, so the text + Summary view stays and
the pages sit behind a Text/Pages toggle), and the document-content
payload carries it as `page_strip_mode`. The key is ABSENT — never
`null` — when there is no strip, which is also how a row written before
this phase serializes; the viewer treats an absent class as `primary`,
so every existing document renders exactly as before until the next
sync stamps it (the sync's self-heal pass, phase 163).
- **The strip repairs itself (phase 163).** An unchanged PDF is not
assumed to still have its page images on disk. On every sync, a row
that claims a strip is re-gated deterministically from the current
file bytes and the current settings — the same gate the changed path
uses (`BOR_PDF_PICTORIAL` and `BOR_PDF_PICTORIAL_MIN_CHARS`
included), with **zero vision or LLM calls** (rendering is CPU-only):
the class is re-stamped, any `<doc-id>-p{N}.png` missing from
`BOR_PDF_PAGE_DIR` is re-rendered (write-before-delete, atomically per
file), and a strip that no longer qualifies — the kill switch turned
off, the sparse-page bar lowered past it — is cleared along with its
copies. So a wiped pages directory, or a copy set lost to an
interrupted sync, repairs itself on the next sync instead of leaving
"Page N unavailable" forever. It logs one `PDF strip re-gate …` line
only when it actually acts, and a failure there never fails the sync.
PDFs without a strip pay nothing extra (no extra read at all).
--- ---
@@ -975,7 +1008,7 @@ to `.env` and adjust.
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) | | `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly | | `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
| `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) | | `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) |
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view | | `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view. Clearing the directory is self-healing — the next sync re-gates any strip that still claims pages and re-renders what is missing, with zero vision calls (phase 163) |
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** | | `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled | | `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked | | `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
@@ -0,0 +1,68 @@
"""documents.page_strip_mode: the strip CLASS behind page_count (phase 163, task 01)
Revision ID: 0027
Revises: 0026
Create Date: 2026-10-08
Phase 163 (the PDF page strip is ADDITIVE for a text-bearing book —
task 01, storage only):
* ``documents.page_strip_mode`` — VARCHAR(16) NULLABLE, NO server
default, NO index: the CLASS of the page strip a non-NULL
``page_count`` describes, so the viewer knows whether the strip
REPLACES the text or sits one toggle behind it.
``"primary"`` — the phase-133 FALLBACK class (an unusable text layer:
the screenshots ARE the content), i.e. exactly the pre-phase view: the
strip is the document's view and no text is shown. ``"additive"`` —
the phase-140 PICTORIAL class (a usable text layer with ≥1 sparse page
described): the document keeps its text view and its Summary panel,
and the strip is one Text/Pages toggle away.
NULL means NO strip (every non-qualifying document — text documents,
images, text-heavy PDFs — and every qualifying PDF whose render/write
failed). A row written before this phase keeps NULL while its
``page_count`` stands; the wire omits the key in that case and the
viewer treats an absent mode as ``"primary"`` (today's DOM), so
existing rows render exactly as before until the fast-path re-gate
(task 03) stamps them. Stamped and cleared EXACTLY with
``page_count`` — the importer's persist (task 02) and the unchanged-
path re-gate (task 03) are the only writers.
A display fact, never a query surface, like the 0026 column it
accompanies: the content endpoint reads it from the ROW after the
``(source, path)`` lookup, so it carries no index of its own (the
``file_hash`` convention, 0025).
One additive, fully reversible migration (A13); no other schema change.
The ordering of the copies on disk (task 02) and the re-gate itself
(task 03) are importer code — this revision only carries the column.
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "0027"
down_revision = "0026"
branch_labels = None
depends_on = None
def upgrade() -> None:
# No index (a display fact, never a query surface — the content
# endpoint reads it from the row) and no server default (NULL is the
# meaningful state for every non-qualifying document, every fail-soft
# miss, and every pre-phase row).
op.add_column(
"documents", sa.Column("page_strip_mode", sa.String(length=16), nullable=True)
)
def downgrade() -> None:
# The page_strip_mode column is the only 0027 artefact — dropping it
# leaves 0026's schema byte-identical (A13, fully reversible). The
# class is re-derivable from the source file on the next sync (the
# deterministic re-gate of task 03), so no data is lost by the drop.
op.drop_column("documents", "page_strip_mode")
+20
View File
@@ -206,6 +206,16 @@ def get_document_content(
basename matches). ``None`` — a non-git source, no row, or an basename matches). ``None`` — a non-git source, no row, or an
ambiguous basename — keeps the payload byte-identical to pre-phase ambiguous basename — keeps the payload byte-identical to pre-phase
(both fields are dropped by the serializer, never ``null``). (both fields are dropped by the serializer, never ``null``).
Phase 163 (task 01): a page-qualifying PDF's payload additionally
carries ``page_strip_mode`` — ``"primary"`` (the phase-133 fallback
class: the strip IS the view) or ``"additive"`` (the phase-140
pictorial class: the text + Summary view with the strip behind a
Text/Pages toggle). Both strip keys are built off ``page_count`` and
dropped when it is NULL, so every non-qualifying payload stays
byte-identical to pre-phase (and a pre-phase row, whose mode column
is NULL, keeps the strip-only DOM the viewer assumes for an absent
mode).
""" """
row = db.execute( row = db.execute(
select(Document, func.count(Chunk.id).label("chunks")) select(Document, func.count(Chunk.id).label("chunks"))
@@ -254,6 +264,16 @@ def get_document_content(
if doc.page_count is not None if doc.page_count is not None
else None else None
), ),
# Phase 163 (task 01): the strip CLASS (``"primary"`` = the
# fallback class, the strip IS the view; ``"additive"`` = the
# pictorial class, text + a Text/Pages toggle). The whole strip
# block keys off ``page_count`` — a row with a stray mode but no
# strip serializes NEITHER key — and a pre-phase row (mode NULL)
# serializes the key ABSENT (never null), which the viewer reads
# as ``"primary"`` (task 04).
page_strip_mode=(
doc.page_strip_mode if doc.page_count is not None else None
),
) )
+18
View File
@@ -198,6 +198,24 @@ class Document(Base):
#: cleanup deletes the copies with the row. No index: a display #: cleanup deletes the copies with the row. No index: a display
#: fact, never a query surface (the ``file_hash`` convention, 0025). #: fact, never a query surface (the ``file_hash`` convention, 0025).
page_count: Mapped[int | None] = mapped_column(Integer, default=None) page_count: Mapped[int | None] = mapped_column(Integer, default=None)
#: The STRIP CLASS behind a non-NULL :py:attr:`page_count` (phase 163,
#: LOCKED A1/A3) — the signal the viewer branches on so a text-bearing
#: book never loses its text:
#:
#: * ``"primary"`` — the phase-133 FALLBACK class (unusable text layer:
#: the screenshots ARE the content): the strip IS the view, exactly
#: as phase 148 shipped it;
#: * ``"additive"`` — the phase-140 PICTORIAL class (usable text layer,
#: ≥1 sparse page described): the text view + the Summary panel, with
#: the strip one Text/Pages toggle away (Text is the default).
#:
#: NULL = no strip (every non-qualifying document) — and a pre-phase-163
#: row keeps NULL while its ``page_count`` stands, which the wire treats
#: as ``"primary"`` (today's DOM) until the fast-path re-gate stamps it.
#: Stamped/cleared EXACTLY with ``page_count``: written by the importer's
#: persist (task 02) and the unchanged-path re-gate (task 03) only. Like
#: ``page_count`` it is a display fact, never a query surface — no index.
page_strip_mode: Mapped[str | None] = mapped_column(String(16), default=None)
chunks: Mapped[list[Chunk]] = relationship( chunks: Mapped[list[Chunk]] = relationship(
back_populates="document", cascade="all, delete-orphan" back_populates="document", cascade="all, delete-orphan"
+409 -71
View File
@@ -190,15 +190,37 @@ dpi, every page) and persisted to ``settings.pdf_page_dir``
``<doc-id>-p{N}.png`` (N 1-based), with ``documents.page_count`` ``<doc-id>-p{N}.png`` (N 1-based), with ``documents.page_count``
(migration 0026) recording the page count — a per-page render (migration 0026) recording the page count — a per-page render
failure leaves a GAP (no file — the route 404s that N, the viewer's failure leaves a GAP (no file — the route 404s that N, the viewer's
note covers it). The existing copies are deleted before replacement note covers it). WRITE-FIRST (phase 163, task 02, LOCKED A4): the
(a shrunk PDF's old ``p{N}`` tail cannot leak), a document that fresh render's bytes land ATOMICALLY per file (``<name>.png.tmp`` +
STOPS qualifying is cleared (``page_count`` NULL + copies deleted), ``os.replace``) BEFORE any old copy is removed — only the old-extent
and the prune deletes the copies with the row files the fresh render did NOT write (a shrunk PDF's ``p{N}`` tail)
go, so a rollback or crash can never restore a stale ``page_count``
over ZERO files. The row is stamped with the strip's CLASS beside the
count (``documents.page_strip_mode``, migration 0027 — ``primary`` for
the fallback class, ``additive`` for the pictorial class, phase 163);
a document that STOPS qualifying is cleared (both columns NULL +
copies deleted — deletion-first is correct where there is no
replacement set), and the prune deletes the copies with the row
(:func:`_delete_page_copies`). FAIL-SOFT: a render or write failure (:func:`_delete_page_copies`). FAIL-SOFT: a render or write failure
leaves ``page_count`` NULL and logs a warning — the document indexes leaves ``page_count``/``page_strip_mode`` NULL and logs a warning —
EXACTLY as pre-phase (the render is a display concern, it never the document indexes EXACTLY as pre-phase (the render is a display
gates the indexing). The UNCHANGED fast path renders nothing (the concern, it never gates the indexing).
copies persist — zero render work, zero orphans).
PDF strip re-gate (phase 163, task 03, LOCKED A5): the UNCHANGED paths
never assume "the copies persist" any more — a row that still claims a
strip (``page_count`` set, a ``.pdf`` path) is re-gated
deterministically from the CURRENT bytes + CURRENT settings
(:func:`_regate_pdf_strip`, run at the tail of
:func:`_unchanged_doc`): the class is re-stamped (``primary`` for the
unusable-layer fallback class, ``additive`` for the sparse-page
class), any missing ``<doc-id>-p{N}.png`` is RE-RENDERED through the
same write-first atomic writer, and a strip that no longer qualifies
(lowered ``pdf_pictorial_min_chars``, the kill switch off) is cleared
with its copies. ZERO vision/LLM calls (the render is pure CPU, PLAN
A21-safe), one greppable INFO line only when the re-gate ACTS, and
fail-soft on every exception — the loss state (an external wipe, a
pre-phase delete-first rollback) cannot survive one sync. Every other
file's fast path still performs ZERO PDF parsing (the guard).
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
description, the same source's importable markdown files are scanned for description, the same source's importable markdown files are scanned for
@@ -298,6 +320,7 @@ from __future__ import annotations
import hashlib import hashlib
import logging import logging
import os
import posixpath import posixpath
import re import re
import uuid import uuid
@@ -1021,6 +1044,7 @@ async def _unchanged_doc(
summary: ImportSummary, summary: ImportSummary,
raw_date: datetime, raw_date: datetime,
file_digest: str, file_digest: str,
full_path: Path | None = None,
) -> None: ) -> None:
"""Phase 147 (task 02): the ONE shared unchanged path — the three """Phase 147 (task 02): the ONE shared unchanged path — the three
branches' near-identical unchanged blocks, one code site, reached branches' near-identical unchanged blocks, one code site, reached
@@ -1064,7 +1088,22 @@ async def _unchanged_doc(
the date block (the stamp set ``file_hash`` and the date did the date block (the stamp set ``file_hash`` and the date did
not change): quiet bookkeeping — no count, no log (the not change): quiet bookkeeping — no count, no log (the
``unchanged`` line already fired; a date-change run already ``unchanged`` line already fired; a date-change run already
committed in step 5). committed in step 5);
7. the phase-163 (task 03) PDF strip RE-GATE — a row that still
claims a strip (``page_count`` set) whose path ends in ``.pdf``
goes through :func:`_regate_pdf_strip`: the strip is re-derived
from the CURRENT bytes + CURRENT settings (deterministic, ZERO
LLM/vision calls) — a missing/changed ``page_strip_mode`` is
re-stamped, copies that vanished from disk are RE-RENDERED, a
strip that no longer qualifies is cleared — and whatever it
wrote commits in this helper's transaction. Unchanged files
re-gate the PDF strip deterministically (phase 163): the
pre-phase "the copies persist" assumption is gone, so a wipe of
``pdf_page_dir`` or a pre-phase delete-first rollback cannot
outlive one sync. Every other file (``page_count`` NULL) pays
ZERO extra read — the guard keeps the fast path's contract, and
BOTH exits (the D1 manual-date return and the normal tail) run
the re-gate.
""" """
summary.unchanged += 1 summary.unchanged += 1
logger.info("import: unchanged source=%s path=%s", source, rel) logger.info("import: unchanged source=%s path=%s", source, rel)
@@ -1078,10 +1117,32 @@ async def _unchanged_doc(
session, doc=doc, source=source, rel=rel, content=doc.content, session, doc=doc, source=source, rel=rel, content=doc.content,
llm=llm, summary=summary, backfill=True, llm=llm, summary=summary, backfill=True,
) )
def _regate_strip() -> None:
"""Step 7 (phase 163, task 03): the GUARDED strip re-gate — only
a row that carries a strip (``page_count`` set) on a ``.pdf``
path reaches :func:`_regate_pdf_strip`, so every other file's
fast path stays byte-identical (zero extra read, zero PDF
parse). The re-gate never raises (fail-soft inside); its writes
ride this helper's transaction."""
if doc.page_count is None or not str(doc.path).lower().endswith(".pdf"):
return
_regate_pdf_strip(
session,
doc,
full_path=full_path if full_path is not None else Path(doc.full_path),
llm=llm,
summary=summary,
)
if session.is_modified(doc):
session.commit()
# 4 — the phase-106 (D1) manual-date lock: no date write; the # 4 — the phase-106 (D1) manual-date lock: no date write; the
# stamp, if dirty, is the only write (committed — see step 2's # stamp, if dirty, is the only write (committed — see step 2's
# "next sync takes the fast path"). # "next sync takes the fast path") — plus whatever the phase-163
# re-gate repaired.
if doc.created_at_manual: if doc.created_at_manual:
_regate_strip()
if session.is_modified(doc): if session.is_modified(doc):
session.commit() session.commit()
return return
@@ -1103,6 +1164,165 @@ async def _unchanged_doc(
# 6 — the stamp's commit (the date did not move): quiet # 6 — the stamp's commit (the date did not move): quiet
# bookkeeping — the ``unchanged`` line already fired. # bookkeeping — the ``unchanged`` line already fired.
session.commit() session.commit()
# 7 — the phase-163 (task 03) strip re-gate (see the local helper).
_regate_strip()
def _regate_pdf_strip(
session: Session,
doc: Document,
*,
full_path: Path,
llm: Embedder,
summary: ImportSummary,
) -> None:
"""Phase 163 (task 03, LOCKED A5): the DETERMINISTIC re-gate of a
PDF's display strip — run from the unchanged paths
(:func:`_unchanged_doc`) whenever a row still claims a strip
(``page_count`` set), so a strip can never die silently.
WHY (the 2026-10-07 live state, owner report): the ``Letumdei`` row
carried ``page_count = 36`` over ZERO files — the pre-phase-163
:func:`_persist_pdf_pages` deleted the old copies BEFORE rendering,
OUTSIDE the DB transaction, so a later per-file rollback (the
import loop's ``EmbeddingError`` handler) restored the stale count
over an empty directory; an external wipe of ``pdf_page_dir`` lands
exactly the same state. The phase-147 fast path believed "the
copies persist" and never re-rendered, so every
``GET /api/documents/<id>/pages/N`` 404'd and the viewer showed
"Page N unavailable." on all 36 pages — on every sync, forever.
WHAT IT DOES — the changed-path gate mirrored exactly, from the
CURRENT file bytes + CURRENT settings, with ZERO vision/LLM calls
(:func:`app.rag.pdf.render_pdf_pages` is pure CPU — PLAN A21-safe,
no model, no budget):
* ``not is_usable_pdf_text(extract_pdf_text(full_path))`` → the
phase-133 fallback class → mode ``"primary"`` (the strip IS the
view — the same doc as today);
* else ``settings.pdf_pictorial`` ON and
:func:`app.rag.pdf.pictorial_pages` NON-EMPTY over
:func:`app.rag.pdf.extract_pdf_page_texts` → the phase-140
pictorial class → mode ``"additive"`` (the text view + the strip
behind a toggle). The candidates are the existing gate's
VERBATIM (text-carrying pages under ``min_chars``; a page with no
text has no key in the map and is not a candidate) — the re-gate
adds NO new eligibility semantics;
* else the strip no longer qualifies (the owner lowered
``pdf_pictorial_min_chars``, the kill switch went off, the layer
gained substance) → the REPAIR arm: the copies go
(``_delete_page_copies`` reads ``doc.page_count``, so it runs
while the count still describes the extent) and BOTH columns are
cleared.
For a re-gated-ELIGIBLE doc the copies are VERIFIED: any missing
``<doc-id>-p{N}.png`` in 1..``page_count`` re-renders EVERY page
through the changed path's own write-first atomic writer
(:func:`_write_page_copies` — tmp + ``os.replace``), then the
stale tail beyond the new count goes and ``page_count`` follows the
fresh render's length (a drift names both counts in a WARNING — on
unchanged bytes a drift is pathological). A fresh render producing
NO pages (unopenable right after the fast path hashed the file)
leaves the row exactly as it was + a WARNING: the re-gate never
clears a strip on a transient.
The mode re-stamp is SILENT when it only fills a NULL (a pre-phase
row learning its class is beneficial bookkeeping — it rides the
same INFO line only when real work followed); a change of an
ALREADY-SET mode (the kill switch, a class flip) says so
(``mode <old>-><new>``). The ONE greppable INFO line fires only
when the re-gate ACTS — never per unchanged doc.
FAIL-SOFT: the whole body is wrapped — any exception is a WARNING
and the sync continues (the strip is a display concern). A repair's
DB writes ride the caller's transaction, so an exception raised
after them rolls back with the file like any other per-file
failure. *summary* keeps the unchanged path's signature shape (and
a future counter site); the re-gate books nothing there — its stats
are log-only (the phase-133/140 convention).
"""
source, rel = doc.source, doc.path
try:
page_dir = Path(llm.settings.pdf_page_dir).expanduser()
# The gate, mirrored from the changed path (the phase-133
# precedence over the phase-140 pass, the kill switch included).
if not is_usable_pdf_text(extract_pdf_text(full_path)):
strip_class: str | None = "fallback"
elif llm.settings.pdf_pictorial and pictorial_pages(
extract_pdf_page_texts(full_path),
min_chars=llm.settings.pdf_pictorial_min_chars,
):
strip_class = "pictorial"
else:
strip_class = None
if strip_class is None:
# The REPAIR arm — the strip no longer qualifies: delete
# while the count still names the extent, then clear both
# columns (the same clearing the changed path does for a
# doc that stopped qualifying).
_delete_page_copies(doc, page_dir)
doc.page_count = None
doc.page_strip_mode = None
logger.info(
"import: PDF strip re-gate source=%s path=%s — cleared stale strip",
source, rel,
)
return
mode = "primary" if strip_class == "fallback" else "additive"
old_mode = doc.page_strip_mode
old_count = doc.page_count or 0
actions: list[str] = []
missing = [
n
for n in range(1, old_count + 1)
if not (page_dir / f"{doc.id}-p{n}.png").is_file()
]
if missing:
pages = render_pdf_pages(full_path)
if not pages or not any(pages):
# Unopenable (or every page a placeholder) — right
# after the fast path hashed the file this is
# pathological: KEEP the row as-is (never clear a strip
# on a transient) and let the next sync try again.
logger.warning(
"import: PDF strip re-gate source=%s path=%s — re-render "
"produced no pages (state kept)",
source, rel,
)
return
page_dir.mkdir(parents=True, exist_ok=True)
written_ns = _write_page_copies(doc, page_dir, pages)
_delete_stale_page_tail(
doc_id=doc.id,
page_dir=page_dir,
old_extent=old_count,
written_ns=written_ns,
)
doc.page_count = len(pages)
actions.append(f"healed {len(missing)} pages")
if doc.page_count != old_count:
logger.warning(
"import: PDF strip re-gate source=%s path=%s — page_count "
"drift %s->%s on unchanged bytes",
source, rel, old_count, doc.page_count,
)
if doc.page_strip_mode != mode:
# NULL -> stamped stays SILENT (it rides the line only when
# work followed); a change of an already-set mode is news.
doc.page_strip_mode = mode
if old_mode is not None:
actions.append(f"mode {old_mode}->{mode}")
if actions:
logger.info(
"import: PDF strip re-gate source=%s path=%s — %s",
source, rel, " | ".join(actions),
)
except Exception as e:
# Fail-soft: the strip never breaks a sync.
logger.warning(
"import: PDF strip re-gate failed source=%s path=%s — %s",
source, rel, e,
)
async def _index_file( async def _index_file(
@@ -1186,7 +1406,7 @@ async def _index_file(
# helper, nothing else does. # helper, nothing else does.
await _unchanged_doc( await _unchanged_doc(
session, doc, source=source, rel=rel, llm=llm, summary=summary, session, doc, source=source, rel=rel, llm=llm, summary=summary,
raw_date=raw_date, file_digest=file_digest, raw_date=raw_date, file_digest=file_digest, full_path=full_path,
) )
return return
# Phase 122 (task 02): the image branch FIRST. Only reachable while # Phase 122 (task 02): the image branch FIRST. Only reachable while
@@ -1223,7 +1443,7 @@ async def _index_file(
# file hash, so the NEXT sync takes the fast path above. # file hash, so the NEXT sync takes the fast path above.
await _unchanged_doc( await _unchanged_doc(
session, doc, source=source, rel=rel, llm=llm, summary=summary, session, doc, source=source, rel=rel, llm=llm, summary=summary,
raw_date=raw_date, file_digest=file_digest, raw_date=raw_date, file_digest=file_digest, full_path=full_path,
) )
return return
@@ -1506,19 +1726,24 @@ async def _index_pdf_file(
(:func:`_unchanged_doc`) — no re-extract, no re-embed, no image (:func:`_unchanged_doc`) — no re-extract, no re-embed, no image
pass, the backfill/manual-date/date-refresh semantics in one pass, the backfill/manual-date/date-refresh semantics in one
code site — and the helper stamps the new file hash (the next code site — and the helper stamps the new file hash (the next
sync takes the fast path). This path renders NO pages and sync takes the fast path). This path renders NO pages through
touches NO ``page_count`` (the copies persist — zero render THIS branch, but a row that carries a strip is re-gated by the
work). helper's phase-163 tail (:func:`_regate_pdf_strip` — the class
re-stamped, missing copies healed, a stale strip cleared; zero
vision/LLM work).
* Phase 148 (task 01): in this NEW/CHANGED path, RIGHT AFTER the * Phase 148 (task 01): in this NEW/CHANGED path, RIGHT AFTER the
row's flush and BEFORE the chunk pipeline, a page-QUALIFYING row's flush and BEFORE the chunk pipeline, a page-QUALIFYING
PDF (the phase-133 fallback fired, or the phase-140 pictorial PDF (the phase-133 fallback fired, or the phase-140 pictorial
pass described ≥1 page — the signals recorded above) gets its pass described ≥1 page — the signals recorded above) gets its
every page rendered to ``settings.pdf_page_dir`` as every page rendered to ``settings.pdf_page_dir`` as
``<doc-id>-p{N}.png`` and ``documents.page_count`` set ``<doc-id>-p{N}.png`` and ``documents.page_count`` +
(:func:`_persist_pdf_pages` — stale copies deleted first, a ``documents.page_strip_mode`` set
stopped-qualifier cleared, a per-page ``b""`` a gap). FAIL-SOFT (:func:`_persist_pdf_pages` — WRITE-FIRST (phase 163, LOCKED
— a render/write exception leaves ``page_count`` NULL + a A4): the new copies land atomically per file BEFORE any old copy
warning; the document indexes exactly as pre-phase. is removed, a stopped-qualifier cleared, a per-page ``b""`` a
gap). FAIL-SOFT — a render/write exception leaves
``page_count``/``page_strip_mode`` NULL + a warning; the
document indexes exactly as pre-phase.
* THEN the embedded images: :func:`_index_pdf_embedded_images` * THEN the embedded images: :func:`_index_pdf_embedded_images`
(A3 — the phase-122 pipeline, gated by the ``images`` toggle). (A3 — the phase-122 pipeline, gated by the ``images`` toggle).
@@ -1631,7 +1856,7 @@ async def _index_pdf_file(
# fast path). # fast path).
await _unchanged_doc( await _unchanged_doc(
session, doc, source=source, rel=rel, llm=llm, summary=summary, session, doc, source=source, rel=rel, llm=llm, summary=summary,
raw_date=raw_date, file_digest=file_digest, raw_date=raw_date, file_digest=file_digest, full_path=full_path,
) )
return return
@@ -1673,23 +1898,34 @@ async def _index_pdf_file(
# its id — the copy names need it) and BEFORE the chunk pipeline # its id — the copy names need it) and BEFORE the chunk pipeline
# (the count commits ATOMICALLY with the doc + chunks). Runs ONLY # (the count commits ATOMICALLY with the doc + chunks). Runs ONLY
# in this new/changed path: the phase-147 fast path returned # in this new/changed path: the phase-147 fast path returned
# before the branch (an unchanged PDF re-renders NOTHING — the # before the branch and the content-digest hit above took the
# copies persist) and the content-digest hit above took the shared # shared unchanged path — neither renders here; the phase-163
# unchanged path (no page work there either). FAIL-SOFT: the # task-03 re-gate (:func:`_regate_pdf_strip`, run by
# render is a DISPLAY concern — any exception leaves # :func:`_unchanged_doc`) owns the repair of an unchanged doc's
# ``page_count`` NULL, the warning is the signal, and the document # strip. FAIL-SOFT: the render is a DISPLAY
# proceeds to chunk/embed/summary EXACTLY as pre-phase. # concern — any exception leaves ``page_count``/``page_strip_mode``
# NULL, the warning is the signal, and the document proceeds to
# chunk/embed/summary EXACTLY as pre-phase.
# Phase 163 (task 02): the eligibility decision carried as a CLASS
# (LOCKED A7 — the truth table is the phase-148 rule verbatim: a
# non-None class ⇔ fallback_fired or pictorial_described). The
# class decides the stamped `page_strip_mode` ("fallback" →
# "primary", "pictorial" → "additive"), nothing else.
strip_class = (
"fallback" if fallback_fired else ("pictorial" if pictorial_described else None)
)
try: try:
_persist_pdf_pages( _persist_pdf_pages(
doc, doc,
full_path=full_path, full_path=full_path,
page_dir=Path(llm.settings.pdf_page_dir).expanduser(), page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
eligible=fallback_fired or pictorial_described, strip_class=strip_class,
source=source, source=source,
rel=rel, rel=rel,
) )
except Exception as e: except Exception as e:
doc.page_count = None doc.page_count = None
doc.page_strip_mode = None
logger.warning( logger.warning(
"import: PDF page render failed source=%s path=%s — %s", "import: PDF page render failed source=%s path=%s — %s",
source, rel, e, source, rel, e,
@@ -1991,84 +2227,186 @@ def _persist_pdf_pages(
*, *,
full_path: Path, full_path: Path,
page_dir: Path, page_dir: Path,
eligible: bool, strip_class: str | None,
source: str, source: str,
rel: str, rel: str,
) -> None: ) -> None:
"""Phase 148 (task 01): the persistent page copies of one NEW/CHANGED """Phase 148 (task 01) + phase 163 (task 02): the persistent page
PDF — the display strip's bytes (task 02's public route serves them). copies of one NEW/CHANGED PDF — the display strip's bytes (task 02's
public route serves them) — written WRITE-FIRST and stamped with the
strip's CLASS.
* ELIGIBLE (the phase-133 fallback fired, or the phase-140 *strip_class* is the sync's eligibility decision carried as a CLASS
pictorial pass described ≥1 page — the sync's own decisions, see (phase 163, LOCKED A7): ``"fallback"`` (the phase-133 fallback fired
the caller's signals): EVERY page of the PDF is rendered (150 dpi, — the screenshots ARE the content → stamped mode ``"primary"``),
``"pictorial"`` (the phase-140 pass described ≥1 sparse page →
stamped mode ``"additive"``), or ``None`` (no strip). The truth
table is the phase-148 eligibility rule VERBATIM: a non-None class
⇔ the old ``fallback_fired or pictorial_described`` — this phase
changes only what the class DOES (the stamp), never who qualifies.
* ELIGIBLE: EVERY page of the PDF is rendered (150 dpi,
:func:`render_pdf_pages`' default — the phase-133 constant; the :func:`render_pdf_pages`' default — the phase-133 constant; the
``pages=None`` "list index *i* is page *i*+1" contract) and each ``pages=None`` "list index *i* is page *i*+1" contract) and each
NON-EMPTY PNG is written to *page_dir* as NON-EMPTY PNG lands in *page_dir* as ``<doc-id>-p{N}.png``
``<doc-id>-p{N}.png`` (N 1-based), then ``doc.page_count`` is set (N 1-based) through an ATOMIC per-file write — the shared writer
to the PDF's page count (``len(pages)`` — one slot per page, so :func:`_write_page_copies` puts the bytes on ``<name>.png.tmp``
the list IS the page count for an openable file). A per-page first and ``os.replace`` moves them onto the final name, so a
concurrent reader of the route NEVER sees a torn PNG (the phase
163 task-03 re-gate heals a strip through THE SAME writer) — and
the success tail stamps ``doc.page_count = len(pages)``
(one slot per page — the list IS the page count for an openable
file) and ``doc.page_strip_mode`` from the class. A per-page
``b""`` (an unreadable page) is a GAP, not an error: no file is ``b""`` (an unreadable page) is a GAP, not an error: no file is
written for it (the route 404s that N, the viewer's note covers written for it (the route 404s that N, the viewer's note covers
it), and the count still covers the full range. it), and the count still covers the full range.
* the STALE-REPLACEMENT convention (phase 122's image copies): the * WRITE-FIRST (phase 163, LOCKED A4 — the 2026-10-07 live loss): the
doc's EXISTING copies (1..``doc.page_count`` — the row knows its OLD copies are deleted LAST, never first. The pre-phase order
own extent, read BEFORE any change) are deleted FIRST, so a (blanket delete → render → write) ran OUTSIDE the DB transaction,
changed PDF's old page set (e.g. 30 pages → 24) cannot leak so any later per-file rollback (the import loop's
``p25..p30``; a fresh doc's NULL count is a no-op. ``EmbeddingError`` handler) — or a crash between delete and write —
* NOT eligible, or an unopenable file (:func:`render_pdf_pages` → restored the stale ``page_count`` over ZERO files and the strip
``[]``), or a render with ZERO non-empty pages: ``page_count`` is 404ed forever. Now only the old-extent files (1..the count read
None + the existing copies deleted — a document that STOPPED BEFORE any change) that the fresh render did NOT write — a shrunk
qualifying loses its strip (e.g. a new version gained a real text PDF's ``p{N}`` tail, a new gap — are removed, and only once every
layer). replacement byte is durably in place; a file about to be
* FAIL-SOFT: a write failure removes the copies THIS CALL wrote overwritten is never deleted at all.
(no partial-write orphans) and re-raises — the caller * NOT eligible (``strip_class is None``), an unopenable file
(:func:`_index_pdf_file`) catches, leaves ``page_count`` NULL, (:func:`render_pdf_pages` → ``[]``), or a render with ZERO
logs the warning, and the document indexes exactly as pre-phase non-empty pages: the existing copies are deleted (the helper reads
(the render is a display concern — it never gates indexing). ``doc.page_count``, so it runs BEFORE the nulling) and BOTH
``page_count`` and ``page_strip_mode`` are cleared — the
deletion-first order is correct HERE, there is no replacement set
to protect; a document that STOPPED qualifying loses its strip
(e.g. a new version gained a real text layer).
* FAIL-SOFT: a write failure removes this call's ``.tmp`` files and
the final-name files it CREATED — but NEVER a file it OVERWROTE
(the replaced copy keeps standing as the durable new bytes; a
failed call may change a copy's bytes, it may never remove a
copy) — and re-raises: the caller (:func:`_index_pdf_file`)
catches, leaves ``page_count`` AND ``page_strip_mode`` NULL, logs
the warning, and the document indexes exactly as pre-phase (the
render is a display concern — it never gates indexing).
*page_dir* is ``Path(settings.pdf_page_dir).expanduser()`` (the *page_dir* is ``Path(settings.pdf_page_dir).expanduser()`` (the
``image_dir`` convention — raw string, expanduser'd by the ``image_dir`` convention — raw string, expanduser'd by the
importer); the directory is created on demand, only when a page is importer); the directory is created on demand, only when a page is
actually written (an ineligible doc never creates it). actually written (an ineligible doc never creates it).
""" """
# 1 — the stale-replacement: this doc's EXISTING copies go first — if strip_class is None:
# a changed PDF's old page set must not leak, and a doc that # No strip — the DELETION-first order is correct here (there is
# stopped qualifying loses its strip (a fresh doc's NULL count is # no replacement set to protect); the helper reads
# a no-op; missing files are a silent no-op). # doc.page_count, so it runs BEFORE the nulling (a fresh doc's
_delete_page_copies(doc, page_dir) # NULL count is a no-op; missing files are a silent no-op).
if not eligible: _delete_page_copies(doc, page_dir)
doc.page_count = None doc.page_count = None
doc.page_strip_mode = None
return return
pages = render_pdf_pages(full_path) pages = render_pdf_pages(full_path)
if not pages or not any(pages): if not pages or not any(pages):
# Unopenable file (the render's ``[]``) or every page a # Unopenable file (the render's ``[]``) or every page a
# placeholder: no strip — fail-soft, the index is untouched # placeholder: no strip and NO replacement set — the existing
# (the render helper already logged the per-page failures). # copies go with the cleared row (helper BEFORE the nulling),
# fail-soft, the index is untouched (the render helper already
# logged the per-page failures).
logger.warning( logger.warning(
"import: PDF page render produced no pages source=%s path=%s", "import: PDF page render produced no pages source=%s path=%s",
source, rel, source, rel,
) )
_delete_page_copies(doc, page_dir)
doc.page_count = None doc.page_count = None
doc.page_strip_mode = None
return return
# The row's OWN old extent, read BEFORE any change — the stale-tail
# delete below walks it (the write-first contract never deletes a
# file outside this extent, and never before its replacement is
# durable).
old_extent = doc.page_count or 0
# 1 — WRITE FIRST, atomically per file (the shared writer, phase
# 163 task 02 — the phase-163 task-03 re-gate heals through it).
written_ns = _write_page_copies(doc, page_dir, pages)
# 2 — the stale tail goes LAST.
_delete_stale_page_tail(
doc_id=doc.id,
page_dir=page_dir,
old_extent=old_extent,
written_ns=written_ns,
)
doc.page_count = len(pages)
doc.page_strip_mode = "primary" if strip_class == "fallback" else "additive"
def _write_page_copies(doc: Document, page_dir: Path, pages: Sequence[bytes]) -> set[int]:
"""Phase 163 (task 02, LOCKED A4; extracted for the task-03 re-gate,
which heals a strip through the SAME writer): write every NON-EMPTY
PNG of *pages* to ``page_dir/<doc-id>-p{N}.png`` (N 1-based) and
return the set of page numbers written.
* ATOMIC per file — the bytes land on ``<name>.png.tmp`` and
``os.replace`` moves them onto the final name, so a concurrent
reader of the ``/pages/{n}`` route never sees a torn PNG.
* A per-page ``b""`` (the render's failure placeholder) is the GAP:
no file, no entry in the returned set.
* The directory is created on demand (a doc that never writes never
creates it).
* A failure leaves NO orphan: this call's ``.tmp`` files and the
finals it CREATED are removed — an OVERWRITTEN final keeps
standing (deleting it would re-open the 2026-10-07 hole with one
page instead of the whole strip) — and the exception re-raises so
the caller keeps its fail-soft contract.
"""
page_dir.mkdir(parents=True, exist_ok=True) page_dir.mkdir(parents=True, exist_ok=True)
written: list[Path] = [] written_tmps: list[Path] = []
created_finals: list[Path] = [] # finals that did NOT exist before
written_ns: set[int] = set()
try: try:
for n, png in enumerate(pages, start=1): for n, png in enumerate(pages, start=1):
if not png: if not png:
# A per-page render failure: the GAP (no file — the # A per-page render failure: the GAP (no file — the
# route 404s that N, the viewer's note covers it). # route 404s that N, the viewer's note covers it).
continue continue
path = page_dir / f"{doc.id}-p{n}.png" final = page_dir / f"{doc.id}-p{n}.png"
path.write_bytes(png) tmp = page_dir / f"{doc.id}-p{n}.png.tmp"
written.append(path) # Track the tmp BEFORE the write: a failure mid-write may
# leave it on disk, and the cleanup must find it either way.
written_tmps.append(tmp)
existed = final.exists()
tmp.write_bytes(png)
os.replace(tmp, final) # atomic per file — no torn reads
written_ns.add(n)
if not existed:
created_finals.append(final)
except Exception: except Exception:
# A partial write leaves no orphan: the copies this call wrote # A partial write leaves no orphan: this call's tmp files and
# go, then the re-raise hands the failure to the caller's # the finals it CREATED go (an OVERWRITTEN final keeps standing
# fail-soft handler (page_count NULL + the warning). # — deleting it would re-open the 2026-10-07 hole with one page
for path in written: # instead of the whole strip), then the re-raise hands the
# failure to the caller's fail-soft handler (page_count +
# page_strip_mode NULL + the warning).
for path in [*written_tmps, *created_finals]:
path.unlink(missing_ok=True) path.unlink(missing_ok=True)
raise raise
doc.page_count = len(pages) return written_ns
def _delete_stale_page_tail(
*, doc_id: uuid.UUID, page_dir: Path, old_extent: int, written_ns: set[int]
) -> None:
"""Phase 163 (task 02, LOCKED A4 — the LAST step of the write-first
order): remove the files of the row's OLD extent (1..*old_extent*)
the fresh render did NOT write — a shrunk PDF's ``p{N}`` tail, a new
gap. Runs only once every replacement byte is durable, never touches
a file outside the old extent, and never touches a file that was just
overwritten. Best-effort like :func:`_delete_page_copies`: an
unreadable file is logged, never raised (the sync survives)."""
for n in range(1, old_extent + 1):
if n in written_ns:
continue
path = page_dir / f"{doc_id}-p{n}.png"
try:
path.unlink(missing_ok=True)
except OSError as e:
logger.warning("import: could not delete PDF page copy %s — %s", path, e)
async def _index_pdf_embedded_images( async def _index_pdf_embedded_images(
@@ -3271,7 +3609,7 @@ async def _index_image_file(
# ``_store_summary`` backfill re-stores it verbatim. # ``_store_summary`` backfill re-stores it verbatim.
await _unchanged_doc( await _unchanged_doc(
session, doc, source=source, rel=rel, llm=llm, summary=summary, session, doc, source=source, rel=rel, llm=llm, summary=summary,
raw_date=raw_date, file_digest=file_digest, raw_date=raw_date, file_digest=file_digest, full_path=full_path,
) )
return return
+18
View File
@@ -636,6 +636,19 @@ class DocContent(BaseModel):
#: ``null``, the :attr:`image_url` omission precedent), so a #: ``null``, the :attr:`image_url` omission precedent), so a
#: non-qualifying doc's payload is byte-identical to pre-phase. #: non-qualifying doc's payload is byte-identical to pre-phase.
page_urls: list[str] | None = None page_urls: list[str] | None = None
#: Phase 163 (task 01, LOCKED A2/A3) — the STRIP CLASS behind
#: :attr:`page_urls`: ``"primary"`` = the phase-133 fallback class
#: (an unusable text layer — the screenshots ARE the content, so the
#: strip IS the view, exactly as phase 148 shipped it); ``"additive"``
#: = the phase-140 pictorial class (a usable text layer with ≥1 sparse
#: page described — the text view + the Summary panel, the strip one
#: Text/Pages toggle away). ABSENT from the wire whenever there is no
#: strip (``None`` → dropped by the serializer — never ``null``, the
#: :attr:`image_url` omission rule), which is ALSO how a pre-phase row
#: (``page_count`` set, the column NULL) serializes: the viewer treats
#: an absent mode as ``"primary"`` (task 04), so every existing row
#: keeps today's DOM until the fast-path re-gate stamps it.
page_strip_mode: str | None = None
@model_serializer(mode="wrap") @model_serializer(mode="wrap")
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any: def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
@@ -648,6 +661,11 @@ class DocContent(BaseModel):
data.pop("origin_branch", None) data.pop("origin_branch", None)
if self.page_urls is None: if self.page_urls is None:
data.pop("page_urls", None) data.pop("page_urls", None)
# Phase 163 (task 01): the strip class rides the strip — ABSENT
# (never null) with no strip, so a non-qualifying doc's payload is
# byte-identical to pre-phase.
if self.page_strip_mode is None:
data.pop("page_strip_mode", None)
return data return data
+101 -8
View File
@@ -144,8 +144,9 @@
* wire for every other document, so they render the EXACT pre-phase * wire for every other document, so they render the EXACT pre-phase
* DOM on BOTH surfaces — the load-bearing byte-identity property), * DOM on BOTH surfaces — the load-bearing byte-identity property),
* #doc-content renders a .doc-pages strip INSTEAD of the text * #doc-content renders a .doc-pages strip INSTEAD of the text
* content AND the .doc-summary panel: one <figure class="doc-page"> * content AND the .doc-summary panel FOR THE PRIMARY CLASS ONLY
* per URL — a lazy <img> (alt = "Page N of <title>", the WCAG alt * (phase 163 — the absent/`primary` mode; see the next paragraph):
* one <figure class="doc-page"> per URL — a lazy <img> (alt = "Page N of <title>", the WCAG alt
* contract — the title as the fallback context) + a "Page N" * contract — the title as the fallback context) + a "Page N"
* <figcaption> — while the meta badge row renders exactly as today. * <figcaption> — while the meta badge row renders exactly as today.
* The owner's contract: the per-page descriptions stay in the DB for * The owner's contract: the per-page descriptions stay in the DB for
@@ -160,6 +161,26 @@
* per-surface copy); createElement + properties/textContent only (the * per-surface copy); createElement + properties/textContent only (the
* never-build-HTML rule — no innerHTML). * never-build-HTML rule — no innerHTML).
* *
* Phase 163 (task 04, LOCKED A1/A2/A6): the strip's CLASS decides the
* view. page_strip_mode === "additive" (the pictorial class — a USABLE
* text layer, so the pages are a convenience, not the content) keeps
* the TEXT doc EXACTLY as a non-qualifying doc renders it (Summary
* panel + content — the pre-148 branch reused), and the strip sits one
* Text/Pages toggle away: a .doc-view-toggle group (role=group,
* aria-label="Document view", two <button aria-pressed>) above a
* .doc-views wrapper holding the text view + the strip; Text is
* pressed by DEFAULT (A6 — never a sparse-ratio flip), Pages flips the
* wrapper's data-view (CSS shows the strip / hides the text — the
* strip node stays MOUNTED, its lazy imgs never refetch on flip-flop).
* The text is NEVER hidden for this class. The absent/"primary" mode
* (the fallback class — the screenshots ARE the content, and every
* pre-163 stamped row) keeps the strip-only DOM above byte-identically
* (the tolerant read: only the exact "additive" string switches on the
* toggle); falsy page_urls stay untouched. The wrapper exists ONLY on
* additive docs — every other document's DOM is unchanged. Toggle
* labels are fixed strings (no doc-derived text); createElement +
* properties/textContent only.
*
* Phase 153 (task 02): the meta row's "Copy path" control — the * Phase 153 (task 02): the meta row's "Copy path" control — the
* read-tool `source/path` form in the clipboard with one click * read-tool `source/path` form in the clipboard with one click
* (ASSUMPTION A1 — the exact `source/path` the read tool, the * (ASSUMPTION A1 — the exact `source/path` the read tool, the
@@ -361,7 +382,9 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
// Phase 148 (task 03): the page strip — the payload's page_urls // Phase 148 (task 03): the page strip — the payload's page_urls
// (task 02's endpoint: the exact 1-based route list for a // (task 02's endpoint: the exact 1-based route list for a
// page-qualifying PDF; ABSENT on the wire for every other // page-qualifying PDF; ABSENT on the wire for every other
// document) is the ONE branch that replaces the content slot: the // document) is the ONE branch that replaces the content slot (phase
// 163: the PRIMARY class only — an "additive" doc falls through to
// the text rendering with the strip behind a toggle, D4/A1): the
// markdown/.doc-raw content block AND the labeled Summary panel // markdown/.doc-raw content block AND the labeled Summary panel
// (including its admin edit wiring — moot without the panel) are // (including its admin edit wiring — moot without the panel) are
// NOT built. The per-page descriptions stay in the DB for Q&A (the // NOT built. The per-page descriptions stay in the DB for Q&A (the
@@ -372,16 +395,40 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
// below (byte-identical for every other doc — the load-bearing // below (byte-identical for every other doc — the load-bearing
// property). // property).
const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null; const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;
if (pageUrls && pageUrls.length > 0) { const stripEligible = pageUrls !== null && pageUrls.length > 0;
// Phase 163 (task 04, LOCKED A2): the strip's CLASS picks the view.
// Only the exact "additive" string switches on the toggle — absent
// (pre-163 rows, and the omission rule's non-qualifying docs) or
// "primary" read as primary (the tolerant-reading house rule), so
// the fallback class keeps the strip-only DOM byte-identically.
const additive = stripEligible && doc.page_strip_mode === "additive";
if (stripEligible && !additive) {
contentEl.appendChild(docPagesStrip(doc, pageUrls)); contentEl.appendChild(docPagesStrip(doc, pageUrls));
return; return;
} }
// Phase 163 (task 04, LOCKED A1/A6): an ADDITIVE doc renders the
// text doc EXACTLY as a non-qualifying one does (Summary panel +
// content below, reused unchanged) INSIDE a .doc-views wrapper —
// the Text/Pages toggle above it flips the wrapper's data-view
// (CSS-driven show/hide, never a re-render); the strip mounts once
// AFTER the text view and stays mounted across flips (lazy imgs,
// display:none while hidden, never fetch). Every non-additive doc
// keeps slot === contentEl — its DOM is byte-identical to pre-phase.
let views = null;
let slot = contentEl;
if (additive) {
views = document.createElement("div");
views.className = "doc-views";
views.setAttribute("data-view", "text"); // LOCKED A6: Text is the default view
contentEl.append(docViewToggle(views), views);
slot = views;
}
// Phase 122 (task 04): an image document — the persistent bytes // Phase 122 (task 04): an image document — the persistent bytes
// render as the <img> block FIRST; the description (= doc.content) // render as the <img> block FIRST; the description (= doc.content)
// follows in the normal content slot below (the plain-content path // follows in the normal content slot below (the plain-content path
// — the last branch in this function). // — the last branch in this function).
if (doc.is_image) { if (doc.is_image) {
contentEl.appendChild(docImageBlock(doc)); slot.appendChild(docImageBlock(doc));
} }
// Phase 36: the summary panel — labeled section ABOVE the original // Phase 36: the summary panel — labeled section ABOVE the original
// content, on BOTH surfaces (page + modal) through this one core. // content, on BOTH surfaces (page + modal) through this one core.
@@ -405,7 +452,7 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
body.className = "doc-summary-text"; body.className = "doc-summary-text";
body.textContent = doc.summary; // text node — XSS contract unchanged body.textContent = doc.summary; // text node — XSS contract unchanged
section.append(title, body); section.append(title, body);
contentEl.appendChild(section); slot.appendChild(section);
// Phase 57 (task 02, D4): the section above is the phase-36 shape // Phase 57 (task 02, D4): the section above is the phase-36 shape
// for EVERYONE — only an authenticated admin (docAdminReady(), the // for EVERYONE — only an authenticated admin (docAdminReady(), the
// module-cached whoami promise) then gains the header row + Edit // module-cached whoami promise) then gains the header row + Edit
@@ -419,13 +466,59 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
const wrap = document.createElement("div"); const wrap = document.createElement("div");
wrap.className = "doc-md"; wrap.className = "doc-md";
wrap.innerHTML = renderMarkdown(doc.content); // escape-first: XSS-safe wrap.innerHTML = renderMarkdown(doc.content); // escape-first: XSS-safe
contentEl.appendChild(wrap); slot.appendChild(wrap);
} else { } else {
const pre = document.createElement("pre"); const pre = document.createElement("pre");
pre.className = "doc-raw"; pre.className = "doc-raw";
pre.textContent = doc.content; // text node: never parsed as HTML pre.textContent = doc.content; // text node: never parsed as HTML
contentEl.appendChild(pre); slot.appendChild(pre);
} }
// Phase 163 (task 04): the additive strip mounts once, AFTER the
// text view, inside the wrapper — hidden by [data-view="text"]
// (the lazy imgs never fetch until Pages is chosen; the node
// persists across flips — no refetch).
if (views !== null) {
views.appendChild(docPagesStrip(doc, pageUrls));
}
}
/* ---------- view toggle (phase 163, task 04 — LOCKED A1/A6) ----------
* The Text/Pages toggle of an ADDITIVE strip doc: one
* <div class="doc-view-toggle" role="group" aria-label="Document
* view"> with two <button type="button" class="doc-view-btn"
* aria-pressed>. Text is pressed initially (A6 — no sparse-ratio
* default flip). A click mutates ONLY the two aria-pressed states and
* the .doc-views wrapper's data-view attribute — the CSS shows/hides
* the text view and the strip (never a re-render, never a refetch).
* The labels are FIXED strings (no doc-derived text inside the
* toggle); createElement + properties/textContent only (the
* never-build-HTML rule — no innerHTML). */
function docViewToggle(viewsEl) {
const group = document.createElement("div");
group.className = "doc-view-toggle";
group.setAttribute("role", "group");
group.setAttribute("aria-label", "Document view");
const textBtn = viewToggleButton("Text");
const pagesBtn = viewToggleButton("Pages");
const select = (view) => {
viewsEl.setAttribute("data-view", view);
textBtn.setAttribute("aria-pressed", view === "text" ? "true" : "false");
pagesBtn.setAttribute("aria-pressed", view === "pages" ? "true" : "false");
};
textBtn.setAttribute("aria-pressed", "true"); // LOCKED A6: Text pressed by default
pagesBtn.setAttribute("aria-pressed", "false");
textBtn.addEventListener("click", () => select("text"));
pagesBtn.addEventListener("click", () => select("pages"));
group.append(textBtn, pagesBtn);
return group;
}
function viewToggleButton(label) {
const btn = document.createElement("button");
btn.type = "button";
btn.className = "doc-view-btn";
btn.textContent = label; // fixed string — never doc-derived
return btn;
} }
/* ---------- image block (phase 122, task 04) ---------- /* ---------- image block (phase 122, task 04) ----------
+53
View File
@@ -5311,6 +5311,59 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
padding: 0.35rem 0 0; padding: 0.35rem 0 0;
} }
/* Additive strip view (phase 163, task 04): the pictorial-class PDF
— a USABLE text layer — keeps the text doc (Summary panel +
content) with the page strip one Text/Pages toggle away (LOCKED
A1/A6). The toggle is the house ghost-pill idiom (the .auth-link
family): the two pills sit adjacent in the labeled group, ink-soft
on the transparent pill (AA on --bg/--surface), hover the
brand-soft/brand-ink pair; the PRESSED button is the brand fill +
heavier weight — text + style, never color alone (B5) — with --bg
ink on --brand (5.2:1, the .share-chat-btn pair). ≥44px targets
(§7.1); :focus-visible rides the global 3px ring. The view swap is
CSS-driven off the .doc-views wrapper's data-view: the strip is
display:none by default (its lazy imgs never fetch while hidden)
and shows on [data-view="pages"], which hides the text view instead
— the nodes stay mounted across flips (no re-render, no refetch).
Phase-08 tokens only (no new hue — the monochrome invariant). */
.doc-view-toggle {
display: flex;
gap: 0.4rem;
width: 100%;
max-width: var(--chat-column);
margin-inline: auto;
margin-bottom: 0.75rem;
}
.doc-view-btn {
display: inline-flex;
align-items: center;
justify-content: center;
min-height: 44px;
padding: 0.5rem 0.9rem;
border-radius: 999px;
border: 1px solid var(--line);
background: transparent;
color: var(--ink-soft);
font: inherit;
font-weight: 600;
font-size: 0.95rem;
white-space: nowrap;
cursor: pointer;
}
.doc-view-btn:hover { background: var(--brand-soft); color: var(--brand-ink); }
.doc-view-btn[aria-pressed="true"] {
background: var(--brand);
border-color: var(--brand);
color: var(--bg); /* dark ink on brand: 5.2:1 (AA) */
font-weight: 700; /* pressed = weight + fill, never color alone (B5) */
}
.doc-views .doc-pages { display: none; } /* Text default — hidden lazy imgs never fetch */
.doc-views[data-view="pages"] .doc-summary,
.doc-views[data-view="pages"] .doc-md,
.doc-views[data-view="pages"] .doc-raw,
.doc-views[data-view="pages"] .doc-image { display: none; }
.doc-views[data-view="pages"] .doc-pages { display: grid; }
/* Designed not-found state (no emoji — plain SVG mark, phase 08 rule). */ /* Designed not-found state (no emoji — plain SVG mark, phase 08 rule). */
.doc-not-found { .doc-not-found {
width: 100%; width: 100%;
+83 -48
View File
@@ -36,33 +36,38 @@ token (the phase-126 convention — the owner's local ``.env`` may lack
it, and without it the walk is blind to the fixture). it, and without it the walk is blind to the fixture).
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
blocks (and the seed's DB witness that the descriptions ARE indexed blocks (and the seed's DB witness that the descriptions ARE indexed)
but NOT displayed) are exact only against the mock's deterministic are exact only against the mock's deterministic
``PICTORIAL_PAGE_MODE`` canned reply. ``PICTORIAL_PAGE_MODE`` canned reply. (Phase 163 re-cut: the viewer
assertions moved from the phase-148 suppression contract to the
ADDITIVE contract — the pictorial class shows its text INCLUDING the
page descriptions, with the strip behind the Text/Pages toggle.)
Test → observable mapping (Playwright Mapping Rule): Test → observable mapping (Playwright Mapping Rule):
1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module) 1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module)
asserts the DB + filesystem first: the pictorial row has asserts the DB + filesystem first: the pictorial row has
``page_count == 3`` and the scratch page dir holds EXACTLY ``page_count == 3`` and the scratch page dir holds EXACTLY
``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy ``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy
doc's copies do not exist); then the Sources-tree doc modal shows doc's copies do not exist); then the Sources-tree doc modal opens
the ``.doc-pages`` strip: exactly three ``.doc-page`` figures, the ADDITIVE view (phase 163 — a usable text layer: TEXT first,
each a LOADED lazy ``<img>`` (``complete && naturalWidth > 0`` — the strip one toggle away): the Text/Pages group with ``Text``
the route served a real PNG, not a 404) with the pressed, the Summary panel + content visible (the mock's page
``Page N of <title>`` alt, the ``Page N`` caption, the NO descriptions RIDE the shown text — LOCKED A1/A6), the
``.doc-raw`` / NO ``.doc-md`` / NO ``.doc-summary`` suppression ``.doc-pages`` strip mounted but hidden; pressing ``Pages`` shows
contract (the mock's page descriptions sit in the DB content but exactly three ``.doc-page`` figures, each a LOADED lazy ``<img>``
nowhere in the rendered modal), the meta badge row (source / (``complete && naturalWidth > 0`` — the route served a real PNG,
Created / Indexed) untouched. not a 404) with the ``Page N of <title>`` alt and the ``Page N``
caption while the text view is hidden (mounted for the flip-
back), the meta badge row (source / Created / Indexed) untouched.
2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the 2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the
PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the
Summary panel (the mock's lite digest), NO ``.doc-pages`` — and Summary panel (the mock's lite digest), NO ``.doc-pages`` — and
the content payload's ``page_urls`` KEY is ABSENT (never ``null`` — the content payload's ``page_urls`` KEY is ABSENT (never ``null`` —
the omission pin the viewer's strip gate rides on). the omission pin the viewer's strip gate rides on).
3. ``test_full_page_surface`` — the same strip on the dedicated 3. ``test_full_page_surface`` — the same additive view + strip on the
``/document.html`` page (admin-logged-in, the phase-79 gate dedicated ``/document.html`` page (admin-logged-in, the phase-79
satisfied): one ``renderDocument`` core — no per-surface drift — gate satisfied): one ``renderDocument`` core — no per-surface
same figures, same suppression. drift — same toggle, same figures, same hidden-while-Text shape.
4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct 4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct
request on ``/api/documents/<id>/pages/1``: 200, ``image/png``, request on ``/api/documents/<id>/pages/1``: 200, ``image/png``,
the ``Cache-Control: private, max-age=3600`` header, a non-empty the ``Cache-Control: private, max-age=3600`` header, a non-empty
@@ -597,15 +602,25 @@ def _wait_img_loaded(img: Locator) -> None:
def _assert_page_strip( def _assert_page_strip(
content: Locator, title: str, doc_id: str, page_count: int = 3 content: Locator, title: str, doc_id: str, page_count: int = 3
) -> None: ) -> None:
"""The phase-148 strip contract inside a content container (the """The strip contract inside a content container (the ONE
ONE ``renderDocument`` core — modal and full page, no ``renderDocument`` core — modal and full page, no per-surface
per-surface drift): a ``.doc-pages`` strip with exactly drift): a ``.doc-pages`` strip with exactly *page_count*
*page_count* ``.doc-page`` figures; each figure's lazy ``<img>`` ``.doc-page`` figures; each figure's lazy ``<img>`` is LOADED (the
is LOADED (the route served a real PNG) with the ``Page N of route served a real PNG) with the ``Page N of <title>`` alt and
<title>`` alt and the exact 1-based route src, and the the exact 1-based route src, and the figcaption reads ``Page N``.
figcaption reads ``Page N``; the text content block AND the
Summary panel are SUPPRESSED (the descriptions stay in the DB, Phase 163 (task 04) re-cut: this fixture is the PICTORIAL class —
not displayed).""" an ADDITIVE doc (``page_strip_mode == "additive"``) that opens on
its TEXT view, so the strip is revealed by pressing the toggle's
``Pages`` button first (a primary/absent doc has no toggle — the
strip already IS the content). The text content block and the
Summary panel are HIDDEN while the strip shows (they stay mounted
for the flip back — LOCKED A1); a primary-class doc has no such
nodes at all, which the same ``not_to_be_visible`` assertion
covers."""
toggle = content.locator(".doc-view-toggle")
if toggle.count() > 0:
toggle.get_by_role("button", name="Pages").click()
strip = content.locator(".doc-pages") strip = content.locator(".doc-pages")
expect(strip).to_have_count(1) expect(strip).to_have_count(1)
figures = strip.locator(".doc-page") figures = strip.locator(".doc-page")
@@ -624,12 +639,12 @@ def _assert_page_strip(
) )
_wait_img_loaded(img) _wait_img_loaded(img)
expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}") expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}")
# The suppression contract (the owner's TODO L5): no text content # While the strip shows, the text view is not (phase 163: hidden,
# block, no Summary panel — the meta badge row is asserted by the # not unmounted — the additive flip-back; a primary doc has no
# caller (it renders exactly as today). # such nodes at all). The meta badge row is asserted by the caller.
expect(content.locator("pre.doc-raw")).to_have_count(0) expect(content.locator("pre.doc-raw")).not_to_be_visible()
expect(content.locator(".doc-md")).to_have_count(0) expect(content.locator(".doc-md")).not_to_be_visible()
expect(content.locator(".doc-summary")).to_have_count(0) expect(content.locator(".doc-summary")).not_to_be_visible()
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -640,15 +655,18 @@ def _assert_page_strip(
def test_pictorial_pdf_shows_pages_in_modal( def test_pictorial_pdf_shows_pages_in_modal(
page: Page, app_url: str, seeded: dict[str, Any] page: Page, app_url: str, seeded: dict[str, Any]
) -> None: ) -> None:
"""The IKEA-shaped PDF opens in the doc modal as a strip of its """The IKEA-shaped PDF opens in the doc modal as an ADDITIVE doc
REAL pages (the owner's L5 ask): ``.doc-pages`` with exactly three (phase 163 — a usable text layer: the pictorial class): the TEXT
view first (Summary panel + content visible, the toggle's ``Text``
pressed — LOCKED A6, the mock's page descriptions ride the shown
text), with the strip of its REAL pages (the owner's L5 ask) one
``Pages`` click away: ``.doc-pages`` with exactly three
``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth ``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth
> 0 proves the full pipeline: sync → render → persist → route → > 0 proves the full pipeline: sync → render → persist → route →
viewer) with the ``Page N of <title>`` alt and the ``Page N`` viewer) with the ``Page N of <title>`` alt and the ``Page N``
caption — and the suppression contract: NO text content caption — while the text view is hidden (mounted for the
(``.doc-raw``), NO Summary panel (the mock's page descriptions are flip-back); the meta badge row (source / Created / Indexed)
in the DB content, nowhere in the rendered modal); the meta badge renders as today."""
row (source / Created / Indexed) renders as today."""
login(page, app_url) # lands on /sources.html login(page, app_url) # lands on /sources.html
_drill_to_source(page, SOURCE_NAME) _drill_to_source(page, SOURCE_NAME)
expect(page.locator("#docs-tbody tr")).to_have_count(2) expect(page.locator("#docs-tbody tr")).to_have_count(2)
@@ -658,6 +676,29 @@ def test_pictorial_pdf_shows_pages_in_modal(
) )
content = page.locator("#doc-modal-content") content = page.locator("#doc-modal-content")
# Phase 163 (task 04, LOCKED A1/A6): the pictorial class is an
# ADDITIVE doc — it opens on its TEXT view: Summary panel +
# content visible, the toggle's Text pressed, the strip mounted
# but hidden (its lazy imgs never fetched while hidden).
toggle = content.locator(".doc-view-toggle")
expect(toggle).to_have_count(1)
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
"aria-pressed", "true"
)
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
"aria-pressed", "false"
)
expect(content.locator("pre.doc-raw")).to_be_visible()
expect(content.locator(".doc-summary")).to_be_visible()
expect(content.locator(".doc-pages")).to_be_hidden()
# The text view carries the indexed content — the phase-140 page
# descriptions ride the SHOWN text for an additive doc (the text
# is never hidden: A1).
assert PICTORIAL_PAGE_ANSWER in content.inner_text(), (
"the additive text view shows the indexed content including "
"the described sparse pages"
)
# …and the strip is one Pages click away (the helper flips).
_assert_page_strip(content, seeded["title"], seeded["pictorial_id"]) _assert_page_strip(content, seeded["title"], seeded["pictorial_id"])
# The meta badge row is untouched by the strip (the pre-phase row). # The meta badge row is untouched by the strip (the pre-phase row).
@@ -666,13 +707,6 @@ def test_pictorial_pdf_shows_pages_in_modal(
expect(meta.locator(".doc-created")).to_contain_text("Created") expect(meta.locator(".doc-created")).to_contain_text("Created")
expect(meta.locator(".doc-indexed")).to_contain_text("Indexed") expect(meta.locator(".doc-indexed")).to_contain_text("Indexed")
# The suppression witness: the mock's canned page descriptions sit
# in the DB (the seed asserted it) but are NOT displayed.
assert PICTORIAL_PAGE_ANSWER not in content.inner_text(), (
"the described pages' text is in the DB for Q&A — it must not "
"be shown to the user (the owner's contract)"
)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# 2. The text-heavy PDF: byte-identical to pre-phase # 2. The text-heavy PDF: byte-identical to pre-phase
@@ -730,10 +764,11 @@ def test_textheavy_pdf_unchanged(
def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None: def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None:
"""The dedicated ``/document.html`` page (admin-logged-in — the """The dedicated ``/document.html`` page (admin-logged-in — the
phase-79 gate satisfied) renders the SAME strip through the ONE phase-79 gate satisfied) renders the SAME additive view through
shared ``renderDocument`` core: the same three loaded figures with the ONE shared ``renderDocument`` core (no per-surface drift): the
captions/alt texts, the same suppression (no text content, no text view first, the strip one ``Pages`` click away with the same
Summary panel), the meta row as today.""" three loaded figures + captions/alt texts and the text view
hidden while it shows, the meta row as today."""
login(page, app_url, next="/") login(page, app_url, next="/")
page.goto( page.goto(
f"{app_url}/document.html?source={quote(SOURCE_NAME)}" f"{app_url}/document.html?source={quote(SOURCE_NAME)}"
File diff suppressed because it is too large Load Diff
+64 -44
View File
@@ -24,11 +24,14 @@ format — the upload gate admits it) and arriving through the REAL admin
upload flow → the RAG page's Sync button: upload flow → the RAG page's Sync button:
* (a) the document appears on the Sources page and the document viewer * (a) the document appears on the Sources page and the document viewer
shows the PDF's ACTUAL pages (phase 148 — a described sparse page opens the PDF in its ADDITIVE view (phase 163 — a described sparse
makes the doc page-QUALIFYING: the page strip REPLACES the text page makes the doc page-QUALIFYING, and a usable text layer makes
content AND the Summary panel, the owner's TODO L5 contract): two the strip ADDITIVE: the text stays, the pages sit behind a
``.doc-page`` figures with loaded lazy images + ``Page N`` captions, Text/Pages toggle — LOCKED A1/A6): the Text view first (``.doc-raw``
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself + ``.doc-summary`` visible, ``Text`` pressed), the strip one
``Pages`` click away — two ``.doc-page`` figures with loaded lazy
images + ``Page N`` captions, the text view hidden while it shows;
the described content itself
(page 1's prose WITHOUT an illustration block, page 2 WITH (page 1's prose WITHOUT an illustration block, page 2 WITH
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]`` ``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in — the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
@@ -83,14 +86,17 @@ Test → observable mapping (Playwright Mapping Rule):
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks + Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
the summary chunk — no image doc), the drill-down table lists the PDF the summary chunk — no image doc), the drill-down table lists the PDF
as a plain text row (LOCKED A4 — a text document, no thumbnail), and as a plain text row (LOCKED A4 — a text document, no thumbnail), and
the same-page document viewer (the row's path link) shows the the same-page document viewer (the row's path link) opens the
PHASE-148 PAGE STRIP under the METADATA title (the fixture is PHASE-163 ADDITIVE VIEW under the METADATA title (the fixture is
page-qualifying — the pictorial pass described its sparse page): page-qualifying with a usable text layer — the pictorial pass
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>`` described its sparse page): the TEXT view first — ``.doc-raw`` +
(naturalWidth > 0) with the ``Page N of <title>`` alt + the ``.doc-summary`` visible with the ``[Page 2 illustration: …]``
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the block among them, the toggle's ``Text`` pressed, the strip mounted
page descriptions stay in the DB for Q&A, not displayed — the but hidden; pressing ``Pages`` shows exactly two ``.doc-page``
owner's contract). figures, each a LOADED lazy ``<img>`` (naturalWidth > 0) with the
``Page N of <title>`` alt + the ``Page N`` caption while the text
view hides (mounted for the flip-back — the text is never
destroyed).
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw 2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
SSE over the scripted seeded-read question (httpx — the ``done`` SSE over the scripted seeded-read question (httpx — the ``done``
event is the wire lens the task pins; the UI flow's premise is the event is the wire lens the task pins; the UI flow's premise is the
@@ -578,8 +584,8 @@ def seeded(
) )
# The non-markdown doc got its lite summary (the mock's # The non-markdown doc got its lite summary (the mock's
# SUMMARY_MODE digest) — the summary chunk exists (the # SUMMARY_MODE digest) — the summary chunk exists (the
# viewer's Summary panel is suppressed by the phase-148 # additive view shows the Summary panel; the summary
# strip — the summary itself stays in the DB). # itself stays in the DB regardless).
assert doc.summary assert doc.summary
# Phase 148: the described sparse page makes this doc # Phase 148: the described sparse page makes this doc
# page-QUALIFYING — the strip's payload (page_urls) rides # page-QUALIFYING — the strip's payload (page_urls) rides
@@ -602,18 +608,19 @@ def test_sources_page_and_viewer_show_the_described_page(
content chunks + the phase-30 summary chunk — no image doc; the content chunks + the phase-30 summary chunk — no image doc; the
vector figure is invisible to the embedded-image pipeline). The vector figure is invisible to the embedded-image pipeline). The
drill-down table lists the PDF as a plain text row (LOCKED A4), and drill-down table lists the PDF as a plain text row (LOCKED A4), and
the same-page document viewer (the row's path link, phase 26) shows the same-page document viewer (the row's path link, phase 26)
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is opens the PHASE-163 ADDITIVE VIEW (LOCKED A1/A6): the TEXT view
page-qualifying (the pictorial pass described its sparse page), so first — Summary panel + ``.doc-raw`` visible, the Text/Pages
the strip REPLACES the text content AND the Summary panel — two toggle's ``Text`` pressed — with the ``[Page 2 illustration:
``.doc-page`` figures, each a loaded lazy ``<img>`` with the <canned>]`` block riding the SHOWN text (the text is never hidden
``Page N of <title>`` alt + the ``Page N`` caption, NO for the pictorial class). Pressing ``Pages`` reveals the strip —
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's two ``.doc-page`` figures, each a loaded lazy ``<img>`` with the
prose WITHOUT an illustration block, page 2 WITH ``Page N of <title>`` alt + the ``Page N`` caption — while the
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB text view is hidden (mounted, not removed: flipping back shows it
for Q&A: the seed asserted it byte-exact, the chat test proves it with zero refetch). The described content (page 1's prose WITHOUT
reaches the model — it is simply not displayed (the owner's an illustration block, page 2 WITH the block — LOCKED A4) also
contract).""" stays in the DB for Q&A: the seed asserted it byte-exact, the chat
test proves it reaches the model."""
login(page, app_url) # lands on /sources.html login(page, app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("1") expect(page.locator("#stat-docs")).to_have_text("1")
expect(page.locator("#stat-chunks")).to_have_text("3") expect(page.locator("#stat-chunks")).to_have_text("3")
@@ -630,13 +637,31 @@ def test_sources_page_and_viewer_show_the_described_page(
expect(modal).to_be_visible(timeout=30_000) expect(modal).to_be_visible(timeout=30_000)
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE) expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
# Phase 148: the page-qualifying fixture shows its ACTUAL pages — # Phase 163 (task 04, LOCKED A1/A6): the page-qualifying fixture
# a .doc-pages strip with exactly two .doc-page figures (the # with a usable text layer is an ADDITIVE doc — it opens on its
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 — # TEXT view: Summary panel + content visible, Text pressed, the
# the route served a real PNG) with the "Page N of <title>" alt + # strip mounted but hidden (its lazy imgs never fetched).
# the "Page N" caption. The text content AND the Summary panel are toggle = modal.locator(".doc-view-toggle")
# SUPPRESSED (the owner's contract — the descriptions stay in the expect(toggle).to_have_count(1)
# DB for Q&A, not displayed). expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
"aria-pressed", "true"
)
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
"aria-pressed", "false"
)
expect(modal.locator("pre.doc-raw")).to_be_visible()
expect(modal.locator(".doc-summary")).to_be_visible()
expect(modal.locator(".doc-pages")).to_be_hidden()
# The canned page description RIDES the shown text (phase-140
# blocks are part of the content — the additive view never hides
# the text). The node stays in the DOM, so a visibility-scoped
# inner_text() check would miss it — use the DOM text.
assert PICTORIAL_PAGE_ANSWER in modal.evaluate("el => el.textContent"), (
"the additive text view shows the indexed content including "
"the described sparse page"
)
# …and the strip is one Pages click away (phase 148's ask):
toggle.get_by_role("button", name="Pages").click()
strip = modal.locator(".doc-pages") strip = modal.locator(".doc-pages")
expect(strip).to_have_count(1) expect(strip).to_have_count(1)
figures = strip.locator(".doc-page") figures = strip.locator(".doc-page")
@@ -657,17 +682,12 @@ def test_sources_page_and_viewer_show_the_described_page(
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text( expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
f"Page {n}" f"Page {n}"
) )
# The suppression contract: no text content block, no Summary # While the strip shows, the text view is HIDDEN — mounted for
# panel, no image block (the PDF is not an image document). # the flip back (LOCKED A1), never removed. No image block (the
expect(modal.locator("pre.doc-raw")).to_have_count(0) # PDF is not an image document).
expect(modal.locator(".doc-summary")).to_have_count(0) expect(modal.locator("pre.doc-raw")).not_to_be_visible()
expect(modal.locator(".doc-summary")).not_to_be_visible()
expect(modal.locator(".doc-image-img")).to_have_count(0) expect(modal.locator(".doc-image-img")).to_have_count(0)
# The canned description is in the DB (the seed asserted it)
# but nowhere in the rendered modal.
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
"the described page's text is in the DB for Q&A — it must not "
"be shown to the user (the owner's contract)"
)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+83 -3
View File
@@ -10,7 +10,10 @@ per phase 124 A4) and the phase-148 (task 02) PDF page route
(``GET /api/documents/{id}/pages/{n}`` — the same trust model and 404 (``GET /api/documents/{id}/pages/{n}`` — the same trust model and 404
posture for a page-qualifying document's rendered page PNGs) + the posture for a page-qualifying document's rendered page PNGs) + the
content endpoint's ``image_url`` / ``page_urls`` keys (present for the content endpoint's ``image_url`` / ``page_urls`` keys (present for the
qualifying row, ABSENT otherwise — never ``null``). qualifying row, ABSENT otherwise — never ``null``), plus the phase-163
(task 01) ``page_strip_mode`` class signal (``"primary"`` /``"additive"``
for a stamped strip, ABSENT for a pre-phase row and for every
non-qualifying document).
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient. Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
""" """
@@ -1381,7 +1384,8 @@ def test_content_endpoint_exposes_the_image_affordance(
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Phase 148 (task 02) — the PDF page BYTES route + the content payload's # Phase 148 (task 02) — the PDF page BYTES route + the content payload's
# ``page_urls`` affordance # ``page_urls`` affordance (phase 163 task 01 adds the ``page_strip_mode``
# class signal beside it)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -1404,6 +1408,7 @@ def _seed_page_doc(
path: str = "besta.pdf", path: str = "besta.pdf",
page_count: int | None = 2, page_count: int | None = 2,
write_copies: bool = True, write_copies: bool = True,
page_strip_mode: str | None = None,
) -> Document: ) -> Document:
"""One PDF document row with ``page_count`` + its rendered page """One PDF document row with ``page_count`` + its rendered page
copies (the importer's ``pdf_page_dir`` layout — copies (the importer's ``pdf_page_dir`` layout —
@@ -1411,7 +1416,8 @@ def _seed_page_doc(
sentinel bytes, so a page mix-up between the two copies is caught. sentinel bytes, so a page mix-up between the two copies is caught.
``page_count=None`` is a non-qualifying PDF (no copies); ``page_count=None`` is a non-qualifying PDF (no copies);
``write_copies=False`` leaves a qualifying row without its files ``write_copies=False`` leaves a qualifying row without its files
(the lost-copy corner).""" (the lost-copy corner); ``page_strip_mode`` (phase 163) stamps the
strip CLASS — ``None`` is the pre-phase row."""
doc = Document( doc = Document(
id=uuid.uuid4(), id=uuid.uuid4(),
source=source, source=source,
@@ -1424,6 +1430,7 @@ def _seed_page_doc(
created_at=datetime(2026, 1, 1, tzinfo=UTC), created_at=datetime(2026, 1, 1, tzinfo=UTC),
summary="IKEA BESTA frame assembly manual (page descriptions).", summary="IKEA BESTA frame assembly manual (page descriptions).",
page_count=page_count, page_count=page_count,
page_strip_mode=page_strip_mode,
) )
db.add(doc) db.add(doc)
db.flush() db.flush()
@@ -1610,6 +1617,79 @@ def test_content_endpoint_exposes_the_page_urls_affordance(
) )
assert r.status_code == 200 assert r.status_code == 200
assert "page_urls" not in r.json() # absent — never null (non-qualifying PDF) assert "page_urls" not in r.json() # absent — never null (non-qualifying PDF)
# Phase 163 (task 01): the class key follows the same rule — a
# pre-phase row (mode NULL) omits it while keeping page_urls.
assert "page_strip_mode" not in r.json()
finally:
_cleanup_kb(db)
@pytest.mark.parametrize("mode", ["primary", "additive"])
def test_content_endpoint_carries_the_strip_class(
admin_client: TestClient, db, tmp_path: Path, mode: str
) -> None:
"""Phase 163 (task 01, LOCKED A2/A3) — the real-DB round-trip of the
strip CLASS: a stamped row (``page_strip_mode`` = ``"primary"`` — the
phase-133 fallback class, the strip IS the view — or ``"additive"`` —
the phase-140 pictorial class, text + a Text/Pages toggle) serves the
class VERBATIM beside the untouched ``page_urls`` list (the viewer's
branch signal, task 04); every other key stands where phase 148 left
it."""
_cleanup_kb(db)
try:
pages_dir = tmp_path / "pages"
_seed_page_doc(db, pages_dir, page_count=3, page_strip_mode=mode)
r = admin_client.get(
"/api/documents/content", params={"source": "PdfSrc", "path": "besta.pdf"}
)
assert r.status_code == 200
body = r.json()
assert body["page_strip_mode"] == mode
assert len(body["page_urls"]) == 3 # the strip rides along, unchanged
finally:
_cleanup_kb(db)
def test_content_strip_class_never_travels_without_the_strip(
admin_client: TestClient, db, tmp_path: Path
) -> None:
"""Phase 163 (task 01) — the two strip keys are ONE block keyed off
``page_count``: a row with a stray ``page_strip_mode`` and no strip
(``page_count`` NULL — the corner a partial repair could leave)
serializes NEITHER key, byte-identically to a plain non-qualifying
document (the viewer must never be handed a class for pages that do
not exist)."""
_cleanup_kb(db)
try:
pages_dir = tmp_path / "pages"
_seed_page_doc(
db,
pages_dir,
path="stray.pdf",
page_count=None,
write_copies=False,
page_strip_mode="additive",
)
r = admin_client.get(
"/api/documents/content", params={"source": "PdfSrc", "path": "stray.pdf"}
)
assert r.status_code == 200
body = r.json()
assert "page_strip_mode" not in body
assert "page_urls" not in body
# The exact pre-phase key set (the byte-identity pin).
assert set(body) == {
"source",
"path",
"title",
"format",
"summary",
"created_at",
"content",
"indexed_at",
"chunks",
"is_image",
}
finally: finally:
_cleanup_kb(db) _cleanup_kb(db)
+370
View File
@@ -0,0 +1,370 @@
"""Integration: migration 0027 (documents.page_strip_mode) schema contract
(phase 163, task 01).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0026.py`` (information_schema assertions on the state
the migration must leave). The tests target the 0026 → 0027 step
explicitly so later migrations cannot break the pins:
* upgrade 0026 → 0027 → ``page_strip_mode`` exists with the full
contract — VARCHAR(16), NULLABLE, NO server default (NULL = "no strip"
for every pre-phase row, every non-qualifying document and every
fail-soft miss) and UNINDEXED (a display fact, never a query surface —
the content endpoint reads it from the ROW, the ``file_hash`` /
``page_count`` convention) — while the 0026 ``documents`` schema
survives, in particular ``page_count`` keeping its 0026 contract and
``content_hash`` staying NOT NULL and INDEXED;
* rows inserted while the DB is at 0026 backfill ``page_strip_mode`` to
NULL — including a row whose ``page_count`` IS set (the pre-phase strip
with no stamped class: the wire omits the key and the viewer keeps
today's "primary" DOM until the task-03 re-gate stamps it);
* the ORM contract agrees: a freshly inserted ``Document`` with no mode
reads ``page_strip_mode is None`` through a fresh session, and each of
the two classes (``"primary"`` / ``"additive"``) round-trips verbatim;
* downgrade to 0026 → the column is GONE (A13) while the row + its
content + its ``page_count`` survive; upgrade back to 0027 → the column
is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
import hashlib
import uuid
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.orm import Session
from alembic import command
from app.db import SessionLocal, db_available
from app.models import Document
SOURCE = "mig0027"
CONTENT = "content here"
CONTENT_HASH = hashlib.sha256(CONTENT.encode()).hexdigest()
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown
upgrades to head no matter what happened, so the dev DB is never
left below head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
# on ``documents``, which would deadlock the repair's
# ``ALTER TABLE`` forever.
db.rollback()
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default, character_maximum_length)
for one documents column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default, character_maximum_length"
" FROM information_schema.columns"
" WHERE table_name = 'documents' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _index_covering(db: Session, column: str) -> str | None:
"""The name of any documents index whose definition mentions the
column (None = UNINDEXED)."""
return db.execute(
text(
"SELECT indexname FROM pg_indexes"
" WHERE tablename = 'documents' AND indexdef ILIKE :like"
),
{"like": f"%{column}%"},
).scalar()
def _insert_sql(db: Session, path: str, page_count: int | None = None) -> uuid.UUID:
"""Insert one documents row with the PRE-0027 column set (the 0026
shape — the mode column, when present, is omitted so a NULL backfill
is what the row reads; ``page_count`` is 0026's own column and may be
set to model a pre-phase strip)."""
doc_id = uuid.uuid4()
db.execute(
text(
"INSERT INTO documents (id, source, path, full_path, title, content,"
" content_hash, page_count)"
" VALUES (:id, :s, :p, :fp, :t, :c, :h, :pc)"
),
{
"id": doc_id,
"s": SOURCE,
"p": path,
"fp": f"/tmp/{path}",
"t": f"Doc {path}",
"c": CONTENT,
"h": CONTENT_HASH,
"pc": page_count,
},
)
db.commit()
return doc_id
def _delete(db: Session, doc_id: uuid.UUID) -> None:
db.execute(text("DELETE FROM documents WHERE id = :id"), {"id": doc_id})
db.commit()
def _assert_0026_schema_survives(db: Session) -> None:
"""The 0026 documents contract must survive the additive upgrade —
``page_count`` keeps its INTEGER / NULLABLE / NO-default / UNINDEXED
shape (the sibling column this one rides) and ``content_hash`` stays
NOT NULL and indexed (the retriever's identity)."""
page_count = _column(db, "page_count")
assert page_count is not None, "documents.page_count (0026) must survive the upgrade"
assert page_count[0] == "integer", "page_count must stay INTEGER"
assert page_count[1] == "YES", "page_count must stay NULLABLE"
assert page_count[2] is None, "page_count must keep NO server default"
assert _index_covering(db, "page_count") is None, "page_count must stay UNINDEXED"
content = _column(db, "content_hash")
assert content is not None and content[0] == "character varying" and content[1] == "NO", (
"documents.content_hash (0001) must keep its contract"
)
assert (
_index_covering(db, "content_hash") is not None
), "the ix_documents_content_hash index must survive the upgrade"
indexed = _column(db, "indexed_at")
assert indexed is not None and indexed[0] == "timestamp with time zone", (
"documents.indexed_at (0001) must survive the upgrade"
)
is_image = _column(db, "is_image")
assert is_image is not None and is_image[0] == "boolean" and is_image[1] == "NO", (
"documents.is_image (0022) must survive the upgrade"
)
def test_upgrade_to_0027_adds_page_strip_mode(db: Session, alembic: Config) -> None:
"""Upgrade 0026 → 0027: ``page_strip_mode`` exists with the full
contract (VARCHAR(16), NULLABLE, NO server default — NULL = "no
strip" — and UNINDEXED: a display fact, never a query surface), is
ABSENT at 0026, and every pre-0027 row backfills it to NULL —
including the row whose ``page_count`` IS set (the pre-phase strip
with no stamped class), while the 0026 table contract (the
``page_count`` sibling, the indexed ``content_hash``) survives.
"""
command.downgrade(alembic, "0026") # start from the pre-0027 state
assert _version(db) == "0026"
assert _column(db, "page_strip_mode") is None, "page_strip_mode must be absent at 0026"
plain_id = _insert_sql(db, "plain.pdf") # the 0026 column set, no strip
pre_strip_id = _insert_sql(db, "prephase.pdf", page_count=36) # a live pre-phase strip
try:
command.upgrade(alembic, "0027")
assert _version(db) == "0027", "alembic_version must be at 0027"
col = _column(db, "page_strip_mode")
assert col is not None, "documents.page_strip_mode is missing"
assert col[0] == "character varying", "page_strip_mode must be VARCHAR"
assert col[3] == 16, "page_strip_mode must be VARCHAR(16) — the two class names"
assert col[1] == "YES", (
"page_strip_mode must be NULLABLE (NULL = no strip, and the pre-phase shape)"
)
assert col[2] is None, (
"page_strip_mode must carry NO server default — NULL is the meaningful"
" pre-phase state"
)
assert _index_covering(db, "page_strip_mode") is None, (
"page_strip_mode must be UNINDEXED — a display fact, never a query"
" surface (the content endpoint reads it from the row)"
)
# Every pre-0027 row backfills to NULL — no backfill, no default.
assert (
db.execute(
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
{"id": plain_id},
).fetchone()
== (None, None)
), "a pre-0027 row without a strip must read both columns NULL"
assert (
db.execute(
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
{"id": pre_strip_id},
).fetchone()
== (36, None)
), (
"a pre-phase strip (page_count set) must backfill page_strip_mode to"
" NULL — the wire omits the key and the viewer keeps today's DOM"
)
_assert_0026_schema_survives(db)
finally:
_delete(db, plain_id)
_delete(db, pre_strip_id)
@pytest.mark.parametrize("mode", ["primary", "additive"])
def test_orm_page_strip_mode_round_trips(db: Session, alembic: Config, mode: str) -> None:
"""The ORM contract agrees with the column contract: each class
(``"primary"`` = the fallback class, ``"additive"`` = the pictorial
class) round-trips VERBATIM through a fresh session, and a row
created with no mode reads ``page_strip_mode is None`` (the
pre-phase shape — NULL until the importer stamps it)."""
command.upgrade(alembic, "head")
stamped = Document(
source=SOURCE,
path="stamped.pdf",
full_path="/tmp/stamped.pdf",
title="stamped",
content=CONTENT,
content_hash=CONTENT_HASH,
page_count=36,
page_strip_mode=mode,
)
unstamped = Document(
source=SOURCE,
path="unstamped.md",
full_path="/tmp/unstamped.md",
title="unstamped",
content=CONTENT,
content_hash=CONTENT_HASH,
)
db.add(stamped)
db.add(unstamped)
db.commit()
try:
with SessionLocal() as fresh:
reloaded = fresh.get(Document, stamped.id)
assert reloaded is not None, "the stamped row must be readable"
assert reloaded.page_strip_mode == mode, (
f"the {mode!r} class must round-trip through the DB verbatim"
)
assert reloaded.page_count == 36, "the sibling page_count must round-trip too"
reloaded_unstamped = fresh.get(Document, unstamped.id)
assert reloaded_unstamped is not None, "the unstamped row must be readable"
assert reloaded_unstamped.page_strip_mode is None, (
"a row created without a mode must read NULL (the pre-phase shape)"
)
assert reloaded_unstamped.content_hash == CONTENT_HASH, (
"content_hash must be untouched by the page_strip_mode addition"
)
finally:
_delete(db, stamped.id)
_delete(db, unstamped.id)
def test_downgrade_to_0026_drops_the_column(db: Session, alembic: Config) -> None:
"""Downgrade 0027 → 0026: the page_strip_mode column is gone (A13 —
fully reversible, the class is re-derivable from the source file by
the deterministic re-gate) while the row + its content + its 0026
``page_count`` survive, and the rest of the table contract is intact."""
command.upgrade(alembic, "head")
row = Document(
source=SOURCE,
path="down.pdf",
full_path="/tmp/down.pdf",
title="down",
content=CONTENT,
content_hash=CONTENT_HASH,
page_count=36,
page_strip_mode="additive",
)
db.add(row)
db.commit()
try:
command.downgrade(alembic, "0026")
assert _version(db) == "0026"
assert _column(db, "page_strip_mode") is None, "page_strip_mode must be dropped"
surviving = db.execute(
text(
"SELECT source, path, content, content_hash, page_count, is_image"
" FROM documents WHERE id = :id"
),
{"id": row.id},
).fetchone()
assert surviving is not None and surviving[0] == SOURCE, (
"the row must survive the column drop"
)
assert surviving[1] == "down.pdf" and surviving[2] == CONTENT, (
"path + content must survive the drop"
)
assert surviving[3] == CONTENT_HASH, (
"content_hash (the retriever's identity) must survive the drop"
)
assert surviving[4] == 36, (
"page_count (the 0026 sibling) must survive the drop — the strip"
" itself is untouched by this migration's reversal"
)
assert surviving[5] is False, "is_image must survive the drop"
assert _index_covering(db, "content_hash") is not None, (
"the content_hash index must survive the downgrade"
)
finally:
_delete(db, row.id)
# Repair: the fixture teardown re-upgrades to head.
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
"""Downgrade to 0026, then upgrade back to 0027: ``page_strip_mode``
is back with the full contract (VARCHAR(16), NULLABLE, NO server
default, UNINDEXED), a row seeded at 0026 — strip included — reads
the mode as NULL (the pre-phase shape on the round-trip), and the
0026 schema (the ``page_count`` sibling, the indexed
``content_hash``) is intact."""
command.downgrade(alembic, "0026")
pre_id = _insert_sql(db, "round.pdf", page_count=36)
try:
command.upgrade(alembic, "0027")
assert _version(db) == "0027", "round-trip upgrade must land at 0027"
col = _column(db, "page_strip_mode")
assert col is not None, "documents.page_strip_mode must be back"
assert col[0] == "character varying", "page_strip_mode must be VARCHAR after the round-trip"
assert col[3] == 16, "page_strip_mode must be VARCHAR(16) after the round-trip"
assert col[1] == "YES", "page_strip_mode must be NULLABLE after the round-trip"
assert col[2] is None, (
"page_strip_mode must still carry NO server default after the round-trip"
)
assert _index_covering(db, "page_strip_mode") is None, (
"page_strip_mode must stay UNINDEXED after the round-trip"
)
assert (
db.execute(
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
{"id": pre_id},
).fetchone()
== (36, None)
), (
"a row seeded at 0026 must read page_strip_mode NULL after the"
" round-trip while its page_count stands"
)
_assert_0026_schema_survives(db)
finally:
_delete(db, pre_id)
+112 -5
View File
@@ -30,6 +30,17 @@ page-qualifying PDF row (``page_count`` set) adds ``page_urls`` = the
exact 1-based ``/api/documents/<id>/pages/<i>`` list over exact 1-based ``/api/documents/<id>/pages/<i>`` list over
``1..page_count``. ``1..page_count``.
Phase 163 (task 01) — the content payload's STRIP CLASS signal
(``page_strip_mode``: ``"primary"`` = the fallback class, the strip IS
the view; ``"additive"`` = the pictorial class, text + a Text/Pages
toggle): the model-level omission rule (ABSENT when ``None``, never
``null``), and the endpoint (stubbed session) — the key rides
``page_urls`` for a qualifying row, a PRE-PHASE row (``page_count`` set,
mode NULL — the exact live ``Letumdei`` state) serializes it ABSENT while
keeping ``page_urls``, and the whole strip block keys off
``page_count``, so a row with a stray mode and no strip carries neither
key (every non-qualifying payload byte-identical to pre-phase).
Phase 145 (task 03) — the content payload's git origin fields + the Phase 145 (task 03) — the content payload's git origin fields + the
``renderDocument`` "Original" anchor (modal AND /document.html, one ``renderDocument`` "Original" anchor (modal AND /document.html, one
site): site):
@@ -316,10 +327,11 @@ def _text_doc_content(
origin_url: str | None = None, origin_url: str | None = None,
origin_branch: str | None = None, origin_branch: str | None = None,
page_urls: list[str] | None = None, page_urls: list[str] | None = None,
page_strip_mode: str | None = None,
) -> DocContent: ) -> DocContent:
"""One text doc's content payload (the pre-phase shape) with the """One text doc's content payload (the pre-phase shape) with the
phase-145 origin fields + the phase-148 ``page_urls`` optionally phase-145 origin fields + the phase-148 ``page_urls`` + the
set.""" phase-163 ``page_strip_mode`` optionally set."""
return DocContent( return DocContent(
source="S", source="S",
path="a.md", path="a.md",
@@ -332,6 +344,7 @@ def _text_doc_content(
origin_url=origin_url, origin_url=origin_url,
origin_branch=origin_branch, origin_branch=origin_branch,
page_urls=page_urls, page_urls=page_urls,
page_strip_mode=page_strip_mode,
) )
@@ -367,9 +380,29 @@ def test_doc_content_page_urls_absent_on_the_wire_when_unset() -> None:
assert dumped["page_urls"] == urls # the exact 1-based route list assert dumped["page_urls"] == urls # the exact 1-based route list
def _page_pdf_doc() -> Document: def test_doc_content_page_strip_mode_absent_on_the_wire_when_unset() -> None:
"""Phase 163 (task 01, LOCKED A2) — the model-level omission rule
(the ``image_url`` / ``page_urls`` precedent): no strip class
(``page_strip_mode`` ``None`` — every non-qualifying doc AND every
pre-phase row) serializes the key ABSENT — never ``null`` — so those
payloads are byte-identical to pre-phase. A stamped row carries the
class verbatim, beside ``page_urls``."""
dumped = _text_doc_content().model_dump()
assert "page_strip_mode" not in dumped # absent — never null
for mode in ("primary", "additive"):
dumped = _text_doc_content(
page_urls=["/api/documents/d/pages/1"], page_strip_mode=mode
).model_dump()
assert dumped["page_strip_mode"] == mode # the class verbatim
def _page_pdf_doc(
page_count: int | None = 2, page_strip_mode: str | None = None
) -> Document:
"""A page-QUALIFYING PDF row (phase 148): ``page_count`` set — the """A page-QUALIFYING PDF row (phase 148): ``page_count`` set — the
endpoint derives ``page_urls`` from it (the exact 1-based list).""" endpoint derives ``page_urls`` from it (the exact 1-based list).
``page_strip_mode`` (phase 163) is the strip CLASS; ``None`` is the
PRE-PHASE row shape (the column NULL while the strip stands)."""
doc = Document( doc = Document(
id=uuid.uuid4(), id=uuid.uuid4(),
source="Homelab", source="Homelab",
@@ -379,7 +412,8 @@ def _page_pdf_doc() -> Document:
content="[Page 1 illustration: the parts tray]…", content="[Page 1 illustration: the parts tray]…",
content_hash="e" * 64, content_hash="e" * 64,
is_image=False, # phase 122: the stub session never applies defaults is_image=False, # phase 122: the stub session never applies defaults
page_count=2, page_count=page_count,
page_strip_mode=page_strip_mode,
) )
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC) doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC) doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC)
@@ -410,6 +444,79 @@ database): a page-qualifying PDF row (``page_count`` set) carries
] # the exact 1-based list, count == page_count ] # the exact 1-based list, count == page_count
assert body["is_image"] is False assert body["is_image"] is False
assert "image_url" not in body # absent — never null (not an image doc) assert "image_url" not in body # absent — never null (not an image doc)
# Phase 163 (task 01): the pre-phase row (mode NULL) keeps the strip
# and serializes the class ABSENT — the viewer treats that as
# "primary", i.e. today's DOM.
assert "page_strip_mode" not in body
@pytest.mark.parametrize("mode", ["primary", "additive"])
def test_content_page_strip_mode_carried_for_a_stamped_strip(mode: str) -> None:
"""Phase 163 (task 01, LOCKED A2/A3) — the endpoint (stubbed
session, no
database): a page-qualifying PDF whose row carries the class
(``"primary"`` = the fallback class, ``"additive"`` = the pictorial
class) carries ``page_strip_mode`` VERBATIM next to ``page_urls`` —
the signal the viewer branches on so an additive strip never hides
the text (task 04). Nothing else in the payload moves."""
doc = _page_pdf_doc(page_strip_mode=mode)
with _client_with_row((doc, 4)) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "ikea/besta.pdf"},
)
assert r.status_code == 200
body = r.json()
assert body["page_strip_mode"] == mode # the class verbatim
assert body["page_urls"] == [
f"/api/documents/{doc.id}/pages/1",
f"/api/documents/{doc.id}/pages/2",
] # the strip rides along, unchanged
def test_content_strip_block_keys_off_page_count_never_mode_alone() -> None:
"""Phase 163 (task 01) — the class never appears alone: the whole
strip block keys off ``page_count``, so a row whose column somehow
holds a mode while ``page_count`` is NULL serializes NEITHER
``page_strip_mode`` nor ``page_urls`` — the payload is byte-identical
to a plain non-qualifying document (there is no strip to show, and
the viewer must never be told a class for pages that do not exist)."""
doc = _page_pdf_doc(page_count=None, page_strip_mode="additive")
with _client_with_row((doc, 1)) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "ikea/besta.pdf"},
)
assert r.status_code == 200
body = r.json()
assert "page_strip_mode" not in body # absent — the strip block is off
assert "page_urls" not in body # absent — never null
def test_content_non_qualifying_payload_key_set_is_byte_identical() -> None:
"""Phase 163 (task 01) — the byte-identity pin: a non-qualifying PDF
row (``page_count`` NULL, mode NULL) carries EXACTLY the pre-phase
key set — neither strip key joins it (``is_image`` always present,
``image_url`` / the origin keys / both strip keys absent)."""
doc = _page_pdf_doc(page_count=None)
with _client_with_row((doc, 1)) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "ikea/besta.pdf"},
)
assert r.status_code == 200
assert set(r.json()) == {
"source",
"path",
"title",
"format",
"summary",
"created_at",
"content",
"indexed_at",
"chunks",
"is_image",
}
def test_content_origin_url_carried_for_a_matching_git_row() -> None: def test_content_origin_url_carried_for_a_matching_git_row() -> None:
@@ -31,6 +31,16 @@ the second sync):
* the streamed hash is transparent (a >1 MiB file hashes identically * the streamed hash is transparent (a >1 MiB file hashes identically
to ``hashlib.sha256(bytes).hexdigest()``). to ``hashlib.sha256(bytes).hexdigest()``).
Phase 163 (task 03) extends this file with the PDF strip RE-GATE the
unchanged paths now run (``_regate_pdf_strip``) — the pre-phase "the
copies persist" assumption is what broke in production (a row claiming
``page_count = 36`` over ZERO files, every page 404ing forever): the
re-gate re-derives the strip from the current bytes + settings (ZERO
vision/LLM calls), heals missing copies, re-stamps a changed class,
clears a strip that no longer qualifies (kill switch included), stays
fail-soft, logs only when it ACTS, and costs a non-strip file zero PDF
parsing (spy-pinned).
The E2E surface (the idempotent second sync through the real UI + The E2E surface (the idempotent second sync through the real UI +
mock LLM) is pinned in ``tests/e2e/test_sync_changed_only.py`` mock LLM) is pinned in ``tests/e2e/test_sync_changed_only.py``
(task 03). (task 03).
@@ -43,6 +53,7 @@ import logging
import os import os
from datetime import UTC, datetime, timedelta from datetime import UTC, datetime, timedelta
from pathlib import Path from pathlib import Path
from typing import Any
import pymupdf as fitz import pymupdf as fitz
import pytest import pytest
@@ -50,6 +61,7 @@ from sqlalchemy import select
from app.config import Settings from app.config import Settings
from app.models import Document from app.models import Document
from app.rag import importer
from app.rag.importer import _file_hash, import_sources from app.rag.importer import _file_hash, import_sources
from tests.fakes import FakeEmbedder from tests.fakes import FakeEmbedder
from tests.unit.test_importer import ( from tests.unit.test_importer import (
@@ -60,6 +72,7 @@ from tests.unit.test_importer import (
_VisionLLM, _VisionLLM,
) )
from tests.unit.test_pictorial_import import ( from tests.unit.test_pictorial_import import (
_build_dense_pdf,
_build_ikea_like_pdf, _build_ikea_like_pdf,
_PictorialLLM, _PictorialLLM,
_settings, _settings,
@@ -454,6 +467,520 @@ def test_pre_phase_null_file_hash_image_runs_branch_and_stamps(
_cleanup_source(db, root.name) _cleanup_source(db, root.name)
# ---------- phase 163 (task 03): the deterministic PDF strip re-gate ----------
#
# The live defect (owner report, 2026-10-07): the `Letumdei` row carried
# `page_count = 36` over ZERO files on disk (the pre-phase-163 persist
# deleted the old copies BEFORE rendering, outside the DB transaction —
# a later per-file rollback restored the stale count over an empty
# directory; an external wipe of `pdf_page_dir` lands the same state).
# The phase-147 fast path believed "the copies persist" and never
# re-rendered, so the strip 404'd forever. `_regate_pdf_strip` now runs
# at the tail of `_unchanged_doc`: the changed-path gate mirrored from
# the current bytes + settings (ZERO vision/LLM calls), missing copies
# healed, a stale strip cleared.
#: The PNG signature (a healed copy is a real render, not a placeholder).
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
def _regate_settings(tmp_path: Path, **kwargs: Any) -> Settings:
"""Settings for the re-gate matrix: the ``pdf`` token admitted, the
phase-140 knobs at their CODE DEFAULTS (``pdf_pictorial`` ON), and
the strip's home ``pdf_page_dir`` pinned under *tmp_path* (never the
real home)."""
kwargs.setdefault("_env_file", None)
kwargs.setdefault("import_extensions", "md,txt,pdf")
kwargs.setdefault("images", False)
kwargs.setdefault("image_dir", str(tmp_path / "images"))
kwargs.setdefault("pdf_page_dir", str(tmp_path / "pages"))
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
def _pages_dir(tmp_path: Path) -> Path:
return tmp_path / "pages"
def _strip_copies(doc: Document, page_dir: Path, count: int) -> list[Path]:
return [page_dir / f"{doc.id}-p{n}.png" for n in range(1, count + 1)]
def test_regate_heals_deleted_copies_of_a_pictorial_strip(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""THE live-loss reproduction (heal arm): a pictorial-class PDF is
indexed (``page_count`` set, copies written), its copies are deleted
out from under the row (the exact live state — mode NULL on a
pre-phase row), and an UNCHANGED re-sync restores every file with
real PNG bytes, stamps the mode ``additive``, keeps ``page_count``
(no drift), burns ZERO llm/vision calls, and logs the one greppable
INFO line. The NULL-mode stamp is SILENT (no ``mode`` segment — it
rides the line only when work followed)."""
root = tmp_path / "rgheal"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgheal", "ikea.pdf")
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
page_dir = _pages_dir(tmp_path)
doc.page_strip_mode = None # the pre-phase row (the live shape)
db.commit()
for path in _strip_copies(doc, page_dir, 10):
path.unlink()
assert not list(page_dir.glob(f"{doc.id}-*.png"))
embeds1, chats1, vision1 = len(llm.calls), len(llm.chat_calls), llm.pictorial_calls
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
# ZERO llm work of any kind (the re-gate is deterministic, A5).
assert len(llm.calls) == embeds1, "the re-gate never re-embeds"
assert len(llm.chat_calls) == chats1, "the re-gate never chats"
assert llm.pictorial_calls == vision1, "the re-gate issues ZERO vision calls"
db.expire_all()
doc2 = _doc(db, "rgheal", "ikea.pdf")
assert doc2.page_count == 10, "no drift — the count is untouched"
assert doc2.page_strip_mode == "additive", "the mode is stamped"
for path in _strip_copies(doc2, page_dir, 10):
assert path.is_file(), f"copy {path.name} was not healed"
assert path.read_bytes()[:8] == PNG_MAGIC, f"{path.name} is not a PNG"
assert not list(page_dir.glob("*.tmp")), "no .tmp orphan from the heal"
messages = [r.getMessage() for r in caplog.records]
assert (
"import: PDF strip re-gate source=rgheal path=ikea.pdf — "
"healed 10 pages"
) in messages, messages
assert not any("mode " in m for m in messages if "re-gate" in m), (
"a NULL->stamp mode re-stamp is silent-beneficial " + str(messages)
)
finally:
_cleanup_source(db, "rgheal")
def test_regate_heals_a_fallback_strip_as_primary(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""Heal arm, the FALLBACK class (phase 133's unusable text layer):
the copies of a garbage-layer PDF's strip are restored on an
unchanged re-sync and the mode lands ``primary`` (the strip IS the
view) — and the phase-133 fallback vision does NOT re-fire (the
class is re-derived from the bytes, no vision call)."""
root = tmp_path / "rgfall"
root.mkdir()
_build_garbage_pdf(root / "garbage.pdf") # 2 pages
llm = _PageVisionLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgfall", "garbage.pdf")
assert (doc.page_count, doc.page_strip_mode) == (2, "primary")
page_dir = _pages_dir(tmp_path)
for path in _strip_copies(doc, page_dir, 2):
path.unlink()
vision1, chats1, embeds1 = llm.vision_calls, len(llm.chat_calls), len(llm.calls)
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
assert second.pdfs_screenshotted == 0, "the fallback gate did not re-run"
assert llm.vision_calls == vision1, "ZERO vision calls on the re-gate"
assert len(llm.chat_calls) == chats1 and len(llm.calls) == embeds1
db.expire_all()
doc2 = _doc(db, "rgfall", "garbage.pdf")
assert doc2.page_strip_mode == "primary", "the unusable layer is primary"
assert doc2.page_count == 2
for path in _strip_copies(doc2, page_dir, 2):
assert path.is_file() and path.read_bytes()[:8] == PNG_MAGIC
assert any(
"healed 2 pages" in m and "re-gate" in m for m in caplog.messages
), caplog.messages
finally:
_cleanup_source(db, "rgfall")
def test_regate_noop_when_the_strip_is_healthy(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""No-op arm (the IKEA-class doc whose copies all stand): an
unchanged re-sync writes NOTHING (every copy's mtime and bytes are
untouched), renders NOTHING, and logs NO re-gate line — the INFO is
never per-unchanged-doc — while the pre-existing ``import:
unchanged`` line still fires."""
root = tmp_path / "rgnoop"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgnoop", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
before = {
p: (p.stat().st_mtime_ns, p.read_bytes())
for p in _strip_copies(doc, page_dir, 10)
}
assert len(before) == 10
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
db.expire_all()
doc2 = _doc(db, "rgnoop", "ikea.pdf")
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
after = {
p: (p.stat().st_mtime_ns, p.read_bytes())
for p in _strip_copies(doc2, page_dir, 10)
}
assert after == before, "a healthy strip is not written at all"
messages = [r.getMessage() for r in caplog.records]
assert any(m.startswith("import: unchanged ") for m in messages), messages
assert not any("re-gate" in m for m in messages), (
f"the re-gate logged without acting: {messages}"
)
finally:
_cleanup_source(db, "rgnoop")
def test_regate_restamps_a_changed_mode(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""Mode re-stamp (the class flip): a row whose stamped mode no
longer matches the gate's class (here a pictorial-class row carrying
``primary`` — a legacy/mis-stamped row) is re-stamped to
``additive``, and because the mode was ALREADY SET the change is
reported (``mode primary->additive``) — the only thing the re-gate
did, so that is the whole INFO line (no copies were missing, none
were written)."""
root = tmp_path / "rgmode"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgmode", "ikea.pdf")
assert doc.page_strip_mode == "additive"
doc.page_strip_mode = "primary" # the mis-stamp the re-gate fixes
db.commit()
page_dir = _pages_dir(tmp_path)
before = {
p: p.stat().st_mtime_ns for p in _strip_copies(doc, page_dir, 10)
}
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.unchanged) == (0, 1)
db.expire_all()
doc2 = _doc(db, "rgmode", "ikea.pdf")
assert doc2.page_strip_mode == "additive", "the class is re-stamped"
assert doc2.page_count == 10
assert {
p: p.stat().st_mtime_ns for p in _strip_copies(doc2, page_dir, 10)
} == before, "a mode fix writes no files"
assert (
"import: PDF strip re-gate source=rgmode path=ikea.pdf — "
"mode primary->additive"
) in [r.getMessage() for r in caplog.records], caplog.messages
finally:
_cleanup_source(db, "rgmode")
def test_regate_clears_a_strip_that_no_longer_qualifies(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""REPAIR arm: a row claims a strip (``page_count`` set, copies on
disk) on a PDF whose GATE now says ineligible (a clean, dense text
layer — zero sparse pages). The unchanged re-sync clears BOTH
columns and deletes the copies (the strip must not linger over a
document the viewer shows as text) + the ``cleared stale strip``
INFO line, with zero llm calls."""
root = tmp_path / "rgclear"
root.mkdir()
pdf = root / "clean.pdf"
_build_dense_pdf(pdf) # usable layer, zero candidates — never qualifies
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgclear", "clean.pdf")
assert (doc.page_count, doc.page_strip_mode) == (None, None)
# The stale state the re-gate must repair (a strip stamped by an
# older gate/settings, copies still on disk).
page_dir = _pages_dir(tmp_path)
page_dir.mkdir(parents=True, exist_ok=True)
for path in _strip_copies(doc, page_dir, 2):
path.write_bytes(b"STALE-COPY")
doc.page_count = 2
doc.page_strip_mode = "additive"
db.commit()
embeds1, chats1 = len(llm.calls), len(llm.chat_calls)
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
assert len(llm.calls) == embeds1 and len(llm.chat_calls) == chats1
db.expire_all()
doc2 = _doc(db, "rgclear", "clean.pdf")
assert doc2.page_count is None, "the stale strip is cleared"
assert doc2.page_strip_mode is None, "the class is cleared with it"
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies are deleted"
assert (
"import: PDF strip re-gate source=rgclear path=clean.pdf — "
"cleared stale strip"
) in [r.getMessage() for r in caplog.records], caplog.messages
finally:
_cleanup_source(db, "rgclear")
def test_regate_kill_switch_clears_the_pictorial_strip(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""REPAIR arm via the kill switch (``pdf_pictorial=False``): the
changed-path semantics mirrored — with the pass off a sparse-page
PDF is NOT eligible, so its existing strip is cleared with its
copies on the next (unchanged) sync. The bytes never moved: the
setting alone decided."""
root = tmp_path / "rgkill"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgkill", "ikea.pdf")
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
page_dir = _pages_dir(tmp_path)
assert len(list(page_dir.glob(f"{doc.id}-p*.png"))) == 10
vision1 = llm.pictorial_calls
llm.settings = _regate_settings(tmp_path, pdf_pictorial=False)
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
assert llm.pictorial_calls == vision1, "ZERO vision calls (the pass is off)"
db.expire_all()
doc2 = _doc(db, "rgkill", "ikea.pdf")
assert doc2.page_count is None and doc2.page_strip_mode is None
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies went too"
assert any(
"cleared stale strip" in m for m in caplog.messages
), caplog.messages
finally:
_cleanup_source(db, "rgkill")
def test_regate_render_exception_is_fail_soft(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Fail-soft: the heal's render RAISES — the sync completes
(``unchanged``, zero errors), the row keeps its prior state
(``page_count`` + mode untouched — the re-gate never clears a strip
on a failure), no copies are touched, and the WARNING is the only
signal."""
root = tmp_path / "rgboom"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgboom", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
for path in _strip_copies(doc, page_dir, 10):
path.unlink()
def boom(*args: object, **kwargs: object) -> list[bytes]:
raise RuntimeError("simulated render failure (test sentinel)")
monkeypatch.setattr(importer, "render_pdf_pages", boom)
with caplog.at_level(logging.WARNING, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged, second.errors) == (
0, 0, 1, 0,
)
db.expire_all()
doc2 = _doc(db, "rgboom", "ikea.pdf")
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive"), (
"the row keeps its pre-re-gate state"
)
assert not list(page_dir.glob(f"{doc.id}-p*.png")), "no copies were touched"
assert any(
"PDF strip re-gate failed" in m and "test sentinel" in m
for m in caplog.messages
), caplog.messages
finally:
_cleanup_source(db, "rgboom")
def test_regate_empty_render_keeps_the_row(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Fail-soft on the no-pages render (unopenable right after the fast
path hashed the file — pathological): the row is kept EXACTLY as it
was (never cleared on a transient), nothing is written, and the
WARNING names the no-pages state."""
root = tmp_path / "rgempty"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgempty", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
for path in _strip_copies(doc, page_dir, 10):
path.unlink()
monkeypatch.setattr(importer, "render_pdf_pages", lambda *a, **k: [])
with caplog.at_level(logging.WARNING, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.unchanged, second.errors) == (0, 1, 0)
db.expire_all()
doc2 = _doc(db, "rgempty", "ikea.pdf")
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
assert not page_dir.exists() or not list(page_dir.glob(f"{doc.id}-*"))
assert any(
"re-render produced no pages (state kept)" in m
for m in caplog.messages
), caplog.messages
finally:
_cleanup_source(db, "rgempty")
def test_regate_reports_a_page_count_drift_on_the_heal(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Drift (pathological on unchanged bytes, so it is LOUD): the heal
renders 6 pages for a row that claimed 10 — ``page_count`` follows
the fresh render, the ``p7..p10`` tail beyond it is deleted, and a
WARNING names both counts beside the ``healed`` INFO line."""
root = tmp_path / "rgdrift"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgdrift", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
for path in _strip_copies(doc, page_dir, 10):
path.unlink()
monkeypatch.setattr(
importer,
"render_pdf_pages",
lambda *a, **k: [PNG_MAGIC + f"{i}".encode() for i in range(6)],
)
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.unchanged) == (0, 1)
db.expire_all()
doc2 = _doc(db, "rgdrift", "ikea.pdf")
assert doc2.page_count == 6, "the count follows the fresh render"
for n in range(1, 7):
assert (page_dir / f"{doc2.id}-p{n}.png").is_file()
for n in range(7, 11):
assert not (page_dir / f"{doc2.id}-p{n}.png").exists(), f"p{n} leaked"
messages = [r.getMessage() for r in caplog.records]
assert any("page_count drift 10->6" in m for m in messages), messages
assert any("healed 10 pages" in m for m in messages), messages
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
_cleanup_source(db, "rgdrift")
def test_regate_fires_on_the_content_digest_hit_path(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The SECOND unchanged path (the PDF branch's content-digest hit —
a pre-phase row with ``file_hash`` NULL): the branch re-runs its own
gate (the byte-change's honest cost) and lands unchanged, and the
shared helper's tail still heals the strip deleted out from under
the row — the re-gate is not fast-path-only."""
root = tmp_path / "rgbranch"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
try:
asyncio.run(import_sources([root], llm, session=db))
doc = _doc(db, "rgbranch", "ikea.pdf")
doc.file_hash = None # the pre-phase row: the fast path cannot fire
db.commit()
page_dir = _pages_dir(tmp_path)
for path in _strip_copies(doc, page_dir, 10):
path.unlink()
with caplog.at_level(logging.INFO, logger="app.importer"):
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
db.expire_all()
doc2 = _doc(db, "rgbranch", "ikea.pdf")
assert doc2.file_hash is not None, "the helper stamped the hash"
for path in _strip_copies(doc2, page_dir, 10):
assert path.is_file(), f"copy {path.name} was not healed"
assert any("healed 10 pages" in m for m in caplog.messages), caplog.messages
finally:
_cleanup_source(db, "rgbranch")
def test_regate_guard_costs_a_non_strip_file_nothing(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The GUARD (A5): a non-PDF unchanged file and a PDF whose
``page_count`` is NULL reach the tail of ``_unchanged_doc`` and
``extract_pdf_text`` is NEVER called for either — every non-strip
file's fast path keeps its zero-extra-read contract (one sync of
each class, then the spy covers the second sync only)."""
root = tmp_path / "rgguard"
root.mkdir()
(root / "note.md").write_text("# Note\n\nplain text body\n")
_build_dense_pdf(root / "clean.pdf") # usable layer, no strip
llm = _PictorialLLM()
llm.settings = _regate_settings(tmp_path)
seen: list[str] = []
real_extract = importer.extract_pdf_text
def spy(path: Path) -> str:
seen.append(str(path))
return real_extract(path)
try:
asyncio.run(import_sources([root], llm, session=db))
monkeypatch.setattr(importer, "extract_pdf_text", spy)
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 2)
assert seen == [], f"the guard let a PDF parse through: {seen}"
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
_cleanup_source(db, "rgguard")
# ---------- the preserved semantics on the fast path -------------------------- # ---------- the preserved semantics on the fast path --------------------------
+76
View File
@@ -293,6 +293,82 @@ def test_git_source_python_default_include_hidden_false() -> None:
assert col.default.arg is False, "the default must resolve to False" assert col.default.arg is False, "the default must resolve to False"
def test_documents_page_strip_mode_column_contract() -> None:
"""Phase 163 (task 01): ``documents.page_strip_mode`` — the CLASS of
the page strip a non-NULL ``page_count`` describes (``"primary"`` =
the phase-133 fallback class, the strip IS the view; ``"additive"``
= the phase-140 pictorial class, text + a Text/Pages toggle):
VARCHAR(16), NULLABLE (NULL = no strip — and the pre-phase shape for
a row whose strip stands), NO server default, and NO index (the
content endpoint reads it from the ROW — a display fact, never a
query surface, the ``file_hash`` / ``page_count`` convention).
``page_count`` (0026) stays exactly as phase 148 left it."""
from sqlalchemy import String as SAString
documents = Base.metadata.tables["documents"]
assert "page_strip_mode" in documents.c, (
"documents must have the page_strip_mode column (phase 163)"
)
col = documents.c["page_strip_mode"]
assert col.nullable is True, (
"page_strip_mode must be NULLABLE — NULL means no strip (and the"
" pre-phase row whose strip stands)"
)
assert isinstance(col.type, SAString), "page_strip_mode must be a String column"
assert col.type.length == 16, (
"page_strip_mode must be String(16) — the two class names, bounded"
)
assert col.server_default is None, (
"page_strip_mode must carry NO server default — NULL is the"
" meaningful pre-phase state"
)
assert not col.index, (
"page_strip_mode must be UNINDEXED — the content endpoint reads it"
" from the row, no query filters on it"
)
covering = [
idx
for idx in documents.indexes
if any(c.name == "page_strip_mode" for c in idx.columns)
]
assert not covering, "no table-level index may cover page_strip_mode"
# page_count is untouched: still the nullable, unindexed sibling.
page_count = documents.c["page_count"]
assert page_count.nullable is True, "page_count must stay NULLABLE (0026)"
assert not page_count.index, "page_count must stay UNINDEXED (0026)"
def test_document_page_strip_mode_orm_defaults_to_none() -> None:
"""A freshly constructed ``Document`` (no DB) exposes
``page_strip_mode``: an omitted kwarg and an explicit ``None`` both
read NULL (no strip — the shape every non-qualifying document keeps,
and the shape a pre-phase-163 row keeps WHILE its ``page_count``
stands, which the wire serializes ABSENT), and each class name
round-trips verbatim as a plain string (the real DB round-trip is
pinned in ``test_migration_0027.py``)."""
from app.models import Document
base = dict(
source="s",
path="p.pdf",
full_path="/tmp/s/p.pdf",
title="p",
content="c",
content_hash="1" * 64,
page_count=36,
)
for mode in ("primary", "additive"):
assert Document(**base, page_strip_mode=mode).page_strip_mode == mode
assert Document(**base, page_strip_mode=None).page_strip_mode is None
omitted = Document(**base)
assert omitted.page_strip_mode is None, (
"an omitted page_strip_mode must read NULL while page_count stands"
" (the pre-phase strip the viewer keeps showing as 'primary')"
)
def test_doc_drafts_column_contract() -> None: def test_doc_drafts_column_contract() -> None:
"""Phase 59: the editable triple (title/path/body) + status + """Phase 59: the editable triple (title/path/body) + status +
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
+312 -18
View File
@@ -21,14 +21,20 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
layer, ≥1 sparse page) and the fallback class (unusable layer) get layer, ≥1 sparse page) and the fallback class (unusable layer) get
``<doc-id>-p{N}.png`` for EVERY page (not just the described ``<doc-id>-p{N}.png`` for EVERY page (not just the described
candidates — the strip shows the whole manual) + ``page_count`` = candidates — the strip shows the whole manual) + ``page_count`` =
the page count + real PNG bytes; a text-heavy PDF gets neither the page count + real PNG bytes + the phase-163 CLASS stamp
(no copies, NULL count, ZERO render calls, byte-identical content); (``page_strip_mode`` ``additive`` / ``primary``); a text-heavy PDF
gets neither (no copies, NULL count, NULL mode, ZERO render calls,
byte-identical content);
* the phase-147 fast path — an unchanged qualifying PDF's second sync * the phase-147 fast path — an unchanged qualifying PDF's second sync
re-renders NOTHING (zero render calls — the copies persist, re-renders NOTHING (zero render calls — the copies persist,
``page_count`` and the bytes untouched); ``page_count`` and the bytes untouched);
* stale replacement — a changed (shrunk) PDF's old copy set is * WRITE-FIRST replacement (phase 163, task 02, LOCKED A4) — a changed
replaced: the new set covers the new pages, the old ``p{N}`` tail (shrunk) PDF's new copies land atomically (tmp + ``os.replace``,
is GONE (no orphans), ``page_count`` follows the new count; pinned) BEFORE any old copy is removed; the old ``p{N}`` tail is
GONE only after the replacement bytes are durable (no orphans),
``page_count`` follows the new count; a MID-WRITE failure leaves the
prior copy set intact (the rollback-orphan state — the row over
zero files — is structurally unrepresentable);
* qualifier loss — a qualifying PDF whose new version gained a dense * qualifier loss — a qualifying PDF whose new version gained a dense
text layer loses its strip: ``page_count`` NULL + copies deleted; text layer loses its strip: ``page_count`` NULL + copies deleted;
* unopenable — a render that yields no pages (``render_pdf_pages`` → * unopenable — a render that yields no pages (``render_pdf_pages`` →
@@ -40,9 +46,10 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
the FULL count (the gap is a display-level 404 — task 02 pins the the FULL count (the gap is a display-level 404 — task 02 pins the
route's side); route's side);
* fail-soft — the render raising (warning is the signal, ``page_count`` * fail-soft — the render raising (warning is the signal, ``page_count``
NULL, no copies, the doc indexed fully — chunks + summary) and the + ``page_strip_mode`` NULL, no copies, the doc indexed fully — chunks
WRITER raising (the copies this call wrote are removed — no partial + summary) and the WRITER raising (this call's ``.tmp`` files and the
orphans, ``page_count`` NULL, the doc indexed fully); finals it CREATED are removed — no partial orphans, ``page_count`` +
``page_strip_mode`` NULL, the doc indexed fully);
* prune — a qualifying doc's file removed from the source: the row * prune — a qualifying doc's file removed from the source: the row
goes and its page copies go with it (no orphans in the tmp dir). goes and its page copies go with it (no orphans in the tmp dir).
@@ -54,6 +61,7 @@ from __future__ import annotations
import asyncio import asyncio
import logging import logging
import os
import uuid import uuid
from collections.abc import Callable, Sequence from collections.abc import Callable, Sequence
from pathlib import Path from pathlib import Path
@@ -161,6 +169,35 @@ def _render_spy(
return calls return calls
# ---------------------------------------------------------------------------
# Phase 163 (task 01) — the strip-class column beside page_count
# ---------------------------------------------------------------------------
def test_pictorial_index_stamps_the_additive_class(db, tmp_path: Path) -> None:
"""Phase 163 (task 02, re-cut of the task-01 pin) — the STAMP: a
qualifying PDF indexed through the PICTORIAL class gets
``page_strip_mode == "additive"`` beside ``page_count`` (the text +
toggle view); the strip's copies themselves are unchanged by the
phase (task 01's shape, mode stamped exactly with the count).
"""
root = tmp_path / "pgm"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
doc = _doc(db, "pgm", "ikea.pdf")
assert doc.page_count == 10, "the strip itself is unchanged by task 02"
assert doc.page_strip_mode == "additive", (
"the pictorial class stamps the additive mode (LOCKED A1/A3)"
)
finally:
_cleanup_source(db, "pgm")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Eligibility × new index # Eligibility × new index
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -219,6 +256,10 @@ def test_fallback_class_garbage_pdf_writes_every_page_copy(db, tmp_path: Path) -
assert summary.pdfs_screenshotted == 1 # the fallback fired (the class) assert summary.pdfs_screenshotted == 1 # the fallback fired (the class)
doc = _doc(db, "pgf", "garbage.pdf") doc = _doc(db, "pgf", "garbage.pdf")
assert doc.page_count == 2 assert doc.page_count == 2
assert doc.page_strip_mode == "primary", (
"the fallback class stamps the primary mode — the strip IS the "
"view, byte-identical to phase 148 otherwise (LOCKED A1)"
)
page_dir = _pages_dir(tmp_path) page_dir = _pages_dir(tmp_path)
for n in (1, 2): for n in (1, 2):
path = page_dir / f"{doc.id}-p{n}.png" path = page_dir / f"{doc.id}-p{n}.png"
@@ -253,6 +294,7 @@ def test_text_heavy_pdf_keeps_the_text_view(
assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops
doc = _doc(db, "pgd", "clean.pdf") doc = _doc(db, "pgd", "clean.pdf")
assert doc.page_count is None, "a text-heavy PDF keeps the text view" assert doc.page_count is None, "a text-heavy PDF keeps the text view"
assert doc.page_strip_mode is None, "no strip — no class either"
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created" assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
assert calls == [], "zero render calls (no fallback, zero candidates)" assert calls == [], "zero render calls (no fallback, zero candidates)"
assert doc.content == expected, "byte-identical to the extraction" assert doc.content == expected, "byte-identical to the extraction"
@@ -315,9 +357,10 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
) -> None: ) -> None:
"""A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index """A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index
replaces the copy set — the new set covers p1..p6, the old replaces the copy set — the new set covers p1..p6, the old
version's ``p7..p10`` tail is GONE (the stale-replacement runs version's ``p7..p10`` tail is GONE (the stale-tail delete runs
before the write), the doc keeps its id, and ``page_count`` LAST, after the write-first loop — phase 163, task 02), the doc
follows the new page count.""" keeps its id, ``page_count`` follows the new page count, and
``page_strip_mode`` is (re-)stamped."""
root = tmp_path / "pgc" root = tmp_path / "pgc"
root.mkdir() root.mkdir()
pdf = root / "ikea.pdf" pdf = root / "ikea.pdf"
@@ -336,6 +379,7 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
doc2 = _doc(db, "pgc", "ikea.pdf") doc2 = _doc(db, "pgc", "ikea.pdf")
assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)" assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)"
assert doc2.page_count == 6, "page_count follows the new page count" assert doc2.page_count == 6, "page_count follows the new page count"
assert doc2.page_strip_mode == "additive", "the class is (re-)stamped"
page_dir = _pages_dir(tmp_path) page_dir = _pages_dir(tmp_path)
for n in range(1, 7): for n in range(1, 7):
assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing" assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing"
@@ -379,6 +423,7 @@ def test_qualifier_loss_clears_page_count_and_copies(
doc2 = _doc(db, "pgl", "ikea.pdf") doc2 = _doc(db, "pgl", "ikea.pdf")
assert doc2.id == doc1.id assert doc2.id == doc1.id
assert doc2.page_count is None, "a doc that stopped qualifying loses its strip" assert doc2.page_count is None, "a doc that stopped qualifying loses its strip"
assert doc2.page_strip_mode is None, "the class is cleared with the count"
assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), ( assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), (
"the copies must be deleted on qualifier loss" "the copies must be deleted on qualifier loss"
) )
@@ -411,6 +456,7 @@ def test_unopenable_pdf_page_render_is_fail_soft(
assert (summary.files, summary.added, summary.errors) == (1, 1, 0) assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
doc = _doc(db, "pgn", "garbage.pdf") doc = _doc(db, "pgn", "garbage.pdf")
assert doc.page_count is None, "no pages rendered — no strip" assert doc.page_count is None, "no pages rendered — no strip"
assert doc.page_strip_mode is None, "the class is cleared with the count"
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created" assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
assert doc.content == "", "the fallback's all-failed render lands empty content" assert doc.content == "", "the fallback's all-failed render lands empty content"
assert summary.summaries == 1, "the document is indexed exactly as pre-phase" assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
@@ -493,10 +539,11 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
monkeypatch: pytest.MonkeyPatch, monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture, caplog: pytest.LogCaptureFixture,
) -> None: ) -> None:
"""The PNG WRITER raises (a full disk) after the FIRST copy """The PNG WRITER raises (a full disk) after the FIRST atomic copy
landed: the copies this call wrote are removed (no partial landed: this call's ``.tmp`` files and the final-name files it
orphans), ``page_count`` NULL, the warning is the signal, and the CREATED are removed (no partial orphans — phase 163, task 02),
document indexes exactly as pre-phase.""" ``page_count`` AND ``page_strip_mode`` NULL, the warning is the
signal, and the document indexes exactly as pre-phase."""
root = tmp_path / "pgw" root = tmp_path / "pgw"
root.mkdir() root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf") _build_ikea_like_pdf(root / "ikea.pdf")
@@ -507,22 +554,29 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
def boom(self: Path, data: bytes) -> int: def boom(self: Path, data: bytes) -> int:
written_paths.append(self) written_paths.append(self)
n = original_write(self, data) # the tmp file really lands first
if len(written_paths) > 1: if len(written_paths) > 1:
raise OSError("simulated disk full (test sentinel)") raise OSError("simulated disk full (test sentinel)")
return original_write(self, data) return n
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType] monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
try: try:
with caplog.at_level(logging.WARNING, logger="app.importer"): with caplog.at_level(logging.WARNING, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db)) summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0) assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert len(written_paths) == 2, "p1 landed, p2's write failed" assert len(written_paths) == 2, "p1's tmp landed + replaced, p2's write failed"
assert all(p.name.endswith(".png.tmp") for p in written_paths), (
"the writer only ever touches .tmp paths (atomic replace)"
)
doc = _doc(db, "pgw", "ikea.pdf") doc = _doc(db, "pgw", "ikea.pdf")
assert doc.page_count is None assert doc.page_count is None
assert doc.page_strip_mode is None, "the class is cleared with the count"
assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), ( assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), (
"the partial write was cleaned up — no orphans" "the partial write was cleaned up — no orphans"
) )
assert not written_paths[0].exists(), "the landed p1 copy was removed" assert not list(_pages_dir(tmp_path).glob("*.tmp")), (
"the failed call's .tmp file was cleaned up too"
)
assert summary.summaries == 1, "the document is indexed exactly as pre-phase" assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
assert any("PDF page render failed" in m for m in caplog.messages), ( assert any("PDF page render failed" in m for m in caplog.messages), (
caplog.messages caplog.messages
@@ -531,6 +585,246 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
_cleanup_source(db, "pgw") _cleanup_source(db, "pgw")
# ---------------------------------------------------------------------------
# Phase 163 (task 02) — WRITE-FIRST, atomic per file (LOCKED A4)
# ---------------------------------------------------------------------------
def test_write_first_new_bytes_are_durable_before_any_stale_delete(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE order pin (LOCKED A4 — the 2026-10-07 live loss): re-rendering
a CHANGED (shrunk, 10 → 6) PDF writes every replacement page FIRST
(tmp + ``os.replace``) and only then deletes the stale tail — the
event log shows ALL replaces of the fresh bytes before the FIRST
unlink of an old-extent file, the overwritten pages keep the NEW
bytes, the shrunk tail (p7..p10) is gone, and nothing else is
deleted."""
root = tmp_path / "pgord"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
asyncio.run(import_sources([root], llm, session=db))
doc1 = _doc(db, "pgord", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
sentinel = b"OLD-COPY-SENTINEL"
for n in range(1, 11): # make the old copies unmistakably distinguishable
(page_dir / f"{doc1.id}-p{n}.png").write_bytes(sentinel)
events: list[tuple[str, str]] = []
real_replace = os.replace
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
events.append(("replace", Path(os.fspath(dst)).name))
return real_replace(src, dst, *args, **kwargs)
real_unlink = Path.unlink
def unlink_spy(self: Path, **kwargs: Any) -> None:
if self.parent == page_dir and self.name.startswith(str(doc1.id)):
events.append(("unlink", self.name))
return real_unlink(self, **kwargs)
monkeypatch.setattr(importer.os, "replace", replace_spy)
monkeypatch.setattr(Path, "unlink", unlink_spy) # pyright: ignore[reportArgumentType]
try:
_build_ikea_shrunk(pdf) # same path, 6 pages — still qualifying
s2 = asyncio.run(import_sources([root], llm, session=db))
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
_cleanup_source(db, "pgord")
replaces = [name for kind, name in events if kind == "replace"]
unlinks = [name for kind, name in events if kind == "unlink"]
assert len(replaces) == 6, "the 6 fresh pages are written via os.replace"
assert set(unlinks) == {f"{doc1.id}-p{n}.png" for n in range(7, 11)}, (
"the stale-tail delete touches ONLY the old extent the render did "
"not rewrite — a file about to be overwritten is never deleted"
)
first_unlink = next(i for i, e in enumerate(events) if e[0] == "unlink")
last_replace = max(i for i, e in enumerate(events) if e[0] == "replace")
assert last_replace < first_unlink, (
f"write-first violated — event order: {events}"
)
for n in range(1, 7):
blob = (page_dir / f"{doc1.id}-p{n}.png").read_bytes()
assert blob != sentinel and blob[:8] == PNG_MAGIC, (
f"p{n} keeps the NEW rendered bytes"
)
for n in range(7, 11):
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"stale p{n} leaked"
assert not list(page_dir.glob("*.tmp")), "no .tmp orphans"
def test_mid_write_failure_keeps_the_prior_copies_intact(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""THE failure pin (LOCKED A4): the writer raises on the SECOND
page's tmp write of a changed re-render — the prior copy set stays
on disk INTACT (the rollback-orphan state — the row over zero
files — is structurally unrepresentable): every one of the 10 prior
files still exists (p1 with the new bytes already atomically
replaced before the failure, p2..p10 byte-for-byte the prior
copies), zero ``.tmp`` orphans from this call, and the caller's
fail-soft fires (``page_count``/``page_strip_mode`` NULL + the
warning, the doc indexed fully)."""
root = tmp_path / "pgfail"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
asyncio.run(import_sources([root], llm, session=db))
doc1 = _doc(db, "pgfail", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
prior = {
n: (page_dir / f"{doc1.id}-p{n}.png").read_bytes() for n in range(1, 11)
}
# Tag the re-render's bytes so the new p1 is recognizable, then
# blow up the SECOND tmp write (page 2).
_render_spy(
monkeypatch,
mutate_full=lambda pages: [
PNG_MAGIC + f"NEW-{i}".encode() for i in range(len(pages))
],
)
original_write = Path.write_bytes
tmp_writes: list[Path] = []
def boom(self: Path, data: bytes) -> int:
if self.suffix == ".tmp":
tmp_writes.append(self)
if len(tmp_writes) > 1:
raise OSError("simulated disk full (test sentinel)")
return original_write(self, data)
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
_build_ikea_shrunk(pdf)
try:
with caplog.at_level(logging.WARNING, logger="app.importer"):
s2 = asyncio.run(import_sources([root], llm, session=db))
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
assert len(tmp_writes) == 2, "p1's tmp wrote, p2's tmp write raised"
doc2 = _doc(db, "pgfail", "ikea.pdf")
assert doc2.page_count is None, "the caller's fail-soft nulled the count"
assert doc2.page_strip_mode is None, "the caller's fail-soft nulled the class"
assert any("PDF page render failed" in m for m in caplog.messages), caplog.messages
# The prior set is INTACT: exactly the 10 names, none lost, none
# orphaned on top.
assert {p.name for p in page_dir.iterdir()} == {
f"{doc1.id}-p{n}.png" for n in range(1, 11)
}, "files on disk exactly the prior set"
assert (
page_dir / f"{doc1.id}-p1.png"
).read_bytes() == PNG_MAGIC + b"NEW-0", (
"p1's replacement bytes were durable before the failure (atomic "
"replace) and were NOT removed by the cleanup"
)
for n in range(2, 11):
assert (
page_dir / f"{doc1.id}-p{n}.png"
).read_bytes() == prior[n], f"prior copy p{n} was disturbed"
assert not list(page_dir.glob("*.tmp")), "zero .tmp orphans from this call"
assert doc2.chunks, "the document is indexed fully (fail-soft)"
def test_page_copies_are_written_atomically_tmp_then_replace(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE atomic pin (LOCKED A4): the writer NEVER writes a final
``<doc-id>-p{N}.png`` path directly — every page's bytes land on
``<name>.png.tmp`` and ``os.replace`` moves them onto the final
name in order (a reader opening the final path never sees truncated
bytes), and no ``.tmp`` survives the successful call."""
root = tmp_path / "pgatom"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
written: list[Path] = []
original_write = Path.write_bytes
def write_spy(self: Path, data: bytes) -> int:
written.append(self)
return original_write(self, data)
replaced: list[tuple[str, str]] = []
real_replace = os.replace
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
replaced.append((os.fspath(src), os.fspath(dst)))
return real_replace(src, dst, *args, **kwargs)
monkeypatch.setattr(Path, "write_bytes", write_spy) # pyright: ignore[reportArgumentType]
monkeypatch.setattr(importer.os, "replace", replace_spy)
try:
asyncio.run(import_sources([root], llm, session=db))
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
doc = _doc(db, "pgatom", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
assert len(written) == 10 and len(replaced) == 10
assert all(p.name.endswith(".png.tmp") for p in written), (
"no final path is ever written directly"
)
for n, (src, dst) in enumerate(replaced, start=1):
assert dst == str(page_dir / f"{doc.id}-p{n}.png")
assert src == f"{dst}.tmp", "the tmp name is <final>.tmp"
assert not list(page_dir.glob("*.tmp")), "no .tmp survives the success path"
def test_stale_tail_delete_failure_is_best_effort(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""An unreadable STALE-TAIL copy (the write-first tail pass, the
``_delete_page_copies`` convention): the failure is logged, never
raised — the sync completes, the row is stamped with the fresh
count, and the reachable tail files still go."""
root = tmp_path / "pgtail"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
asyncio.run(import_sources([root], llm, session=db))
doc1 = _doc(db, "pgtail", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
p10 = page_dir / f"{doc1.id}-p10.png"
real_unlink = Path.unlink
def boom(self: Path, **kwargs: Any) -> None:
if self == p10:
raise OSError("simulated unreadable file (test sentinel)")
return real_unlink(self, **kwargs)
monkeypatch.setattr(Path, "unlink", boom) # pyright: ignore[reportArgumentType]
_build_ikea_shrunk(pdf)
try:
with caplog.at_level(logging.WARNING, logger="app.importer"):
s2 = asyncio.run(import_sources([root], llm, session=db))
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
doc2 = _doc(db, "pgtail", "ikea.pdf")
assert doc2.page_count == 6 and doc2.page_strip_mode == "additive"
assert p10.exists(), "the unreadable tail file survived (never raised)"
for n in range(7, 10):
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"p{n} still leaked"
assert any(
"could not delete PDF page copy" in m for m in caplog.messages
), caplog.messages
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# The shared cleanup helper (the _delete_image_copy convention) # The shared cleanup helper (the _delete_image_copy convention)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+4 -1
View File
@@ -117,7 +117,10 @@ def test_gate_runs_after_the_phase36_base_construction() -> None:
label = js.find('title.textContent = "Summary"') label = js.find('title.textContent = "Summary"')
text_node = js.find('body.className = "doc-summary-text"') text_node = js.find('body.className = "doc-summary-text"')
append = js.find("section.append(title, body)") append = js.find("section.append(title, body)")
mount = js.find("contentEl.appendChild(section)") # Phase 163 (task 04): the mount rides the `slot` variable —
# slot === contentEl for every non-additive doc (the byte-identity
# property), views only for the additive strip class.
mount = js.find("slot.appendChild(section)")
gate = js.find("void docAdminReady().then(", mount) gate = js.find("void docAdminReady().then(", mount)
wiring = js.find("if (admin) wireSummaryEdit(section, doc);") wiring = js.find("if (admin) wireSummaryEdit(section, doc);")
assert 0 < base < label < text_node < append < mount < gate < wiring, ( assert 0 < base < label < text_node < append < mount < gate < wiring, (
+431
View File
@@ -0,0 +1,431 @@
"""Unit: the additive PDF strip view (phase 163, task 04 — D4, LOCKED A1/A6).
Frontend source-contract pins + node behavioral runs (the
``test_document_viewer.py`` house pattern) over ``document.js``'s
``renderDocument`` — the ONE shared core (Sources modal AND
/document.html):
* the MODE branch — ``page_urls`` + ``page_strip_mode === "additive"``
falls through to the text-doc rendering (Summary panel + content,
the pre-148 branch reused) wrapped in a ``.doc-views`` container
with the ``.doc-view-toggle`` group above it (Text pressed by
default, LOCKED A6); ``page_urls`` with the mode ABSENT or
``"primary"`` keeps the phase-148 strip-only branch VERBATIM (append
the strip to ``contentEl`` + return — no toggle, no wrapper, no
Summary panel); falsy ``page_urls`` is untouched;
* the toggle semantics — ``role="group"`` + ``aria-label="Document
view"``, two ``<button type="button">`` labeled ``Text`` / ``Pages``
(fixed strings, no doc-derived text), ``aria-pressed`` true/false
initial state, and a click mutating ONLY the two ``aria-pressed``
states + the wrapper's ``data-view`` attribute (CSS-driven show/
hide — the strip node stays mounted, no re-render, no refetch);
* the never-build-HTML rule — createElement + properties/textContent
only in the new code path (the module's ONE innerHTML stays the
escape-first markdown render); the strip imgs keep
``loading = "lazy"`` (the hidden strip never fetches);
* the ``.doc-view-toggle`` CSS — the house ghost-pill idiom, the
pressed state as text + style (never color alone, B5), ≥44px
targets, and the ``data-view`` show/hide contract.
"""
from __future__ import annotations
import json
import re
import shutil
import subprocess
from pathlib import Path
import pytest
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
DOCUMENT_JS = FRONTEND / "assets" / "document.js"
STYLES_CSS = FRONTEND / "assets" / "styles.css"
HAVE_NODE = shutil.which("node") is not None
def _read(path: Path) -> str:
return path.read_text(encoding="utf-8")
def _fn(js: str, name: str) -> str:
"""The source of a top-level function via balanced-brace counting
(the ``test_document_viewer.py`` pattern)."""
start = js.find(f"function {name}(")
assert start != -1, f"{name}() must exist in document.js"
i = js.find("(", start)
depth = 0
close = i
while i < len(js):
if js[i] == "(":
depth += 1
elif js[i] == ")":
depth -= 1
if depth == 0:
close = i
break
i += 1
brace = js.find("{", close)
depth = 0
for j in range(brace, len(js)):
if js[j] == "{":
depth += 1
elif js[j] == "}":
depth -= 1
if depth == 0:
return js[start : j + 1]
raise AssertionError(f"unbalanced braces in {name}()")
def _code(js: str) -> str:
"""The source with comments stripped (the never-build-HTML pins)."""
return re.sub(r"//.*?$|/\*.*?\*/", "", js, flags=re.S | re.M)
def _run_node(script: str) -> str:
proc = subprocess.run(
["node", "-e", script], capture_output=True, text=True, timeout=60
)
assert proc.returncode == 0, f"node failed: {proc.stderr}"
return proc.stdout
# ---------------------------------------------------------------------------
# Source pins — the renderDocument mode branch
# ---------------------------------------------------------------------------
def test_render_document_branches_on_strip_mode() -> None:
"""The mode branch (LOCKED A2 — the tolerant read): the additive
flag keys on the EXACT ``doc.page_strip_mode === "additive"``
string (absent / "primary" / anything else ⇒ primary), and the
primary/absent strip branch stays the phase-148 shape — append
``docPagesStrip`` to ``contentEl`` and RETURN — evaluated BEFORE
any wrapper/toggle construction."""
render = _fn(_read(DOCUMENT_JS), "renderDocument")
assert 'doc.page_strip_mode === "additive"' in render, (
"only the exact additive string switches on the toggle"
)
gate_i = render.find('const additive = stripEligible && doc.page_strip_mode === "additive";')
primary_i = render.find("if (stripEligible && !additive) {")
strip_i = render.find("contentEl.appendChild(docPagesStrip(doc, pageUrls));")
return_i = render.find("return;", strip_i)
wrapper_i = render.find('views.className = "doc-views";')
assert -1 < gate_i < primary_i < strip_i < return_i < wrapper_i, (
"the primary/absent branch (strip into contentEl + return) precedes "
"the additive wrapper — its DOM stays the phase-148 shape verbatim"
)
# The wrapper + toggle: ONE .doc-views container, the toggle BEFORE
# it, both into contentEl; the text view lands in the wrapper.
append_i = render.find("contentEl.append(docViewToggle(views), views);")
assert wrapper_i < append_i, "the wrapper is built, then appended after its toggle"
assert 'views.setAttribute("data-view", "text");' in render, (
"LOCKED A6 — Text is the default view"
)
# The strip mounts ONCE inside the wrapper, after the text view —
# never rebuilt on a flip (no docPagesStrip call outside the two
# branches).
assert render.count("docPagesStrip(doc, pageUrls)") == 2
mount_i = render.rfind("views.appendChild(docPagesStrip(doc, pageUrls));")
assert mount_i > render.find('slot.appendChild(pre);') or mount_i > render.rfind(
"slot.appendChild(wrap);"
), "the strip mounts AFTER the text view inside the wrapper"
# The non-additive path is untouched: slot === contentEl (the
# pre-148 appends keep their exact shape through the slot var).
assert "let slot = contentEl;" in render
def test_primary_absent_branch_builds_no_toggle() -> None:
"""The primary/absent branch is toggle-free by construction: the
strip-branch block (from the ``pageUrls`` read to its ``return``)
contains NO ``doc-view-toggle`` / ``doc-views`` construction — the
strip-only DOM stays byte-identical to pre-phase-163."""
render = _fn(_read(DOCUMENT_JS), "renderDocument")
primary_start = render.find("if (stripEligible && !additive)")
branch = render[render.find("const pageUrls =") : render.find("return;", primary_start)]
assert "doc-view-toggle" not in branch
assert "doc-views" not in branch
assert branch.count("appendChild") == 1, (
"the primary branch is exactly the phase-148 append + return"
)
def test_toggle_never_builds_html_and_labels_are_fixed() -> None:
"""The never-build-HTML rule: the toggle builders use
createElement + properties/textContent only — no innerHTML /
outerHTML — and the labels are the FIXED strings ``Text`` /
``Pages`` (no doc-derived text inside the toggle). The module's
ONE innerHTML (the escape-first markdown render) survives."""
js = _read(DOCUMENT_JS)
toggle = _fn(js, "docViewToggle")
btn = _fn(js, "viewToggleButton")
for body in (toggle, btn):
code = _code(body)
assert "innerHTML" not in code and "outerHTML" not in code
assert 'document.createElement("div")' in toggle
assert 'group.setAttribute("role", "group")' in toggle
assert 'group.setAttribute("aria-label", "Document view")' in toggle
assert 'document.createElement("button")' in btn
assert 'btn.type = "button"' in btn
assert 'btn.className = "doc-view-btn"' in btn
assert 'viewToggleButton("Text")' in toggle
assert 'viewToggleButton("Pages")' in toggle
# The module-wide XSS contract: exactly one innerHTML, the
# renderMarkdown line.
whole = _code(js)
assert whole.count("innerHTML") == 1
assert "outerHTML" not in whole
def test_strip_imgs_stay_lazy() -> None:
"""The phase-148 lazy-img pin re-runs green: the strip builder is
UNMODIFIED — every page ``<img>`` keeps ``loading = "lazy"`` (the
additive default view never fetches the strip)."""
strip = _fn(_read(DOCUMENT_JS), "docPagesStrip")
assert 'img.loading = "lazy";' in strip
def test_view_toggle_css() -> None:
"""styles.css: the ``.doc-view-toggle`` group (the chat-column
reading width, the pill pair adjacent) + the ``.doc-view-btn``
ghost pill (≥44px target, ink-soft text — AA) + the PRESSED state
(``[aria-pressed="true"]`` — brand fill + font-weight 700: text +
style, never color alone, B5) + the CSS-driven view swap
(``.doc-views .doc-pages`` hidden by default; ``[data-view=
"pages"]`` hides the Summary panel / .doc-md / .doc-raw /
.doc-image and shows the strip as the grid). Provenance comment
above the block (house style)."""
css = _read(STYLES_CSS)
toggle_i = css.find(".doc-view-toggle {")
assert toggle_i != -1, "styles.css must style .doc-view-toggle"
comment_start = css.rfind("/*", 0, toggle_i)
assert -1 < comment_start < toggle_i and "phase 163" in css[comment_start:toggle_i]
btn_i = css.find(".doc-view-btn {")
pressed_i = css.find('.doc-view-btn[aria-pressed="true"]')
assert toggle_i < btn_i < pressed_i
btn_rule = css[btn_i : css.find("}", btn_i)]
for decl in ("min-height: 44px", "color: var(--ink-soft)", "border: 1px solid var(--line)"):
assert decl in btn_rule
pressed_rule = css[pressed_i : css.find("}", pressed_i)]
assert "background: var(--brand)" in pressed_rule
assert "font-weight: 700" in pressed_rule, (
"the pressed state is text weight + fill — never color alone (B5)"
)
# The CSS-driven swap: hidden strip by default; pages hides the
# text view and shows the strip.
assert '.doc-views .doc-pages { display: none; }' in css
for sel in (".doc-summary", ".doc-md", ".doc-raw", ".doc-image"):
assert f'.doc-views[data-view="pages"] {sel}' in css
assert '.doc-views[data-view="pages"] .doc-pages { display: grid; }' in css
# ---------------------------------------------------------------------------
# Node behavioral runs — the real renderDocument over a minimal fake DOM
# ---------------------------------------------------------------------------
_FAKE_DOM = """
function makeEl(tag) {
return {
tag, className: "", textContent: "", attrs: {}, children: [], handlers: {},
setAttribute(k, v) { this.attrs[k] = String(v); },
getAttribute(k) { return k in this.attrs ? this.attrs[k] : null; },
appendChild(c) { this.children.push(c); return c; },
append(...cs) { this.children.push(...cs); },
replaceChildren(...cs) { this.children = cs; },
addEventListener(ev, fn) { this.handlers[ev] = fn; },
click() { this.handlers.click(); },
};
}
globalThis.document = { createElement: (t) => makeEl(t) };
function fmtDate(x) { return "DATE:" + x; }
function metaBadge(cls, text, title) {
const s = makeEl("span");
s.className = cls;
s.textContent = text;
if (title !== undefined) s.setAttribute("title", title);
return s;
}
function copyPathButton() { return makeEl("button"); }
function originAnchor() { return makeEl("a"); }
async function docAdminReady() { return false; }
function wireDateEdit() {}
function wireSummaryEdit() {}
function renderMarkdown(s) { return s; }
function docImageBlock() { return makeEl("div"); }
function describe(el) {
return {
tag: el.tag,
className: el.className,
attrs: el.attrs,
text: el.textContent,
children: el.children.map(describe),
};
}
"""
def _render_script() -> str:
js = _read(DOCUMENT_JS)
parts = [
_fn(js, n)
for n in ("renderDocument", "docViewToggle", "viewToggleButton", "docPagesStrip")
]
return _FAKE_DOM + "\n".join(parts)
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_additive_doc_renders_text_views_and_toggle() -> None:
"""Behavioral (node, fake DOM): an ADDITIVE doc renders the TEXT
doc exactly as a non-qualifying one — Summary panel + content —
inside ``.doc-views`` (data-view="text"), with the toggle group
FIRST in the content slot: role=group, aria-label="Document
view", the two ``doc-view-btn`` buttons labeled Text/Pages, Text
aria-pressed="true" (LOCKED A6), and the strip MOUNTED inside the
wrapper (figures + lazy imgs) after the text view."""
out = _run_node(
_render_script()
+ """
const doc = {
title: "Letumdei", path: "book.pdf", source: "Library", format: "pdf",
summary: "A book.", content: "BODY TEXT", created_at: "c", indexed_at: "i",
chunks: 9, page_urls: ["/api/documents/x/pages/1", "/api/documents/x/pages/2"],
page_strip_mode: "additive",
};
const contentEl = makeEl("div");
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
console.log(JSON.stringify(describe(contentEl)));
"""
)
content = json.loads(out)
assert [c["className"] for c in content["children"]] == [
"doc-view-toggle",
"doc-views",
], "the toggle group sits above the .doc-views wrapper"
toggle, views = content["children"]
assert toggle["attrs"]["role"] == "group"
assert toggle["attrs"]["aria-label"] == "Document view"
labels = [c["text"] for c in toggle["children"]]
assert labels == ["Text", "Pages"]
assert [c["attrs"]["aria-pressed"] for c in toggle["children"]] == ["true", "false"], (
"Text pressed by default (LOCKED A6)"
)
for c in toggle["children"]:
assert c["tag"] == "button"
assert views["attrs"]["data-view"] == "text"
assert [c["className"] for c in views["children"]] == [
"doc-summary",
"doc-raw",
"doc-pages",
], "additive = Summary panel + text content + the mounted strip"
# pre.textContent — a text node, never parsed as HTML
assert views["children"][1]["text"] == "BODY TEXT"
strip = views["children"][2]
assert len(strip["children"]) == 2, "one figure per page_urls entry"
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_primary_and_absent_modes_keep_the_strip_only_dom() -> None:
"""Behavioral: mode "primary" AND an absent mode (the pre-163
stamped row — the live Letumdei pre-sync shape) keep the phase-148
DOM byte-identically: the content slot holds EXACTLY the
``.doc-pages`` strip — no wrapper, no toggle, no Summary panel,
no text block."""
out = _run_node(
_render_script()
+ """
const results = [];
for (const mode of ["primary", undefined]) {
const doc = {
title: "Manual", path: "m.pdf", source: "Library", format: "pdf",
summary: "A manual.", content: "SHOULD NOT SHOW", created_at: "c",
indexed_at: "i", chunks: 4,
page_urls: ["/api/documents/y/pages/1"],
};
if (mode !== undefined) doc.page_strip_mode = mode;
const contentEl = makeEl("div");
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
results.push(describe(contentEl));
}
console.log(JSON.stringify(results));
"""
)
primary, absent = json.loads(out)
for content in (primary, absent):
assert len(content["children"]) == 1
assert content["children"][0]["className"] == "doc-pages"
classes = json.dumps(content)
assert "doc-view-toggle" not in classes
assert "doc-views" not in classes
assert "doc-summary" not in classes
assert "doc-raw" not in classes, "the strip replaces the text block (primary class)"
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_text_doc_dom_is_untouched() -> None:
"""Behavioral: a doc with no page_urls is UNTOUCHED — Summary
panel + content appended straight to the content slot, no
wrapper, no toggle (the byte-identity property for every
non-PDF)."""
out = _run_node(
_render_script()
+ """
const doc = {
title: "Note", path: "n.pdf", source: "Library", format: "pdf",
summary: "S.", content: "TEXT", created_at: "c", indexed_at: "i", chunks: 1,
};
const contentEl = makeEl("div");
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
console.log(JSON.stringify(describe(contentEl)));
"""
)
content = json.loads(out)
assert [c["className"] for c in content["children"]] == ["doc-summary", "doc-raw"]
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_toggle_flips_view_without_rerender_or_refetch() -> None:
"""Behavioral: the toggle handler mutates ONLY the two
aria-pressed states + the wrapper's data-view. Clicking Pages
flips data-view to "pages"; clicking back to Text reverses; the
strip node STAYS MOUNTED the whole time (same object, no
re-append — the flip-flop fetches nothing) and the wrapper's
children never change count."""
out = _run_node(
_render_script()
+ """
const doc = {
title: "Book", path: "b.pdf", source: "Library", format: "pdf",
summary: "S.", content: "TEXT", created_at: "c", indexed_at: "i", chunks: 2,
page_urls: ["/api/documents/z/pages/1", "/api/documents/z/pages/2"],
page_strip_mode: "additive",
};
const contentEl = makeEl("div");
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
const toggle = contentEl.children[0];
const views = contentEl.children[1];
const [textBtn, pagesBtn] = toggle.children;
const stripBefore = views.children[2];
const countBefore = views.children.length;
pagesBtn.click();
const afterPages = {
view: views.getAttribute("data-view"),
pressed: toggle.children.map((b) => b.getAttribute("aria-pressed")),
};
textBtn.click();
const afterText = {
view: views.getAttribute("data-view"),
pressed: toggle.children.map((b) => b.getAttribute("aria-pressed")),
stripSame: views.children[2] === stripBefore,
countSame: views.children.length === countBefore,
};
console.log(JSON.stringify({ afterPages, afterText }));
"""
)
res = json.loads(out)
assert res["afterPages"] == {"view": "pages", "pressed": ["false", "true"]}
assert res["afterText"] == {
"view": "text",
"pressed": ["true", "false"],
"stripSame": True,
"countSame": True,
}, "the strip node persists across flips — no re-render, no refetch"
+12 -8
View File
@@ -13,12 +13,13 @@ without a browser:
* NO ``@media (min-width: 1500px)`` block anywhere in the file — the * NO ``@media (min-width: 1500px)`` block anywhere in the file — the
phase-58 wide-desktop doubling is deleted in full (the token is phase-58 wide-desktop doubling is deleted in full (the token is
72rem at every viewport); 72rem at every viewport);
* the six reading-column selectors — ``.chat-shell``, * the reading-column selectors — ``.chat-shell``,
``.shared-shell``, ``.doc-md``, ``.doc-image`` (phase 122, task 04), ``.shared-shell``, ``.doc-md``, ``.doc-image`` (phase 122, task 04),
``.doc-pages`` (phase 148, task 03 — the page strip), and ``.doc-pages`` (phase 148, task 03 — the page strip),
``.doc-summary:has(+ .doc-md)`` — each capped with ``.doc-view-toggle`` (phase 163, task 04 — the Text/Pages toggle
``max-width: var(--chat-column)`` and NOTHING else in the file uses above the additive views), and ``.doc-summary:has(+ .doc-md)`` —
the token for a max-width (exactly six rules); each capped with ``max-width: var(--chat-column)`` and NOTHING else
in the file uses the token for a max-width (exactly seven rules);
* the flipped negative pin — ZERO literal ``max-width: 46rem`` rules * the flipped negative pin — ZERO literal ``max-width: 46rem`` rules
remain (the phase-27/59/91 form-column caps are retired): the form remain (the phase-27/59/91 form-column caps are retired): the form
shells (``.tuning-shell``, ``.theme-shell``, ``.doc-edit-shell``) shells (``.tuning-shell``, ``.theme-shell``, ``.doc-edit-shell``)
@@ -121,8 +122,10 @@ def test_the_reading_columns_use_the_token() -> None:
task 04 — the viewer's image block rides the SAME reading column task 04 — the viewer's image block rides the SAME reading column
as its .doc-md sibling), .doc-pages (phase 148, task 03 — the as its .doc-md sibling), .doc-pages (phase 148, task 03 — the
viewer's page strip rides the SAME reading column in the shared viewer's page strip rides the SAME reading column in the shared
#doc-content slot) and .doc-summary:has(+ .doc-md) each cap with #doc-content slot), .doc-view-toggle (phase 163, task 04 — the
max-width: var(--chat-column) — and exactly those six rules use Text/Pages toggle sits above the additive views, aligned with
them) and .doc-summary:has(+ .doc-md) each cap with
max-width: var(--chat-column) — and exactly those seven rules use
the token for a max-width (no other selector).""" the token for a max-width (no other selector)."""
css = _css() css = _css()
for selector in ( for selector in (
@@ -131,12 +134,13 @@ def test_the_reading_columns_use_the_token() -> None:
".doc-md", ".doc-md",
".doc-image", ".doc-image",
".doc-pages", ".doc-pages",
".doc-view-toggle",
".doc-summary:has(+ .doc-md)", ".doc-summary:has(+ .doc-md)",
): ):
assert "max-width: var(--chat-column)" in _rule_block(css, selector), ( assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
f"{selector} must cap with max-width: var(--chat-column)" f"{selector} must cap with max-width: var(--chat-column)"
) )
assert css.count("max-width: var(--chat-column)") == 6, ( assert css.count("max-width: var(--chat-column)") == 7, (
"exactly the reading-column selectors use the token" "exactly the reading-column selectors use the token"
) )