Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6ba46e735a | ||
|
|
74eb2e265c | ||
|
|
855fa27a30 |
@@ -0,0 +1,73 @@
|
|||||||
|
# Phase 163 — PDF page strip: additive text+pages view, write-first atomic copies, deterministic fast-path re-gate (self-heal)
|
||||||
|
|
||||||
|
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, the document modal for `Letumdei_ Heirs to the Imperium/Letumdei_ Heirs to the Imperium.pdf` (doc id `b72f5515-dc40-4167-98ff-179a6503e6e5`). Every page rendered "Page N unavailable." and the document was unreadable in the viewer. **Pre-fix evidence (measured live, 2026-10-07):** the payload carries 36 `page_urls` (so `page_count = 36`); `GET /api/documents/<id>/pages/1…36` **all 404**; the content is genuine extracted text (114 389 chars, 36 `---PAGE-N---` markers, exactly ONE `[Page 36 illustration: …]` block — a map page under `pdf_pictorial_min_chars`); the same server serves the 10/3-indexed IKEA manual's pages 200 (the pages dir was NOT wiped — only this doc's copies are gone). Two stacked defects:
|
||||||
|
|
||||||
|
1. **Eligibility consequence (the "shouldn't be a pictorial pdf" complaint).** The phase-140 gate is per-page (`len(text.strip()) < pdf_pictorial_min_chars`, default 100); ONE sparse page in a text-heavy 36-page book set `pictorial_described = True` → the doc joined the strip class → the phase-148 viewer renders the strip **INSTEAD OF** the text content and the Summary panel. The owner's call (2026-10-07): **additive strip** — a usable-layer (pictorial-class) doc shows its TEXT (plus the Summary panel, parity with any text doc) with a **Text/Pages toggle** for the strip; the text is never hidden. The fallback class (unusable text layer — the phase-133 screenshots ARE the content) keeps today's strip-only view, DOM byte-identical.
|
||||||
|
2. **Lost copies, stale `page_count`, no self-heal.** `_persist_pdf_pages` **deletes the old copies from disk first**, and the filesystem work is NOT transactional with the DB session: any later rollback of the file's transaction (the per-file `EmbeddingError` handler, a killed sync) — or an external wipe of `pdf_page_dir` — leaves the row claiming `page_count = 36` over ZERO files. The phase-147 fast path ("the copies persist") never re-renders, so the strip 404s forever. The owner's call: **write-before-delete + fast-path re-gate** — the changed path writes the new copies (atomically, per file) BEFORE any deletion; the unchanged paths deterministically re-gate the strip (no LLM/vision call) — re-stamp the class, re-render missing copies, or clear a strip that no longer qualifies.
|
||||||
|
|
||||||
|
**Story:** n/a (owner-directed bug fix; extends `148_pdf_pages_in_sources`, `147_sync_changed_only`, `140_pictorial_pdf_pages`, `133_pdf_screenshot_fallback`).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A pictorial-class PDF is a **readable text document with the pages one toggle away** (Text default; Pages shows the phase-148 strip); a fallback-class PDF keeps the strip-only view byte-identically; and a strip never dies silently — the next sync **re-gates and re-renders** missing copies (or clears a strip that no longer qualifies) with zero LLM calls. The live `Letumdei` doc is repaired by the next production Sync (no manual script): additive mode stamped, 36 copies re-rendered, the book's text back in the viewer.
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `148_pdf_pages_in_sources` (complete) — `page_count`, `pdf_page_dir`, `<doc-id>-p{N}.png`, the `/pages/{n}` route, the viewer's `docPagesStrip`, and the persist/regate seams this phase changes.
|
||||||
|
- `147_sync_changed_only` (complete) — the `file_hash` fast path + shared `_unchanged_doc` (the re-gate hook site) whose "the copies persist" assumption this phase replaces.
|
||||||
|
- `140_pictorial_pdf_pages` (complete) — the deterministic `pictorial_pages` gate + the kill switch `settings.pdf_pictorial` (the re-gate mirrors this gate exactly).
|
||||||
|
- `133_pdf_screenshot_fallback` (complete) — `is_usable_pdf_text` (the fallback-class signal the re-gate mirrors).
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — the strip-mode wire signal (task 01, LOCKED A2/A3).** New nullable `documents.page_strip_mode` VARCHAR(16) (migration **0027**, tested downgrade, A13): `"primary"` = fallback class (strip replaces the text — today's behavior), `"additive"` = pictorial class (text + toggle). Set exactly when `page_count` is set (cleared with it). `DocContent` gains `page_strip_mode: str | None = None` with the `image_url` omission rule: **ABSENT (never null) when `page_count` is NULL** — every non-qualifying doc's payload stays byte-identical to pre-phase. For a qualifying doc the key is present; a **NULL-mode pre-phase row serializes ABSENT** and the viewer treats absent as `"primary"` (existing rows keep today's DOM until the re-gate stamps them). The `/pages/{n}` route is untouched.
|
||||||
|
|
||||||
|
**D2 — write-first atomic persist (task 02, LOCKED A4).** `_persist_pdf_pages` reorders: **render → write every page (atomic per file: write to `<name>.png.tmp`, `os.replace`) → only then delete stale copies** outside the written set (the shrunk-PDF tail; the current blanket `_delete_page_copies` BEFORE the render is removed — a rollback or crash can no longer leave the row over zero files). The partial-write failure path unlinks BOTH the new final files and the `.tmp` files before re-raising (no orphans; the existing fail-soft contract — caller sets `page_count` NULL + warning — unchanged). The function also stamps `doc.page_strip_mode`: `"primary"` when `fallback_fired`, `"additive"` when `pictorial_described` (the caller passes the CLASS, not just the boolean — `eligible=fallback_fired or pictorial_described` is replaced by `strip_class="fallback"|"pictorial"|None`); the ineligible arm clears mode with `page_count`. **The eligibility RULE itself is unchanged** (the phase-148 lock `fallback_fired or pictorial_described` stands — only its VIEW consequence changes, via D1/D4).
|
||||||
|
|
||||||
|
**D3 — the deterministic fast-path re-gate (task 03, LOCKED A5).** A new `_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths — inside `_unchanged_doc` (an optional `full_path` kwarg, defaulting to `Path(doc.full_path)`), so it fires on the `file_hash` fast path AND the content-digest-hit path — guarded by `doc.page_count is not None` (only PDF rows ever carry it; every other file's fast path keeps its zero-extra-read contract). It mirrors the changed-path gate **deterministically from the current file bytes + current settings — ZERO vision/LLM calls** (`render_pdf_pages` is CPU-only, A21-safe):
|
||||||
|
- `not is_usable_pdf_text(extract_pdf_text(full_path))` → class `fallback` (mode `primary`) — same doc as today;
|
||||||
|
- elif `settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=settings.pdf_pictorial_min_chars)` is non-empty → class `pictorial` (mode `additive`) — **candidates are exactly the existing gate's** (text-carrying pages under `min_chars`; no-text pages are NOT candidates, mirroring `pictorial_pages` — the re-gate adds no new semantics);
|
||||||
|
- else → **repair arm**: the strip no longer qualifies (e.g. the owner lowered `pdf_pictorial_min_chars`) → `page_count = None`, `page_strip_mode = None`, `_delete_page_copies(doc, page_dir)`.
|
||||||
|
|
||||||
|
For a re-gated-eligible doc, **verify the copies**: any of `page_dir/<doc.id>-p1..p{page_count}.png` missing → re-render ALL pages (`render_pdf_pages`), atomic write-first (D2's writer), and set `page_count` to the fresh render's length (a drift names the doc in a warning; bytes unchanged makes drift pathological). A fresh render producing NO pages (unopenable despite the just-computed hash — pathological) keeps the row as-is with a warning (fail-soft, never clears on a transient). One greppable INFO line whenever re-gate ACTS (never per-unchanged-doc): `import: PDF strip re-gate source=… path=… — healed N pages | cleared stale strip | mode <old>-><new>`. The re-gate mutates the row INSIDE the unchanged path's transaction (the commit already owned by `_unchanged_doc`'s caller flow); a copy-verify/render exception is caught, warned, and never fails the sync.
|
||||||
|
|
||||||
|
**D4 — the additive viewer (task 04, LOCKED A1/A6).** `renderDocument` (the ONE shared core — modal AND `/document.html`): the strip branch now keys on the mode —
|
||||||
|
- `page_strip_mode` absent or `"primary"` (with `page_urls`): the EXACT current DOM (`.doc-pages` strip, no toggle, no Summary panel) — byte-identical, pinned;
|
||||||
|
- `"additive"` (with `page_urls`): the text doc renders **exactly as a non-qualifying text doc** (Summary panel + content, the pre-148 branch reused), PLUS a `Text`/`Pages` toggle above the content: `<div class="doc-view-toggle" role="group" aria-label="Document view">` with two `<button type="button" aria-pressed>`; **default `Text`** (LOCKED A6 — no sparse-ratio default flip); switching to `Pages` appends/reveals the `docPagesStrip` (imgs `loading="lazy"` — hidden-until-shown never fetches) and hides the text block; switching back reverses. State is per-render (no persistence). createElement + properties/textContent only (the never-build-HTML rule); styles.css gains `.doc-view-toggle` per the house button idiom (focus-visible ring, ≥4.5:1, pressed state is text + style, never color alone — B5).
|
||||||
|
- no `page_urls`: byte-identical today's text doc (untouched).
|
||||||
|
|
||||||
|
**D5 — the live repair (task 05, records; no code).** With D3 shipped, the next production Sync takes the `Letumdei` doc's unchanged fast path → re-gate: usable text layer + ≥1 sparse candidate (page 36) → mode `additive` stamped → all 36 copies missing → re-rendered → the viewer shows the book's text with the Pages toggle. The IKEA-class doc (bytes unchanged, copies present, mode NULL pre-phase) gets mode `additive` stamped with zero render work. The E2E fixture reproduces the exact live shape (one illustration block on the last page, copies deleted out from under a stamped `page_count`).
|
||||||
|
|
||||||
|
**Invariants (all tasks).** The changed-path eligibility rule (`fallback_fired or pictorial_described`), every vision pass, the `/pages/{n}` route, the fallback-class strip DOM, the non-qualifying payloads, all SSE frame shapes, the mock-LLM markers, and the prune (`_delete_page_copies` at prune time) are byte-identical. `PLAN.md` untouched (the §7 viewer line describing the strip is folded at the next redo; AGENTS.md's viewer bullets get the additive note in the final pass).
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_schema_wire_strip_mode.md` — the `page_strip_mode` column + migration 0027 + the `DocContent` field + the payload builder + unit pins (D1).
|
||||||
|
2. `02_persist_write_first.md` — `_persist_pdf_pages` write-before-delete + atomic replace + the `strip_class` stamp (D2).
|
||||||
|
3. `03_fast_path_regate.md` — `_regate_pdf_strip` on the unchanged paths: mirror gate, heal, repair, log line (D3).
|
||||||
|
4. `04_viewer_additive_toggle.md` — `renderDocument` mode branch + Text/Pages toggle + styles + frontend unit pins (D4).
|
||||||
|
5. `05_e2e_selfheal_pins.md` — the dedicated Playwright suite (additive view, primary byte-identity, heal, repair arm), the final gates, the docs pass (D5 + gates).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_pdf_page_copies.py` (D2: write-first order, atomic replace, partial-write cleanup, the class stamp; D1: column defaults), `tests/unit/test_importer_unchanged_fast_path.py` (D3: every re-gate arm incl. kill-switch, heal, repair, fail-softs, the zero-extra-read guard for non-strip docs), `tests/unit/test_docs_api.py` or the phase-148 payload-pin file (D1: the key present/absent rule, the byte-identity of non-qualifying payloads), a frontend source-contract file per the house style (D4: the mode branch, the toggle semantics, the never-build-HTML rule, the primary-DOM byte-identity).
|
||||||
|
- E2E: `tests/e2e/test_pdf_strip_additive_selfheal.py`, **in isolation** (`uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov`); `test_pdf_pages_in_sources.py` / `test_pictorial_pdf.py` / `test_garbage_pdf.py` stay green (the re-cut pins included).
|
||||||
|
- Coverage: **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A pictorial-class PDF opens in the viewer showing its text + Summary panel with a Text/Pages toggle (Text pressed by default); Pages shows the strip and every page image loads (200); a fallback-class PDF's strip-only DOM is byte-identical to pre-phase (no toggle).
|
||||||
|
- [ ] The changed path writes copies before deleting any (measured by the unit order pin); a mid-write failure leaves the prior copies intact and no orphans.
|
||||||
|
- [ ] Deleting a strip's copies on disk + a re-sync (unchanged bytes) restores them and stamps the mode (the E2E heal test); a stale `page_count` on a now-ineligible doc is cleared on re-sync with the copies removed (the E2E repair test); zero vision/LLM calls on the fast path (asserted via the mock).
|
||||||
|
- [ ] Non-qualifying-doc payloads and the primary-class DOM are byte-identical; the full `uv run pytest` suite is green; coverage >90%; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass (the viewer description gains the additive-mode sentence; `PLAN.md` untouched per the invariants).
|
||||||
|
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate. Post-deploy (owner runtime, recorded here): the next production Sync self-heals `Letumdei` per D5 — no manual script.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — additive strip**: usable-layer (pictorial-class) PDFs show text + Summary + a Text/Pages toggle; the text is NEVER hidden for them; fallback-class keeps the strip-only view byte-identical (owner-confirmed 2026-10-07 — supersedes phase-148's "strip instead of the text content" for the pictorial class only).
|
||||||
|
- **A2 — the wire signal**: `page_strip_mode: "primary"|"additive"` ABSENT (never null) when `page_count` is NULL; absent ⇒ viewer treats `"primary"`; non-qualifying payloads byte-identical (owner-accepted default, 2026-10-07).
|
||||||
|
- **A3 — the column**: `documents.page_strip_mode` VARCHAR(16) nullable, migration 0027 with a tested downgrade (A13); stamped/cleared exactly with `page_count`.
|
||||||
|
- **A4 — write-first atomic copies**: render → atomic per-file write (tmp + `os.replace`) → stale-tail delete; a rollback can never leave the row over zero files; the fail-soft contract unchanged (owner-confirmed 2026-10-07).
|
||||||
|
- **A5 — the deterministic re-gate**: unchanged paths mirror the changed-path gate exactly (`is_usable_pdf_text` / `pictorial_pages` + the `pdf_pictorial` kill switch, current settings), re-render missing copies (CPU-only, no vision), or clear a stale strip; only rows with `page_count` pay the extra PDF parse; fail-soft on any re-gate exception (owner-confirmed 2026-10-07).
|
||||||
|
- **A6 — default view = Text** for additive docs (no sparse-ratio default flip; the illustration descriptions already live in the text via phase 140) (owner-accepted default, 2026-10-07).
|
||||||
|
- **A7 — the changed-path eligibility RULE is unchanged** (`fallback_fired or pictorial_described`, phase-148 lock intact); this phase changes only the persist ordering, the recovery, and the view consequence.
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ alembic/ .agents/phases/ && git commit --no-gpg-sign -m "fix(pdf): additive text+pages viewer for pictorial PDFs, write-first atomic page copies, and a deterministic fast-path re-gate that self-heals lost strip images"
|
||||||
|
```
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Task 01 — The `page_strip_mode` column + wire signal (phase-163 D1, LOCKED A2/A3)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D1 — the live modal showed the strip replacing the text of a genuinely textual book; the viewer needs an explicit class signal (fallback class = strip primary; pictorial class = additive) that never disturbs non-qualifying payloads.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`documents.page_strip_mode` (VARCHAR(16), nullable, migration 0027 with a tested downgrade) exists and `DocContent` carries `page_strip_mode` with the `image_url` omission rule — ABSENT (never null) when `page_count` is NULL, present for qualifying docs — so the frontend (task 04) can branch on the class, and every non-qualifying doc's payload stays byte-identical to pre-phase.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/models.py` — `Document.page_strip_mode: Mapped[str | None]` (VARCHAR(16), nullable, default NULL), docstring right after `page_count` (phase 148's column): the strip CLASS — `"primary"` = the phase-133 fallback class (the strip IS the view — the screenshots are the content), `"additive"` = the phase-140 pictorial class (the text view + the strip behind a toggle); NULL = no strip. Stamped/cleared exactly with `page_count` (tasks 02/03 own the writes; this task only adds the column).
|
||||||
|
2. `alembic/versions/0027_documents_page_strip_mode.py` — add the column; downgrade drops it. Follow `0026_documents_page_count.py` exactly (revision naming, `nullable=True`, no index). `uv run alembic upgrade head && uv run alembic downgrade -1 && uv run alembic upgrade head` round-trips.
|
||||||
|
3. `app/schemas.py` — `DocContent` gains `page_strip_mode: str | None = None` beside `page_urls`, docstring: the phase-163 strip class; ABSENT (never null — the `image_url` omission rule, `exclude_none`-style serialization already in place for `image_url` — verify how `DocContent` drops `image_url=None` and apply the SAME mechanism) when there is no strip; `"primary"`/`"additive"` otherwise. A pre-phase row (column NULL while `page_count` is set) serializes ABSENT — the viewer treats absent as `"primary"` (task 04).
|
||||||
|
4. `app/api/docs.py` — the content payload builder (the `page_urls=(…)` block, ~line 250): pass `page_strip_mode=doc.page_strip_mode` (the None-drops-the-key serialization handles the absent rule). The `/documents/{doc_id}/pages/{n}` route is NOT touched.
|
||||||
|
5. Unit tests — extend the phase-148 payload-pin file (find where `page_urls` serialization is pinned — `tests/unit/test_docs_api.py` or `test_pdf_page_copies.py`):
|
||||||
|
- a qualifying doc with mode set: the serialized payload carries `"page_strip_mode": "primary"` / `"additive"` next to `page_urls`;
|
||||||
|
- a doc with `page_count` set and mode NULL (pre-phase row): the key is ABSENT (not null) and `page_urls` still present — the exact live `Letumdei` state;
|
||||||
|
- a non-qualifying doc (`page_count` NULL): NO `page_strip_mode` and NO `page_urls` keys — pin the payload byte-identity against the current expected dict;
|
||||||
|
- `page_strip_mode` never appears alone: with `page_count` NULL it is absent even if the column somehow holds a value (the builder keys the whole strip block off `page_count` — if the current builder does NOT, make it so).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; the existing `page_urls` pins stay green unmodified except where a test asserts an exact payload dict (then re-cut in this task). Coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] `uv run alembic upgrade head` applies 0027; downgrade removes the column; upgrade again is clean.
|
||||||
|
- [ ] The three payload shapes (mode set / mode NULL with strip / no strip) serialize exactly per D1, pinned by unit tests.
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# Task 02 — Write-first atomic page-copy persist + the strip-class stamp (phase-163 D2, LOCKED A4/A7)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D2 — the live loss: the changed path deletes the old copies BEFORE rendering, outside the DB transaction, so any later per-file rollback (the `EmbeddingError` handler in the import loop) restores the stale `page_count` over ZERO files on disk (filesystem ops are not transactional). External wipes (`pdf_page_dir` cleaned) land the same state; the heal is task 03.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`_persist_pdf_pages` never removes a live copy before the replacement bytes are durably written: render → atomic per-file write (tmp + `os.replace`) → stale-tail delete, and the function stamps `page_strip_mode` from a `strip_class` keyword instead of the bare `eligible` boolean — the eligibility RULE (`fallback_fired or pictorial_described`, phase-148 lock) is unchanged at the call site.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — `_persist_pdf_pages(doc, *, full_path, page_dir, strip_class, source, rel)` (replaces the `eligible: bool` keyword):
|
||||||
|
- `strip_class` is `"fallback" | "pictorial" | None` (the caller passes the CLASS — see 2).
|
||||||
|
- `strip_class is None`: clear BOTH `doc.page_count = None` and `doc.page_strip_mode = None`, THEN `_delete_page_copies(doc, page_dir)` (the deletion-first order is correct HERE — there is no replacement set to protect; note the helper reads `doc.page_count`, so call it BEFORE nulling or pass the old count — check the current order and keep the deletion actually deleting: today the helper runs FIRST while the count is still set; preserve that behavior).
|
||||||
|
- eligible: `render_pdf_pages(full_path)` → the no-pages guard unchanged (`page_count`/mode NULL, warning) → `page_dir.mkdir` → **write loop FIRST**: every non-empty page to `<doc-id>-p{n}.png.tmp` then `os.replace` onto the final name (atomic per file — a reader never sees a torn PNG); collect the written names → **then** delete the stale set: `for n in range(1, doc.page_count + 1)` beyond/present-but-not-written → unlink (the shrunk-PDF tail; reuse `_delete_page_copies` semantics by deleting the old extent files that are NOT in the written set — a file about to be overwritten is not deleted at all).
|
||||||
|
- the failure path: on any exception in the render/write, unlink this call's `.tmp` AND final-name files it wrote (track both), then re-raise (the caller's fail-soft handler — `page_count` NULL + warning — is extended to also null `page_strip_mode`). The OLD copies are untouched by a failed call (they were never deleted up-front — the crash/rollback orphan state is structurally impossible now).
|
||||||
|
- success tail: `doc.page_count = len(pages)` AND `doc.page_strip_mode = "primary"` if `strip_class == "fallback"` else `"additive"`.
|
||||||
|
- docstring: the write-first rationale (the 2026-10-07 live loss: delete-first + a later rollback = the row over zero files) + the per-file atomicity note.
|
||||||
|
2. `app/rag/importer.py` — the `_index_pdf_file` call site (~line 1671): `strip_class = "fallback" if fallback_fired else ("pictorial" if pictorial_described else None)` passed as the keyword; the `eligible=fallback_fired or pictorial_described` expression is GONE (the class carries the same truth table — pin it in the docstring: class non-None ⇔ eligible, the phase-148 rule verbatim). The `except Exception` fail-soft handler nulls mode too.
|
||||||
|
3. `tests/unit/test_pdf_page_copies.py` — extend (the file's existing fixture pattern):
|
||||||
|
- **the order pin**: patch `_delete_page_copies`-class file ops / use a real tmp dir — after a successful re-render of a CHANGED doc, the new files exist and the stale tail (old count > new count) is gone; the overwritten pages keep the NEW bytes;
|
||||||
|
- **the failure pin**: a render that raises mid-write (the 2nd of 3 pages) → the FIRST render's old copies still on disk (the rollback-orphan state is unrepresentable), no `.tmp`/partial orphans from this call, the caller's fail-soft fires (`page_count`/mode NULL + the warning);
|
||||||
|
- **the atomic pin**: the writer uses tmp + `os.replace` (spy or a failing-replace test — a reader opening the final path never sees truncated bytes; assert the tmp name pattern is used and `os.replace` is called);
|
||||||
|
- **the class stamp**: fallback → `page_strip_mode == "primary"`; pictorial → `"additive"`; ineligible → both NULL and old copies deleted;
|
||||||
|
- the existing phase-148 pins (gap = per-page `b""` leaves no file and counts in `page_count`; the shrunk tail) stay green, re-cut only where the deletion order changes expectations.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A successful changed-path render: new copies durable BEFORE any old copy is removed (measured by the order pin); the shrunk tail still cannot leak.
|
||||||
|
- [ ] A mid-write failure leaves the prior copies intact and zero orphans (files on disk exactly the prior set), `page_count`/`page_strip_mode` NULL, sync continues (the fail-soft unchanged).
|
||||||
|
- [ ] `strip_class` carries the exact phase-148 eligibility truth table; no call site passes a bare `eligible` anymore.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_pdf_page_copies.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
# Task 03 — The deterministic fast-path re-gate: heal & repair (phase-163 D3, LOCKED A5)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D3 — with the copies lost (external wipe, or the pre-phase-163 delete-first window), the phase-147 fast path assumes "the copies persist" and NEVER re-renders: the live `Letumdei` row (36 `page_urls`, zero files, unchanged bytes on every sync since) is the proof. The unchanged paths must re-gate the strip deterministically — no LLM/vision calls — heal missing copies, and clear strips that no longer qualify.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`_regate_pdf_strip(session, doc, *, full_path, llm, summary)` runs on BOTH unchanged paths (the `file_hash` fast path and the content-digest-hit path), guarded by `doc.page_count is not None`: it mirrors the changed-path gate from the current bytes + settings, re-stamps `page_strip_mode`, re-renders when any copy is missing, or clears a stale strip — one greppable INFO line only when it ACTS, fail-soft on every exception.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/rag/importer.py` — new `_regate_pdf_strip(session, doc, *, full_path, llm, summary) -> None`:
|
||||||
|
- **mirror the changed-path gate exactly** (the truth table in the phase overview D3):
|
||||||
|
`text = extract_pdf_text(full_path)`; `is_usable = is_usable_pdf_text(text)`;
|
||||||
|
- `not is_usable` → class `fallback` (mode `"primary"`) — the phase-133 signal verbatim;
|
||||||
|
- elif `llm.settings.pdf_pictorial` and `pictorial_pages(extract_pdf_page_texts(full_path), min_chars=llm.settings.pdf_pictorial_min_chars)` non-empty → class `pictorial` (mode `"additive"`) — **the existing gate verbatim** (text-carrying pages under `min_chars` only; no-text pages are NOT candidates — `pictorial_pages`'s own contract; the re-gate adds no new eligibility semantics);
|
||||||
|
- else → **repair arm**: `_delete_page_copies(doc, page_dir)` (while `page_count` is still set — the helper reads it), then `doc.page_count = None`, `doc.page_strip_mode = None`; INFO `import: PDF strip re-gate source=… path=… — cleared stale strip`.
|
||||||
|
- **mode re-stamp** (eligible arms): a mode change (NULL→stamped, or the owner's kill-switch/class flip) writes `doc.page_strip_mode` and, when it changes an already-set mode, INFO `… mode <old>-><new>`; a NULL→stamp on existing rows is silent-beneficial (fold into the same line only when work follows).
|
||||||
|
- **copy verify + heal** (eligible arms): `page_dir = Path(llm.settings.pdf_page_dir).expanduser()`; missing = any of `page_dir/f"{doc.id}-p{n}.png"` for `n in 1..doc.page_count` absent. Any missing → `render_pdf_pages(full_path)`: the no-pages result (unopenable — pathological right after the fast path hashed the file) keeps the row untouched + WARNING `import: PDF strip re-gate source=… path=… — re-render produced no pages (state kept)`; else WRITE EVERY page via task-02's atomic writer (tmp + `os.replace`) — reuse it (extract the shared writer helper from task 02 rather than duplicating), then delete the stale tail beyond the new count; `doc.page_count = len(pages)` — a drift from the old count adds a WARNING naming both counts; INFO `… healed N pages`.
|
||||||
|
- **fail-soft**: the whole body in `try/except Exception` → `logger.warning` (the re-gate never fails a sync; the row keeps its pre-re-gate state — but note the repair arm's DB writes and the heal's row writes are in the caller's transaction, so an exception AFTER them rolls back naturally with the file anyway).
|
||||||
|
- docstring: the 2026-10-07 live state as the rationale; the zero-LLM-call property (render is CPU — A21-safe); the guard means every non-strip file's fast path keeps its zero-extra-read contract.
|
||||||
|
2. `app/rag/importer.py` — `_unchanged_doc(...)`: gains the optional keyword `full_path: Path | None = None` (defaults to `Path(doc.full_path)`); at its tail, `if doc.page_count is not None and str(doc.path).lower().endswith(".pdf"): _regate_pdf_strip(session, doc, full_path=…, llm=llm, summary=summary)`. Call sites: the `_index_file` fast path passes the live `full_path`; the branches' content-digest-hit calls pass their own `full_path` (every call site already has it). Docstring: the phase-147 "the copies persist" paragraph is replaced — "unchanged files re-gate the PDF strip deterministically (phase 163)".
|
||||||
|
3. `tests/unit/test_importer_unchanged_fast_path.py` — extend (the file's fast-path fixture pattern; the mock embedder asserts ZERO chat/vision calls across every case):
|
||||||
|
- **heal**: a stamped strip doc (`page_count=3`, mode NULL or set), copies deleted out from under it, unchanged bytes → sync → the 3 files exist (PNG bytes), mode stamped `"additive"` (usable layer + sparse candidate), INFO `healed 3 pages`, `page_count` unchanged (no drift);
|
||||||
|
- **fallback heal**: an unusable-layer doc → mode `"primary"`, copies restored;
|
||||||
|
- **no-op**: all copies present, mode set → zero file writes, zero INFO lines (pin "INFO never fires per unchanged doc" — assert the log records contain no re-gate line);
|
||||||
|
- **repair**: a row with `page_count` set (copies present) on a PDF whose gate now says ineligible (clean text, zero sparse pages) → cleared (both columns NULL, copies deleted), INFO `cleared stale strip`;
|
||||||
|
- **kill switch**: `pdf_pictorial=False` + pictorial-class doc → repair arm clears the strip (the changed-path semantics mirrored);
|
||||||
|
- **fail-soft**: `render_pdf_pages` patched to raise → sync completes, row keeps its prior state, WARNING, no copies touched;
|
||||||
|
- **guard**: a non-PDF unchanged file and a PDF with `page_count` NULL → `extract_pdf_text` NEVER called (spy) — the zero-extra-read contract intact.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the cases above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Unchanged re-sync restores deleted copies (files + row + route-serveable) and stamps the mode — the live-loss state cannot persist past one sync (unit-proven; the E2E pins it in task 05).
|
||||||
|
- [ ] A stale strip on a now-ineligible doc (incl. via the kill switch) is cleared with its copies; the re-gate fires ZERO LLM/vision calls and no INFO when there is nothing to do.
|
||||||
|
- [ ] Non-strip fast paths perform zero PDF parsing (spy-pinned); every re-gate exception is fail-soft.
|
||||||
|
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py tests/unit/test_pdf_page_copies.py -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 04 — The additive viewer: text + Summary + the Text/Pages toggle (phase-163 D4, LOCKED A1/A6)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D4 — the live complaint: "Notice how all the pages are unavailable… it shouldn't be a pictorial pdf." One short map page flipped a 114 KB-text book into the phase-148 strip-only view (the strip renders INSTEAD of the text content AND the Summary panel). The owner's call: additive for the pictorial class — the text is never hidden; the strip is a toggle away.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`renderDocument` (the ONE shared core — the Sources modal AND `/document.html`) branches on `doc.page_strip_mode`: absent/`"primary"` keeps the current strip-only DOM byte-identically; `"additive"` renders the doc exactly as a text doc (Summary panel + content) plus a `Text`/`Pages` toggle (default `Text`) revealing the existing `docPagesStrip`.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/document.js` — the `renderDocument` strip branch (~line 374, `const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;`):
|
||||||
|
- `pageUrls` truthy AND `doc.page_strip_mode === "additive"`: fall through to the EXISTING text-doc rendering (the pre-148 branch — Summary panel + content, reused unchanged), THEN append the view toggle + the strip container:
|
||||||
|
- `<div class="doc-view-toggle" role="group" aria-label="Document view">` + two `<button type="button" class="doc-view-btn" aria-pressed>` — labels `Text` / `Pages`; initial state `Text` pressed (`"true"`), the strip NOT in the content flow;
|
||||||
|
- `Pages` click: `aria-pressed` flips; the text block + Summary panel hide (a class toggle, e.g. a `.doc-views[data-view="pages"]` wrapper — CSS-driven, never re-render), the `docPagesStrip(doc, pageUrls)` (the EXISTING builder, unmodified — lazy imgs never fetch while hidden) is appended once and shown;
|
||||||
|
- `Text` click reverses (the strip node stays mounted — no re-fetch on flip-flop);
|
||||||
|
- structure: wrap the text view + the strip in one `.doc-views` container appended to `contentEl` (both surfaces get it via the shared core; the wrapper exists ONLY on additive docs — every other doc's DOM is untouched).
|
||||||
|
- `pageUrls` truthy AND mode absent or `"primary"`: the CURRENT branch verbatim (strip only, no Summary panel, no toggle) — byte-identical DOM, pinned;
|
||||||
|
- falsy `pageUrls`: untouched.
|
||||||
|
- the never-build-HTML rule: createElement + properties/textContent only (the labels are fixed strings — no doc-derived text inside the toggle); the `strip_mode` read is `doc.page_strip_mode === "additive"` (anything else ⇒ primary — the tolerant-reading house rule).
|
||||||
|
- the module docstring: the phase-163 paragraph (the additive class contract, LOCKED A1/A6) replacing the phase-148 "INSTEAD of the text content AND the .doc-summary panel" sentence's scope (it now reads: primary class only).
|
||||||
|
2. `frontend/assets/styles.css` — `.doc-view-toggle` (house button idiom beside the existing segmented/pill controls — check `.format-badge`/the modal action buttons for the closest idiom): the two buttons adjacent, pressed state via `aria-pressed="true"` styling (weight/background pair — **text + style, never color alone**, B5), `:focus-visible` 3px ring, ≥4.5:1 text contrast on both states, ≥44px touch targets on mobile (§7.1); `.doc-views[data-view="pages"]` hides the text block + Summary panel and shows the strip; default (`data-view="text"`) hides the strip container entirely.
|
||||||
|
3. Unit (frontend source-contract — the house pattern; extend the file that already pins the phase-148 strip DOM — locate it via `docPagesStrip` in `tests/unit/`, likely `test_pdf_page_copies.py` / a viewer-contract file):
|
||||||
|
- the additive branch builds the `.doc-views` wrapper + the toggle with the exact labels/roles/`aria-pressed` initial state (source-regex or the file's established DOM-assertion style);
|
||||||
|
- the primary/absent branch contains NO `doc-view-toggle` string and the strip-only structure is byte-identical to the pre-phase pins;
|
||||||
|
- the toggle handler mutates only `aria-pressed` + the `data-view` attribute (no innerHTML anywhere in the new code path — the never-build-HTML pin);
|
||||||
|
- `loading = "lazy"` remains on the strip imgs (the existing pin re-runs green).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the contract pins above; coverage **>90%** on `app/` (no `app/` change in this task — the gate must simply stay green).
|
||||||
|
- The phase's E2E (task 05) exercises the real DOM; `test_pdf_pages_in_sources.py` (the phase-148 E2E) is re-cut ONLY where a modal-DOM assertion meets the additive fixture (the fixture's mode stamp decides; prefer fixture docs that keep primary semantics for the existing pins).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] An additive doc shows Summary + text with `Text` pressed by default; one click shows the strip (every page img loads); clicking back shows the text with zero refetch (the strip node persists — E2E-pinned in task 05).
|
||||||
|
- [ ] A primary/absent-mode doc's viewer DOM is byte-identical to the pre-phase structure (unit pin green); a text doc and every non-PDF are untouched.
|
||||||
|
- [ ] The toggle passes the WCAG basics: labeled group, `aria-pressed` state as text + style, focus-visible ring (checked against §7.2; the E2E asserts the attributes).
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Task 05 — The dedicated E2E: additive view, primary byte-identity, heal & repair; the final gates (phase-163 D5)
|
||||||
|
|
||||||
|
**Phase:** `163_pdf_strip_additive_selfheal` · **Source:** phase overview D5 + the AGENTS.md per-phase E2E rule (A16).
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
**Pre-fix evidence (measured on the live server, 2026-10-07 — recorded here per house practice):** `Letumdei_ Heirs to the Imperium.pdf` — payload carries 36 `page_urls`; `GET /api/documents/b72f5515-dc40-4167-98ff-179a6503e6e5/pages/1,2,18,36` → **404/404/404/404** (page 37 → 404, the out-of-range arm); the content is 114 389 chars of genuine extracted text with 36 `---PAGE-N---` markers and exactly ONE `[Page 36 illustration: …]` block; the same server serves `besta-frame-white__AA-1272068-11-100.pdf` page 1 → **200** (the dir is intact; only the 10/1-indexed doc's copies were lost). The viewer showed "Page N unavailable." on every page with the text hidden.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One Playwright suite proves the whole phase end to end — the additive view with the toggle, the fallback class's byte-identical strip-only DOM, the disk-level heal, and the repair arm — and the phase's final pass runs every gate plus the full documentation consistency review.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_pdf_strip_additive_selfheal.py` — new suite (run in isolation; read `tests/e2e/test_pdf_pages_in_sources.py` and `tests/e2e/test_pictorial_pdf.py` first and reuse their synthetic-PDF builder + fixture-KB pattern; deterministic mock LLM, the default):
|
||||||
|
- **fixture**: source `strip_fixture` with (a) `manual.pdf` — usable text layer, 3 text pages + 1 sparse `<100-char` map-style page (the pictorial class — the Letumdei shape in miniature); (b) `scanned.pdf` — the garbage/unusable-layer fixture (the phase-133 pattern) → fallback class; (c) `plain.pdf` — clean text, zero sparse pages → no strip.
|
||||||
|
- **assert 1 (additive view)**: index → the modal for `manual.pdf`: mode stamped `additive` (via the content API), the text + Summary panel visible by default, the `Text`/`Pages` toggle present with `Text` `aria-pressed="true"`; click `Pages` → the strip renders, EVERY page `<img>` has `naturalWidth > 0` (routes serve — no "unavailable" note anywhere); click `Text` → text visible, strip hidden, the strip node still mounted (no refetch: assert the img count does not reset).
|
||||||
|
- **assert 2 (primary byte-identity)**: the modal for `scanned.pdf`: the strip-only DOM (no `.doc-view-toggle`, no Summary panel) and the "Page N" figures match the phase-148 structure (the existing suite's DOM assertions adapted); `plain.pdf` renders the plain text doc with no strip artifacts.
|
||||||
|
- **assert 3 (heal)**: for `manual.pdf`, delete ALL `pdf_page_dir/<id>-p*.png` (the fixture's page dir) + confirm a page route 404s → run a sync (unchanged bytes → the fast path) → the files exist, the page route 200s, the payload still carries `page_urls` + `additive`, the viewer's Pages view loads every img (the live-loss state, healed by the next sync — the exact D5 promise).
|
||||||
|
- **assert 4 (repair)**: direct-DB seed — set `page_count` (and stale copies on disk) on `plain.pdf`'s row (the stale-eligible state no current index produces) → sync → `page_count`/`page_strip_mode` NULL (read through the fixture session), the stale copies deleted, the modal shows the plain text (the strip gone).
|
||||||
|
- **assert 5 (no vision)**: the whole suite's syncs ran against the deterministic mock — no chat/vision endpoint hits beyond the initial index's own passes (the heal burns zero LLM calls; assert via the mock's call counter or the suite's request log, the house pattern).
|
||||||
|
2. Final pass (the phase pipeline runs this after the tasks): full-suite + gates:
|
||||||
|
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**;
|
||||||
|
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` green **in isolation**; `tests/e2e/test_pdf_pages_in_sources.py`, `test_pictorial_pdf.py`, `test_garbage_pdf.py`, `test_document_viewer*.py`, `test_image_documents.py` green;
|
||||||
|
- `uv run ruff check . && uv run pyright` clean; the alembic 0027 up/down/up round-trip clean.
|
||||||
|
3. Documentation & consistency (mandatory): review `README.md` and `AGENTS.md` IN FULL against the post-phase app — the viewer description gains the additive-mode sentence (text + Text/Pages toggle for pictorial-class PDFs; the fallback class keeps the strip), the importer notes (write-first copies, the fast-path re-gate + its INFO line) — and fix EVERY inconsistency found, including pre-existing ones not caused by this phase. `PLAN.md` is NOT touched (its §7 strip line folds at the next redo — recorded in the phase overview invariants).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the E2E + gates task; its own new-code coverage rides the suite (page route + fixture helpers are test-side; `app/` coverage stays **>90%**).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All five E2E asserts green in isolation; the four existing viewer/PDF suites green unmodified (except explicitly re-cut pins, each named in the commit body).
|
||||||
|
- [ ] The heal assert proves: copies deleted externally → one unchanged-byte sync → files restored, routes serve, viewer complete (the live `Letumdei` failure mode is impossible after this phase).
|
||||||
|
- [ ] `uv run pytest` fully green; coverage >90%; ruff + pyright clean; migration round-trip clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full, every inconsistency fixed (the additive-viewer sentence included); the phase dir moves to `complete/` and the single `--no-gpg-sign` commit (see the overview) is made by the pipeline gate.
|
||||||
|
- [ ] Owner follow-up recorded (not code): after deploy, the next production Sync self-heals `Letumdei` (D5) — verify once via the Sources modal (text visible, Pages loads all 36) and `grep` the server log for the one `import: PDF strip re-gate … healed 36 pages` line.
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# Phase 164 — Chat failure honesty: the shared error-detail serializer, a distinct pre-stream failure line, and caps that accept a full-length answer
|
||||||
|
|
||||||
|
**Source:** owner live report, 2026-10-07 — brain.reeseapps.com, right after the live answer cap was raised to `BOR_LLM_MAX_OUTPUT_TOKENS=131072`: when an answer hit the length limit, the next turn's bubble read *"My answer didn't make it — the connection dropped. Use Retry to ask again. Failed [object Object]."* The failure chain (traced in code, 2026-10-07): a long answer (> 32 000 chars) lives in `bor.chat.v1` and rides back to the server as `history` on the next `POST /api/chat`; `HistoryTurn.text` (`max_length=32_000`) rejects it with a **422 whose `detail` is a LIST OF OBJECTS** (`[{loc, msg, type}, …]`); the chat's pre-stream error path (`frontend/assets/app.js` ~3956: `if (body.detail) detail = body.detail`) has **no type guard** — `new Error(<array>)` stringifies to `"[object Object]"`, which the zero-frame settle shows as the fixed `FAILED_TURN_TEXT` ("the connection dropped" — untrue: the server WAS reachable and answered 422) plus the garbage detail. This is the SECOND occurrence of the same class: `HistoryTurn`'s own docstring records the 4 000 → 32 000 raise for an identical 422 (phase-42 E2E); the 32 000 ceiling broke again the moment outputs could exceed it — **at the DEFAULT `max_output_tokens=32_768` (~130k chars), not only at 131072**.
|
||||||
|
|
||||||
|
**Story:** n/a (owner-directed bug fix; extends the phase-120 failed-turn contract, the phase-114 hint channel, the phase-83 SEC-05 boundary caps, the phase-74 history budgets).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A failed turn states **what actually happened**: pre-stream HTTP rejections get their own honest line (not "the connection dropped") with the validation message serialized verbatim — never `[object Object]` — and a full-length answer (up to a 131 072-token cap ≈ 524k chars) flows through the whole chain: accepted as `history` on the next turn, and saveable/restorable as part of a saved chat. The anonymous saved-chat surface keeps a bounded worst case (the phase-83 SEC-05 property survives the raise).
|
||||||
|
|
||||||
|
## Dependencies
|
||||||
|
- `83_chat_save_payload_limits` (complete) — the SEC-05 caps this phase raises TOGETHER (`ChatMessage.text`/`thinking` mirror `HistoryTurn`; the mirror invariant must stay true).
|
||||||
|
- `74`-line history budgets (complete; `history_to_messages`, `history_max_turns=40` / `history_max_chars=24_000`) — the LOCKED budget trim (owner 2026-09-08: overflow turns dropped WHOLE, never cut) — **untouched**; it is the real prompt-side gate and stays so.
|
||||||
|
- `120_failed_turn_persistence` (complete; the `FAILED_TURN_TEXT` / `finalizeFailedTurn` / `.failed-note` contract) and `114` hint channel (`err.hint` → banner) — the bubble/note mechanics this phase extends.
|
||||||
|
|
||||||
|
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||||
|
|
||||||
|
**D1 — the caps chain up, together (task 01, LOCKED A1).** `app/schemas.py`:
|
||||||
|
- `HistoryTurn.text` / `HistoryTurn.thinking`: `max_length=600_000` (rationale IN the docstring: a full-cap answer at the owner's `BOR_LLM_MAX_OUTPUT_TOKENS` up to 131 072 ≈ 524 300 chars at ~4 chars/token — 600 000 is that with margin; the budget trim (locked A3, 2026-09-08) remains the real prompt gate — a turn over `history_max_chars` is dropped WHOLE, so a long answer trims itself out of later prompts; this phase fixes the 422, NOT the trim (its locked semantics are a separate owner decision, recorded in the overview));
|
||||||
|
- `ChatRequest.history` list: `max_length=100` → `500` (the same "DoS sanity ceiling only" docstring, now also noting the client sends the whole `bor.chat.v1` minus the current question);
|
||||||
|
- `ChatMessage.text` / `ChatMessage.thinking`: `max_length=600_000` (MIRRORS `HistoryTurn` — the phase-83 invariant "a saved chat can never legitimately carry more than the chat path accepts" must stay TRUE: raise both or neither);
|
||||||
|
- `SavedChatCreate.messages` / `SavedChatUpdate.messages`: `max_length=200` → `500` (symmetry with the history list) PLUS a new `model_validator` on both: **total chars across all messages (`len(text) + len(thinking or "")`) ≤ 8_000_000** — this bounds the ANONYMOUS SEC-05 surface at the same order as the audited pre-phase worst case (200 × 64k ≈ 12.8 MB) while fitting ~15 full-cap answers; the validator error surfaces as the standard 422 (`ValueError` message names the budget);
|
||||||
|
- `ChatRequest.message`: `max_length=4000` UNCHANGED (the composer textarea carries the cap client-side — pin it).
|
||||||
|
|
||||||
|
**D2 — the shared `errorDetailText` serializer (task 02, LOCKED A2).** `frontend/assets/app.js` exports `errorDetailText(value, fallback)` (exported const/function, the `TURN_TIMEOUT_MS` unit-pin idiom): a non-empty **string** → verbatim; an **array** → per-entry `entry.msg` when the entry is an object else `String(entry)`, empties dropped, `"; "`-joined, all-empty → `fallback`; an **object** → `String(entry.msg)` when present, else `fallback`; nullish / empty / anything else → `fallback`. NEVER `JSON.stringify` a raw detail (the FastAPI 422 entries carry the FULL `input` — the whole conversation — which must never reach the DOM; only `msg` and plain strings serialize). Applied at the TWO unguarded chat paths: the pre-stream `!res.ok` handler (~3956 — `detail = errorDetailText(body.detail, \`Brain's API answered with HTTP ${res.status}.\`)`) and the SSE `error`-frame throw (~4353 — `new Error(errorDetailText(ev.detail, "Something went wrong on my side."))`, the `err.hint` line untouched). The seven EXISTING guarded copies (`app.js` tuning form, `tuning.js`, `theme.js`, `document.js`, `git-sources.js`, `doc-edit.js`) are behaviorally correct and stay byte-identical (no opportunistic refactor — the new helper matches their semantics for the shared shapes, generalized to arrays-of-strings and multi-entry joins).
|
||||||
|
|
||||||
|
**D3 — the distinct pre-stream failure line (task 03, LOCKED A3/A4).** `app.js`:
|
||||||
|
- `const FAILED_REQUEST_TEXT = "I couldn't take that — my API rejected the request. Use Retry to ask again.";` (sibling of `FAILED_TURN_TEXT`, its docstring: the PRE-STREAM HTTP-rejection settle — the server answered, the request did not pass; the real detail rides the `.failed-note` + persisted `error` key exactly as `FAILED_TURN_TEXT` does);
|
||||||
|
- the pre-stream `!res.ok` throw carries `err.bubbleText = FAILED_REQUEST_TEXT` and `err.hint = "The app answered — it rejected the request before the model ran. Nothing was lost but this send."` (the phase-114 banner-hint channel);
|
||||||
|
- `runTurn`'s catch passes `bubbleText: err?.bubbleText` into `finalizeFailedTurn` (new options key); the zero-frame settle uses `bubbleText || FAILED_TURN_TEXT` — the persisted record text stays "what the user saw is what is stored";
|
||||||
|
- UNCHANGED settles: a transport-level failure (the `fetch` itself rejects — `TypeError`) has no `bubbleText` → keeps `FAILED_TURN_TEXT` (that one IS the connection); the SSE `error`-frame zero-frame path (a model-side failure) keeps `FAILED_TURN_TEXT`; the stop / abort paths are untouched. A partial-stream settle (`wrap` + `acc`) never shows a bubble text at all — unchanged.
|
||||||
|
|
||||||
|
**D4 — verification (task 04).** Dedicated Playwright suite `tests/e2e/test_chat_error_honesty.py` (isolation; deterministic mock LLM) with (a) a `page.route` stub of `POST /api/chat` returning the EXACT FastAPI 422 shape for a long-history turn → assert the bubble reads `FAILED_REQUEST_TEXT`, the `.failed-note` detail contains the pydantic `msg` verbatim ("String should have at most 600000 characters" — never `[object Object]`, never `input` text), the banner hint shows, Retry + Edit land, composer re-enables (§7.4); (b) a 500 stub with a non-JSON body → the `HTTP 500` default string line (the serializer's fallback path); (c) a real-server acceptance: send a turn whose history carries a ~525k-char prior brain answer → the request is ACCEPTED (no 422) and the turn settles normally with the mock (the budget trim may drop the giant turn WHOLE from the prompt — the locked A3 behavior; the test asserts acceptance + settle, not prompt content); (d) a stubbed SSE `error` frame carrying an ARRAY detail → the note serializes (client tolerance). Server-side acceptance/422 matrix for every raised cap + the total-chars validator rides integration pins in task 01 (the `test_chats_api.py` pattern).
|
||||||
|
|
||||||
|
**Invariants (all tasks).** The budget trim (`history_to_messages`, budgets, whole-drop, contiguous window) is byte-identical (locked A3 2026-09-08). `ChatRequest.message` 4000. The `FAILED_TURN_TEXT` wording, the phase-120 persistence keys (`failed`, `error` ≤500), Retry/Edit pairing, the SSE frame shapes, and the seven guarded frontend detail-readers are byte-identical. `PLAN.md` untouched.
|
||||||
|
|
||||||
|
## Tasks
|
||||||
|
1. `01_caps_long_answer_chain.md` — the `schemas.py` raise + the total-chars validator + docstrings + the unit/integration acceptance-rejection matrix (D1).
|
||||||
|
2. `02_error_detail_serializer.md` — the exported `errorDetailText` + the two chat call sites + unit pins (D2).
|
||||||
|
3. `03_prestream_failure_copy.md` — `FAILED_REQUEST_TEXT` + the `bubbleText` threading + the banner hint + the failed-turn pins (D3).
|
||||||
|
4. `04_e2e_error_honesty.md` — the dedicated E2E (D4) + the final gates + the docs consistency pass (incl. the proxy body-size deploy note).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: `tests/unit/test_schemas.py` (or the file pinning `HistoryTurn`/`ChatMessage` bounds — locate via `32_000` in `tests/`): every cap's boundary (600_000 accept / +1 reject; 500 vs 501 turns; the total-8M validator: accept ~15 full-cap messages, reject beyond), the 422 `detail` shapes; `tests/unit/test_frontend_error_detail.py` (new; the `test_frontend_*` house pattern): all serializer shapes incl. the exact live FastAPI 422 body; `tests/unit/test_frontend_failed_turn.py` extended: the `bubbleText` threading (pre-stream vs transport vs SSE-frame settles), the never-`innerHTML` rule.
|
||||||
|
- Integration: `tests/integration/test_chats_api.py` extended — a saved chat carrying ONE 525k-char answer saves 201 (the phase-83 surface stays bounded by the total validator), over-total rejects 422.
|
||||||
|
- E2E: `tests/e2e/test_chat_error_honesty.py` green **in isolation**; the phase-120 failed-turn E2E stays green (the `FAILED_TURN_TEXT` path unchanged for transport failures).
|
||||||
|
- Coverage **>90%** on `app/`; `uv run ruff check . && uv run pyright` clean.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] A pre-stream 422 shows the distinct honest line + the validation `msg` verbatim in the note — `[object Object]` is structurally impossible on the chat path (unit + E2E pinned; no raw `input` reaches the DOM).
|
||||||
|
- [ ] A history turn with a 525k-char brain answer is ACCEPTED (200 turn settles); >600k rejects 422; the saved-chat chain accepts the same answer inside a 201 and the anonymous surface stays bounded (total-8M validator pinned).
|
||||||
|
- [ ] Transport failures and SSE error frames keep `FAILED_TURN_TEXT` verbatim; stop/abort paths byte-identical; §7.4 failure feedback re-enables every control.
|
||||||
|
- [ ] The budget trim, `message` 4000, the seven guarded detail-readers, and all SSE shapes are byte-identical; full `uv run pytest` green; coverage >90%; ruff + pyright clean.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full in the final pass — includes the deploy note: the proxy `client_max_body_size` must accommodate the now-legal large chat POSTs (recommend ≥ 16 MB) — and every other inconsistency fixed.
|
||||||
|
- [ ] One `--no-gpg-sign` commit; the phase dir moves to `complete/` by the pipeline gate.
|
||||||
|
|
||||||
|
## Locked decisions
|
||||||
|
- **A1 — the caps chain together**: `HistoryTurn.text/thinking` AND `ChatMessage.text/thinking` → 600_000 (≈ 131 072 × 4 chars with margin); history list → 500; saved-messages list → 500 with a total-chars ≤ 8_000_000 model validator (the SEC-05 bounded-anonymous property preserved at the audited order); `message` 4000 unchanged; the phase-83 mirror invariant explicitly honored (owner-confirmed 2026-10-07 — "static generous caps", with the SEC-05 coupling added by this phase's audit of the chain — raise ONE side would break saves).
|
||||||
|
- **A2 — `errorDetailText`**: the shared serializer's exact semantics (string verbatim; array entries `msg`-else-String joined `"; "`; object `msg`-else-fallback; never `JSON.stringify`; never the `input`), exported from `app.js`, applied at the two unguarded chat paths ONLY; the seven guarded copies untouched (owner-confirmed 2026-10-07).
|
||||||
|
- **A3 — `FAILED_REQUEST_TEXT`**: distinct pre-stream HTTP-rejection bubble text verbatim; the persisted record is what the user saw (the phase-120 contract extended, not changed); transport failures keep `FAILED_TURN_TEXT` (owner-confirmed 2026-10-07 — "distinct pre-stream line").
|
||||||
|
- **A4 — the budget trim is NOT touched**: the locked 2026-09-08 whole-drop contiguous-window semantics stand; the consequence — a >24k-char answer is dropped whole from later prompts (follow-ups WORK with a trimmed window, without that answer's context) — is recorded here; making long answers useful in later prompts (bigger `BOR_HISTORY_MAX_CHARS` is pure env, or a smarter cut) is an owner call outside this phase.
|
||||||
|
- **A5 — PLAN.md untouched** (the caps rationale lives in the schema docstrings; folded at the next redo).
|
||||||
|
|
||||||
|
## Commit
|
||||||
|
```bash
|
||||||
|
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(chat): honest pre-stream failure line, a shared error-detail serializer, and history caps that accept a full-length answer"
|
||||||
|
```
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 01 — The caps chain up: 600k text, 500 turns, a bounded anonymous total (phase-164 D1, LOCKED A1/A4)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D1 — the live 422: a >32k-char answer rode back as `history` and rejected the NEXT turn. `HistoryTurn`'s docstring already records this failure class happening once (4 000 → 32 000, phase-42 E2E); the ceiling broke again the moment outputs could exceed 32k — at the DEFAULT `max_output_tokens=32_768` too.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
Every schema cap in the long-answer chain (`HistoryTurn` → `ChatRequest.history` → `ChatMessage` → saved-chat lists) accepts a full-cap answer (~524k chars at `max_output_tokens=131072`), the phase-83 mirror invariant between the chat path and the saved-chat path stays TRUE, and the anonymous saved-chat worst case stays bounded by a new total-chars validator.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `app/schemas.py` — `HistoryTurn`: `text`/`thinking` `max_length=32_000`/`32000` → **600_000** both; docstring: the cap accepts a full-length answer (`BOR_LLM_MAX_OUTPUT_TOKENS` up to 131 072 ≈ 524 300 chars at ~4 chars/token, plus margin — the 2026-10-07 live 422: the 32 000 ceiling rejected the FOLLOW-UP turn of a long answer, the same class as the recorded 4 000 → 32 000 raise); the real gate stays the locked budget trim (`history_to_messages` — overflow turns dropped WHOLE; a long answer trims itself out of later prompts, never a 422).
|
||||||
|
2. `app/schemas.py` — `ChatRequest.history`: `max_length=100` → **500**; the docstring sentence updates: still "a DoS sanity ceiling only — the config budgets do the real trimming (A3)", plus: the client sends the whole `bor.chat.v1` minus the current question, so the ceiling must clear long conversations.
|
||||||
|
3. `app/schemas.py` — `ChatMessage.text`/`thinking`: → **600_000** (docstring: the phase-83 SEC-05 mirror RAISED with `HistoryTurn` — the invariant "a saved chat can never legitimately carry more than the chat path accepts" survives the raise; raise ONE side and every done-time auto-save of a long answer 422s into the phase-113 quiet-failure).
|
||||||
|
4. `app/schemas.py` — `SavedChatCreate.messages` / `SavedChatUpdate.messages`: `max_length=200` → **500** AND a `model_validator(mode="after")` on each: total chars `sum(len(m.text) + len(m.thinking or "") for m in messages) <= 8_000_000` else `raise ValueError("saved chat exceeds the 8,000,000-character budget")` — docstring: bounds the ANONYMOUS SEC-05 worst case at the audited order (the pre-phase 200×64k ≈ 12.8 MB) while fitting ~15 full-cap answers; the 422 is the framework's standard validation response (the phase-83 A2 boundary-only style — NO route changes).
|
||||||
|
5. `ChatRequest.message`: **unchanged** `max_length=4000` (the composer textarea caps client-side — `app.js` ~1122 comment; pin it stays).
|
||||||
|
6. Unit — extend the file pinning these bounds (locate via `32_000` under `tests/unit/`):
|
||||||
|
- `HistoryTurn.text` 600_000 accept / 600_001 reject; same for `thinking`;
|
||||||
|
- `history` 500 accept / 501 reject; `message` 4000 boundary re-pinned unchanged;
|
||||||
|
- `ChatMessage.text/thinking` 600_000 accept / +1 reject;
|
||||||
|
- messages 500 accept; the total validator: ~15 × 524k messages REJECT with the budget-naming `ValueError` surfaced in the 422 `detail`, a saved chat with ONE 525k answer + normal turns ACCEPTS;
|
||||||
|
- the 422 `detail` SHAPE of a cap rejection is the pydantic list-of-objects (the serializer's task-02 input — pinned here as the server-side half of the contract).
|
||||||
|
7. Integration — `tests/integration/test_chats_api.py`: `POST /api/chats` with a one-525k-answer conversation → 201 (round-trips through JSONB — the DB is JSONB, no column changes needed — verify); over-total → 422; the existing phase-83 boundary matrix green (re-cut only the pins that assert the literal `32_000`/`200` values, naming each in the task's completion).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit + integration above; coverage **>90%** on `app/`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] The full-cap-answer conversation flows: accepted as `/api/chat` history (integration-level) AND saveable as a saved chat (201), with the 8M total cap the only new anonymous bound.
|
||||||
|
- [ ] `ChatMessage` mirrors `HistoryTurn` exactly (the phase-83 invariant re-stated in both docstrings, verified by identical-boundary tests).
|
||||||
|
- [ ] The budget-trim suites (`history_to_messages`) are untouched and green — D1 changes what ARRIVES, never what the trim keeps.
|
||||||
|
- [ ] `uv run pytest tests/unit tests/integration -v` green; full suite green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Task 02 — The shared `errorDetailText` serializer for the two chat paths (phase-164 D2, LOCKED A2)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D2 — the live "[object Object]": `frontend/assets/app.js` ~3956 does `if (body.detail) detail = body.detail` on a FastAPI 422 whose `detail` is a LIST OF OBJECTS; `new Error(<array>)` → message `"[object Object]"`. The SSE `error`-frame path (~4353 `new Error(ev.detail || …)`) is the second unguarded reader. The guarded idiom ALREADY exists in seven copies (`tuning.js`, `theme.js`, `document.js`, `git-sources.js`, `doc-edit.js`, `app.js`'s tuning form) — this task generalizes it ONCE and applies it ONLY where it is missing.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
`app.js` exports `errorDetailText(value, fallback)` with exact, total semantics (every input shape yields a human string or the fallback — `[object Object]` and raw `input` leakage are structurally impossible), wired at the pre-stream HTTP-error handler and the SSE `error`-frame throw; the seven existing guarded copies stay byte-identical.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — new exported function (the `THINKING_NEAR_BOTTOM_PX` / `TURN_TIMEOUT_MS` export-for-unit-pin idiom; place beside the other error constants):
|
||||||
|
```
|
||||||
|
errorDetailText(value, fallback) → string
|
||||||
|
```
|
||||||
|
- non-empty **string** → the string verbatim;
|
||||||
|
- **array** → per entry: `String(entry.msg)` when the entry is an object with a non-empty `msg`, else (non-object entry) `String(entry)`; entries with neither drop; join with `"; "`; empty result → `fallback`;
|
||||||
|
- **object** → `String(value.msg)` when a non-empty `msg` exists, else `fallback`;
|
||||||
|
- nullish / empty string / any other type → `fallback`;
|
||||||
|
- NEVER `JSON.stringify` anything, NEVER touch `input`/`loc`/`ctx` (the FastAPI entries carry the full rejected input — the whole conversation — which must never reach the DOM). Docstring names the live 2026-10-07 `[object Object]` failure and the `input`-leak rule.
|
||||||
|
2. `frontend/assets/app.js` — the pre-stream handler (~3956, the `!res.ok || !res.body` block): replace the unguarded assignment with `const text = errorDetailText(body.detail, \`Brain's API answered with HTTP ${res.status}.\`); throw new Error(text);` (the `try`-around-`res.json` and the non-JSON catch keep their current shape; the default per-status string remains the fallback). The `bubbleText`/hint threading is task 03 — do NOT add it here.
|
||||||
|
3. `frontend/assets/app.js` — the SSE `error` frame (~4353): `const err = new Error(errorDetailText(ev.detail, "Something went wrong on my side."));` — the `err.hint = ev.hint;` line byte-identical. A string `detail` (every server-built frame, `ChatErrorEvent`) serializes VERBATIM — pin that the user-visible strings for "Question too long", "dropped the connection" etc. are unchanged.
|
||||||
|
4. `tests/unit/test_frontend_error_detail.py` (new — read `tests/unit/test_frontend_failed_turn.py` first and reuse its harness/assertion style):
|
||||||
|
- the EXACT live 422 body (`{"detail":[{"type":"string_too_long","loc":["body","history",5,"text"],"msg":"String should have at most 600000 characters","input":"<huge>"}]}`) → the `msg` verbatim, and the result contains NO `input` text and no `"[object Object]"`;
|
||||||
|
- multi-entry array → the `msg`s joined `"; "`; mixed entries (object + plain string) → joined; all-empty array → fallback;
|
||||||
|
- object with `msg` → verbatim; object without → fallback;
|
||||||
|
- string → verbatim (incl. the server's fixed copy — pinned); `""`/null/undefined/number → fallback;
|
||||||
|
- source-contract pins: the two call sites now read through `errorDetailText`; `JSON.stringify` appears NOWHERE in the new helper; the seven guarded copies are byte-identical (hash their regions or assert the untouched lines).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the shape matrix above (the harness the `test_frontend_*` files use); no `app/` change — coverage gate stays green.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Every serialized shape yields a human string or the named fallback — `[object Object]` is unrepresentable at the two chat call sites (unit-proven).
|
||||||
|
- [ ] A 422 `detail` never puts request `input` text into the DOM (pinned); server string frames render byte-identical copy.
|
||||||
|
- [ ] The seven existing guarded readers are untouched; the `err.hint` channel is untouched.
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Task 03 — The distinct pre-stream failure line: `FAILED_REQUEST_TEXT` (phase-164 D3, LOCKED A3)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D3 — the live bubble said *"My answer didn't make it — the connection dropped"* for a **422**: the server was reachable and answered deliberately; the copy lied, and the owner-confirmed remedy is a distinct honest line for pre-stream HTTP rejections.
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
A pre-stream HTTP rejection (the `!res.ok` settle) bubbles `FAILED_REQUEST_TEXT` — "I couldn't take that — my API rejected the request. Use Retry to ask again." — with the serialized detail on the `.failed-note` and the phase-114 hint on the banner; transport failures (the `fetch` itself rejects) and SSE `error` frames keep `FAILED_TURN_TEXT` byte-identically; the phase-120 persistence contract ("what the user saw is what is stored") extends to the new text.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `frontend/assets/app.js` — new constant beside `FAILED_TURN_TEXT`: `const FAILED_REQUEST_TEXT = "I couldn't take that — my API rejected the request. Use Retry to ask again.";` docstring: the PRE-STREAM HTTP-rejection settle (the server ANSWERED with a 4xx/5xx — the request died before the model; the 2026-10-07 live 422-after-a-long-answer is the finding); the real detail rides the in-bubble `.failed-note` + the persisted `error` key exactly as `FAILED_TURN_TEXT`'s does; NOT for transport failures (no answer happened at all — `FAILED_TURN_TEXT` is honest there).
|
||||||
|
2. `frontend/assets/app.js` — the pre-stream throw (task 02's site): attach `err.bubbleText = FAILED_REQUEST_TEXT;` and `err.hint = "The app answered — it rejected the request before the model ran. Nothing was lost but this send.";` to the thrown `Error` (the phase-114 `hint` channel already flows from `err.hint` to `showErrorBanner` via the catch's `setUiState` opts — verify, do not duplicate).
|
||||||
|
3. `frontend/assets/app.js` — `finalizeFailedTurn(detail, { acc, thinking, tools, segments, wrap, leavePartialIndex, bubbleText })`: the ZERO-frame branch (`else` — no `wrap`) renders `bubbleText || FAILED_TURN_TEXT` (the `addMessage` text AND the `rememberBrainTurn` record text — same value, the stored-what-you-saw contract) and the note/persistence/Retry/Edit machinery is otherwise untouched (partial-stream settles never receive a `bubbleText` by construction — a pre-stream throw has no `wrap`; still, the partial branch must ignore the key: read it ONLY in the zero-frame branch). Docstring update (the `bubbleText` key, its caller-only meaning).
|
||||||
|
4. `frontend/assets/app.js` — `runTurn`'s catch: pass `bubbleText: err?.bubbleText` into the `finalizeFailedTurn` options (the ONE line-level change; the detail/hint/banner sequencing after the call is untouched).
|
||||||
|
5. `tests/unit/test_frontend_failed_turn.py` (extend; the file's harness):
|
||||||
|
- a thrown `Error` with `bubbleText` → the zero-frame bubble + persisted record text are `FAILED_REQUEST_TEXT`, note = the detail, `failed: true` + the `error` key unchanged in shape (≤500 cap intact);
|
||||||
|
- a thrown `Error` WITHOUT `bubbleText` (the transport-failure shape) → `FAILED_TURN_TEXT` byte-identical (the existing pins re-run green);
|
||||||
|
- the SSE-frame throw (task 02's site) sets NO `bubbleText` → model-side failures keep `FAILED_TURN_TEXT`;
|
||||||
|
- the banner: `err.hint` reaches the opts the same way for both throw kinds (the phase-114 pin re-runs);
|
||||||
|
- `FAILED_REQUEST_TEXT`'s exact string pinned ONCE (the copy is owner-locked); the never-`innerHTML` rule holds.
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- Unit: the pins above; the phase-120 suite green WITHOUT modification (the default path byte-identical); coverage gate unchanged (no `app/` code).
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] Pre-stream HTTP error → bubble `FAILED_REQUEST_TEXT`, note carries the serialized detail (task 02's serializer), hint on the banner, Retry + Edit land, composer re-enables (§7.4).
|
||||||
|
- [ ] Transport failures and SSE error frames are byte-identical to the phase-120 behavior (`FAILED_TURN_TEXT`, same persistence).
|
||||||
|
- [ ] The persisted `bor.chat.v1` record for a new-line failure stores the new text + `failed: true` + the capped `error` — a refresh restores it identically (the same property the existing suite pins for `FAILED_TURN_TEXT`).
|
||||||
|
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Task 04 — The dedicated E2E: error honesty + long-answer acceptance; the final gates (phase-164 D4)
|
||||||
|
|
||||||
|
**Phase:** `164_chat_error_honesty` · **Source:** phase overview D4 + the AGENTS.md per-phase E2E rule (A16).
|
||||||
|
**Story:** n/a
|
||||||
|
|
||||||
|
**Pre-fix evidence (2026-10-07, recorded per house practice):** owner-live instance after raising `BOR_LLM_MAX_OUTPUT_TOKENS=131072` — the post-long-answer turn's bubble read *"My answer didn't make it — the connection dropped. Use Retry to ask again. Failed [object Object]."*; code trace: the 422's list-of-objects `detail` hit the unguarded `body.detail` assignment (`app.js` ~3956) → `new Error(<array>)` → `"[object Object]"` (the example was not preserved live — the cap raise removed the repro; the 32 000 ceiling repros it at DEFAULT settings with any >32k-char answer). `HistoryTurn`'s own docstring records the identical failure class once already (4 000 ceiling, phase-42 E2E).
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
One Playwright suite proves the phase end to end against the REAL client code paths — the distinct pre-stream line with the serialized validation message (never `[object Object]`, never `input` leakage), the fallback and tolerance paths, and the real-server acceptance of a full-length-answer history — and the final pass runs every gate plus the full documentation consistency review.
|
||||||
|
|
||||||
|
## Work
|
||||||
|
1. `tests/e2e/test_chat_error_honesty.py` (new; isolation; deterministic mock LLM; read `tests/e2e/test_frontend_failed_turn*.py`-adjacent suites and any existing `page.route` usage first and reuse the established stubbing approach — if no route-stub precedent exists, intercept `POST /api/chat` with Playwright `page.route` fulfilling the fetch; the assertions must drive the real `app.js` send path):
|
||||||
|
- **assert 1 (the live bug)**: stub `POST /api/chat` → 422 with the EXACT FastAPI shape for a long-history rejection (`{"detail":[{"type":"string_too_long","loc":["body","history",5,"text"],"msg":"String should have at most 600000 characters","input":"x".repeat(1000)}]}`) → send a question: the bubble text is `FAILED_REQUEST_TEXT` verbatim; the `.failed-note` detail contains the `msg` substring; NO `[object Object]` anywhere in the DOM; the rejected `input` text is NOT in the DOM; the banner shows the hint; Retry + Edit are present; the composer is re-enabled (§7.4).
|
||||||
|
- **assert 2 (fallback)**: stub → 500 with a NON-JSON body → the note/line settle on the `HTTP 500` default string, `FAILED_REQUEST_TEXT` bubble (still pre-stream), no crash.
|
||||||
|
- **assert 3 (acceptance — the D1 fix)**: seed `localStorage` `bor.chat.v1` with a conversation whose LAST brain record carries a ~525k-char answer (the seeded-restore pattern the saved-chat suites use), reload, send a short question → the request is NOT 422 (assert the response status 200 via the route/network log) and the turn settles normally with the mock answer (the locked budget trim may drop the giant turn WHOLE from the prompt — asserting settle, never prompt content, per A4).
|
||||||
|
- **assert 4 (SSE-frame tolerance)**: stub → a valid SSE stream whose `error` frame carries an ARRAY `detail` → the note serializes the `msg` (client tolerance; server-built string frames are pinned in task 02 units).
|
||||||
|
2. Final pass (the phase pipeline runs this after the tasks): full-suite + gates:
|
||||||
|
- `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` **>90%**;
|
||||||
|
- `uv run pytest tests/e2e/test_chat_error_honesty.py -v --no-cov` green **in isolation**; the phase-120/114 failed-turn + banner suites and `test_chats_api.py` green (re-cut pins named in the commit body);
|
||||||
|
- `uv run ruff check . && uv run pyright` clean.
|
||||||
|
3. Documentation & consistency (mandatory): review `README.md` and `AGENTS.md` IN FULL against the post-phase app — add the **deploy note**: *the proxy in front of the app must accept chat POSTs sized to the new caps (recommend `client_max_body_size ≥ 16m` — a follow-up turn after a full-cap answer posts ~0.5 MB+, a saved chat up to the 8 MB budget); a proxy with a smaller limit turns a legal request into a 413* — review the error-handling descriptions for accuracy (the failed-turn copy is documented where phase 120's bubbles are), and fix EVERY inconsistency found, including pre-existing ones not caused by this phase. `PLAN.md` untouched (A5).
|
||||||
|
|
||||||
|
## Testing & Quality
|
||||||
|
- This task IS the E2E + gates task; its fixtures are test-side; `app/` coverage stays **>90%**.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
- [ ] All four asserts green in isolation; assert 1 proves the exact live message renders honestly end to end (bubble + note + hint + controls).
|
||||||
|
- [ ] The seeded 525k-answer conversation sends, is accepted, and settles — the live "every follow-up dies after a long answer" failure mode is impossible after this phase.
|
||||||
|
- [ ] `uv run pytest` fully green; coverage >90%; ruff + pyright clean; no existing suite modified except re-cut pins named in the commit body.
|
||||||
|
- [ ] `README.md` + `AGENTS.md` reviewed in full, the proxy body-size deploy note added, every inconsistency fixed; the phase dir moves to `complete/` and the single `--no-gpg-sign` commit (overview) is made by the pipeline gate.
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
All gates verified. Final report:
|
||||||
|
|
||||||
|
**Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)**
|
||||||
|
|
||||||
|
- Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs).
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%)
|
||||||
|
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation
|
||||||
|
- `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed
|
||||||
|
- `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins)
|
||||||
|
- Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅
|
||||||
|
- **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**.
|
||||||
|
- Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit.
|
||||||
|
- **Next pending phase:** `164_chat_error_honesty`
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+124
@@ -0,0 +1,124 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 73%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
.................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 277 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 196 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 266 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 68 0 100%
|
||||||
|
app/models.py 136 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 841 4 99%
|
||||||
|
app/rag/llm.py 246 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/schemas.py 473 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 5869 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+21
@@ -0,0 +1,21 @@
|
|||||||
|
All green. Final state:
|
||||||
|
|
||||||
|
**Implemented (task 01 — phase 163 D1, LOCKED A2/A3)**
|
||||||
|
- `app/models.py`: `Document.page_strip_mode` (VARCHAR(16), nullable, no server default, unindexed) right after `page_count`, docstringed as the strip CLASS (`primary` = fallback class, `additive` = pictorial class, NULL = no strip).
|
||||||
|
- `alembic/versions/0027_documents_page_strip_mode.py`: additive add/drop migration mirroring 0026; `upgrade head → downgrade -1 → upgrade head` round-trips clean (DB now at `0027 (head)`).
|
||||||
|
- `app/schemas.py`: `DocContent.page_strip_mode` + the `image_url`-style omission in the existing `model_serializer` (ABSENT, never `null`).
|
||||||
|
- `app/api/docs.py`: content payload passes `page_strip_mode=doc.page_strip_mode if doc.page_count is not None else None` — the whole strip block keys off `page_count`, so a stray mode never travels alone; `/pages/{n}` untouched; endpoint docstring + README "Which kind of strip it is (phase 163)" section updated. No AGENTS.md change (its viewer bullets are the final pass's).
|
||||||
|
- Tests: `tests/integration/test_migration_0027.py` (new, house pattern: contract + pre-phase `page_count=36`/mode-NULL backfill + ORM round-trip both classes + tested downgrade + round-trip), `tests/unit/test_models.py` (column contract + ORM defaults), `tests/unit/test_document_viewer.py` (model omission rule; endpoint: stamped primary/additive, pre-phase mode-NULL row → key absent with `page_urls` kept, stray mode + `page_count` NULL → neither key, exact non-qualifying key-set), `tests/unit/test_pdf_page_copies.py` (importer leaves the class unstamped until task 02), `tests/integration/test_docs_api.py` (`_seed_page_doc` gains `page_strip_mode`; real-DB class pins + never-alone + exact key set).
|
||||||
|
|
||||||
|
**Gates**
|
||||||
|
- `uv run pytest` → **3772 passed**; `uv run pytest --cov=app --cov-report=term-missing` → **3772 passed, app TOTAL 99%** (`app/schemas.py` 100%).
|
||||||
|
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
|
||||||
|
- `uv run alembic upgrade head && downgrade -1 && upgrade head` → clean, `current` = `0027 (head)`.
|
||||||
|
- E2E regression (isolation, `--no-cov`): `test_pdf_pages_in_sources.py` 4 passed, `test_pictorial_pdf.py`+`test_garbage_pdf.py` 6 passed, `test_smoke.py` 3 passed.
|
||||||
|
|
||||||
|
**Notes / decisions**
|
||||||
|
- Non-qualifying payloads stay byte-identical (both strip keys dropped when `page_count` is NULL); pre-phase rows serialize the class ABSENT and the viewer will read that as `primary` (task 04) — pinned, no data backfill.
|
||||||
|
- Column carries no CHECK constraint (purely additive per A13); the class values are enforced by the writers (tasks 02/03).
|
||||||
|
- No live-infrastructure changes needed for this task.
|
||||||
|
|
||||||
|
**Next pending task:** `163_pdf_strip_additive_selfheal/02_persist_write_first.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+124
@@ -0,0 +1,124 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 19%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
............................ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 277 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 196 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 266 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 68 0 100%
|
||||||
|
app/models.py 136 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 764 4 99%
|
||||||
|
app/rag/llm.py 246 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/schemas.py 473 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 5792 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
**Task 02 complete — `_persist_pdf_pages` write-first atomic persist + `strip_class` stamp (D2, A4/A7).**
|
||||||
|
|
||||||
|
- `app/rag/importer.py`: `_persist_pdf_pages` rewritten — render → per-file atomic write (`<name>.png.tmp` + `os.replace`, never a direct final-path write) → stale-tail delete LAST (only old-extent files not rewritten; a file about to be overwritten is never deleted); failure path unlinks this call's `.tmp`s and files it **created** (an overwritten copy keeps standing — deleting it would re-open the live-loss hole); `strip_class="fallback"|"pictorial"|None` replaces `eligible`, stamping `page_strip_mode` `primary`/`additive` exactly with `page_count`; the None arm and no-pages guard delete-then-null (helper reads the count); module + function docstrings carry the write-first rationale and the verbatim phase-148 truth table.
|
||||||
|
- Call site (`_index_pdf_file`): `strip_class = "fallback" if fallback_fired else ("pictorial" if pictorial_described else None)`; fail-soft handler nulls `page_strip_mode` too; comments updated.
|
||||||
|
- `tests/unit/test_pdf_page_copies.py`: re-cut task-01 pin → additive stamp; fallback→`primary`, ineligible/qualifier-loss/no-pages→NULL pins; new pins — order (event log: all `os.replace`s strictly before the first stale `unlink`, new bytes win), mid-write failure (prior 10-file set intact, zero `.tmp` orphans, count/mode NULL, doc indexed), atomic tmp+replace writer (every write to `.png.tmp`, `dst`/`src` pairing), best-effort stale-tail OSError branch; writer-exception test re-cut for tmp semantics.
|
||||||
|
|
||||||
|
Gates: `uv run pytest --cov=app --cov-report=term-missing` → **3776 passed, app coverage 99%** (importer 99%; remaining misses pre-existing); `uv run ruff check . && uv run pyright` → clean; E2E `test_pdf_pages_in_sources` (4), `test_pictorial_pdf` (3), `test_garbage_pdf` (3) all pass in isolation.
|
||||||
|
|
||||||
|
Docs: no further doc impact — README's phase-163 strip-class section (task 01) already covers the user-visible contract; deletion ordering is internal.
|
||||||
|
|
||||||
|
Notable decision: the partial-write cleanup keeps an **overwritten** final (keeps only newly-created files) — strictly satisfies "a failed call never leaves more files than before, and never removes a live copy" (LOCKED A4).
|
||||||
|
|
||||||
|
Next pending task: `163_pdf_strip_additive_selfheal/03_fast_path_regate.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+124
@@ -0,0 +1,124 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 19%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
................................ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 277 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 196 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 266 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 68 0 100%
|
||||||
|
app/models.py 136 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 788 4 99%
|
||||||
|
app/rag/llm.py 246 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/schemas.py 473 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 5816 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+10
@@ -0,0 +1,10 @@
|
|||||||
|
**Task 03 complete — all green.**
|
||||||
|
|
||||||
|
- **`app/rag/importer.py`**: new `_regate_pdf_strip(session, doc, *, full_path, llm, summary)` — mirrors the changed-path gate verbatim from current bytes + settings (phase-133 `is_usable_pdf_text` → phase-140 `pictorial_pages` + `pdf_pictorial` kill switch), **zero LLM/vision calls**; heals any missing `<id>-p{N}.png` (full re-render), stamps `page_strip_mode` (silent NULL→stamp, `mode old->new` on a set mode), clears a strip that no longer qualifies + deletes its copies, one greppable INFO only when it **acts**, whole body fail-soft (`re-gate failed` / `re-render produced no pages (state kept)` / `page_count drift a->b` warnings).
|
||||||
|
- `_unchanged_doc` gained `full_path: Path | None = None`; the **guarded** re-gate (`page_count` set + `.pdf`) runs at **both** exits (incl. the D1 manual-date return) and commits what it wrote; all 4 call sites (fast path, text/PDF/image content-digest hits) pass their `full_path`. Docstrings: the phase-147 "the copies persist" assumption is replaced (module doc + `_unchanged_doc` step 7 + `_index_pdf_file`).
|
||||||
|
- De-duplicated task 02's writer: extracted `_write_page_copies` (tmp + `os.replace`, orphan cleanup) and `_delete_stale_page_tail`; `_persist_pdf_pages` and the heal now share them (behavior unchanged).
|
||||||
|
- **Tests**: 11 new pins in `tests/unit/test_importer_unchanged_fast_path.py` — heal (pictorial, live NULL-mode), fallback heal (`primary`), healthy no-op (zero writes, zero re-gate log), mode re-stamp, repair arm, kill-switch clear, render-raise fail-soft, empty-render keeps row, count drift, content-digest-hit path, non-strip zero-parse guard (spy).
|
||||||
|
- `uv run pytest tests/unit/test_importer_unchanged_fast_path.py tests/unit/test_pdf_page_copies.py --no-cov` → exit 0 (24 + 15 tests); `uv run pytest --cov=app` → **3787 passed**, `app/rag/importer.py` **99%** (only pre-existing misses: 883, 3506-3507, 3793), TOTAL 99%; `uv run ruff check .` → clean; `uv run pyright` → **0 errors**; PDF E2E files pass in isolation (`test_pdf_pages_in_sources`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_sync_changed_only`, exit 0).
|
||||||
|
- **Docs**: README — new "The strip repairs itself (phase 163)" viewer bullet, the "Updating the documents" fast-path qualifier, and the `BOR_PDF_PAGE_DIR` env-row note (dedup avoided). **AGENTS.md: no change** (per the phase overview its viewer bullet lands in the phase's final pass).
|
||||||
|
- **Notable**: (1) running all of `tests/e2e` in one process fails on a **pre-existing** design artifact — `tests/e2e/test_extensionless_import.py:70` mutates `os.environ["BOR_IMPORT_EXTENSIONS"]` at import, so other suites' 13-doc seed drops to 5; unaffected by this task, and the project's gate is per-file isolation (flagging for the owner/final pass). (2) Restored `.agents/screenshots/doc_modal_scrollbar_1280.png` (a tracked artifact my E2E runs overwrote) from HEAD so the phase commit stays clean.
|
||||||
|
- **Next pending task**: `163_pdf_strip_additive_selfheal/04_viewer_additive_toggle.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+124
@@ -0,0 +1,124 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 19%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
........................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 277 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 196 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 266 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 68 0 100%
|
||||||
|
app/models.py 136 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 841 4 99%
|
||||||
|
app/rag/llm.py 246 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/schemas.py 473 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 5869 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
**Task 04 complete — report**
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `frontend/assets/document.js`: `renderDocument` now branches on `doc.page_strip_mode` — absent/`"primary"` keeps the phase-148 strip-only branch verbatim (byte-identical); `"additive"` renders the text doc (Summary panel + content) inside a `.doc-views` wrapper with a `Text`/`Pages` toggle (`role=group`, `aria-pressed`, Text default per LOCKED A6); strip mounts once, hidden via CSS `data-view` swap (lazy imgs never fetch while hidden, no refetch on flip-flop); never-build-HTML rule kept (module still has exactly one `innerHTML`); module docstring gained the phase-163 paragraph and re-scoped the phase-148 "INSTEAD of" sentence to the primary class.
|
||||||
|
- `frontend/assets/styles.css`: `.doc-view-toggle` ghost-pill pair (house `.auth-link`/`.share-chat-btn` idiom, chat-column cap, ≥44px targets, global 3px focus ring, pressed = brand fill + weight 700 — text + style, never color alone) + `.doc-views[data-view]` show/hide rules.
|
||||||
|
- New `tests/unit/test_viewer_additive_toggle.py`: 9 pins (source-contract + node behavioral runs of the real core: additive shape, primary/absent byte-identity, text-doc untouched, flip semantics with strip-node persistence, lazy imgs, XSS, CSS contract).
|
||||||
|
- Re-cut stale/newly-affected pins: `test_summary_edit_ui.py` (slot mount), `test_wide_column_css.py` (7th reading-column selector), E2E additive-view contracts in `test_pdf_pages_in_sources.py` and `test_pictorial_pdf.py` (text-first default, Pages flip reveals loaded strip, description rides the shown text).
|
||||||
|
|
||||||
|
**Gates**
|
||||||
|
- `uv run pytest` → **3796 passed**, 0 failed; coverage `app/` **99%** (>90%).
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
|
||||||
|
- E2E in isolation: `test_pdf_pages_in_sources.py` 4✓, `test_pictorial_pdf.py` 3✓, `test_garbage_pdf.py` 3✓, plus `test_pdf_support.py` 6✓, `test_document_viewer.py` 8✓, `test_summary_in_viewer.py` 4✓.
|
||||||
|
|
||||||
|
**Docs**: README — phase-148 viewer bullet scope-fixed to "primary class only" (the phase-163 bullets from tasks 01–03 already describe the toggle); AGENTS.md — no doc impact (viewer bullet note is reserved for the phase final pass).
|
||||||
|
|
||||||
|
**Decisions**: additive E2E witnesses flipped from "descriptions not displayed" to "descriptions ride the shown text" (intended consequence of LOCKED A1/A6); toggle lives in a `.doc-views` wrapper only on additive docs — all other DOM untouched.
|
||||||
|
|
||||||
|
**Next pending task**: `163_pdf_strip_additive_selfheal/05_e2e_selfheal_pins.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+124
@@ -0,0 +1,124 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 73%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
.................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 277 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 196 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 266 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 68 0 100%
|
||||||
|
app/models.py 136 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 841 4 99%
|
||||||
|
app/rag/llm.py 246 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/schemas.py 473 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 5869 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+16
@@ -0,0 +1,16 @@
|
|||||||
|
**Task 05 complete — report**
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- New `tests/e2e/test_pdf_strip_additive_selfheal.py` (5 tests, phase-163 A16 suite): fixture `strip_fixture` = `manual.pdf` (3 dense pages + 1 sparse map page → additive), `scanned.pdf` (XML-garbage layer → primary), `plain.pdf` (no strip); asserts 1 additive view (payload stamp, Text default, Pages flip loads all 4 imgs, flip-back keeps the mounted strip node — dataset-probed, zero refetch); 2 primary/plain DOM byte-identity incl. the key-omission payload pins; 3 heal (wipe copies → route 404 → one unchanged-byte sync → files restored, route 200, viewer complete); 4 repair (direct-DB stale `page_count`/mode + fake copies → cleared + deleted); 5 zero-LLM (mock `/v1/e2e/captured`: index burned both vision passes, heal/repair captured zero content calls — the sync prelude's fixed `ping` health probe is the one documented exemption).
|
||||||
|
- Docs pass: README verified in full (viewer + importer + env-table claims cross-checked against code; defaults script-audited vs `Settings` — no drift); AGENTS.md rule 5 gained the phase-163 viewer-class note.
|
||||||
|
|
||||||
|
**Gates** (all run this session, DB up)
|
||||||
|
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation.
|
||||||
|
- `test_pdf_pages_in_sources / test_pictorial_pdf / test_garbage_pdf / test_document_viewer / test_document_viewer_context / test_image_documents` → all exit 0, unmodified.
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 passed**, coverage **99%** (>90%); `uv run ruff check .` clean; `uv run pyright` 0 errors; alembic **0027 downgrade→upgrade round-trip clean**.
|
||||||
|
|
||||||
|
**Docs**: README — no further edits needed (phase bullets from tasks 01–03 verified accurate); AGENTS.md — viewer-class note added.
|
||||||
|
|
||||||
|
**Decisions**: heal/repair zero-call pins exempt only `check_models`' `ping` (every sync pays it, predates the phase); anything else fails the pin. Pre-existing `.agents/screenshots/doc_modal_scrollbar_1280.png` modification (earlier E2E run) left for the pipeline commit.
|
||||||
|
|
||||||
|
**Next pending task**: `164_chat_error_honesty/01_caps_long_answer_chain.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+124
@@ -0,0 +1,124 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 7%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 34%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 73%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
.................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 277 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 110 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 196 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 266 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 39 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/theming.py 38 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 68 0 100%
|
||||||
|
app/models.py 136 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 841 4 99%
|
||||||
|
app/rag/llm.py 246 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/schemas.py 473 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 5869 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 42 KiB After Width: | Height: | Size: 43 KiB |
@@ -21,7 +21,11 @@
|
|||||||
panels — `max-height: min(70vh, 64rem)` — with a sticky header row;
|
panels — `max-height: min(70vh, 64rem)` — with a sticky header row;
|
||||||
a clipped action label shows an ellipsis hint (`.cell-clip`) with
|
a clipped action label shows an ellipsis hint (`.cell-clip`) with
|
||||||
the full text kept reachable via the control's aria-label
|
the full text kept reachable via the control's aria-label
|
||||||
(phase 156)) and
|
(phase 156); the document viewer's page strip is CLASS-scoped —
|
||||||
|
the primary class shows the strip in place of the text, the
|
||||||
|
additive class never hides the text: Text/Pages toggle group
|
||||||
|
(`aria-pressed`, Text default) with the strip behind it
|
||||||
|
(phase 163)) and
|
||||||
meets WCAG 2.1 AA basics (semantic
|
meets WCAG 2.1 AA basics (semantic
|
||||||
landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the
|
landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the
|
||||||
stream).
|
stream).
|
||||||
|
|||||||
@@ -239,9 +239,11 @@ idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
|
|||||||
streamed) and checked against the last index's stamp *before* any
|
streamed) and checked against the last index's stamp *before* any
|
||||||
extraction or vision work — an unchanged file costs one hash + one row
|
extraction or vision work — an unchanged file costs one hash + one row
|
||||||
lookup (no re-extract, no vision, no re-embed), while a changed file
|
lookup (no re-extract, no vision, no re-embed), while a changed file
|
||||||
re-indexes fully. A refresh after a normal editing session takes
|
re-indexes fully. The one exception is a PDF row that still claims a
|
||||||
seconds — pictorial PDFs included (their expensive first-index vision
|
page strip: it is re-gated from its current bytes (no vision, no LLM —
|
||||||
burns only once per file version):
|
see "The strip repairs itself" below). A refresh after a normal editing
|
||||||
|
session takes seconds — pictorial PDFs included (their expensive
|
||||||
|
first-index vision burns only once per file version):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# After editing/adding/removing notes:
|
# After editing/adding/removing notes:
|
||||||
@@ -844,11 +846,42 @@ uploaded archives):
|
|||||||
shows the PAGES: a vertical strip of lazy page images with
|
shows the PAGES: a vertical strip of lazy page images with
|
||||||
"Page N" captions, in place of the text content and the Summary
|
"Page N" captions, in place of the text content and the Summary
|
||||||
panel (the per-page descriptions stay in the DB for Q&A — the
|
panel (the per-page descriptions stay in the DB for Q&A — the
|
||||||
embeddings and summary are untouched, just not displayed). A lost
|
embeddings and summary are untouched, just not displayed). Since
|
||||||
|
phase 163 this strip-only shape is the PRIMARY class only — the
|
||||||
|
pictorial class keeps its text behind a Text/Pages toggle, per the
|
||||||
|
next bullet. A lost
|
||||||
copy shows a "Page N unavailable" note for that page only. Text-
|
copy shows a "Page N unavailable" note for that page only. Text-
|
||||||
heavy PDFs (usable layer, zero pictorial candidates — and the whole
|
heavy PDFs (usable layer, zero pictorial candidates — and the whole
|
||||||
sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view,
|
sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view,
|
||||||
byte-identical to pre-phase.
|
byte-identical to pre-phase.
|
||||||
|
- **Which kind of strip it is (phase 163).** The two qualifying classes
|
||||||
|
are not the same problem: the screenshot class has no usable text to
|
||||||
|
show, while a book with one sparse page (an illustration map page in a
|
||||||
|
36-page text) does. The document row therefore records the strip's
|
||||||
|
CLASS beside the page count (`documents.page_strip_mode` —
|
||||||
|
`primary` = the screenshots ARE the content, so the strip is the view;
|
||||||
|
`additive` = a usable text layer, so the text + Summary view stays and
|
||||||
|
the pages sit behind a Text/Pages toggle), and the document-content
|
||||||
|
payload carries it as `page_strip_mode`. The key is ABSENT — never
|
||||||
|
`null` — when there is no strip, which is also how a row written before
|
||||||
|
this phase serializes; the viewer treats an absent class as `primary`,
|
||||||
|
so every existing document renders exactly as before until the next
|
||||||
|
sync stamps it (the sync's self-heal pass, phase 163).
|
||||||
|
- **The strip repairs itself (phase 163).** An unchanged PDF is not
|
||||||
|
assumed to still have its page images on disk. On every sync, a row
|
||||||
|
that claims a strip is re-gated deterministically from the current
|
||||||
|
file bytes and the current settings — the same gate the changed path
|
||||||
|
uses (`BOR_PDF_PICTORIAL` and `BOR_PDF_PICTORIAL_MIN_CHARS`
|
||||||
|
included), with **zero vision or LLM calls** (rendering is CPU-only):
|
||||||
|
the class is re-stamped, any `<doc-id>-p{N}.png` missing from
|
||||||
|
`BOR_PDF_PAGE_DIR` is re-rendered (write-before-delete, atomically per
|
||||||
|
file), and a strip that no longer qualifies — the kill switch turned
|
||||||
|
off, the sparse-page bar lowered past it — is cleared along with its
|
||||||
|
copies. So a wiped pages directory, or a copy set lost to an
|
||||||
|
interrupted sync, repairs itself on the next sync instead of leaving
|
||||||
|
"Page N unavailable" forever. It logs one `PDF strip re-gate …` line
|
||||||
|
only when it actually acts, and a failure there never fails the sync.
|
||||||
|
PDFs without a strip pay nothing extra (no extra read at all).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -975,7 +1008,7 @@ to `.env` and adjust.
|
|||||||
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
|
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
|
||||||
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
|
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
|
||||||
| `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) |
|
| `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) |
|
||||||
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view |
|
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view. Clearing the directory is self-healing — the next sync re-gates any strip that still claims pages and re-renders what is missing, with zero vision calls (phase 163) |
|
||||||
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
|
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
|
||||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
|
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
|
||||||
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
|
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""documents.page_strip_mode: the strip CLASS behind page_count (phase 163, task 01)
|
||||||
|
|
||||||
|
Revision ID: 0027
|
||||||
|
Revises: 0026
|
||||||
|
Create Date: 2026-10-08
|
||||||
|
|
||||||
|
Phase 163 (the PDF page strip is ADDITIVE for a text-bearing book —
|
||||||
|
task 01, storage only):
|
||||||
|
|
||||||
|
* ``documents.page_strip_mode`` — VARCHAR(16) NULLABLE, NO server
|
||||||
|
default, NO index: the CLASS of the page strip a non-NULL
|
||||||
|
``page_count`` describes, so the viewer knows whether the strip
|
||||||
|
REPLACES the text or sits one toggle behind it.
|
||||||
|
|
||||||
|
``"primary"`` — the phase-133 FALLBACK class (an unusable text layer:
|
||||||
|
the screenshots ARE the content), i.e. exactly the pre-phase view: the
|
||||||
|
strip is the document's view and no text is shown. ``"additive"`` —
|
||||||
|
the phase-140 PICTORIAL class (a usable text layer with ≥1 sparse page
|
||||||
|
described): the document keeps its text view and its Summary panel,
|
||||||
|
and the strip is one Text/Pages toggle away.
|
||||||
|
|
||||||
|
NULL means NO strip (every non-qualifying document — text documents,
|
||||||
|
images, text-heavy PDFs — and every qualifying PDF whose render/write
|
||||||
|
failed). A row written before this phase keeps NULL while its
|
||||||
|
``page_count`` stands; the wire omits the key in that case and the
|
||||||
|
viewer treats an absent mode as ``"primary"`` (today's DOM), so
|
||||||
|
existing rows render exactly as before until the fast-path re-gate
|
||||||
|
(task 03) stamps them. Stamped and cleared EXACTLY with
|
||||||
|
``page_count`` — the importer's persist (task 02) and the unchanged-
|
||||||
|
path re-gate (task 03) are the only writers.
|
||||||
|
|
||||||
|
A display fact, never a query surface, like the 0026 column it
|
||||||
|
accompanies: the content endpoint reads it from the ROW after the
|
||||||
|
``(source, path)`` lookup, so it carries no index of its own (the
|
||||||
|
``file_hash`` convention, 0025).
|
||||||
|
|
||||||
|
One additive, fully reversible migration (A13); no other schema change.
|
||||||
|
The ordering of the copies on disk (task 02) and the re-gate itself
|
||||||
|
(task 03) are importer code — this revision only carries the column.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlalchemy as sa
|
||||||
|
|
||||||
|
from alembic import op
|
||||||
|
|
||||||
|
revision = "0027"
|
||||||
|
down_revision = "0026"
|
||||||
|
branch_labels = None
|
||||||
|
depends_on = None
|
||||||
|
|
||||||
|
|
||||||
|
def upgrade() -> None:
|
||||||
|
# No index (a display fact, never a query surface — the content
|
||||||
|
# endpoint reads it from the row) and no server default (NULL is the
|
||||||
|
# meaningful state for every non-qualifying document, every fail-soft
|
||||||
|
# miss, and every pre-phase row).
|
||||||
|
op.add_column(
|
||||||
|
"documents", sa.Column("page_strip_mode", sa.String(length=16), nullable=True)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def downgrade() -> None:
|
||||||
|
# The page_strip_mode column is the only 0027 artefact — dropping it
|
||||||
|
# leaves 0026's schema byte-identical (A13, fully reversible). The
|
||||||
|
# class is re-derivable from the source file on the next sync (the
|
||||||
|
# deterministic re-gate of task 03), so no data is lost by the drop.
|
||||||
|
op.drop_column("documents", "page_strip_mode")
|
||||||
@@ -206,6 +206,16 @@ def get_document_content(
|
|||||||
basename matches). ``None`` — a non-git source, no row, or an
|
basename matches). ``None`` — a non-git source, no row, or an
|
||||||
ambiguous basename — keeps the payload byte-identical to pre-phase
|
ambiguous basename — keeps the payload byte-identical to pre-phase
|
||||||
(both fields are dropped by the serializer, never ``null``).
|
(both fields are dropped by the serializer, never ``null``).
|
||||||
|
|
||||||
|
Phase 163 (task 01): a page-qualifying PDF's payload additionally
|
||||||
|
carries ``page_strip_mode`` — ``"primary"`` (the phase-133 fallback
|
||||||
|
class: the strip IS the view) or ``"additive"`` (the phase-140
|
||||||
|
pictorial class: the text + Summary view with the strip behind a
|
||||||
|
Text/Pages toggle). Both strip keys are built off ``page_count`` and
|
||||||
|
dropped when it is NULL, so every non-qualifying payload stays
|
||||||
|
byte-identical to pre-phase (and a pre-phase row, whose mode column
|
||||||
|
is NULL, keeps the strip-only DOM the viewer assumes for an absent
|
||||||
|
mode).
|
||||||
"""
|
"""
|
||||||
row = db.execute(
|
row = db.execute(
|
||||||
select(Document, func.count(Chunk.id).label("chunks"))
|
select(Document, func.count(Chunk.id).label("chunks"))
|
||||||
@@ -254,6 +264,16 @@ def get_document_content(
|
|||||||
if doc.page_count is not None
|
if doc.page_count is not None
|
||||||
else None
|
else None
|
||||||
),
|
),
|
||||||
|
# Phase 163 (task 01): the strip CLASS (``"primary"`` = the
|
||||||
|
# fallback class, the strip IS the view; ``"additive"`` = the
|
||||||
|
# pictorial class, text + a Text/Pages toggle). The whole strip
|
||||||
|
# block keys off ``page_count`` — a row with a stray mode but no
|
||||||
|
# strip serializes NEITHER key — and a pre-phase row (mode NULL)
|
||||||
|
# serializes the key ABSENT (never null), which the viewer reads
|
||||||
|
# as ``"primary"`` (task 04).
|
||||||
|
page_strip_mode=(
|
||||||
|
doc.page_strip_mode if doc.page_count is not None else None
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -198,6 +198,24 @@ class Document(Base):
|
|||||||
#: cleanup deletes the copies with the row. No index: a display
|
#: cleanup deletes the copies with the row. No index: a display
|
||||||
#: fact, never a query surface (the ``file_hash`` convention, 0025).
|
#: fact, never a query surface (the ``file_hash`` convention, 0025).
|
||||||
page_count: Mapped[int | None] = mapped_column(Integer, default=None)
|
page_count: Mapped[int | None] = mapped_column(Integer, default=None)
|
||||||
|
#: The STRIP CLASS behind a non-NULL :py:attr:`page_count` (phase 163,
|
||||||
|
#: LOCKED A1/A3) — the signal the viewer branches on so a text-bearing
|
||||||
|
#: book never loses its text:
|
||||||
|
#:
|
||||||
|
#: * ``"primary"`` — the phase-133 FALLBACK class (unusable text layer:
|
||||||
|
#: the screenshots ARE the content): the strip IS the view, exactly
|
||||||
|
#: as phase 148 shipped it;
|
||||||
|
#: * ``"additive"`` — the phase-140 PICTORIAL class (usable text layer,
|
||||||
|
#: ≥1 sparse page described): the text view + the Summary panel, with
|
||||||
|
#: the strip one Text/Pages toggle away (Text is the default).
|
||||||
|
#:
|
||||||
|
#: NULL = no strip (every non-qualifying document) — and a pre-phase-163
|
||||||
|
#: row keeps NULL while its ``page_count`` stands, which the wire treats
|
||||||
|
#: as ``"primary"`` (today's DOM) until the fast-path re-gate stamps it.
|
||||||
|
#: Stamped/cleared EXACTLY with ``page_count``: written by the importer's
|
||||||
|
#: persist (task 02) and the unchanged-path re-gate (task 03) only. Like
|
||||||
|
#: ``page_count`` it is a display fact, never a query surface — no index.
|
||||||
|
page_strip_mode: Mapped[str | None] = mapped_column(String(16), default=None)
|
||||||
|
|
||||||
chunks: Mapped[list[Chunk]] = relationship(
|
chunks: Mapped[list[Chunk]] = relationship(
|
||||||
back_populates="document", cascade="all, delete-orphan"
|
back_populates="document", cascade="all, delete-orphan"
|
||||||
|
|||||||
+408
-70
@@ -190,15 +190,37 @@ dpi, every page) and persisted to ``settings.pdf_page_dir``
|
|||||||
``<doc-id>-p{N}.png`` (N 1-based), with ``documents.page_count``
|
``<doc-id>-p{N}.png`` (N 1-based), with ``documents.page_count``
|
||||||
(migration 0026) recording the page count — a per-page render
|
(migration 0026) recording the page count — a per-page render
|
||||||
failure leaves a GAP (no file — the route 404s that N, the viewer's
|
failure leaves a GAP (no file — the route 404s that N, the viewer's
|
||||||
note covers it). The existing copies are deleted before replacement
|
note covers it). WRITE-FIRST (phase 163, task 02, LOCKED A4): the
|
||||||
(a shrunk PDF's old ``p{N}`` tail cannot leak), a document that
|
fresh render's bytes land ATOMICALLY per file (``<name>.png.tmp`` +
|
||||||
STOPS qualifying is cleared (``page_count`` NULL + copies deleted),
|
``os.replace``) BEFORE any old copy is removed — only the old-extent
|
||||||
and the prune deletes the copies with the row
|
files the fresh render did NOT write (a shrunk PDF's ``p{N}`` tail)
|
||||||
|
go, so a rollback or crash can never restore a stale ``page_count``
|
||||||
|
over ZERO files. The row is stamped with the strip's CLASS beside the
|
||||||
|
count (``documents.page_strip_mode``, migration 0027 — ``primary`` for
|
||||||
|
the fallback class, ``additive`` for the pictorial class, phase 163);
|
||||||
|
a document that STOPS qualifying is cleared (both columns NULL +
|
||||||
|
copies deleted — deletion-first is correct where there is no
|
||||||
|
replacement set), and the prune deletes the copies with the row
|
||||||
(:func:`_delete_page_copies`). FAIL-SOFT: a render or write failure
|
(:func:`_delete_page_copies`). FAIL-SOFT: a render or write failure
|
||||||
leaves ``page_count`` NULL and logs a warning — the document indexes
|
leaves ``page_count``/``page_strip_mode`` NULL and logs a warning —
|
||||||
EXACTLY as pre-phase (the render is a display concern, it never
|
the document indexes EXACTLY as pre-phase (the render is a display
|
||||||
gates the indexing). The UNCHANGED fast path renders nothing (the
|
concern, it never gates the indexing).
|
||||||
copies persist — zero render work, zero orphans).
|
|
||||||
|
PDF strip re-gate (phase 163, task 03, LOCKED A5): the UNCHANGED paths
|
||||||
|
never assume "the copies persist" any more — a row that still claims a
|
||||||
|
strip (``page_count`` set, a ``.pdf`` path) is re-gated
|
||||||
|
deterministically from the CURRENT bytes + CURRENT settings
|
||||||
|
(:func:`_regate_pdf_strip`, run at the tail of
|
||||||
|
:func:`_unchanged_doc`): the class is re-stamped (``primary`` for the
|
||||||
|
unusable-layer fallback class, ``additive`` for the sparse-page
|
||||||
|
class), any missing ``<doc-id>-p{N}.png`` is RE-RENDERED through the
|
||||||
|
same write-first atomic writer, and a strip that no longer qualifies
|
||||||
|
(lowered ``pdf_pictorial_min_chars``, the kill switch off) is cleared
|
||||||
|
with its copies. ZERO vision/LLM calls (the render is pure CPU, PLAN
|
||||||
|
A21-safe), one greppable INFO line only when the re-gate ACTS, and
|
||||||
|
fail-soft on every exception — the loss state (an external wipe, a
|
||||||
|
pre-phase delete-first rollback) cannot survive one sync. Every other
|
||||||
|
file's fast path still performs ZERO PDF parsing (the guard).
|
||||||
|
|
||||||
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
|
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
|
||||||
description, the same source's importable markdown files are scanned for
|
description, the same source's importable markdown files are scanned for
|
||||||
@@ -298,6 +320,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import hashlib
|
import hashlib
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
import posixpath
|
import posixpath
|
||||||
import re
|
import re
|
||||||
import uuid
|
import uuid
|
||||||
@@ -1021,6 +1044,7 @@ async def _unchanged_doc(
|
|||||||
summary: ImportSummary,
|
summary: ImportSummary,
|
||||||
raw_date: datetime,
|
raw_date: datetime,
|
||||||
file_digest: str,
|
file_digest: str,
|
||||||
|
full_path: Path | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Phase 147 (task 02): the ONE shared unchanged path — the three
|
"""Phase 147 (task 02): the ONE shared unchanged path — the three
|
||||||
branches' near-identical unchanged blocks, one code site, reached
|
branches' near-identical unchanged blocks, one code site, reached
|
||||||
@@ -1064,7 +1088,22 @@ async def _unchanged_doc(
|
|||||||
the date block (the stamp set ``file_hash`` and the date did
|
the date block (the stamp set ``file_hash`` and the date did
|
||||||
not change): quiet bookkeeping — no count, no log (the
|
not change): quiet bookkeeping — no count, no log (the
|
||||||
``unchanged`` line already fired; a date-change run already
|
``unchanged`` line already fired; a date-change run already
|
||||||
committed in step 5).
|
committed in step 5);
|
||||||
|
7. the phase-163 (task 03) PDF strip RE-GATE — a row that still
|
||||||
|
claims a strip (``page_count`` set) whose path ends in ``.pdf``
|
||||||
|
goes through :func:`_regate_pdf_strip`: the strip is re-derived
|
||||||
|
from the CURRENT bytes + CURRENT settings (deterministic, ZERO
|
||||||
|
LLM/vision calls) — a missing/changed ``page_strip_mode`` is
|
||||||
|
re-stamped, copies that vanished from disk are RE-RENDERED, a
|
||||||
|
strip that no longer qualifies is cleared — and whatever it
|
||||||
|
wrote commits in this helper's transaction. Unchanged files
|
||||||
|
re-gate the PDF strip deterministically (phase 163): the
|
||||||
|
pre-phase "the copies persist" assumption is gone, so a wipe of
|
||||||
|
``pdf_page_dir`` or a pre-phase delete-first rollback cannot
|
||||||
|
outlive one sync. Every other file (``page_count`` NULL) pays
|
||||||
|
ZERO extra read — the guard keeps the fast path's contract, and
|
||||||
|
BOTH exits (the D1 manual-date return and the normal tail) run
|
||||||
|
the re-gate.
|
||||||
"""
|
"""
|
||||||
summary.unchanged += 1
|
summary.unchanged += 1
|
||||||
logger.info("import: unchanged source=%s path=%s", source, rel)
|
logger.info("import: unchanged source=%s path=%s", source, rel)
|
||||||
@@ -1078,10 +1117,32 @@ async def _unchanged_doc(
|
|||||||
session, doc=doc, source=source, rel=rel, content=doc.content,
|
session, doc=doc, source=source, rel=rel, content=doc.content,
|
||||||
llm=llm, summary=summary, backfill=True,
|
llm=llm, summary=summary, backfill=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def _regate_strip() -> None:
|
||||||
|
"""Step 7 (phase 163, task 03): the GUARDED strip re-gate — only
|
||||||
|
a row that carries a strip (``page_count`` set) on a ``.pdf``
|
||||||
|
path reaches :func:`_regate_pdf_strip`, so every other file's
|
||||||
|
fast path stays byte-identical (zero extra read, zero PDF
|
||||||
|
parse). The re-gate never raises (fail-soft inside); its writes
|
||||||
|
ride this helper's transaction."""
|
||||||
|
if doc.page_count is None or not str(doc.path).lower().endswith(".pdf"):
|
||||||
|
return
|
||||||
|
_regate_pdf_strip(
|
||||||
|
session,
|
||||||
|
doc,
|
||||||
|
full_path=full_path if full_path is not None else Path(doc.full_path),
|
||||||
|
llm=llm,
|
||||||
|
summary=summary,
|
||||||
|
)
|
||||||
|
if session.is_modified(doc):
|
||||||
|
session.commit()
|
||||||
|
|
||||||
# 4 — the phase-106 (D1) manual-date lock: no date write; the
|
# 4 — the phase-106 (D1) manual-date lock: no date write; the
|
||||||
# stamp, if dirty, is the only write (committed — see step 2's
|
# stamp, if dirty, is the only write (committed — see step 2's
|
||||||
# "next sync takes the fast path").
|
# "next sync takes the fast path") — plus whatever the phase-163
|
||||||
|
# re-gate repaired.
|
||||||
if doc.created_at_manual:
|
if doc.created_at_manual:
|
||||||
|
_regate_strip()
|
||||||
if session.is_modified(doc):
|
if session.is_modified(doc):
|
||||||
session.commit()
|
session.commit()
|
||||||
return
|
return
|
||||||
@@ -1103,6 +1164,165 @@ async def _unchanged_doc(
|
|||||||
# 6 — the stamp's commit (the date did not move): quiet
|
# 6 — the stamp's commit (the date did not move): quiet
|
||||||
# bookkeeping — the ``unchanged`` line already fired.
|
# bookkeeping — the ``unchanged`` line already fired.
|
||||||
session.commit()
|
session.commit()
|
||||||
|
# 7 — the phase-163 (task 03) strip re-gate (see the local helper).
|
||||||
|
_regate_strip()
|
||||||
|
|
||||||
|
|
||||||
|
def _regate_pdf_strip(
|
||||||
|
session: Session,
|
||||||
|
doc: Document,
|
||||||
|
*,
|
||||||
|
full_path: Path,
|
||||||
|
llm: Embedder,
|
||||||
|
summary: ImportSummary,
|
||||||
|
) -> None:
|
||||||
|
"""Phase 163 (task 03, LOCKED A5): the DETERMINISTIC re-gate of a
|
||||||
|
PDF's display strip — run from the unchanged paths
|
||||||
|
(:func:`_unchanged_doc`) whenever a row still claims a strip
|
||||||
|
(``page_count`` set), so a strip can never die silently.
|
||||||
|
|
||||||
|
WHY (the 2026-10-07 live state, owner report): the ``Letumdei`` row
|
||||||
|
carried ``page_count = 36`` over ZERO files — the pre-phase-163
|
||||||
|
:func:`_persist_pdf_pages` deleted the old copies BEFORE rendering,
|
||||||
|
OUTSIDE the DB transaction, so a later per-file rollback (the
|
||||||
|
import loop's ``EmbeddingError`` handler) restored the stale count
|
||||||
|
over an empty directory; an external wipe of ``pdf_page_dir`` lands
|
||||||
|
exactly the same state. The phase-147 fast path believed "the
|
||||||
|
copies persist" and never re-rendered, so every
|
||||||
|
``GET /api/documents/<id>/pages/N`` 404'd and the viewer showed
|
||||||
|
"Page N unavailable." on all 36 pages — on every sync, forever.
|
||||||
|
|
||||||
|
WHAT IT DOES — the changed-path gate mirrored exactly, from the
|
||||||
|
CURRENT file bytes + CURRENT settings, with ZERO vision/LLM calls
|
||||||
|
(:func:`app.rag.pdf.render_pdf_pages` is pure CPU — PLAN A21-safe,
|
||||||
|
no model, no budget):
|
||||||
|
|
||||||
|
* ``not is_usable_pdf_text(extract_pdf_text(full_path))`` → the
|
||||||
|
phase-133 fallback class → mode ``"primary"`` (the strip IS the
|
||||||
|
view — the same doc as today);
|
||||||
|
* else ``settings.pdf_pictorial`` ON and
|
||||||
|
:func:`app.rag.pdf.pictorial_pages` NON-EMPTY over
|
||||||
|
:func:`app.rag.pdf.extract_pdf_page_texts` → the phase-140
|
||||||
|
pictorial class → mode ``"additive"`` (the text view + the strip
|
||||||
|
behind a toggle). The candidates are the existing gate's
|
||||||
|
VERBATIM (text-carrying pages under ``min_chars``; a page with no
|
||||||
|
text has no key in the map and is not a candidate) — the re-gate
|
||||||
|
adds NO new eligibility semantics;
|
||||||
|
* else the strip no longer qualifies (the owner lowered
|
||||||
|
``pdf_pictorial_min_chars``, the kill switch went off, the layer
|
||||||
|
gained substance) → the REPAIR arm: the copies go
|
||||||
|
(``_delete_page_copies`` reads ``doc.page_count``, so it runs
|
||||||
|
while the count still describes the extent) and BOTH columns are
|
||||||
|
cleared.
|
||||||
|
|
||||||
|
For a re-gated-ELIGIBLE doc the copies are VERIFIED: any missing
|
||||||
|
``<doc-id>-p{N}.png`` in 1..``page_count`` re-renders EVERY page
|
||||||
|
through the changed path's own write-first atomic writer
|
||||||
|
(:func:`_write_page_copies` — tmp + ``os.replace``), then the
|
||||||
|
stale tail beyond the new count goes and ``page_count`` follows the
|
||||||
|
fresh render's length (a drift names both counts in a WARNING — on
|
||||||
|
unchanged bytes a drift is pathological). A fresh render producing
|
||||||
|
NO pages (unopenable right after the fast path hashed the file)
|
||||||
|
leaves the row exactly as it was + a WARNING: the re-gate never
|
||||||
|
clears a strip on a transient.
|
||||||
|
|
||||||
|
The mode re-stamp is SILENT when it only fills a NULL (a pre-phase
|
||||||
|
row learning its class is beneficial bookkeeping — it rides the
|
||||||
|
same INFO line only when real work followed); a change of an
|
||||||
|
ALREADY-SET mode (the kill switch, a class flip) says so
|
||||||
|
(``mode <old>-><new>``). The ONE greppable INFO line fires only
|
||||||
|
when the re-gate ACTS — never per unchanged doc.
|
||||||
|
|
||||||
|
FAIL-SOFT: the whole body is wrapped — any exception is a WARNING
|
||||||
|
and the sync continues (the strip is a display concern). A repair's
|
||||||
|
DB writes ride the caller's transaction, so an exception raised
|
||||||
|
after them rolls back with the file like any other per-file
|
||||||
|
failure. *summary* keeps the unchanged path's signature shape (and
|
||||||
|
a future counter site); the re-gate books nothing there — its stats
|
||||||
|
are log-only (the phase-133/140 convention).
|
||||||
|
"""
|
||||||
|
source, rel = doc.source, doc.path
|
||||||
|
try:
|
||||||
|
page_dir = Path(llm.settings.pdf_page_dir).expanduser()
|
||||||
|
# The gate, mirrored from the changed path (the phase-133
|
||||||
|
# precedence over the phase-140 pass, the kill switch included).
|
||||||
|
if not is_usable_pdf_text(extract_pdf_text(full_path)):
|
||||||
|
strip_class: str | None = "fallback"
|
||||||
|
elif llm.settings.pdf_pictorial and pictorial_pages(
|
||||||
|
extract_pdf_page_texts(full_path),
|
||||||
|
min_chars=llm.settings.pdf_pictorial_min_chars,
|
||||||
|
):
|
||||||
|
strip_class = "pictorial"
|
||||||
|
else:
|
||||||
|
strip_class = None
|
||||||
|
if strip_class is None:
|
||||||
|
# The REPAIR arm — the strip no longer qualifies: delete
|
||||||
|
# while the count still names the extent, then clear both
|
||||||
|
# columns (the same clearing the changed path does for a
|
||||||
|
# doc that stopped qualifying).
|
||||||
|
_delete_page_copies(doc, page_dir)
|
||||||
|
doc.page_count = None
|
||||||
|
doc.page_strip_mode = None
|
||||||
|
logger.info(
|
||||||
|
"import: PDF strip re-gate source=%s path=%s — cleared stale strip",
|
||||||
|
source, rel,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
mode = "primary" if strip_class == "fallback" else "additive"
|
||||||
|
old_mode = doc.page_strip_mode
|
||||||
|
old_count = doc.page_count or 0
|
||||||
|
actions: list[str] = []
|
||||||
|
missing = [
|
||||||
|
n
|
||||||
|
for n in range(1, old_count + 1)
|
||||||
|
if not (page_dir / f"{doc.id}-p{n}.png").is_file()
|
||||||
|
]
|
||||||
|
if missing:
|
||||||
|
pages = render_pdf_pages(full_path)
|
||||||
|
if not pages or not any(pages):
|
||||||
|
# Unopenable (or every page a placeholder) — right
|
||||||
|
# after the fast path hashed the file this is
|
||||||
|
# pathological: KEEP the row as-is (never clear a strip
|
||||||
|
# on a transient) and let the next sync try again.
|
||||||
|
logger.warning(
|
||||||
|
"import: PDF strip re-gate source=%s path=%s — re-render "
|
||||||
|
"produced no pages (state kept)",
|
||||||
|
source, rel,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
page_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
written_ns = _write_page_copies(doc, page_dir, pages)
|
||||||
|
_delete_stale_page_tail(
|
||||||
|
doc_id=doc.id,
|
||||||
|
page_dir=page_dir,
|
||||||
|
old_extent=old_count,
|
||||||
|
written_ns=written_ns,
|
||||||
|
)
|
||||||
|
doc.page_count = len(pages)
|
||||||
|
actions.append(f"healed {len(missing)} pages")
|
||||||
|
if doc.page_count != old_count:
|
||||||
|
logger.warning(
|
||||||
|
"import: PDF strip re-gate source=%s path=%s — page_count "
|
||||||
|
"drift %s->%s on unchanged bytes",
|
||||||
|
source, rel, old_count, doc.page_count,
|
||||||
|
)
|
||||||
|
if doc.page_strip_mode != mode:
|
||||||
|
# NULL -> stamped stays SILENT (it rides the line only when
|
||||||
|
# work followed); a change of an already-set mode is news.
|
||||||
|
doc.page_strip_mode = mode
|
||||||
|
if old_mode is not None:
|
||||||
|
actions.append(f"mode {old_mode}->{mode}")
|
||||||
|
if actions:
|
||||||
|
logger.info(
|
||||||
|
"import: PDF strip re-gate source=%s path=%s — %s",
|
||||||
|
source, rel, " | ".join(actions),
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
# Fail-soft: the strip never breaks a sync.
|
||||||
|
logger.warning(
|
||||||
|
"import: PDF strip re-gate failed source=%s path=%s — %s",
|
||||||
|
source, rel, e,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
async def _index_file(
|
async def _index_file(
|
||||||
@@ -1186,7 +1406,7 @@ async def _index_file(
|
|||||||
# helper, nothing else does.
|
# helper, nothing else does.
|
||||||
await _unchanged_doc(
|
await _unchanged_doc(
|
||||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||||
raw_date=raw_date, file_digest=file_digest,
|
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
# Phase 122 (task 02): the image branch FIRST. Only reachable while
|
# Phase 122 (task 02): the image branch FIRST. Only reachable while
|
||||||
@@ -1223,7 +1443,7 @@ async def _index_file(
|
|||||||
# file hash, so the NEXT sync takes the fast path above.
|
# file hash, so the NEXT sync takes the fast path above.
|
||||||
await _unchanged_doc(
|
await _unchanged_doc(
|
||||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||||
raw_date=raw_date, file_digest=file_digest,
|
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -1506,19 +1726,24 @@ async def _index_pdf_file(
|
|||||||
(:func:`_unchanged_doc`) — no re-extract, no re-embed, no image
|
(:func:`_unchanged_doc`) — no re-extract, no re-embed, no image
|
||||||
pass, the backfill/manual-date/date-refresh semantics in one
|
pass, the backfill/manual-date/date-refresh semantics in one
|
||||||
code site — and the helper stamps the new file hash (the next
|
code site — and the helper stamps the new file hash (the next
|
||||||
sync takes the fast path). This path renders NO pages and
|
sync takes the fast path). This path renders NO pages through
|
||||||
touches NO ``page_count`` (the copies persist — zero render
|
THIS branch, but a row that carries a strip is re-gated by the
|
||||||
work).
|
helper's phase-163 tail (:func:`_regate_pdf_strip` — the class
|
||||||
|
re-stamped, missing copies healed, a stale strip cleared; zero
|
||||||
|
vision/LLM work).
|
||||||
* Phase 148 (task 01): in this NEW/CHANGED path, RIGHT AFTER the
|
* Phase 148 (task 01): in this NEW/CHANGED path, RIGHT AFTER the
|
||||||
row's flush and BEFORE the chunk pipeline, a page-QUALIFYING
|
row's flush and BEFORE the chunk pipeline, a page-QUALIFYING
|
||||||
PDF (the phase-133 fallback fired, or the phase-140 pictorial
|
PDF (the phase-133 fallback fired, or the phase-140 pictorial
|
||||||
pass described ≥1 page — the signals recorded above) gets its
|
pass described ≥1 page — the signals recorded above) gets its
|
||||||
every page rendered to ``settings.pdf_page_dir`` as
|
every page rendered to ``settings.pdf_page_dir`` as
|
||||||
``<doc-id>-p{N}.png`` and ``documents.page_count`` set
|
``<doc-id>-p{N}.png`` and ``documents.page_count`` +
|
||||||
(:func:`_persist_pdf_pages` — stale copies deleted first, a
|
``documents.page_strip_mode`` set
|
||||||
stopped-qualifier cleared, a per-page ``b""`` a gap). FAIL-SOFT
|
(:func:`_persist_pdf_pages` — WRITE-FIRST (phase 163, LOCKED
|
||||||
— a render/write exception leaves ``page_count`` NULL + a
|
A4): the new copies land atomically per file BEFORE any old copy
|
||||||
warning; the document indexes exactly as pre-phase.
|
is removed, a stopped-qualifier cleared, a per-page ``b""`` a
|
||||||
|
gap). FAIL-SOFT — a render/write exception leaves
|
||||||
|
``page_count``/``page_strip_mode`` NULL + a warning; the
|
||||||
|
document indexes exactly as pre-phase.
|
||||||
* THEN the embedded images: :func:`_index_pdf_embedded_images`
|
* THEN the embedded images: :func:`_index_pdf_embedded_images`
|
||||||
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
|
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
|
||||||
|
|
||||||
@@ -1631,7 +1856,7 @@ async def _index_pdf_file(
|
|||||||
# fast path).
|
# fast path).
|
||||||
await _unchanged_doc(
|
await _unchanged_doc(
|
||||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||||
raw_date=raw_date, file_digest=file_digest,
|
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -1673,23 +1898,34 @@ async def _index_pdf_file(
|
|||||||
# its id — the copy names need it) and BEFORE the chunk pipeline
|
# its id — the copy names need it) and BEFORE the chunk pipeline
|
||||||
# (the count commits ATOMICALLY with the doc + chunks). Runs ONLY
|
# (the count commits ATOMICALLY with the doc + chunks). Runs ONLY
|
||||||
# in this new/changed path: the phase-147 fast path returned
|
# in this new/changed path: the phase-147 fast path returned
|
||||||
# before the branch (an unchanged PDF re-renders NOTHING — the
|
# before the branch and the content-digest hit above took the
|
||||||
# copies persist) and the content-digest hit above took the shared
|
# shared unchanged path — neither renders here; the phase-163
|
||||||
# unchanged path (no page work there either). FAIL-SOFT: the
|
# task-03 re-gate (:func:`_regate_pdf_strip`, run by
|
||||||
# render is a DISPLAY concern — any exception leaves
|
# :func:`_unchanged_doc`) owns the repair of an unchanged doc's
|
||||||
# ``page_count`` NULL, the warning is the signal, and the document
|
# strip. FAIL-SOFT: the render is a DISPLAY
|
||||||
# proceeds to chunk/embed/summary EXACTLY as pre-phase.
|
# concern — any exception leaves ``page_count``/``page_strip_mode``
|
||||||
|
# NULL, the warning is the signal, and the document proceeds to
|
||||||
|
# chunk/embed/summary EXACTLY as pre-phase.
|
||||||
|
# Phase 163 (task 02): the eligibility decision carried as a CLASS
|
||||||
|
# (LOCKED A7 — the truth table is the phase-148 rule verbatim: a
|
||||||
|
# non-None class ⇔ fallback_fired or pictorial_described). The
|
||||||
|
# class decides the stamped `page_strip_mode` ("fallback" →
|
||||||
|
# "primary", "pictorial" → "additive"), nothing else.
|
||||||
|
strip_class = (
|
||||||
|
"fallback" if fallback_fired else ("pictorial" if pictorial_described else None)
|
||||||
|
)
|
||||||
try:
|
try:
|
||||||
_persist_pdf_pages(
|
_persist_pdf_pages(
|
||||||
doc,
|
doc,
|
||||||
full_path=full_path,
|
full_path=full_path,
|
||||||
page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
|
page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
|
||||||
eligible=fallback_fired or pictorial_described,
|
strip_class=strip_class,
|
||||||
source=source,
|
source=source,
|
||||||
rel=rel,
|
rel=rel,
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
doc.page_count = None
|
doc.page_count = None
|
||||||
|
doc.page_strip_mode = None
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"import: PDF page render failed source=%s path=%s — %s",
|
"import: PDF page render failed source=%s path=%s — %s",
|
||||||
source, rel, e,
|
source, rel, e,
|
||||||
@@ -1991,84 +2227,186 @@ def _persist_pdf_pages(
|
|||||||
*,
|
*,
|
||||||
full_path: Path,
|
full_path: Path,
|
||||||
page_dir: Path,
|
page_dir: Path,
|
||||||
eligible: bool,
|
strip_class: str | None,
|
||||||
source: str,
|
source: str,
|
||||||
rel: str,
|
rel: str,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Phase 148 (task 01): the persistent page copies of one NEW/CHANGED
|
"""Phase 148 (task 01) + phase 163 (task 02): the persistent page
|
||||||
PDF — the display strip's bytes (task 02's public route serves them).
|
copies of one NEW/CHANGED PDF — the display strip's bytes (task 02's
|
||||||
|
public route serves them) — written WRITE-FIRST and stamped with the
|
||||||
|
strip's CLASS.
|
||||||
|
|
||||||
* ELIGIBLE (the phase-133 fallback fired, or the phase-140
|
*strip_class* is the sync's eligibility decision carried as a CLASS
|
||||||
pictorial pass described ≥1 page — the sync's own decisions, see
|
(phase 163, LOCKED A7): ``"fallback"`` (the phase-133 fallback fired
|
||||||
the caller's signals): EVERY page of the PDF is rendered (150 dpi,
|
— the screenshots ARE the content → stamped mode ``"primary"``),
|
||||||
|
``"pictorial"`` (the phase-140 pass described ≥1 sparse page →
|
||||||
|
stamped mode ``"additive"``), or ``None`` (no strip). The truth
|
||||||
|
table is the phase-148 eligibility rule VERBATIM: a non-None class
|
||||||
|
⇔ the old ``fallback_fired or pictorial_described`` — this phase
|
||||||
|
changes only what the class DOES (the stamp), never who qualifies.
|
||||||
|
|
||||||
|
* ELIGIBLE: EVERY page of the PDF is rendered (150 dpi,
|
||||||
:func:`render_pdf_pages`' default — the phase-133 constant; the
|
:func:`render_pdf_pages`' default — the phase-133 constant; the
|
||||||
``pages=None`` "list index *i* is page *i*+1" contract) and each
|
``pages=None`` "list index *i* is page *i*+1" contract) and each
|
||||||
NON-EMPTY PNG is written to *page_dir* as
|
NON-EMPTY PNG lands in *page_dir* as ``<doc-id>-p{N}.png``
|
||||||
``<doc-id>-p{N}.png`` (N 1-based), then ``doc.page_count`` is set
|
(N 1-based) through an ATOMIC per-file write — the shared writer
|
||||||
to the PDF's page count (``len(pages)`` — one slot per page, so
|
:func:`_write_page_copies` puts the bytes on ``<name>.png.tmp``
|
||||||
the list IS the page count for an openable file). A per-page
|
first and ``os.replace`` moves them onto the final name, so a
|
||||||
|
concurrent reader of the route NEVER sees a torn PNG (the phase
|
||||||
|
163 task-03 re-gate heals a strip through THE SAME writer) — and
|
||||||
|
the success tail stamps ``doc.page_count = len(pages)``
|
||||||
|
(one slot per page — the list IS the page count for an openable
|
||||||
|
file) and ``doc.page_strip_mode`` from the class. A per-page
|
||||||
``b""`` (an unreadable page) is a GAP, not an error: no file is
|
``b""`` (an unreadable page) is a GAP, not an error: no file is
|
||||||
written for it (the route 404s that N, the viewer's note covers
|
written for it (the route 404s that N, the viewer's note covers
|
||||||
it), and the count still covers the full range.
|
it), and the count still covers the full range.
|
||||||
* the STALE-REPLACEMENT convention (phase 122's image copies): the
|
* WRITE-FIRST (phase 163, LOCKED A4 — the 2026-10-07 live loss): the
|
||||||
doc's EXISTING copies (1..``doc.page_count`` — the row knows its
|
OLD copies are deleted LAST, never first. The pre-phase order
|
||||||
own extent, read BEFORE any change) are deleted FIRST, so a
|
(blanket delete → render → write) ran OUTSIDE the DB transaction,
|
||||||
changed PDF's old page set (e.g. 30 pages → 24) cannot leak
|
so any later per-file rollback (the import loop's
|
||||||
``p25..p30``; a fresh doc's NULL count is a no-op.
|
``EmbeddingError`` handler) — or a crash between delete and write —
|
||||||
* NOT eligible, or an unopenable file (:func:`render_pdf_pages` →
|
restored the stale ``page_count`` over ZERO files and the strip
|
||||||
``[]``), or a render with ZERO non-empty pages: ``page_count`` is
|
404ed forever. Now only the old-extent files (1..the count read
|
||||||
None + the existing copies deleted — a document that STOPPED
|
BEFORE any change) that the fresh render did NOT write — a shrunk
|
||||||
qualifying loses its strip (e.g. a new version gained a real text
|
PDF's ``p{N}`` tail, a new gap — are removed, and only once every
|
||||||
layer).
|
replacement byte is durably in place; a file about to be
|
||||||
* FAIL-SOFT: a write failure removes the copies THIS CALL wrote
|
overwritten is never deleted at all.
|
||||||
(no partial-write orphans) and re-raises — the caller
|
* NOT eligible (``strip_class is None``), an unopenable file
|
||||||
(:func:`_index_pdf_file`) catches, leaves ``page_count`` NULL,
|
(:func:`render_pdf_pages` → ``[]``), or a render with ZERO
|
||||||
logs the warning, and the document indexes exactly as pre-phase
|
non-empty pages: the existing copies are deleted (the helper reads
|
||||||
(the render is a display concern — it never gates indexing).
|
``doc.page_count``, so it runs BEFORE the nulling) and BOTH
|
||||||
|
``page_count`` and ``page_strip_mode`` are cleared — the
|
||||||
|
deletion-first order is correct HERE, there is no replacement set
|
||||||
|
to protect; a document that STOPPED qualifying loses its strip
|
||||||
|
(e.g. a new version gained a real text layer).
|
||||||
|
* FAIL-SOFT: a write failure removes this call's ``.tmp`` files and
|
||||||
|
the final-name files it CREATED — but NEVER a file it OVERWROTE
|
||||||
|
(the replaced copy keeps standing as the durable new bytes; a
|
||||||
|
failed call may change a copy's bytes, it may never remove a
|
||||||
|
copy) — and re-raises: the caller (:func:`_index_pdf_file`)
|
||||||
|
catches, leaves ``page_count`` AND ``page_strip_mode`` NULL, logs
|
||||||
|
the warning, and the document indexes exactly as pre-phase (the
|
||||||
|
render is a display concern — it never gates indexing).
|
||||||
|
|
||||||
*page_dir* is ``Path(settings.pdf_page_dir).expanduser()`` (the
|
*page_dir* is ``Path(settings.pdf_page_dir).expanduser()`` (the
|
||||||
``image_dir`` convention — raw string, expanduser'd by the
|
``image_dir`` convention — raw string, expanduser'd by the
|
||||||
importer); the directory is created on demand, only when a page is
|
importer); the directory is created on demand, only when a page is
|
||||||
actually written (an ineligible doc never creates it).
|
actually written (an ineligible doc never creates it).
|
||||||
"""
|
"""
|
||||||
# 1 — the stale-replacement: this doc's EXISTING copies go first —
|
if strip_class is None:
|
||||||
# a changed PDF's old page set must not leak, and a doc that
|
# No strip — the DELETION-first order is correct here (there is
|
||||||
# stopped qualifying loses its strip (a fresh doc's NULL count is
|
# no replacement set to protect); the helper reads
|
||||||
# a no-op; missing files are a silent no-op).
|
# doc.page_count, so it runs BEFORE the nulling (a fresh doc's
|
||||||
|
# NULL count is a no-op; missing files are a silent no-op).
|
||||||
_delete_page_copies(doc, page_dir)
|
_delete_page_copies(doc, page_dir)
|
||||||
if not eligible:
|
|
||||||
doc.page_count = None
|
doc.page_count = None
|
||||||
|
doc.page_strip_mode = None
|
||||||
return
|
return
|
||||||
pages = render_pdf_pages(full_path)
|
pages = render_pdf_pages(full_path)
|
||||||
if not pages or not any(pages):
|
if not pages or not any(pages):
|
||||||
# Unopenable file (the render's ``[]``) or every page a
|
# Unopenable file (the render's ``[]``) or every page a
|
||||||
# placeholder: no strip — fail-soft, the index is untouched
|
# placeholder: no strip and NO replacement set — the existing
|
||||||
# (the render helper already logged the per-page failures).
|
# copies go with the cleared row (helper BEFORE the nulling),
|
||||||
|
# fail-soft, the index is untouched (the render helper already
|
||||||
|
# logged the per-page failures).
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"import: PDF page render produced no pages source=%s path=%s",
|
"import: PDF page render produced no pages source=%s path=%s",
|
||||||
source, rel,
|
source, rel,
|
||||||
)
|
)
|
||||||
|
_delete_page_copies(doc, page_dir)
|
||||||
doc.page_count = None
|
doc.page_count = None
|
||||||
|
doc.page_strip_mode = None
|
||||||
return
|
return
|
||||||
|
# The row's OWN old extent, read BEFORE any change — the stale-tail
|
||||||
|
# delete below walks it (the write-first contract never deletes a
|
||||||
|
# file outside this extent, and never before its replacement is
|
||||||
|
# durable).
|
||||||
|
old_extent = doc.page_count or 0
|
||||||
|
# 1 — WRITE FIRST, atomically per file (the shared writer, phase
|
||||||
|
# 163 task 02 — the phase-163 task-03 re-gate heals through it).
|
||||||
|
written_ns = _write_page_copies(doc, page_dir, pages)
|
||||||
|
# 2 — the stale tail goes LAST.
|
||||||
|
_delete_stale_page_tail(
|
||||||
|
doc_id=doc.id,
|
||||||
|
page_dir=page_dir,
|
||||||
|
old_extent=old_extent,
|
||||||
|
written_ns=written_ns,
|
||||||
|
)
|
||||||
|
doc.page_count = len(pages)
|
||||||
|
doc.page_strip_mode = "primary" if strip_class == "fallback" else "additive"
|
||||||
|
|
||||||
|
|
||||||
|
def _write_page_copies(doc: Document, page_dir: Path, pages: Sequence[bytes]) -> set[int]:
|
||||||
|
"""Phase 163 (task 02, LOCKED A4; extracted for the task-03 re-gate,
|
||||||
|
which heals a strip through the SAME writer): write every NON-EMPTY
|
||||||
|
PNG of *pages* to ``page_dir/<doc-id>-p{N}.png`` (N 1-based) and
|
||||||
|
return the set of page numbers written.
|
||||||
|
|
||||||
|
* ATOMIC per file — the bytes land on ``<name>.png.tmp`` and
|
||||||
|
``os.replace`` moves them onto the final name, so a concurrent
|
||||||
|
reader of the ``/pages/{n}`` route never sees a torn PNG.
|
||||||
|
* A per-page ``b""`` (the render's failure placeholder) is the GAP:
|
||||||
|
no file, no entry in the returned set.
|
||||||
|
* The directory is created on demand (a doc that never writes never
|
||||||
|
creates it).
|
||||||
|
* A failure leaves NO orphan: this call's ``.tmp`` files and the
|
||||||
|
finals it CREATED are removed — an OVERWRITTEN final keeps
|
||||||
|
standing (deleting it would re-open the 2026-10-07 hole with one
|
||||||
|
page instead of the whole strip) — and the exception re-raises so
|
||||||
|
the caller keeps its fail-soft contract.
|
||||||
|
"""
|
||||||
page_dir.mkdir(parents=True, exist_ok=True)
|
page_dir.mkdir(parents=True, exist_ok=True)
|
||||||
written: list[Path] = []
|
written_tmps: list[Path] = []
|
||||||
|
created_finals: list[Path] = [] # finals that did NOT exist before
|
||||||
|
written_ns: set[int] = set()
|
||||||
try:
|
try:
|
||||||
for n, png in enumerate(pages, start=1):
|
for n, png in enumerate(pages, start=1):
|
||||||
if not png:
|
if not png:
|
||||||
# A per-page render failure: the GAP (no file — the
|
# A per-page render failure: the GAP (no file — the
|
||||||
# route 404s that N, the viewer's note covers it).
|
# route 404s that N, the viewer's note covers it).
|
||||||
continue
|
continue
|
||||||
path = page_dir / f"{doc.id}-p{n}.png"
|
final = page_dir / f"{doc.id}-p{n}.png"
|
||||||
path.write_bytes(png)
|
tmp = page_dir / f"{doc.id}-p{n}.png.tmp"
|
||||||
written.append(path)
|
# Track the tmp BEFORE the write: a failure mid-write may
|
||||||
|
# leave it on disk, and the cleanup must find it either way.
|
||||||
|
written_tmps.append(tmp)
|
||||||
|
existed = final.exists()
|
||||||
|
tmp.write_bytes(png)
|
||||||
|
os.replace(tmp, final) # atomic per file — no torn reads
|
||||||
|
written_ns.add(n)
|
||||||
|
if not existed:
|
||||||
|
created_finals.append(final)
|
||||||
except Exception:
|
except Exception:
|
||||||
# A partial write leaves no orphan: the copies this call wrote
|
# A partial write leaves no orphan: this call's tmp files and
|
||||||
# go, then the re-raise hands the failure to the caller's
|
# the finals it CREATED go (an OVERWRITTEN final keeps standing
|
||||||
# fail-soft handler (page_count NULL + the warning).
|
# — deleting it would re-open the 2026-10-07 hole with one page
|
||||||
for path in written:
|
# instead of the whole strip), then the re-raise hands the
|
||||||
|
# failure to the caller's fail-soft handler (page_count +
|
||||||
|
# page_strip_mode NULL + the warning).
|
||||||
|
for path in [*written_tmps, *created_finals]:
|
||||||
path.unlink(missing_ok=True)
|
path.unlink(missing_ok=True)
|
||||||
raise
|
raise
|
||||||
doc.page_count = len(pages)
|
return written_ns
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_stale_page_tail(
|
||||||
|
*, doc_id: uuid.UUID, page_dir: Path, old_extent: int, written_ns: set[int]
|
||||||
|
) -> None:
|
||||||
|
"""Phase 163 (task 02, LOCKED A4 — the LAST step of the write-first
|
||||||
|
order): remove the files of the row's OLD extent (1..*old_extent*)
|
||||||
|
the fresh render did NOT write — a shrunk PDF's ``p{N}`` tail, a new
|
||||||
|
gap. Runs only once every replacement byte is durable, never touches
|
||||||
|
a file outside the old extent, and never touches a file that was just
|
||||||
|
overwritten. Best-effort like :func:`_delete_page_copies`: an
|
||||||
|
unreadable file is logged, never raised (the sync survives)."""
|
||||||
|
for n in range(1, old_extent + 1):
|
||||||
|
if n in written_ns:
|
||||||
|
continue
|
||||||
|
path = page_dir / f"{doc_id}-p{n}.png"
|
||||||
|
try:
|
||||||
|
path.unlink(missing_ok=True)
|
||||||
|
except OSError as e:
|
||||||
|
logger.warning("import: could not delete PDF page copy %s — %s", path, e)
|
||||||
|
|
||||||
|
|
||||||
async def _index_pdf_embedded_images(
|
async def _index_pdf_embedded_images(
|
||||||
@@ -3271,7 +3609,7 @@ async def _index_image_file(
|
|||||||
# ``_store_summary`` backfill re-stores it verbatim.
|
# ``_store_summary`` backfill re-stores it verbatim.
|
||||||
await _unchanged_doc(
|
await _unchanged_doc(
|
||||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||||
raw_date=raw_date, file_digest=file_digest,
|
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
|
|||||||
@@ -636,6 +636,19 @@ class DocContent(BaseModel):
|
|||||||
#: ``null``, the :attr:`image_url` omission precedent), so a
|
#: ``null``, the :attr:`image_url` omission precedent), so a
|
||||||
#: non-qualifying doc's payload is byte-identical to pre-phase.
|
#: non-qualifying doc's payload is byte-identical to pre-phase.
|
||||||
page_urls: list[str] | None = None
|
page_urls: list[str] | None = None
|
||||||
|
#: Phase 163 (task 01, LOCKED A2/A3) — the STRIP CLASS behind
|
||||||
|
#: :attr:`page_urls`: ``"primary"`` = the phase-133 fallback class
|
||||||
|
#: (an unusable text layer — the screenshots ARE the content, so the
|
||||||
|
#: strip IS the view, exactly as phase 148 shipped it); ``"additive"``
|
||||||
|
#: = the phase-140 pictorial class (a usable text layer with ≥1 sparse
|
||||||
|
#: page described — the text view + the Summary panel, the strip one
|
||||||
|
#: Text/Pages toggle away). ABSENT from the wire whenever there is no
|
||||||
|
#: strip (``None`` → dropped by the serializer — never ``null``, the
|
||||||
|
#: :attr:`image_url` omission rule), which is ALSO how a pre-phase row
|
||||||
|
#: (``page_count`` set, the column NULL) serializes: the viewer treats
|
||||||
|
#: an absent mode as ``"primary"`` (task 04), so every existing row
|
||||||
|
#: keeps today's DOM until the fast-path re-gate stamps it.
|
||||||
|
page_strip_mode: str | None = None
|
||||||
|
|
||||||
@model_serializer(mode="wrap")
|
@model_serializer(mode="wrap")
|
||||||
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
|
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
|
||||||
@@ -648,6 +661,11 @@ class DocContent(BaseModel):
|
|||||||
data.pop("origin_branch", None)
|
data.pop("origin_branch", None)
|
||||||
if self.page_urls is None:
|
if self.page_urls is None:
|
||||||
data.pop("page_urls", None)
|
data.pop("page_urls", None)
|
||||||
|
# Phase 163 (task 01): the strip class rides the strip — ABSENT
|
||||||
|
# (never null) with no strip, so a non-qualifying doc's payload is
|
||||||
|
# byte-identical to pre-phase.
|
||||||
|
if self.page_strip_mode is None:
|
||||||
|
data.pop("page_strip_mode", None)
|
||||||
return data
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+101
-8
@@ -144,8 +144,9 @@
|
|||||||
* wire for every other document, so they render the EXACT pre-phase
|
* wire for every other document, so they render the EXACT pre-phase
|
||||||
* DOM on BOTH surfaces — the load-bearing byte-identity property),
|
* DOM on BOTH surfaces — the load-bearing byte-identity property),
|
||||||
* #doc-content renders a .doc-pages strip INSTEAD of the text
|
* #doc-content renders a .doc-pages strip INSTEAD of the text
|
||||||
* content AND the .doc-summary panel: one <figure class="doc-page">
|
* content AND the .doc-summary panel FOR THE PRIMARY CLASS ONLY
|
||||||
* per URL — a lazy <img> (alt = "Page N of <title>", the WCAG alt
|
* (phase 163 — the absent/`primary` mode; see the next paragraph):
|
||||||
|
* one <figure class="doc-page"> per URL — a lazy <img> (alt = "Page N of <title>", the WCAG alt
|
||||||
* contract — the title as the fallback context) + a "Page N"
|
* contract — the title as the fallback context) + a "Page N"
|
||||||
* <figcaption> — while the meta badge row renders exactly as today.
|
* <figcaption> — while the meta badge row renders exactly as today.
|
||||||
* The owner's contract: the per-page descriptions stay in the DB for
|
* The owner's contract: the per-page descriptions stay in the DB for
|
||||||
@@ -160,6 +161,26 @@
|
|||||||
* per-surface copy); createElement + properties/textContent only (the
|
* per-surface copy); createElement + properties/textContent only (the
|
||||||
* never-build-HTML rule — no innerHTML).
|
* never-build-HTML rule — no innerHTML).
|
||||||
*
|
*
|
||||||
|
* Phase 163 (task 04, LOCKED A1/A2/A6): the strip's CLASS decides the
|
||||||
|
* view. page_strip_mode === "additive" (the pictorial class — a USABLE
|
||||||
|
* text layer, so the pages are a convenience, not the content) keeps
|
||||||
|
* the TEXT doc EXACTLY as a non-qualifying doc renders it (Summary
|
||||||
|
* panel + content — the pre-148 branch reused), and the strip sits one
|
||||||
|
* Text/Pages toggle away: a .doc-view-toggle group (role=group,
|
||||||
|
* aria-label="Document view", two <button aria-pressed>) above a
|
||||||
|
* .doc-views wrapper holding the text view + the strip; Text is
|
||||||
|
* pressed by DEFAULT (A6 — never a sparse-ratio flip), Pages flips the
|
||||||
|
* wrapper's data-view (CSS shows the strip / hides the text — the
|
||||||
|
* strip node stays MOUNTED, its lazy imgs never refetch on flip-flop).
|
||||||
|
* The text is NEVER hidden for this class. The absent/"primary" mode
|
||||||
|
* (the fallback class — the screenshots ARE the content, and every
|
||||||
|
* pre-163 stamped row) keeps the strip-only DOM above byte-identically
|
||||||
|
* (the tolerant read: only the exact "additive" string switches on the
|
||||||
|
* toggle); falsy page_urls stay untouched. The wrapper exists ONLY on
|
||||||
|
* additive docs — every other document's DOM is unchanged. Toggle
|
||||||
|
* labels are fixed strings (no doc-derived text); createElement +
|
||||||
|
* properties/textContent only.
|
||||||
|
*
|
||||||
* Phase 153 (task 02): the meta row's "Copy path" control — the
|
* Phase 153 (task 02): the meta row's "Copy path" control — the
|
||||||
* read-tool `source/path` form in the clipboard with one click
|
* read-tool `source/path` form in the clipboard with one click
|
||||||
* (ASSUMPTION A1 — the exact `source/path` the read tool, the
|
* (ASSUMPTION A1 — the exact `source/path` the read tool, the
|
||||||
@@ -361,7 +382,9 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
|
|||||||
// Phase 148 (task 03): the page strip — the payload's page_urls
|
// Phase 148 (task 03): the page strip — the payload's page_urls
|
||||||
// (task 02's endpoint: the exact 1-based route list for a
|
// (task 02's endpoint: the exact 1-based route list for a
|
||||||
// page-qualifying PDF; ABSENT on the wire for every other
|
// page-qualifying PDF; ABSENT on the wire for every other
|
||||||
// document) is the ONE branch that replaces the content slot: the
|
// document) is the ONE branch that replaces the content slot (phase
|
||||||
|
// 163: the PRIMARY class only — an "additive" doc falls through to
|
||||||
|
// the text rendering with the strip behind a toggle, D4/A1): the
|
||||||
// markdown/.doc-raw content block AND the labeled Summary panel
|
// markdown/.doc-raw content block AND the labeled Summary panel
|
||||||
// (including its admin edit wiring — moot without the panel) are
|
// (including its admin edit wiring — moot without the panel) are
|
||||||
// NOT built. The per-page descriptions stay in the DB for Q&A (the
|
// NOT built. The per-page descriptions stay in the DB for Q&A (the
|
||||||
@@ -372,16 +395,40 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
|
|||||||
// below (byte-identical for every other doc — the load-bearing
|
// below (byte-identical for every other doc — the load-bearing
|
||||||
// property).
|
// property).
|
||||||
const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;
|
const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;
|
||||||
if (pageUrls && pageUrls.length > 0) {
|
const stripEligible = pageUrls !== null && pageUrls.length > 0;
|
||||||
|
// Phase 163 (task 04, LOCKED A2): the strip's CLASS picks the view.
|
||||||
|
// Only the exact "additive" string switches on the toggle — absent
|
||||||
|
// (pre-163 rows, and the omission rule's non-qualifying docs) or
|
||||||
|
// "primary" read as primary (the tolerant-reading house rule), so
|
||||||
|
// the fallback class keeps the strip-only DOM byte-identically.
|
||||||
|
const additive = stripEligible && doc.page_strip_mode === "additive";
|
||||||
|
if (stripEligible && !additive) {
|
||||||
contentEl.appendChild(docPagesStrip(doc, pageUrls));
|
contentEl.appendChild(docPagesStrip(doc, pageUrls));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
// Phase 163 (task 04, LOCKED A1/A6): an ADDITIVE doc renders the
|
||||||
|
// text doc EXACTLY as a non-qualifying one does (Summary panel +
|
||||||
|
// content below, reused unchanged) INSIDE a .doc-views wrapper —
|
||||||
|
// the Text/Pages toggle above it flips the wrapper's data-view
|
||||||
|
// (CSS-driven show/hide, never a re-render); the strip mounts once
|
||||||
|
// AFTER the text view and stays mounted across flips (lazy imgs,
|
||||||
|
// display:none while hidden, never fetch). Every non-additive doc
|
||||||
|
// keeps slot === contentEl — its DOM is byte-identical to pre-phase.
|
||||||
|
let views = null;
|
||||||
|
let slot = contentEl;
|
||||||
|
if (additive) {
|
||||||
|
views = document.createElement("div");
|
||||||
|
views.className = "doc-views";
|
||||||
|
views.setAttribute("data-view", "text"); // LOCKED A6: Text is the default view
|
||||||
|
contentEl.append(docViewToggle(views), views);
|
||||||
|
slot = views;
|
||||||
|
}
|
||||||
// Phase 122 (task 04): an image document — the persistent bytes
|
// Phase 122 (task 04): an image document — the persistent bytes
|
||||||
// render as the <img> block FIRST; the description (= doc.content)
|
// render as the <img> block FIRST; the description (= doc.content)
|
||||||
// follows in the normal content slot below (the plain-content path
|
// follows in the normal content slot below (the plain-content path
|
||||||
// — the last branch in this function).
|
// — the last branch in this function).
|
||||||
if (doc.is_image) {
|
if (doc.is_image) {
|
||||||
contentEl.appendChild(docImageBlock(doc));
|
slot.appendChild(docImageBlock(doc));
|
||||||
}
|
}
|
||||||
// Phase 36: the summary panel — labeled section ABOVE the original
|
// Phase 36: the summary panel — labeled section ABOVE the original
|
||||||
// content, on BOTH surfaces (page + modal) through this one core.
|
// content, on BOTH surfaces (page + modal) through this one core.
|
||||||
@@ -405,7 +452,7 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
|
|||||||
body.className = "doc-summary-text";
|
body.className = "doc-summary-text";
|
||||||
body.textContent = doc.summary; // text node — XSS contract unchanged
|
body.textContent = doc.summary; // text node — XSS contract unchanged
|
||||||
section.append(title, body);
|
section.append(title, body);
|
||||||
contentEl.appendChild(section);
|
slot.appendChild(section);
|
||||||
// Phase 57 (task 02, D4): the section above is the phase-36 shape
|
// Phase 57 (task 02, D4): the section above is the phase-36 shape
|
||||||
// for EVERYONE — only an authenticated admin (docAdminReady(), the
|
// for EVERYONE — only an authenticated admin (docAdminReady(), the
|
||||||
// module-cached whoami promise) then gains the header row + Edit
|
// module-cached whoami promise) then gains the header row + Edit
|
||||||
@@ -419,13 +466,59 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) {
|
|||||||
const wrap = document.createElement("div");
|
const wrap = document.createElement("div");
|
||||||
wrap.className = "doc-md";
|
wrap.className = "doc-md";
|
||||||
wrap.innerHTML = renderMarkdown(doc.content); // escape-first: XSS-safe
|
wrap.innerHTML = renderMarkdown(doc.content); // escape-first: XSS-safe
|
||||||
contentEl.appendChild(wrap);
|
slot.appendChild(wrap);
|
||||||
} else {
|
} else {
|
||||||
const pre = document.createElement("pre");
|
const pre = document.createElement("pre");
|
||||||
pre.className = "doc-raw";
|
pre.className = "doc-raw";
|
||||||
pre.textContent = doc.content; // text node: never parsed as HTML
|
pre.textContent = doc.content; // text node: never parsed as HTML
|
||||||
contentEl.appendChild(pre);
|
slot.appendChild(pre);
|
||||||
}
|
}
|
||||||
|
// Phase 163 (task 04): the additive strip mounts once, AFTER the
|
||||||
|
// text view, inside the wrapper — hidden by [data-view="text"]
|
||||||
|
// (the lazy imgs never fetch until Pages is chosen; the node
|
||||||
|
// persists across flips — no refetch).
|
||||||
|
if (views !== null) {
|
||||||
|
views.appendChild(docPagesStrip(doc, pageUrls));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ---------- view toggle (phase 163, task 04 — LOCKED A1/A6) ----------
|
||||||
|
* The Text/Pages toggle of an ADDITIVE strip doc: one
|
||||||
|
* <div class="doc-view-toggle" role="group" aria-label="Document
|
||||||
|
* view"> with two <button type="button" class="doc-view-btn"
|
||||||
|
* aria-pressed>. Text is pressed initially (A6 — no sparse-ratio
|
||||||
|
* default flip). A click mutates ONLY the two aria-pressed states and
|
||||||
|
* the .doc-views wrapper's data-view attribute — the CSS shows/hides
|
||||||
|
* the text view and the strip (never a re-render, never a refetch).
|
||||||
|
* The labels are FIXED strings (no doc-derived text inside the
|
||||||
|
* toggle); createElement + properties/textContent only (the
|
||||||
|
* never-build-HTML rule — no innerHTML). */
|
||||||
|
function docViewToggle(viewsEl) {
|
||||||
|
const group = document.createElement("div");
|
||||||
|
group.className = "doc-view-toggle";
|
||||||
|
group.setAttribute("role", "group");
|
||||||
|
group.setAttribute("aria-label", "Document view");
|
||||||
|
const textBtn = viewToggleButton("Text");
|
||||||
|
const pagesBtn = viewToggleButton("Pages");
|
||||||
|
const select = (view) => {
|
||||||
|
viewsEl.setAttribute("data-view", view);
|
||||||
|
textBtn.setAttribute("aria-pressed", view === "text" ? "true" : "false");
|
||||||
|
pagesBtn.setAttribute("aria-pressed", view === "pages" ? "true" : "false");
|
||||||
|
};
|
||||||
|
textBtn.setAttribute("aria-pressed", "true"); // LOCKED A6: Text pressed by default
|
||||||
|
pagesBtn.setAttribute("aria-pressed", "false");
|
||||||
|
textBtn.addEventListener("click", () => select("text"));
|
||||||
|
pagesBtn.addEventListener("click", () => select("pages"));
|
||||||
|
group.append(textBtn, pagesBtn);
|
||||||
|
return group;
|
||||||
|
}
|
||||||
|
|
||||||
|
function viewToggleButton(label) {
|
||||||
|
const btn = document.createElement("button");
|
||||||
|
btn.type = "button";
|
||||||
|
btn.className = "doc-view-btn";
|
||||||
|
btn.textContent = label; // fixed string — never doc-derived
|
||||||
|
return btn;
|
||||||
}
|
}
|
||||||
|
|
||||||
/* ---------- image block (phase 122, task 04) ----------
|
/* ---------- image block (phase 122, task 04) ----------
|
||||||
|
|||||||
@@ -5311,6 +5311,59 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
|
|||||||
padding: 0.35rem 0 0;
|
padding: 0.35rem 0 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* Additive strip view (phase 163, task 04): the pictorial-class PDF
|
||||||
|
— a USABLE text layer — keeps the text doc (Summary panel +
|
||||||
|
content) with the page strip one Text/Pages toggle away (LOCKED
|
||||||
|
A1/A6). The toggle is the house ghost-pill idiom (the .auth-link
|
||||||
|
family): the two pills sit adjacent in the labeled group, ink-soft
|
||||||
|
on the transparent pill (AA on --bg/--surface), hover the
|
||||||
|
brand-soft/brand-ink pair; the PRESSED button is the brand fill +
|
||||||
|
heavier weight — text + style, never color alone (B5) — with --bg
|
||||||
|
ink on --brand (5.2:1, the .share-chat-btn pair). ≥44px targets
|
||||||
|
(§7.1); :focus-visible rides the global 3px ring. The view swap is
|
||||||
|
CSS-driven off the .doc-views wrapper's data-view: the strip is
|
||||||
|
display:none by default (its lazy imgs never fetch while hidden)
|
||||||
|
and shows on [data-view="pages"], which hides the text view instead
|
||||||
|
— the nodes stay mounted across flips (no re-render, no refetch).
|
||||||
|
Phase-08 tokens only (no new hue — the monochrome invariant). */
|
||||||
|
.doc-view-toggle {
|
||||||
|
display: flex;
|
||||||
|
gap: 0.4rem;
|
||||||
|
width: 100%;
|
||||||
|
max-width: var(--chat-column);
|
||||||
|
margin-inline: auto;
|
||||||
|
margin-bottom: 0.75rem;
|
||||||
|
}
|
||||||
|
.doc-view-btn {
|
||||||
|
display: inline-flex;
|
||||||
|
align-items: center;
|
||||||
|
justify-content: center;
|
||||||
|
min-height: 44px;
|
||||||
|
padding: 0.5rem 0.9rem;
|
||||||
|
border-radius: 999px;
|
||||||
|
border: 1px solid var(--line);
|
||||||
|
background: transparent;
|
||||||
|
color: var(--ink-soft);
|
||||||
|
font: inherit;
|
||||||
|
font-weight: 600;
|
||||||
|
font-size: 0.95rem;
|
||||||
|
white-space: nowrap;
|
||||||
|
cursor: pointer;
|
||||||
|
}
|
||||||
|
.doc-view-btn:hover { background: var(--brand-soft); color: var(--brand-ink); }
|
||||||
|
.doc-view-btn[aria-pressed="true"] {
|
||||||
|
background: var(--brand);
|
||||||
|
border-color: var(--brand);
|
||||||
|
color: var(--bg); /* dark ink on brand: 5.2:1 (AA) */
|
||||||
|
font-weight: 700; /* pressed = weight + fill, never color alone (B5) */
|
||||||
|
}
|
||||||
|
.doc-views .doc-pages { display: none; } /* Text default — hidden lazy imgs never fetch */
|
||||||
|
.doc-views[data-view="pages"] .doc-summary,
|
||||||
|
.doc-views[data-view="pages"] .doc-md,
|
||||||
|
.doc-views[data-view="pages"] .doc-raw,
|
||||||
|
.doc-views[data-view="pages"] .doc-image { display: none; }
|
||||||
|
.doc-views[data-view="pages"] .doc-pages { display: grid; }
|
||||||
|
|
||||||
/* Designed not-found state (no emoji — plain SVG mark, phase 08 rule). */
|
/* Designed not-found state (no emoji — plain SVG mark, phase 08 rule). */
|
||||||
.doc-not-found {
|
.doc-not-found {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
|
|||||||
@@ -36,33 +36,38 @@ token (the phase-126 convention — the owner's local ``.env`` may lack
|
|||||||
it, and without it the walk is blind to the fixture).
|
it, and without it the walk is blind to the fixture).
|
||||||
|
|
||||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
|
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
|
||||||
blocks (and the seed's DB witness that the descriptions ARE indexed
|
blocks (and the seed's DB witness that the descriptions ARE indexed)
|
||||||
but NOT displayed) are exact only against the mock's deterministic
|
are exact only against the mock's deterministic
|
||||||
``PICTORIAL_PAGE_MODE`` canned reply.
|
``PICTORIAL_PAGE_MODE`` canned reply. (Phase 163 re-cut: the viewer
|
||||||
|
assertions moved from the phase-148 suppression contract to the
|
||||||
|
ADDITIVE contract — the pictorial class shows its text INCLUDING the
|
||||||
|
page descriptions, with the strip behind the Text/Pages toggle.)
|
||||||
|
|
||||||
Test → observable mapping (Playwright Mapping Rule):
|
Test → observable mapping (Playwright Mapping Rule):
|
||||||
1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module)
|
1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module)
|
||||||
asserts the DB + filesystem first: the pictorial row has
|
asserts the DB + filesystem first: the pictorial row has
|
||||||
``page_count == 3`` and the scratch page dir holds EXACTLY
|
``page_count == 3`` and the scratch page dir holds EXACTLY
|
||||||
``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy
|
``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy
|
||||||
doc's copies do not exist); then the Sources-tree doc modal shows
|
doc's copies do not exist); then the Sources-tree doc modal opens
|
||||||
the ``.doc-pages`` strip: exactly three ``.doc-page`` figures,
|
the ADDITIVE view (phase 163 — a usable text layer: TEXT first,
|
||||||
each a LOADED lazy ``<img>`` (``complete && naturalWidth > 0`` —
|
the strip one toggle away): the Text/Pages group with ``Text``
|
||||||
the route served a real PNG, not a 404) with the
|
pressed, the Summary panel + content visible (the mock's page
|
||||||
``Page N of <title>`` alt, the ``Page N`` caption, the NO
|
descriptions RIDE the shown text — LOCKED A1/A6), the
|
||||||
``.doc-raw`` / NO ``.doc-md`` / NO ``.doc-summary`` suppression
|
``.doc-pages`` strip mounted but hidden; pressing ``Pages`` shows
|
||||||
contract (the mock's page descriptions sit in the DB content but
|
exactly three ``.doc-page`` figures, each a LOADED lazy ``<img>``
|
||||||
nowhere in the rendered modal), the meta badge row (source /
|
(``complete && naturalWidth > 0`` — the route served a real PNG,
|
||||||
Created / Indexed) untouched.
|
not a 404) with the ``Page N of <title>`` alt and the ``Page N``
|
||||||
|
caption while the text view is hidden (mounted for the flip-
|
||||||
|
back), the meta badge row (source / Created / Indexed) untouched.
|
||||||
2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the
|
2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the
|
||||||
PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the
|
PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the
|
||||||
Summary panel (the mock's lite digest), NO ``.doc-pages`` — and
|
Summary panel (the mock's lite digest), NO ``.doc-pages`` — and
|
||||||
the content payload's ``page_urls`` KEY is ABSENT (never ``null`` —
|
the content payload's ``page_urls`` KEY is ABSENT (never ``null`` —
|
||||||
the omission pin the viewer's strip gate rides on).
|
the omission pin the viewer's strip gate rides on).
|
||||||
3. ``test_full_page_surface`` — the same strip on the dedicated
|
3. ``test_full_page_surface`` — the same additive view + strip on the
|
||||||
``/document.html`` page (admin-logged-in, the phase-79 gate
|
dedicated ``/document.html`` page (admin-logged-in, the phase-79
|
||||||
satisfied): one ``renderDocument`` core — no per-surface drift —
|
gate satisfied): one ``renderDocument`` core — no per-surface
|
||||||
same figures, same suppression.
|
drift — same toggle, same figures, same hidden-while-Text shape.
|
||||||
4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct
|
4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct
|
||||||
request on ``/api/documents/<id>/pages/1``: 200, ``image/png``,
|
request on ``/api/documents/<id>/pages/1``: 200, ``image/png``,
|
||||||
the ``Cache-Control: private, max-age=3600`` header, a non-empty
|
the ``Cache-Control: private, max-age=3600`` header, a non-empty
|
||||||
@@ -597,15 +602,25 @@ def _wait_img_loaded(img: Locator) -> None:
|
|||||||
def _assert_page_strip(
|
def _assert_page_strip(
|
||||||
content: Locator, title: str, doc_id: str, page_count: int = 3
|
content: Locator, title: str, doc_id: str, page_count: int = 3
|
||||||
) -> None:
|
) -> None:
|
||||||
"""The phase-148 strip contract inside a content container (the
|
"""The strip contract inside a content container (the ONE
|
||||||
ONE ``renderDocument`` core — modal and full page, no
|
``renderDocument`` core — modal and full page, no per-surface
|
||||||
per-surface drift): a ``.doc-pages`` strip with exactly
|
drift): a ``.doc-pages`` strip with exactly *page_count*
|
||||||
*page_count* ``.doc-page`` figures; each figure's lazy ``<img>``
|
``.doc-page`` figures; each figure's lazy ``<img>`` is LOADED (the
|
||||||
is LOADED (the route served a real PNG) with the ``Page N of
|
route served a real PNG) with the ``Page N of <title>`` alt and
|
||||||
<title>`` alt and the exact 1-based route src, and the
|
the exact 1-based route src, and the figcaption reads ``Page N``.
|
||||||
figcaption reads ``Page N``; the text content block AND the
|
|
||||||
Summary panel are SUPPRESSED (the descriptions stay in the DB,
|
Phase 163 (task 04) re-cut: this fixture is the PICTORIAL class —
|
||||||
not displayed)."""
|
an ADDITIVE doc (``page_strip_mode == "additive"``) that opens on
|
||||||
|
its TEXT view, so the strip is revealed by pressing the toggle's
|
||||||
|
``Pages`` button first (a primary/absent doc has no toggle — the
|
||||||
|
strip already IS the content). The text content block and the
|
||||||
|
Summary panel are HIDDEN while the strip shows (they stay mounted
|
||||||
|
for the flip back — LOCKED A1); a primary-class doc has no such
|
||||||
|
nodes at all, which the same ``not_to_be_visible`` assertion
|
||||||
|
covers."""
|
||||||
|
toggle = content.locator(".doc-view-toggle")
|
||||||
|
if toggle.count() > 0:
|
||||||
|
toggle.get_by_role("button", name="Pages").click()
|
||||||
strip = content.locator(".doc-pages")
|
strip = content.locator(".doc-pages")
|
||||||
expect(strip).to_have_count(1)
|
expect(strip).to_have_count(1)
|
||||||
figures = strip.locator(".doc-page")
|
figures = strip.locator(".doc-page")
|
||||||
@@ -624,12 +639,12 @@ def _assert_page_strip(
|
|||||||
)
|
)
|
||||||
_wait_img_loaded(img)
|
_wait_img_loaded(img)
|
||||||
expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}")
|
expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}")
|
||||||
# The suppression contract (the owner's TODO L5): no text content
|
# While the strip shows, the text view is not (phase 163: hidden,
|
||||||
# block, no Summary panel — the meta badge row is asserted by the
|
# not unmounted — the additive flip-back; a primary doc has no
|
||||||
# caller (it renders exactly as today).
|
# such nodes at all). The meta badge row is asserted by the caller.
|
||||||
expect(content.locator("pre.doc-raw")).to_have_count(0)
|
expect(content.locator("pre.doc-raw")).not_to_be_visible()
|
||||||
expect(content.locator(".doc-md")).to_have_count(0)
|
expect(content.locator(".doc-md")).not_to_be_visible()
|
||||||
expect(content.locator(".doc-summary")).to_have_count(0)
|
expect(content.locator(".doc-summary")).not_to_be_visible()
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -640,15 +655,18 @@ def _assert_page_strip(
|
|||||||
def test_pictorial_pdf_shows_pages_in_modal(
|
def test_pictorial_pdf_shows_pages_in_modal(
|
||||||
page: Page, app_url: str, seeded: dict[str, Any]
|
page: Page, app_url: str, seeded: dict[str, Any]
|
||||||
) -> None:
|
) -> None:
|
||||||
"""The IKEA-shaped PDF opens in the doc modal as a strip of its
|
"""The IKEA-shaped PDF opens in the doc modal as an ADDITIVE doc
|
||||||
REAL pages (the owner's L5 ask): ``.doc-pages`` with exactly three
|
(phase 163 — a usable text layer: the pictorial class): the TEXT
|
||||||
|
view first (Summary panel + content visible, the toggle's ``Text``
|
||||||
|
pressed — LOCKED A6, the mock's page descriptions ride the shown
|
||||||
|
text), with the strip of its REAL pages (the owner's L5 ask) one
|
||||||
|
``Pages`` click away: ``.doc-pages`` with exactly three
|
||||||
``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth
|
``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth
|
||||||
> 0 proves the full pipeline: sync → render → persist → route →
|
> 0 proves the full pipeline: sync → render → persist → route →
|
||||||
viewer) with the ``Page N of <title>`` alt and the ``Page N``
|
viewer) with the ``Page N of <title>`` alt and the ``Page N``
|
||||||
caption — and the suppression contract: NO text content
|
caption — while the text view is hidden (mounted for the
|
||||||
(``.doc-raw``), NO Summary panel (the mock's page descriptions are
|
flip-back); the meta badge row (source / Created / Indexed)
|
||||||
in the DB content, nowhere in the rendered modal); the meta badge
|
renders as today."""
|
||||||
row (source / Created / Indexed) renders as today."""
|
|
||||||
login(page, app_url) # lands on /sources.html
|
login(page, app_url) # lands on /sources.html
|
||||||
_drill_to_source(page, SOURCE_NAME)
|
_drill_to_source(page, SOURCE_NAME)
|
||||||
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
||||||
@@ -658,6 +676,29 @@ def test_pictorial_pdf_shows_pages_in_modal(
|
|||||||
)
|
)
|
||||||
|
|
||||||
content = page.locator("#doc-modal-content")
|
content = page.locator("#doc-modal-content")
|
||||||
|
# Phase 163 (task 04, LOCKED A1/A6): the pictorial class is an
|
||||||
|
# ADDITIVE doc — it opens on its TEXT view: Summary panel +
|
||||||
|
# content visible, the toggle's Text pressed, the strip mounted
|
||||||
|
# but hidden (its lazy imgs never fetched while hidden).
|
||||||
|
toggle = content.locator(".doc-view-toggle")
|
||||||
|
expect(toggle).to_have_count(1)
|
||||||
|
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
|
||||||
|
"aria-pressed", "true"
|
||||||
|
)
|
||||||
|
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
|
||||||
|
"aria-pressed", "false"
|
||||||
|
)
|
||||||
|
expect(content.locator("pre.doc-raw")).to_be_visible()
|
||||||
|
expect(content.locator(".doc-summary")).to_be_visible()
|
||||||
|
expect(content.locator(".doc-pages")).to_be_hidden()
|
||||||
|
# The text view carries the indexed content — the phase-140 page
|
||||||
|
# descriptions ride the SHOWN text for an additive doc (the text
|
||||||
|
# is never hidden: A1).
|
||||||
|
assert PICTORIAL_PAGE_ANSWER in content.inner_text(), (
|
||||||
|
"the additive text view shows the indexed content including "
|
||||||
|
"the described sparse pages"
|
||||||
|
)
|
||||||
|
# …and the strip is one Pages click away (the helper flips).
|
||||||
_assert_page_strip(content, seeded["title"], seeded["pictorial_id"])
|
_assert_page_strip(content, seeded["title"], seeded["pictorial_id"])
|
||||||
|
|
||||||
# The meta badge row is untouched by the strip (the pre-phase row).
|
# The meta badge row is untouched by the strip (the pre-phase row).
|
||||||
@@ -666,13 +707,6 @@ def test_pictorial_pdf_shows_pages_in_modal(
|
|||||||
expect(meta.locator(".doc-created")).to_contain_text("Created")
|
expect(meta.locator(".doc-created")).to_contain_text("Created")
|
||||||
expect(meta.locator(".doc-indexed")).to_contain_text("Indexed")
|
expect(meta.locator(".doc-indexed")).to_contain_text("Indexed")
|
||||||
|
|
||||||
# The suppression witness: the mock's canned page descriptions sit
|
|
||||||
# in the DB (the seed asserted it) but are NOT displayed.
|
|
||||||
assert PICTORIAL_PAGE_ANSWER not in content.inner_text(), (
|
|
||||||
"the described pages' text is in the DB for Q&A — it must not "
|
|
||||||
"be shown to the user (the owner's contract)"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# 2. The text-heavy PDF: byte-identical to pre-phase
|
# 2. The text-heavy PDF: byte-identical to pre-phase
|
||||||
@@ -730,10 +764,11 @@ def test_textheavy_pdf_unchanged(
|
|||||||
|
|
||||||
def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None:
|
def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None:
|
||||||
"""The dedicated ``/document.html`` page (admin-logged-in — the
|
"""The dedicated ``/document.html`` page (admin-logged-in — the
|
||||||
phase-79 gate satisfied) renders the SAME strip through the ONE
|
phase-79 gate satisfied) renders the SAME additive view through
|
||||||
shared ``renderDocument`` core: the same three loaded figures with
|
the ONE shared ``renderDocument`` core (no per-surface drift): the
|
||||||
captions/alt texts, the same suppression (no text content, no
|
text view first, the strip one ``Pages`` click away with the same
|
||||||
Summary panel), the meta row as today."""
|
three loaded figures + captions/alt texts and the text view
|
||||||
|
hidden while it shows, the meta row as today."""
|
||||||
login(page, app_url, next="/")
|
login(page, app_url, next="/")
|
||||||
page.goto(
|
page.goto(
|
||||||
f"{app_url}/document.html?source={quote(SOURCE_NAME)}"
|
f"{app_url}/document.html?source={quote(SOURCE_NAME)}"
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -24,11 +24,14 @@ format — the upload gate admits it) and arriving through the REAL admin
|
|||||||
upload flow → the RAG page's Sync button:
|
upload flow → the RAG page's Sync button:
|
||||||
|
|
||||||
* (a) the document appears on the Sources page and the document viewer
|
* (a) the document appears on the Sources page and the document viewer
|
||||||
shows the PDF's ACTUAL pages (phase 148 — a described sparse page
|
opens the PDF in its ADDITIVE view (phase 163 — a described sparse
|
||||||
makes the doc page-QUALIFYING: the page strip REPLACES the text
|
page makes the doc page-QUALIFYING, and a usable text layer makes
|
||||||
content AND the Summary panel, the owner's TODO L5 contract): two
|
the strip ADDITIVE: the text stays, the pages sit behind a
|
||||||
``.doc-page`` figures with loaded lazy images + ``Page N`` captions,
|
Text/Pages toggle — LOCKED A1/A6): the Text view first (``.doc-raw``
|
||||||
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself
|
+ ``.doc-summary`` visible, ``Text`` pressed), the strip one
|
||||||
|
``Pages`` click away — two ``.doc-page`` figures with loaded lazy
|
||||||
|
images + ``Page N`` captions, the text view hidden while it shows;
|
||||||
|
the described content itself
|
||||||
(page 1's prose WITHOUT an illustration block, page 2 WITH
|
(page 1's prose WITHOUT an illustration block, page 2 WITH
|
||||||
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
|
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
|
||||||
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
|
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
|
||||||
@@ -83,14 +86,17 @@ Test → observable mapping (Playwright Mapping Rule):
|
|||||||
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
|
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
|
||||||
the summary chunk — no image doc), the drill-down table lists the PDF
|
the summary chunk — no image doc), the drill-down table lists the PDF
|
||||||
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
|
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
|
||||||
the same-page document viewer (the row's path link) shows the
|
the same-page document viewer (the row's path link) opens the
|
||||||
PHASE-148 PAGE STRIP under the METADATA title (the fixture is
|
PHASE-163 ADDITIVE VIEW under the METADATA title (the fixture is
|
||||||
page-qualifying — the pictorial pass described its sparse page):
|
page-qualifying with a usable text layer — the pictorial pass
|
||||||
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>``
|
described its sparse page): the TEXT view first — ``.doc-raw`` +
|
||||||
(naturalWidth > 0) with the ``Page N of <title>`` alt + the
|
``.doc-summary`` visible with the ``[Page 2 illustration: …]``
|
||||||
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the
|
block among them, the toggle's ``Text`` pressed, the strip mounted
|
||||||
page descriptions stay in the DB for Q&A, not displayed — the
|
but hidden; pressing ``Pages`` shows exactly two ``.doc-page``
|
||||||
owner's contract).
|
figures, each a LOADED lazy ``<img>`` (naturalWidth > 0) with the
|
||||||
|
``Page N of <title>`` alt + the ``Page N`` caption while the text
|
||||||
|
view hides (mounted for the flip-back — the text is never
|
||||||
|
destroyed).
|
||||||
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
|
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
|
||||||
SSE over the scripted seeded-read question (httpx — the ``done``
|
SSE over the scripted seeded-read question (httpx — the ``done``
|
||||||
event is the wire lens the task pins; the UI flow's premise is the
|
event is the wire lens the task pins; the UI flow's premise is the
|
||||||
@@ -578,8 +584,8 @@ def seeded(
|
|||||||
)
|
)
|
||||||
# The non-markdown doc got its lite summary (the mock's
|
# The non-markdown doc got its lite summary (the mock's
|
||||||
# SUMMARY_MODE digest) — the summary chunk exists (the
|
# SUMMARY_MODE digest) — the summary chunk exists (the
|
||||||
# viewer's Summary panel is suppressed by the phase-148
|
# additive view shows the Summary panel; the summary
|
||||||
# strip — the summary itself stays in the DB).
|
# itself stays in the DB regardless).
|
||||||
assert doc.summary
|
assert doc.summary
|
||||||
# Phase 148: the described sparse page makes this doc
|
# Phase 148: the described sparse page makes this doc
|
||||||
# page-QUALIFYING — the strip's payload (page_urls) rides
|
# page-QUALIFYING — the strip's payload (page_urls) rides
|
||||||
@@ -602,18 +608,19 @@ def test_sources_page_and_viewer_show_the_described_page(
|
|||||||
content chunks + the phase-30 summary chunk — no image doc; the
|
content chunks + the phase-30 summary chunk — no image doc; the
|
||||||
vector figure is invisible to the embedded-image pipeline). The
|
vector figure is invisible to the embedded-image pipeline). The
|
||||||
drill-down table lists the PDF as a plain text row (LOCKED A4), and
|
drill-down table lists the PDF as a plain text row (LOCKED A4), and
|
||||||
the same-page document viewer (the row's path link, phase 26) shows
|
the same-page document viewer (the row's path link, phase 26)
|
||||||
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is
|
opens the PHASE-163 ADDITIVE VIEW (LOCKED A1/A6): the TEXT view
|
||||||
page-qualifying (the pictorial pass described its sparse page), so
|
first — Summary panel + ``.doc-raw`` visible, the Text/Pages
|
||||||
the strip REPLACES the text content AND the Summary panel — two
|
toggle's ``Text`` pressed — with the ``[Page 2 illustration:
|
||||||
``.doc-page`` figures, each a loaded lazy ``<img>`` with the
|
<canned>]`` block riding the SHOWN text (the text is never hidden
|
||||||
``Page N of <title>`` alt + the ``Page N`` caption, NO
|
for the pictorial class). Pressing ``Pages`` reveals the strip —
|
||||||
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's
|
two ``.doc-page`` figures, each a loaded lazy ``<img>`` with the
|
||||||
prose WITHOUT an illustration block, page 2 WITH
|
``Page N of <title>`` alt + the ``Page N`` caption — while the
|
||||||
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB
|
text view is hidden (mounted, not removed: flipping back shows it
|
||||||
for Q&A: the seed asserted it byte-exact, the chat test proves it
|
with zero refetch). The described content (page 1's prose WITHOUT
|
||||||
reaches the model — it is simply not displayed (the owner's
|
an illustration block, page 2 WITH the block — LOCKED A4) also
|
||||||
contract)."""
|
stays in the DB for Q&A: the seed asserted it byte-exact, the chat
|
||||||
|
test proves it reaches the model."""
|
||||||
login(page, app_url) # lands on /sources.html
|
login(page, app_url) # lands on /sources.html
|
||||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||||
expect(page.locator("#stat-chunks")).to_have_text("3")
|
expect(page.locator("#stat-chunks")).to_have_text("3")
|
||||||
@@ -630,13 +637,31 @@ def test_sources_page_and_viewer_show_the_described_page(
|
|||||||
expect(modal).to_be_visible(timeout=30_000)
|
expect(modal).to_be_visible(timeout=30_000)
|
||||||
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
||||||
|
|
||||||
# Phase 148: the page-qualifying fixture shows its ACTUAL pages —
|
# Phase 163 (task 04, LOCKED A1/A6): the page-qualifying fixture
|
||||||
# a .doc-pages strip with exactly two .doc-page figures (the
|
# with a usable text layer is an ADDITIVE doc — it opens on its
|
||||||
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 —
|
# TEXT view: Summary panel + content visible, Text pressed, the
|
||||||
# the route served a real PNG) with the "Page N of <title>" alt +
|
# strip mounted but hidden (its lazy imgs never fetched).
|
||||||
# the "Page N" caption. The text content AND the Summary panel are
|
toggle = modal.locator(".doc-view-toggle")
|
||||||
# SUPPRESSED (the owner's contract — the descriptions stay in the
|
expect(toggle).to_have_count(1)
|
||||||
# DB for Q&A, not displayed).
|
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
|
||||||
|
"aria-pressed", "true"
|
||||||
|
)
|
||||||
|
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
|
||||||
|
"aria-pressed", "false"
|
||||||
|
)
|
||||||
|
expect(modal.locator("pre.doc-raw")).to_be_visible()
|
||||||
|
expect(modal.locator(".doc-summary")).to_be_visible()
|
||||||
|
expect(modal.locator(".doc-pages")).to_be_hidden()
|
||||||
|
# The canned page description RIDES the shown text (phase-140
|
||||||
|
# blocks are part of the content — the additive view never hides
|
||||||
|
# the text). The node stays in the DOM, so a visibility-scoped
|
||||||
|
# inner_text() check would miss it — use the DOM text.
|
||||||
|
assert PICTORIAL_PAGE_ANSWER in modal.evaluate("el => el.textContent"), (
|
||||||
|
"the additive text view shows the indexed content including "
|
||||||
|
"the described sparse page"
|
||||||
|
)
|
||||||
|
# …and the strip is one Pages click away (phase 148's ask):
|
||||||
|
toggle.get_by_role("button", name="Pages").click()
|
||||||
strip = modal.locator(".doc-pages")
|
strip = modal.locator(".doc-pages")
|
||||||
expect(strip).to_have_count(1)
|
expect(strip).to_have_count(1)
|
||||||
figures = strip.locator(".doc-page")
|
figures = strip.locator(".doc-page")
|
||||||
@@ -657,17 +682,12 @@ def test_sources_page_and_viewer_show_the_described_page(
|
|||||||
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
|
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
|
||||||
f"Page {n}"
|
f"Page {n}"
|
||||||
)
|
)
|
||||||
# The suppression contract: no text content block, no Summary
|
# While the strip shows, the text view is HIDDEN — mounted for
|
||||||
# panel, no image block (the PDF is not an image document).
|
# the flip back (LOCKED A1), never removed. No image block (the
|
||||||
expect(modal.locator("pre.doc-raw")).to_have_count(0)
|
# PDF is not an image document).
|
||||||
expect(modal.locator(".doc-summary")).to_have_count(0)
|
expect(modal.locator("pre.doc-raw")).not_to_be_visible()
|
||||||
|
expect(modal.locator(".doc-summary")).not_to_be_visible()
|
||||||
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
||||||
# The canned description is in the DB (the seed asserted it)
|
|
||||||
# but nowhere in the rendered modal.
|
|
||||||
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
|
|
||||||
"the described page's text is in the DB for Q&A — it must not "
|
|
||||||
"be shown to the user (the owner's contract)"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -10,7 +10,10 @@ per phase 124 A4) and the phase-148 (task 02) PDF page route
|
|||||||
(``GET /api/documents/{id}/pages/{n}`` — the same trust model and 404
|
(``GET /api/documents/{id}/pages/{n}`` — the same trust model and 404
|
||||||
posture for a page-qualifying document's rendered page PNGs) + the
|
posture for a page-qualifying document's rendered page PNGs) + the
|
||||||
content endpoint's ``image_url`` / ``page_urls`` keys (present for the
|
content endpoint's ``image_url`` / ``page_urls`` keys (present for the
|
||||||
qualifying row, ABSENT otherwise — never ``null``).
|
qualifying row, ABSENT otherwise — never ``null``), plus the phase-163
|
||||||
|
(task 01) ``page_strip_mode`` class signal (``"primary"`` /``"additive"``
|
||||||
|
for a stamped strip, ABSENT for a pre-phase row and for every
|
||||||
|
non-qualifying document).
|
||||||
|
|
||||||
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
|
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
|
||||||
"""
|
"""
|
||||||
@@ -1381,7 +1384,8 @@ def test_content_endpoint_exposes_the_image_affordance(
|
|||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Phase 148 (task 02) — the PDF page BYTES route + the content payload's
|
# Phase 148 (task 02) — the PDF page BYTES route + the content payload's
|
||||||
# ``page_urls`` affordance
|
# ``page_urls`` affordance (phase 163 task 01 adds the ``page_strip_mode``
|
||||||
|
# class signal beside it)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
@@ -1404,6 +1408,7 @@ def _seed_page_doc(
|
|||||||
path: str = "besta.pdf",
|
path: str = "besta.pdf",
|
||||||
page_count: int | None = 2,
|
page_count: int | None = 2,
|
||||||
write_copies: bool = True,
|
write_copies: bool = True,
|
||||||
|
page_strip_mode: str | None = None,
|
||||||
) -> Document:
|
) -> Document:
|
||||||
"""One PDF document row with ``page_count`` + its rendered page
|
"""One PDF document row with ``page_count`` + its rendered page
|
||||||
copies (the importer's ``pdf_page_dir`` layout —
|
copies (the importer's ``pdf_page_dir`` layout —
|
||||||
@@ -1411,7 +1416,8 @@ def _seed_page_doc(
|
|||||||
sentinel bytes, so a page mix-up between the two copies is caught.
|
sentinel bytes, so a page mix-up between the two copies is caught.
|
||||||
``page_count=None`` is a non-qualifying PDF (no copies);
|
``page_count=None`` is a non-qualifying PDF (no copies);
|
||||||
``write_copies=False`` leaves a qualifying row without its files
|
``write_copies=False`` leaves a qualifying row without its files
|
||||||
(the lost-copy corner)."""
|
(the lost-copy corner); ``page_strip_mode`` (phase 163) stamps the
|
||||||
|
strip CLASS — ``None`` is the pre-phase row."""
|
||||||
doc = Document(
|
doc = Document(
|
||||||
id=uuid.uuid4(),
|
id=uuid.uuid4(),
|
||||||
source=source,
|
source=source,
|
||||||
@@ -1424,6 +1430,7 @@ def _seed_page_doc(
|
|||||||
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
||||||
summary="IKEA BESTA frame assembly manual (page descriptions).",
|
summary="IKEA BESTA frame assembly manual (page descriptions).",
|
||||||
page_count=page_count,
|
page_count=page_count,
|
||||||
|
page_strip_mode=page_strip_mode,
|
||||||
)
|
)
|
||||||
db.add(doc)
|
db.add(doc)
|
||||||
db.flush()
|
db.flush()
|
||||||
@@ -1610,6 +1617,79 @@ def test_content_endpoint_exposes_the_page_urls_affordance(
|
|||||||
)
|
)
|
||||||
assert r.status_code == 200
|
assert r.status_code == 200
|
||||||
assert "page_urls" not in r.json() # absent — never null (non-qualifying PDF)
|
assert "page_urls" not in r.json() # absent — never null (non-qualifying PDF)
|
||||||
|
# Phase 163 (task 01): the class key follows the same rule — a
|
||||||
|
# pre-phase row (mode NULL) omits it while keeping page_urls.
|
||||||
|
assert "page_strip_mode" not in r.json()
|
||||||
|
finally:
|
||||||
|
_cleanup_kb(db)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mode", ["primary", "additive"])
|
||||||
|
def test_content_endpoint_carries_the_strip_class(
|
||||||
|
admin_client: TestClient, db, tmp_path: Path, mode: str
|
||||||
|
) -> None:
|
||||||
|
"""Phase 163 (task 01, LOCKED A2/A3) — the real-DB round-trip of the
|
||||||
|
strip CLASS: a stamped row (``page_strip_mode`` = ``"primary"`` — the
|
||||||
|
phase-133 fallback class, the strip IS the view — or ``"additive"`` —
|
||||||
|
the phase-140 pictorial class, text + a Text/Pages toggle) serves the
|
||||||
|
class VERBATIM beside the untouched ``page_urls`` list (the viewer's
|
||||||
|
branch signal, task 04); every other key stands where phase 148 left
|
||||||
|
it."""
|
||||||
|
_cleanup_kb(db)
|
||||||
|
try:
|
||||||
|
pages_dir = tmp_path / "pages"
|
||||||
|
_seed_page_doc(db, pages_dir, page_count=3, page_strip_mode=mode)
|
||||||
|
r = admin_client.get(
|
||||||
|
"/api/documents/content", params={"source": "PdfSrc", "path": "besta.pdf"}
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["page_strip_mode"] == mode
|
||||||
|
assert len(body["page_urls"]) == 3 # the strip rides along, unchanged
|
||||||
|
finally:
|
||||||
|
_cleanup_kb(db)
|
||||||
|
|
||||||
|
|
||||||
|
def test_content_strip_class_never_travels_without_the_strip(
|
||||||
|
admin_client: TestClient, db, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 163 (task 01) — the two strip keys are ONE block keyed off
|
||||||
|
``page_count``: a row with a stray ``page_strip_mode`` and no strip
|
||||||
|
(``page_count`` NULL — the corner a partial repair could leave)
|
||||||
|
serializes NEITHER key, byte-identically to a plain non-qualifying
|
||||||
|
document (the viewer must never be handed a class for pages that do
|
||||||
|
not exist)."""
|
||||||
|
_cleanup_kb(db)
|
||||||
|
try:
|
||||||
|
pages_dir = tmp_path / "pages"
|
||||||
|
_seed_page_doc(
|
||||||
|
db,
|
||||||
|
pages_dir,
|
||||||
|
path="stray.pdf",
|
||||||
|
page_count=None,
|
||||||
|
write_copies=False,
|
||||||
|
page_strip_mode="additive",
|
||||||
|
)
|
||||||
|
r = admin_client.get(
|
||||||
|
"/api/documents/content", params={"source": "PdfSrc", "path": "stray.pdf"}
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert "page_strip_mode" not in body
|
||||||
|
assert "page_urls" not in body
|
||||||
|
# The exact pre-phase key set (the byte-identity pin).
|
||||||
|
assert set(body) == {
|
||||||
|
"source",
|
||||||
|
"path",
|
||||||
|
"title",
|
||||||
|
"format",
|
||||||
|
"summary",
|
||||||
|
"created_at",
|
||||||
|
"content",
|
||||||
|
"indexed_at",
|
||||||
|
"chunks",
|
||||||
|
"is_image",
|
||||||
|
}
|
||||||
finally:
|
finally:
|
||||||
_cleanup_kb(db)
|
_cleanup_kb(db)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,370 @@
|
|||||||
|
"""Integration: migration 0027 (documents.page_strip_mode) schema contract
|
||||||
|
(phase 163, task 01).
|
||||||
|
|
||||||
|
Drives the **real Alembic engine** against the live dev database
|
||||||
|
(``podman compose up -d db``), mirroring the house pattern of
|
||||||
|
``test_migration_0026.py`` (information_schema assertions on the state
|
||||||
|
the migration must leave). The tests target the 0026 → 0027 step
|
||||||
|
explicitly so later migrations cannot break the pins:
|
||||||
|
|
||||||
|
* upgrade 0026 → 0027 → ``page_strip_mode`` exists with the full
|
||||||
|
contract — VARCHAR(16), NULLABLE, NO server default (NULL = "no strip"
|
||||||
|
for every pre-phase row, every non-qualifying document and every
|
||||||
|
fail-soft miss) and UNINDEXED (a display fact, never a query surface —
|
||||||
|
the content endpoint reads it from the ROW, the ``file_hash`` /
|
||||||
|
``page_count`` convention) — while the 0026 ``documents`` schema
|
||||||
|
survives, in particular ``page_count`` keeping its 0026 contract and
|
||||||
|
``content_hash`` staying NOT NULL and INDEXED;
|
||||||
|
* rows inserted while the DB is at 0026 backfill ``page_strip_mode`` to
|
||||||
|
NULL — including a row whose ``page_count`` IS set (the pre-phase strip
|
||||||
|
with no stamped class: the wire omits the key and the viewer keeps
|
||||||
|
today's "primary" DOM until the task-03 re-gate stamps it);
|
||||||
|
* the ORM contract agrees: a freshly inserted ``Document`` with no mode
|
||||||
|
reads ``page_strip_mode is None`` through a fresh session, and each of
|
||||||
|
the two classes (``"primary"`` / ``"additive"``) round-trips verbatim;
|
||||||
|
* downgrade to 0026 → the column is GONE (A13) while the row + its
|
||||||
|
content + its ``page_count`` survive; upgrade back to 0027 → the column
|
||||||
|
is back (round-trip).
|
||||||
|
|
||||||
|
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||||
|
fails or the process is interrupted.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import uuid
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from alembic.config import Config
|
||||||
|
from sqlalchemy import text
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from alembic import command
|
||||||
|
from app.db import SessionLocal, db_available
|
||||||
|
from app.models import Document
|
||||||
|
|
||||||
|
SOURCE = "mig0027"
|
||||||
|
CONTENT = "content here"
|
||||||
|
CONTENT_HASH = hashlib.sha256(CONTENT.encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def alembic(db: Session) -> Iterator[Config]:
|
||||||
|
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||||
|
|
||||||
|
Starts at head (repairs an interrupted earlier run); teardown
|
||||||
|
upgrades to head no matter what happened, so the dev DB is never
|
||||||
|
left below head.
|
||||||
|
"""
|
||||||
|
if not db_available():
|
||||||
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||||
|
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||||
|
cfg.set_main_option("script_location", "alembic")
|
||||||
|
command.upgrade(cfg, "head")
|
||||||
|
try:
|
||||||
|
yield cfg
|
||||||
|
finally:
|
||||||
|
# Release the test session's open transaction BEFORE the repair
|
||||||
|
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
|
||||||
|
# on ``documents``, which would deadlock the repair's
|
||||||
|
# ``ALTER TABLE`` forever.
|
||||||
|
db.rollback()
|
||||||
|
command.upgrade(cfg, "head")
|
||||||
|
|
||||||
|
|
||||||
|
def _version(db: Session) -> str | None:
|
||||||
|
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||||
|
|
||||||
|
|
||||||
|
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||||
|
"""(data_type, is_nullable, column_default, character_maximum_length)
|
||||||
|
for one documents column."""
|
||||||
|
row = db.execute(
|
||||||
|
text(
|
||||||
|
"SELECT data_type, is_nullable, column_default, character_maximum_length"
|
||||||
|
" FROM information_schema.columns"
|
||||||
|
" WHERE table_name = 'documents' AND column_name = :c"
|
||||||
|
),
|
||||||
|
{"c": column},
|
||||||
|
).fetchone()
|
||||||
|
return tuple(row) if row is not None else None
|
||||||
|
|
||||||
|
|
||||||
|
def _index_covering(db: Session, column: str) -> str | None:
|
||||||
|
"""The name of any documents index whose definition mentions the
|
||||||
|
column (None = UNINDEXED)."""
|
||||||
|
return db.execute(
|
||||||
|
text(
|
||||||
|
"SELECT indexname FROM pg_indexes"
|
||||||
|
" WHERE tablename = 'documents' AND indexdef ILIKE :like"
|
||||||
|
),
|
||||||
|
{"like": f"%{column}%"},
|
||||||
|
).scalar()
|
||||||
|
|
||||||
|
|
||||||
|
def _insert_sql(db: Session, path: str, page_count: int | None = None) -> uuid.UUID:
|
||||||
|
"""Insert one documents row with the PRE-0027 column set (the 0026
|
||||||
|
shape — the mode column, when present, is omitted so a NULL backfill
|
||||||
|
is what the row reads; ``page_count`` is 0026's own column and may be
|
||||||
|
set to model a pre-phase strip)."""
|
||||||
|
doc_id = uuid.uuid4()
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"INSERT INTO documents (id, source, path, full_path, title, content,"
|
||||||
|
" content_hash, page_count)"
|
||||||
|
" VALUES (:id, :s, :p, :fp, :t, :c, :h, :pc)"
|
||||||
|
),
|
||||||
|
{
|
||||||
|
"id": doc_id,
|
||||||
|
"s": SOURCE,
|
||||||
|
"p": path,
|
||||||
|
"fp": f"/tmp/{path}",
|
||||||
|
"t": f"Doc {path}",
|
||||||
|
"c": CONTENT,
|
||||||
|
"h": CONTENT_HASH,
|
||||||
|
"pc": page_count,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
return doc_id
|
||||||
|
|
||||||
|
|
||||||
|
def _delete(db: Session, doc_id: uuid.UUID) -> None:
|
||||||
|
db.execute(text("DELETE FROM documents WHERE id = :id"), {"id": doc_id})
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _assert_0026_schema_survives(db: Session) -> None:
|
||||||
|
"""The 0026 documents contract must survive the additive upgrade —
|
||||||
|
``page_count`` keeps its INTEGER / NULLABLE / NO-default / UNINDEXED
|
||||||
|
shape (the sibling column this one rides) and ``content_hash`` stays
|
||||||
|
NOT NULL and indexed (the retriever's identity)."""
|
||||||
|
page_count = _column(db, "page_count")
|
||||||
|
assert page_count is not None, "documents.page_count (0026) must survive the upgrade"
|
||||||
|
assert page_count[0] == "integer", "page_count must stay INTEGER"
|
||||||
|
assert page_count[1] == "YES", "page_count must stay NULLABLE"
|
||||||
|
assert page_count[2] is None, "page_count must keep NO server default"
|
||||||
|
assert _index_covering(db, "page_count") is None, "page_count must stay UNINDEXED"
|
||||||
|
|
||||||
|
content = _column(db, "content_hash")
|
||||||
|
assert content is not None and content[0] == "character varying" and content[1] == "NO", (
|
||||||
|
"documents.content_hash (0001) must keep its contract"
|
||||||
|
)
|
||||||
|
assert (
|
||||||
|
_index_covering(db, "content_hash") is not None
|
||||||
|
), "the ix_documents_content_hash index must survive the upgrade"
|
||||||
|
|
||||||
|
indexed = _column(db, "indexed_at")
|
||||||
|
assert indexed is not None and indexed[0] == "timestamp with time zone", (
|
||||||
|
"documents.indexed_at (0001) must survive the upgrade"
|
||||||
|
)
|
||||||
|
is_image = _column(db, "is_image")
|
||||||
|
assert is_image is not None and is_image[0] == "boolean" and is_image[1] == "NO", (
|
||||||
|
"documents.is_image (0022) must survive the upgrade"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_to_0027_adds_page_strip_mode(db: Session, alembic: Config) -> None:
|
||||||
|
"""Upgrade 0026 → 0027: ``page_strip_mode`` exists with the full
|
||||||
|
contract (VARCHAR(16), NULLABLE, NO server default — NULL = "no
|
||||||
|
strip" — and UNINDEXED: a display fact, never a query surface), is
|
||||||
|
ABSENT at 0026, and every pre-0027 row backfills it to NULL —
|
||||||
|
including the row whose ``page_count`` IS set (the pre-phase strip
|
||||||
|
with no stamped class), while the 0026 table contract (the
|
||||||
|
``page_count`` sibling, the indexed ``content_hash``) survives.
|
||||||
|
"""
|
||||||
|
command.downgrade(alembic, "0026") # start from the pre-0027 state
|
||||||
|
assert _version(db) == "0026"
|
||||||
|
assert _column(db, "page_strip_mode") is None, "page_strip_mode must be absent at 0026"
|
||||||
|
|
||||||
|
plain_id = _insert_sql(db, "plain.pdf") # the 0026 column set, no strip
|
||||||
|
pre_strip_id = _insert_sql(db, "prephase.pdf", page_count=36) # a live pre-phase strip
|
||||||
|
|
||||||
|
try:
|
||||||
|
command.upgrade(alembic, "0027")
|
||||||
|
assert _version(db) == "0027", "alembic_version must be at 0027"
|
||||||
|
|
||||||
|
col = _column(db, "page_strip_mode")
|
||||||
|
assert col is not None, "documents.page_strip_mode is missing"
|
||||||
|
assert col[0] == "character varying", "page_strip_mode must be VARCHAR"
|
||||||
|
assert col[3] == 16, "page_strip_mode must be VARCHAR(16) — the two class names"
|
||||||
|
assert col[1] == "YES", (
|
||||||
|
"page_strip_mode must be NULLABLE (NULL = no strip, and the pre-phase shape)"
|
||||||
|
)
|
||||||
|
assert col[2] is None, (
|
||||||
|
"page_strip_mode must carry NO server default — NULL is the meaningful"
|
||||||
|
" pre-phase state"
|
||||||
|
)
|
||||||
|
assert _index_covering(db, "page_strip_mode") is None, (
|
||||||
|
"page_strip_mode must be UNINDEXED — a display fact, never a query"
|
||||||
|
" surface (the content endpoint reads it from the row)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Every pre-0027 row backfills to NULL — no backfill, no default.
|
||||||
|
assert (
|
||||||
|
db.execute(
|
||||||
|
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
|
||||||
|
{"id": plain_id},
|
||||||
|
).fetchone()
|
||||||
|
== (None, None)
|
||||||
|
), "a pre-0027 row without a strip must read both columns NULL"
|
||||||
|
assert (
|
||||||
|
db.execute(
|
||||||
|
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
|
||||||
|
{"id": pre_strip_id},
|
||||||
|
).fetchone()
|
||||||
|
== (36, None)
|
||||||
|
), (
|
||||||
|
"a pre-phase strip (page_count set) must backfill page_strip_mode to"
|
||||||
|
" NULL — the wire omits the key and the viewer keeps today's DOM"
|
||||||
|
)
|
||||||
|
|
||||||
|
_assert_0026_schema_survives(db)
|
||||||
|
finally:
|
||||||
|
_delete(db, plain_id)
|
||||||
|
_delete(db, pre_strip_id)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mode", ["primary", "additive"])
|
||||||
|
def test_orm_page_strip_mode_round_trips(db: Session, alembic: Config, mode: str) -> None:
|
||||||
|
"""The ORM contract agrees with the column contract: each class
|
||||||
|
(``"primary"`` = the fallback class, ``"additive"`` = the pictorial
|
||||||
|
class) round-trips VERBATIM through a fresh session, and a row
|
||||||
|
created with no mode reads ``page_strip_mode is None`` (the
|
||||||
|
pre-phase shape — NULL until the importer stamps it)."""
|
||||||
|
command.upgrade(alembic, "head")
|
||||||
|
stamped = Document(
|
||||||
|
source=SOURCE,
|
||||||
|
path="stamped.pdf",
|
||||||
|
full_path="/tmp/stamped.pdf",
|
||||||
|
title="stamped",
|
||||||
|
content=CONTENT,
|
||||||
|
content_hash=CONTENT_HASH,
|
||||||
|
page_count=36,
|
||||||
|
page_strip_mode=mode,
|
||||||
|
)
|
||||||
|
unstamped = Document(
|
||||||
|
source=SOURCE,
|
||||||
|
path="unstamped.md",
|
||||||
|
full_path="/tmp/unstamped.md",
|
||||||
|
title="unstamped",
|
||||||
|
content=CONTENT,
|
||||||
|
content_hash=CONTENT_HASH,
|
||||||
|
)
|
||||||
|
db.add(stamped)
|
||||||
|
db.add(unstamped)
|
||||||
|
db.commit()
|
||||||
|
try:
|
||||||
|
with SessionLocal() as fresh:
|
||||||
|
reloaded = fresh.get(Document, stamped.id)
|
||||||
|
assert reloaded is not None, "the stamped row must be readable"
|
||||||
|
assert reloaded.page_strip_mode == mode, (
|
||||||
|
f"the {mode!r} class must round-trip through the DB verbatim"
|
||||||
|
)
|
||||||
|
assert reloaded.page_count == 36, "the sibling page_count must round-trip too"
|
||||||
|
reloaded_unstamped = fresh.get(Document, unstamped.id)
|
||||||
|
assert reloaded_unstamped is not None, "the unstamped row must be readable"
|
||||||
|
assert reloaded_unstamped.page_strip_mode is None, (
|
||||||
|
"a row created without a mode must read NULL (the pre-phase shape)"
|
||||||
|
)
|
||||||
|
assert reloaded_unstamped.content_hash == CONTENT_HASH, (
|
||||||
|
"content_hash must be untouched by the page_strip_mode addition"
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_delete(db, stamped.id)
|
||||||
|
_delete(db, unstamped.id)
|
||||||
|
|
||||||
|
|
||||||
|
def test_downgrade_to_0026_drops_the_column(db: Session, alembic: Config) -> None:
|
||||||
|
"""Downgrade 0027 → 0026: the page_strip_mode column is gone (A13 —
|
||||||
|
fully reversible, the class is re-derivable from the source file by
|
||||||
|
the deterministic re-gate) while the row + its content + its 0026
|
||||||
|
``page_count`` survive, and the rest of the table contract is intact."""
|
||||||
|
command.upgrade(alembic, "head")
|
||||||
|
row = Document(
|
||||||
|
source=SOURCE,
|
||||||
|
path="down.pdf",
|
||||||
|
full_path="/tmp/down.pdf",
|
||||||
|
title="down",
|
||||||
|
content=CONTENT,
|
||||||
|
content_hash=CONTENT_HASH,
|
||||||
|
page_count=36,
|
||||||
|
page_strip_mode="additive",
|
||||||
|
)
|
||||||
|
db.add(row)
|
||||||
|
db.commit()
|
||||||
|
try:
|
||||||
|
command.downgrade(alembic, "0026")
|
||||||
|
assert _version(db) == "0026"
|
||||||
|
assert _column(db, "page_strip_mode") is None, "page_strip_mode must be dropped"
|
||||||
|
|
||||||
|
surviving = db.execute(
|
||||||
|
text(
|
||||||
|
"SELECT source, path, content, content_hash, page_count, is_image"
|
||||||
|
" FROM documents WHERE id = :id"
|
||||||
|
),
|
||||||
|
{"id": row.id},
|
||||||
|
).fetchone()
|
||||||
|
assert surviving is not None and surviving[0] == SOURCE, (
|
||||||
|
"the row must survive the column drop"
|
||||||
|
)
|
||||||
|
assert surviving[1] == "down.pdf" and surviving[2] == CONTENT, (
|
||||||
|
"path + content must survive the drop"
|
||||||
|
)
|
||||||
|
assert surviving[3] == CONTENT_HASH, (
|
||||||
|
"content_hash (the retriever's identity) must survive the drop"
|
||||||
|
)
|
||||||
|
assert surviving[4] == 36, (
|
||||||
|
"page_count (the 0026 sibling) must survive the drop — the strip"
|
||||||
|
" itself is untouched by this migration's reversal"
|
||||||
|
)
|
||||||
|
assert surviving[5] is False, "is_image must survive the drop"
|
||||||
|
assert _index_covering(db, "content_hash") is not None, (
|
||||||
|
"the content_hash index must survive the downgrade"
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_delete(db, row.id)
|
||||||
|
# Repair: the fixture teardown re-upgrades to head.
|
||||||
|
|
||||||
|
|
||||||
|
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
|
||||||
|
"""Downgrade to 0026, then upgrade back to 0027: ``page_strip_mode``
|
||||||
|
is back with the full contract (VARCHAR(16), NULLABLE, NO server
|
||||||
|
default, UNINDEXED), a row seeded at 0026 — strip included — reads
|
||||||
|
the mode as NULL (the pre-phase shape on the round-trip), and the
|
||||||
|
0026 schema (the ``page_count`` sibling, the indexed
|
||||||
|
``content_hash``) is intact."""
|
||||||
|
command.downgrade(alembic, "0026")
|
||||||
|
pre_id = _insert_sql(db, "round.pdf", page_count=36)
|
||||||
|
try:
|
||||||
|
command.upgrade(alembic, "0027")
|
||||||
|
assert _version(db) == "0027", "round-trip upgrade must land at 0027"
|
||||||
|
|
||||||
|
col = _column(db, "page_strip_mode")
|
||||||
|
assert col is not None, "documents.page_strip_mode must be back"
|
||||||
|
assert col[0] == "character varying", "page_strip_mode must be VARCHAR after the round-trip"
|
||||||
|
assert col[3] == 16, "page_strip_mode must be VARCHAR(16) after the round-trip"
|
||||||
|
assert col[1] == "YES", "page_strip_mode must be NULLABLE after the round-trip"
|
||||||
|
assert col[2] is None, (
|
||||||
|
"page_strip_mode must still carry NO server default after the round-trip"
|
||||||
|
)
|
||||||
|
assert _index_covering(db, "page_strip_mode") is None, (
|
||||||
|
"page_strip_mode must stay UNINDEXED after the round-trip"
|
||||||
|
)
|
||||||
|
|
||||||
|
assert (
|
||||||
|
db.execute(
|
||||||
|
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
|
||||||
|
{"id": pre_id},
|
||||||
|
).fetchone()
|
||||||
|
== (36, None)
|
||||||
|
), (
|
||||||
|
"a row seeded at 0026 must read page_strip_mode NULL after the"
|
||||||
|
" round-trip while its page_count stands"
|
||||||
|
)
|
||||||
|
|
||||||
|
_assert_0026_schema_survives(db)
|
||||||
|
finally:
|
||||||
|
_delete(db, pre_id)
|
||||||
@@ -30,6 +30,17 @@ page-qualifying PDF row (``page_count`` set) adds ``page_urls`` = the
|
|||||||
exact 1-based ``/api/documents/<id>/pages/<i>`` list over
|
exact 1-based ``/api/documents/<id>/pages/<i>`` list over
|
||||||
``1..page_count``.
|
``1..page_count``.
|
||||||
|
|
||||||
|
Phase 163 (task 01) — the content payload's STRIP CLASS signal
|
||||||
|
(``page_strip_mode``: ``"primary"`` = the fallback class, the strip IS
|
||||||
|
the view; ``"additive"`` = the pictorial class, text + a Text/Pages
|
||||||
|
toggle): the model-level omission rule (ABSENT when ``None``, never
|
||||||
|
``null``), and the endpoint (stubbed session) — the key rides
|
||||||
|
``page_urls`` for a qualifying row, a PRE-PHASE row (``page_count`` set,
|
||||||
|
mode NULL — the exact live ``Letumdei`` state) serializes it ABSENT while
|
||||||
|
keeping ``page_urls``, and the whole strip block keys off
|
||||||
|
``page_count``, so a row with a stray mode and no strip carries neither
|
||||||
|
key (every non-qualifying payload byte-identical to pre-phase).
|
||||||
|
|
||||||
Phase 145 (task 03) — the content payload's git origin fields + the
|
Phase 145 (task 03) — the content payload's git origin fields + the
|
||||||
``renderDocument`` "Original" anchor (modal AND /document.html, one
|
``renderDocument`` "Original" anchor (modal AND /document.html, one
|
||||||
site):
|
site):
|
||||||
@@ -316,10 +327,11 @@ def _text_doc_content(
|
|||||||
origin_url: str | None = None,
|
origin_url: str | None = None,
|
||||||
origin_branch: str | None = None,
|
origin_branch: str | None = None,
|
||||||
page_urls: list[str] | None = None,
|
page_urls: list[str] | None = None,
|
||||||
|
page_strip_mode: str | None = None,
|
||||||
) -> DocContent:
|
) -> DocContent:
|
||||||
"""One text doc's content payload (the pre-phase shape) with the
|
"""One text doc's content payload (the pre-phase shape) with the
|
||||||
phase-145 origin fields + the phase-148 ``page_urls`` optionally
|
phase-145 origin fields + the phase-148 ``page_urls`` + the
|
||||||
set."""
|
phase-163 ``page_strip_mode`` optionally set."""
|
||||||
return DocContent(
|
return DocContent(
|
||||||
source="S",
|
source="S",
|
||||||
path="a.md",
|
path="a.md",
|
||||||
@@ -332,6 +344,7 @@ def _text_doc_content(
|
|||||||
origin_url=origin_url,
|
origin_url=origin_url,
|
||||||
origin_branch=origin_branch,
|
origin_branch=origin_branch,
|
||||||
page_urls=page_urls,
|
page_urls=page_urls,
|
||||||
|
page_strip_mode=page_strip_mode,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -367,9 +380,29 @@ def test_doc_content_page_urls_absent_on_the_wire_when_unset() -> None:
|
|||||||
assert dumped["page_urls"] == urls # the exact 1-based route list
|
assert dumped["page_urls"] == urls # the exact 1-based route list
|
||||||
|
|
||||||
|
|
||||||
def _page_pdf_doc() -> Document:
|
def test_doc_content_page_strip_mode_absent_on_the_wire_when_unset() -> None:
|
||||||
|
"""Phase 163 (task 01, LOCKED A2) — the model-level omission rule
|
||||||
|
(the ``image_url`` / ``page_urls`` precedent): no strip class
|
||||||
|
(``page_strip_mode`` ``None`` — every non-qualifying doc AND every
|
||||||
|
pre-phase row) serializes the key ABSENT — never ``null`` — so those
|
||||||
|
payloads are byte-identical to pre-phase. A stamped row carries the
|
||||||
|
class verbatim, beside ``page_urls``."""
|
||||||
|
dumped = _text_doc_content().model_dump()
|
||||||
|
assert "page_strip_mode" not in dumped # absent — never null
|
||||||
|
for mode in ("primary", "additive"):
|
||||||
|
dumped = _text_doc_content(
|
||||||
|
page_urls=["/api/documents/d/pages/1"], page_strip_mode=mode
|
||||||
|
).model_dump()
|
||||||
|
assert dumped["page_strip_mode"] == mode # the class verbatim
|
||||||
|
|
||||||
|
|
||||||
|
def _page_pdf_doc(
|
||||||
|
page_count: int | None = 2, page_strip_mode: str | None = None
|
||||||
|
) -> Document:
|
||||||
"""A page-QUALIFYING PDF row (phase 148): ``page_count`` set — the
|
"""A page-QUALIFYING PDF row (phase 148): ``page_count`` set — the
|
||||||
endpoint derives ``page_urls`` from it (the exact 1-based list)."""
|
endpoint derives ``page_urls`` from it (the exact 1-based list).
|
||||||
|
``page_strip_mode`` (phase 163) is the strip CLASS; ``None`` is the
|
||||||
|
PRE-PHASE row shape (the column NULL while the strip stands)."""
|
||||||
doc = Document(
|
doc = Document(
|
||||||
id=uuid.uuid4(),
|
id=uuid.uuid4(),
|
||||||
source="Homelab",
|
source="Homelab",
|
||||||
@@ -379,7 +412,8 @@ def _page_pdf_doc() -> Document:
|
|||||||
content="[Page 1 illustration: the parts tray]…",
|
content="[Page 1 illustration: the parts tray]…",
|
||||||
content_hash="e" * 64,
|
content_hash="e" * 64,
|
||||||
is_image=False, # phase 122: the stub session never applies defaults
|
is_image=False, # phase 122: the stub session never applies defaults
|
||||||
page_count=2,
|
page_count=page_count,
|
||||||
|
page_strip_mode=page_strip_mode,
|
||||||
)
|
)
|
||||||
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
|
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
|
||||||
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC)
|
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC)
|
||||||
@@ -410,6 +444,79 @@ database): a page-qualifying PDF row (``page_count`` set) carries
|
|||||||
] # the exact 1-based list, count == page_count
|
] # the exact 1-based list, count == page_count
|
||||||
assert body["is_image"] is False
|
assert body["is_image"] is False
|
||||||
assert "image_url" not in body # absent — never null (not an image doc)
|
assert "image_url" not in body # absent — never null (not an image doc)
|
||||||
|
# Phase 163 (task 01): the pre-phase row (mode NULL) keeps the strip
|
||||||
|
# and serializes the class ABSENT — the viewer treats that as
|
||||||
|
# "primary", i.e. today's DOM.
|
||||||
|
assert "page_strip_mode" not in body
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("mode", ["primary", "additive"])
|
||||||
|
def test_content_page_strip_mode_carried_for_a_stamped_strip(mode: str) -> None:
|
||||||
|
"""Phase 163 (task 01, LOCKED A2/A3) — the endpoint (stubbed
|
||||||
|
session, no
|
||||||
|
database): a page-qualifying PDF whose row carries the class
|
||||||
|
(``"primary"`` = the fallback class, ``"additive"`` = the pictorial
|
||||||
|
class) carries ``page_strip_mode`` VERBATIM next to ``page_urls`` —
|
||||||
|
the signal the viewer branches on so an additive strip never hides
|
||||||
|
the text (task 04). Nothing else in the payload moves."""
|
||||||
|
doc = _page_pdf_doc(page_strip_mode=mode)
|
||||||
|
with _client_with_row((doc, 4)) as client:
|
||||||
|
r = client.get(
|
||||||
|
"/api/documents/content",
|
||||||
|
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["page_strip_mode"] == mode # the class verbatim
|
||||||
|
assert body["page_urls"] == [
|
||||||
|
f"/api/documents/{doc.id}/pages/1",
|
||||||
|
f"/api/documents/{doc.id}/pages/2",
|
||||||
|
] # the strip rides along, unchanged
|
||||||
|
|
||||||
|
|
||||||
|
def test_content_strip_block_keys_off_page_count_never_mode_alone() -> None:
|
||||||
|
"""Phase 163 (task 01) — the class never appears alone: the whole
|
||||||
|
strip block keys off ``page_count``, so a row whose column somehow
|
||||||
|
holds a mode while ``page_count`` is NULL serializes NEITHER
|
||||||
|
``page_strip_mode`` nor ``page_urls`` — the payload is byte-identical
|
||||||
|
to a plain non-qualifying document (there is no strip to show, and
|
||||||
|
the viewer must never be told a class for pages that do not exist)."""
|
||||||
|
doc = _page_pdf_doc(page_count=None, page_strip_mode="additive")
|
||||||
|
with _client_with_row((doc, 1)) as client:
|
||||||
|
r = client.get(
|
||||||
|
"/api/documents/content",
|
||||||
|
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert "page_strip_mode" not in body # absent — the strip block is off
|
||||||
|
assert "page_urls" not in body # absent — never null
|
||||||
|
|
||||||
|
|
||||||
|
def test_content_non_qualifying_payload_key_set_is_byte_identical() -> None:
|
||||||
|
"""Phase 163 (task 01) — the byte-identity pin: a non-qualifying PDF
|
||||||
|
row (``page_count`` NULL, mode NULL) carries EXACTLY the pre-phase
|
||||||
|
key set — neither strip key joins it (``is_image`` always present,
|
||||||
|
``image_url`` / the origin keys / both strip keys absent)."""
|
||||||
|
doc = _page_pdf_doc(page_count=None)
|
||||||
|
with _client_with_row((doc, 1)) as client:
|
||||||
|
r = client.get(
|
||||||
|
"/api/documents/content",
|
||||||
|
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||||
|
)
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert set(r.json()) == {
|
||||||
|
"source",
|
||||||
|
"path",
|
||||||
|
"title",
|
||||||
|
"format",
|
||||||
|
"summary",
|
||||||
|
"created_at",
|
||||||
|
"content",
|
||||||
|
"indexed_at",
|
||||||
|
"chunks",
|
||||||
|
"is_image",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def test_content_origin_url_carried_for_a_matching_git_row() -> None:
|
def test_content_origin_url_carried_for_a_matching_git_row() -> None:
|
||||||
|
|||||||
@@ -31,6 +31,16 @@ the second sync):
|
|||||||
* the streamed hash is transparent (a >1 MiB file hashes identically
|
* the streamed hash is transparent (a >1 MiB file hashes identically
|
||||||
to ``hashlib.sha256(bytes).hexdigest()``).
|
to ``hashlib.sha256(bytes).hexdigest()``).
|
||||||
|
|
||||||
|
Phase 163 (task 03) extends this file with the PDF strip RE-GATE the
|
||||||
|
unchanged paths now run (``_regate_pdf_strip``) — the pre-phase "the
|
||||||
|
copies persist" assumption is what broke in production (a row claiming
|
||||||
|
``page_count = 36`` over ZERO files, every page 404ing forever): the
|
||||||
|
re-gate re-derives the strip from the current bytes + settings (ZERO
|
||||||
|
vision/LLM calls), heals missing copies, re-stamps a changed class,
|
||||||
|
clears a strip that no longer qualifies (kill switch included), stays
|
||||||
|
fail-soft, logs only when it ACTS, and costs a non-strip file zero PDF
|
||||||
|
parsing (spy-pinned).
|
||||||
|
|
||||||
The E2E surface (the idempotent second sync through the real UI +
|
The E2E surface (the idempotent second sync through the real UI +
|
||||||
mock LLM) is pinned in ``tests/e2e/test_sync_changed_only.py``
|
mock LLM) is pinned in ``tests/e2e/test_sync_changed_only.py``
|
||||||
(task 03).
|
(task 03).
|
||||||
@@ -43,6 +53,7 @@ import logging
|
|||||||
import os
|
import os
|
||||||
from datetime import UTC, datetime, timedelta
|
from datetime import UTC, datetime, timedelta
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
import pymupdf as fitz
|
import pymupdf as fitz
|
||||||
import pytest
|
import pytest
|
||||||
@@ -50,6 +61,7 @@ from sqlalchemy import select
|
|||||||
|
|
||||||
from app.config import Settings
|
from app.config import Settings
|
||||||
from app.models import Document
|
from app.models import Document
|
||||||
|
from app.rag import importer
|
||||||
from app.rag.importer import _file_hash, import_sources
|
from app.rag.importer import _file_hash, import_sources
|
||||||
from tests.fakes import FakeEmbedder
|
from tests.fakes import FakeEmbedder
|
||||||
from tests.unit.test_importer import (
|
from tests.unit.test_importer import (
|
||||||
@@ -60,6 +72,7 @@ from tests.unit.test_importer import (
|
|||||||
_VisionLLM,
|
_VisionLLM,
|
||||||
)
|
)
|
||||||
from tests.unit.test_pictorial_import import (
|
from tests.unit.test_pictorial_import import (
|
||||||
|
_build_dense_pdf,
|
||||||
_build_ikea_like_pdf,
|
_build_ikea_like_pdf,
|
||||||
_PictorialLLM,
|
_PictorialLLM,
|
||||||
_settings,
|
_settings,
|
||||||
@@ -454,6 +467,520 @@ def test_pre_phase_null_file_hash_image_runs_branch_and_stamps(
|
|||||||
_cleanup_source(db, root.name)
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- phase 163 (task 03): the deterministic PDF strip re-gate ----------
|
||||||
|
#
|
||||||
|
# The live defect (owner report, 2026-10-07): the `Letumdei` row carried
|
||||||
|
# `page_count = 36` over ZERO files on disk (the pre-phase-163 persist
|
||||||
|
# deleted the old copies BEFORE rendering, outside the DB transaction —
|
||||||
|
# a later per-file rollback restored the stale count over an empty
|
||||||
|
# directory; an external wipe of `pdf_page_dir` lands the same state).
|
||||||
|
# The phase-147 fast path believed "the copies persist" and never
|
||||||
|
# re-rendered, so the strip 404'd forever. `_regate_pdf_strip` now runs
|
||||||
|
# at the tail of `_unchanged_doc`: the changed-path gate mirrored from
|
||||||
|
# the current bytes + settings (ZERO vision/LLM calls), missing copies
|
||||||
|
# healed, a stale strip cleared.
|
||||||
|
|
||||||
|
#: The PNG signature (a healed copy is a real render, not a placeholder).
|
||||||
|
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
|
||||||
|
|
||||||
|
|
||||||
|
def _regate_settings(tmp_path: Path, **kwargs: Any) -> Settings:
|
||||||
|
"""Settings for the re-gate matrix: the ``pdf`` token admitted, the
|
||||||
|
phase-140 knobs at their CODE DEFAULTS (``pdf_pictorial`` ON), and
|
||||||
|
the strip's home ``pdf_page_dir`` pinned under *tmp_path* (never the
|
||||||
|
real home)."""
|
||||||
|
kwargs.setdefault("_env_file", None)
|
||||||
|
kwargs.setdefault("import_extensions", "md,txt,pdf")
|
||||||
|
kwargs.setdefault("images", False)
|
||||||
|
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||||
|
kwargs.setdefault("pdf_page_dir", str(tmp_path / "pages"))
|
||||||
|
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||||
|
|
||||||
|
|
||||||
|
def _pages_dir(tmp_path: Path) -> Path:
|
||||||
|
return tmp_path / "pages"
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_copies(doc: Document, page_dir: Path, count: int) -> list[Path]:
|
||||||
|
return [page_dir / f"{doc.id}-p{n}.png" for n in range(1, count + 1)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_heals_deleted_copies_of_a_pictorial_strip(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""THE live-loss reproduction (heal arm): a pictorial-class PDF is
|
||||||
|
indexed (``page_count`` set, copies written), its copies are deleted
|
||||||
|
out from under the row (the exact live state — mode NULL on a
|
||||||
|
pre-phase row), and an UNCHANGED re-sync restores every file with
|
||||||
|
real PNG bytes, stamps the mode ``additive``, keeps ``page_count``
|
||||||
|
(no drift), burns ZERO llm/vision calls, and logs the one greppable
|
||||||
|
INFO line. The NULL-mode stamp is SILENT (no ``mode`` segment — it
|
||||||
|
rides the line only when work followed)."""
|
||||||
|
root = tmp_path / "rgheal"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgheal", "ikea.pdf")
|
||||||
|
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
doc.page_strip_mode = None # the pre-phase row (the live shape)
|
||||||
|
db.commit()
|
||||||
|
for path in _strip_copies(doc, page_dir, 10):
|
||||||
|
path.unlink()
|
||||||
|
assert not list(page_dir.glob(f"{doc.id}-*.png"))
|
||||||
|
embeds1, chats1, vision1 = len(llm.calls), len(llm.chat_calls), llm.pictorial_calls
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
# ZERO llm work of any kind (the re-gate is deterministic, A5).
|
||||||
|
assert len(llm.calls) == embeds1, "the re-gate never re-embeds"
|
||||||
|
assert len(llm.chat_calls) == chats1, "the re-gate never chats"
|
||||||
|
assert llm.pictorial_calls == vision1, "the re-gate issues ZERO vision calls"
|
||||||
|
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgheal", "ikea.pdf")
|
||||||
|
assert doc2.page_count == 10, "no drift — the count is untouched"
|
||||||
|
assert doc2.page_strip_mode == "additive", "the mode is stamped"
|
||||||
|
for path in _strip_copies(doc2, page_dir, 10):
|
||||||
|
assert path.is_file(), f"copy {path.name} was not healed"
|
||||||
|
assert path.read_bytes()[:8] == PNG_MAGIC, f"{path.name} is not a PNG"
|
||||||
|
assert not list(page_dir.glob("*.tmp")), "no .tmp orphan from the heal"
|
||||||
|
messages = [r.getMessage() for r in caplog.records]
|
||||||
|
assert (
|
||||||
|
"import: PDF strip re-gate source=rgheal path=ikea.pdf — "
|
||||||
|
"healed 10 pages"
|
||||||
|
) in messages, messages
|
||||||
|
assert not any("mode " in m for m in messages if "re-gate" in m), (
|
||||||
|
"a NULL->stamp mode re-stamp is silent-beneficial " + str(messages)
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgheal")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_heals_a_fallback_strip_as_primary(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""Heal arm, the FALLBACK class (phase 133's unusable text layer):
|
||||||
|
the copies of a garbage-layer PDF's strip are restored on an
|
||||||
|
unchanged re-sync and the mode lands ``primary`` (the strip IS the
|
||||||
|
view) — and the phase-133 fallback vision does NOT re-fire (the
|
||||||
|
class is re-derived from the bytes, no vision call)."""
|
||||||
|
root = tmp_path / "rgfall"
|
||||||
|
root.mkdir()
|
||||||
|
_build_garbage_pdf(root / "garbage.pdf") # 2 pages
|
||||||
|
llm = _PageVisionLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgfall", "garbage.pdf")
|
||||||
|
assert (doc.page_count, doc.page_strip_mode) == (2, "primary")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
for path in _strip_copies(doc, page_dir, 2):
|
||||||
|
path.unlink()
|
||||||
|
vision1, chats1, embeds1 = llm.vision_calls, len(llm.chat_calls), len(llm.calls)
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
assert second.pdfs_screenshotted == 0, "the fallback gate did not re-run"
|
||||||
|
assert llm.vision_calls == vision1, "ZERO vision calls on the re-gate"
|
||||||
|
assert len(llm.chat_calls) == chats1 and len(llm.calls) == embeds1
|
||||||
|
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgfall", "garbage.pdf")
|
||||||
|
assert doc2.page_strip_mode == "primary", "the unusable layer is primary"
|
||||||
|
assert doc2.page_count == 2
|
||||||
|
for path in _strip_copies(doc2, page_dir, 2):
|
||||||
|
assert path.is_file() and path.read_bytes()[:8] == PNG_MAGIC
|
||||||
|
assert any(
|
||||||
|
"healed 2 pages" in m and "re-gate" in m for m in caplog.messages
|
||||||
|
), caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgfall")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_noop_when_the_strip_is_healthy(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""No-op arm (the IKEA-class doc whose copies all stand): an
|
||||||
|
unchanged re-sync writes NOTHING (every copy's mtime and bytes are
|
||||||
|
untouched), renders NOTHING, and logs NO re-gate line — the INFO is
|
||||||
|
never per-unchanged-doc — while the pre-existing ``import:
|
||||||
|
unchanged`` line still fires."""
|
||||||
|
root = tmp_path / "rgnoop"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgnoop", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
before = {
|
||||||
|
p: (p.stat().st_mtime_ns, p.read_bytes())
|
||||||
|
for p in _strip_copies(doc, page_dir, 10)
|
||||||
|
}
|
||||||
|
assert len(before) == 10
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgnoop", "ikea.pdf")
|
||||||
|
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
|
||||||
|
after = {
|
||||||
|
p: (p.stat().st_mtime_ns, p.read_bytes())
|
||||||
|
for p in _strip_copies(doc2, page_dir, 10)
|
||||||
|
}
|
||||||
|
assert after == before, "a healthy strip is not written at all"
|
||||||
|
messages = [r.getMessage() for r in caplog.records]
|
||||||
|
assert any(m.startswith("import: unchanged ") for m in messages), messages
|
||||||
|
assert not any("re-gate" in m for m in messages), (
|
||||||
|
f"the re-gate logged without acting: {messages}"
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgnoop")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_restamps_a_changed_mode(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""Mode re-stamp (the class flip): a row whose stamped mode no
|
||||||
|
longer matches the gate's class (here a pictorial-class row carrying
|
||||||
|
``primary`` — a legacy/mis-stamped row) is re-stamped to
|
||||||
|
``additive``, and because the mode was ALREADY SET the change is
|
||||||
|
reported (``mode primary->additive``) — the only thing the re-gate
|
||||||
|
did, so that is the whole INFO line (no copies were missing, none
|
||||||
|
were written)."""
|
||||||
|
root = tmp_path / "rgmode"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgmode", "ikea.pdf")
|
||||||
|
assert doc.page_strip_mode == "additive"
|
||||||
|
doc.page_strip_mode = "primary" # the mis-stamp the re-gate fixes
|
||||||
|
db.commit()
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
before = {
|
||||||
|
p: p.stat().st_mtime_ns for p in _strip_copies(doc, page_dir, 10)
|
||||||
|
}
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.unchanged) == (0, 1)
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgmode", "ikea.pdf")
|
||||||
|
assert doc2.page_strip_mode == "additive", "the class is re-stamped"
|
||||||
|
assert doc2.page_count == 10
|
||||||
|
assert {
|
||||||
|
p: p.stat().st_mtime_ns for p in _strip_copies(doc2, page_dir, 10)
|
||||||
|
} == before, "a mode fix writes no files"
|
||||||
|
assert (
|
||||||
|
"import: PDF strip re-gate source=rgmode path=ikea.pdf — "
|
||||||
|
"mode primary->additive"
|
||||||
|
) in [r.getMessage() for r in caplog.records], caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgmode")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_clears_a_strip_that_no_longer_qualifies(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""REPAIR arm: a row claims a strip (``page_count`` set, copies on
|
||||||
|
disk) on a PDF whose GATE now says ineligible (a clean, dense text
|
||||||
|
layer — zero sparse pages). The unchanged re-sync clears BOTH
|
||||||
|
columns and deletes the copies (the strip must not linger over a
|
||||||
|
document the viewer shows as text) + the ``cleared stale strip``
|
||||||
|
INFO line, with zero llm calls."""
|
||||||
|
root = tmp_path / "rgclear"
|
||||||
|
root.mkdir()
|
||||||
|
pdf = root / "clean.pdf"
|
||||||
|
_build_dense_pdf(pdf) # usable layer, zero candidates — never qualifies
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgclear", "clean.pdf")
|
||||||
|
assert (doc.page_count, doc.page_strip_mode) == (None, None)
|
||||||
|
# The stale state the re-gate must repair (a strip stamped by an
|
||||||
|
# older gate/settings, copies still on disk).
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
page_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
for path in _strip_copies(doc, page_dir, 2):
|
||||||
|
path.write_bytes(b"STALE-COPY")
|
||||||
|
doc.page_count = 2
|
||||||
|
doc.page_strip_mode = "additive"
|
||||||
|
db.commit()
|
||||||
|
embeds1, chats1 = len(llm.calls), len(llm.chat_calls)
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
assert len(llm.calls) == embeds1 and len(llm.chat_calls) == chats1
|
||||||
|
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgclear", "clean.pdf")
|
||||||
|
assert doc2.page_count is None, "the stale strip is cleared"
|
||||||
|
assert doc2.page_strip_mode is None, "the class is cleared with it"
|
||||||
|
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies are deleted"
|
||||||
|
assert (
|
||||||
|
"import: PDF strip re-gate source=rgclear path=clean.pdf — "
|
||||||
|
"cleared stale strip"
|
||||||
|
) in [r.getMessage() for r in caplog.records], caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgclear")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_kill_switch_clears_the_pictorial_strip(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""REPAIR arm via the kill switch (``pdf_pictorial=False``): the
|
||||||
|
changed-path semantics mirrored — with the pass off a sparse-page
|
||||||
|
PDF is NOT eligible, so its existing strip is cleared with its
|
||||||
|
copies on the next (unchanged) sync. The bytes never moved: the
|
||||||
|
setting alone decided."""
|
||||||
|
root = tmp_path / "rgkill"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgkill", "ikea.pdf")
|
||||||
|
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
assert len(list(page_dir.glob(f"{doc.id}-p*.png"))) == 10
|
||||||
|
vision1 = llm.pictorial_calls
|
||||||
|
|
||||||
|
llm.settings = _regate_settings(tmp_path, pdf_pictorial=False)
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
assert llm.pictorial_calls == vision1, "ZERO vision calls (the pass is off)"
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgkill", "ikea.pdf")
|
||||||
|
assert doc2.page_count is None and doc2.page_strip_mode is None
|
||||||
|
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies went too"
|
||||||
|
assert any(
|
||||||
|
"cleared stale strip" in m for m in caplog.messages
|
||||||
|
), caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgkill")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_render_exception_is_fail_soft(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
caplog: pytest.LogCaptureFixture,
|
||||||
|
) -> None:
|
||||||
|
"""Fail-soft: the heal's render RAISES — the sync completes
|
||||||
|
(``unchanged``, zero errors), the row keeps its prior state
|
||||||
|
(``page_count`` + mode untouched — the re-gate never clears a strip
|
||||||
|
on a failure), no copies are touched, and the WARNING is the only
|
||||||
|
signal."""
|
||||||
|
root = tmp_path / "rgboom"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgboom", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
for path in _strip_copies(doc, page_dir, 10):
|
||||||
|
path.unlink()
|
||||||
|
|
||||||
|
def boom(*args: object, **kwargs: object) -> list[bytes]:
|
||||||
|
raise RuntimeError("simulated render failure (test sentinel)")
|
||||||
|
|
||||||
|
monkeypatch.setattr(importer, "render_pdf_pages", boom)
|
||||||
|
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged, second.errors) == (
|
||||||
|
0, 0, 1, 0,
|
||||||
|
)
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgboom", "ikea.pdf")
|
||||||
|
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive"), (
|
||||||
|
"the row keeps its pre-re-gate state"
|
||||||
|
)
|
||||||
|
assert not list(page_dir.glob(f"{doc.id}-p*.png")), "no copies were touched"
|
||||||
|
assert any(
|
||||||
|
"PDF strip re-gate failed" in m and "test sentinel" in m
|
||||||
|
for m in caplog.messages
|
||||||
|
), caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgboom")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_empty_render_keeps_the_row(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
caplog: pytest.LogCaptureFixture,
|
||||||
|
) -> None:
|
||||||
|
"""Fail-soft on the no-pages render (unopenable right after the fast
|
||||||
|
path hashed the file — pathological): the row is kept EXACTLY as it
|
||||||
|
was (never cleared on a transient), nothing is written, and the
|
||||||
|
WARNING names the no-pages state."""
|
||||||
|
root = tmp_path / "rgempty"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgempty", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
for path in _strip_copies(doc, page_dir, 10):
|
||||||
|
path.unlink()
|
||||||
|
monkeypatch.setattr(importer, "render_pdf_pages", lambda *a, **k: [])
|
||||||
|
|
||||||
|
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.unchanged, second.errors) == (0, 1, 0)
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgempty", "ikea.pdf")
|
||||||
|
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
|
||||||
|
assert not page_dir.exists() or not list(page_dir.glob(f"{doc.id}-*"))
|
||||||
|
assert any(
|
||||||
|
"re-render produced no pages (state kept)" in m
|
||||||
|
for m in caplog.messages
|
||||||
|
), caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgempty")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_reports_a_page_count_drift_on_the_heal(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
caplog: pytest.LogCaptureFixture,
|
||||||
|
) -> None:
|
||||||
|
"""Drift (pathological on unchanged bytes, so it is LOUD): the heal
|
||||||
|
renders 6 pages for a row that claimed 10 — ``page_count`` follows
|
||||||
|
the fresh render, the ``p7..p10`` tail beyond it is deleted, and a
|
||||||
|
WARNING names both counts beside the ``healed`` INFO line."""
|
||||||
|
root = tmp_path / "rgdrift"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgdrift", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
for path in _strip_copies(doc, page_dir, 10):
|
||||||
|
path.unlink()
|
||||||
|
monkeypatch.setattr(
|
||||||
|
importer,
|
||||||
|
"render_pdf_pages",
|
||||||
|
lambda *a, **k: [PNG_MAGIC + f"{i}".encode() for i in range(6)],
|
||||||
|
)
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.unchanged) == (0, 1)
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgdrift", "ikea.pdf")
|
||||||
|
assert doc2.page_count == 6, "the count follows the fresh render"
|
||||||
|
for n in range(1, 7):
|
||||||
|
assert (page_dir / f"{doc2.id}-p{n}.png").is_file()
|
||||||
|
for n in range(7, 11):
|
||||||
|
assert not (page_dir / f"{doc2.id}-p{n}.png").exists(), f"p{n} leaked"
|
||||||
|
messages = [r.getMessage() for r in caplog.records]
|
||||||
|
assert any("page_count drift 10->6" in m for m in messages), messages
|
||||||
|
assert any("healed 10 pages" in m for m in messages), messages
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||||
|
_cleanup_source(db, "rgdrift")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_fires_on_the_content_digest_hit_path(
|
||||||
|
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""The SECOND unchanged path (the PDF branch's content-digest hit —
|
||||||
|
a pre-phase row with ``file_hash`` NULL): the branch re-runs its own
|
||||||
|
gate (the byte-change's honest cost) and lands unchanged, and the
|
||||||
|
shared helper's tail still heals the strip deleted out from under
|
||||||
|
the row — the re-gate is not fast-path-only."""
|
||||||
|
root = tmp_path / "rgbranch"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc = _doc(db, "rgbranch", "ikea.pdf")
|
||||||
|
doc.file_hash = None # the pre-phase row: the fast path cannot fire
|
||||||
|
db.commit()
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
for path in _strip_copies(doc, page_dir, 10):
|
||||||
|
path.unlink()
|
||||||
|
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
db.expire_all()
|
||||||
|
doc2 = _doc(db, "rgbranch", "ikea.pdf")
|
||||||
|
assert doc2.file_hash is not None, "the helper stamped the hash"
|
||||||
|
for path in _strip_copies(doc2, page_dir, 10):
|
||||||
|
assert path.is_file(), f"copy {path.name} was not healed"
|
||||||
|
assert any("healed 10 pages" in m for m in caplog.messages), caplog.messages
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "rgbranch")
|
||||||
|
|
||||||
|
|
||||||
|
def test_regate_guard_costs_a_non_strip_file_nothing(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""The GUARD (A5): a non-PDF unchanged file and a PDF whose
|
||||||
|
``page_count`` is NULL reach the tail of ``_unchanged_doc`` and
|
||||||
|
``extract_pdf_text`` is NEVER called for either — every non-strip
|
||||||
|
file's fast path keeps its zero-extra-read contract (one sync of
|
||||||
|
each class, then the spy covers the second sync only)."""
|
||||||
|
root = tmp_path / "rgguard"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "note.md").write_text("# Note\n\nplain text body\n")
|
||||||
|
_build_dense_pdf(root / "clean.pdf") # usable layer, no strip
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _regate_settings(tmp_path)
|
||||||
|
seen: list[str] = []
|
||||||
|
real_extract = importer.extract_pdf_text
|
||||||
|
|
||||||
|
def spy(path: Path) -> str:
|
||||||
|
seen.append(str(path))
|
||||||
|
return real_extract(path)
|
||||||
|
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
monkeypatch.setattr(importer, "extract_pdf_text", spy)
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 2)
|
||||||
|
assert seen == [], f"the guard let a PDF parse through: {seen}"
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||||
|
_cleanup_source(db, "rgguard")
|
||||||
|
|
||||||
|
|
||||||
# ---------- the preserved semantics on the fast path --------------------------
|
# ---------- the preserved semantics on the fast path --------------------------
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -293,6 +293,82 @@ def test_git_source_python_default_include_hidden_false() -> None:
|
|||||||
assert col.default.arg is False, "the default must resolve to False"
|
assert col.default.arg is False, "the default must resolve to False"
|
||||||
|
|
||||||
|
|
||||||
|
def test_documents_page_strip_mode_column_contract() -> None:
|
||||||
|
"""Phase 163 (task 01): ``documents.page_strip_mode`` — the CLASS of
|
||||||
|
the page strip a non-NULL ``page_count`` describes (``"primary"`` =
|
||||||
|
the phase-133 fallback class, the strip IS the view; ``"additive"``
|
||||||
|
= the phase-140 pictorial class, text + a Text/Pages toggle):
|
||||||
|
VARCHAR(16), NULLABLE (NULL = no strip — and the pre-phase shape for
|
||||||
|
a row whose strip stands), NO server default, and NO index (the
|
||||||
|
content endpoint reads it from the ROW — a display fact, never a
|
||||||
|
query surface, the ``file_hash`` / ``page_count`` convention).
|
||||||
|
``page_count`` (0026) stays exactly as phase 148 left it."""
|
||||||
|
from sqlalchemy import String as SAString
|
||||||
|
|
||||||
|
documents = Base.metadata.tables["documents"]
|
||||||
|
assert "page_strip_mode" in documents.c, (
|
||||||
|
"documents must have the page_strip_mode column (phase 163)"
|
||||||
|
)
|
||||||
|
col = documents.c["page_strip_mode"]
|
||||||
|
assert col.nullable is True, (
|
||||||
|
"page_strip_mode must be NULLABLE — NULL means no strip (and the"
|
||||||
|
" pre-phase row whose strip stands)"
|
||||||
|
)
|
||||||
|
assert isinstance(col.type, SAString), "page_strip_mode must be a String column"
|
||||||
|
assert col.type.length == 16, (
|
||||||
|
"page_strip_mode must be String(16) — the two class names, bounded"
|
||||||
|
)
|
||||||
|
assert col.server_default is None, (
|
||||||
|
"page_strip_mode must carry NO server default — NULL is the"
|
||||||
|
" meaningful pre-phase state"
|
||||||
|
)
|
||||||
|
assert not col.index, (
|
||||||
|
"page_strip_mode must be UNINDEXED — the content endpoint reads it"
|
||||||
|
" from the row, no query filters on it"
|
||||||
|
)
|
||||||
|
covering = [
|
||||||
|
idx
|
||||||
|
for idx in documents.indexes
|
||||||
|
if any(c.name == "page_strip_mode" for c in idx.columns)
|
||||||
|
]
|
||||||
|
assert not covering, "no table-level index may cover page_strip_mode"
|
||||||
|
|
||||||
|
# page_count is untouched: still the nullable, unindexed sibling.
|
||||||
|
page_count = documents.c["page_count"]
|
||||||
|
assert page_count.nullable is True, "page_count must stay NULLABLE (0026)"
|
||||||
|
assert not page_count.index, "page_count must stay UNINDEXED (0026)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_document_page_strip_mode_orm_defaults_to_none() -> None:
|
||||||
|
"""A freshly constructed ``Document`` (no DB) exposes
|
||||||
|
``page_strip_mode``: an omitted kwarg and an explicit ``None`` both
|
||||||
|
read NULL (no strip — the shape every non-qualifying document keeps,
|
||||||
|
and the shape a pre-phase-163 row keeps WHILE its ``page_count``
|
||||||
|
stands, which the wire serializes ABSENT), and each class name
|
||||||
|
round-trips verbatim as a plain string (the real DB round-trip is
|
||||||
|
pinned in ``test_migration_0027.py``)."""
|
||||||
|
from app.models import Document
|
||||||
|
|
||||||
|
base = dict(
|
||||||
|
source="s",
|
||||||
|
path="p.pdf",
|
||||||
|
full_path="/tmp/s/p.pdf",
|
||||||
|
title="p",
|
||||||
|
content="c",
|
||||||
|
content_hash="1" * 64,
|
||||||
|
page_count=36,
|
||||||
|
)
|
||||||
|
for mode in ("primary", "additive"):
|
||||||
|
assert Document(**base, page_strip_mode=mode).page_strip_mode == mode
|
||||||
|
|
||||||
|
assert Document(**base, page_strip_mode=None).page_strip_mode is None
|
||||||
|
omitted = Document(**base)
|
||||||
|
assert omitted.page_strip_mode is None, (
|
||||||
|
"an omitted page_strip_mode must read NULL while page_count stands"
|
||||||
|
" (the pre-phase strip the viewer keeps showing as 'primary')"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_doc_drafts_column_contract() -> None:
|
def test_doc_drafts_column_contract() -> None:
|
||||||
"""Phase 59: the editable triple (title/path/body) + status +
|
"""Phase 59: the editable triple (title/path/body) + status +
|
||||||
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
|
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
|
||||||
|
|||||||
@@ -21,14 +21,20 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
|
|||||||
layer, ≥1 sparse page) and the fallback class (unusable layer) get
|
layer, ≥1 sparse page) and the fallback class (unusable layer) get
|
||||||
``<doc-id>-p{N}.png`` for EVERY page (not just the described
|
``<doc-id>-p{N}.png`` for EVERY page (not just the described
|
||||||
candidates — the strip shows the whole manual) + ``page_count`` =
|
candidates — the strip shows the whole manual) + ``page_count`` =
|
||||||
the page count + real PNG bytes; a text-heavy PDF gets neither
|
the page count + real PNG bytes + the phase-163 CLASS stamp
|
||||||
(no copies, NULL count, ZERO render calls, byte-identical content);
|
(``page_strip_mode`` ``additive`` / ``primary``); a text-heavy PDF
|
||||||
|
gets neither (no copies, NULL count, NULL mode, ZERO render calls,
|
||||||
|
byte-identical content);
|
||||||
* the phase-147 fast path — an unchanged qualifying PDF's second sync
|
* the phase-147 fast path — an unchanged qualifying PDF's second sync
|
||||||
re-renders NOTHING (zero render calls — the copies persist,
|
re-renders NOTHING (zero render calls — the copies persist,
|
||||||
``page_count`` and the bytes untouched);
|
``page_count`` and the bytes untouched);
|
||||||
* stale replacement — a changed (shrunk) PDF's old copy set is
|
* WRITE-FIRST replacement (phase 163, task 02, LOCKED A4) — a changed
|
||||||
replaced: the new set covers the new pages, the old ``p{N}`` tail
|
(shrunk) PDF's new copies land atomically (tmp + ``os.replace``,
|
||||||
is GONE (no orphans), ``page_count`` follows the new count;
|
pinned) BEFORE any old copy is removed; the old ``p{N}`` tail is
|
||||||
|
GONE only after the replacement bytes are durable (no orphans),
|
||||||
|
``page_count`` follows the new count; a MID-WRITE failure leaves the
|
||||||
|
prior copy set intact (the rollback-orphan state — the row over
|
||||||
|
zero files — is structurally unrepresentable);
|
||||||
* qualifier loss — a qualifying PDF whose new version gained a dense
|
* qualifier loss — a qualifying PDF whose new version gained a dense
|
||||||
text layer loses its strip: ``page_count`` NULL + copies deleted;
|
text layer loses its strip: ``page_count`` NULL + copies deleted;
|
||||||
* unopenable — a render that yields no pages (``render_pdf_pages`` →
|
* unopenable — a render that yields no pages (``render_pdf_pages`` →
|
||||||
@@ -40,9 +46,10 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
|
|||||||
the FULL count (the gap is a display-level 404 — task 02 pins the
|
the FULL count (the gap is a display-level 404 — task 02 pins the
|
||||||
route's side);
|
route's side);
|
||||||
* fail-soft — the render raising (warning is the signal, ``page_count``
|
* fail-soft — the render raising (warning is the signal, ``page_count``
|
||||||
NULL, no copies, the doc indexed fully — chunks + summary) and the
|
+ ``page_strip_mode`` NULL, no copies, the doc indexed fully — chunks
|
||||||
WRITER raising (the copies this call wrote are removed — no partial
|
+ summary) and the WRITER raising (this call's ``.tmp`` files and the
|
||||||
orphans, ``page_count`` NULL, the doc indexed fully);
|
finals it CREATED are removed — no partial orphans, ``page_count`` +
|
||||||
|
``page_strip_mode`` NULL, the doc indexed fully);
|
||||||
* prune — a qualifying doc's file removed from the source: the row
|
* prune — a qualifying doc's file removed from the source: the row
|
||||||
goes and its page copies go with it (no orphans in the tmp dir).
|
goes and its page copies go with it (no orphans in the tmp dir).
|
||||||
|
|
||||||
@@ -54,6 +61,7 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import logging
|
import logging
|
||||||
|
import os
|
||||||
import uuid
|
import uuid
|
||||||
from collections.abc import Callable, Sequence
|
from collections.abc import Callable, Sequence
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@@ -161,6 +169,35 @@ def _render_spy(
|
|||||||
return calls
|
return calls
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Phase 163 (task 01) — the strip-class column beside page_count
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_pictorial_index_stamps_the_additive_class(db, tmp_path: Path) -> None:
|
||||||
|
"""Phase 163 (task 02, re-cut of the task-01 pin) — the STAMP: a
|
||||||
|
qualifying PDF indexed through the PICTORIAL class gets
|
||||||
|
``page_strip_mode == "additive"`` beside ``page_count`` (the text +
|
||||||
|
toggle view); the strip's copies themselves are unchanged by the
|
||||||
|
phase (task 01's shape, mode stamped exactly with the count).
|
||||||
|
"""
|
||||||
|
root = tmp_path / "pgm"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _settings(tmp_path)
|
||||||
|
try:
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||||
|
doc = _doc(db, "pgm", "ikea.pdf")
|
||||||
|
assert doc.page_count == 10, "the strip itself is unchanged by task 02"
|
||||||
|
assert doc.page_strip_mode == "additive", (
|
||||||
|
"the pictorial class stamps the additive mode (LOCKED A1/A3)"
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, "pgm")
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Eligibility × new index
|
# Eligibility × new index
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -219,6 +256,10 @@ def test_fallback_class_garbage_pdf_writes_every_page_copy(db, tmp_path: Path) -
|
|||||||
assert summary.pdfs_screenshotted == 1 # the fallback fired (the class)
|
assert summary.pdfs_screenshotted == 1 # the fallback fired (the class)
|
||||||
doc = _doc(db, "pgf", "garbage.pdf")
|
doc = _doc(db, "pgf", "garbage.pdf")
|
||||||
assert doc.page_count == 2
|
assert doc.page_count == 2
|
||||||
|
assert doc.page_strip_mode == "primary", (
|
||||||
|
"the fallback class stamps the primary mode — the strip IS the "
|
||||||
|
"view, byte-identical to phase 148 otherwise (LOCKED A1)"
|
||||||
|
)
|
||||||
page_dir = _pages_dir(tmp_path)
|
page_dir = _pages_dir(tmp_path)
|
||||||
for n in (1, 2):
|
for n in (1, 2):
|
||||||
path = page_dir / f"{doc.id}-p{n}.png"
|
path = page_dir / f"{doc.id}-p{n}.png"
|
||||||
@@ -253,6 +294,7 @@ def test_text_heavy_pdf_keeps_the_text_view(
|
|||||||
assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops
|
assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops
|
||||||
doc = _doc(db, "pgd", "clean.pdf")
|
doc = _doc(db, "pgd", "clean.pdf")
|
||||||
assert doc.page_count is None, "a text-heavy PDF keeps the text view"
|
assert doc.page_count is None, "a text-heavy PDF keeps the text view"
|
||||||
|
assert doc.page_strip_mode is None, "no strip — no class either"
|
||||||
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
||||||
assert calls == [], "zero render calls (no fallback, zero candidates)"
|
assert calls == [], "zero render calls (no fallback, zero candidates)"
|
||||||
assert doc.content == expected, "byte-identical to the extraction"
|
assert doc.content == expected, "byte-identical to the extraction"
|
||||||
@@ -315,9 +357,10 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index
|
"""A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index
|
||||||
replaces the copy set — the new set covers p1..p6, the old
|
replaces the copy set — the new set covers p1..p6, the old
|
||||||
version's ``p7..p10`` tail is GONE (the stale-replacement runs
|
version's ``p7..p10`` tail is GONE (the stale-tail delete runs
|
||||||
before the write), the doc keeps its id, and ``page_count``
|
LAST, after the write-first loop — phase 163, task 02), the doc
|
||||||
follows the new page count."""
|
keeps its id, ``page_count`` follows the new page count, and
|
||||||
|
``page_strip_mode`` is (re-)stamped."""
|
||||||
root = tmp_path / "pgc"
|
root = tmp_path / "pgc"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
pdf = root / "ikea.pdf"
|
pdf = root / "ikea.pdf"
|
||||||
@@ -336,6 +379,7 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
|
|||||||
doc2 = _doc(db, "pgc", "ikea.pdf")
|
doc2 = _doc(db, "pgc", "ikea.pdf")
|
||||||
assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)"
|
assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)"
|
||||||
assert doc2.page_count == 6, "page_count follows the new page count"
|
assert doc2.page_count == 6, "page_count follows the new page count"
|
||||||
|
assert doc2.page_strip_mode == "additive", "the class is (re-)stamped"
|
||||||
page_dir = _pages_dir(tmp_path)
|
page_dir = _pages_dir(tmp_path)
|
||||||
for n in range(1, 7):
|
for n in range(1, 7):
|
||||||
assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing"
|
assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing"
|
||||||
@@ -379,6 +423,7 @@ def test_qualifier_loss_clears_page_count_and_copies(
|
|||||||
doc2 = _doc(db, "pgl", "ikea.pdf")
|
doc2 = _doc(db, "pgl", "ikea.pdf")
|
||||||
assert doc2.id == doc1.id
|
assert doc2.id == doc1.id
|
||||||
assert doc2.page_count is None, "a doc that stopped qualifying loses its strip"
|
assert doc2.page_count is None, "a doc that stopped qualifying loses its strip"
|
||||||
|
assert doc2.page_strip_mode is None, "the class is cleared with the count"
|
||||||
assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), (
|
assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), (
|
||||||
"the copies must be deleted on qualifier loss"
|
"the copies must be deleted on qualifier loss"
|
||||||
)
|
)
|
||||||
@@ -411,6 +456,7 @@ def test_unopenable_pdf_page_render_is_fail_soft(
|
|||||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||||
doc = _doc(db, "pgn", "garbage.pdf")
|
doc = _doc(db, "pgn", "garbage.pdf")
|
||||||
assert doc.page_count is None, "no pages rendered — no strip"
|
assert doc.page_count is None, "no pages rendered — no strip"
|
||||||
|
assert doc.page_strip_mode is None, "the class is cleared with the count"
|
||||||
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
||||||
assert doc.content == "", "the fallback's all-failed render lands empty content"
|
assert doc.content == "", "the fallback's all-failed render lands empty content"
|
||||||
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
||||||
@@ -493,10 +539,11 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
|||||||
monkeypatch: pytest.MonkeyPatch,
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
caplog: pytest.LogCaptureFixture,
|
caplog: pytest.LogCaptureFixture,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""The PNG WRITER raises (a full disk) after the FIRST copy
|
"""The PNG WRITER raises (a full disk) after the FIRST atomic copy
|
||||||
landed: the copies this call wrote are removed (no partial
|
landed: this call's ``.tmp`` files and the final-name files it
|
||||||
orphans), ``page_count`` NULL, the warning is the signal, and the
|
CREATED are removed (no partial orphans — phase 163, task 02),
|
||||||
document indexes exactly as pre-phase."""
|
``page_count`` AND ``page_strip_mode`` NULL, the warning is the
|
||||||
|
signal, and the document indexes exactly as pre-phase."""
|
||||||
root = tmp_path / "pgw"
|
root = tmp_path / "pgw"
|
||||||
root.mkdir()
|
root.mkdir()
|
||||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
@@ -507,22 +554,29 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
|||||||
|
|
||||||
def boom(self: Path, data: bytes) -> int:
|
def boom(self: Path, data: bytes) -> int:
|
||||||
written_paths.append(self)
|
written_paths.append(self)
|
||||||
|
n = original_write(self, data) # the tmp file really lands first
|
||||||
if len(written_paths) > 1:
|
if len(written_paths) > 1:
|
||||||
raise OSError("simulated disk full (test sentinel)")
|
raise OSError("simulated disk full (test sentinel)")
|
||||||
return original_write(self, data)
|
return n
|
||||||
|
|
||||||
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
|
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
|
||||||
try:
|
try:
|
||||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||||
assert len(written_paths) == 2, "p1 landed, p2's write failed"
|
assert len(written_paths) == 2, "p1's tmp landed + replaced, p2's write failed"
|
||||||
|
assert all(p.name.endswith(".png.tmp") for p in written_paths), (
|
||||||
|
"the writer only ever touches .tmp paths (atomic replace)"
|
||||||
|
)
|
||||||
doc = _doc(db, "pgw", "ikea.pdf")
|
doc = _doc(db, "pgw", "ikea.pdf")
|
||||||
assert doc.page_count is None
|
assert doc.page_count is None
|
||||||
|
assert doc.page_strip_mode is None, "the class is cleared with the count"
|
||||||
assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), (
|
assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), (
|
||||||
"the partial write was cleaned up — no orphans"
|
"the partial write was cleaned up — no orphans"
|
||||||
)
|
)
|
||||||
assert not written_paths[0].exists(), "the landed p1 copy was removed"
|
assert not list(_pages_dir(tmp_path).glob("*.tmp")), (
|
||||||
|
"the failed call's .tmp file was cleaned up too"
|
||||||
|
)
|
||||||
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
||||||
assert any("PDF page render failed" in m for m in caplog.messages), (
|
assert any("PDF page render failed" in m for m in caplog.messages), (
|
||||||
caplog.messages
|
caplog.messages
|
||||||
@@ -531,6 +585,246 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
|||||||
_cleanup_source(db, "pgw")
|
_cleanup_source(db, "pgw")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Phase 163 (task 02) — WRITE-FIRST, atomic per file (LOCKED A4)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_first_new_bytes_are_durable_before_any_stale_delete(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""THE order pin (LOCKED A4 — the 2026-10-07 live loss): re-rendering
|
||||||
|
a CHANGED (shrunk, 10 → 6) PDF writes every replacement page FIRST
|
||||||
|
(tmp + ``os.replace``) and only then deletes the stale tail — the
|
||||||
|
event log shows ALL replaces of the fresh bytes before the FIRST
|
||||||
|
unlink of an old-extent file, the overwritten pages keep the NEW
|
||||||
|
bytes, the shrunk tail (p7..p10) is gone, and nothing else is
|
||||||
|
deleted."""
|
||||||
|
root = tmp_path / "pgord"
|
||||||
|
root.mkdir()
|
||||||
|
pdf = root / "ikea.pdf"
|
||||||
|
_build_ikea_like_pdf(pdf)
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _settings(tmp_path)
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc1 = _doc(db, "pgord", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
sentinel = b"OLD-COPY-SENTINEL"
|
||||||
|
for n in range(1, 11): # make the old copies unmistakably distinguishable
|
||||||
|
(page_dir / f"{doc1.id}-p{n}.png").write_bytes(sentinel)
|
||||||
|
|
||||||
|
events: list[tuple[str, str]] = []
|
||||||
|
real_replace = os.replace
|
||||||
|
|
||||||
|
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
|
||||||
|
events.append(("replace", Path(os.fspath(dst)).name))
|
||||||
|
return real_replace(src, dst, *args, **kwargs)
|
||||||
|
|
||||||
|
real_unlink = Path.unlink
|
||||||
|
|
||||||
|
def unlink_spy(self: Path, **kwargs: Any) -> None:
|
||||||
|
if self.parent == page_dir and self.name.startswith(str(doc1.id)):
|
||||||
|
events.append(("unlink", self.name))
|
||||||
|
return real_unlink(self, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(importer.os, "replace", replace_spy)
|
||||||
|
monkeypatch.setattr(Path, "unlink", unlink_spy) # pyright: ignore[reportArgumentType]
|
||||||
|
try:
|
||||||
|
_build_ikea_shrunk(pdf) # same path, 6 pages — still qualifying
|
||||||
|
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||||
|
_cleanup_source(db, "pgord")
|
||||||
|
replaces = [name for kind, name in events if kind == "replace"]
|
||||||
|
unlinks = [name for kind, name in events if kind == "unlink"]
|
||||||
|
assert len(replaces) == 6, "the 6 fresh pages are written via os.replace"
|
||||||
|
assert set(unlinks) == {f"{doc1.id}-p{n}.png" for n in range(7, 11)}, (
|
||||||
|
"the stale-tail delete touches ONLY the old extent the render did "
|
||||||
|
"not rewrite — a file about to be overwritten is never deleted"
|
||||||
|
)
|
||||||
|
first_unlink = next(i for i, e in enumerate(events) if e[0] == "unlink")
|
||||||
|
last_replace = max(i for i, e in enumerate(events) if e[0] == "replace")
|
||||||
|
assert last_replace < first_unlink, (
|
||||||
|
f"write-first violated — event order: {events}"
|
||||||
|
)
|
||||||
|
for n in range(1, 7):
|
||||||
|
blob = (page_dir / f"{doc1.id}-p{n}.png").read_bytes()
|
||||||
|
assert blob != sentinel and blob[:8] == PNG_MAGIC, (
|
||||||
|
f"p{n} keeps the NEW rendered bytes"
|
||||||
|
)
|
||||||
|
for n in range(7, 11):
|
||||||
|
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"stale p{n} leaked"
|
||||||
|
assert not list(page_dir.glob("*.tmp")), "no .tmp orphans"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mid_write_failure_keeps_the_prior_copies_intact(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
caplog: pytest.LogCaptureFixture,
|
||||||
|
) -> None:
|
||||||
|
"""THE failure pin (LOCKED A4): the writer raises on the SECOND
|
||||||
|
page's tmp write of a changed re-render — the prior copy set stays
|
||||||
|
on disk INTACT (the rollback-orphan state — the row over zero
|
||||||
|
files — is structurally unrepresentable): every one of the 10 prior
|
||||||
|
files still exists (p1 with the new bytes already atomically
|
||||||
|
replaced before the failure, p2..p10 byte-for-byte the prior
|
||||||
|
copies), zero ``.tmp`` orphans from this call, and the caller's
|
||||||
|
fail-soft fires (``page_count``/``page_strip_mode`` NULL + the
|
||||||
|
warning, the doc indexed fully)."""
|
||||||
|
root = tmp_path / "pgfail"
|
||||||
|
root.mkdir()
|
||||||
|
pdf = root / "ikea.pdf"
|
||||||
|
_build_ikea_like_pdf(pdf)
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _settings(tmp_path)
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc1 = _doc(db, "pgfail", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
prior = {
|
||||||
|
n: (page_dir / f"{doc1.id}-p{n}.png").read_bytes() for n in range(1, 11)
|
||||||
|
}
|
||||||
|
# Tag the re-render's bytes so the new p1 is recognizable, then
|
||||||
|
# blow up the SECOND tmp write (page 2).
|
||||||
|
_render_spy(
|
||||||
|
monkeypatch,
|
||||||
|
mutate_full=lambda pages: [
|
||||||
|
PNG_MAGIC + f"NEW-{i}".encode() for i in range(len(pages))
|
||||||
|
],
|
||||||
|
)
|
||||||
|
original_write = Path.write_bytes
|
||||||
|
tmp_writes: list[Path] = []
|
||||||
|
|
||||||
|
def boom(self: Path, data: bytes) -> int:
|
||||||
|
if self.suffix == ".tmp":
|
||||||
|
tmp_writes.append(self)
|
||||||
|
if len(tmp_writes) > 1:
|
||||||
|
raise OSError("simulated disk full (test sentinel)")
|
||||||
|
return original_write(self, data)
|
||||||
|
|
||||||
|
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
|
||||||
|
_build_ikea_shrunk(pdf)
|
||||||
|
try:
|
||||||
|
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||||
|
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||||
|
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||||
|
assert len(tmp_writes) == 2, "p1's tmp wrote, p2's tmp write raised"
|
||||||
|
doc2 = _doc(db, "pgfail", "ikea.pdf")
|
||||||
|
assert doc2.page_count is None, "the caller's fail-soft nulled the count"
|
||||||
|
assert doc2.page_strip_mode is None, "the caller's fail-soft nulled the class"
|
||||||
|
assert any("PDF page render failed" in m for m in caplog.messages), caplog.messages
|
||||||
|
# The prior set is INTACT: exactly the 10 names, none lost, none
|
||||||
|
# orphaned on top.
|
||||||
|
assert {p.name for p in page_dir.iterdir()} == {
|
||||||
|
f"{doc1.id}-p{n}.png" for n in range(1, 11)
|
||||||
|
}, "files on disk exactly the prior set"
|
||||||
|
assert (
|
||||||
|
page_dir / f"{doc1.id}-p1.png"
|
||||||
|
).read_bytes() == PNG_MAGIC + b"NEW-0", (
|
||||||
|
"p1's replacement bytes were durable before the failure (atomic "
|
||||||
|
"replace) and were NOT removed by the cleanup"
|
||||||
|
)
|
||||||
|
for n in range(2, 11):
|
||||||
|
assert (
|
||||||
|
page_dir / f"{doc1.id}-p{n}.png"
|
||||||
|
).read_bytes() == prior[n], f"prior copy p{n} was disturbed"
|
||||||
|
assert not list(page_dir.glob("*.tmp")), "zero .tmp orphans from this call"
|
||||||
|
assert doc2.chunks, "the document is indexed fully (fail-soft)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_page_copies_are_written_atomically_tmp_then_replace(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""THE atomic pin (LOCKED A4): the writer NEVER writes a final
|
||||||
|
``<doc-id>-p{N}.png`` path directly — every page's bytes land on
|
||||||
|
``<name>.png.tmp`` and ``os.replace`` moves them onto the final
|
||||||
|
name in order (a reader opening the final path never sees truncated
|
||||||
|
bytes), and no ``.tmp`` survives the successful call."""
|
||||||
|
root = tmp_path / "pgatom"
|
||||||
|
root.mkdir()
|
||||||
|
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _settings(tmp_path)
|
||||||
|
written: list[Path] = []
|
||||||
|
original_write = Path.write_bytes
|
||||||
|
|
||||||
|
def write_spy(self: Path, data: bytes) -> int:
|
||||||
|
written.append(self)
|
||||||
|
return original_write(self, data)
|
||||||
|
|
||||||
|
replaced: list[tuple[str, str]] = []
|
||||||
|
real_replace = os.replace
|
||||||
|
|
||||||
|
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
|
||||||
|
replaced.append((os.fspath(src), os.fspath(dst)))
|
||||||
|
return real_replace(src, dst, *args, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(Path, "write_bytes", write_spy) # pyright: ignore[reportArgumentType]
|
||||||
|
monkeypatch.setattr(importer.os, "replace", replace_spy)
|
||||||
|
try:
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||||
|
doc = _doc(db, "pgatom", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
assert len(written) == 10 and len(replaced) == 10
|
||||||
|
assert all(p.name.endswith(".png.tmp") for p in written), (
|
||||||
|
"no final path is ever written directly"
|
||||||
|
)
|
||||||
|
for n, (src, dst) in enumerate(replaced, start=1):
|
||||||
|
assert dst == str(page_dir / f"{doc.id}-p{n}.png")
|
||||||
|
assert src == f"{dst}.tmp", "the tmp name is <final>.tmp"
|
||||||
|
assert not list(page_dir.glob("*.tmp")), "no .tmp survives the success path"
|
||||||
|
|
||||||
|
|
||||||
|
def test_stale_tail_delete_failure_is_best_effort(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
caplog: pytest.LogCaptureFixture,
|
||||||
|
) -> None:
|
||||||
|
"""An unreadable STALE-TAIL copy (the write-first tail pass, the
|
||||||
|
``_delete_page_copies`` convention): the failure is logged, never
|
||||||
|
raised — the sync completes, the row is stamped with the fresh
|
||||||
|
count, and the reachable tail files still go."""
|
||||||
|
root = tmp_path / "pgtail"
|
||||||
|
root.mkdir()
|
||||||
|
pdf = root / "ikea.pdf"
|
||||||
|
_build_ikea_like_pdf(pdf)
|
||||||
|
llm = _PictorialLLM()
|
||||||
|
llm.settings = _settings(tmp_path)
|
||||||
|
asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
doc1 = _doc(db, "pgtail", "ikea.pdf")
|
||||||
|
page_dir = _pages_dir(tmp_path)
|
||||||
|
p10 = page_dir / f"{doc1.id}-p10.png"
|
||||||
|
real_unlink = Path.unlink
|
||||||
|
|
||||||
|
def boom(self: Path, **kwargs: Any) -> None:
|
||||||
|
if self == p10:
|
||||||
|
raise OSError("simulated unreadable file (test sentinel)")
|
||||||
|
return real_unlink(self, **kwargs)
|
||||||
|
|
||||||
|
monkeypatch.setattr(Path, "unlink", boom) # pyright: ignore[reportArgumentType]
|
||||||
|
_build_ikea_shrunk(pdf)
|
||||||
|
try:
|
||||||
|
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||||
|
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||||
|
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||||
|
doc2 = _doc(db, "pgtail", "ikea.pdf")
|
||||||
|
assert doc2.page_count == 6 and doc2.page_strip_mode == "additive"
|
||||||
|
assert p10.exists(), "the unreadable tail file survived (never raised)"
|
||||||
|
for n in range(7, 10):
|
||||||
|
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"p{n} still leaked"
|
||||||
|
assert any(
|
||||||
|
"could not delete PDF page copy" in m for m in caplog.messages
|
||||||
|
), caplog.messages
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# The shared cleanup helper (the _delete_image_copy convention)
|
# The shared cleanup helper (the _delete_image_copy convention)
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -117,7 +117,10 @@ def test_gate_runs_after_the_phase36_base_construction() -> None:
|
|||||||
label = js.find('title.textContent = "Summary"')
|
label = js.find('title.textContent = "Summary"')
|
||||||
text_node = js.find('body.className = "doc-summary-text"')
|
text_node = js.find('body.className = "doc-summary-text"')
|
||||||
append = js.find("section.append(title, body)")
|
append = js.find("section.append(title, body)")
|
||||||
mount = js.find("contentEl.appendChild(section)")
|
# Phase 163 (task 04): the mount rides the `slot` variable —
|
||||||
|
# slot === contentEl for every non-additive doc (the byte-identity
|
||||||
|
# property), views only for the additive strip class.
|
||||||
|
mount = js.find("slot.appendChild(section)")
|
||||||
gate = js.find("void docAdminReady().then(", mount)
|
gate = js.find("void docAdminReady().then(", mount)
|
||||||
wiring = js.find("if (admin) wireSummaryEdit(section, doc);")
|
wiring = js.find("if (admin) wireSummaryEdit(section, doc);")
|
||||||
assert 0 < base < label < text_node < append < mount < gate < wiring, (
|
assert 0 < base < label < text_node < append < mount < gate < wiring, (
|
||||||
|
|||||||
@@ -0,0 +1,431 @@
|
|||||||
|
"""Unit: the additive PDF strip view (phase 163, task 04 — D4, LOCKED A1/A6).
|
||||||
|
|
||||||
|
Frontend source-contract pins + node behavioral runs (the
|
||||||
|
``test_document_viewer.py`` house pattern) over ``document.js``'s
|
||||||
|
``renderDocument`` — the ONE shared core (Sources modal AND
|
||||||
|
/document.html):
|
||||||
|
|
||||||
|
* the MODE branch — ``page_urls`` + ``page_strip_mode === "additive"``
|
||||||
|
falls through to the text-doc rendering (Summary panel + content,
|
||||||
|
the pre-148 branch reused) wrapped in a ``.doc-views`` container
|
||||||
|
with the ``.doc-view-toggle`` group above it (Text pressed by
|
||||||
|
default, LOCKED A6); ``page_urls`` with the mode ABSENT or
|
||||||
|
``"primary"`` keeps the phase-148 strip-only branch VERBATIM (append
|
||||||
|
the strip to ``contentEl`` + return — no toggle, no wrapper, no
|
||||||
|
Summary panel); falsy ``page_urls`` is untouched;
|
||||||
|
* the toggle semantics — ``role="group"`` + ``aria-label="Document
|
||||||
|
view"``, two ``<button type="button">`` labeled ``Text`` / ``Pages``
|
||||||
|
(fixed strings, no doc-derived text), ``aria-pressed`` true/false
|
||||||
|
initial state, and a click mutating ONLY the two ``aria-pressed``
|
||||||
|
states + the wrapper's ``data-view`` attribute (CSS-driven show/
|
||||||
|
hide — the strip node stays mounted, no re-render, no refetch);
|
||||||
|
* the never-build-HTML rule — createElement + properties/textContent
|
||||||
|
only in the new code path (the module's ONE innerHTML stays the
|
||||||
|
escape-first markdown render); the strip imgs keep
|
||||||
|
``loading = "lazy"`` (the hidden strip never fetches);
|
||||||
|
* the ``.doc-view-toggle`` CSS — the house ghost-pill idiom, the
|
||||||
|
pressed state as text + style (never color alone, B5), ≥44px
|
||||||
|
targets, and the ``data-view`` show/hide contract.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||||
|
DOCUMENT_JS = FRONTEND / "assets" / "document.js"
|
||||||
|
STYLES_CSS = FRONTEND / "assets" / "styles.css"
|
||||||
|
|
||||||
|
HAVE_NODE = shutil.which("node") is not None
|
||||||
|
|
||||||
|
|
||||||
|
def _read(path: Path) -> str:
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def _fn(js: str, name: str) -> str:
|
||||||
|
"""The source of a top-level function via balanced-brace counting
|
||||||
|
(the ``test_document_viewer.py`` pattern)."""
|
||||||
|
start = js.find(f"function {name}(")
|
||||||
|
assert start != -1, f"{name}() must exist in document.js"
|
||||||
|
i = js.find("(", start)
|
||||||
|
depth = 0
|
||||||
|
close = i
|
||||||
|
while i < len(js):
|
||||||
|
if js[i] == "(":
|
||||||
|
depth += 1
|
||||||
|
elif js[i] == ")":
|
||||||
|
depth -= 1
|
||||||
|
if depth == 0:
|
||||||
|
close = i
|
||||||
|
break
|
||||||
|
i += 1
|
||||||
|
brace = js.find("{", close)
|
||||||
|
depth = 0
|
||||||
|
for j in range(brace, len(js)):
|
||||||
|
if js[j] == "{":
|
||||||
|
depth += 1
|
||||||
|
elif js[j] == "}":
|
||||||
|
depth -= 1
|
||||||
|
if depth == 0:
|
||||||
|
return js[start : j + 1]
|
||||||
|
raise AssertionError(f"unbalanced braces in {name}()")
|
||||||
|
|
||||||
|
|
||||||
|
def _code(js: str) -> str:
|
||||||
|
"""The source with comments stripped (the never-build-HTML pins)."""
|
||||||
|
return re.sub(r"//.*?$|/\*.*?\*/", "", js, flags=re.S | re.M)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_node(script: str) -> str:
|
||||||
|
proc = subprocess.run(
|
||||||
|
["node", "-e", script], capture_output=True, text=True, timeout=60
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, f"node failed: {proc.stderr}"
|
||||||
|
return proc.stdout
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Source pins — the renderDocument mode branch
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_render_document_branches_on_strip_mode() -> None:
|
||||||
|
"""The mode branch (LOCKED A2 — the tolerant read): the additive
|
||||||
|
flag keys on the EXACT ``doc.page_strip_mode === "additive"``
|
||||||
|
string (absent / "primary" / anything else ⇒ primary), and the
|
||||||
|
primary/absent strip branch stays the phase-148 shape — append
|
||||||
|
``docPagesStrip`` to ``contentEl`` and RETURN — evaluated BEFORE
|
||||||
|
any wrapper/toggle construction."""
|
||||||
|
render = _fn(_read(DOCUMENT_JS), "renderDocument")
|
||||||
|
assert 'doc.page_strip_mode === "additive"' in render, (
|
||||||
|
"only the exact additive string switches on the toggle"
|
||||||
|
)
|
||||||
|
gate_i = render.find('const additive = stripEligible && doc.page_strip_mode === "additive";')
|
||||||
|
primary_i = render.find("if (stripEligible && !additive) {")
|
||||||
|
strip_i = render.find("contentEl.appendChild(docPagesStrip(doc, pageUrls));")
|
||||||
|
return_i = render.find("return;", strip_i)
|
||||||
|
wrapper_i = render.find('views.className = "doc-views";')
|
||||||
|
assert -1 < gate_i < primary_i < strip_i < return_i < wrapper_i, (
|
||||||
|
"the primary/absent branch (strip into contentEl + return) precedes "
|
||||||
|
"the additive wrapper — its DOM stays the phase-148 shape verbatim"
|
||||||
|
)
|
||||||
|
# The wrapper + toggle: ONE .doc-views container, the toggle BEFORE
|
||||||
|
# it, both into contentEl; the text view lands in the wrapper.
|
||||||
|
append_i = render.find("contentEl.append(docViewToggle(views), views);")
|
||||||
|
assert wrapper_i < append_i, "the wrapper is built, then appended after its toggle"
|
||||||
|
assert 'views.setAttribute("data-view", "text");' in render, (
|
||||||
|
"LOCKED A6 — Text is the default view"
|
||||||
|
)
|
||||||
|
# The strip mounts ONCE inside the wrapper, after the text view —
|
||||||
|
# never rebuilt on a flip (no docPagesStrip call outside the two
|
||||||
|
# branches).
|
||||||
|
assert render.count("docPagesStrip(doc, pageUrls)") == 2
|
||||||
|
mount_i = render.rfind("views.appendChild(docPagesStrip(doc, pageUrls));")
|
||||||
|
assert mount_i > render.find('slot.appendChild(pre);') or mount_i > render.rfind(
|
||||||
|
"slot.appendChild(wrap);"
|
||||||
|
), "the strip mounts AFTER the text view inside the wrapper"
|
||||||
|
# The non-additive path is untouched: slot === contentEl (the
|
||||||
|
# pre-148 appends keep their exact shape through the slot var).
|
||||||
|
assert "let slot = contentEl;" in render
|
||||||
|
|
||||||
|
|
||||||
|
def test_primary_absent_branch_builds_no_toggle() -> None:
|
||||||
|
"""The primary/absent branch is toggle-free by construction: the
|
||||||
|
strip-branch block (from the ``pageUrls`` read to its ``return``)
|
||||||
|
contains NO ``doc-view-toggle`` / ``doc-views`` construction — the
|
||||||
|
strip-only DOM stays byte-identical to pre-phase-163."""
|
||||||
|
render = _fn(_read(DOCUMENT_JS), "renderDocument")
|
||||||
|
primary_start = render.find("if (stripEligible && !additive)")
|
||||||
|
branch = render[render.find("const pageUrls =") : render.find("return;", primary_start)]
|
||||||
|
assert "doc-view-toggle" not in branch
|
||||||
|
assert "doc-views" not in branch
|
||||||
|
assert branch.count("appendChild") == 1, (
|
||||||
|
"the primary branch is exactly the phase-148 append + return"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_toggle_never_builds_html_and_labels_are_fixed() -> None:
|
||||||
|
"""The never-build-HTML rule: the toggle builders use
|
||||||
|
createElement + properties/textContent only — no innerHTML /
|
||||||
|
outerHTML — and the labels are the FIXED strings ``Text`` /
|
||||||
|
``Pages`` (no doc-derived text inside the toggle). The module's
|
||||||
|
ONE innerHTML (the escape-first markdown render) survives."""
|
||||||
|
js = _read(DOCUMENT_JS)
|
||||||
|
toggle = _fn(js, "docViewToggle")
|
||||||
|
btn = _fn(js, "viewToggleButton")
|
||||||
|
for body in (toggle, btn):
|
||||||
|
code = _code(body)
|
||||||
|
assert "innerHTML" not in code and "outerHTML" not in code
|
||||||
|
assert 'document.createElement("div")' in toggle
|
||||||
|
assert 'group.setAttribute("role", "group")' in toggle
|
||||||
|
assert 'group.setAttribute("aria-label", "Document view")' in toggle
|
||||||
|
assert 'document.createElement("button")' in btn
|
||||||
|
assert 'btn.type = "button"' in btn
|
||||||
|
assert 'btn.className = "doc-view-btn"' in btn
|
||||||
|
assert 'viewToggleButton("Text")' in toggle
|
||||||
|
assert 'viewToggleButton("Pages")' in toggle
|
||||||
|
# The module-wide XSS contract: exactly one innerHTML, the
|
||||||
|
# renderMarkdown line.
|
||||||
|
whole = _code(js)
|
||||||
|
assert whole.count("innerHTML") == 1
|
||||||
|
assert "outerHTML" not in whole
|
||||||
|
|
||||||
|
|
||||||
|
def test_strip_imgs_stay_lazy() -> None:
|
||||||
|
"""The phase-148 lazy-img pin re-runs green: the strip builder is
|
||||||
|
UNMODIFIED — every page ``<img>`` keeps ``loading = "lazy"`` (the
|
||||||
|
additive default view never fetches the strip)."""
|
||||||
|
strip = _fn(_read(DOCUMENT_JS), "docPagesStrip")
|
||||||
|
assert 'img.loading = "lazy";' in strip
|
||||||
|
|
||||||
|
|
||||||
|
def test_view_toggle_css() -> None:
|
||||||
|
"""styles.css: the ``.doc-view-toggle`` group (the chat-column
|
||||||
|
reading width, the pill pair adjacent) + the ``.doc-view-btn``
|
||||||
|
ghost pill (≥44px target, ink-soft text — AA) + the PRESSED state
|
||||||
|
(``[aria-pressed="true"]`` — brand fill + font-weight 700: text +
|
||||||
|
style, never color alone, B5) + the CSS-driven view swap
|
||||||
|
(``.doc-views .doc-pages`` hidden by default; ``[data-view=
|
||||||
|
"pages"]`` hides the Summary panel / .doc-md / .doc-raw /
|
||||||
|
.doc-image and shows the strip as the grid). Provenance comment
|
||||||
|
above the block (house style)."""
|
||||||
|
css = _read(STYLES_CSS)
|
||||||
|
toggle_i = css.find(".doc-view-toggle {")
|
||||||
|
assert toggle_i != -1, "styles.css must style .doc-view-toggle"
|
||||||
|
comment_start = css.rfind("/*", 0, toggle_i)
|
||||||
|
assert -1 < comment_start < toggle_i and "phase 163" in css[comment_start:toggle_i]
|
||||||
|
btn_i = css.find(".doc-view-btn {")
|
||||||
|
pressed_i = css.find('.doc-view-btn[aria-pressed="true"]')
|
||||||
|
assert toggle_i < btn_i < pressed_i
|
||||||
|
btn_rule = css[btn_i : css.find("}", btn_i)]
|
||||||
|
for decl in ("min-height: 44px", "color: var(--ink-soft)", "border: 1px solid var(--line)"):
|
||||||
|
assert decl in btn_rule
|
||||||
|
pressed_rule = css[pressed_i : css.find("}", pressed_i)]
|
||||||
|
assert "background: var(--brand)" in pressed_rule
|
||||||
|
assert "font-weight: 700" in pressed_rule, (
|
||||||
|
"the pressed state is text weight + fill — never color alone (B5)"
|
||||||
|
)
|
||||||
|
# The CSS-driven swap: hidden strip by default; pages hides the
|
||||||
|
# text view and shows the strip.
|
||||||
|
assert '.doc-views .doc-pages { display: none; }' in css
|
||||||
|
for sel in (".doc-summary", ".doc-md", ".doc-raw", ".doc-image"):
|
||||||
|
assert f'.doc-views[data-view="pages"] {sel}' in css
|
||||||
|
assert '.doc-views[data-view="pages"] .doc-pages { display: grid; }' in css
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Node behavioral runs — the real renderDocument over a minimal fake DOM
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
_FAKE_DOM = """
|
||||||
|
function makeEl(tag) {
|
||||||
|
return {
|
||||||
|
tag, className: "", textContent: "", attrs: {}, children: [], handlers: {},
|
||||||
|
setAttribute(k, v) { this.attrs[k] = String(v); },
|
||||||
|
getAttribute(k) { return k in this.attrs ? this.attrs[k] : null; },
|
||||||
|
appendChild(c) { this.children.push(c); return c; },
|
||||||
|
append(...cs) { this.children.push(...cs); },
|
||||||
|
replaceChildren(...cs) { this.children = cs; },
|
||||||
|
addEventListener(ev, fn) { this.handlers[ev] = fn; },
|
||||||
|
click() { this.handlers.click(); },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
globalThis.document = { createElement: (t) => makeEl(t) };
|
||||||
|
function fmtDate(x) { return "DATE:" + x; }
|
||||||
|
function metaBadge(cls, text, title) {
|
||||||
|
const s = makeEl("span");
|
||||||
|
s.className = cls;
|
||||||
|
s.textContent = text;
|
||||||
|
if (title !== undefined) s.setAttribute("title", title);
|
||||||
|
return s;
|
||||||
|
}
|
||||||
|
function copyPathButton() { return makeEl("button"); }
|
||||||
|
function originAnchor() { return makeEl("a"); }
|
||||||
|
async function docAdminReady() { return false; }
|
||||||
|
function wireDateEdit() {}
|
||||||
|
function wireSummaryEdit() {}
|
||||||
|
function renderMarkdown(s) { return s; }
|
||||||
|
function docImageBlock() { return makeEl("div"); }
|
||||||
|
function describe(el) {
|
||||||
|
return {
|
||||||
|
tag: el.tag,
|
||||||
|
className: el.className,
|
||||||
|
attrs: el.attrs,
|
||||||
|
text: el.textContent,
|
||||||
|
children: el.children.map(describe),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def _render_script() -> str:
|
||||||
|
js = _read(DOCUMENT_JS)
|
||||||
|
parts = [
|
||||||
|
_fn(js, n)
|
||||||
|
for n in ("renderDocument", "docViewToggle", "viewToggleButton", "docPagesStrip")
|
||||||
|
]
|
||||||
|
return _FAKE_DOM + "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||||
|
def test_additive_doc_renders_text_views_and_toggle() -> None:
|
||||||
|
"""Behavioral (node, fake DOM): an ADDITIVE doc renders the TEXT
|
||||||
|
doc exactly as a non-qualifying one — Summary panel + content —
|
||||||
|
inside ``.doc-views`` (data-view="text"), with the toggle group
|
||||||
|
FIRST in the content slot: role=group, aria-label="Document
|
||||||
|
view", the two ``doc-view-btn`` buttons labeled Text/Pages, Text
|
||||||
|
aria-pressed="true" (LOCKED A6), and the strip MOUNTED inside the
|
||||||
|
wrapper (figures + lazy imgs) after the text view."""
|
||||||
|
out = _run_node(
|
||||||
|
_render_script()
|
||||||
|
+ """
|
||||||
|
const doc = {
|
||||||
|
title: "Letumdei", path: "book.pdf", source: "Library", format: "pdf",
|
||||||
|
summary: "A book.", content: "BODY TEXT", created_at: "c", indexed_at: "i",
|
||||||
|
chunks: 9, page_urls: ["/api/documents/x/pages/1", "/api/documents/x/pages/2"],
|
||||||
|
page_strip_mode: "additive",
|
||||||
|
};
|
||||||
|
const contentEl = makeEl("div");
|
||||||
|
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||||
|
console.log(JSON.stringify(describe(contentEl)));
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
content = json.loads(out)
|
||||||
|
assert [c["className"] for c in content["children"]] == [
|
||||||
|
"doc-view-toggle",
|
||||||
|
"doc-views",
|
||||||
|
], "the toggle group sits above the .doc-views wrapper"
|
||||||
|
toggle, views = content["children"]
|
||||||
|
assert toggle["attrs"]["role"] == "group"
|
||||||
|
assert toggle["attrs"]["aria-label"] == "Document view"
|
||||||
|
labels = [c["text"] for c in toggle["children"]]
|
||||||
|
assert labels == ["Text", "Pages"]
|
||||||
|
assert [c["attrs"]["aria-pressed"] for c in toggle["children"]] == ["true", "false"], (
|
||||||
|
"Text pressed by default (LOCKED A6)"
|
||||||
|
)
|
||||||
|
for c in toggle["children"]:
|
||||||
|
assert c["tag"] == "button"
|
||||||
|
assert views["attrs"]["data-view"] == "text"
|
||||||
|
assert [c["className"] for c in views["children"]] == [
|
||||||
|
"doc-summary",
|
||||||
|
"doc-raw",
|
||||||
|
"doc-pages",
|
||||||
|
], "additive = Summary panel + text content + the mounted strip"
|
||||||
|
# pre.textContent — a text node, never parsed as HTML
|
||||||
|
assert views["children"][1]["text"] == "BODY TEXT"
|
||||||
|
strip = views["children"][2]
|
||||||
|
assert len(strip["children"]) == 2, "one figure per page_urls entry"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||||
|
def test_primary_and_absent_modes_keep_the_strip_only_dom() -> None:
|
||||||
|
"""Behavioral: mode "primary" AND an absent mode (the pre-163
|
||||||
|
stamped row — the live Letumdei pre-sync shape) keep the phase-148
|
||||||
|
DOM byte-identically: the content slot holds EXACTLY the
|
||||||
|
``.doc-pages`` strip — no wrapper, no toggle, no Summary panel,
|
||||||
|
no text block."""
|
||||||
|
out = _run_node(
|
||||||
|
_render_script()
|
||||||
|
+ """
|
||||||
|
const results = [];
|
||||||
|
for (const mode of ["primary", undefined]) {
|
||||||
|
const doc = {
|
||||||
|
title: "Manual", path: "m.pdf", source: "Library", format: "pdf",
|
||||||
|
summary: "A manual.", content: "SHOULD NOT SHOW", created_at: "c",
|
||||||
|
indexed_at: "i", chunks: 4,
|
||||||
|
page_urls: ["/api/documents/y/pages/1"],
|
||||||
|
};
|
||||||
|
if (mode !== undefined) doc.page_strip_mode = mode;
|
||||||
|
const contentEl = makeEl("div");
|
||||||
|
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||||
|
results.push(describe(contentEl));
|
||||||
|
}
|
||||||
|
console.log(JSON.stringify(results));
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
primary, absent = json.loads(out)
|
||||||
|
for content in (primary, absent):
|
||||||
|
assert len(content["children"]) == 1
|
||||||
|
assert content["children"][0]["className"] == "doc-pages"
|
||||||
|
classes = json.dumps(content)
|
||||||
|
assert "doc-view-toggle" not in classes
|
||||||
|
assert "doc-views" not in classes
|
||||||
|
assert "doc-summary" not in classes
|
||||||
|
assert "doc-raw" not in classes, "the strip replaces the text block (primary class)"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||||
|
def test_text_doc_dom_is_untouched() -> None:
|
||||||
|
"""Behavioral: a doc with no page_urls is UNTOUCHED — Summary
|
||||||
|
panel + content appended straight to the content slot, no
|
||||||
|
wrapper, no toggle (the byte-identity property for every
|
||||||
|
non-PDF)."""
|
||||||
|
out = _run_node(
|
||||||
|
_render_script()
|
||||||
|
+ """
|
||||||
|
const doc = {
|
||||||
|
title: "Note", path: "n.pdf", source: "Library", format: "pdf",
|
||||||
|
summary: "S.", content: "TEXT", created_at: "c", indexed_at: "i", chunks: 1,
|
||||||
|
};
|
||||||
|
const contentEl = makeEl("div");
|
||||||
|
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||||
|
console.log(JSON.stringify(describe(contentEl)));
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
content = json.loads(out)
|
||||||
|
assert [c["className"] for c in content["children"]] == ["doc-summary", "doc-raw"]
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||||
|
def test_toggle_flips_view_without_rerender_or_refetch() -> None:
|
||||||
|
"""Behavioral: the toggle handler mutates ONLY the two
|
||||||
|
aria-pressed states + the wrapper's data-view. Clicking Pages
|
||||||
|
flips data-view to "pages"; clicking back to Text reverses; the
|
||||||
|
strip node STAYS MOUNTED the whole time (same object, no
|
||||||
|
re-append — the flip-flop fetches nothing) and the wrapper's
|
||||||
|
children never change count."""
|
||||||
|
out = _run_node(
|
||||||
|
_render_script()
|
||||||
|
+ """
|
||||||
|
const doc = {
|
||||||
|
title: "Book", path: "b.pdf", source: "Library", format: "pdf",
|
||||||
|
summary: "S.", content: "TEXT", created_at: "c", indexed_at: "i", chunks: 2,
|
||||||
|
page_urls: ["/api/documents/z/pages/1", "/api/documents/z/pages/2"],
|
||||||
|
page_strip_mode: "additive",
|
||||||
|
};
|
||||||
|
const contentEl = makeEl("div");
|
||||||
|
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||||
|
const toggle = contentEl.children[0];
|
||||||
|
const views = contentEl.children[1];
|
||||||
|
const [textBtn, pagesBtn] = toggle.children;
|
||||||
|
const stripBefore = views.children[2];
|
||||||
|
const countBefore = views.children.length;
|
||||||
|
pagesBtn.click();
|
||||||
|
const afterPages = {
|
||||||
|
view: views.getAttribute("data-view"),
|
||||||
|
pressed: toggle.children.map((b) => b.getAttribute("aria-pressed")),
|
||||||
|
};
|
||||||
|
textBtn.click();
|
||||||
|
const afterText = {
|
||||||
|
view: views.getAttribute("data-view"),
|
||||||
|
pressed: toggle.children.map((b) => b.getAttribute("aria-pressed")),
|
||||||
|
stripSame: views.children[2] === stripBefore,
|
||||||
|
countSame: views.children.length === countBefore,
|
||||||
|
};
|
||||||
|
console.log(JSON.stringify({ afterPages, afterText }));
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
res = json.loads(out)
|
||||||
|
assert res["afterPages"] == {"view": "pages", "pressed": ["false", "true"]}
|
||||||
|
assert res["afterText"] == {
|
||||||
|
"view": "text",
|
||||||
|
"pressed": ["true", "false"],
|
||||||
|
"stripSame": True,
|
||||||
|
"countSame": True,
|
||||||
|
}, "the strip node persists across flips — no re-render, no refetch"
|
||||||
@@ -13,12 +13,13 @@ without a browser:
|
|||||||
* NO ``@media (min-width: 1500px)`` block anywhere in the file — the
|
* NO ``@media (min-width: 1500px)`` block anywhere in the file — the
|
||||||
phase-58 wide-desktop doubling is deleted in full (the token is
|
phase-58 wide-desktop doubling is deleted in full (the token is
|
||||||
72rem at every viewport);
|
72rem at every viewport);
|
||||||
* the six reading-column selectors — ``.chat-shell``,
|
* the reading-column selectors — ``.chat-shell``,
|
||||||
``.shared-shell``, ``.doc-md``, ``.doc-image`` (phase 122, task 04),
|
``.shared-shell``, ``.doc-md``, ``.doc-image`` (phase 122, task 04),
|
||||||
``.doc-pages`` (phase 148, task 03 — the page strip), and
|
``.doc-pages`` (phase 148, task 03 — the page strip),
|
||||||
``.doc-summary:has(+ .doc-md)`` — each capped with
|
``.doc-view-toggle`` (phase 163, task 04 — the Text/Pages toggle
|
||||||
``max-width: var(--chat-column)`` and NOTHING else in the file uses
|
above the additive views), and ``.doc-summary:has(+ .doc-md)`` —
|
||||||
the token for a max-width (exactly six rules);
|
each capped with ``max-width: var(--chat-column)`` and NOTHING else
|
||||||
|
in the file uses the token for a max-width (exactly seven rules);
|
||||||
* the flipped negative pin — ZERO literal ``max-width: 46rem`` rules
|
* the flipped negative pin — ZERO literal ``max-width: 46rem`` rules
|
||||||
remain (the phase-27/59/91 form-column caps are retired): the form
|
remain (the phase-27/59/91 form-column caps are retired): the form
|
||||||
shells (``.tuning-shell``, ``.theme-shell``, ``.doc-edit-shell``)
|
shells (``.tuning-shell``, ``.theme-shell``, ``.doc-edit-shell``)
|
||||||
@@ -121,8 +122,10 @@ def test_the_reading_columns_use_the_token() -> None:
|
|||||||
task 04 — the viewer's image block rides the SAME reading column
|
task 04 — the viewer's image block rides the SAME reading column
|
||||||
as its .doc-md sibling), .doc-pages (phase 148, task 03 — the
|
as its .doc-md sibling), .doc-pages (phase 148, task 03 — the
|
||||||
viewer's page strip rides the SAME reading column in the shared
|
viewer's page strip rides the SAME reading column in the shared
|
||||||
#doc-content slot) and .doc-summary:has(+ .doc-md) each cap with
|
#doc-content slot), .doc-view-toggle (phase 163, task 04 — the
|
||||||
max-width: var(--chat-column) — and exactly those six rules use
|
Text/Pages toggle sits above the additive views, aligned with
|
||||||
|
them) and .doc-summary:has(+ .doc-md) each cap with
|
||||||
|
max-width: var(--chat-column) — and exactly those seven rules use
|
||||||
the token for a max-width (no other selector)."""
|
the token for a max-width (no other selector)."""
|
||||||
css = _css()
|
css = _css()
|
||||||
for selector in (
|
for selector in (
|
||||||
@@ -131,12 +134,13 @@ def test_the_reading_columns_use_the_token() -> None:
|
|||||||
".doc-md",
|
".doc-md",
|
||||||
".doc-image",
|
".doc-image",
|
||||||
".doc-pages",
|
".doc-pages",
|
||||||
|
".doc-view-toggle",
|
||||||
".doc-summary:has(+ .doc-md)",
|
".doc-summary:has(+ .doc-md)",
|
||||||
):
|
):
|
||||||
assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
|
assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
|
||||||
f"{selector} must cap with max-width: var(--chat-column)"
|
f"{selector} must cap with max-width: var(--chat-column)"
|
||||||
)
|
)
|
||||||
assert css.count("max-width: var(--chat-column)") == 6, (
|
assert css.count("max-width: var(--chat-column)") == 7, (
|
||||||
"exactly the reading-column selectors use the token"
|
"exactly the reading-column selectors use the token"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user