phase: 148_pdf_pages_in_sources
Build and Push Containers / build-and-push-app (push) Successful in 2m28s
Build and Push Containers / build-and-push-db (push) Successful in 16s

**Phase 148 — final verification pass: complete, 100% green.**

- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
This commit is contained in:
2026-10-01 16:58:01 -04:00
parent 863519e980
commit b233c8d714
39 changed files with 3426 additions and 56 deletions
+77 -40
View File
@@ -24,13 +24,19 @@ format — the upload gate admits it) and arriving through the REAL admin
upload flow → the RAG page's Sync button:
* (a) the document appears on the Sources page and the document viewer
renders the stored content: page 1's prose WITHOUT an illustration
block, page 2 WITH ``[Page 2 illustration: <the mock's canned
PICTORIAL_PAGE_ANSWER>]`` (the task-02 ``PICTORIAL_PAGE_MODE`` branch
— deterministic; the stored content is asserted BYTE-EXACT against
the fixture's own extraction in the seed — the candidates are computed
from ``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
hardcoded page list);
shows the PDF's ACTUAL pages (phase 148 — a described sparse page
makes the doc page-QUALIFYING: the page strip REPLACES the text
content AND the Summary panel, the owner's TODO L5 contract): two
``.doc-page`` figures with loaded lazy images + ``Page N`` captions,
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself
(page 1's prose WITHOUT an illustration block, page 2 WITH
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
the DB: the seed asserts it BYTE-EXACT against the fixture's own
extraction (the candidates computed from
``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
hardcoded page list) and the chat test below proves it still reaches
the model;
* (b) a chat question that ONLY the illustration answers ("how are the
two parts joined on page two?" — the phrase lives NOWHERE in the text
layer) grounds (the described page's embedding retrieves — the mock's
@@ -77,11 +83,14 @@ Test → observable mapping (Playwright Mapping Rule):
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
the summary chunk — no image doc), the drill-down table lists the PDF
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
the same-page document viewer (the row's path link) renders the
stored content under the METADATA title: the page-1 prose (NO
``[Page 1 illustration`` block), the page-2 marker + its
``[Page 2 illustration: <canned>]`` block, and the Summary panel
(a non-image doc with a non-empty summary).
the same-page document viewer (the row's path link) shows the
PHASE-148 PAGE STRIP under the METADATA title (the fixture is
page-qualifying — the pictorial pass described its sparse page):
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>``
(naturalWidth > 0) with the ``Page N of <title>`` alt + the
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the
page descriptions stay in the DB for Q&A, not displayed — the
owner's contract).
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
SSE over the scripted seeded-read question (httpx — the ``done``
event is the wire lens the task pins; the UI flow's premise is the
@@ -554,9 +563,14 @@ def seeded(
"described sparse page — the pictorial pass misfired"
)
# The non-markdown doc got its lite summary (the mock's
# SUMMARY_MODE digest) — the summary chunk + the viewer's
# Summary panel exist.
# SUMMARY_MODE digest) — the summary chunk exists (the
# viewer's Summary panel is suppressed by the phase-148
# strip — the summary itself stays in the DB).
assert doc.summary
# Phase 148: the described sparse page makes this doc
# page-QUALIFYING — the strip's payload (page_urls) rides
# on page_count (the 2-page fixture's count).
assert doc.page_count == 2
finally:
page.close()
yield {"title": PDF_TITLE}
@@ -574,12 +588,18 @@ def test_sources_page_and_viewer_show_the_described_page(
content chunks + the phase-30 summary chunk — no image doc; the
vector figure is invisible to the embedded-image pipeline). The
drill-down table lists the PDF as a plain text row (LOCKED A4), and
the same-page document viewer (the row's path link, phase 26)
renders the stored content under the METADATA title: page 1's prose
WITHOUT an illustration block (not a candidate) and page 2 WITH
``[Page 2 illustration: <canned>]`` (the described sparse page —
LOCKED A4's user surface), plus the Summary panel (a non-image doc
with a non-empty summary)."""
the same-page document viewer (the row's path link, phase 26) shows
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is
page-qualifying (the pictorial pass described its sparse page), so
the strip REPLACES the text content AND the Summary panel — two
``.doc-page`` figures, each a loaded lazy ``<img>`` with the
``Page N of <title>`` alt + the ``Page N`` caption, NO
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's
prose WITHOUT an illustration block, page 2 WITH
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB
for Q&A: the seed asserted it byte-exact, the chat test proves it
reaches the model — it is simply not displayed (the owner's
contract)."""
login(page, app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("1")
expect(page.locator("#stat-chunks")).to_have_text("3")
@@ -596,27 +616,44 @@ def test_sources_page_and_viewer_show_the_described_page(
expect(modal).to_be_visible(timeout=30_000)
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
raw = modal.locator("pre.doc-raw")
# Page 1 — the prose WITHOUT any illustration block. The viewer
# renders the content verbatim (``insert_textbox`` wraps mid-
# sentence, so compare whitespace-normalized).
expect(raw).to_contain_text("---PAGE-1---")
rendered = " ".join(raw.inner_text().split())
assert " ".join(PAGE_ONE.split()) in rendered, (
"the page-1 prose is not what the viewer renders"
)
assert "[Page 1 illustration" not in rendered, (
"page 1 (over the sparseness bar) must carry NO illustration block"
)
# Page 2 — the sparse page WITH the described block (a) — the mock's
# canned reply, deterministic.
expect(raw).to_contain_text("---PAGE-2---")
expect(raw).to_contain_text(PAGE_TWO)
expect(raw).to_contain_text(ILLUSTRATION_BLOCK)
# The Summary panel renders (a non-image doc with a non-empty
# summary); no image block (the PDF is not an image document).
expect(modal.locator(".doc-summary")).to_have_count(1)
# Phase 148: the page-qualifying fixture shows its ACTUAL pages —
# a .doc-pages strip with exactly two .doc-page figures (the
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 —
# the route served a real PNG) with the "Page N of <title>" alt +
# the "Page N" caption. The text content AND the Summary panel are
# SUPPRESSED (the owner's contract — the descriptions stay in the
# DB for Q&A, not displayed).
strip = modal.locator(".doc-pages")
expect(strip).to_have_count(1)
figures = strip.locator(".doc-page")
expect(figures).to_have_count(2)
for i in range(2):
n = i + 1
img = figures.nth(i).locator("img")
# The figure still carries its <img> (a lost copy would have
# been swapped for the "Page N unavailable" note instead).
expect(img).to_have_count(1)
expect(img).to_have_attribute("loading", "lazy")
expect(img).to_have_attribute("alt", f"Page {n} of {PDF_TITLE}")
img.scroll_into_view_if_needed()
img.evaluate("el => el.decode()")
assert img.evaluate("el => el.naturalWidth") > 0, (
f"page {n} image did not load (the route must serve a real PNG)"
)
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
f"Page {n}"
)
# The suppression contract: no text content block, no Summary
# panel, no image block (the PDF is not an image document).
expect(modal.locator("pre.doc-raw")).to_have_count(0)
expect(modal.locator(".doc-summary")).to_have_count(0)
expect(modal.locator(".doc-image-img")).to_have_count(0)
# The canned description is in the DB (the seed asserted it)
# but nowhere in the rendered modal.
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
"the described page's text is in the DB for Q&A — it must not "
"be shown to the user (the owner's contract)"
)
# ---------------------------------------------------------------------------