phase: 148_pdf_pages_in_sources
**Phase 148 — final verification pass: complete, 100% green.**
- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
This commit is contained in:
@@ -24,13 +24,19 @@ format — the upload gate admits it) and arriving through the REAL admin
|
||||
upload flow → the RAG page's Sync button:
|
||||
|
||||
* (a) the document appears on the Sources page and the document viewer
|
||||
renders the stored content: page 1's prose WITHOUT an illustration
|
||||
block, page 2 WITH ``[Page 2 illustration: <the mock's canned
|
||||
PICTORIAL_PAGE_ANSWER>]`` (the task-02 ``PICTORIAL_PAGE_MODE`` branch
|
||||
— deterministic; the stored content is asserted BYTE-EXACT against
|
||||
the fixture's own extraction in the seed — the candidates are computed
|
||||
from ``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
|
||||
hardcoded page list);
|
||||
shows the PDF's ACTUAL pages (phase 148 — a described sparse page
|
||||
makes the doc page-QUALIFYING: the page strip REPLACES the text
|
||||
content AND the Summary panel, the owner's TODO L5 contract): two
|
||||
``.doc-page`` figures with loaded lazy images + ``Page N`` captions,
|
||||
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself
|
||||
(page 1's prose WITHOUT an illustration block, page 2 WITH
|
||||
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
|
||||
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
|
||||
the DB: the seed asserts it BYTE-EXACT against the fixture's own
|
||||
extraction (the candidates computed from
|
||||
``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
|
||||
hardcoded page list) and the chat test below proves it still reaches
|
||||
the model;
|
||||
* (b) a chat question that ONLY the illustration answers ("how are the
|
||||
two parts joined on page two?" — the phrase lives NOWHERE in the text
|
||||
layer) grounds (the described page's embedding retrieves — the mock's
|
||||
@@ -77,11 +83,14 @@ Test → observable mapping (Playwright Mapping Rule):
|
||||
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
|
||||
the summary chunk — no image doc), the drill-down table lists the PDF
|
||||
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
|
||||
the same-page document viewer (the row's path link) renders the
|
||||
stored content under the METADATA title: the page-1 prose (NO
|
||||
``[Page 1 illustration`` block), the page-2 marker + its
|
||||
``[Page 2 illustration: <canned>]`` block, and the Summary panel
|
||||
(a non-image doc with a non-empty summary).
|
||||
the same-page document viewer (the row's path link) shows the
|
||||
PHASE-148 PAGE STRIP under the METADATA title (the fixture is
|
||||
page-qualifying — the pictorial pass described its sparse page):
|
||||
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>``
|
||||
(naturalWidth > 0) with the ``Page N of <title>`` alt + the
|
||||
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the
|
||||
page descriptions stay in the DB for Q&A, not displayed — the
|
||||
owner's contract).
|
||||
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
|
||||
SSE over the scripted seeded-read question (httpx — the ``done``
|
||||
event is the wire lens the task pins; the UI flow's premise is the
|
||||
@@ -554,9 +563,14 @@ def seeded(
|
||||
"described sparse page — the pictorial pass misfired"
|
||||
)
|
||||
# The non-markdown doc got its lite summary (the mock's
|
||||
# SUMMARY_MODE digest) — the summary chunk + the viewer's
|
||||
# Summary panel exist.
|
||||
# SUMMARY_MODE digest) — the summary chunk exists (the
|
||||
# viewer's Summary panel is suppressed by the phase-148
|
||||
# strip — the summary itself stays in the DB).
|
||||
assert doc.summary
|
||||
# Phase 148: the described sparse page makes this doc
|
||||
# page-QUALIFYING — the strip's payload (page_urls) rides
|
||||
# on page_count (the 2-page fixture's count).
|
||||
assert doc.page_count == 2
|
||||
finally:
|
||||
page.close()
|
||||
yield {"title": PDF_TITLE}
|
||||
@@ -574,12 +588,18 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
content chunks + the phase-30 summary chunk — no image doc; the
|
||||
vector figure is invisible to the embedded-image pipeline). The
|
||||
drill-down table lists the PDF as a plain text row (LOCKED A4), and
|
||||
the same-page document viewer (the row's path link, phase 26)
|
||||
renders the stored content under the METADATA title: page 1's prose
|
||||
WITHOUT an illustration block (not a candidate) and page 2 WITH
|
||||
``[Page 2 illustration: <canned>]`` (the described sparse page —
|
||||
LOCKED A4's user surface), plus the Summary panel (a non-image doc
|
||||
with a non-empty summary)."""
|
||||
the same-page document viewer (the row's path link, phase 26) shows
|
||||
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is
|
||||
page-qualifying (the pictorial pass described its sparse page), so
|
||||
the strip REPLACES the text content AND the Summary panel — two
|
||||
``.doc-page`` figures, each a loaded lazy ``<img>`` with the
|
||||
``Page N of <title>`` alt + the ``Page N`` caption, NO
|
||||
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's
|
||||
prose WITHOUT an illustration block, page 2 WITH
|
||||
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB
|
||||
for Q&A: the seed asserted it byte-exact, the chat test proves it
|
||||
reaches the model — it is simply not displayed (the owner's
|
||||
contract)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("3")
|
||||
@@ -596,27 +616,44 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
||||
|
||||
raw = modal.locator("pre.doc-raw")
|
||||
# Page 1 — the prose WITHOUT any illustration block. The viewer
|
||||
# renders the content verbatim (``insert_textbox`` wraps mid-
|
||||
# sentence, so compare whitespace-normalized).
|
||||
expect(raw).to_contain_text("---PAGE-1---")
|
||||
rendered = " ".join(raw.inner_text().split())
|
||||
assert " ".join(PAGE_ONE.split()) in rendered, (
|
||||
"the page-1 prose is not what the viewer renders"
|
||||
)
|
||||
assert "[Page 1 illustration" not in rendered, (
|
||||
"page 1 (over the sparseness bar) must carry NO illustration block"
|
||||
)
|
||||
# Page 2 — the sparse page WITH the described block (a) — the mock's
|
||||
# canned reply, deterministic.
|
||||
expect(raw).to_contain_text("---PAGE-2---")
|
||||
expect(raw).to_contain_text(PAGE_TWO)
|
||||
expect(raw).to_contain_text(ILLUSTRATION_BLOCK)
|
||||
# The Summary panel renders (a non-image doc with a non-empty
|
||||
# summary); no image block (the PDF is not an image document).
|
||||
expect(modal.locator(".doc-summary")).to_have_count(1)
|
||||
# Phase 148: the page-qualifying fixture shows its ACTUAL pages —
|
||||
# a .doc-pages strip with exactly two .doc-page figures (the
|
||||
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 —
|
||||
# the route served a real PNG) with the "Page N of <title>" alt +
|
||||
# the "Page N" caption. The text content AND the Summary panel are
|
||||
# SUPPRESSED (the owner's contract — the descriptions stay in the
|
||||
# DB for Q&A, not displayed).
|
||||
strip = modal.locator(".doc-pages")
|
||||
expect(strip).to_have_count(1)
|
||||
figures = strip.locator(".doc-page")
|
||||
expect(figures).to_have_count(2)
|
||||
for i in range(2):
|
||||
n = i + 1
|
||||
img = figures.nth(i).locator("img")
|
||||
# The figure still carries its <img> (a lost copy would have
|
||||
# been swapped for the "Page N unavailable" note instead).
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_have_attribute("loading", "lazy")
|
||||
expect(img).to_have_attribute("alt", f"Page {n} of {PDF_TITLE}")
|
||||
img.scroll_into_view_if_needed()
|
||||
img.evaluate("el => el.decode()")
|
||||
assert img.evaluate("el => el.naturalWidth") > 0, (
|
||||
f"page {n} image did not load (the route must serve a real PNG)"
|
||||
)
|
||||
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
|
||||
f"Page {n}"
|
||||
)
|
||||
# The suppression contract: no text content block, no Summary
|
||||
# panel, no image block (the PDF is not an image document).
|
||||
expect(modal.locator("pre.doc-raw")).to_have_count(0)
|
||||
expect(modal.locator(".doc-summary")).to_have_count(0)
|
||||
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
||||
# The canned description is in the DB (the seed asserted it)
|
||||
# but nowhere in the rendered modal.
|
||||
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
|
||||
"the described page's text is in the DB for Q&A — it must not "
|
||||
"be shown to the user (the owner's contract)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user