phase: 163_pdf_strip_additive_selfheal
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Successful in 21s

All gates verified. Final report:

**Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)**

- Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs).
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%)
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation
- `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed
- `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins)
- Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅
- **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**.
- Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit.
- **Next pending phase:** `164_chat_error_honesty`
This commit is contained in:
2026-10-07 13:57:30 -04:00
parent 74eb2e265c
commit 6ba46e735a
52 changed files with 4729 additions and 212 deletions
+83 -48
View File
@@ -36,33 +36,38 @@ token (the phase-126 convention — the owner's local ``.env`` may lack
it, and without it the walk is blind to the fixture).
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
blocks (and the seed's DB witness that the descriptions ARE indexed
but NOT displayed) are exact only against the mock's deterministic
``PICTORIAL_PAGE_MODE`` canned reply.
blocks (and the seed's DB witness that the descriptions ARE indexed)
are exact only against the mock's deterministic
``PICTORIAL_PAGE_MODE`` canned reply. (Phase 163 re-cut: the viewer
assertions moved from the phase-148 suppression contract to the
ADDITIVE contract — the pictorial class shows its text INCLUDING the
page descriptions, with the strip behind the Text/Pages toggle.)
Test → observable mapping (Playwright Mapping Rule):
1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module)
asserts the DB + filesystem first: the pictorial row has
``page_count == 3`` and the scratch page dir holds EXACTLY
``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy
doc's copies do not exist); then the Sources-tree doc modal shows
the ``.doc-pages`` strip: exactly three ``.doc-page`` figures,
each a LOADED lazy ``<img>`` (``complete && naturalWidth > 0`` —
the route served a real PNG, not a 404) with the
``Page N of <title>`` alt, the ``Page N`` caption, the NO
``.doc-raw`` / NO ``.doc-md`` / NO ``.doc-summary`` suppression
contract (the mock's page descriptions sit in the DB content but
nowhere in the rendered modal), the meta badge row (source /
Created / Indexed) untouched.
doc's copies do not exist); then the Sources-tree doc modal opens
the ADDITIVE view (phase 163 — a usable text layer: TEXT first,
the strip one toggle away): the Text/Pages group with ``Text``
pressed, the Summary panel + content visible (the mock's page
descriptions RIDE the shown text — LOCKED A1/A6), the
``.doc-pages`` strip mounted but hidden; pressing ``Pages`` shows
exactly three ``.doc-page`` figures, each a LOADED lazy ``<img>``
(``complete && naturalWidth > 0`` — the route served a real PNG,
not a 404) with the ``Page N of <title>`` alt and the ``Page N``
caption while the text view is hidden (mounted for the flip-
back), the meta badge row (source / Created / Indexed) untouched.
2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the
PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the
Summary panel (the mock's lite digest), NO ``.doc-pages`` — and
the content payload's ``page_urls`` KEY is ABSENT (never ``null`` —
the omission pin the viewer's strip gate rides on).
3. ``test_full_page_surface`` — the same strip on the dedicated
``/document.html`` page (admin-logged-in, the phase-79 gate
satisfied): one ``renderDocument`` core — no per-surface drift —
same figures, same suppression.
3. ``test_full_page_surface`` — the same additive view + strip on the
dedicated ``/document.html`` page (admin-logged-in, the phase-79
gate satisfied): one ``renderDocument`` core — no per-surface
drift — same toggle, same figures, same hidden-while-Text shape.
4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct
request on ``/api/documents/<id>/pages/1``: 200, ``image/png``,
the ``Cache-Control: private, max-age=3600`` header, a non-empty
@@ -597,15 +602,25 @@ def _wait_img_loaded(img: Locator) -> None:
def _assert_page_strip(
content: Locator, title: str, doc_id: str, page_count: int = 3
) -> None:
"""The phase-148 strip contract inside a content container (the
ONE ``renderDocument`` core — modal and full page, no
per-surface drift): a ``.doc-pages`` strip with exactly
*page_count* ``.doc-page`` figures; each figure's lazy ``<img>``
is LOADED (the route served a real PNG) with the ``Page N of
<title>`` alt and the exact 1-based route src, and the
figcaption reads ``Page N``; the text content block AND the
Summary panel are SUPPRESSED (the descriptions stay in the DB,
not displayed)."""
"""The strip contract inside a content container (the ONE
``renderDocument`` core — modal and full page, no per-surface
drift): a ``.doc-pages`` strip with exactly *page_count*
``.doc-page`` figures; each figure's lazy ``<img>`` is LOADED (the
route served a real PNG) with the ``Page N of <title>`` alt and
the exact 1-based route src, and the figcaption reads ``Page N``.
Phase 163 (task 04) re-cut: this fixture is the PICTORIAL class —
an ADDITIVE doc (``page_strip_mode == "additive"``) that opens on
its TEXT view, so the strip is revealed by pressing the toggle's
``Pages`` button first (a primary/absent doc has no toggle — the
strip already IS the content). The text content block and the
Summary panel are HIDDEN while the strip shows (they stay mounted
for the flip back — LOCKED A1); a primary-class doc has no such
nodes at all, which the same ``not_to_be_visible`` assertion
covers."""
toggle = content.locator(".doc-view-toggle")
if toggle.count() > 0:
toggle.get_by_role("button", name="Pages").click()
strip = content.locator(".doc-pages")
expect(strip).to_have_count(1)
figures = strip.locator(".doc-page")
@@ -624,12 +639,12 @@ def _assert_page_strip(
)
_wait_img_loaded(img)
expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}")
# The suppression contract (the owner's TODO L5): no text content
# block, no Summary panel — the meta badge row is asserted by the
# caller (it renders exactly as today).
expect(content.locator("pre.doc-raw")).to_have_count(0)
expect(content.locator(".doc-md")).to_have_count(0)
expect(content.locator(".doc-summary")).to_have_count(0)
# While the strip shows, the text view is not (phase 163: hidden,
# not unmounted — the additive flip-back; a primary doc has no
# such nodes at all). The meta badge row is asserted by the caller.
expect(content.locator("pre.doc-raw")).not_to_be_visible()
expect(content.locator(".doc-md")).not_to_be_visible()
expect(content.locator(".doc-summary")).not_to_be_visible()
# ---------------------------------------------------------------------------
@@ -640,15 +655,18 @@ def _assert_page_strip(
def test_pictorial_pdf_shows_pages_in_modal(
page: Page, app_url: str, seeded: dict[str, Any]
) -> None:
"""The IKEA-shaped PDF opens in the doc modal as a strip of its
REAL pages (the owner's L5 ask): ``.doc-pages`` with exactly three
"""The IKEA-shaped PDF opens in the doc modal as an ADDITIVE doc
(phase 163 — a usable text layer: the pictorial class): the TEXT
view first (Summary panel + content visible, the toggle's ``Text``
pressed — LOCKED A6, the mock's page descriptions ride the shown
text), with the strip of its REAL pages (the owner's L5 ask) one
``Pages`` click away: ``.doc-pages`` with exactly three
``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth
> 0 proves the full pipeline: sync → render → persist → route →
viewer) with the ``Page N of <title>`` alt and the ``Page N``
caption — and the suppression contract: NO text content
(``.doc-raw``), NO Summary panel (the mock's page descriptions are
in the DB content, nowhere in the rendered modal); the meta badge
row (source / Created / Indexed) renders as today."""
caption — while the text view is hidden (mounted for the
flip-back); the meta badge row (source / Created / Indexed)
renders as today."""
login(page, app_url) # lands on /sources.html
_drill_to_source(page, SOURCE_NAME)
expect(page.locator("#docs-tbody tr")).to_have_count(2)
@@ -658,6 +676,29 @@ def test_pictorial_pdf_shows_pages_in_modal(
)
content = page.locator("#doc-modal-content")
# Phase 163 (task 04, LOCKED A1/A6): the pictorial class is an
# ADDITIVE doc — it opens on its TEXT view: Summary panel +
# content visible, the toggle's Text pressed, the strip mounted
# but hidden (its lazy imgs never fetched while hidden).
toggle = content.locator(".doc-view-toggle")
expect(toggle).to_have_count(1)
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
"aria-pressed", "true"
)
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
"aria-pressed", "false"
)
expect(content.locator("pre.doc-raw")).to_be_visible()
expect(content.locator(".doc-summary")).to_be_visible()
expect(content.locator(".doc-pages")).to_be_hidden()
# The text view carries the indexed content — the phase-140 page
# descriptions ride the SHOWN text for an additive doc (the text
# is never hidden: A1).
assert PICTORIAL_PAGE_ANSWER in content.inner_text(), (
"the additive text view shows the indexed content including "
"the described sparse pages"
)
# …and the strip is one Pages click away (the helper flips).
_assert_page_strip(content, seeded["title"], seeded["pictorial_id"])
# The meta badge row is untouched by the strip (the pre-phase row).
@@ -666,13 +707,6 @@ def test_pictorial_pdf_shows_pages_in_modal(
expect(meta.locator(".doc-created")).to_contain_text("Created")
expect(meta.locator(".doc-indexed")).to_contain_text("Indexed")
# The suppression witness: the mock's canned page descriptions sit
# in the DB (the seed asserted it) but are NOT displayed.
assert PICTORIAL_PAGE_ANSWER not in content.inner_text(), (
"the described pages' text is in the DB for Q&A — it must not "
"be shown to the user (the owner's contract)"
)
# ---------------------------------------------------------------------------
# 2. The text-heavy PDF: byte-identical to pre-phase
@@ -730,10 +764,11 @@ def test_textheavy_pdf_unchanged(
def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None:
"""The dedicated ``/document.html`` page (admin-logged-in — the
phase-79 gate satisfied) renders the SAME strip through the ONE
shared ``renderDocument`` core: the same three loaded figures with
captions/alt texts, the same suppression (no text content, no
Summary panel), the meta row as today."""
phase-79 gate satisfied) renders the SAME additive view through
the ONE shared ``renderDocument`` core (no per-surface drift): the
text view first, the strip one ``Pages`` click away with the same
three loaded figures + captions/alt texts and the text view
hidden while it shows, the meta row as today."""
login(page, app_url, next="/")
page.goto(
f"{app_url}/document.html?source={quote(SOURCE_NAME)}"
File diff suppressed because it is too large Load Diff
+64 -44
View File
@@ -24,11 +24,14 @@ format — the upload gate admits it) and arriving through the REAL admin
upload flow → the RAG page's Sync button:
* (a) the document appears on the Sources page and the document viewer
shows the PDF's ACTUAL pages (phase 148 — a described sparse page
makes the doc page-QUALIFYING: the page strip REPLACES the text
content AND the Summary panel, the owner's TODO L5 contract): two
``.doc-page`` figures with loaded lazy images + ``Page N`` captions,
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself
opens the PDF in its ADDITIVE view (phase 163 — a described sparse
page makes the doc page-QUALIFYING, and a usable text layer makes
the strip ADDITIVE: the text stays, the pages sit behind a
Text/Pages toggle — LOCKED A1/A6): the Text view first (``.doc-raw``
+ ``.doc-summary`` visible, ``Text`` pressed), the strip one
``Pages`` click away — two ``.doc-page`` figures with loaded lazy
images + ``Page N`` captions, the text view hidden while it shows;
the described content itself
(page 1's prose WITHOUT an illustration block, page 2 WITH
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
@@ -83,14 +86,17 @@ Test → observable mapping (Playwright Mapping Rule):
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
the summary chunk — no image doc), the drill-down table lists the PDF
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
the same-page document viewer (the row's path link) shows the
PHASE-148 PAGE STRIP under the METADATA title (the fixture is
page-qualifying — the pictorial pass described its sparse page):
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>``
(naturalWidth > 0) with the ``Page N of <title>`` alt + the
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the
page descriptions stay in the DB for Q&A, not displayed — the
owner's contract).
the same-page document viewer (the row's path link) opens the
PHASE-163 ADDITIVE VIEW under the METADATA title (the fixture is
page-qualifying with a usable text layer — the pictorial pass
described its sparse page): the TEXT view first — ``.doc-raw`` +
``.doc-summary`` visible with the ``[Page 2 illustration: …]``
block among them, the toggle's ``Text`` pressed, the strip mounted
but hidden; pressing ``Pages`` shows exactly two ``.doc-page``
figures, each a LOADED lazy ``<img>`` (naturalWidth > 0) with the
``Page N of <title>`` alt + the ``Page N`` caption while the text
view hides (mounted for the flip-back — the text is never
destroyed).
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
SSE over the scripted seeded-read question (httpx — the ``done``
event is the wire lens the task pins; the UI flow's premise is the
@@ -578,8 +584,8 @@ def seeded(
)
# The non-markdown doc got its lite summary (the mock's
# SUMMARY_MODE digest) — the summary chunk exists (the
# viewer's Summary panel is suppressed by the phase-148
# strip — the summary itself stays in the DB).
# additive view shows the Summary panel; the summary
# itself stays in the DB regardless).
assert doc.summary
# Phase 148: the described sparse page makes this doc
# page-QUALIFYING — the strip's payload (page_urls) rides
@@ -602,18 +608,19 @@ def test_sources_page_and_viewer_show_the_described_page(
content chunks + the phase-30 summary chunk — no image doc; the
vector figure is invisible to the embedded-image pipeline). The
drill-down table lists the PDF as a plain text row (LOCKED A4), and
the same-page document viewer (the row's path link, phase 26) shows
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is
page-qualifying (the pictorial pass described its sparse page), so
the strip REPLACES the text content AND the Summary panel — two
``.doc-page`` figures, each a loaded lazy ``<img>`` with the
``Page N of <title>`` alt + the ``Page N`` caption, NO
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's
prose WITHOUT an illustration block, page 2 WITH
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB
for Q&A: the seed asserted it byte-exact, the chat test proves it
reaches the model — it is simply not displayed (the owner's
contract)."""
the same-page document viewer (the row's path link, phase 26)
opens the PHASE-163 ADDITIVE VIEW (LOCKED A1/A6): the TEXT view
first — Summary panel + ``.doc-raw`` visible, the Text/Pages
toggle's ``Text`` pressed — with the ``[Page 2 illustration:
<canned>]`` block riding the SHOWN text (the text is never hidden
for the pictorial class). Pressing ``Pages`` reveals the strip —
two ``.doc-page`` figures, each a loaded lazy ``<img>`` with the
``Page N of <title>`` alt + the ``Page N`` caption — while the
text view is hidden (mounted, not removed: flipping back shows it
with zero refetch). The described content (page 1's prose WITHOUT
an illustration block, page 2 WITH the block — LOCKED A4) also
stays in the DB for Q&A: the seed asserted it byte-exact, the chat
test proves it reaches the model."""
login(page, app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("1")
expect(page.locator("#stat-chunks")).to_have_text("3")
@@ -630,13 +637,31 @@ def test_sources_page_and_viewer_show_the_described_page(
expect(modal).to_be_visible(timeout=30_000)
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
# Phase 148: the page-qualifying fixture shows its ACTUAL pages —
# a .doc-pages strip with exactly two .doc-page figures (the
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 —
# the route served a real PNG) with the "Page N of <title>" alt +
# the "Page N" caption. The text content AND the Summary panel are
# SUPPRESSED (the owner's contract — the descriptions stay in the
# DB for Q&A, not displayed).
# Phase 163 (task 04, LOCKED A1/A6): the page-qualifying fixture
# with a usable text layer is an ADDITIVE doc — it opens on its
# TEXT view: Summary panel + content visible, Text pressed, the
# strip mounted but hidden (its lazy imgs never fetched).
toggle = modal.locator(".doc-view-toggle")
expect(toggle).to_have_count(1)
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
"aria-pressed", "true"
)
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
"aria-pressed", "false"
)
expect(modal.locator("pre.doc-raw")).to_be_visible()
expect(modal.locator(".doc-summary")).to_be_visible()
expect(modal.locator(".doc-pages")).to_be_hidden()
# The canned page description RIDES the shown text (phase-140
# blocks are part of the content — the additive view never hides
# the text). The node stays in the DOM, so a visibility-scoped
# inner_text() check would miss it — use the DOM text.
assert PICTORIAL_PAGE_ANSWER in modal.evaluate("el => el.textContent"), (
"the additive text view shows the indexed content including "
"the described sparse page"
)
# …and the strip is one Pages click away (phase 148's ask):
toggle.get_by_role("button", name="Pages").click()
strip = modal.locator(".doc-pages")
expect(strip).to_have_count(1)
figures = strip.locator(".doc-page")
@@ -657,17 +682,12 @@ def test_sources_page_and_viewer_show_the_described_page(
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
f"Page {n}"
)
# The suppression contract: no text content block, no Summary
# panel, no image block (the PDF is not an image document).
expect(modal.locator("pre.doc-raw")).to_have_count(0)
expect(modal.locator(".doc-summary")).to_have_count(0)
# While the strip shows, the text view is HIDDEN — mounted for
# the flip back (LOCKED A1), never removed. No image block (the
# PDF is not an image document).
expect(modal.locator("pre.doc-raw")).not_to_be_visible()
expect(modal.locator(".doc-summary")).not_to_be_visible()
expect(modal.locator(".doc-image-img")).to_have_count(0)
# The canned description is in the DB (the seed asserted it)
# but nowhere in the rendered modal.
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
"the described page's text is in the DB for Q&A — it must not "
"be shown to the user (the owner's contract)"
)
# ---------------------------------------------------------------------------