phase: 163_pdf_strip_additive_selfheal
All gates verified. Final report: **Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)** - Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs). - `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%) - `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation - `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed - `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins) - Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅ - **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**. - Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit. - **Next pending phase:** `164_chat_error_honesty`
This commit is contained in:
@@ -36,33 +36,38 @@ token (the phase-126 convention — the owner's local ``.env`` may lack
|
||||
it, and without it the walk is blind to the fixture).
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
|
||||
blocks (and the seed's DB witness that the descriptions ARE indexed
|
||||
but NOT displayed) are exact only against the mock's deterministic
|
||||
``PICTORIAL_PAGE_MODE`` canned reply.
|
||||
blocks (and the seed's DB witness that the descriptions ARE indexed)
|
||||
are exact only against the mock's deterministic
|
||||
``PICTORIAL_PAGE_MODE`` canned reply. (Phase 163 re-cut: the viewer
|
||||
assertions moved from the phase-148 suppression contract to the
|
||||
ADDITIVE contract — the pictorial class shows its text INCLUDING the
|
||||
page descriptions, with the strip behind the Text/Pages toggle.)
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module)
|
||||
asserts the DB + filesystem first: the pictorial row has
|
||||
``page_count == 3`` and the scratch page dir holds EXACTLY
|
||||
``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy
|
||||
doc's copies do not exist); then the Sources-tree doc modal shows
|
||||
the ``.doc-pages`` strip: exactly three ``.doc-page`` figures,
|
||||
each a LOADED lazy ``<img>`` (``complete && naturalWidth > 0`` —
|
||||
the route served a real PNG, not a 404) with the
|
||||
``Page N of <title>`` alt, the ``Page N`` caption, the NO
|
||||
``.doc-raw`` / NO ``.doc-md`` / NO ``.doc-summary`` suppression
|
||||
contract (the mock's page descriptions sit in the DB content but
|
||||
nowhere in the rendered modal), the meta badge row (source /
|
||||
Created / Indexed) untouched.
|
||||
doc's copies do not exist); then the Sources-tree doc modal opens
|
||||
the ADDITIVE view (phase 163 — a usable text layer: TEXT first,
|
||||
the strip one toggle away): the Text/Pages group with ``Text``
|
||||
pressed, the Summary panel + content visible (the mock's page
|
||||
descriptions RIDE the shown text — LOCKED A1/A6), the
|
||||
``.doc-pages`` strip mounted but hidden; pressing ``Pages`` shows
|
||||
exactly three ``.doc-page`` figures, each a LOADED lazy ``<img>``
|
||||
(``complete && naturalWidth > 0`` — the route served a real PNG,
|
||||
not a 404) with the ``Page N of <title>`` alt and the ``Page N``
|
||||
caption while the text view is hidden (mounted for the flip-
|
||||
back), the meta badge row (source / Created / Indexed) untouched.
|
||||
2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the
|
||||
PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the
|
||||
Summary panel (the mock's lite digest), NO ``.doc-pages`` — and
|
||||
the content payload's ``page_urls`` KEY is ABSENT (never ``null`` —
|
||||
the omission pin the viewer's strip gate rides on).
|
||||
3. ``test_full_page_surface`` — the same strip on the dedicated
|
||||
``/document.html`` page (admin-logged-in, the phase-79 gate
|
||||
satisfied): one ``renderDocument`` core — no per-surface drift —
|
||||
same figures, same suppression.
|
||||
3. ``test_full_page_surface`` — the same additive view + strip on the
|
||||
dedicated ``/document.html`` page (admin-logged-in, the phase-79
|
||||
gate satisfied): one ``renderDocument`` core — no per-surface
|
||||
drift — same toggle, same figures, same hidden-while-Text shape.
|
||||
4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct
|
||||
request on ``/api/documents/<id>/pages/1``: 200, ``image/png``,
|
||||
the ``Cache-Control: private, max-age=3600`` header, a non-empty
|
||||
@@ -597,15 +602,25 @@ def _wait_img_loaded(img: Locator) -> None:
|
||||
def _assert_page_strip(
|
||||
content: Locator, title: str, doc_id: str, page_count: int = 3
|
||||
) -> None:
|
||||
"""The phase-148 strip contract inside a content container (the
|
||||
ONE ``renderDocument`` core — modal and full page, no
|
||||
per-surface drift): a ``.doc-pages`` strip with exactly
|
||||
*page_count* ``.doc-page`` figures; each figure's lazy ``<img>``
|
||||
is LOADED (the route served a real PNG) with the ``Page N of
|
||||
<title>`` alt and the exact 1-based route src, and the
|
||||
figcaption reads ``Page N``; the text content block AND the
|
||||
Summary panel are SUPPRESSED (the descriptions stay in the DB,
|
||||
not displayed)."""
|
||||
"""The strip contract inside a content container (the ONE
|
||||
``renderDocument`` core — modal and full page, no per-surface
|
||||
drift): a ``.doc-pages`` strip with exactly *page_count*
|
||||
``.doc-page`` figures; each figure's lazy ``<img>`` is LOADED (the
|
||||
route served a real PNG) with the ``Page N of <title>`` alt and
|
||||
the exact 1-based route src, and the figcaption reads ``Page N``.
|
||||
|
||||
Phase 163 (task 04) re-cut: this fixture is the PICTORIAL class —
|
||||
an ADDITIVE doc (``page_strip_mode == "additive"``) that opens on
|
||||
its TEXT view, so the strip is revealed by pressing the toggle's
|
||||
``Pages`` button first (a primary/absent doc has no toggle — the
|
||||
strip already IS the content). The text content block and the
|
||||
Summary panel are HIDDEN while the strip shows (they stay mounted
|
||||
for the flip back — LOCKED A1); a primary-class doc has no such
|
||||
nodes at all, which the same ``not_to_be_visible`` assertion
|
||||
covers."""
|
||||
toggle = content.locator(".doc-view-toggle")
|
||||
if toggle.count() > 0:
|
||||
toggle.get_by_role("button", name="Pages").click()
|
||||
strip = content.locator(".doc-pages")
|
||||
expect(strip).to_have_count(1)
|
||||
figures = strip.locator(".doc-page")
|
||||
@@ -624,12 +639,12 @@ def _assert_page_strip(
|
||||
)
|
||||
_wait_img_loaded(img)
|
||||
expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}")
|
||||
# The suppression contract (the owner's TODO L5): no text content
|
||||
# block, no Summary panel — the meta badge row is asserted by the
|
||||
# caller (it renders exactly as today).
|
||||
expect(content.locator("pre.doc-raw")).to_have_count(0)
|
||||
expect(content.locator(".doc-md")).to_have_count(0)
|
||||
expect(content.locator(".doc-summary")).to_have_count(0)
|
||||
# While the strip shows, the text view is not (phase 163: hidden,
|
||||
# not unmounted — the additive flip-back; a primary doc has no
|
||||
# such nodes at all). The meta badge row is asserted by the caller.
|
||||
expect(content.locator("pre.doc-raw")).not_to_be_visible()
|
||||
expect(content.locator(".doc-md")).not_to_be_visible()
|
||||
expect(content.locator(".doc-summary")).not_to_be_visible()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -640,15 +655,18 @@ def _assert_page_strip(
|
||||
def test_pictorial_pdf_shows_pages_in_modal(
|
||||
page: Page, app_url: str, seeded: dict[str, Any]
|
||||
) -> None:
|
||||
"""The IKEA-shaped PDF opens in the doc modal as a strip of its
|
||||
REAL pages (the owner's L5 ask): ``.doc-pages`` with exactly three
|
||||
"""The IKEA-shaped PDF opens in the doc modal as an ADDITIVE doc
|
||||
(phase 163 — a usable text layer: the pictorial class): the TEXT
|
||||
view first (Summary panel + content visible, the toggle's ``Text``
|
||||
pressed — LOCKED A6, the mock's page descriptions ride the shown
|
||||
text), with the strip of its REAL pages (the owner's L5 ask) one
|
||||
``Pages`` click away: ``.doc-pages`` with exactly three
|
||||
``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth
|
||||
> 0 proves the full pipeline: sync → render → persist → route →
|
||||
viewer) with the ``Page N of <title>`` alt and the ``Page N``
|
||||
caption — and the suppression contract: NO text content
|
||||
(``.doc-raw``), NO Summary panel (the mock's page descriptions are
|
||||
in the DB content, nowhere in the rendered modal); the meta badge
|
||||
row (source / Created / Indexed) renders as today."""
|
||||
caption — while the text view is hidden (mounted for the
|
||||
flip-back); the meta badge row (source / Created / Indexed)
|
||||
renders as today."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
||||
@@ -658,6 +676,29 @@ def test_pictorial_pdf_shows_pages_in_modal(
|
||||
)
|
||||
|
||||
content = page.locator("#doc-modal-content")
|
||||
# Phase 163 (task 04, LOCKED A1/A6): the pictorial class is an
|
||||
# ADDITIVE doc — it opens on its TEXT view: Summary panel +
|
||||
# content visible, the toggle's Text pressed, the strip mounted
|
||||
# but hidden (its lazy imgs never fetched while hidden).
|
||||
toggle = content.locator(".doc-view-toggle")
|
||||
expect(toggle).to_have_count(1)
|
||||
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
|
||||
"aria-pressed", "true"
|
||||
)
|
||||
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
|
||||
"aria-pressed", "false"
|
||||
)
|
||||
expect(content.locator("pre.doc-raw")).to_be_visible()
|
||||
expect(content.locator(".doc-summary")).to_be_visible()
|
||||
expect(content.locator(".doc-pages")).to_be_hidden()
|
||||
# The text view carries the indexed content — the phase-140 page
|
||||
# descriptions ride the SHOWN text for an additive doc (the text
|
||||
# is never hidden: A1).
|
||||
assert PICTORIAL_PAGE_ANSWER in content.inner_text(), (
|
||||
"the additive text view shows the indexed content including "
|
||||
"the described sparse pages"
|
||||
)
|
||||
# …and the strip is one Pages click away (the helper flips).
|
||||
_assert_page_strip(content, seeded["title"], seeded["pictorial_id"])
|
||||
|
||||
# The meta badge row is untouched by the strip (the pre-phase row).
|
||||
@@ -666,13 +707,6 @@ def test_pictorial_pdf_shows_pages_in_modal(
|
||||
expect(meta.locator(".doc-created")).to_contain_text("Created")
|
||||
expect(meta.locator(".doc-indexed")).to_contain_text("Indexed")
|
||||
|
||||
# The suppression witness: the mock's canned page descriptions sit
|
||||
# in the DB (the seed asserted it) but are NOT displayed.
|
||||
assert PICTORIAL_PAGE_ANSWER not in content.inner_text(), (
|
||||
"the described pages' text is in the DB for Q&A — it must not "
|
||||
"be shown to the user (the owner's contract)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The text-heavy PDF: byte-identical to pre-phase
|
||||
@@ -730,10 +764,11 @@ def test_textheavy_pdf_unchanged(
|
||||
|
||||
def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None:
|
||||
"""The dedicated ``/document.html`` page (admin-logged-in — the
|
||||
phase-79 gate satisfied) renders the SAME strip through the ONE
|
||||
shared ``renderDocument`` core: the same three loaded figures with
|
||||
captions/alt texts, the same suppression (no text content, no
|
||||
Summary panel), the meta row as today."""
|
||||
phase-79 gate satisfied) renders the SAME additive view through
|
||||
the ONE shared ``renderDocument`` core (no per-surface drift): the
|
||||
text view first, the strip one ``Pages`` click away with the same
|
||||
three loaded figures + captions/alt texts and the text view
|
||||
hidden while it shows, the meta row as today."""
|
||||
login(page, app_url, next="/")
|
||||
page.goto(
|
||||
f"{app_url}/document.html?source={quote(SOURCE_NAME)}"
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -24,11 +24,14 @@ format — the upload gate admits it) and arriving through the REAL admin
|
||||
upload flow → the RAG page's Sync button:
|
||||
|
||||
* (a) the document appears on the Sources page and the document viewer
|
||||
shows the PDF's ACTUAL pages (phase 148 — a described sparse page
|
||||
makes the doc page-QUALIFYING: the page strip REPLACES the text
|
||||
content AND the Summary panel, the owner's TODO L5 contract): two
|
||||
``.doc-page`` figures with loaded lazy images + ``Page N`` captions,
|
||||
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself
|
||||
opens the PDF in its ADDITIVE view (phase 163 — a described sparse
|
||||
page makes the doc page-QUALIFYING, and a usable text layer makes
|
||||
the strip ADDITIVE: the text stays, the pages sit behind a
|
||||
Text/Pages toggle — LOCKED A1/A6): the Text view first (``.doc-raw``
|
||||
+ ``.doc-summary`` visible, ``Text`` pressed), the strip one
|
||||
``Pages`` click away — two ``.doc-page`` figures with loaded lazy
|
||||
images + ``Page N`` captions, the text view hidden while it shows;
|
||||
the described content itself
|
||||
(page 1's prose WITHOUT an illustration block, page 2 WITH
|
||||
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
|
||||
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
|
||||
@@ -83,14 +86,17 @@ Test → observable mapping (Playwright Mapping Rule):
|
||||
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
|
||||
the summary chunk — no image doc), the drill-down table lists the PDF
|
||||
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
|
||||
the same-page document viewer (the row's path link) shows the
|
||||
PHASE-148 PAGE STRIP under the METADATA title (the fixture is
|
||||
page-qualifying — the pictorial pass described its sparse page):
|
||||
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>``
|
||||
(naturalWidth > 0) with the ``Page N of <title>`` alt + the
|
||||
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the
|
||||
page descriptions stay in the DB for Q&A, not displayed — the
|
||||
owner's contract).
|
||||
the same-page document viewer (the row's path link) opens the
|
||||
PHASE-163 ADDITIVE VIEW under the METADATA title (the fixture is
|
||||
page-qualifying with a usable text layer — the pictorial pass
|
||||
described its sparse page): the TEXT view first — ``.doc-raw`` +
|
||||
``.doc-summary`` visible with the ``[Page 2 illustration: …]``
|
||||
block among them, the toggle's ``Text`` pressed, the strip mounted
|
||||
but hidden; pressing ``Pages`` shows exactly two ``.doc-page``
|
||||
figures, each a LOADED lazy ``<img>`` (naturalWidth > 0) with the
|
||||
``Page N of <title>`` alt + the ``Page N`` caption while the text
|
||||
view hides (mounted for the flip-back — the text is never
|
||||
destroyed).
|
||||
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
|
||||
SSE over the scripted seeded-read question (httpx — the ``done``
|
||||
event is the wire lens the task pins; the UI flow's premise is the
|
||||
@@ -578,8 +584,8 @@ def seeded(
|
||||
)
|
||||
# The non-markdown doc got its lite summary (the mock's
|
||||
# SUMMARY_MODE digest) — the summary chunk exists (the
|
||||
# viewer's Summary panel is suppressed by the phase-148
|
||||
# strip — the summary itself stays in the DB).
|
||||
# additive view shows the Summary panel; the summary
|
||||
# itself stays in the DB regardless).
|
||||
assert doc.summary
|
||||
# Phase 148: the described sparse page makes this doc
|
||||
# page-QUALIFYING — the strip's payload (page_urls) rides
|
||||
@@ -602,18 +608,19 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
content chunks + the phase-30 summary chunk — no image doc; the
|
||||
vector figure is invisible to the embedded-image pipeline). The
|
||||
drill-down table lists the PDF as a plain text row (LOCKED A4), and
|
||||
the same-page document viewer (the row's path link, phase 26) shows
|
||||
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is
|
||||
page-qualifying (the pictorial pass described its sparse page), so
|
||||
the strip REPLACES the text content AND the Summary panel — two
|
||||
``.doc-page`` figures, each a loaded lazy ``<img>`` with the
|
||||
``Page N of <title>`` alt + the ``Page N`` caption, NO
|
||||
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's
|
||||
prose WITHOUT an illustration block, page 2 WITH
|
||||
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB
|
||||
for Q&A: the seed asserted it byte-exact, the chat test proves it
|
||||
reaches the model — it is simply not displayed (the owner's
|
||||
contract)."""
|
||||
the same-page document viewer (the row's path link, phase 26)
|
||||
opens the PHASE-163 ADDITIVE VIEW (LOCKED A1/A6): the TEXT view
|
||||
first — Summary panel + ``.doc-raw`` visible, the Text/Pages
|
||||
toggle's ``Text`` pressed — with the ``[Page 2 illustration:
|
||||
<canned>]`` block riding the SHOWN text (the text is never hidden
|
||||
for the pictorial class). Pressing ``Pages`` reveals the strip —
|
||||
two ``.doc-page`` figures, each a loaded lazy ``<img>`` with the
|
||||
``Page N of <title>`` alt + the ``Page N`` caption — while the
|
||||
text view is hidden (mounted, not removed: flipping back shows it
|
||||
with zero refetch). The described content (page 1's prose WITHOUT
|
||||
an illustration block, page 2 WITH the block — LOCKED A4) also
|
||||
stays in the DB for Q&A: the seed asserted it byte-exact, the chat
|
||||
test proves it reaches the model."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("3")
|
||||
@@ -630,13 +637,31 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
||||
|
||||
# Phase 148: the page-qualifying fixture shows its ACTUAL pages —
|
||||
# a .doc-pages strip with exactly two .doc-page figures (the
|
||||
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 —
|
||||
# the route served a real PNG) with the "Page N of <title>" alt +
|
||||
# the "Page N" caption. The text content AND the Summary panel are
|
||||
# SUPPRESSED (the owner's contract — the descriptions stay in the
|
||||
# DB for Q&A, not displayed).
|
||||
# Phase 163 (task 04, LOCKED A1/A6): the page-qualifying fixture
|
||||
# with a usable text layer is an ADDITIVE doc — it opens on its
|
||||
# TEXT view: Summary panel + content visible, Text pressed, the
|
||||
# strip mounted but hidden (its lazy imgs never fetched).
|
||||
toggle = modal.locator(".doc-view-toggle")
|
||||
expect(toggle).to_have_count(1)
|
||||
expect(toggle.get_by_role("button", name="Text")).to_have_attribute(
|
||||
"aria-pressed", "true"
|
||||
)
|
||||
expect(toggle.get_by_role("button", name="Pages")).to_have_attribute(
|
||||
"aria-pressed", "false"
|
||||
)
|
||||
expect(modal.locator("pre.doc-raw")).to_be_visible()
|
||||
expect(modal.locator(".doc-summary")).to_be_visible()
|
||||
expect(modal.locator(".doc-pages")).to_be_hidden()
|
||||
# The canned page description RIDES the shown text (phase-140
|
||||
# blocks are part of the content — the additive view never hides
|
||||
# the text). The node stays in the DOM, so a visibility-scoped
|
||||
# inner_text() check would miss it — use the DOM text.
|
||||
assert PICTORIAL_PAGE_ANSWER in modal.evaluate("el => el.textContent"), (
|
||||
"the additive text view shows the indexed content including "
|
||||
"the described sparse page"
|
||||
)
|
||||
# …and the strip is one Pages click away (phase 148's ask):
|
||||
toggle.get_by_role("button", name="Pages").click()
|
||||
strip = modal.locator(".doc-pages")
|
||||
expect(strip).to_have_count(1)
|
||||
figures = strip.locator(".doc-page")
|
||||
@@ -657,17 +682,12 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
|
||||
f"Page {n}"
|
||||
)
|
||||
# The suppression contract: no text content block, no Summary
|
||||
# panel, no image block (the PDF is not an image document).
|
||||
expect(modal.locator("pre.doc-raw")).to_have_count(0)
|
||||
expect(modal.locator(".doc-summary")).to_have_count(0)
|
||||
# While the strip shows, the text view is HIDDEN — mounted for
|
||||
# the flip back (LOCKED A1), never removed. No image block (the
|
||||
# PDF is not an image document).
|
||||
expect(modal.locator("pre.doc-raw")).not_to_be_visible()
|
||||
expect(modal.locator(".doc-summary")).not_to_be_visible()
|
||||
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
||||
# The canned description is in the DB (the seed asserted it)
|
||||
# but nowhere in the rendered modal.
|
||||
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
|
||||
"the described page's text is in the DB for Q&A — it must not "
|
||||
"be shown to the user (the owner's contract)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -10,7 +10,10 @@ per phase 124 A4) and the phase-148 (task 02) PDF page route
|
||||
(``GET /api/documents/{id}/pages/{n}`` — the same trust model and 404
|
||||
posture for a page-qualifying document's rendered page PNGs) + the
|
||||
content endpoint's ``image_url`` / ``page_urls`` keys (present for the
|
||||
qualifying row, ABSENT otherwise — never ``null``).
|
||||
qualifying row, ABSENT otherwise — never ``null``), plus the phase-163
|
||||
(task 01) ``page_strip_mode`` class signal (``"primary"`` /``"additive"``
|
||||
for a stamped strip, ABSENT for a pre-phase row and for every
|
||||
non-qualifying document).
|
||||
|
||||
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
|
||||
"""
|
||||
@@ -1381,7 +1384,8 @@ def test_content_endpoint_exposes_the_image_affordance(
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 148 (task 02) — the PDF page BYTES route + the content payload's
|
||||
# ``page_urls`` affordance
|
||||
# ``page_urls`` affordance (phase 163 task 01 adds the ``page_strip_mode``
|
||||
# class signal beside it)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@@ -1404,6 +1408,7 @@ def _seed_page_doc(
|
||||
path: str = "besta.pdf",
|
||||
page_count: int | None = 2,
|
||||
write_copies: bool = True,
|
||||
page_strip_mode: str | None = None,
|
||||
) -> Document:
|
||||
"""One PDF document row with ``page_count`` + its rendered page
|
||||
copies (the importer's ``pdf_page_dir`` layout —
|
||||
@@ -1411,7 +1416,8 @@ def _seed_page_doc(
|
||||
sentinel bytes, so a page mix-up between the two copies is caught.
|
||||
``page_count=None`` is a non-qualifying PDF (no copies);
|
||||
``write_copies=False`` leaves a qualifying row without its files
|
||||
(the lost-copy corner)."""
|
||||
(the lost-copy corner); ``page_strip_mode`` (phase 163) stamps the
|
||||
strip CLASS — ``None`` is the pre-phase row."""
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source=source,
|
||||
@@ -1424,6 +1430,7 @@ def _seed_page_doc(
|
||||
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
||||
summary="IKEA BESTA frame assembly manual (page descriptions).",
|
||||
page_count=page_count,
|
||||
page_strip_mode=page_strip_mode,
|
||||
)
|
||||
db.add(doc)
|
||||
db.flush()
|
||||
@@ -1610,6 +1617,79 @@ def test_content_endpoint_exposes_the_page_urls_affordance(
|
||||
)
|
||||
assert r.status_code == 200
|
||||
assert "page_urls" not in r.json() # absent — never null (non-qualifying PDF)
|
||||
# Phase 163 (task 01): the class key follows the same rule — a
|
||||
# pre-phase row (mode NULL) omits it while keeping page_urls.
|
||||
assert "page_strip_mode" not in r.json()
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["primary", "additive"])
|
||||
def test_content_endpoint_carries_the_strip_class(
|
||||
admin_client: TestClient, db, tmp_path: Path, mode: str
|
||||
) -> None:
|
||||
"""Phase 163 (task 01, LOCKED A2/A3) — the real-DB round-trip of the
|
||||
strip CLASS: a stamped row (``page_strip_mode`` = ``"primary"`` — the
|
||||
phase-133 fallback class, the strip IS the view — or ``"additive"`` —
|
||||
the phase-140 pictorial class, text + a Text/Pages toggle) serves the
|
||||
class VERBATIM beside the untouched ``page_urls`` list (the viewer's
|
||||
branch signal, task 04); every other key stands where phase 148 left
|
||||
it."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
pages_dir = tmp_path / "pages"
|
||||
_seed_page_doc(db, pages_dir, page_count=3, page_strip_mode=mode)
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "PdfSrc", "path": "besta.pdf"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["page_strip_mode"] == mode
|
||||
assert len(body["page_urls"]) == 3 # the strip rides along, unchanged
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_content_strip_class_never_travels_without_the_strip(
|
||||
admin_client: TestClient, db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 163 (task 01) — the two strip keys are ONE block keyed off
|
||||
``page_count``: a row with a stray ``page_strip_mode`` and no strip
|
||||
(``page_count`` NULL — the corner a partial repair could leave)
|
||||
serializes NEITHER key, byte-identically to a plain non-qualifying
|
||||
document (the viewer must never be handed a class for pages that do
|
||||
not exist)."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
pages_dir = tmp_path / "pages"
|
||||
_seed_page_doc(
|
||||
db,
|
||||
pages_dir,
|
||||
path="stray.pdf",
|
||||
page_count=None,
|
||||
write_copies=False,
|
||||
page_strip_mode="additive",
|
||||
)
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "PdfSrc", "path": "stray.pdf"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert "page_strip_mode" not in body
|
||||
assert "page_urls" not in body
|
||||
# The exact pre-phase key set (the byte-identity pin).
|
||||
assert set(body) == {
|
||||
"source",
|
||||
"path",
|
||||
"title",
|
||||
"format",
|
||||
"summary",
|
||||
"created_at",
|
||||
"content",
|
||||
"indexed_at",
|
||||
"chunks",
|
||||
"is_image",
|
||||
}
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
@@ -0,0 +1,370 @@
|
||||
"""Integration: migration 0027 (documents.page_strip_mode) schema contract
|
||||
(phase 163, task 01).
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the house pattern of
|
||||
``test_migration_0026.py`` (information_schema assertions on the state
|
||||
the migration must leave). The tests target the 0026 → 0027 step
|
||||
explicitly so later migrations cannot break the pins:
|
||||
|
||||
* upgrade 0026 → 0027 → ``page_strip_mode`` exists with the full
|
||||
contract — VARCHAR(16), NULLABLE, NO server default (NULL = "no strip"
|
||||
for every pre-phase row, every non-qualifying document and every
|
||||
fail-soft miss) and UNINDEXED (a display fact, never a query surface —
|
||||
the content endpoint reads it from the ROW, the ``file_hash`` /
|
||||
``page_count`` convention) — while the 0026 ``documents`` schema
|
||||
survives, in particular ``page_count`` keeping its 0026 contract and
|
||||
``content_hash`` staying NOT NULL and INDEXED;
|
||||
* rows inserted while the DB is at 0026 backfill ``page_strip_mode`` to
|
||||
NULL — including a row whose ``page_count`` IS set (the pre-phase strip
|
||||
with no stamped class: the wire omits the key and the viewer keeps
|
||||
today's "primary" DOM until the task-03 re-gate stamps it);
|
||||
* the ORM contract agrees: a freshly inserted ``Document`` with no mode
|
||||
reads ``page_strip_mode is None`` through a fresh session, and each of
|
||||
the two classes (``"primary"`` / ``"additive"``) round-trips verbatim;
|
||||
* downgrade to 0026 → the column is GONE (A13) while the row + its
|
||||
content + its ``page_count`` survive; upgrade back to 0027 → the column
|
||||
is back (round-trip).
|
||||
|
||||
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||
fails or the process is interrupted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import Document
|
||||
|
||||
SOURCE = "mig0027"
|
||||
CONTENT = "content here"
|
||||
CONTENT_HASH = hashlib.sha256(CONTENT.encode()).hexdigest()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run); teardown
|
||||
upgrades to head no matter what happened, so the dev DB is never
|
||||
left below head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
|
||||
# on ``documents``, which would deadlock the repair's
|
||||
# ``ALTER TABLE`` forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||
"""(data_type, is_nullable, column_default, character_maximum_length)
|
||||
for one documents column."""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT data_type, is_nullable, column_default, character_maximum_length"
|
||||
" FROM information_schema.columns"
|
||||
" WHERE table_name = 'documents' AND column_name = :c"
|
||||
),
|
||||
{"c": column},
|
||||
).fetchone()
|
||||
return tuple(row) if row is not None else None
|
||||
|
||||
|
||||
def _index_covering(db: Session, column: str) -> str | None:
|
||||
"""The name of any documents index whose definition mentions the
|
||||
column (None = UNINDEXED)."""
|
||||
return db.execute(
|
||||
text(
|
||||
"SELECT indexname FROM pg_indexes"
|
||||
" WHERE tablename = 'documents' AND indexdef ILIKE :like"
|
||||
),
|
||||
{"like": f"%{column}%"},
|
||||
).scalar()
|
||||
|
||||
|
||||
def _insert_sql(db: Session, path: str, page_count: int | None = None) -> uuid.UUID:
|
||||
"""Insert one documents row with the PRE-0027 column set (the 0026
|
||||
shape — the mode column, when present, is omitted so a NULL backfill
|
||||
is what the row reads; ``page_count`` is 0026's own column and may be
|
||||
set to model a pre-phase strip)."""
|
||||
doc_id = uuid.uuid4()
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO documents (id, source, path, full_path, title, content,"
|
||||
" content_hash, page_count)"
|
||||
" VALUES (:id, :s, :p, :fp, :t, :c, :h, :pc)"
|
||||
),
|
||||
{
|
||||
"id": doc_id,
|
||||
"s": SOURCE,
|
||||
"p": path,
|
||||
"fp": f"/tmp/{path}",
|
||||
"t": f"Doc {path}",
|
||||
"c": CONTENT,
|
||||
"h": CONTENT_HASH,
|
||||
"pc": page_count,
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
return doc_id
|
||||
|
||||
|
||||
def _delete(db: Session, doc_id: uuid.UUID) -> None:
|
||||
db.execute(text("DELETE FROM documents WHERE id = :id"), {"id": doc_id})
|
||||
db.commit()
|
||||
|
||||
|
||||
def _assert_0026_schema_survives(db: Session) -> None:
|
||||
"""The 0026 documents contract must survive the additive upgrade —
|
||||
``page_count`` keeps its INTEGER / NULLABLE / NO-default / UNINDEXED
|
||||
shape (the sibling column this one rides) and ``content_hash`` stays
|
||||
NOT NULL and indexed (the retriever's identity)."""
|
||||
page_count = _column(db, "page_count")
|
||||
assert page_count is not None, "documents.page_count (0026) must survive the upgrade"
|
||||
assert page_count[0] == "integer", "page_count must stay INTEGER"
|
||||
assert page_count[1] == "YES", "page_count must stay NULLABLE"
|
||||
assert page_count[2] is None, "page_count must keep NO server default"
|
||||
assert _index_covering(db, "page_count") is None, "page_count must stay UNINDEXED"
|
||||
|
||||
content = _column(db, "content_hash")
|
||||
assert content is not None and content[0] == "character varying" and content[1] == "NO", (
|
||||
"documents.content_hash (0001) must keep its contract"
|
||||
)
|
||||
assert (
|
||||
_index_covering(db, "content_hash") is not None
|
||||
), "the ix_documents_content_hash index must survive the upgrade"
|
||||
|
||||
indexed = _column(db, "indexed_at")
|
||||
assert indexed is not None and indexed[0] == "timestamp with time zone", (
|
||||
"documents.indexed_at (0001) must survive the upgrade"
|
||||
)
|
||||
is_image = _column(db, "is_image")
|
||||
assert is_image is not None and is_image[0] == "boolean" and is_image[1] == "NO", (
|
||||
"documents.is_image (0022) must survive the upgrade"
|
||||
)
|
||||
|
||||
|
||||
def test_upgrade_to_0027_adds_page_strip_mode(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0026 → 0027: ``page_strip_mode`` exists with the full
|
||||
contract (VARCHAR(16), NULLABLE, NO server default — NULL = "no
|
||||
strip" — and UNINDEXED: a display fact, never a query surface), is
|
||||
ABSENT at 0026, and every pre-0027 row backfills it to NULL —
|
||||
including the row whose ``page_count`` IS set (the pre-phase strip
|
||||
with no stamped class), while the 0026 table contract (the
|
||||
``page_count`` sibling, the indexed ``content_hash``) survives.
|
||||
"""
|
||||
command.downgrade(alembic, "0026") # start from the pre-0027 state
|
||||
assert _version(db) == "0026"
|
||||
assert _column(db, "page_strip_mode") is None, "page_strip_mode must be absent at 0026"
|
||||
|
||||
plain_id = _insert_sql(db, "plain.pdf") # the 0026 column set, no strip
|
||||
pre_strip_id = _insert_sql(db, "prephase.pdf", page_count=36) # a live pre-phase strip
|
||||
|
||||
try:
|
||||
command.upgrade(alembic, "0027")
|
||||
assert _version(db) == "0027", "alembic_version must be at 0027"
|
||||
|
||||
col = _column(db, "page_strip_mode")
|
||||
assert col is not None, "documents.page_strip_mode is missing"
|
||||
assert col[0] == "character varying", "page_strip_mode must be VARCHAR"
|
||||
assert col[3] == 16, "page_strip_mode must be VARCHAR(16) — the two class names"
|
||||
assert col[1] == "YES", (
|
||||
"page_strip_mode must be NULLABLE (NULL = no strip, and the pre-phase shape)"
|
||||
)
|
||||
assert col[2] is None, (
|
||||
"page_strip_mode must carry NO server default — NULL is the meaningful"
|
||||
" pre-phase state"
|
||||
)
|
||||
assert _index_covering(db, "page_strip_mode") is None, (
|
||||
"page_strip_mode must be UNINDEXED — a display fact, never a query"
|
||||
" surface (the content endpoint reads it from the row)"
|
||||
)
|
||||
|
||||
# Every pre-0027 row backfills to NULL — no backfill, no default.
|
||||
assert (
|
||||
db.execute(
|
||||
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
|
||||
{"id": plain_id},
|
||||
).fetchone()
|
||||
== (None, None)
|
||||
), "a pre-0027 row without a strip must read both columns NULL"
|
||||
assert (
|
||||
db.execute(
|
||||
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
|
||||
{"id": pre_strip_id},
|
||||
).fetchone()
|
||||
== (36, None)
|
||||
), (
|
||||
"a pre-phase strip (page_count set) must backfill page_strip_mode to"
|
||||
" NULL — the wire omits the key and the viewer keeps today's DOM"
|
||||
)
|
||||
|
||||
_assert_0026_schema_survives(db)
|
||||
finally:
|
||||
_delete(db, plain_id)
|
||||
_delete(db, pre_strip_id)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["primary", "additive"])
|
||||
def test_orm_page_strip_mode_round_trips(db: Session, alembic: Config, mode: str) -> None:
|
||||
"""The ORM contract agrees with the column contract: each class
|
||||
(``"primary"`` = the fallback class, ``"additive"`` = the pictorial
|
||||
class) round-trips VERBATIM through a fresh session, and a row
|
||||
created with no mode reads ``page_strip_mode is None`` (the
|
||||
pre-phase shape — NULL until the importer stamps it)."""
|
||||
command.upgrade(alembic, "head")
|
||||
stamped = Document(
|
||||
source=SOURCE,
|
||||
path="stamped.pdf",
|
||||
full_path="/tmp/stamped.pdf",
|
||||
title="stamped",
|
||||
content=CONTENT,
|
||||
content_hash=CONTENT_HASH,
|
||||
page_count=36,
|
||||
page_strip_mode=mode,
|
||||
)
|
||||
unstamped = Document(
|
||||
source=SOURCE,
|
||||
path="unstamped.md",
|
||||
full_path="/tmp/unstamped.md",
|
||||
title="unstamped",
|
||||
content=CONTENT,
|
||||
content_hash=CONTENT_HASH,
|
||||
)
|
||||
db.add(stamped)
|
||||
db.add(unstamped)
|
||||
db.commit()
|
||||
try:
|
||||
with SessionLocal() as fresh:
|
||||
reloaded = fresh.get(Document, stamped.id)
|
||||
assert reloaded is not None, "the stamped row must be readable"
|
||||
assert reloaded.page_strip_mode == mode, (
|
||||
f"the {mode!r} class must round-trip through the DB verbatim"
|
||||
)
|
||||
assert reloaded.page_count == 36, "the sibling page_count must round-trip too"
|
||||
reloaded_unstamped = fresh.get(Document, unstamped.id)
|
||||
assert reloaded_unstamped is not None, "the unstamped row must be readable"
|
||||
assert reloaded_unstamped.page_strip_mode is None, (
|
||||
"a row created without a mode must read NULL (the pre-phase shape)"
|
||||
)
|
||||
assert reloaded_unstamped.content_hash == CONTENT_HASH, (
|
||||
"content_hash must be untouched by the page_strip_mode addition"
|
||||
)
|
||||
finally:
|
||||
_delete(db, stamped.id)
|
||||
_delete(db, unstamped.id)
|
||||
|
||||
|
||||
def test_downgrade_to_0026_drops_the_column(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade 0027 → 0026: the page_strip_mode column is gone (A13 —
|
||||
fully reversible, the class is re-derivable from the source file by
|
||||
the deterministic re-gate) while the row + its content + its 0026
|
||||
``page_count`` survive, and the rest of the table contract is intact."""
|
||||
command.upgrade(alembic, "head")
|
||||
row = Document(
|
||||
source=SOURCE,
|
||||
path="down.pdf",
|
||||
full_path="/tmp/down.pdf",
|
||||
title="down",
|
||||
content=CONTENT,
|
||||
content_hash=CONTENT_HASH,
|
||||
page_count=36,
|
||||
page_strip_mode="additive",
|
||||
)
|
||||
db.add(row)
|
||||
db.commit()
|
||||
try:
|
||||
command.downgrade(alembic, "0026")
|
||||
assert _version(db) == "0026"
|
||||
assert _column(db, "page_strip_mode") is None, "page_strip_mode must be dropped"
|
||||
|
||||
surviving = db.execute(
|
||||
text(
|
||||
"SELECT source, path, content, content_hash, page_count, is_image"
|
||||
" FROM documents WHERE id = :id"
|
||||
),
|
||||
{"id": row.id},
|
||||
).fetchone()
|
||||
assert surviving is not None and surviving[0] == SOURCE, (
|
||||
"the row must survive the column drop"
|
||||
)
|
||||
assert surviving[1] == "down.pdf" and surviving[2] == CONTENT, (
|
||||
"path + content must survive the drop"
|
||||
)
|
||||
assert surviving[3] == CONTENT_HASH, (
|
||||
"content_hash (the retriever's identity) must survive the drop"
|
||||
)
|
||||
assert surviving[4] == 36, (
|
||||
"page_count (the 0026 sibling) must survive the drop — the strip"
|
||||
" itself is untouched by this migration's reversal"
|
||||
)
|
||||
assert surviving[5] is False, "is_image must survive the drop"
|
||||
assert _index_covering(db, "content_hash") is not None, (
|
||||
"the content_hash index must survive the downgrade"
|
||||
)
|
||||
finally:
|
||||
_delete(db, row.id)
|
||||
# Repair: the fixture teardown re-upgrades to head.
|
||||
|
||||
|
||||
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0026, then upgrade back to 0027: ``page_strip_mode``
|
||||
is back with the full contract (VARCHAR(16), NULLABLE, NO server
|
||||
default, UNINDEXED), a row seeded at 0026 — strip included — reads
|
||||
the mode as NULL (the pre-phase shape on the round-trip), and the
|
||||
0026 schema (the ``page_count`` sibling, the indexed
|
||||
``content_hash``) is intact."""
|
||||
command.downgrade(alembic, "0026")
|
||||
pre_id = _insert_sql(db, "round.pdf", page_count=36)
|
||||
try:
|
||||
command.upgrade(alembic, "0027")
|
||||
assert _version(db) == "0027", "round-trip upgrade must land at 0027"
|
||||
|
||||
col = _column(db, "page_strip_mode")
|
||||
assert col is not None, "documents.page_strip_mode must be back"
|
||||
assert col[0] == "character varying", "page_strip_mode must be VARCHAR after the round-trip"
|
||||
assert col[3] == 16, "page_strip_mode must be VARCHAR(16) after the round-trip"
|
||||
assert col[1] == "YES", "page_strip_mode must be NULLABLE after the round-trip"
|
||||
assert col[2] is None, (
|
||||
"page_strip_mode must still carry NO server default after the round-trip"
|
||||
)
|
||||
assert _index_covering(db, "page_strip_mode") is None, (
|
||||
"page_strip_mode must stay UNINDEXED after the round-trip"
|
||||
)
|
||||
|
||||
assert (
|
||||
db.execute(
|
||||
text("SELECT page_count, page_strip_mode FROM documents WHERE id = :id"),
|
||||
{"id": pre_id},
|
||||
).fetchone()
|
||||
== (36, None)
|
||||
), (
|
||||
"a row seeded at 0026 must read page_strip_mode NULL after the"
|
||||
" round-trip while its page_count stands"
|
||||
)
|
||||
|
||||
_assert_0026_schema_survives(db)
|
||||
finally:
|
||||
_delete(db, pre_id)
|
||||
@@ -30,6 +30,17 @@ page-qualifying PDF row (``page_count`` set) adds ``page_urls`` = the
|
||||
exact 1-based ``/api/documents/<id>/pages/<i>`` list over
|
||||
``1..page_count``.
|
||||
|
||||
Phase 163 (task 01) — the content payload's STRIP CLASS signal
|
||||
(``page_strip_mode``: ``"primary"`` = the fallback class, the strip IS
|
||||
the view; ``"additive"`` = the pictorial class, text + a Text/Pages
|
||||
toggle): the model-level omission rule (ABSENT when ``None``, never
|
||||
``null``), and the endpoint (stubbed session) — the key rides
|
||||
``page_urls`` for a qualifying row, a PRE-PHASE row (``page_count`` set,
|
||||
mode NULL — the exact live ``Letumdei`` state) serializes it ABSENT while
|
||||
keeping ``page_urls``, and the whole strip block keys off
|
||||
``page_count``, so a row with a stray mode and no strip carries neither
|
||||
key (every non-qualifying payload byte-identical to pre-phase).
|
||||
|
||||
Phase 145 (task 03) — the content payload's git origin fields + the
|
||||
``renderDocument`` "Original" anchor (modal AND /document.html, one
|
||||
site):
|
||||
@@ -316,10 +327,11 @@ def _text_doc_content(
|
||||
origin_url: str | None = None,
|
||||
origin_branch: str | None = None,
|
||||
page_urls: list[str] | None = None,
|
||||
page_strip_mode: str | None = None,
|
||||
) -> DocContent:
|
||||
"""One text doc's content payload (the pre-phase shape) with the
|
||||
phase-145 origin fields + the phase-148 ``page_urls`` optionally
|
||||
set."""
|
||||
phase-145 origin fields + the phase-148 ``page_urls`` + the
|
||||
phase-163 ``page_strip_mode`` optionally set."""
|
||||
return DocContent(
|
||||
source="S",
|
||||
path="a.md",
|
||||
@@ -332,6 +344,7 @@ def _text_doc_content(
|
||||
origin_url=origin_url,
|
||||
origin_branch=origin_branch,
|
||||
page_urls=page_urls,
|
||||
page_strip_mode=page_strip_mode,
|
||||
)
|
||||
|
||||
|
||||
@@ -367,9 +380,29 @@ def test_doc_content_page_urls_absent_on_the_wire_when_unset() -> None:
|
||||
assert dumped["page_urls"] == urls # the exact 1-based route list
|
||||
|
||||
|
||||
def _page_pdf_doc() -> Document:
|
||||
def test_doc_content_page_strip_mode_absent_on_the_wire_when_unset() -> None:
|
||||
"""Phase 163 (task 01, LOCKED A2) — the model-level omission rule
|
||||
(the ``image_url`` / ``page_urls`` precedent): no strip class
|
||||
(``page_strip_mode`` ``None`` — every non-qualifying doc AND every
|
||||
pre-phase row) serializes the key ABSENT — never ``null`` — so those
|
||||
payloads are byte-identical to pre-phase. A stamped row carries the
|
||||
class verbatim, beside ``page_urls``."""
|
||||
dumped = _text_doc_content().model_dump()
|
||||
assert "page_strip_mode" not in dumped # absent — never null
|
||||
for mode in ("primary", "additive"):
|
||||
dumped = _text_doc_content(
|
||||
page_urls=["/api/documents/d/pages/1"], page_strip_mode=mode
|
||||
).model_dump()
|
||||
assert dumped["page_strip_mode"] == mode # the class verbatim
|
||||
|
||||
|
||||
def _page_pdf_doc(
|
||||
page_count: int | None = 2, page_strip_mode: str | None = None
|
||||
) -> Document:
|
||||
"""A page-QUALIFYING PDF row (phase 148): ``page_count`` set — the
|
||||
endpoint derives ``page_urls`` from it (the exact 1-based list)."""
|
||||
endpoint derives ``page_urls`` from it (the exact 1-based list).
|
||||
``page_strip_mode`` (phase 163) is the strip CLASS; ``None`` is the
|
||||
PRE-PHASE row shape (the column NULL while the strip stands)."""
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source="Homelab",
|
||||
@@ -379,7 +412,8 @@ def _page_pdf_doc() -> Document:
|
||||
content="[Page 1 illustration: the parts tray]…",
|
||||
content_hash="e" * 64,
|
||||
is_image=False, # phase 122: the stub session never applies defaults
|
||||
page_count=2,
|
||||
page_count=page_count,
|
||||
page_strip_mode=page_strip_mode,
|
||||
)
|
||||
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
|
||||
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC)
|
||||
@@ -410,6 +444,79 @@ database): a page-qualifying PDF row (``page_count`` set) carries
|
||||
] # the exact 1-based list, count == page_count
|
||||
assert body["is_image"] is False
|
||||
assert "image_url" not in body # absent — never null (not an image doc)
|
||||
# Phase 163 (task 01): the pre-phase row (mode NULL) keeps the strip
|
||||
# and serializes the class ABSENT — the viewer treats that as
|
||||
# "primary", i.e. today's DOM.
|
||||
assert "page_strip_mode" not in body
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["primary", "additive"])
|
||||
def test_content_page_strip_mode_carried_for_a_stamped_strip(mode: str) -> None:
|
||||
"""Phase 163 (task 01, LOCKED A2/A3) — the endpoint (stubbed
|
||||
session, no
|
||||
database): a page-qualifying PDF whose row carries the class
|
||||
(``"primary"`` = the fallback class, ``"additive"`` = the pictorial
|
||||
class) carries ``page_strip_mode`` VERBATIM next to ``page_urls`` —
|
||||
the signal the viewer branches on so an additive strip never hides
|
||||
the text (task 04). Nothing else in the payload moves."""
|
||||
doc = _page_pdf_doc(page_strip_mode=mode)
|
||||
with _client_with_row((doc, 4)) as client:
|
||||
r = client.get(
|
||||
"/api/documents/content",
|
||||
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["page_strip_mode"] == mode # the class verbatim
|
||||
assert body["page_urls"] == [
|
||||
f"/api/documents/{doc.id}/pages/1",
|
||||
f"/api/documents/{doc.id}/pages/2",
|
||||
] # the strip rides along, unchanged
|
||||
|
||||
|
||||
def test_content_strip_block_keys_off_page_count_never_mode_alone() -> None:
|
||||
"""Phase 163 (task 01) — the class never appears alone: the whole
|
||||
strip block keys off ``page_count``, so a row whose column somehow
|
||||
holds a mode while ``page_count`` is NULL serializes NEITHER
|
||||
``page_strip_mode`` nor ``page_urls`` — the payload is byte-identical
|
||||
to a plain non-qualifying document (there is no strip to show, and
|
||||
the viewer must never be told a class for pages that do not exist)."""
|
||||
doc = _page_pdf_doc(page_count=None, page_strip_mode="additive")
|
||||
with _client_with_row((doc, 1)) as client:
|
||||
r = client.get(
|
||||
"/api/documents/content",
|
||||
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert "page_strip_mode" not in body # absent — the strip block is off
|
||||
assert "page_urls" not in body # absent — never null
|
||||
|
||||
|
||||
def test_content_non_qualifying_payload_key_set_is_byte_identical() -> None:
|
||||
"""Phase 163 (task 01) — the byte-identity pin: a non-qualifying PDF
|
||||
row (``page_count`` NULL, mode NULL) carries EXACTLY the pre-phase
|
||||
key set — neither strip key joins it (``is_image`` always present,
|
||||
``image_url`` / the origin keys / both strip keys absent)."""
|
||||
doc = _page_pdf_doc(page_count=None)
|
||||
with _client_with_row((doc, 1)) as client:
|
||||
r = client.get(
|
||||
"/api/documents/content",
|
||||
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
assert set(r.json()) == {
|
||||
"source",
|
||||
"path",
|
||||
"title",
|
||||
"format",
|
||||
"summary",
|
||||
"created_at",
|
||||
"content",
|
||||
"indexed_at",
|
||||
"chunks",
|
||||
"is_image",
|
||||
}
|
||||
|
||||
|
||||
def test_content_origin_url_carried_for_a_matching_git_row() -> None:
|
||||
|
||||
@@ -31,6 +31,16 @@ the second sync):
|
||||
* the streamed hash is transparent (a >1 MiB file hashes identically
|
||||
to ``hashlib.sha256(bytes).hexdigest()``).
|
||||
|
||||
Phase 163 (task 03) extends this file with the PDF strip RE-GATE the
|
||||
unchanged paths now run (``_regate_pdf_strip``) — the pre-phase "the
|
||||
copies persist" assumption is what broke in production (a row claiming
|
||||
``page_count = 36`` over ZERO files, every page 404ing forever): the
|
||||
re-gate re-derives the strip from the current bytes + settings (ZERO
|
||||
vision/LLM calls), heals missing copies, re-stamps a changed class,
|
||||
clears a strip that no longer qualifies (kill switch included), stays
|
||||
fail-soft, logs only when it ACTS, and costs a non-strip file zero PDF
|
||||
parsing (spy-pinned).
|
||||
|
||||
The E2E surface (the idempotent second sync through the real UI +
|
||||
mock LLM) is pinned in ``tests/e2e/test_sync_changed_only.py``
|
||||
(task 03).
|
||||
@@ -43,6 +53,7 @@ import logging
|
||||
import os
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pymupdf as fitz
|
||||
import pytest
|
||||
@@ -50,6 +61,7 @@ from sqlalchemy import select
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag import importer
|
||||
from app.rag.importer import _file_hash, import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
from tests.unit.test_importer import (
|
||||
@@ -60,6 +72,7 @@ from tests.unit.test_importer import (
|
||||
_VisionLLM,
|
||||
)
|
||||
from tests.unit.test_pictorial_import import (
|
||||
_build_dense_pdf,
|
||||
_build_ikea_like_pdf,
|
||||
_PictorialLLM,
|
||||
_settings,
|
||||
@@ -454,6 +467,520 @@ def test_pre_phase_null_file_hash_image_runs_branch_and_stamps(
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 163 (task 03): the deterministic PDF strip re-gate ----------
|
||||
#
|
||||
# The live defect (owner report, 2026-10-07): the `Letumdei` row carried
|
||||
# `page_count = 36` over ZERO files on disk (the pre-phase-163 persist
|
||||
# deleted the old copies BEFORE rendering, outside the DB transaction —
|
||||
# a later per-file rollback restored the stale count over an empty
|
||||
# directory; an external wipe of `pdf_page_dir` lands the same state).
|
||||
# The phase-147 fast path believed "the copies persist" and never
|
||||
# re-rendered, so the strip 404'd forever. `_regate_pdf_strip` now runs
|
||||
# at the tail of `_unchanged_doc`: the changed-path gate mirrored from
|
||||
# the current bytes + settings (ZERO vision/LLM calls), missing copies
|
||||
# healed, a stale strip cleared.
|
||||
|
||||
#: The PNG signature (a healed copy is a real render, not a placeholder).
|
||||
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
|
||||
|
||||
|
||||
def _regate_settings(tmp_path: Path, **kwargs: Any) -> Settings:
|
||||
"""Settings for the re-gate matrix: the ``pdf`` token admitted, the
|
||||
phase-140 knobs at their CODE DEFAULTS (``pdf_pictorial`` ON), and
|
||||
the strip's home ``pdf_page_dir`` pinned under *tmp_path* (never the
|
||||
real home)."""
|
||||
kwargs.setdefault("_env_file", None)
|
||||
kwargs.setdefault("import_extensions", "md,txt,pdf")
|
||||
kwargs.setdefault("images", False)
|
||||
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||
kwargs.setdefault("pdf_page_dir", str(tmp_path / "pages"))
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _pages_dir(tmp_path: Path) -> Path:
|
||||
return tmp_path / "pages"
|
||||
|
||||
|
||||
def _strip_copies(doc: Document, page_dir: Path, count: int) -> list[Path]:
|
||||
return [page_dir / f"{doc.id}-p{n}.png" for n in range(1, count + 1)]
|
||||
|
||||
|
||||
def test_regate_heals_deleted_copies_of_a_pictorial_strip(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""THE live-loss reproduction (heal arm): a pictorial-class PDF is
|
||||
indexed (``page_count`` set, copies written), its copies are deleted
|
||||
out from under the row (the exact live state — mode NULL on a
|
||||
pre-phase row), and an UNCHANGED re-sync restores every file with
|
||||
real PNG bytes, stamps the mode ``additive``, keeps ``page_count``
|
||||
(no drift), burns ZERO llm/vision calls, and logs the one greppable
|
||||
INFO line. The NULL-mode stamp is SILENT (no ``mode`` segment — it
|
||||
rides the line only when work followed)."""
|
||||
root = tmp_path / "rgheal"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgheal", "ikea.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
doc.page_strip_mode = None # the pre-phase row (the live shape)
|
||||
db.commit()
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
assert not list(page_dir.glob(f"{doc.id}-*.png"))
|
||||
embeds1, chats1, vision1 = len(llm.calls), len(llm.chat_calls), llm.pictorial_calls
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
# ZERO llm work of any kind (the re-gate is deterministic, A5).
|
||||
assert len(llm.calls) == embeds1, "the re-gate never re-embeds"
|
||||
assert len(llm.chat_calls) == chats1, "the re-gate never chats"
|
||||
assert llm.pictorial_calls == vision1, "the re-gate issues ZERO vision calls"
|
||||
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgheal", "ikea.pdf")
|
||||
assert doc2.page_count == 10, "no drift — the count is untouched"
|
||||
assert doc2.page_strip_mode == "additive", "the mode is stamped"
|
||||
for path in _strip_copies(doc2, page_dir, 10):
|
||||
assert path.is_file(), f"copy {path.name} was not healed"
|
||||
assert path.read_bytes()[:8] == PNG_MAGIC, f"{path.name} is not a PNG"
|
||||
assert not list(page_dir.glob("*.tmp")), "no .tmp orphan from the heal"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert (
|
||||
"import: PDF strip re-gate source=rgheal path=ikea.pdf — "
|
||||
"healed 10 pages"
|
||||
) in messages, messages
|
||||
assert not any("mode " in m for m in messages if "re-gate" in m), (
|
||||
"a NULL->stamp mode re-stamp is silent-beneficial " + str(messages)
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "rgheal")
|
||||
|
||||
|
||||
def test_regate_heals_a_fallback_strip_as_primary(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Heal arm, the FALLBACK class (phase 133's unusable text layer):
|
||||
the copies of a garbage-layer PDF's strip are restored on an
|
||||
unchanged re-sync and the mode lands ``primary`` (the strip IS the
|
||||
view) — and the phase-133 fallback vision does NOT re-fire (the
|
||||
class is re-derived from the bytes, no vision call)."""
|
||||
root = tmp_path / "rgfall"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "garbage.pdf") # 2 pages
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgfall", "garbage.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (2, "primary")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 2):
|
||||
path.unlink()
|
||||
vision1, chats1, embeds1 = llm.vision_calls, len(llm.chat_calls), len(llm.calls)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert second.pdfs_screenshotted == 0, "the fallback gate did not re-run"
|
||||
assert llm.vision_calls == vision1, "ZERO vision calls on the re-gate"
|
||||
assert len(llm.chat_calls) == chats1 and len(llm.calls) == embeds1
|
||||
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgfall", "garbage.pdf")
|
||||
assert doc2.page_strip_mode == "primary", "the unusable layer is primary"
|
||||
assert doc2.page_count == 2
|
||||
for path in _strip_copies(doc2, page_dir, 2):
|
||||
assert path.is_file() and path.read_bytes()[:8] == PNG_MAGIC
|
||||
assert any(
|
||||
"healed 2 pages" in m and "re-gate" in m for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgfall")
|
||||
|
||||
|
||||
def test_regate_noop_when_the_strip_is_healthy(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""No-op arm (the IKEA-class doc whose copies all stand): an
|
||||
unchanged re-sync writes NOTHING (every copy's mtime and bytes are
|
||||
untouched), renders NOTHING, and logs NO re-gate line — the INFO is
|
||||
never per-unchanged-doc — while the pre-existing ``import:
|
||||
unchanged`` line still fires."""
|
||||
root = tmp_path / "rgnoop"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgnoop", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
before = {
|
||||
p: (p.stat().st_mtime_ns, p.read_bytes())
|
||||
for p in _strip_copies(doc, page_dir, 10)
|
||||
}
|
||||
assert len(before) == 10
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgnoop", "ikea.pdf")
|
||||
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
|
||||
after = {
|
||||
p: (p.stat().st_mtime_ns, p.read_bytes())
|
||||
for p in _strip_copies(doc2, page_dir, 10)
|
||||
}
|
||||
assert after == before, "a healthy strip is not written at all"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any(m.startswith("import: unchanged ") for m in messages), messages
|
||||
assert not any("re-gate" in m for m in messages), (
|
||||
f"the re-gate logged without acting: {messages}"
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "rgnoop")
|
||||
|
||||
|
||||
def test_regate_restamps_a_changed_mode(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Mode re-stamp (the class flip): a row whose stamped mode no
|
||||
longer matches the gate's class (here a pictorial-class row carrying
|
||||
``primary`` — a legacy/mis-stamped row) is re-stamped to
|
||||
``additive``, and because the mode was ALREADY SET the change is
|
||||
reported (``mode primary->additive``) — the only thing the re-gate
|
||||
did, so that is the whole INFO line (no copies were missing, none
|
||||
were written)."""
|
||||
root = tmp_path / "rgmode"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgmode", "ikea.pdf")
|
||||
assert doc.page_strip_mode == "additive"
|
||||
doc.page_strip_mode = "primary" # the mis-stamp the re-gate fixes
|
||||
db.commit()
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
before = {
|
||||
p: p.stat().st_mtime_ns for p in _strip_copies(doc, page_dir, 10)
|
||||
}
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.unchanged) == (0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgmode", "ikea.pdf")
|
||||
assert doc2.page_strip_mode == "additive", "the class is re-stamped"
|
||||
assert doc2.page_count == 10
|
||||
assert {
|
||||
p: p.stat().st_mtime_ns for p in _strip_copies(doc2, page_dir, 10)
|
||||
} == before, "a mode fix writes no files"
|
||||
assert (
|
||||
"import: PDF strip re-gate source=rgmode path=ikea.pdf — "
|
||||
"mode primary->additive"
|
||||
) in [r.getMessage() for r in caplog.records], caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgmode")
|
||||
|
||||
|
||||
def test_regate_clears_a_strip_that_no_longer_qualifies(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""REPAIR arm: a row claims a strip (``page_count`` set, copies on
|
||||
disk) on a PDF whose GATE now says ineligible (a clean, dense text
|
||||
layer — zero sparse pages). The unchanged re-sync clears BOTH
|
||||
columns and deletes the copies (the strip must not linger over a
|
||||
document the viewer shows as text) + the ``cleared stale strip``
|
||||
INFO line, with zero llm calls."""
|
||||
root = tmp_path / "rgclear"
|
||||
root.mkdir()
|
||||
pdf = root / "clean.pdf"
|
||||
_build_dense_pdf(pdf) # usable layer, zero candidates — never qualifies
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgclear", "clean.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (None, None)
|
||||
# The stale state the re-gate must repair (a strip stamped by an
|
||||
# older gate/settings, copies still on disk).
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
page_dir.mkdir(parents=True, exist_ok=True)
|
||||
for path in _strip_copies(doc, page_dir, 2):
|
||||
path.write_bytes(b"STALE-COPY")
|
||||
doc.page_count = 2
|
||||
doc.page_strip_mode = "additive"
|
||||
db.commit()
|
||||
embeds1, chats1 = len(llm.calls), len(llm.chat_calls)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert len(llm.calls) == embeds1 and len(llm.chat_calls) == chats1
|
||||
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgclear", "clean.pdf")
|
||||
assert doc2.page_count is None, "the stale strip is cleared"
|
||||
assert doc2.page_strip_mode is None, "the class is cleared with it"
|
||||
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies are deleted"
|
||||
assert (
|
||||
"import: PDF strip re-gate source=rgclear path=clean.pdf — "
|
||||
"cleared stale strip"
|
||||
) in [r.getMessage() for r in caplog.records], caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgclear")
|
||||
|
||||
|
||||
def test_regate_kill_switch_clears_the_pictorial_strip(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""REPAIR arm via the kill switch (``pdf_pictorial=False``): the
|
||||
changed-path semantics mirrored — with the pass off a sparse-page
|
||||
PDF is NOT eligible, so its existing strip is cleared with its
|
||||
copies on the next (unchanged) sync. The bytes never moved: the
|
||||
setting alone decided."""
|
||||
root = tmp_path / "rgkill"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgkill", "ikea.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
assert len(list(page_dir.glob(f"{doc.id}-p*.png"))) == 10
|
||||
vision1 = llm.pictorial_calls
|
||||
|
||||
llm.settings = _regate_settings(tmp_path, pdf_pictorial=False)
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert llm.pictorial_calls == vision1, "ZERO vision calls (the pass is off)"
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgkill", "ikea.pdf")
|
||||
assert doc2.page_count is None and doc2.page_strip_mode is None
|
||||
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies went too"
|
||||
assert any(
|
||||
"cleared stale strip" in m for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgkill")
|
||||
|
||||
|
||||
def test_regate_render_exception_is_fail_soft(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Fail-soft: the heal's render RAISES — the sync completes
|
||||
(``unchanged``, zero errors), the row keeps its prior state
|
||||
(``page_count`` + mode untouched — the re-gate never clears a strip
|
||||
on a failure), no copies are touched, and the WARNING is the only
|
||||
signal."""
|
||||
root = tmp_path / "rgboom"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgboom", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
|
||||
def boom(*args: object, **kwargs: object) -> list[bytes]:
|
||||
raise RuntimeError("simulated render failure (test sentinel)")
|
||||
|
||||
monkeypatch.setattr(importer, "render_pdf_pages", boom)
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged, second.errors) == (
|
||||
0, 0, 1, 0,
|
||||
)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgboom", "ikea.pdf")
|
||||
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive"), (
|
||||
"the row keeps its pre-re-gate state"
|
||||
)
|
||||
assert not list(page_dir.glob(f"{doc.id}-p*.png")), "no copies were touched"
|
||||
assert any(
|
||||
"PDF strip re-gate failed" in m and "test sentinel" in m
|
||||
for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgboom")
|
||||
|
||||
|
||||
def test_regate_empty_render_keeps_the_row(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Fail-soft on the no-pages render (unopenable right after the fast
|
||||
path hashed the file — pathological): the row is kept EXACTLY as it
|
||||
was (never cleared on a transient), nothing is written, and the
|
||||
WARNING names the no-pages state."""
|
||||
root = tmp_path / "rgempty"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgempty", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
monkeypatch.setattr(importer, "render_pdf_pages", lambda *a, **k: [])
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.unchanged, second.errors) == (0, 1, 0)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgempty", "ikea.pdf")
|
||||
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
|
||||
assert not page_dir.exists() or not list(page_dir.glob(f"{doc.id}-*"))
|
||||
assert any(
|
||||
"re-render produced no pages (state kept)" in m
|
||||
for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgempty")
|
||||
|
||||
|
||||
def test_regate_reports_a_page_count_drift_on_the_heal(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Drift (pathological on unchanged bytes, so it is LOUD): the heal
|
||||
renders 6 pages for a row that claimed 10 — ``page_count`` follows
|
||||
the fresh render, the ``p7..p10`` tail beyond it is deleted, and a
|
||||
WARNING names both counts beside the ``healed`` INFO line."""
|
||||
root = tmp_path / "rgdrift"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgdrift", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
monkeypatch.setattr(
|
||||
importer,
|
||||
"render_pdf_pages",
|
||||
lambda *a, **k: [PNG_MAGIC + f"{i}".encode() for i in range(6)],
|
||||
)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.unchanged) == (0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgdrift", "ikea.pdf")
|
||||
assert doc2.page_count == 6, "the count follows the fresh render"
|
||||
for n in range(1, 7):
|
||||
assert (page_dir / f"{doc2.id}-p{n}.png").is_file()
|
||||
for n in range(7, 11):
|
||||
assert not (page_dir / f"{doc2.id}-p{n}.png").exists(), f"p{n} leaked"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any("page_count drift 10->6" in m for m in messages), messages
|
||||
assert any("healed 10 pages" in m for m in messages), messages
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
_cleanup_source(db, "rgdrift")
|
||||
|
||||
|
||||
def test_regate_fires_on_the_content_digest_hit_path(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The SECOND unchanged path (the PDF branch's content-digest hit —
|
||||
a pre-phase row with ``file_hash`` NULL): the branch re-runs its own
|
||||
gate (the byte-change's honest cost) and lands unchanged, and the
|
||||
shared helper's tail still heals the strip deleted out from under
|
||||
the row — the re-gate is not fast-path-only."""
|
||||
root = tmp_path / "rgbranch"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgbranch", "ikea.pdf")
|
||||
doc.file_hash = None # the pre-phase row: the fast path cannot fire
|
||||
db.commit()
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgbranch", "ikea.pdf")
|
||||
assert doc2.file_hash is not None, "the helper stamped the hash"
|
||||
for path in _strip_copies(doc2, page_dir, 10):
|
||||
assert path.is_file(), f"copy {path.name} was not healed"
|
||||
assert any("healed 10 pages" in m for m in caplog.messages), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgbranch")
|
||||
|
||||
|
||||
def test_regate_guard_costs_a_non_strip_file_nothing(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The GUARD (A5): a non-PDF unchanged file and a PDF whose
|
||||
``page_count`` is NULL reach the tail of ``_unchanged_doc`` and
|
||||
``extract_pdf_text`` is NEVER called for either — every non-strip
|
||||
file's fast path keeps its zero-extra-read contract (one sync of
|
||||
each class, then the spy covers the second sync only)."""
|
||||
root = tmp_path / "rgguard"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nplain text body\n")
|
||||
_build_dense_pdf(root / "clean.pdf") # usable layer, no strip
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
seen: list[str] = []
|
||||
real_extract = importer.extract_pdf_text
|
||||
|
||||
def spy(path: Path) -> str:
|
||||
seen.append(str(path))
|
||||
return real_extract(path)
|
||||
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
monkeypatch.setattr(importer, "extract_pdf_text", spy)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 2)
|
||||
assert seen == [], f"the guard let a PDF parse through: {seen}"
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
_cleanup_source(db, "rgguard")
|
||||
|
||||
|
||||
# ---------- the preserved semantics on the fast path --------------------------
|
||||
|
||||
|
||||
|
||||
@@ -293,6 +293,82 @@ def test_git_source_python_default_include_hidden_false() -> None:
|
||||
assert col.default.arg is False, "the default must resolve to False"
|
||||
|
||||
|
||||
def test_documents_page_strip_mode_column_contract() -> None:
|
||||
"""Phase 163 (task 01): ``documents.page_strip_mode`` — the CLASS of
|
||||
the page strip a non-NULL ``page_count`` describes (``"primary"`` =
|
||||
the phase-133 fallback class, the strip IS the view; ``"additive"``
|
||||
= the phase-140 pictorial class, text + a Text/Pages toggle):
|
||||
VARCHAR(16), NULLABLE (NULL = no strip — and the pre-phase shape for
|
||||
a row whose strip stands), NO server default, and NO index (the
|
||||
content endpoint reads it from the ROW — a display fact, never a
|
||||
query surface, the ``file_hash`` / ``page_count`` convention).
|
||||
``page_count`` (0026) stays exactly as phase 148 left it."""
|
||||
from sqlalchemy import String as SAString
|
||||
|
||||
documents = Base.metadata.tables["documents"]
|
||||
assert "page_strip_mode" in documents.c, (
|
||||
"documents must have the page_strip_mode column (phase 163)"
|
||||
)
|
||||
col = documents.c["page_strip_mode"]
|
||||
assert col.nullable is True, (
|
||||
"page_strip_mode must be NULLABLE — NULL means no strip (and the"
|
||||
" pre-phase row whose strip stands)"
|
||||
)
|
||||
assert isinstance(col.type, SAString), "page_strip_mode must be a String column"
|
||||
assert col.type.length == 16, (
|
||||
"page_strip_mode must be String(16) — the two class names, bounded"
|
||||
)
|
||||
assert col.server_default is None, (
|
||||
"page_strip_mode must carry NO server default — NULL is the"
|
||||
" meaningful pre-phase state"
|
||||
)
|
||||
assert not col.index, (
|
||||
"page_strip_mode must be UNINDEXED — the content endpoint reads it"
|
||||
" from the row, no query filters on it"
|
||||
)
|
||||
covering = [
|
||||
idx
|
||||
for idx in documents.indexes
|
||||
if any(c.name == "page_strip_mode" for c in idx.columns)
|
||||
]
|
||||
assert not covering, "no table-level index may cover page_strip_mode"
|
||||
|
||||
# page_count is untouched: still the nullable, unindexed sibling.
|
||||
page_count = documents.c["page_count"]
|
||||
assert page_count.nullable is True, "page_count must stay NULLABLE (0026)"
|
||||
assert not page_count.index, "page_count must stay UNINDEXED (0026)"
|
||||
|
||||
|
||||
def test_document_page_strip_mode_orm_defaults_to_none() -> None:
|
||||
"""A freshly constructed ``Document`` (no DB) exposes
|
||||
``page_strip_mode``: an omitted kwarg and an explicit ``None`` both
|
||||
read NULL (no strip — the shape every non-qualifying document keeps,
|
||||
and the shape a pre-phase-163 row keeps WHILE its ``page_count``
|
||||
stands, which the wire serializes ABSENT), and each class name
|
||||
round-trips verbatim as a plain string (the real DB round-trip is
|
||||
pinned in ``test_migration_0027.py``)."""
|
||||
from app.models import Document
|
||||
|
||||
base = dict(
|
||||
source="s",
|
||||
path="p.pdf",
|
||||
full_path="/tmp/s/p.pdf",
|
||||
title="p",
|
||||
content="c",
|
||||
content_hash="1" * 64,
|
||||
page_count=36,
|
||||
)
|
||||
for mode in ("primary", "additive"):
|
||||
assert Document(**base, page_strip_mode=mode).page_strip_mode == mode
|
||||
|
||||
assert Document(**base, page_strip_mode=None).page_strip_mode is None
|
||||
omitted = Document(**base)
|
||||
assert omitted.page_strip_mode is None, (
|
||||
"an omitted page_strip_mode must read NULL while page_count stands"
|
||||
" (the pre-phase strip the viewer keeps showing as 'primary')"
|
||||
)
|
||||
|
||||
|
||||
def test_doc_drafts_column_contract() -> None:
|
||||
"""Phase 59: the editable triple (title/path/body) + status +
|
||||
timestamps are NOT NULL; ``branch`` / ``commit_sha`` are NULL
|
||||
|
||||
@@ -21,14 +21,20 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
|
||||
layer, ≥1 sparse page) and the fallback class (unusable layer) get
|
||||
``<doc-id>-p{N}.png`` for EVERY page (not just the described
|
||||
candidates — the strip shows the whole manual) + ``page_count`` =
|
||||
the page count + real PNG bytes; a text-heavy PDF gets neither
|
||||
(no copies, NULL count, ZERO render calls, byte-identical content);
|
||||
the page count + real PNG bytes + the phase-163 CLASS stamp
|
||||
(``page_strip_mode`` ``additive`` / ``primary``); a text-heavy PDF
|
||||
gets neither (no copies, NULL count, NULL mode, ZERO render calls,
|
||||
byte-identical content);
|
||||
* the phase-147 fast path — an unchanged qualifying PDF's second sync
|
||||
re-renders NOTHING (zero render calls — the copies persist,
|
||||
``page_count`` and the bytes untouched);
|
||||
* stale replacement — a changed (shrunk) PDF's old copy set is
|
||||
replaced: the new set covers the new pages, the old ``p{N}`` tail
|
||||
is GONE (no orphans), ``page_count`` follows the new count;
|
||||
* WRITE-FIRST replacement (phase 163, task 02, LOCKED A4) — a changed
|
||||
(shrunk) PDF's new copies land atomically (tmp + ``os.replace``,
|
||||
pinned) BEFORE any old copy is removed; the old ``p{N}`` tail is
|
||||
GONE only after the replacement bytes are durable (no orphans),
|
||||
``page_count`` follows the new count; a MID-WRITE failure leaves the
|
||||
prior copy set intact (the rollback-orphan state — the row over
|
||||
zero files — is structurally unrepresentable);
|
||||
* qualifier loss — a qualifying PDF whose new version gained a dense
|
||||
text layer loses its strip: ``page_count`` NULL + copies deleted;
|
||||
* unopenable — a render that yields no pages (``render_pdf_pages`` →
|
||||
@@ -40,9 +46,10 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
|
||||
the FULL count (the gap is a display-level 404 — task 02 pins the
|
||||
route's side);
|
||||
* fail-soft — the render raising (warning is the signal, ``page_count``
|
||||
NULL, no copies, the doc indexed fully — chunks + summary) and the
|
||||
WRITER raising (the copies this call wrote are removed — no partial
|
||||
orphans, ``page_count`` NULL, the doc indexed fully);
|
||||
+ ``page_strip_mode`` NULL, no copies, the doc indexed fully — chunks
|
||||
+ summary) and the WRITER raising (this call's ``.tmp`` files and the
|
||||
finals it CREATED are removed — no partial orphans, ``page_count`` +
|
||||
``page_strip_mode`` NULL, the doc indexed fully);
|
||||
* prune — a qualifying doc's file removed from the source: the row
|
||||
goes and its page copies go with it (no orphans in the tmp dir).
|
||||
|
||||
@@ -54,6 +61,7 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from collections.abc import Callable, Sequence
|
||||
from pathlib import Path
|
||||
@@ -161,6 +169,35 @@ def _render_spy(
|
||||
return calls
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 163 (task 01) — the strip-class column beside page_count
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pictorial_index_stamps_the_additive_class(db, tmp_path: Path) -> None:
|
||||
"""Phase 163 (task 02, re-cut of the task-01 pin) — the STAMP: a
|
||||
qualifying PDF indexed through the PICTORIAL class gets
|
||||
``page_strip_mode == "additive"`` beside ``page_count`` (the text +
|
||||
toggle view); the strip's copies themselves are unchanged by the
|
||||
phase (task 01's shape, mode stamped exactly with the count).
|
||||
"""
|
||||
root = tmp_path / "pgm"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
doc = _doc(db, "pgm", "ikea.pdf")
|
||||
assert doc.page_count == 10, "the strip itself is unchanged by task 02"
|
||||
assert doc.page_strip_mode == "additive", (
|
||||
"the pictorial class stamps the additive mode (LOCKED A1/A3)"
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "pgm")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Eligibility × new index
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -219,6 +256,10 @@ def test_fallback_class_garbage_pdf_writes_every_page_copy(db, tmp_path: Path) -
|
||||
assert summary.pdfs_screenshotted == 1 # the fallback fired (the class)
|
||||
doc = _doc(db, "pgf", "garbage.pdf")
|
||||
assert doc.page_count == 2
|
||||
assert doc.page_strip_mode == "primary", (
|
||||
"the fallback class stamps the primary mode — the strip IS the "
|
||||
"view, byte-identical to phase 148 otherwise (LOCKED A1)"
|
||||
)
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for n in (1, 2):
|
||||
path = page_dir / f"{doc.id}-p{n}.png"
|
||||
@@ -253,6 +294,7 @@ def test_text_heavy_pdf_keeps_the_text_view(
|
||||
assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops
|
||||
doc = _doc(db, "pgd", "clean.pdf")
|
||||
assert doc.page_count is None, "a text-heavy PDF keeps the text view"
|
||||
assert doc.page_strip_mode is None, "no strip — no class either"
|
||||
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
||||
assert calls == [], "zero render calls (no fallback, zero candidates)"
|
||||
assert doc.content == expected, "byte-identical to the extraction"
|
||||
@@ -315,9 +357,10 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
|
||||
) -> None:
|
||||
"""A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index
|
||||
replaces the copy set — the new set covers p1..p6, the old
|
||||
version's ``p7..p10`` tail is GONE (the stale-replacement runs
|
||||
before the write), the doc keeps its id, and ``page_count``
|
||||
follows the new page count."""
|
||||
version's ``p7..p10`` tail is GONE (the stale-tail delete runs
|
||||
LAST, after the write-first loop — phase 163, task 02), the doc
|
||||
keeps its id, ``page_count`` follows the new page count, and
|
||||
``page_strip_mode`` is (re-)stamped."""
|
||||
root = tmp_path / "pgc"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
@@ -336,6 +379,7 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
|
||||
doc2 = _doc(db, "pgc", "ikea.pdf")
|
||||
assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)"
|
||||
assert doc2.page_count == 6, "page_count follows the new page count"
|
||||
assert doc2.page_strip_mode == "additive", "the class is (re-)stamped"
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for n in range(1, 7):
|
||||
assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing"
|
||||
@@ -379,6 +423,7 @@ def test_qualifier_loss_clears_page_count_and_copies(
|
||||
doc2 = _doc(db, "pgl", "ikea.pdf")
|
||||
assert doc2.id == doc1.id
|
||||
assert doc2.page_count is None, "a doc that stopped qualifying loses its strip"
|
||||
assert doc2.page_strip_mode is None, "the class is cleared with the count"
|
||||
assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), (
|
||||
"the copies must be deleted on qualifier loss"
|
||||
)
|
||||
@@ -411,6 +456,7 @@ def test_unopenable_pdf_page_render_is_fail_soft(
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
doc = _doc(db, "pgn", "garbage.pdf")
|
||||
assert doc.page_count is None, "no pages rendered — no strip"
|
||||
assert doc.page_strip_mode is None, "the class is cleared with the count"
|
||||
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
||||
assert doc.content == "", "the fallback's all-failed render lands empty content"
|
||||
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
||||
@@ -493,10 +539,11 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The PNG WRITER raises (a full disk) after the FIRST copy
|
||||
landed: the copies this call wrote are removed (no partial
|
||||
orphans), ``page_count`` NULL, the warning is the signal, and the
|
||||
document indexes exactly as pre-phase."""
|
||||
"""The PNG WRITER raises (a full disk) after the FIRST atomic copy
|
||||
landed: this call's ``.tmp`` files and the final-name files it
|
||||
CREATED are removed (no partial orphans — phase 163, task 02),
|
||||
``page_count`` AND ``page_strip_mode`` NULL, the warning is the
|
||||
signal, and the document indexes exactly as pre-phase."""
|
||||
root = tmp_path / "pgw"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
@@ -507,22 +554,29 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
||||
|
||||
def boom(self: Path, data: bytes) -> int:
|
||||
written_paths.append(self)
|
||||
n = original_write(self, data) # the tmp file really lands first
|
||||
if len(written_paths) > 1:
|
||||
raise OSError("simulated disk full (test sentinel)")
|
||||
return original_write(self, data)
|
||||
return n
|
||||
|
||||
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
|
||||
try:
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert len(written_paths) == 2, "p1 landed, p2's write failed"
|
||||
assert len(written_paths) == 2, "p1's tmp landed + replaced, p2's write failed"
|
||||
assert all(p.name.endswith(".png.tmp") for p in written_paths), (
|
||||
"the writer only ever touches .tmp paths (atomic replace)"
|
||||
)
|
||||
doc = _doc(db, "pgw", "ikea.pdf")
|
||||
assert doc.page_count is None
|
||||
assert doc.page_strip_mode is None, "the class is cleared with the count"
|
||||
assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), (
|
||||
"the partial write was cleaned up — no orphans"
|
||||
)
|
||||
assert not written_paths[0].exists(), "the landed p1 copy was removed"
|
||||
assert not list(_pages_dir(tmp_path).glob("*.tmp")), (
|
||||
"the failed call's .tmp file was cleaned up too"
|
||||
)
|
||||
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
||||
assert any("PDF page render failed" in m for m in caplog.messages), (
|
||||
caplog.messages
|
||||
@@ -531,6 +585,246 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
||||
_cleanup_source(db, "pgw")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 163 (task 02) — WRITE-FIRST, atomic per file (LOCKED A4)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_write_first_new_bytes_are_durable_before_any_stale_delete(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE order pin (LOCKED A4 — the 2026-10-07 live loss): re-rendering
|
||||
a CHANGED (shrunk, 10 → 6) PDF writes every replacement page FIRST
|
||||
(tmp + ``os.replace``) and only then deletes the stale tail — the
|
||||
event log shows ALL replaces of the fresh bytes before the FIRST
|
||||
unlink of an old-extent file, the overwritten pages keep the NEW
|
||||
bytes, the shrunk tail (p7..p10) is gone, and nothing else is
|
||||
deleted."""
|
||||
root = tmp_path / "pgord"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
_build_ikea_like_pdf(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc1 = _doc(db, "pgord", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
sentinel = b"OLD-COPY-SENTINEL"
|
||||
for n in range(1, 11): # make the old copies unmistakably distinguishable
|
||||
(page_dir / f"{doc1.id}-p{n}.png").write_bytes(sentinel)
|
||||
|
||||
events: list[tuple[str, str]] = []
|
||||
real_replace = os.replace
|
||||
|
||||
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
|
||||
events.append(("replace", Path(os.fspath(dst)).name))
|
||||
return real_replace(src, dst, *args, **kwargs)
|
||||
|
||||
real_unlink = Path.unlink
|
||||
|
||||
def unlink_spy(self: Path, **kwargs: Any) -> None:
|
||||
if self.parent == page_dir and self.name.startswith(str(doc1.id)):
|
||||
events.append(("unlink", self.name))
|
||||
return real_unlink(self, **kwargs)
|
||||
|
||||
monkeypatch.setattr(importer.os, "replace", replace_spy)
|
||||
monkeypatch.setattr(Path, "unlink", unlink_spy) # pyright: ignore[reportArgumentType]
|
||||
try:
|
||||
_build_ikea_shrunk(pdf) # same path, 6 pages — still qualifying
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
_cleanup_source(db, "pgord")
|
||||
replaces = [name for kind, name in events if kind == "replace"]
|
||||
unlinks = [name for kind, name in events if kind == "unlink"]
|
||||
assert len(replaces) == 6, "the 6 fresh pages are written via os.replace"
|
||||
assert set(unlinks) == {f"{doc1.id}-p{n}.png" for n in range(7, 11)}, (
|
||||
"the stale-tail delete touches ONLY the old extent the render did "
|
||||
"not rewrite — a file about to be overwritten is never deleted"
|
||||
)
|
||||
first_unlink = next(i for i, e in enumerate(events) if e[0] == "unlink")
|
||||
last_replace = max(i for i, e in enumerate(events) if e[0] == "replace")
|
||||
assert last_replace < first_unlink, (
|
||||
f"write-first violated — event order: {events}"
|
||||
)
|
||||
for n in range(1, 7):
|
||||
blob = (page_dir / f"{doc1.id}-p{n}.png").read_bytes()
|
||||
assert blob != sentinel and blob[:8] == PNG_MAGIC, (
|
||||
f"p{n} keeps the NEW rendered bytes"
|
||||
)
|
||||
for n in range(7, 11):
|
||||
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"stale p{n} leaked"
|
||||
assert not list(page_dir.glob("*.tmp")), "no .tmp orphans"
|
||||
|
||||
|
||||
def test_mid_write_failure_keeps_the_prior_copies_intact(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""THE failure pin (LOCKED A4): the writer raises on the SECOND
|
||||
page's tmp write of a changed re-render — the prior copy set stays
|
||||
on disk INTACT (the rollback-orphan state — the row over zero
|
||||
files — is structurally unrepresentable): every one of the 10 prior
|
||||
files still exists (p1 with the new bytes already atomically
|
||||
replaced before the failure, p2..p10 byte-for-byte the prior
|
||||
copies), zero ``.tmp`` orphans from this call, and the caller's
|
||||
fail-soft fires (``page_count``/``page_strip_mode`` NULL + the
|
||||
warning, the doc indexed fully)."""
|
||||
root = tmp_path / "pgfail"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
_build_ikea_like_pdf(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc1 = _doc(db, "pgfail", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
prior = {
|
||||
n: (page_dir / f"{doc1.id}-p{n}.png").read_bytes() for n in range(1, 11)
|
||||
}
|
||||
# Tag the re-render's bytes so the new p1 is recognizable, then
|
||||
# blow up the SECOND tmp write (page 2).
|
||||
_render_spy(
|
||||
monkeypatch,
|
||||
mutate_full=lambda pages: [
|
||||
PNG_MAGIC + f"NEW-{i}".encode() for i in range(len(pages))
|
||||
],
|
||||
)
|
||||
original_write = Path.write_bytes
|
||||
tmp_writes: list[Path] = []
|
||||
|
||||
def boom(self: Path, data: bytes) -> int:
|
||||
if self.suffix == ".tmp":
|
||||
tmp_writes.append(self)
|
||||
if len(tmp_writes) > 1:
|
||||
raise OSError("simulated disk full (test sentinel)")
|
||||
return original_write(self, data)
|
||||
|
||||
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
|
||||
_build_ikea_shrunk(pdf)
|
||||
try:
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||
assert len(tmp_writes) == 2, "p1's tmp wrote, p2's tmp write raised"
|
||||
doc2 = _doc(db, "pgfail", "ikea.pdf")
|
||||
assert doc2.page_count is None, "the caller's fail-soft nulled the count"
|
||||
assert doc2.page_strip_mode is None, "the caller's fail-soft nulled the class"
|
||||
assert any("PDF page render failed" in m for m in caplog.messages), caplog.messages
|
||||
# The prior set is INTACT: exactly the 10 names, none lost, none
|
||||
# orphaned on top.
|
||||
assert {p.name for p in page_dir.iterdir()} == {
|
||||
f"{doc1.id}-p{n}.png" for n in range(1, 11)
|
||||
}, "files on disk exactly the prior set"
|
||||
assert (
|
||||
page_dir / f"{doc1.id}-p1.png"
|
||||
).read_bytes() == PNG_MAGIC + b"NEW-0", (
|
||||
"p1's replacement bytes were durable before the failure (atomic "
|
||||
"replace) and were NOT removed by the cleanup"
|
||||
)
|
||||
for n in range(2, 11):
|
||||
assert (
|
||||
page_dir / f"{doc1.id}-p{n}.png"
|
||||
).read_bytes() == prior[n], f"prior copy p{n} was disturbed"
|
||||
assert not list(page_dir.glob("*.tmp")), "zero .tmp orphans from this call"
|
||||
assert doc2.chunks, "the document is indexed fully (fail-soft)"
|
||||
|
||||
|
||||
def test_page_copies_are_written_atomically_tmp_then_replace(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""THE atomic pin (LOCKED A4): the writer NEVER writes a final
|
||||
``<doc-id>-p{N}.png`` path directly — every page's bytes land on
|
||||
``<name>.png.tmp`` and ``os.replace`` moves them onto the final
|
||||
name in order (a reader opening the final path never sees truncated
|
||||
bytes), and no ``.tmp`` survives the successful call."""
|
||||
root = tmp_path / "pgatom"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
written: list[Path] = []
|
||||
original_write = Path.write_bytes
|
||||
|
||||
def write_spy(self: Path, data: bytes) -> int:
|
||||
written.append(self)
|
||||
return original_write(self, data)
|
||||
|
||||
replaced: list[tuple[str, str]] = []
|
||||
real_replace = os.replace
|
||||
|
||||
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
|
||||
replaced.append((os.fspath(src), os.fspath(dst)))
|
||||
return real_replace(src, dst, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(Path, "write_bytes", write_spy) # pyright: ignore[reportArgumentType]
|
||||
monkeypatch.setattr(importer.os, "replace", replace_spy)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
doc = _doc(db, "pgatom", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
assert len(written) == 10 and len(replaced) == 10
|
||||
assert all(p.name.endswith(".png.tmp") for p in written), (
|
||||
"no final path is ever written directly"
|
||||
)
|
||||
for n, (src, dst) in enumerate(replaced, start=1):
|
||||
assert dst == str(page_dir / f"{doc.id}-p{n}.png")
|
||||
assert src == f"{dst}.tmp", "the tmp name is <final>.tmp"
|
||||
assert not list(page_dir.glob("*.tmp")), "no .tmp survives the success path"
|
||||
|
||||
|
||||
def test_stale_tail_delete_failure_is_best_effort(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""An unreadable STALE-TAIL copy (the write-first tail pass, the
|
||||
``_delete_page_copies`` convention): the failure is logged, never
|
||||
raised — the sync completes, the row is stamped with the fresh
|
||||
count, and the reachable tail files still go."""
|
||||
root = tmp_path / "pgtail"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
_build_ikea_like_pdf(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc1 = _doc(db, "pgtail", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
p10 = page_dir / f"{doc1.id}-p10.png"
|
||||
real_unlink = Path.unlink
|
||||
|
||||
def boom(self: Path, **kwargs: Any) -> None:
|
||||
if self == p10:
|
||||
raise OSError("simulated unreadable file (test sentinel)")
|
||||
return real_unlink(self, **kwargs)
|
||||
|
||||
monkeypatch.setattr(Path, "unlink", boom) # pyright: ignore[reportArgumentType]
|
||||
_build_ikea_shrunk(pdf)
|
||||
try:
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||
doc2 = _doc(db, "pgtail", "ikea.pdf")
|
||||
assert doc2.page_count == 6 and doc2.page_strip_mode == "additive"
|
||||
assert p10.exists(), "the unreadable tail file survived (never raised)"
|
||||
for n in range(7, 10):
|
||||
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"p{n} still leaked"
|
||||
assert any(
|
||||
"could not delete PDF page copy" in m for m in caplog.messages
|
||||
), caplog.messages
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The shared cleanup helper (the _delete_image_copy convention)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -117,7 +117,10 @@ def test_gate_runs_after_the_phase36_base_construction() -> None:
|
||||
label = js.find('title.textContent = "Summary"')
|
||||
text_node = js.find('body.className = "doc-summary-text"')
|
||||
append = js.find("section.append(title, body)")
|
||||
mount = js.find("contentEl.appendChild(section)")
|
||||
# Phase 163 (task 04): the mount rides the `slot` variable —
|
||||
# slot === contentEl for every non-additive doc (the byte-identity
|
||||
# property), views only for the additive strip class.
|
||||
mount = js.find("slot.appendChild(section)")
|
||||
gate = js.find("void docAdminReady().then(", mount)
|
||||
wiring = js.find("if (admin) wireSummaryEdit(section, doc);")
|
||||
assert 0 < base < label < text_node < append < mount < gate < wiring, (
|
||||
|
||||
@@ -0,0 +1,431 @@
|
||||
"""Unit: the additive PDF strip view (phase 163, task 04 — D4, LOCKED A1/A6).
|
||||
|
||||
Frontend source-contract pins + node behavioral runs (the
|
||||
``test_document_viewer.py`` house pattern) over ``document.js``'s
|
||||
``renderDocument`` — the ONE shared core (Sources modal AND
|
||||
/document.html):
|
||||
|
||||
* the MODE branch — ``page_urls`` + ``page_strip_mode === "additive"``
|
||||
falls through to the text-doc rendering (Summary panel + content,
|
||||
the pre-148 branch reused) wrapped in a ``.doc-views`` container
|
||||
with the ``.doc-view-toggle`` group above it (Text pressed by
|
||||
default, LOCKED A6); ``page_urls`` with the mode ABSENT or
|
||||
``"primary"`` keeps the phase-148 strip-only branch VERBATIM (append
|
||||
the strip to ``contentEl`` + return — no toggle, no wrapper, no
|
||||
Summary panel); falsy ``page_urls`` is untouched;
|
||||
* the toggle semantics — ``role="group"`` + ``aria-label="Document
|
||||
view"``, two ``<button type="button">`` labeled ``Text`` / ``Pages``
|
||||
(fixed strings, no doc-derived text), ``aria-pressed`` true/false
|
||||
initial state, and a click mutating ONLY the two ``aria-pressed``
|
||||
states + the wrapper's ``data-view`` attribute (CSS-driven show/
|
||||
hide — the strip node stays mounted, no re-render, no refetch);
|
||||
* the never-build-HTML rule — createElement + properties/textContent
|
||||
only in the new code path (the module's ONE innerHTML stays the
|
||||
escape-first markdown render); the strip imgs keep
|
||||
``loading = "lazy"`` (the hidden strip never fetches);
|
||||
* the ``.doc-view-toggle`` CSS — the house ghost-pill idiom, the
|
||||
pressed state as text + style (never color alone, B5), ≥44px
|
||||
targets, and the ``data-view`` show/hide contract.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||
DOCUMENT_JS = FRONTEND / "assets" / "document.js"
|
||||
STYLES_CSS = FRONTEND / "assets" / "styles.css"
|
||||
|
||||
HAVE_NODE = shutil.which("node") is not None
|
||||
|
||||
|
||||
def _read(path: Path) -> str:
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _fn(js: str, name: str) -> str:
|
||||
"""The source of a top-level function via balanced-brace counting
|
||||
(the ``test_document_viewer.py`` pattern)."""
|
||||
start = js.find(f"function {name}(")
|
||||
assert start != -1, f"{name}() must exist in document.js"
|
||||
i = js.find("(", start)
|
||||
depth = 0
|
||||
close = i
|
||||
while i < len(js):
|
||||
if js[i] == "(":
|
||||
depth += 1
|
||||
elif js[i] == ")":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
close = i
|
||||
break
|
||||
i += 1
|
||||
brace = js.find("{", close)
|
||||
depth = 0
|
||||
for j in range(brace, len(js)):
|
||||
if js[j] == "{":
|
||||
depth += 1
|
||||
elif js[j] == "}":
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return js[start : j + 1]
|
||||
raise AssertionError(f"unbalanced braces in {name}()")
|
||||
|
||||
|
||||
def _code(js: str) -> str:
|
||||
"""The source with comments stripped (the never-build-HTML pins)."""
|
||||
return re.sub(r"//.*?$|/\*.*?\*/", "", js, flags=re.S | re.M)
|
||||
|
||||
|
||||
def _run_node(script: str) -> str:
|
||||
proc = subprocess.run(
|
||||
["node", "-e", script], capture_output=True, text=True, timeout=60
|
||||
)
|
||||
assert proc.returncode == 0, f"node failed: {proc.stderr}"
|
||||
return proc.stdout
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Source pins — the renderDocument mode branch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_render_document_branches_on_strip_mode() -> None:
|
||||
"""The mode branch (LOCKED A2 — the tolerant read): the additive
|
||||
flag keys on the EXACT ``doc.page_strip_mode === "additive"``
|
||||
string (absent / "primary" / anything else ⇒ primary), and the
|
||||
primary/absent strip branch stays the phase-148 shape — append
|
||||
``docPagesStrip`` to ``contentEl`` and RETURN — evaluated BEFORE
|
||||
any wrapper/toggle construction."""
|
||||
render = _fn(_read(DOCUMENT_JS), "renderDocument")
|
||||
assert 'doc.page_strip_mode === "additive"' in render, (
|
||||
"only the exact additive string switches on the toggle"
|
||||
)
|
||||
gate_i = render.find('const additive = stripEligible && doc.page_strip_mode === "additive";')
|
||||
primary_i = render.find("if (stripEligible && !additive) {")
|
||||
strip_i = render.find("contentEl.appendChild(docPagesStrip(doc, pageUrls));")
|
||||
return_i = render.find("return;", strip_i)
|
||||
wrapper_i = render.find('views.className = "doc-views";')
|
||||
assert -1 < gate_i < primary_i < strip_i < return_i < wrapper_i, (
|
||||
"the primary/absent branch (strip into contentEl + return) precedes "
|
||||
"the additive wrapper — its DOM stays the phase-148 shape verbatim"
|
||||
)
|
||||
# The wrapper + toggle: ONE .doc-views container, the toggle BEFORE
|
||||
# it, both into contentEl; the text view lands in the wrapper.
|
||||
append_i = render.find("contentEl.append(docViewToggle(views), views);")
|
||||
assert wrapper_i < append_i, "the wrapper is built, then appended after its toggle"
|
||||
assert 'views.setAttribute("data-view", "text");' in render, (
|
||||
"LOCKED A6 — Text is the default view"
|
||||
)
|
||||
# The strip mounts ONCE inside the wrapper, after the text view —
|
||||
# never rebuilt on a flip (no docPagesStrip call outside the two
|
||||
# branches).
|
||||
assert render.count("docPagesStrip(doc, pageUrls)") == 2
|
||||
mount_i = render.rfind("views.appendChild(docPagesStrip(doc, pageUrls));")
|
||||
assert mount_i > render.find('slot.appendChild(pre);') or mount_i > render.rfind(
|
||||
"slot.appendChild(wrap);"
|
||||
), "the strip mounts AFTER the text view inside the wrapper"
|
||||
# The non-additive path is untouched: slot === contentEl (the
|
||||
# pre-148 appends keep their exact shape through the slot var).
|
||||
assert "let slot = contentEl;" in render
|
||||
|
||||
|
||||
def test_primary_absent_branch_builds_no_toggle() -> None:
|
||||
"""The primary/absent branch is toggle-free by construction: the
|
||||
strip-branch block (from the ``pageUrls`` read to its ``return``)
|
||||
contains NO ``doc-view-toggle`` / ``doc-views`` construction — the
|
||||
strip-only DOM stays byte-identical to pre-phase-163."""
|
||||
render = _fn(_read(DOCUMENT_JS), "renderDocument")
|
||||
primary_start = render.find("if (stripEligible && !additive)")
|
||||
branch = render[render.find("const pageUrls =") : render.find("return;", primary_start)]
|
||||
assert "doc-view-toggle" not in branch
|
||||
assert "doc-views" not in branch
|
||||
assert branch.count("appendChild") == 1, (
|
||||
"the primary branch is exactly the phase-148 append + return"
|
||||
)
|
||||
|
||||
|
||||
def test_toggle_never_builds_html_and_labels_are_fixed() -> None:
|
||||
"""The never-build-HTML rule: the toggle builders use
|
||||
createElement + properties/textContent only — no innerHTML /
|
||||
outerHTML — and the labels are the FIXED strings ``Text`` /
|
||||
``Pages`` (no doc-derived text inside the toggle). The module's
|
||||
ONE innerHTML (the escape-first markdown render) survives."""
|
||||
js = _read(DOCUMENT_JS)
|
||||
toggle = _fn(js, "docViewToggle")
|
||||
btn = _fn(js, "viewToggleButton")
|
||||
for body in (toggle, btn):
|
||||
code = _code(body)
|
||||
assert "innerHTML" not in code and "outerHTML" not in code
|
||||
assert 'document.createElement("div")' in toggle
|
||||
assert 'group.setAttribute("role", "group")' in toggle
|
||||
assert 'group.setAttribute("aria-label", "Document view")' in toggle
|
||||
assert 'document.createElement("button")' in btn
|
||||
assert 'btn.type = "button"' in btn
|
||||
assert 'btn.className = "doc-view-btn"' in btn
|
||||
assert 'viewToggleButton("Text")' in toggle
|
||||
assert 'viewToggleButton("Pages")' in toggle
|
||||
# The module-wide XSS contract: exactly one innerHTML, the
|
||||
# renderMarkdown line.
|
||||
whole = _code(js)
|
||||
assert whole.count("innerHTML") == 1
|
||||
assert "outerHTML" not in whole
|
||||
|
||||
|
||||
def test_strip_imgs_stay_lazy() -> None:
|
||||
"""The phase-148 lazy-img pin re-runs green: the strip builder is
|
||||
UNMODIFIED — every page ``<img>`` keeps ``loading = "lazy"`` (the
|
||||
additive default view never fetches the strip)."""
|
||||
strip = _fn(_read(DOCUMENT_JS), "docPagesStrip")
|
||||
assert 'img.loading = "lazy";' in strip
|
||||
|
||||
|
||||
def test_view_toggle_css() -> None:
|
||||
"""styles.css: the ``.doc-view-toggle`` group (the chat-column
|
||||
reading width, the pill pair adjacent) + the ``.doc-view-btn``
|
||||
ghost pill (≥44px target, ink-soft text — AA) + the PRESSED state
|
||||
(``[aria-pressed="true"]`` — brand fill + font-weight 700: text +
|
||||
style, never color alone, B5) + the CSS-driven view swap
|
||||
(``.doc-views .doc-pages`` hidden by default; ``[data-view=
|
||||
"pages"]`` hides the Summary panel / .doc-md / .doc-raw /
|
||||
.doc-image and shows the strip as the grid). Provenance comment
|
||||
above the block (house style)."""
|
||||
css = _read(STYLES_CSS)
|
||||
toggle_i = css.find(".doc-view-toggle {")
|
||||
assert toggle_i != -1, "styles.css must style .doc-view-toggle"
|
||||
comment_start = css.rfind("/*", 0, toggle_i)
|
||||
assert -1 < comment_start < toggle_i and "phase 163" in css[comment_start:toggle_i]
|
||||
btn_i = css.find(".doc-view-btn {")
|
||||
pressed_i = css.find('.doc-view-btn[aria-pressed="true"]')
|
||||
assert toggle_i < btn_i < pressed_i
|
||||
btn_rule = css[btn_i : css.find("}", btn_i)]
|
||||
for decl in ("min-height: 44px", "color: var(--ink-soft)", "border: 1px solid var(--line)"):
|
||||
assert decl in btn_rule
|
||||
pressed_rule = css[pressed_i : css.find("}", pressed_i)]
|
||||
assert "background: var(--brand)" in pressed_rule
|
||||
assert "font-weight: 700" in pressed_rule, (
|
||||
"the pressed state is text weight + fill — never color alone (B5)"
|
||||
)
|
||||
# The CSS-driven swap: hidden strip by default; pages hides the
|
||||
# text view and shows the strip.
|
||||
assert '.doc-views .doc-pages { display: none; }' in css
|
||||
for sel in (".doc-summary", ".doc-md", ".doc-raw", ".doc-image"):
|
||||
assert f'.doc-views[data-view="pages"] {sel}' in css
|
||||
assert '.doc-views[data-view="pages"] .doc-pages { display: grid; }' in css
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node behavioral runs — the real renderDocument over a minimal fake DOM
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_FAKE_DOM = """
|
||||
function makeEl(tag) {
|
||||
return {
|
||||
tag, className: "", textContent: "", attrs: {}, children: [], handlers: {},
|
||||
setAttribute(k, v) { this.attrs[k] = String(v); },
|
||||
getAttribute(k) { return k in this.attrs ? this.attrs[k] : null; },
|
||||
appendChild(c) { this.children.push(c); return c; },
|
||||
append(...cs) { this.children.push(...cs); },
|
||||
replaceChildren(...cs) { this.children = cs; },
|
||||
addEventListener(ev, fn) { this.handlers[ev] = fn; },
|
||||
click() { this.handlers.click(); },
|
||||
};
|
||||
}
|
||||
globalThis.document = { createElement: (t) => makeEl(t) };
|
||||
function fmtDate(x) { return "DATE:" + x; }
|
||||
function metaBadge(cls, text, title) {
|
||||
const s = makeEl("span");
|
||||
s.className = cls;
|
||||
s.textContent = text;
|
||||
if (title !== undefined) s.setAttribute("title", title);
|
||||
return s;
|
||||
}
|
||||
function copyPathButton() { return makeEl("button"); }
|
||||
function originAnchor() { return makeEl("a"); }
|
||||
async function docAdminReady() { return false; }
|
||||
function wireDateEdit() {}
|
||||
function wireSummaryEdit() {}
|
||||
function renderMarkdown(s) { return s; }
|
||||
function docImageBlock() { return makeEl("div"); }
|
||||
function describe(el) {
|
||||
return {
|
||||
tag: el.tag,
|
||||
className: el.className,
|
||||
attrs: el.attrs,
|
||||
text: el.textContent,
|
||||
children: el.children.map(describe),
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _render_script() -> str:
|
||||
js = _read(DOCUMENT_JS)
|
||||
parts = [
|
||||
_fn(js, n)
|
||||
for n in ("renderDocument", "docViewToggle", "viewToggleButton", "docPagesStrip")
|
||||
]
|
||||
return _FAKE_DOM + "\n".join(parts)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_additive_doc_renders_text_views_and_toggle() -> None:
|
||||
"""Behavioral (node, fake DOM): an ADDITIVE doc renders the TEXT
|
||||
doc exactly as a non-qualifying one — Summary panel + content —
|
||||
inside ``.doc-views`` (data-view="text"), with the toggle group
|
||||
FIRST in the content slot: role=group, aria-label="Document
|
||||
view", the two ``doc-view-btn`` buttons labeled Text/Pages, Text
|
||||
aria-pressed="true" (LOCKED A6), and the strip MOUNTED inside the
|
||||
wrapper (figures + lazy imgs) after the text view."""
|
||||
out = _run_node(
|
||||
_render_script()
|
||||
+ """
|
||||
const doc = {
|
||||
title: "Letumdei", path: "book.pdf", source: "Library", format: "pdf",
|
||||
summary: "A book.", content: "BODY TEXT", created_at: "c", indexed_at: "i",
|
||||
chunks: 9, page_urls: ["/api/documents/x/pages/1", "/api/documents/x/pages/2"],
|
||||
page_strip_mode: "additive",
|
||||
};
|
||||
const contentEl = makeEl("div");
|
||||
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||
console.log(JSON.stringify(describe(contentEl)));
|
||||
"""
|
||||
)
|
||||
content = json.loads(out)
|
||||
assert [c["className"] for c in content["children"]] == [
|
||||
"doc-view-toggle",
|
||||
"doc-views",
|
||||
], "the toggle group sits above the .doc-views wrapper"
|
||||
toggle, views = content["children"]
|
||||
assert toggle["attrs"]["role"] == "group"
|
||||
assert toggle["attrs"]["aria-label"] == "Document view"
|
||||
labels = [c["text"] for c in toggle["children"]]
|
||||
assert labels == ["Text", "Pages"]
|
||||
assert [c["attrs"]["aria-pressed"] for c in toggle["children"]] == ["true", "false"], (
|
||||
"Text pressed by default (LOCKED A6)"
|
||||
)
|
||||
for c in toggle["children"]:
|
||||
assert c["tag"] == "button"
|
||||
assert views["attrs"]["data-view"] == "text"
|
||||
assert [c["className"] for c in views["children"]] == [
|
||||
"doc-summary",
|
||||
"doc-raw",
|
||||
"doc-pages",
|
||||
], "additive = Summary panel + text content + the mounted strip"
|
||||
# pre.textContent — a text node, never parsed as HTML
|
||||
assert views["children"][1]["text"] == "BODY TEXT"
|
||||
strip = views["children"][2]
|
||||
assert len(strip["children"]) == 2, "one figure per page_urls entry"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_primary_and_absent_modes_keep_the_strip_only_dom() -> None:
|
||||
"""Behavioral: mode "primary" AND an absent mode (the pre-163
|
||||
stamped row — the live Letumdei pre-sync shape) keep the phase-148
|
||||
DOM byte-identically: the content slot holds EXACTLY the
|
||||
``.doc-pages`` strip — no wrapper, no toggle, no Summary panel,
|
||||
no text block."""
|
||||
out = _run_node(
|
||||
_render_script()
|
||||
+ """
|
||||
const results = [];
|
||||
for (const mode of ["primary", undefined]) {
|
||||
const doc = {
|
||||
title: "Manual", path: "m.pdf", source: "Library", format: "pdf",
|
||||
summary: "A manual.", content: "SHOULD NOT SHOW", created_at: "c",
|
||||
indexed_at: "i", chunks: 4,
|
||||
page_urls: ["/api/documents/y/pages/1"],
|
||||
};
|
||||
if (mode !== undefined) doc.page_strip_mode = mode;
|
||||
const contentEl = makeEl("div");
|
||||
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||
results.push(describe(contentEl));
|
||||
}
|
||||
console.log(JSON.stringify(results));
|
||||
"""
|
||||
)
|
||||
primary, absent = json.loads(out)
|
||||
for content in (primary, absent):
|
||||
assert len(content["children"]) == 1
|
||||
assert content["children"][0]["className"] == "doc-pages"
|
||||
classes = json.dumps(content)
|
||||
assert "doc-view-toggle" not in classes
|
||||
assert "doc-views" not in classes
|
||||
assert "doc-summary" not in classes
|
||||
assert "doc-raw" not in classes, "the strip replaces the text block (primary class)"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_text_doc_dom_is_untouched() -> None:
|
||||
"""Behavioral: a doc with no page_urls is UNTOUCHED — Summary
|
||||
panel + content appended straight to the content slot, no
|
||||
wrapper, no toggle (the byte-identity property for every
|
||||
non-PDF)."""
|
||||
out = _run_node(
|
||||
_render_script()
|
||||
+ """
|
||||
const doc = {
|
||||
title: "Note", path: "n.pdf", source: "Library", format: "pdf",
|
||||
summary: "S.", content: "TEXT", created_at: "c", indexed_at: "i", chunks: 1,
|
||||
};
|
||||
const contentEl = makeEl("div");
|
||||
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||
console.log(JSON.stringify(describe(contentEl)));
|
||||
"""
|
||||
)
|
||||
content = json.loads(out)
|
||||
assert [c["className"] for c in content["children"]] == ["doc-summary", "doc-raw"]
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_toggle_flips_view_without_rerender_or_refetch() -> None:
|
||||
"""Behavioral: the toggle handler mutates ONLY the two
|
||||
aria-pressed states + the wrapper's data-view. Clicking Pages
|
||||
flips data-view to "pages"; clicking back to Text reverses; the
|
||||
strip node STAYS MOUNTED the whole time (same object, no
|
||||
re-append — the flip-flop fetches nothing) and the wrapper's
|
||||
children never change count."""
|
||||
out = _run_node(
|
||||
_render_script()
|
||||
+ """
|
||||
const doc = {
|
||||
title: "Book", path: "b.pdf", source: "Library", format: "pdf",
|
||||
summary: "S.", content: "TEXT", created_at: "c", indexed_at: "i", chunks: 2,
|
||||
page_urls: ["/api/documents/z/pages/1", "/api/documents/z/pages/2"],
|
||||
page_strip_mode: "additive",
|
||||
};
|
||||
const contentEl = makeEl("div");
|
||||
renderDocument(doc, { titleEl: makeEl("h1"), metaEl: makeEl("div"), contentEl });
|
||||
const toggle = contentEl.children[0];
|
||||
const views = contentEl.children[1];
|
||||
const [textBtn, pagesBtn] = toggle.children;
|
||||
const stripBefore = views.children[2];
|
||||
const countBefore = views.children.length;
|
||||
pagesBtn.click();
|
||||
const afterPages = {
|
||||
view: views.getAttribute("data-view"),
|
||||
pressed: toggle.children.map((b) => b.getAttribute("aria-pressed")),
|
||||
};
|
||||
textBtn.click();
|
||||
const afterText = {
|
||||
view: views.getAttribute("data-view"),
|
||||
pressed: toggle.children.map((b) => b.getAttribute("aria-pressed")),
|
||||
stripSame: views.children[2] === stripBefore,
|
||||
countSame: views.children.length === countBefore,
|
||||
};
|
||||
console.log(JSON.stringify({ afterPages, afterText }));
|
||||
"""
|
||||
)
|
||||
res = json.loads(out)
|
||||
assert res["afterPages"] == {"view": "pages", "pressed": ["false", "true"]}
|
||||
assert res["afterText"] == {
|
||||
"view": "text",
|
||||
"pressed": ["true", "false"],
|
||||
"stripSame": True,
|
||||
"countSame": True,
|
||||
}, "the strip node persists across flips — no re-render, no refetch"
|
||||
@@ -13,12 +13,13 @@ without a browser:
|
||||
* NO ``@media (min-width: 1500px)`` block anywhere in the file — the
|
||||
phase-58 wide-desktop doubling is deleted in full (the token is
|
||||
72rem at every viewport);
|
||||
* the six reading-column selectors — ``.chat-shell``,
|
||||
* the reading-column selectors — ``.chat-shell``,
|
||||
``.shared-shell``, ``.doc-md``, ``.doc-image`` (phase 122, task 04),
|
||||
``.doc-pages`` (phase 148, task 03 — the page strip), and
|
||||
``.doc-summary:has(+ .doc-md)`` — each capped with
|
||||
``max-width: var(--chat-column)`` and NOTHING else in the file uses
|
||||
the token for a max-width (exactly six rules);
|
||||
``.doc-pages`` (phase 148, task 03 — the page strip),
|
||||
``.doc-view-toggle`` (phase 163, task 04 — the Text/Pages toggle
|
||||
above the additive views), and ``.doc-summary:has(+ .doc-md)`` —
|
||||
each capped with ``max-width: var(--chat-column)`` and NOTHING else
|
||||
in the file uses the token for a max-width (exactly seven rules);
|
||||
* the flipped negative pin — ZERO literal ``max-width: 46rem`` rules
|
||||
remain (the phase-27/59/91 form-column caps are retired): the form
|
||||
shells (``.tuning-shell``, ``.theme-shell``, ``.doc-edit-shell``)
|
||||
@@ -121,8 +122,10 @@ def test_the_reading_columns_use_the_token() -> None:
|
||||
task 04 — the viewer's image block rides the SAME reading column
|
||||
as its .doc-md sibling), .doc-pages (phase 148, task 03 — the
|
||||
viewer's page strip rides the SAME reading column in the shared
|
||||
#doc-content slot) and .doc-summary:has(+ .doc-md) each cap with
|
||||
max-width: var(--chat-column) — and exactly those six rules use
|
||||
#doc-content slot), .doc-view-toggle (phase 163, task 04 — the
|
||||
Text/Pages toggle sits above the additive views, aligned with
|
||||
them) and .doc-summary:has(+ .doc-md) each cap with
|
||||
max-width: var(--chat-column) — and exactly those seven rules use
|
||||
the token for a max-width (no other selector)."""
|
||||
css = _css()
|
||||
for selector in (
|
||||
@@ -131,12 +134,13 @@ def test_the_reading_columns_use_the_token() -> None:
|
||||
".doc-md",
|
||||
".doc-image",
|
||||
".doc-pages",
|
||||
".doc-view-toggle",
|
||||
".doc-summary:has(+ .doc-md)",
|
||||
):
|
||||
assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
|
||||
f"{selector} must cap with max-width: var(--chat-column)"
|
||||
)
|
||||
assert css.count("max-width: var(--chat-column)") == 6, (
|
||||
assert css.count("max-width: var(--chat-column)") == 7, (
|
||||
"exactly the reading-column selectors use the token"
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user