phase: 148_pdf_pages_in_sources
**Phase 148 — final verification pass: complete, 100% green.**
- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
This commit is contained in:
@@ -0,0 +1,775 @@
|
||||
"""Phase 148 task 04 E2E (Playwright, mock-only): the pictorial PDF
|
||||
opens as its PAGES — a text-heavy PDF renders byte-identical to
|
||||
pre-phase.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov
|
||||
|
||||
The dedicated suite for ``148_pdf_pages_in_sources`` (A16 — one
|
||||
Playwright file per phase, run in isolation): the owner's TODO L5
|
||||
scenario end to end on SYNTHETIC fixtures in the owner's named
|
||||
``ikea_instructions.tar``'s exact CLASS (the real fixture is pinned at
|
||||
the integration level — the E2E keeps the deterministic canned-reply
|
||||
premise). Two PyMuPDF PDFs (the house recipe) land in a LOCAL
|
||||
DIRECTORY source (``BOR_SOURCES_DIR``-sibling temp dir, API-registered
|
||||
— local dirs are the owner's named case) and one real sync indexes
|
||||
them:
|
||||
|
||||
* ``pictorial.pdf`` — the IKEA shape (the pictorial class): page 1 a
|
||||
dense prose paragraph (over the 100-char sparseness bar — NOT a
|
||||
candidate; the whole layer passes ``is_usable_pdf_text``), pages
|
||||
2–3 SPARSE (a short marker + a drawn VECTOR figure — rectangle +
|
||||
arrow, <100 stripped chars — the ``pictorial_pages(texts,
|
||||
min_chars=100)`` candidates the mock's vision describes with the
|
||||
canned ``PICTORIAL_PAGE_ANSWER``);
|
||||
* ``textheavy.pdf`` — 2 dense text pages (ZERO candidates — the
|
||||
negative class: the text view, the pre-phase rendering).
|
||||
|
||||
App env: the conftest pattern with the phase-140 opt-in (``BOR_PDF_
|
||||
PICTORIAL`` pinned EXPLICITLY ``1`` — the conftest pins it off in
|
||||
``os.environ`` for the pre-phase-140 suites) + the phase-148
|
||||
``BOR_PDF_PAGE_DIR`` pointed at a suite-private scratch dir (the seed
|
||||
asserts the copies THERE; the owner's ``~/bor-sources/pages`` default
|
||||
is never touched). ``BOR_IMPORT_EXTENSIONS`` carries the ``pdf``
|
||||
token (the phase-126 convention — the owner's local ``.env`` may lack
|
||||
it, and without it the walk is blind to the fixture).
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the described
|
||||
blocks (and the seed's DB witness that the descriptions ARE indexed
|
||||
but NOT displayed) are exact only against the mock's deterministic
|
||||
``PICTORIAL_PAGE_MODE`` canned reply.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_pictorial_pdf_shows_pages_in_modal`` — the seed (module)
|
||||
asserts the DB + filesystem first: the pictorial row has
|
||||
``page_count == 3`` and the scratch page dir holds EXACTLY
|
||||
``<id>-p{1..3}.png`` (real PNG magic, non-empty — the textheavy
|
||||
doc's copies do not exist); then the Sources-tree doc modal shows
|
||||
the ``.doc-pages`` strip: exactly three ``.doc-page`` figures,
|
||||
each a LOADED lazy ``<img>`` (``complete && naturalWidth > 0`` —
|
||||
the route served a real PNG, not a 404) with the
|
||||
``Page N of <title>`` alt, the ``Page N`` caption, the NO
|
||||
``.doc-raw`` / NO ``.doc-md`` / NO ``.doc-summary`` suppression
|
||||
contract (the mock's page descriptions sit in the DB content but
|
||||
nowhere in the rendered modal), the meta badge row (source /
|
||||
Created / Indexed) untouched.
|
||||
2. ``test_textheavy_pdf_unchanged`` — the textheavy modal renders the
|
||||
PRE-PHASE shape: the content in ``pre.doc-raw`` (both pages), the
|
||||
Summary panel (the mock's lite digest), NO ``.doc-pages`` — and
|
||||
the content payload's ``page_urls`` KEY is ABSENT (never ``null`` —
|
||||
the omission pin the viewer's strip gate rides on).
|
||||
3. ``test_full_page_surface`` — the same strip on the dedicated
|
||||
``/document.html`` page (admin-logged-in, the phase-79 gate
|
||||
satisfied): one ``renderDocument`` core — no per-surface drift —
|
||||
same figures, same suppression.
|
||||
4. ``test_page_route_serves_png`` — an ANONYMOUS (no-login) direct
|
||||
request on ``/api/documents/<id>/pages/1``: 200, ``image/png``,
|
||||
the ``Cache-Control: private, max-age=3600`` header, a non-empty
|
||||
PNG body byte-equal to the stored copy (the public
|
||||
uuid4-credential bytes route, from the UI's perspective).
|
||||
|
||||
House pattern: one module app over the shared E2E Postgres (its own
|
||||
port, module env — the conftest session app is never started in this
|
||||
isolated run, so no port clash). The four tests SHARE the module seed
|
||||
(one source, two documents, one sync), so the shared registry + KB is
|
||||
truncated ONCE, after the last test (a module-granularity autouse — a
|
||||
per-test truncate would wipe the seed from under its siblings), with a
|
||||
no-running-sync wait so a leaked run cannot race it.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
|
||||
import httpx
|
||||
import pymupdf as fitz # PyMuPDF — the synthetic-fixture builder (the house recipe)
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.pdf import (
|
||||
extract_pdf_page_texts,
|
||||
extract_pdf_text,
|
||||
is_usable_pdf_text,
|
||||
pictorial_pages,
|
||||
render_pdf_pages,
|
||||
)
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import PICTORIAL_PAGE_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The local directory's NAME — the source name the registry stores
|
||||
#: for a local-kind source (the directory basename, the
|
||||
#: test_local_directory_sources convention).
|
||||
SOURCE_NAME = "e2e-p148-pdf"
|
||||
PICTORIAL_PATH = "pictorial.pdf"
|
||||
TEXTHEAVY_PATH = "textheavy.pdf"
|
||||
PICTORIAL_TITLE = "Assembly Instructions" # the metadata title
|
||||
TEXTHEAVY_TITLE = "Dense Notes" # the metadata title
|
||||
|
||||
#: Page 1 — the dense prose paragraph (over the 100-char sparseness
|
||||
#: bar — NOT a pictorial candidate; the whole layer is USABLE).
|
||||
PAGE_ONE = (
|
||||
"This manual covers the complete flat-pack furniture frame, its "
|
||||
"shipped hardware kit, and the safety checks a first-time assembler "
|
||||
"should run before starting. The box holds the two side panels, the "
|
||||
"top board, the adjustable leg set, eight cam locks, fourteen wood "
|
||||
"screws, two bolts, and the wall-anchoring strap. Work on a flat "
|
||||
"surface with soft flooring, keep the hardware sorted by size, and "
|
||||
"read every step through before you begin so you know which panel "
|
||||
"is which. The frame stands roughly a metre tall once assembled, "
|
||||
"weights under twelve kilograms, and needs no further tools beyond "
|
||||
"the included Allen key."
|
||||
)
|
||||
assert len(PAGE_ONE) > 500, "page 1 must be a 500+ char paragraph (the task's recipe)"
|
||||
|
||||
#: The sparse pages' markers — ALL the text layer carries on pages
|
||||
#: 2–3 (6 stripped chars, under the 100-char bar — the candidates);
|
||||
#: the figures are drawn vector (no raster image).
|
||||
PAGE_TWO_MARKER = "Fig. 2"
|
||||
PAGE_THREE_MARKER = "Fig. 3"
|
||||
|
||||
#: The stored illustration blocks (the phase-140 LOCKED A4 shape) —
|
||||
#: the mock's canned ``PICTORIAL_PAGE_MODE`` reply, deterministic.
|
||||
BLOCK_TWO = f"[Page 2 illustration: {PICTORIAL_PAGE_ANSWER}]"
|
||||
BLOCK_THREE = f"[Page 3 illustration: {PICTORIAL_PAGE_ANSWER}]"
|
||||
|
||||
#: The PNG signature (the route/seed byte assertions' magic number).
|
||||
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
|
||||
|
||||
#: Phase 79 (task 04, full inventory): the conftest session app owns
|
||||
#: its port in a combined run — this module app binds its own port
|
||||
#: instead (a same-port second uvicorn dies on bind and would drive
|
||||
#: the wrong server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_PDFPAGES", "8161"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixture builder (the house PyMuPDF recipe — the IKEA class: a dense
|
||||
# prose page + sparse vector-illustrated pages)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _draw_illustration(page: Any) -> None:
|
||||
"""The IKEA pictogram class: a rectangle (a part) + a line arrow
|
||||
with a two-stroke head (a join) — vector-drawn, so
|
||||
``get_image_info()`` sees ZERO embedded raster images (the
|
||||
phase-122/137 pipeline is blind to it)."""
|
||||
page.draw_rect(fitz.Rect(100, 120, 300, 220))
|
||||
page.draw_line(fitz.Point(320, 170), fitz.Point(470, 170))
|
||||
page.draw_line(fitz.Point(470, 170), fitz.Point(450, 158))
|
||||
page.draw_line(fitz.Point(470, 170), fitz.Point(450, 182))
|
||||
|
||||
|
||||
def _build_pictorial(path: Path) -> None:
|
||||
"""The IKEA shape: page 1 = the dense prose paragraph
|
||||
(``insert_textbox`` — word-wrapped), pages 2–3 = a short marker +
|
||||
the drawn vector figure (sparse — the pictorial candidates)."""
|
||||
doc = fitz.open()
|
||||
page1 = doc.new_page()
|
||||
rc = page1.insert_textbox(fitz.Rect(72, 72, 523, 772), PAGE_ONE, fontsize=11)
|
||||
assert rc >= 0, "the page-1 paragraph must fit its text box"
|
||||
for marker in (PAGE_TWO_MARKER, PAGE_THREE_MARKER):
|
||||
p = doc.new_page()
|
||||
p.insert_text((72, 72), marker)
|
||||
_draw_illustration(p)
|
||||
doc.set_metadata({"title": PICTORIAL_TITLE})
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _build_textheavy(path: Path) -> None:
|
||||
"""The negative class: two DENSE text pages (zero candidates — the
|
||||
text view, byte-identical to pre-phase)."""
|
||||
doc = fitz.open()
|
||||
for i in (1, 2):
|
||||
p = doc.new_page()
|
||||
rc = p.insert_textbox(
|
||||
fitz.Rect(72, 72, 523, 772), f"Paragraph {i}: {PAGE_ONE}", fontsize=11
|
||||
)
|
||||
assert rc >= 0, f"the page-{i} paragraph must fit its text box"
|
||||
doc.set_metadata({"title": TEXTHEAVY_TITLE})
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def pdfs_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The local directory source's root: both fixture PDFs, built
|
||||
ONCE (module-scoped). The directory's NAME is the source name
|
||||
(a ``kind='local'`` registry row indexes under the directory's
|
||||
basename — the test_local_directory_sources convention), so it
|
||||
is a named subdirectory, not the mktemp root (whose name carries
|
||||
a random suffix). The premise pins (deterministic — computed from
|
||||
the fixtures THEMSELVES, no hardcoded page list): the pictorial
|
||||
PDF is usable-layer with candidates [2, 3] (3 pages — the seed's
|
||||
``page_count`` target); the textheavy PDF is usable-layer with
|
||||
ZERO candidates."""
|
||||
root = tmp_path_factory.mktemp("bor_p148_pdf") / SOURCE_NAME
|
||||
root.mkdir(parents=True)
|
||||
min_chars = Settings.model_fields["pdf_pictorial_min_chars"].default
|
||||
pic = root / PICTORIAL_PATH
|
||||
heavy = root / TEXTHEAVY_PATH
|
||||
_build_pictorial(pic)
|
||||
_build_textheavy(heavy)
|
||||
pic_texts = extract_pdf_page_texts(pic)
|
||||
heavy_texts = extract_pdf_page_texts(heavy)
|
||||
assert is_usable_pdf_text(extract_pdf_text(pic)), (
|
||||
"the pictorial fixture must PASS the usability gate (the IKEA "
|
||||
"class — the fallback class is the other qualifying class, not "
|
||||
"this one)"
|
||||
)
|
||||
assert pictorial_pages(pic_texts, min_chars=min_chars) == [2, 3], (
|
||||
f"the pictorial fixture's sparse-page premise broke: "
|
||||
f"{pictorial_pages(pic_texts, min_chars=min_chars)}"
|
||||
)
|
||||
assert is_usable_pdf_text(extract_pdf_text(heavy))
|
||||
assert pictorial_pages(heavy_texts, min_chars=min_chars) == [], (
|
||||
"the textheavy fixture must carry ZERO sparse pages (the "
|
||||
"negative class)"
|
||||
)
|
||||
# Every page renders to a real PNG (the pipeline's byte premise).
|
||||
assert len(render_pdf_pages(pic)) == 3
|
||||
assert len(render_pdf_pages(heavy)) == 2
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def scratch(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The suite-private app homes (pages dir, upload dir, checkouts,
|
||||
images) — disposable with the module."""
|
||||
return tmp_path_factory.mktemp("bor_p148_scratch")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def page_dir(scratch: Path) -> Path:
|
||||
"""The app's ``pdf_page_dir`` (``BOR_PDF_PAGE_DIR``) for this
|
||||
suite — the seed asserts the ``<doc-id>-p{N}.png`` copies here."""
|
||||
return scratch / "pages"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module app (the conftest env pattern + the phase-140/148 knobs)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(mock_llm: int, scratch: Path, page_dir: Path) -> dict[str, str]:
|
||||
"""The conftest app env (leak guards included) with the phase-140
|
||||
opt-in (``BOR_PDF_PICTORIAL`` EXPLICITLY on — the conftest pins it
|
||||
off in ``os.environ`` for the pre-phase suites; this suite is the
|
||||
IKEA class) and the phase-148 ``BOR_PDF_PAGE_DIR`` (suite-private
|
||||
— the seed's file assertions target it, the owner's default is
|
||||
never written)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gates (conftest pattern) — the app boots with the
|
||||
# same env shape the story suites use.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Leak guards (conftest pattern — the owner's local gitignored
|
||||
# .env cannot leak the scope).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
# Phase 126: the import scope WITH the ``pdf`` token (the walk is
|
||||
# blind to the fixture without it).
|
||||
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
|
||||
# Phase 140: the pictorial toggle EXPLICITLY on (the opt-in — see
|
||||
# the module docstring; without it the whole premise is void).
|
||||
env["BOR_PDF_PICTORIAL"] = "1"
|
||||
# Phase 148: the rendered-page copies' home (the seed asserts the
|
||||
# files here).
|
||||
env["BOR_PDF_PAGE_DIR"] = str(page_dir)
|
||||
# The embedded-image pipeline is out of scope (the fixtures'
|
||||
# figures are vector-drawn — no image doc either way).
|
||||
env["BOR_IMAGES"] = "false"
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int,
|
||||
pdfs_dir: Path, # noqa: ARG001 — the env only needs the homes; the dir's existence is the seed's job
|
||||
scratch: Path,
|
||||
page_dir: Path,
|
||||
) -> Iterator[str]:
|
||||
"""The app under test (all four tests): one module app with the
|
||||
phase-140/148 env (see :func:`_app_env`)."""
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, scratch, page_dir),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB + sync helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry (the E2E isolation pattern): the suites
|
||||
share one Postgres, and a leftover row would corrupt the counts.
|
||||
``sources_meta`` (the KB generation counter) resets with the KB —
|
||||
the sync re-creates the row."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries, sources_meta"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
|
||||
|
||||
|
||||
def _health_db(app_url: str) -> bool:
|
||||
try:
|
||||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||
return False
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 60.0) -> None:
|
||||
"""No background sync may leak across tests (the run state lives in
|
||||
the app's memory, and a still-running run would keep importing into
|
||||
the NEXT test's truncated KB): wait for the app's sync status to be
|
||||
non-running BEFORE the truncate (the phase-96 pattern)."""
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module", autouse=True)
|
||||
def _clean_kb(app_url: str) -> Iterator[None]:
|
||||
"""Module-end KB cleanup (the phase-96 pattern, module
|
||||
granularity): the four tests SHARE the module seed's single source
|
||||
+ two documents, so there is NO per-test truncate (it would wipe
|
||||
the seed from under the next test — the per-test pattern belongs to
|
||||
the suites where every test imports its own source); instead the
|
||||
shared KB is flushed ONCE, after the last test — no rows leak into
|
||||
the next suite's run. The pre-wait keeps a leaked sync (the module
|
||||
app is still up here) from racing the truncate."""
|
||||
yield
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
|
||||
|
||||
def _register_source(base_url: str, path: Path) -> None:
|
||||
"""Register the local directory through the authenticated API (the
|
||||
``test_local_directory_sources`` / phase-94 pattern — a plain
|
||||
directory is an API/DB-only operation; the page form is gone)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post("/api/git-sources", json={"kind": "local", "path": str(path)})
|
||||
assert r.status_code == 201, r.text
|
||||
|
||||
|
||||
def _started_at(body: dict[str, Any]) -> float:
|
||||
"""The status body's ``started_at`` as a wall-clock epoch (the
|
||||
server stamps it in UTC; this process runs on the same host —
|
||||
same clock)."""
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return 0.0
|
||||
try:
|
||||
return datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
|
||||
run reaches a terminal state (the ``test_garbage_pdf`` pattern over
|
||||
plain httpx — with the stale-terminal freshness gate: a terminal
|
||||
is accepted only after a running tick was observed or its
|
||||
``started_at`` post-dates this POST)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
t0 = time.time()
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
saw_running = False
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
state = body["state"]
|
||||
if state == "running":
|
||||
saw_running = True
|
||||
elif state in ("success", "failed") and (
|
||||
saw_running
|
||||
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
|
||||
):
|
||||
return body
|
||||
time.sleep(0.1)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The module seed: register the local dir + one sync (both PDFs), then
|
||||
# assert the DB rows and the rendered page copies
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded(
|
||||
app_url: str, pdfs_dir: Path, page_dir: Path
|
||||
) -> Iterator[dict[str, Any]]:
|
||||
"""Register the local directory source and run ONE real sync (the
|
||||
mock's vision model describes the two sparse pages in-process —
|
||||
the ``PICTORIAL_PAGE_MODE`` canned reply). Asserts the phase-148
|
||||
index-time contract: the pictorial row qualifies (``page_count ==
|
||||
3``, the canned blocks IN the stored content — the Q&A witness),
|
||||
the textheavy row does not (``page_count`` NULL, no copies), and
|
||||
the scratch page dir holds EXACTLY the pictorial doc's
|
||||
``<id>-p{1..3}.png`` (real PNGs, non-empty — no orphans, no
|
||||
textheavy copies). Yields both docs' ids + the page dir."""
|
||||
if not _health_db(app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
|
||||
_truncate_all()
|
||||
_register_source(app_url, pdfs_dir)
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["files"] == 2, detail
|
||||
assert detail["added"] == 2, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["errors"] == 0, detail
|
||||
|
||||
with SessionLocal() as db:
|
||||
pic = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == PICTORIAL_PATH
|
||||
)
|
||||
)
|
||||
heavy = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == TEXTHEAVY_PATH
|
||||
)
|
||||
)
|
||||
assert pic is not None, "the pictorial PDF must become a document"
|
||||
assert heavy is not None, "the textheavy PDF must become a document"
|
||||
# The pictorial class (IKEA shape): qualifies. The metadata
|
||||
# title (LOCKED A2); a TEXT document (no image doc, no image
|
||||
# path).
|
||||
assert pic.title == PICTORIAL_TITLE
|
||||
assert pic.is_image is False and pic.image_path is None
|
||||
assert pic.page_count == 3, (
|
||||
f"the qualifying PDF's page_count must be its page count: {pic.page_count}"
|
||||
)
|
||||
# The Q&A witness: the described pages ARE in the stored
|
||||
# content (the embeddings/summary ride on them — the strip
|
||||
# just does not display them).
|
||||
assert "---PAGE-1---" in pic.content and "---PAGE-3---" in pic.content
|
||||
assert BLOCK_TWO in pic.content and BLOCK_THREE in pic.content
|
||||
assert pic.summary, "the qualifying doc still gets its lite summary"
|
||||
# The negative class: NO strip (the pre-phase-148 viewer).
|
||||
assert heavy.title == TEXTHEAVY_TITLE
|
||||
assert heavy.page_count is None, (
|
||||
f"a text-heavy PDF must not qualify: page_count={heavy.page_count}"
|
||||
)
|
||||
assert heavy.summary, "the textheavy doc still gets its lite summary"
|
||||
pic_id, heavy_id = str(pic.id), str(heavy.id)
|
||||
|
||||
# The filesystem contract: EXACTLY the qualifying doc's three
|
||||
# copies (no textheavy copies, no orphans) — real, non-empty PNGs.
|
||||
pngs = sorted(page_dir.glob("*.png"))
|
||||
assert [p.name for p in pngs] == [
|
||||
f"{pic_id}-p1.png",
|
||||
f"{pic_id}-p2.png",
|
||||
f"{pic_id}-p3.png",
|
||||
], f"the page dir holds {sorted(p.name for p in pngs)}"
|
||||
for png in pngs:
|
||||
data = png.read_bytes()
|
||||
assert data[:8] == PNG_MAGIC and len(data) > 0, f"{png.name} is not a real PNG"
|
||||
assert not list(page_dir.glob(f"{heavy_id}-*")), (
|
||||
"the non-qualifying PDF must not have page copies"
|
||||
)
|
||||
yield {
|
||||
"pictorial_id": pic_id,
|
||||
"textheavy_id": heavy_id,
|
||||
"page_dir": page_dir,
|
||||
"title": PICTORIAL_TITLE,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared UI helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _drill_to_source(page: Page, name: str) -> None:
|
||||
"""Sources tree: top level → the source (the file table then holds
|
||||
its direct files — both fixtures sit at the directory root)."""
|
||||
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
|
||||
|
||||
|
||||
def _open_doc_modal(page: Page, doc_path: str) -> None:
|
||||
"""The house doc-modal idiom: the row's path link opens the
|
||||
same-page modal (no new tab)."""
|
||||
before = len(page.context.pages)
|
||||
page.locator(f'#docs-tbody a.doc-link:text-is("{doc_path}")').click()
|
||||
assert len(page.context.pages) == before, "the row link must not open a new tab"
|
||||
expect(page.locator("#doc-modal")).to_be_visible(timeout=30_000)
|
||||
|
||||
|
||||
def _wait_img_loaded(img: Locator) -> None:
|
||||
"""Wait until a LAZY ``<img>`` really loaded pixels: scroll it into
|
||||
the viewport (a below-the-fold lazy img is not fetched yet), then
|
||||
await the browser's own ``decode()`` promise (it rejects on a
|
||||
failed fetch — a 404 route) and assert ``naturalWidth > 0`` (a
|
||||
real PNG, not a zero-size error image)."""
|
||||
img.scroll_into_view_if_needed()
|
||||
img.evaluate("el => el.decode()")
|
||||
natural = img.evaluate("el => el.naturalWidth")
|
||||
assert natural > 0, f"the page image did not load (naturalWidth={natural})"
|
||||
|
||||
|
||||
def _assert_page_strip(
|
||||
content: Locator, title: str, doc_id: str, page_count: int = 3
|
||||
) -> None:
|
||||
"""The phase-148 strip contract inside a content container (the
|
||||
ONE ``renderDocument`` core — modal and full page, no
|
||||
per-surface drift): a ``.doc-pages`` strip with exactly
|
||||
*page_count* ``.doc-page`` figures; each figure's lazy ``<img>``
|
||||
is LOADED (the route served a real PNG) with the ``Page N of
|
||||
<title>`` alt and the exact 1-based route src, and the
|
||||
figcaption reads ``Page N``; the text content block AND the
|
||||
Summary panel are SUPPRESSED (the descriptions stay in the DB,
|
||||
not displayed)."""
|
||||
strip = content.locator(".doc-pages")
|
||||
expect(strip).to_have_count(1)
|
||||
figures = strip.locator(".doc-page")
|
||||
expect(figures).to_have_count(page_count)
|
||||
for i in range(page_count):
|
||||
n = i + 1
|
||||
fig = figures.nth(i)
|
||||
img = fig.locator("img")
|
||||
# The figure still carries its <img> (a lost copy would have
|
||||
# been swapped for the "Page N unavailable" note instead).
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_have_attribute("loading", "lazy")
|
||||
expect(img).to_have_attribute("alt", f"Page {n} of {title}")
|
||||
expect(img).to_have_attribute(
|
||||
"src", f"/api/documents/{doc_id}/pages/{n}"
|
||||
)
|
||||
_wait_img_loaded(img)
|
||||
expect(fig.locator("figcaption.doc-page-num")).to_have_text(f"Page {n}")
|
||||
# The suppression contract (the owner's TODO L5): no text content
|
||||
# block, no Summary panel — the meta badge row is asserted by the
|
||||
# caller (it renders exactly as today).
|
||||
expect(content.locator("pre.doc-raw")).to_have_count(0)
|
||||
expect(content.locator(".doc-md")).to_have_count(0)
|
||||
expect(content.locator(".doc-summary")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The pictorial PDF: the modal shows its three rendered pages
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pictorial_pdf_shows_pages_in_modal(
|
||||
page: Page, app_url: str, seeded: dict[str, Any]
|
||||
) -> None:
|
||||
"""The IKEA-shaped PDF opens in the doc modal as a strip of its
|
||||
REAL pages (the owner's L5 ask): ``.doc-pages`` with exactly three
|
||||
``.doc-page`` figures — each a loaded lazy ``<img>`` (naturalWidth
|
||||
> 0 proves the full pipeline: sync → render → persist → route →
|
||||
viewer) with the ``Page N of <title>`` alt and the ``Page N``
|
||||
caption — and the suppression contract: NO text content
|
||||
(``.doc-raw``), NO Summary panel (the mock's page descriptions are
|
||||
in the DB content, nowhere in the rendered modal); the meta badge
|
||||
row (source / Created / Indexed) renders as today."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
||||
_open_doc_modal(page, PICTORIAL_PATH)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(
|
||||
seeded["title"]
|
||||
)
|
||||
|
||||
content = page.locator("#doc-modal-content")
|
||||
_assert_page_strip(content, seeded["title"], seeded["pictorial_id"])
|
||||
|
||||
# The meta badge row is untouched by the strip (the pre-phase row).
|
||||
meta = page.locator("#doc-modal-meta")
|
||||
expect(meta.locator(".doc-source-badge")).to_have_text(SOURCE_NAME)
|
||||
expect(meta.locator(".doc-created")).to_contain_text("Created")
|
||||
expect(meta.locator(".doc-indexed")).to_contain_text("Indexed")
|
||||
|
||||
# The suppression witness: the mock's canned page descriptions sit
|
||||
# in the DB (the seed asserted it) but are NOT displayed.
|
||||
assert PICTORIAL_PAGE_ANSWER not in content.inner_text(), (
|
||||
"the described pages' text is in the DB for Q&A — it must not "
|
||||
"be shown to the user (the owner's contract)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. The text-heavy PDF: byte-identical to pre-phase
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_textheavy_pdf_unchanged(
|
||||
page: Page, app_url: str, seeded: dict[str, Any]
|
||||
) -> None:
|
||||
"""The zero-candidates PDF renders the PRE-PHASE shape: the
|
||||
content in ``pre.doc-raw`` (both pages, verbatim), the Summary
|
||||
panel (the mock's lite digest), NO ``.doc-pages`` strip — and the
|
||||
payload's ``page_urls`` KEY is ABSENT from the content endpoint's
|
||||
JSON (never ``null`` — the omission pin the viewer's strip gate
|
||||
rides on; the non-qualifying doc's payload is byte-identical to
|
||||
pre-phase)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
_open_doc_modal(page, TEXTHEAVY_PATH)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(TEXTHEAVY_TITLE)
|
||||
|
||||
content = page.locator("#doc-modal-content")
|
||||
# No strip for the negative class…
|
||||
expect(content.locator(".doc-pages")).to_have_count(0)
|
||||
# …the pre-phase raw rendering (both pages, the stored content
|
||||
# verbatim).
|
||||
pre = content.locator("pre.doc-raw")
|
||||
expect(pre).to_have_count(1)
|
||||
expect(pre).to_contain_text("---PAGE-1---")
|
||||
expect(pre).to_contain_text("Paragraph 1:")
|
||||
expect(pre).to_contain_text("---PAGE-2---")
|
||||
expect(pre).to_contain_text("Paragraph 2:")
|
||||
# …and the Summary panel (a non-empty lite summary).
|
||||
expect(content.locator(".doc-summary")).to_have_count(1)
|
||||
expect(content.locator(".doc-summary-text")).not_to_be_empty()
|
||||
|
||||
# The payload pin: the page_urls KEY is ABSENT (the admin cookie
|
||||
# rides the browser context — the content endpoint is
|
||||
# require_user-gated).
|
||||
r = page.request.get(
|
||||
f"{app_url}/api/documents/content",
|
||||
params={"source": SOURCE_NAME, "path": TEXTHEAVY_PATH},
|
||||
)
|
||||
assert r.status == 200, r.text
|
||||
assert "page_urls" not in r.json(), (
|
||||
f"the non-qualifying doc's payload must not carry page_urls: "
|
||||
f"{r.json().get('page_urls')!r}"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. The full page surface: the SAME strip, no per-surface drift
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_full_page_surface(page: Page, app_url: str, seeded: dict[str, Any]) -> None:
|
||||
"""The dedicated ``/document.html`` page (admin-logged-in — the
|
||||
phase-79 gate satisfied) renders the SAME strip through the ONE
|
||||
shared ``renderDocument`` core: the same three loaded figures with
|
||||
captions/alt texts, the same suppression (no text content, no
|
||||
Summary panel), the meta row as today."""
|
||||
login(page, app_url, next="/")
|
||||
page.goto(
|
||||
f"{app_url}/document.html?source={quote(SOURCE_NAME)}"
|
||||
f"&path={quote(PICTORIAL_PATH)}"
|
||||
)
|
||||
expect(page.locator("#doc-title")).to_have_text(seeded["title"])
|
||||
_assert_page_strip(
|
||||
page.locator("#doc-content"), seeded["title"], seeded["pictorial_id"]
|
||||
)
|
||||
# The meta row on the full page is the pre-phase shape too.
|
||||
expect(page.locator("#doc-meta .doc-source-badge")).to_have_text(SOURCE_NAME)
|
||||
expect(page.locator("#doc-meta .doc-indexed")).to_contain_text("Indexed")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. The page bytes route: public PNG with the image route's headers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_page_route_serves_png(
|
||||
page: Page, app_url: str, seeded: dict[str, Any]
|
||||
) -> None:
|
||||
"""An ANONYMOUS (no-login — this test's fresh context never signs
|
||||
in) direct request on the first ``page_urls`` entry: 200,
|
||||
``image/png``, the ``Cache-Control: private, max-age=3600``
|
||||
header (the image route's header), a non-empty PNG body byte-equal
|
||||
to the stored copy (the public uuid4-credential bytes route — the
|
||||
document id IS the credential — from the UI's perspective)."""
|
||||
doc_id = seeded["pictorial_id"]
|
||||
r = page.request.get(f"{app_url}/api/documents/{doc_id}/pages/1")
|
||||
assert r.status == 200, r.text
|
||||
assert r.headers["content-type"] == "image/png", r.headers
|
||||
assert r.headers["cache-control"] == "private, max-age=3600", r.headers
|
||||
body = r.body()
|
||||
assert body, "the page route served an empty body"
|
||||
assert body[:8] == PNG_MAGIC, "the page route served non-PNG bytes"
|
||||
# The exact stored bytes (the route streams the persistent copy).
|
||||
stored = (seeded["page_dir"] / f"{doc_id}-p1.png").read_bytes()
|
||||
assert body == stored, "the route's bytes are not the stored copy's bytes"
|
||||
@@ -24,13 +24,19 @@ format — the upload gate admits it) and arriving through the REAL admin
|
||||
upload flow → the RAG page's Sync button:
|
||||
|
||||
* (a) the document appears on the Sources page and the document viewer
|
||||
renders the stored content: page 1's prose WITHOUT an illustration
|
||||
block, page 2 WITH ``[Page 2 illustration: <the mock's canned
|
||||
PICTORIAL_PAGE_ANSWER>]`` (the task-02 ``PICTORIAL_PAGE_MODE`` branch
|
||||
— deterministic; the stored content is asserted BYTE-EXACT against
|
||||
the fixture's own extraction in the seed — the candidates are computed
|
||||
from ``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
|
||||
hardcoded page list);
|
||||
shows the PDF's ACTUAL pages (phase 148 — a described sparse page
|
||||
makes the doc page-QUALIFYING: the page strip REPLACES the text
|
||||
content AND the Summary panel, the owner's TODO L5 contract): two
|
||||
``.doc-page`` figures with loaded lazy images + ``Page N`` captions,
|
||||
NO ``.doc-raw``, NO ``.doc-summary``; the described content itself
|
||||
(page 1's prose WITHOUT an illustration block, page 2 WITH
|
||||
``[Page 2 illustration: <the mock's canned PICTORIAL_PAGE_ANSWER>]``
|
||||
— the task-02 ``PICTORIAL_PAGE_MODE`` branch, deterministic) stays in
|
||||
the DB: the seed asserts it BYTE-EXACT against the fixture's own
|
||||
extraction (the candidates computed from
|
||||
``extract_pdf_page_texts`` + ``pictorial_pages`` IN the test, no
|
||||
hardcoded page list) and the chat test below proves it still reaches
|
||||
the model;
|
||||
* (b) a chat question that ONLY the illustration answers ("how are the
|
||||
two parts joined on page two?" — the phrase lives NOWHERE in the text
|
||||
layer) grounds (the described page's embedding retrieves — the mock's
|
||||
@@ -77,11 +83,14 @@ Test → observable mapping (Playwright Mapping Rule):
|
||||
Sources page reads 1 doc / 3 chunks (2 page-anchored content chunks +
|
||||
the summary chunk — no image doc), the drill-down table lists the PDF
|
||||
as a plain text row (LOCKED A4 — a text document, no thumbnail), and
|
||||
the same-page document viewer (the row's path link) renders the
|
||||
stored content under the METADATA title: the page-1 prose (NO
|
||||
``[Page 1 illustration`` block), the page-2 marker + its
|
||||
``[Page 2 illustration: <canned>]`` block, and the Summary panel
|
||||
(a non-image doc with a non-empty summary).
|
||||
the same-page document viewer (the row's path link) shows the
|
||||
PHASE-148 PAGE STRIP under the METADATA title (the fixture is
|
||||
page-qualifying — the pictorial pass described its sparse page):
|
||||
exactly two ``.doc-page`` figures, each a LOADED lazy ``<img>``
|
||||
(naturalWidth > 0) with the ``Page N of <title>`` alt + the
|
||||
``Page N`` caption, NO ``.doc-raw`` / NO ``.doc-summary`` (the
|
||||
page descriptions stay in the DB for Q&A, not displayed — the
|
||||
owner's contract).
|
||||
2. ``test_chat_question_grounds_and_echoes_the_illustration`` — the raw
|
||||
SSE over the scripted seeded-read question (httpx — the ``done``
|
||||
event is the wire lens the task pins; the UI flow's premise is the
|
||||
@@ -554,9 +563,14 @@ def seeded(
|
||||
"described sparse page — the pictorial pass misfired"
|
||||
)
|
||||
# The non-markdown doc got its lite summary (the mock's
|
||||
# SUMMARY_MODE digest) — the summary chunk + the viewer's
|
||||
# Summary panel exist.
|
||||
# SUMMARY_MODE digest) — the summary chunk exists (the
|
||||
# viewer's Summary panel is suppressed by the phase-148
|
||||
# strip — the summary itself stays in the DB).
|
||||
assert doc.summary
|
||||
# Phase 148: the described sparse page makes this doc
|
||||
# page-QUALIFYING — the strip's payload (page_urls) rides
|
||||
# on page_count (the 2-page fixture's count).
|
||||
assert doc.page_count == 2
|
||||
finally:
|
||||
page.close()
|
||||
yield {"title": PDF_TITLE}
|
||||
@@ -574,12 +588,18 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
content chunks + the phase-30 summary chunk — no image doc; the
|
||||
vector figure is invisible to the embedded-image pipeline). The
|
||||
drill-down table lists the PDF as a plain text row (LOCKED A4), and
|
||||
the same-page document viewer (the row's path link, phase 26)
|
||||
renders the stored content under the METADATA title: page 1's prose
|
||||
WITHOUT an illustration block (not a candidate) and page 2 WITH
|
||||
``[Page 2 illustration: <canned>]`` (the described sparse page —
|
||||
LOCKED A4's user surface), plus the Summary panel (a non-image doc
|
||||
with a non-empty summary)."""
|
||||
the same-page document viewer (the row's path link, phase 26) shows
|
||||
the PHASE-148 PAGE STRIP under the METADATA title: the fixture is
|
||||
page-qualifying (the pictorial pass described its sparse page), so
|
||||
the strip REPLACES the text content AND the Summary panel — two
|
||||
``.doc-page`` figures, each a loaded lazy ``<img>`` with the
|
||||
``Page N of <title>`` alt + the ``Page N`` caption, NO
|
||||
``.doc-raw``, NO ``.doc-summary``. The described content (page 1's
|
||||
prose WITHOUT an illustration block, page 2 WITH
|
||||
``[Page 2 illustration: <canned>]`` — LOCKED A4) stays in the DB
|
||||
for Q&A: the seed asserted it byte-exact, the chat test proves it
|
||||
reaches the model — it is simply not displayed (the owner's
|
||||
contract)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("3")
|
||||
@@ -596,27 +616,44 @@ def test_sources_page_and_viewer_show_the_described_page(
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
||||
|
||||
raw = modal.locator("pre.doc-raw")
|
||||
# Page 1 — the prose WITHOUT any illustration block. The viewer
|
||||
# renders the content verbatim (``insert_textbox`` wraps mid-
|
||||
# sentence, so compare whitespace-normalized).
|
||||
expect(raw).to_contain_text("---PAGE-1---")
|
||||
rendered = " ".join(raw.inner_text().split())
|
||||
assert " ".join(PAGE_ONE.split()) in rendered, (
|
||||
"the page-1 prose is not what the viewer renders"
|
||||
)
|
||||
assert "[Page 1 illustration" not in rendered, (
|
||||
"page 1 (over the sparseness bar) must carry NO illustration block"
|
||||
)
|
||||
# Page 2 — the sparse page WITH the described block (a) — the mock's
|
||||
# canned reply, deterministic.
|
||||
expect(raw).to_contain_text("---PAGE-2---")
|
||||
expect(raw).to_contain_text(PAGE_TWO)
|
||||
expect(raw).to_contain_text(ILLUSTRATION_BLOCK)
|
||||
# The Summary panel renders (a non-image doc with a non-empty
|
||||
# summary); no image block (the PDF is not an image document).
|
||||
expect(modal.locator(".doc-summary")).to_have_count(1)
|
||||
# Phase 148: the page-qualifying fixture shows its ACTUAL pages —
|
||||
# a .doc-pages strip with exactly two .doc-page figures (the
|
||||
# 2-page fixture), each a LOADED lazy <img> (naturalWidth > 0 —
|
||||
# the route served a real PNG) with the "Page N of <title>" alt +
|
||||
# the "Page N" caption. The text content AND the Summary panel are
|
||||
# SUPPRESSED (the owner's contract — the descriptions stay in the
|
||||
# DB for Q&A, not displayed).
|
||||
strip = modal.locator(".doc-pages")
|
||||
expect(strip).to_have_count(1)
|
||||
figures = strip.locator(".doc-page")
|
||||
expect(figures).to_have_count(2)
|
||||
for i in range(2):
|
||||
n = i + 1
|
||||
img = figures.nth(i).locator("img")
|
||||
# The figure still carries its <img> (a lost copy would have
|
||||
# been swapped for the "Page N unavailable" note instead).
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_have_attribute("loading", "lazy")
|
||||
expect(img).to_have_attribute("alt", f"Page {n} of {PDF_TITLE}")
|
||||
img.scroll_into_view_if_needed()
|
||||
img.evaluate("el => el.decode()")
|
||||
assert img.evaluate("el => el.naturalWidth") > 0, (
|
||||
f"page {n} image did not load (the route must serve a real PNG)"
|
||||
)
|
||||
expect(figures.nth(i).locator("figcaption.doc-page-num")).to_have_text(
|
||||
f"Page {n}"
|
||||
)
|
||||
# The suppression contract: no text content block, no Summary
|
||||
# panel, no image block (the PDF is not an image document).
|
||||
expect(modal.locator("pre.doc-raw")).to_have_count(0)
|
||||
expect(modal.locator(".doc-summary")).to_have_count(0)
|
||||
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
||||
# The canned description is in the DB (the seed asserted it)
|
||||
# but nowhere in the rendered modal.
|
||||
assert PICTORIAL_PAGE_ANSWER not in modal.inner_text(), (
|
||||
"the described page's text is in the DB for Q&A — it must not "
|
||||
"be shown to the user (the owner's contract)"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -3,6 +3,15 @@ and GET /api/docs/tree (phase 97, task 02 — the full recursive KB tree
|
||||
the RAG view renders: shape, ordering, counts, summaries, the 403 gate,
|
||||
the superset rule, and the stat-walk equivalence with ``GET /api/docs``).
|
||||
|
||||
Also the BYTES routes and their content-payload affordances: the
|
||||
phase-122 (task 04) image route (``GET /api/documents/{id}/image``
|
||||
— exact bytes, the extension's Content-Type, the 404 matrix, public
|
||||
per phase 124 A4) and the phase-148 (task 02) PDF page route
|
||||
(``GET /api/documents/{id}/pages/{n}`` — the same trust model and 404
|
||||
posture for a page-qualifying document's rendered page PNGs) + the
|
||||
content endpoint's ``image_url`` / ``page_urls`` keys (present for the
|
||||
qualifying row, ABSENT otherwise — never ``null``).
|
||||
|
||||
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
@@ -1370,6 +1379,241 @@ def test_content_endpoint_exposes_the_image_affordance(
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 148 (task 02) — the PDF page BYTES route + the content payload's
|
||||
# ``page_urls`` affordance
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _page_settings(pages_dir: Path) -> Settings:
|
||||
"""Fresh settings (the ``.env`` file ignored) with the phase-148
|
||||
``pdf_page_dir`` pinned to *pages_dir* — the page route reads the
|
||||
setting at request time, and the copies under test never land in
|
||||
the real home (the ``get_settings`` monkeypatch pattern)."""
|
||||
return Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
pdf_page_dir=str(pages_dir),
|
||||
)
|
||||
|
||||
|
||||
def _seed_page_doc(
|
||||
db,
|
||||
pages_dir: Path,
|
||||
*,
|
||||
source: str = "PdfSrc",
|
||||
path: str = "besta.pdf",
|
||||
page_count: int | None = 2,
|
||||
write_copies: bool = True,
|
||||
) -> Document:
|
||||
"""One PDF document row with ``page_count`` + its rendered page
|
||||
copies (the importer's ``pdf_page_dir`` layout —
|
||||
``<doc-id>-p{N}.png``, N 1-based); page N is the ``PNG_1X1 + N``
|
||||
sentinel bytes, so a page mix-up between the two copies is caught.
|
||||
``page_count=None`` is a non-qualifying PDF (no copies);
|
||||
``write_copies=False`` leaves a qualifying row without its files
|
||||
(the lost-copy corner)."""
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source=source,
|
||||
path=path,
|
||||
full_path=f"/tmp/{path}",
|
||||
title=path.rsplit(".", 1)[0],
|
||||
content="[Page 1 illustration: the parts tray] [Page 2 illustration: step 1]",
|
||||
content_hash=hashlib.sha256(path.encode("utf-8")).hexdigest(),
|
||||
indexed_at=datetime.now(UTC),
|
||||
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
||||
summary="IKEA BESTA frame assembly manual (page descriptions).",
|
||||
page_count=page_count,
|
||||
)
|
||||
db.add(doc)
|
||||
db.flush()
|
||||
if write_copies and page_count:
|
||||
pages_dir.mkdir(parents=True, exist_ok=True)
|
||||
for n in range(1, page_count + 1):
|
||||
(pages_dir / f"{doc.id}-p{n}.png").write_bytes(PNG_1X1 + str(n).encode("ascii"))
|
||||
db.add_all(
|
||||
[
|
||||
Chunk(
|
||||
document_id=doc.id, position=0, content=doc.content, embedding=[0.01] * 768
|
||||
),
|
||||
Chunk(
|
||||
document_id=doc.id,
|
||||
position=-1,
|
||||
content=doc.summary,
|
||||
embedding=[0.01] * 768,
|
||||
is_summary=True,
|
||||
),
|
||||
]
|
||||
)
|
||||
db.commit()
|
||||
return doc
|
||||
|
||||
|
||||
def test_page_route_serves_exact_bytes_with_png_type(
|
||||
admin_client: TestClient, db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The serve contract: each page of a qualifying doc streams the
|
||||
EXACT stored PNG bytes (a per-page sentinel — a page mix-up is
|
||||
caught) as ``image/png`` (a page is always a PNG —
|
||||
``render_pdf_pages``'s ``tobytes("png")``) with
|
||||
``Cache-Control: private, max-age=3600`` (the image route's
|
||||
header — content-addressed by the document id + page number).
|
||||
The path is built ONLY from the row's id + ``n`` (the row
|
||||
derives it; ``n`` is the int path parameter) — the copies
|
||||
written under the tmp ``pdf_page_dir`` are what come back."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
pages_dir = tmp_path / "pages"
|
||||
monkeypatch.setattr(docs_api, "get_settings", lambda: _page_settings(pages_dir))
|
||||
doc = _seed_page_doc(db, pages_dir, page_count=2)
|
||||
for n in (1, 2):
|
||||
r = admin_client.get(f"/api/documents/{doc.id}/pages/{n}")
|
||||
assert r.status_code == 200, n
|
||||
assert r.headers["content-type"] == "image/png"
|
||||
assert r.headers["cache-control"] == "private, max-age=3600"
|
||||
assert r.content == PNG_1X1 + str(n).encode("ascii") # the exact stored bytes
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_page_route_404_matrix(
|
||||
admin_client: TestClient, db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Every non-servable case is 404 ``document not found`` (the
|
||||
router's unknown-document shape — the same detail string the
|
||||
content endpoint uses): a missing id, a MALFORMED id (an
|
||||
unparseable string maps here, not to a 422 — the image route's
|
||||
mapping: a guessed id is an unknown document), a text doc, an
|
||||
image doc, a PDF row whose ``page_count`` is NULL (a
|
||||
non-qualifying doc — the route knows nothing of its pages),
|
||||
``n`` outside ``1..page_count`` (``n=0``/``n=-1`` below the range
|
||||
— the int parameter makes fractions impossible — and ``n=3``
|
||||
above a 2-page doc), and a qualifying row whose copy was lost on
|
||||
disk (defensive — the row exists, the bytes don't; the viewer's
|
||||
onerror note covers it). The int parameter is the no-traversal
|
||||
pin: a NON-int ``n`` 422s at the router, and no request value
|
||||
ever reaches a filesystem path (the out-of-range 404s show the
|
||||
path is bounded by the ROW's extent)."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
pages_dir = tmp_path / "pages"
|
||||
monkeypatch.setattr(docs_api, "get_settings", lambda: _page_settings(pages_dir))
|
||||
_seed_doc(db, "TextSrc", "note.md", "Note", 1, datetime.now(UTC))
|
||||
db.commit() # the module's _seed_doc leaves the row uncommitted
|
||||
text_doc_id = db.scalar(
|
||||
select(Document.id).where(Document.source == "TextSrc", Document.path == "note.md")
|
||||
)
|
||||
img = _seed_image_doc(db, tmp_path, path="pic.png")
|
||||
plain = _seed_page_doc(db, pages_dir, path="texty.pdf", page_count=None)
|
||||
lost = _seed_page_doc(db, pages_dir, path="lost.pdf")
|
||||
assert (pages_dir / f"{lost.id}-p1.png").is_file() # the copies were written
|
||||
(pages_dir / f"{lost.id}-p1.png").unlink() # the copy is lost (the row remains)
|
||||
qualifying = _seed_page_doc(db, pages_dir, path="besta.pdf")
|
||||
|
||||
for doc_id, n in (
|
||||
(str(uuid.uuid4()), 1), # missing id
|
||||
("abc", 1), # malformed id → 404, not 422
|
||||
(str(text_doc_id), 1), # text doc
|
||||
(str(img.id), 1), # image doc
|
||||
(str(plain.id), 1), # PDF, page_count NULL (non-qualifying)
|
||||
(str(lost.id), 1), # qualifying doc, copy lost on disk
|
||||
(str(qualifying.id), 0), # below range
|
||||
(str(qualifying.id), -1), # negative — the int parse allows it, the range rejects it
|
||||
(str(qualifying.id), 3), # above range (page_count 2)
|
||||
):
|
||||
r = admin_client.get(f"/api/documents/{doc_id}/pages/{n}")
|
||||
assert r.status_code == 404, (doc_id, n)
|
||||
assert r.json() == {"detail": "document not found"}, (doc_id, n)
|
||||
|
||||
# The int parameter (the no-traversal pin): a NON-int n is a
|
||||
# router-level 422 — no other ``n`` shape reaches the handler.
|
||||
r = admin_client.get(f"/api/documents/{qualifying.id}/pages/abc")
|
||||
assert r.status_code == 422
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_page_route_is_public_like_the_shared_chat_content(
|
||||
admin_client: TestClient, db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The image route's phase-124 A4 trust model verbatim: a shared
|
||||
chat can cite a page FIGURE (the viewer's strip, task 03), so the
|
||||
BYTES serve ANONYMOUSLY — the unguessable ``uuid4`` document id IS
|
||||
the credential, no enumeration value (PLAN A10: the shared chats
|
||||
are the app's ONLY anonymous surface). An ANONYMOUS caller (a
|
||||
FRESH client — no session) gets the page PNG, and a signed-in
|
||||
caller gets the same. The document's TEXT (the content endpoint)
|
||||
stays user-gated — only the page bytes are public."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
pages_dir = tmp_path / "pages"
|
||||
monkeypatch.setattr(docs_api, "get_settings", lambda: _page_settings(pages_dir))
|
||||
doc = _seed_page_doc(db, pages_dir)
|
||||
anonymous = TestClient(fastapi_app)
|
||||
r = anonymous.get(f"/api/documents/{doc.id}/pages/1")
|
||||
assert r.status_code == 200
|
||||
assert r.headers["content-type"] == "image/png"
|
||||
assert r.content == PNG_1X1 + b"1"
|
||||
# The signed-in client (admin) gets the same bytes.
|
||||
r_admin = admin_client.get(f"/api/documents/{doc.id}/pages/1")
|
||||
assert r_admin.status_code == 200
|
||||
assert r_admin.content == PNG_1X1 + b"1"
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def test_content_endpoint_exposes_the_page_urls_affordance(
|
||||
admin_client: TestClient, db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The content endpoint (the viewer's data source — task 03's
|
||||
strip reads from it): a page-qualifying PDF (``page_count`` set)
|
||||
carries ``page_urls`` = the EXACT 1-based route list over
|
||||
``1..page_count`` (the count == ``page_count`` — a per-page render
|
||||
gap is still listed; the route 404s that N); text docs, image
|
||||
docs, and non-qualifying PDFs (``page_count`` NULL) keep the key
|
||||
ABSENT from the JSON — never ``null`` (the ``image_url``
|
||||
omission rule — a non-qualifying doc's payload is byte-identical
|
||||
to pre-phase)."""
|
||||
_cleanup_kb(db)
|
||||
try:
|
||||
pages_dir = tmp_path / "pages"
|
||||
doc = _seed_page_doc(db, pages_dir, page_count=3)
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "PdfSrc", "path": "besta.pdf"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["page_urls"] == [
|
||||
f"/api/documents/{doc.id}/pages/1",
|
||||
f"/api/documents/{doc.id}/pages/2",
|
||||
f"/api/documents/{doc.id}/pages/3",
|
||||
] # the exact 1-based list, count == page_count
|
||||
|
||||
_seed_doc(db, "TextSrc", "note.md", "Note", 1, datetime.now(UTC))
|
||||
db.commit() # the app's endpoint session reads committed data only
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "TextSrc", "path": "note.md"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
assert "page_urls" not in r.json() # absent — never null (text doc)
|
||||
|
||||
_seed_image_doc(db, tmp_path, source="ImgSrc", path="pic.png")
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "ImgSrc", "path": "pic.png"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
assert "page_urls" not in r.json() # absent — never null (image doc)
|
||||
|
||||
_seed_page_doc(db, pages_dir, path="texty.pdf", page_count=None)
|
||||
r = admin_client.get(
|
||||
"/api/documents/content", params={"source": "PdfSrc", "path": "texty.pdf"}
|
||||
)
|
||||
assert r.status_code == 200
|
||||
assert "page_urls" not in r.json() # absent — never null (non-qualifying PDF)
|
||||
finally:
|
||||
_cleanup_kb(db)
|
||||
|
||||
|
||||
def _tree_file_nodes_all(sources) -> list[dict]:
|
||||
"""Every file node of a tree response, walked recursively (all
|
||||
sources — env-registered 0-document sources may join the response
|
||||
|
||||
@@ -0,0 +1,351 @@
|
||||
"""Integration: migration 0026 (documents.page_count) schema contract
|
||||
(phase 148, task 01).
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the house pattern of
|
||||
``test_migration_0025.py`` (information_schema assertions on the state
|
||||
the migration must leave). The tests target the 0025 → 0026 step
|
||||
explicitly so later migrations cannot break the pins:
|
||||
|
||||
* upgrade 0025 → 0026 → ``page_count`` exists with the full contract —
|
||||
INTEGER, NULLABLE, NO server default (NULL = "not page-qualifying"
|
||||
for every pre-phase row and every fail-soft miss) — and is UNINDEXED
|
||||
(a display fact, never a query surface — the page route of task 02
|
||||
reads it from the ROW after the uuid4 id lookup, the ``file_hash``
|
||||
convention of 0025) while the 0025 ``documents`` schema survives —
|
||||
in particular ``content_hash`` stays NOT NULL and INDEXED (the
|
||||
retriever's dedupe/caching identity, untouched) and ``file_hash``
|
||||
keeps its 0025 contract;
|
||||
* rows inserted while the DB is at 0025 backfill ``page_count`` to
|
||||
NULL — the pre-phase shape, no backfill, no default;
|
||||
* the ORM contract agrees: a freshly inserted ``Document`` with no
|
||||
``page_count`` reads ``page_count is None`` through a fresh session
|
||||
(the pre-phase shape), and an explicit count round-trips;
|
||||
* downgrade to 0025 → the column is GONE (A13) while the row + its
|
||||
content survive; upgrade back to 0026 → the column is back
|
||||
(round-trip).
|
||||
|
||||
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||
fails or the process is interrupted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.models import Document
|
||||
|
||||
SOURCE = "mig0026"
|
||||
CONTENT = "content here"
|
||||
CONTENT_HASH = hashlib.sha256(CONTENT.encode()).hexdigest()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run); teardown
|
||||
upgrades to head no matter what happened, so the dev DB is never
|
||||
left below head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
# Release the test session's open transaction BEFORE the repair
|
||||
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
|
||||
# on ``documents``, which would deadlock the repair's
|
||||
# ``ALTER TABLE`` forever.
|
||||
db.rollback()
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||
"""(data_type, is_nullable, column_default) for one documents
|
||||
column."""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT data_type, is_nullable, column_default"
|
||||
" FROM information_schema.columns"
|
||||
" WHERE table_name = 'documents' AND column_name = :c"
|
||||
),
|
||||
{"c": column},
|
||||
).fetchone()
|
||||
return tuple(row) if row is not None else None
|
||||
|
||||
|
||||
def _index_covering(db: Session, column: str) -> str | None:
|
||||
"""The name of any documents index whose definition mentions the
|
||||
column (None = UNINDEXED)."""
|
||||
return db.execute(
|
||||
text(
|
||||
"SELECT indexname FROM pg_indexes"
|
||||
" WHERE tablename = 'documents' AND indexdef ILIKE :like"
|
||||
),
|
||||
{"like": f"%{column}%"},
|
||||
).scalar()
|
||||
|
||||
|
||||
def _insert_sql(db: Session, path: str) -> uuid.UUID:
|
||||
"""Insert one documents row with the PRE-0026 column set (the 0025
|
||||
shape — the page_count column, when present, is omitted so a NULL
|
||||
backfill is what the row reads)."""
|
||||
doc_id = uuid.uuid4()
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO documents (id, source, path, full_path, title, content,"
|
||||
" content_hash) VALUES (:id, :s, :p, :fp, :t, :c, :h)"
|
||||
),
|
||||
{
|
||||
"id": doc_id,
|
||||
"s": SOURCE,
|
||||
"p": path,
|
||||
"fp": f"/tmp/{path}",
|
||||
"t": f"Doc {path}",
|
||||
"c": CONTENT,
|
||||
"h": CONTENT_HASH,
|
||||
},
|
||||
)
|
||||
db.commit()
|
||||
return doc_id
|
||||
|
||||
|
||||
def _delete(db: Session, doc_id: uuid.UUID) -> None:
|
||||
db.execute(text("DELETE FROM documents WHERE id = :id"), {"id": doc_id})
|
||||
db.commit()
|
||||
|
||||
|
||||
def _assert_0025_schema_survives(db: Session) -> None:
|
||||
"""The 0025 documents contract must survive the additive upgrade —
|
||||
in particular ``content_hash`` stays NOT NULL and indexed (the
|
||||
retriever's identity — untouched by phase 148) and ``file_hash``
|
||||
keeps its 0025 contract (VARCHAR(64), nullable, unindexed)."""
|
||||
content = _column(db, "content_hash")
|
||||
assert content is not None and content[0] == "character varying" and content[1] == "NO", (
|
||||
"documents.content_hash (0001) must keep its 0025 contract"
|
||||
)
|
||||
assert (
|
||||
_index_covering(db, "content_hash") is not None
|
||||
), "the ix_documents_content_hash index must survive the upgrade"
|
||||
|
||||
file_hash = _column(db, "file_hash")
|
||||
assert file_hash is not None and file_hash[0] == "character varying", (
|
||||
"documents.file_hash (0025) must survive the upgrade"
|
||||
)
|
||||
assert file_hash[1] == "YES", "file_hash must stay NULLABLE"
|
||||
assert file_hash[2] is None, "file_hash must keep NO server default"
|
||||
assert _index_covering(db, "file_hash") is None, (
|
||||
"file_hash must stay UNINDEXED"
|
||||
)
|
||||
|
||||
indexed = _column(db, "indexed_at")
|
||||
assert indexed is not None and indexed[0] == "timestamp with time zone", (
|
||||
"documents.indexed_at (0001) must survive the upgrade"
|
||||
)
|
||||
is_image = _column(db, "is_image")
|
||||
assert is_image is not None and is_image[0] == "boolean" and is_image[1] == "NO", (
|
||||
"documents.is_image (0022) must survive the upgrade"
|
||||
)
|
||||
image_path = _column(db, "image_path")
|
||||
assert image_path is not None and image_path[1] == "YES", (
|
||||
"documents.image_path (0022) must survive the upgrade"
|
||||
)
|
||||
|
||||
|
||||
def test_upgrade_to_0026_adds_page_count(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0025 → 0026: ``page_count`` exists with the full
|
||||
contract (INTEGER, NULLABLE, NO server default — NULL = "not
|
||||
page-qualifying" for every pre-phase row — and UNINDEXED: a
|
||||
display fact, never a query surface), is ABSENT at 0025, and every
|
||||
pre-0026 row backfills ``page_count`` to NULL (no backfill
|
||||
possible — a migration cannot know which documents qualified) —
|
||||
while the 0025 table contract (incl. the INDEXED ``content_hash``
|
||||
and the unindexed ``file_hash``) survives."""
|
||||
command.downgrade(alembic, "0025") # start from the pre-0026 state
|
||||
assert _version(db) == "0025"
|
||||
assert _column(db, "page_count") is None, "page_count must be absent at 0025"
|
||||
|
||||
pre_id = _insert_sql(db, "pre.md") # the 0025 column set
|
||||
try:
|
||||
command.upgrade(alembic, "0026")
|
||||
assert _version(db) == "0026", "alembic_version must be at 0026"
|
||||
|
||||
col = _column(db, "page_count")
|
||||
assert col is not None, "documents.page_count is missing"
|
||||
assert col[0] == "integer", "page_count must be INTEGER"
|
||||
assert col[1] == "YES", (
|
||||
"page_count must be NULLABLE (NULL = not page-qualifying)"
|
||||
)
|
||||
assert col[2] is None, (
|
||||
"page_count must carry NO server default — NULL is the meaningful"
|
||||
" pre-phase state"
|
||||
)
|
||||
assert _index_covering(db, "page_count") is None, (
|
||||
"page_count must be UNINDEXED — a display fact, never a query"
|
||||
" surface (the page route reads it from the row after the id lookup)"
|
||||
)
|
||||
|
||||
# The pre-0026 row backfilled to NULL (the pre-phase shape).
|
||||
assert db.execute(
|
||||
text(
|
||||
"SELECT content_hash, page_count FROM documents WHERE id = :id"
|
||||
),
|
||||
{"id": pre_id},
|
||||
).fetchone() == (CONTENT_HASH, None), (
|
||||
"every pre-0026 row must backfill page_count to NULL — no"
|
||||
" backfill, no default — while content_hash survives"
|
||||
)
|
||||
|
||||
_assert_0025_schema_survives(db)
|
||||
finally:
|
||||
_delete(db, pre_id)
|
||||
|
||||
|
||||
def test_orm_page_count_round_trips(db: Session, alembic: Config) -> None:
|
||||
"""The ORM contract agrees with the column contract: a freshly
|
||||
inserted ``Document`` with NO ``page_count`` reads
|
||||
``page_count is None`` through a FRESH session (the pre-phase
|
||||
shape — NULL until the importer's PDF branch sets it), and an
|
||||
explicit count round-trips the value."""
|
||||
command.upgrade(alembic, "head")
|
||||
row_counted = Document(
|
||||
source=SOURCE,
|
||||
path="counted.pdf",
|
||||
full_path="/tmp/counted.pdf",
|
||||
title="counted",
|
||||
content=CONTENT,
|
||||
content_hash=CONTENT_HASH,
|
||||
page_count=24,
|
||||
)
|
||||
row_uncounted = Document(
|
||||
source=SOURCE,
|
||||
path="uncounted.md",
|
||||
full_path="/tmp/uncounted.md",
|
||||
title="uncounted",
|
||||
content=CONTENT,
|
||||
content_hash=CONTENT_HASH,
|
||||
)
|
||||
db.add(row_counted)
|
||||
db.add(row_uncounted)
|
||||
db.commit()
|
||||
try:
|
||||
with SessionLocal() as fresh:
|
||||
reloaded_counted = fresh.get(Document, row_counted.id)
|
||||
assert reloaded_counted is not None, "the counted row must be readable"
|
||||
assert reloaded_counted.page_count == 24, (
|
||||
"the explicit page_count must round-trip through the DB"
|
||||
)
|
||||
reloaded_uncounted = fresh.get(Document, row_uncounted.id)
|
||||
assert reloaded_uncounted is not None, "the uncounted row must be readable"
|
||||
assert reloaded_uncounted.page_count is None, (
|
||||
"a row created without page_count must read NULL (the pre-phase"
|
||||
" shape)"
|
||||
)
|
||||
assert reloaded_uncounted.content_hash == CONTENT_HASH, (
|
||||
"content_hash must be untouched by the page_count addition"
|
||||
)
|
||||
finally:
|
||||
_delete(db, row_counted.id)
|
||||
_delete(db, row_uncounted.id)
|
||||
|
||||
|
||||
def test_downgrade_to_0025_drops_the_column(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade 0026 → 0025: the page_count column is gone (A13 —
|
||||
fully reversible, the copies are re-derivable from the source
|
||||
files on the next sync) while the row + its content (incl. the
|
||||
0025 ``file_hash`` sibling) survive, and the rest of the 0025
|
||||
table contract is intact."""
|
||||
command.upgrade(alembic, "head")
|
||||
row = Document(
|
||||
source=SOURCE,
|
||||
path="down.pdf",
|
||||
full_path="/tmp/down.pdf",
|
||||
title="down",
|
||||
content=CONTENT,
|
||||
content_hash=CONTENT_HASH,
|
||||
page_count=24,
|
||||
)
|
||||
db.add(row)
|
||||
db.commit()
|
||||
try:
|
||||
command.downgrade(alembic, "0025")
|
||||
assert _version(db) == "0025"
|
||||
assert _column(db, "page_count") is None, "page_count must be dropped"
|
||||
|
||||
surviving = db.execute(
|
||||
text(
|
||||
"SELECT source, path, content, content_hash, file_hash, is_image,"
|
||||
" image_path FROM documents WHERE id = :id"
|
||||
),
|
||||
{"id": row.id},
|
||||
).fetchone()
|
||||
assert surviving is not None and surviving[0] == SOURCE, (
|
||||
"the row must survive the column drop"
|
||||
)
|
||||
assert surviving[1] == "down.pdf" and surviving[2] == CONTENT, (
|
||||
"path + content must survive the drop"
|
||||
)
|
||||
assert surviving[3] == CONTENT_HASH, (
|
||||
"content_hash (the retriever's identity) must survive the drop"
|
||||
)
|
||||
assert surviving[4] is None and surviving[5] is False and surviving[6] is None, (
|
||||
"file_hash + is_image + image_path must survive the drop"
|
||||
)
|
||||
assert _index_covering(db, "content_hash") is not None, (
|
||||
"the content_hash index must survive the downgrade"
|
||||
)
|
||||
finally:
|
||||
_delete(db, row.id)
|
||||
# Repair: the fixture teardown re-upgrades to head.
|
||||
|
||||
|
||||
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0025, then upgrade back to 0026: ``page_count`` is
|
||||
back with the full contract (INTEGER, NULLABLE, NO server default,
|
||||
UNINDEXED), a row seeded at 0025 reads ``page_count is None`` (the
|
||||
pre-phase shape on the round-trip), and the 0025 schema (incl. the
|
||||
indexed ``content_hash``) is intact."""
|
||||
command.downgrade(alembic, "0025")
|
||||
pre_id = _insert_sql(db, "round.pdf")
|
||||
try:
|
||||
command.upgrade(alembic, "0026")
|
||||
assert _version(db) == "0026", "round-trip upgrade must land at 0026"
|
||||
|
||||
col = _column(db, "page_count")
|
||||
assert col is not None, "documents.page_count must be back"
|
||||
assert col[0] == "integer", "page_count must be INTEGER after the round-trip"
|
||||
assert col[1] == "YES", "page_count must be NULLABLE after the round-trip"
|
||||
assert col[2] is None, (
|
||||
"page_count must still carry NO server default after the round-trip"
|
||||
)
|
||||
assert _index_covering(db, "page_count") is None, (
|
||||
"page_count must stay UNINDEXED after the round-trip"
|
||||
)
|
||||
|
||||
assert db.execute(
|
||||
text("SELECT page_count FROM documents WHERE id = :id"),
|
||||
{"id": pre_id},
|
||||
).scalar() is None, (
|
||||
"a row seeded at 0025 must read page_count NULL after the round-trip"
|
||||
)
|
||||
|
||||
_assert_0025_schema_survives(db)
|
||||
finally:
|
||||
_delete(db, pre_id)
|
||||
@@ -547,6 +547,18 @@ def test_sources_dir_env_override_is_raw_string(monkeypatch: pytest.MonkeyPatch)
|
||||
assert s.sources_dir == "/data/bor/sources"
|
||||
|
||||
|
||||
def test_pdf_page_dir_default_and_env_override(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 148 (task 01): ``pdf_page_dir`` — the persistent home for
|
||||
a page-qualifying PDF's rendered page PNGs — defaults to
|
||||
``~/bor-sources/pages`` and, like ``sources_dir``/``upload_dir``/
|
||||
``image_dir``, is a RAW string (``Path.expanduser()`` is applied by
|
||||
the importer, not here) that ``BOR_PDF_PAGE_DIR`` overrides."""
|
||||
monkeypatch.delenv("BOR_PDF_PAGE_DIR", raising=False)
|
||||
assert _settings().pdf_page_dir == "~/bor-sources/pages"
|
||||
monkeypatch.setenv("BOR_PDF_PAGE_DIR", "/data/bor/pages")
|
||||
assert _settings().pdf_page_dir == "/data/bor/pages"
|
||||
|
||||
|
||||
def test_suggestions_default_is_three_plus_real_questions() -> None:
|
||||
s = _settings()
|
||||
assert len(s.suggestions) >= 3
|
||||
|
||||
@@ -21,6 +21,15 @@ Frontend side:
|
||||
import-safe page guard, the modal skeleton on chat + Sources, and the
|
||||
modal module's close/focus/URL contract.
|
||||
|
||||
Phase 148 (task 02) — the content payload's rendered-page routes
|
||||
(the viewer strip, task 03, reads from them): the ``DocContent``
|
||||
omission rule — ``page_urls`` ABSENT on the wire when ``None`` (never
|
||||
``null``; a non-qualifying doc's payload is byte-identical to
|
||||
pre-phase) — and the content endpoint (stubbed session) — a
|
||||
page-qualifying PDF row (``page_count`` set) adds ``page_urls`` = the
|
||||
exact 1-based ``/api/documents/<id>/pages/<i>`` list over
|
||||
``1..page_count``.
|
||||
|
||||
Phase 145 (task 03) — the content payload's git origin fields + the
|
||||
``renderDocument`` "Original" anchor (modal AND /document.html, one
|
||||
site):
|
||||
@@ -46,6 +55,7 @@ import json
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import uuid
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
@@ -279,10 +289,13 @@ def _text_doc() -> Document:
|
||||
|
||||
|
||||
def _text_doc_content(
|
||||
origin_url: str | None = None, origin_branch: str | None = None
|
||||
origin_url: str | None = None,
|
||||
origin_branch: str | None = None,
|
||||
page_urls: list[str] | None = None,
|
||||
) -> DocContent:
|
||||
"""One text doc's content payload (the pre-phase shape) with the
|
||||
phase-145 origin fields optionally set."""
|
||||
phase-145 origin fields + the phase-148 ``page_urls`` optionally
|
||||
set."""
|
||||
return DocContent(
|
||||
source="S",
|
||||
path="a.md",
|
||||
@@ -294,6 +307,7 @@ def _text_doc_content(
|
||||
chunks=1,
|
||||
origin_url=origin_url,
|
||||
origin_branch=origin_branch,
|
||||
page_urls=page_urls,
|
||||
)
|
||||
|
||||
|
||||
@@ -316,6 +330,64 @@ def test_doc_content_origin_fields_absent_on_the_wire_when_unset() -> None:
|
||||
assert dumped["origin_branch"] == "docs"
|
||||
|
||||
|
||||
def test_doc_content_page_urls_absent_on_the_wire_when_unset() -> None:
|
||||
"""Phase 148 (task 02) — the model-level omission rule (the
|
||||
``image_url`` precedent): a non-qualifying document (``page_urls``
|
||||
``None`` — a text doc, an image, a text-heavy PDF) serializes with
|
||||
the key ABSENT — never ``null`` — so its payload is byte-identical
|
||||
to pre-phase. A page-qualifying PDF keeps the exact route list."""
|
||||
dumped = _text_doc_content().model_dump()
|
||||
assert "page_urls" not in dumped # absent — never null
|
||||
urls = ["/api/documents/d/pages/1", "/api/documents/d/pages/2"]
|
||||
dumped = _text_doc_content(page_urls=urls).model_dump()
|
||||
assert dumped["page_urls"] == urls # the exact 1-based route list
|
||||
|
||||
|
||||
def _page_pdf_doc() -> Document:
|
||||
"""A page-QUALIFYING PDF row (phase 148): ``page_count`` set — the
|
||||
endpoint derives ``page_urls`` from it (the exact 1-based list)."""
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source="Homelab",
|
||||
path="ikea/besta.pdf",
|
||||
full_path="/tmp/besta.pdf",
|
||||
title="BESTA frame (white) assembly",
|
||||
content="[Page 1 illustration: the parts tray]…",
|
||||
content_hash="e" * 64,
|
||||
is_image=False, # phase 122: the stub session never applies defaults
|
||||
page_count=2,
|
||||
)
|
||||
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
|
||||
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC)
|
||||
return doc
|
||||
|
||||
|
||||
def test_content_page_urls_carried_for_a_page_qualifying_pdf() -> None:
|
||||
"""Phase 148 (task 02) — the endpoint (stubbed session, no
|
||||
database): a page-qualifying PDF row (``page_count`` set) carries
|
||||
``page_urls`` = the EXACT 1-based
|
||||
``/api/documents/<id>/pages/<i>`` list over ``1..page_count``
|
||||
(the viewer strip's source, task 03 — a per-page render gap is
|
||||
still listed; the route 404s that N). Every pre-phase key stands
|
||||
untouched (``is_image`` false, ``image_url`` ABSENT — the PDF is
|
||||
not an image doc; the document's TEXT stays user-gated, only the
|
||||
page bytes are public)."""
|
||||
doc = _page_pdf_doc()
|
||||
with _client_with_row((doc, 4)) as client:
|
||||
r = client.get(
|
||||
"/api/documents/content",
|
||||
params={"source": "Homelab", "path": "ikea/besta.pdf"},
|
||||
)
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["page_urls"] == [
|
||||
f"/api/documents/{doc.id}/pages/1",
|
||||
f"/api/documents/{doc.id}/pages/2",
|
||||
] # the exact 1-based list, count == page_count
|
||||
assert body["is_image"] is False
|
||||
assert "image_url" not in body # absent — never null (not an image doc)
|
||||
|
||||
|
||||
def test_content_origin_url_carried_for_a_matching_git_row() -> None:
|
||||
"""The endpoint (stubbed session, no database): a ``kind='git'``
|
||||
row whose URL basename matches the document's source adds
|
||||
|
||||
@@ -0,0 +1,628 @@
|
||||
"""Phase 148, task 01 — render qualifying PDF pages at index time (unit
|
||||
matrix).
|
||||
|
||||
The owner's TODO L5: "For primarily pictographic PDFs it's not super
|
||||
helpful to see the summaries of the pages. I want to see the PDF pages
|
||||
in the sources." The locked rule: a PDF gets rendered pages iff the
|
||||
sync ALREADY decided its text is not the useful view — the phase-133
|
||||
fallback fired (unusable text layer — the screenshot class) OR the
|
||||
phase-140 pictorial pass described ≥1 page (the sparse-page class —
|
||||
the IKEA fixture). Text-heavy PDFs (usable layer, zero candidates)
|
||||
keep the text view, and ``BOR_PDF_PICTORIAL=0`` keeps its kill-switch
|
||||
semantics (no pictorial descriptions → no page strip for the sparse
|
||||
class).
|
||||
|
||||
Pinned here (the house PyMuPDF fixture recipes — the
|
||||
``test_pictorial_import.py`` builder for the pictorial class, the
|
||||
``test_importer.py`` garbage builder for the fallback class, the dense
|
||||
builder for the negative; the copies land in a tmp ``pdf_page_dir``):
|
||||
|
||||
* eligibility × new index — the pictorial class (IKEA shape: usable
|
||||
layer, ≥1 sparse page) and the fallback class (unusable layer) get
|
||||
``<doc-id>-p{N}.png`` for EVERY page (not just the described
|
||||
candidates — the strip shows the whole manual) + ``page_count`` =
|
||||
the page count + real PNG bytes; a text-heavy PDF gets neither
|
||||
(no copies, NULL count, ZERO render calls, byte-identical content);
|
||||
* the phase-147 fast path — an unchanged qualifying PDF's second sync
|
||||
re-renders NOTHING (zero render calls — the copies persist,
|
||||
``page_count`` and the bytes untouched);
|
||||
* stale replacement — a changed (shrunk) PDF's old copy set is
|
||||
replaced: the new set covers the new pages, the old ``p{N}`` tail
|
||||
is GONE (no orphans), ``page_count`` follows the new count;
|
||||
* qualifier loss — a qualifying PDF whose new version gained a dense
|
||||
text layer loses its strip: ``page_count`` NULL + copies deleted;
|
||||
* unopenable — a render that yields no pages (``render_pdf_pages`` →
|
||||
``[]``): ``page_count`` NULL, no copies, NO exception, the doc is
|
||||
still indexed exactly as pre-phase (the row lands, the summary
|
||||
runs);
|
||||
* per-page gap — one unreadable page (the render's ``b""``
|
||||
placeholder): the other copies are written, ``page_count`` keeps
|
||||
the FULL count (the gap is a display-level 404 — task 02 pins the
|
||||
route's side);
|
||||
* fail-soft — the render raising (warning is the signal, ``page_count``
|
||||
NULL, no copies, the doc indexed fully — chunks + summary) and the
|
||||
WRITER raising (the copies this call wrote are removed — no partial
|
||||
orphans, ``page_count`` NULL, the doc indexed fully);
|
||||
* prune — a qualifying doc's file removed from the source: the row
|
||||
goes and its page copies go with it (no orphans in the tmp dir).
|
||||
|
||||
The route + payload (task 02) and the viewer strip (task 03) are
|
||||
pinned in their own task files; the E2E surface is
|
||||
``tests/e2e/test_pdf_pages_in_sources.py`` (task 04).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import uuid
|
||||
from collections.abc import Callable, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pymupdf as fitz
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag import importer
|
||||
from app.rag.importer import _delete_page_copies, import_sources
|
||||
from app.rag.pdf import extract_pdf_text
|
||||
from tests.unit.test_importer import _build_garbage_pdf, _PageVisionLLM
|
||||
from tests.unit.test_pictorial_import import (
|
||||
TITLE,
|
||||
_build_dense_pdf,
|
||||
_build_ikea_like_pdf,
|
||||
_insert_dense,
|
||||
_PictorialLLM,
|
||||
)
|
||||
|
||||
#: The PNG signature (``render_pdf_pages``' output is ``tobytes("png")``).
|
||||
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
|
||||
|
||||
|
||||
def _settings(tmp_path: Path, **kwargs: Any) -> Settings:
|
||||
"""Settings admitting the ``pdf`` import token with the phase-140
|
||||
knobs at their CODE DEFAULTS (``pdf_pictorial`` ON — LOCKED A3) and
|
||||
the phase-148 ``pdf_page_dir`` pinned to *tmp_path* (the copies
|
||||
under test never land in the real home); ``images`` off (the
|
||||
embedded-image pass is not in play — the fixture class is
|
||||
vector-drawn)."""
|
||||
kwargs.setdefault("_env_file", None)
|
||||
kwargs.setdefault("import_extensions", "md,txt,pdf")
|
||||
kwargs.setdefault("images", False)
|
||||
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||
kwargs.setdefault("pdf_page_dir", str(tmp_path / "pages"))
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _pages_dir(tmp_path: Path) -> Path:
|
||||
return tmp_path / "pages"
|
||||
|
||||
|
||||
def _doc(db, source: str, rel: str) -> Document:
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == rel)
|
||||
)
|
||||
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
|
||||
return doc
|
||||
|
||||
|
||||
def _cleanup_source(db, source: str) -> None:
|
||||
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _build_ikea_shrunk(path: Path) -> None:
|
||||
"""A SHRUNK IKEA-class PDF (the stale-replacement target): 6 pages
|
||||
— the title page, two dense warnings, three sparse ``Step N``
|
||||
markers — still qualifying (candidates: 1 and 4–6), but four pages
|
||||
shorter than :func:`_build_ikea_like_pdf`'s 10 (the old ``p7..p10``
|
||||
tail must not leak)."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), "BESTA")
|
||||
_insert_dense(doc.new_page())
|
||||
_insert_dense(doc.new_page())
|
||||
for n in range(4, 7):
|
||||
doc.new_page().insert_text((72, 72), f"Step {n}")
|
||||
doc.set_metadata({"title": TITLE})
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def _render_spy(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
*,
|
||||
mutate_full: Callable[[list[bytes]], list[bytes]] | None = None,
|
||||
raise_full: bool = False,
|
||||
) -> list[Any]:
|
||||
"""Count + steer the ``render_pdf_pages`` calls the importer makes
|
||||
(the pictorial pass's candidate render AND the phase-148
|
||||
full-page render both resolve the module global at call time):
|
||||
returns the list of *pages* arguments seen (a candidate list for
|
||||
the pass, ``None`` for the full render). *mutate_full* transforms
|
||||
the full-render result (e.g. blanks one page's slot);
|
||||
*raise_full* makes the full render raise. The candidate render
|
||||
always runs the real helper."""
|
||||
real = importer.render_pdf_pages
|
||||
calls: list[Any] = []
|
||||
|
||||
def spy(pdf_path: Path, dpi: int = 150, pages: Sequence[int] | None = None) -> list[bytes]:
|
||||
calls.append(pages)
|
||||
if pages is not None:
|
||||
return real(pdf_path, dpi=dpi, pages=pages)
|
||||
if raise_full:
|
||||
raise RuntimeError("simulated render failure (test sentinel)")
|
||||
out = real(pdf_path, dpi=dpi, pages=pages)
|
||||
return mutate_full(out) if mutate_full is not None else out
|
||||
|
||||
monkeypatch.setattr(importer, "render_pdf_pages", spy)
|
||||
return calls
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Eligibility × new index
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pictorial_pdf_new_index_writes_every_page_copy(db, tmp_path: Path) -> None:
|
||||
"""Eligible — the PICTORIAL class (IKEA shape: usable layer, ≥1
|
||||
sparse page): the first index writes EVERY page of the PDF (not
|
||||
just the described candidates — the strip shows the whole manual)
|
||||
as ``<doc-id>-p{N}.png`` under ``pdf_page_dir`` (N 1-based), sets
|
||||
``page_count`` to the page count, and the files carry real PNG
|
||||
bytes — while the indexed content is unchanged by this phase (the
|
||||
phase-140 illustration blocks stay: the descriptions help the
|
||||
embeddings and stay in the DB for Q&A — they are simply not
|
||||
displayed, task 03)."""
|
||||
root = tmp_path / "pgp"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_pictorial == 1 # the pass ran (the eligibility class)
|
||||
doc = _doc(db, "pgp", "ikea.pdf")
|
||||
assert doc.page_count == 10, "page_count = the PDF's page count"
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for n in range(1, 11):
|
||||
path = page_dir / f"{doc.id}-p{n}.png"
|
||||
assert path.is_file(), f"copy p{n} is missing"
|
||||
assert path.read_bytes()[:8] == PNG_MAGIC, f"p{n} is not a real PNG"
|
||||
# No behavior change: the final content still carries the
|
||||
# phase-140 illustration blocks on the sparse pages (1, 6–10)
|
||||
# and not on the dense ones (2–5).
|
||||
assert "[Page 1 illustration:" in doc.content
|
||||
assert "[Page 6 illustration:" in doc.content
|
||||
assert "[Page 2 illustration:" not in doc.content
|
||||
finally:
|
||||
_cleanup_source(db, "pgp")
|
||||
|
||||
|
||||
def test_fallback_class_garbage_pdf_writes_every_page_copy(db, tmp_path: Path) -> None:
|
||||
"""Eligible — the FALLBACK class (unusable text layer — the
|
||||
screenshot class): the garbage PDF takes the phase-133 screenshot
|
||||
fallback UNCHANGED and now gets the page strip too — one copy per
|
||||
page + ``page_count`` (the viewer can show the actual pages of the
|
||||
screenshot class as well)."""
|
||||
root = tmp_path / "pgf"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "garbage.pdf") # 2 pages
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_screenshotted == 1 # the fallback fired (the class)
|
||||
doc = _doc(db, "pgf", "garbage.pdf")
|
||||
assert doc.page_count == 2
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for n in (1, 2):
|
||||
path = page_dir / f"{doc.id}-p{n}.png"
|
||||
assert path.is_file(), f"copy p{n} is missing"
|
||||
assert path.read_bytes()[:8] == PNG_MAGIC
|
||||
# No behavior change: the content is still the fallback's
|
||||
# vision descriptions (the page-anchored separator shape).
|
||||
assert doc.content.startswith("---PAGE-1---")
|
||||
finally:
|
||||
_cleanup_source(db, "pgf")
|
||||
|
||||
|
||||
def test_text_heavy_pdf_keeps_the_text_view(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Ineligible — a TEXT-HEAVY PDF (usable layer, zero candidates):
|
||||
the text view stays — no copies, NULL ``page_count``, ZERO render
|
||||
calls (the pictorial pass no-ops before any render), and the
|
||||
content is byte-identical to ``extract_pdf_text`` (the
|
||||
pre-phase-140/148 pin)."""
|
||||
root = tmp_path / "pgd"
|
||||
root.mkdir()
|
||||
pdf = root / "clean.pdf"
|
||||
_build_dense_pdf(pdf)
|
||||
expected = extract_pdf_text(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
calls = _render_spy(monkeypatch)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops
|
||||
doc = _doc(db, "pgd", "clean.pdf")
|
||||
assert doc.page_count is None, "a text-heavy PDF keeps the text view"
|
||||
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
||||
assert calls == [], "zero render calls (no fallback, zero candidates)"
|
||||
assert doc.content == expected, "byte-identical to the extraction"
|
||||
finally:
|
||||
_cleanup_source(db, "pgd")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lifecycle: unchanged / changed / qualifier loss
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unchanged_second_sync_renders_nothing(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The phase-147 fast path + phase 148: an UNCHANGED qualifying
|
||||
PDF's second sync costs ZERO render calls (the copies persist — no
|
||||
re-render, no orphans), and ``page_count`` and every copy's bytes
|
||||
are untouched. The fast path returns before the branch, so the
|
||||
render block is skipped by construction."""
|
||||
root = tmp_path / "pgu"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
calls = _render_spy(monkeypatch)
|
||||
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s1.added, s1.unchanged) == (1, 0)
|
||||
doc1 = _doc(db, "pgu", "ikea.pdf")
|
||||
assert doc1.page_count == 10
|
||||
assert len([c for c in calls if c is None]) == 1, (
|
||||
"exactly one full-page render on the first index"
|
||||
)
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
copies = {
|
||||
n: (page_dir / f"{doc1.id}-p{n}.png").read_bytes() for n in range(1, 11)
|
||||
}
|
||||
assert all(b[:8] == PNG_MAGIC for b in copies.values())
|
||||
embeds1 = len(llm.calls)
|
||||
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 0, 1)
|
||||
assert len(calls) == 2, "second sync: ZERO render calls (the fast path)"
|
||||
assert len(llm.calls) == embeds1, "no re-embed (the fast path's zero-work)"
|
||||
|
||||
doc2 = _doc(db, "pgu", "ikea.pdf")
|
||||
assert doc2.id == doc1.id
|
||||
assert doc2.page_count == 10, "page_count untouched"
|
||||
for n in range(1, 11):
|
||||
assert (page_dir / f"{doc2.id}-p{n}.png").read_bytes() == copies[n], (
|
||||
f"copy p{n} moved on an unchanged sync"
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "pgu")
|
||||
|
||||
|
||||
def test_changed_pdf_replaces_the_copy_set_without_orphans(
|
||||
db, tmp_path: Path,
|
||||
) -> None:
|
||||
"""A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index
|
||||
replaces the copy set — the new set covers p1..p6, the old
|
||||
version's ``p7..p10`` tail is GONE (the stale-replacement runs
|
||||
before the write), the doc keeps its id, and ``page_count``
|
||||
follows the new page count."""
|
||||
root = tmp_path / "pgc"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
_build_ikea_like_pdf(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert s1.added == 1
|
||||
doc1 = _doc(db, "pgc", "ikea.pdf")
|
||||
assert doc1.page_count == 10
|
||||
|
||||
_build_ikea_shrunk(pdf) # same path, 6 pages — still qualifying
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||
doc2 = _doc(db, "pgc", "ikea.pdf")
|
||||
assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)"
|
||||
assert doc2.page_count == 6, "page_count follows the new page count"
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for n in range(1, 7):
|
||||
assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing"
|
||||
for n in range(7, 11):
|
||||
assert not (page_dir / f"{doc2.id}-p{n}.png").exists(), (
|
||||
f"stale p{n} from the 10-page version leaked"
|
||||
)
|
||||
leftovers = sorted(p.name for p in page_dir.iterdir())
|
||||
assert len(leftovers) == 6, "exactly the new copy set, no orphans"
|
||||
finally:
|
||||
_cleanup_source(db, "pgc")
|
||||
|
||||
|
||||
def test_qualifier_loss_clears_page_count_and_copies(
|
||||
db, tmp_path: Path,
|
||||
) -> None:
|
||||
"""A qualifying PDF whose NEW VERSION gained a dense text layer
|
||||
(zero candidates, ``pdf_pictorial`` ON): the document STOPPED
|
||||
qualifying — ``page_count`` is NULL, the copies are deleted, and
|
||||
the text view returns (the content is the extraction,
|
||||
byte-identical)."""
|
||||
root = tmp_path / "pgl"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
_build_ikea_like_pdf(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert s1.added == 1
|
||||
doc1 = _doc(db, "pgl", "ikea.pdf")
|
||||
assert doc1.page_count == 10
|
||||
assert all(
|
||||
_pages_dir(tmp_path).joinpath(f"{doc1.id}-p{n}.png").is_file()
|
||||
for n in range(1, 11)
|
||||
)
|
||||
|
||||
_build_dense_pdf(pdf) # same path: dense text layer, zero candidates
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
|
||||
doc2 = _doc(db, "pgl", "ikea.pdf")
|
||||
assert doc2.id == doc1.id
|
||||
assert doc2.page_count is None, "a doc that stopped qualifying loses its strip"
|
||||
assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), (
|
||||
"the copies must be deleted on qualifier loss"
|
||||
)
|
||||
assert doc2.content == extract_pdf_text(pdf), "the text view returns"
|
||||
finally:
|
||||
_cleanup_source(db, "pgl")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fail-soft: unopenable / gap / render exception / write exception
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unopenable_pdf_page_render_is_fail_soft(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""A render that yields NO pages (an unopenable file —
|
||||
``render_pdf_pages`` → ``[]``): ``page_count`` NULL, no copies,
|
||||
NO exception — the document still indexes exactly as pre-phase
|
||||
(the fallback's all-failed path: an empty-content row lands, the
|
||||
summary runs)."""
|
||||
root = tmp_path / "pgn"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "garbage.pdf") # the fallback class
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
monkeypatch.setattr(importer, "render_pdf_pages", lambda *a, **k: [])
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
doc = _doc(db, "pgn", "garbage.pdf")
|
||||
assert doc.page_count is None, "no pages rendered — no strip"
|
||||
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
|
||||
assert doc.content == "", "the fallback's all-failed render lands empty content"
|
||||
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
||||
finally:
|
||||
_cleanup_source(db, "pgn")
|
||||
|
||||
|
||||
def test_per_page_gap_writes_the_other_copies_and_keeps_the_full_count(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""One unreadable page (the render's ``b""`` placeholder — page 4
|
||||
here): a per-page failure is NOT an exception — the other copies
|
||||
are written, ``page_count`` keeps the FULL count, and the gap's
|
||||
file is simply absent (a display-level 404 — the route of task 02
|
||||
pins its side)."""
|
||||
root = tmp_path / "pgg"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
|
||||
def gap_page_four(pages: list[bytes]) -> list[bytes]:
|
||||
pages[3] = b"" # index 3 = page 4 (the helper's list-index contract)
|
||||
return pages
|
||||
|
||||
_render_spy(monkeypatch, mutate_full=gap_page_four)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "pgg", "ikea.pdf")
|
||||
assert doc.page_count == 10, "the gap does not shrink the count"
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for n in range(1, 11):
|
||||
path = page_dir / f"{doc.id}-p{n}.png"
|
||||
if n == 4:
|
||||
assert not path.exists(), "the gap has no file"
|
||||
else:
|
||||
assert path.is_file(), f"copy p{n} is missing"
|
||||
finally:
|
||||
_cleanup_source(db, "pgg")
|
||||
|
||||
|
||||
def test_render_exception_is_fail_soft(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The full-page render RAISES: the warning is the signal,
|
||||
``page_count`` NULL, no copies — and the document indexes EXACTLY
|
||||
as pre-phase (the pictorial content, the chunks, the summary) —
|
||||
the render is a display concern, it never gates the indexing."""
|
||||
root = tmp_path / "pge"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
_render_spy(monkeypatch, raise_full=True)
|
||||
try:
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
doc = _doc(db, "pge", "ikea.pdf")
|
||||
assert doc.page_count is None
|
||||
assert not _pages_dir(tmp_path).exists(), "no copies — no partial orphans"
|
||||
assert doc.chunks, "the chunk pipeline ran (fail-soft)"
|
||||
assert summary.summaries == 1, "the summary ran (fail-soft)"
|
||||
assert "[Page 6 illustration:" in doc.content, (
|
||||
"the pictorial pass's content is untouched by the render failure"
|
||||
)
|
||||
assert any("PDF page render failed" in m for m in caplog.messages), (
|
||||
caplog.messages
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "pge")
|
||||
|
||||
|
||||
def test_write_exception_is_fail_soft_and_leaves_no_orphans(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The PNG WRITER raises (a full disk) after the FIRST copy
|
||||
landed: the copies this call wrote are removed (no partial
|
||||
orphans), ``page_count`` NULL, the warning is the signal, and the
|
||||
document indexes exactly as pre-phase."""
|
||||
root = tmp_path / "pgw"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
original_write = Path.write_bytes
|
||||
written_paths: list[Path] = []
|
||||
|
||||
def boom(self: Path, data: bytes) -> int:
|
||||
written_paths.append(self)
|
||||
if len(written_paths) > 1:
|
||||
raise OSError("simulated disk full (test sentinel)")
|
||||
return original_write(self, data)
|
||||
|
||||
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
|
||||
try:
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
|
||||
assert len(written_paths) == 2, "p1 landed, p2's write failed"
|
||||
doc = _doc(db, "pgw", "ikea.pdf")
|
||||
assert doc.page_count is None
|
||||
assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), (
|
||||
"the partial write was cleaned up — no orphans"
|
||||
)
|
||||
assert not written_paths[0].exists(), "the landed p1 copy was removed"
|
||||
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
|
||||
assert any("PDF page render failed" in m for m in caplog.messages), (
|
||||
caplog.messages
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "pgw")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The shared cleanup helper (the _delete_image_copy convention)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_delete_page_copies_helper_is_best_effort(
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""The helper (the ``_delete_image_copy`` convention): the row
|
||||
knows its own extent — copies 1..``page_count`` are deleted, a
|
||||
NULL/zero count is a no-op, missing files are a SILENT no-op
|
||||
(already gone — e.g. the owner cleaned the pages dir), and an
|
||||
unreadable file is logged, never raised (copy cleanup must not
|
||||
break the sync)."""
|
||||
page_dir = tmp_path / "pages"
|
||||
page_dir.mkdir()
|
||||
doc_id = uuid.uuid4()
|
||||
doc = Document(
|
||||
id=doc_id,
|
||||
source="s",
|
||||
path="p.pdf",
|
||||
full_path="/p.pdf",
|
||||
title="t",
|
||||
content="c",
|
||||
content_hash="x",
|
||||
page_count=3,
|
||||
)
|
||||
p1 = page_dir / f"{doc_id}-p1.png"
|
||||
p3 = page_dir / f"{doc_id}-p3.png"
|
||||
p1.write_bytes(b"png1")
|
||||
p3.write_bytes(b"png3") # p2 is missing (the silent no-op)
|
||||
|
||||
original_unlink = Path.unlink
|
||||
|
||||
def boom(self: Path, **kwargs: Any) -> None:
|
||||
if self == p3:
|
||||
raise OSError("simulated unreadable file (test sentinel)")
|
||||
return original_unlink(self, **kwargs)
|
||||
|
||||
monkeypatch.setattr(Path, "unlink", boom) # pyright: ignore[reportArgumentType]
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
_delete_page_copies(doc, page_dir)
|
||||
assert not p1.exists(), "the existing copy was deleted"
|
||||
assert p3.exists(), "the unreadable file survived (cleanup never raises)"
|
||||
assert any(
|
||||
"could not delete PDF page copy" in m for m in caplog.messages
|
||||
), caplog.messages
|
||||
|
||||
# A NULL count is a no-op (fresh docs, non-qualifying docs, every
|
||||
# fail-soft miss) — no exception, no attempt.
|
||||
doc.page_count = None
|
||||
_delete_page_copies(doc, page_dir)
|
||||
assert p3.exists()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Prune
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_prune_deletes_page_copies(db, tmp_path: Path) -> None:
|
||||
"""A qualifying doc is PRUNED (its file removed from the source):
|
||||
the row goes AND its page copies go with it — no orphans in the
|
||||
pages dir (the ``_prune`` hook beside the image-copy deletion)."""
|
||||
root = tmp_path / "pgz"
|
||||
root.mkdir()
|
||||
pdf = root / "ikea.pdf"
|
||||
_build_ikea_like_pdf(pdf)
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _settings(tmp_path)
|
||||
try:
|
||||
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert s1.added == 1
|
||||
doc = _doc(db, "pgz", "ikea.pdf")
|
||||
assert doc.page_count == 10
|
||||
assert len(list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png"))) == 10
|
||||
|
||||
pdf.unlink()
|
||||
s2 = asyncio.run(import_sources([root], llm, session=db, prune=True))
|
||||
assert s2.pruned == 1
|
||||
assert (
|
||||
db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == "pgz", Document.path == "ikea.pdf"
|
||||
)
|
||||
)
|
||||
is None
|
||||
), "the row is pruned"
|
||||
assert not list(_pages_dir(tmp_path).glob("*.png")), (
|
||||
"no orphaned page copies after the prune"
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "pgz")
|
||||
@@ -13,10 +13,12 @@ without a browser:
|
||||
* NO ``@media (min-width: 1500px)`` block anywhere in the file — the
|
||||
phase-58 wide-desktop doubling is deleted in full (the token is
|
||||
72rem at every viewport);
|
||||
* the four reading-column selectors — ``.chat-shell``,
|
||||
``.shared-shell``, ``.doc-md``, ``.doc-summary:has(+ .doc-md)`` —
|
||||
each capped with ``max-width: var(--chat-column)`` and NOTHING else
|
||||
in the file uses the token for a max-width (exactly four rules);
|
||||
* the six reading-column selectors — ``.chat-shell``,
|
||||
``.shared-shell``, ``.doc-md``, ``.doc-image`` (phase 122, task 04),
|
||||
``.doc-pages`` (phase 148, task 03 — the page strip), and
|
||||
``.doc-summary:has(+ .doc-md)`` — each capped with
|
||||
``max-width: var(--chat-column)`` and NOTHING else in the file uses
|
||||
the token for a max-width (exactly six rules);
|
||||
* the flipped negative pin — ZERO literal ``max-width: 46rem`` rules
|
||||
remain (the phase-27/59/91 form-column caps are retired): the form
|
||||
shells (``.tuning-shell``, ``.theme-shell``, ``.doc-edit-shell``)
|
||||
@@ -117,21 +119,24 @@ def test_no_literal_46rem_width_remains() -> None:
|
||||
def test_the_reading_columns_use_the_token() -> None:
|
||||
""".chat-shell, .shared-shell, .doc-md, .doc-image (phase 122,
|
||||
task 04 — the viewer's image block rides the SAME reading column
|
||||
as its .doc-md sibling) and .doc-summary:has(+ .doc-md) each cap
|
||||
with max-width: var(--chat-column) — and exactly those five rules
|
||||
use the token for a max-width (no other selector)."""
|
||||
as its .doc-md sibling), .doc-pages (phase 148, task 03 — the
|
||||
viewer's page strip rides the SAME reading column in the shared
|
||||
#doc-content slot) and .doc-summary:has(+ .doc-md) each cap with
|
||||
max-width: var(--chat-column) — and exactly those six rules use
|
||||
the token for a max-width (no other selector)."""
|
||||
css = _css()
|
||||
for selector in (
|
||||
".chat-shell",
|
||||
".shared-shell",
|
||||
".doc-md",
|
||||
".doc-image",
|
||||
".doc-pages",
|
||||
".doc-summary:has(+ .doc-md)",
|
||||
):
|
||||
assert "max-width: var(--chat-column)" in _rule_block(css, selector), (
|
||||
f"{selector} must cap with max-width: var(--chat-column)"
|
||||
)
|
||||
assert css.count("max-width: var(--chat-column)") == 5, (
|
||||
assert css.count("max-width: var(--chat-column)") == 6, (
|
||||
"exactly the reading-column selectors use the token"
|
||||
)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user