phase: 126_pdf_support
Build and Push Containers / build-and-push-app (push) Failing after 2m36s
Build and Push Containers / build-and-push-db (push) Successful in 1m39s

All gates verified green — no code changes were needed on this pass. Final report:

## Phase 126 — PDF support: final verification pass

**Verified (all tasks 01–05 already complete in working tree):**
- `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token
- `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk

**Test / lint / coverage results:**
- `uv run pytest` → 2997 passed
- `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed
- `uv run ruff check . && uv run pyright` → clean (0 errors)

**Completion criteria:**
1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests)
2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test)
3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites)
4. PDF renders as extracted text in viewer — ✅ (E2E)
5. Test/coverage/lint gates — ✅ (above)
6. Commit + phase move — harness handles (per protocol, no `git` on this pass)

**Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree.

**Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
2026-09-27 09:36:40 -04:00
parent e0307fe3ff
commit e0f11e7aa8
40 changed files with 4092 additions and 0 deletions
+644
View File
@@ -0,0 +1,644 @@
"""Phase 126 E2E (Playwright): PDF support — the user path.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_pdf_support.py -v --no-cov
PDFs are first-class, retrievable documents behind the ``pdf`` token of
``BOR_IMPORT_EXTENSIONS`` (the two module apps pin the token EXPLICITLY
— an operator's local gitignored ``.env`` cannot leak the scope in
either direction, and without the token the walk is blind to the file):
* the story app (``BOR_IMAGES=true``): the fixture PDF (two text pages
+ one embedded PNG on page 2, built with PyMuPDF in the fixture —
LOCKED A1) arrives through the REAL admin upload flow (a zip with a
single PDF member — the archive unpack is content-agnostic, the
extension walk admits it once ``pdf`` is in the token set) and is
first-class at every surface:
- the Sources page lists it (the extracted text's document — a plain
text row, no thumbnail) plus the embedded image's document (the
phase-122 pipeline, LOCKED A3 — the 48px thumbnail from the bytes
route, alt = the mock vision description);
- the document viewer renders the EXTRACTED TEXT (LOCKED A4 — the
``---PAGE-N---`` page markers and the page text, never binary
bytes) with the metadata title (LOCKED A2 — one document per PDF);
- a grounded chat question (the mock's token-overlap cosine against
the summary chunk ≈0.61, over the mock-calibrated gate 0.30 —
FTS-corroborated anyway) scripted to READ it echoes the read result
verbatim (the extracted text reaches the model AND the answer), and
the sources block carries the PDF's citation chip PLUS the embedded
image's auto-cited chip + compact inline figure (phase 124 — the
suggested-tier image doc, best chunk ≈0.22 over the usefulness
floor);
* the default app (``BOR_IMAGES`` forced ``false``): the SAME upload +
sync indexes the PDF's text normally — the embedded image is skipped
(no image document, no copy, the source is a 1-document source) and
the chat turn grounds with the PDF's chip only (no image chip, no
figure — LOCKED A3's toggle gate end-to-end).
"""
from __future__ import annotations
import base64
import json
import os
import re
import subprocess
import sys
import zipfile
from collections.abc import Iterator
from pathlib import Path
import httpx
import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (LOCKED A1)
import pytest
from playwright.sync_api import Browser, Page, expect
from sqlalchemy import select, text
from app.config import Settings
from app.db import SessionLocal
from app.models import Document
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
REPO = Path(__file__).resolve().parents[2]
SOURCE_NAME = "e2e-pdf" # the archive stem (archive_source_name)
DOC_PATH = "report.pdf"
IMAGE_REL = "report.pdf#page=2#img=0.png" # the synthetic embedded-image path
PDF_TITLE = "Quarterly Homelab Report" # the metadata title (LOCKED A2)
#: The fixture PDF's page texts (the chunker's ``Page N:`` anchors make
#: the chunks ``Page 1:\n<text>`` / ``Page 2:\n<text>``). The question
#: below shares its tokens with them (the mock's token-overlap cosine —
#: the summary chunk ≈0.61 — clears the mock-calibrated gate 0.30).
PAGE_ONE = "The quarterly homelab report covers the core switch firmware upgrade."
PAGE_TWO = "The homelab backup schedule runs nightly at 02:00 and keeps seven rotations."
#: The grounded question, carrying the house scripted-read call
#: (``SUMMARY_SEED_READ_TRIGGER`` — the phase-119 A1 convention: a
#: zero-read grounded turn chips NOTHING, so the PDF doc must be READ to
#: earn its citation chip): the mock emits the scripted
#: ``read e2e-pdf/report.pdf``, then echoes the tool result VERBATIM —
#: the read result carries the document's FULL extracted text, so the
#: echo in the answer is the E2E lens on LOCKED A4 (the PDF served as
#: extracted text, page markers and all).
QUESTION = (
"Read the suggested document: read e2e-pdf/report.pdf — "
"what does the homelab report say about the backup schedule?"
)
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
#: the pipeline is content-agnostic; the well-formed bytes keep the
#: extraction + copy + serve + render path honest).
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
UPLOAD_TIMEOUT_MS = 30_000
SYNC_TIMEOUT_MS = 60_000
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
# ---------------------------------------------------------------------------
# Module apps: images ON (the story) and images forced OFF (the default
# contract). Each owns its port + its scratch upload/source/image dirs.
# Both pin BOR_IMPORT_EXTENSIONS WITH the ``pdf`` token explicitly
# (phase 126 — the feature lives behind the token, and process env ranks
# above an operator's local gitignored ``.env``).
# ---------------------------------------------------------------------------
def _app_env(mock_llm: int, app_port: int, *, images: bool, scratch: Path) -> dict[str, str]:
"""The conftest app env (leak guards included) with the phase-126
knobs: ``BOR_IMPORT_EXTENSIONS`` pinned to a set carrying the
``pdf`` token, ``BOR_IMAGES`` pinned EXPLICITLY (true for the story
app, false for the default app — the contract under test cannot
leak in either direction), and the image/upload homes in the suite's
scratch dir (the app under test must not write copies into the
owner's real ``~/bor-sources``)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated gate (conftest pattern) — the story's question
# grounds on these values (see QUESTION).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
# Phase 67: instant retry waits + the code-default budget.
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# Leak guards (conftest pattern).
env["BOR_GIT_SOURCES"] = ""
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
# Phase 126: the import scope WITH the ``pdf`` token (the feature is
# gated by the token — the explicit pin keeps a local ``.env`` from
# changing the premise) + the toggle under test + the suite-private
# homes.
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
env["BOR_IMAGES"] = "true" if images else "false"
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_IMAGE_DIR"] = str(scratch / "images")
return env
@pytest.fixture(scope="module")
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
"""The story app — ``BOR_IMAGES=true`` + the ``pdf`` token (the
module env override; the conftest session app is never started in
this isolated run, so no port clash)."""
scratch = tmp_path_factory.mktemp("bor_pdf_on")
port = int(os.environ.get("E2E_APP_PORT_PDF", "8152"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, images=True, scratch=scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
@pytest.fixture(scope="module")
def default_app_server(
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
) -> Iterator[str]:
"""The default-contract app — ``BOR_IMAGES`` forced ``false`` (the
toggle gate for the embedded images; only the off-app tests start
it)."""
scratch = tmp_path_factory.mktemp("bor_pdf_off")
port = int(os.environ.get("E2E_APP_PORT_PDF_OFF", "8153"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, images=False, scratch=scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def default_app_url(default_app_server: str) -> str:
return default_app_server
@pytest.fixture(scope="module")
def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The fixture archive: ONE PDF at the root (two text pages + one
embedded PNG on page 2, built with PyMuPDF — LOCKED A1) — source
``e2e-pdf``, document path ``report.pdf``."""
root = tmp_path_factory.mktemp("bor_pdf_zip")
pdf_path = root / DOC_PATH
doc = fitz.open()
doc.new_page().insert_text((72, 72), PAGE_ONE)
page2 = doc.new_page()
page2.insert_text((72, 72), PAGE_TWO)
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
doc.set_metadata({"title": PDF_TITLE})
doc.save(pdf_path)
doc.close()
archive = root / f"{SOURCE_NAME}.zip"
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
zf.write(pdf_path, arcname=DOC_PATH)
return archive
# ---------------------------------------------------------------------------
# DB + UI helpers
# ---------------------------------------------------------------------------
def _truncate_all() -> None:
"""Fresh KB + registry per test group (the E2E isolation pattern):
the suites share one Postgres, and a leftover row would corrupt the
counts. ``sources_meta`` (the KB generation counter) resets with
the KB — the sync re-creates the row."""
with SessionLocal() as db:
db.execute(text(
"TRUNCATE chunks, documents, query_log, kb_overview, "
"git_sources, sources_meta, folder_summaries"
))
db.commit()
def _health_db(app_url: str) -> bool:
try:
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
except Exception: # noqa: BLE001 — unreachable is the skip case
return False
def _upload_zip(page: Page, app_url: str, archive: Path) -> None:
"""The real admin upload flow: form login → the git-sources page →
pick the archive → submit → the terminal result line (phase 90:
"Uploaded <source> — press Sync sources to import it.")."""
login(page, app_url, next="/git-sources.html")
expect(page).to_have_url(app_url + "/git-sources.html", timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
page.set_input_files("#archive-upload-file", str(archive))
page.click("#archive-upload-btn")
result = page.locator("#archive-upload-result")
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
expect(result).to_have_text(
f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
timeout=UPLOAD_TIMEOUT_MS,
)
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
"""The RAG page's "Sync sources" button (the scan): click →
"Syncing…" → the terminal "Synced HH:MM" label + the fresh counts
in #sync-result (the never-stale contract)."""
page.goto(app_url + "/sources.html")
btn = page.locator("#sync-btn")
expect(btn).to_be_visible(timeout=30_000)
expect(page.locator("#sync-label")).to_have_text(
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
)
btn.click()
expect(btn).to_be_disabled()
expect(page.locator("#sync-label")).to_have_text("Syncing…")
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
expect(btn).to_be_enabled()
expect(page.locator("#sync-result")).to_have_text(expected_result)
def _drill_to_source(page: Page, name: str) -> None:
"""Top level → the source (the file table then holds its direct
files — ours are at the source root)."""
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
# ---------------------------------------------------------------------------
# The module seeds: upload + sync the fixture PDF through the real UI,
# once per app (the story E2E truncate/re-import-per-module pattern).
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def seeded(
app_url: str,
browser: Browser,
zip_path: Path,
) -> Iterator[dict[str, str]]:
"""Upload the single-PDF zip through the admin UI on the story app,
then scan it with the RAG page's Sync button (the mock's vision
model describes the embedded image in-process). Yields the embedded
image doc's identity (the bytes route + the description) for the
render assertions. The module app's ``/api/config`` flag is
asserted first — the env override must have reached the app under
test, or the whole premise of the suite is void."""
if not _health_db(app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
assert cfg["images"] is True, "the module app must run with BOR_IMAGES=true"
_truncate_all()
page = browser.new_page(viewport={"width": 1280, "height": 800})
try:
_upload_zip(page, app_url, zip_path)
# The PDF + its one embedded image (A3): both count added.
_sync_via_ui(page, app_url, expected_result="2 added")
with SessionLocal() as db:
pdf = db.scalar(
select(Document).where(
Document.source == SOURCE_NAME, Document.path == DOC_PATH
)
)
assert pdf is not None, "the uploaded PDF must become a document"
# LOCKED A2: one document per PDF, the extracted text with
# the page markers; the title is the metadata title.
assert pdf.is_image is False and pdf.image_path is None
assert pdf.title == PDF_TITLE
assert "---PAGE-1---" in pdf.content and PAGE_ONE in pdf.content
assert "---PAGE-2---" in pdf.content and PAGE_TWO in pdf.content
assert pdf.summary is not None
img = db.scalar(
select(Document).where(
Document.source == SOURCE_NAME, Document.path == IMAGE_REL
)
)
assert img is not None, "the embedded image must become an image doc"
assert img.is_image is True and img.image_path is not None
# content == summary == the mock's description (LOCKED A3
# fail-soft shape; no markdown reference exists — the bare
# prompt's answer).
assert img.content == IMAGE_DESCRIPTION_ANSWER
assert img.summary == IMAGE_DESCRIPTION_ANSWER
finally:
page.close()
yield {
"image_url": f"/api/documents/{img.id}/image",
"description": IMAGE_DESCRIPTION_ANSWER,
}
@pytest.fixture(scope="module")
def seeded_off(
default_app_url: str,
browser: Browser,
zip_path: Path,
) -> Iterator[dict[str, str]]:
"""The SAME fixture zip through the default app (images off): the
PDF's text indexes normally, the embedded image is skipped — the
source is a 1-document source. Yields the PDF doc's identity."""
if not _health_db(default_app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = httpx.get(f"{default_app_url}/api/config", timeout=5).json()
assert cfg["images"] is False, "the default app must run with images off"
_truncate_all()
page = browser.new_page(viewport={"width": 1280, "height": 800})
try:
_upload_zip(page, default_app_url, zip_path)
_sync_via_ui(page, default_app_url, expected_result="1 added")
with SessionLocal() as db:
pdf = db.scalar(
select(Document).where(
Document.source == SOURCE_NAME, Document.path == DOC_PATH
)
)
assert pdf is not None, "the uploaded PDF must become a document"
assert PAGE_ONE in pdf.content and PAGE_TWO in pdf.content
assert (
db.scalar(select(Document).where(Document.is_image.is_(True)))
is None
), "images off: the embedded image is skipped (no image doc)"
finally:
page.close()
yield {"title": PDF_TITLE}
# ---------------------------------------------------------------------------
# 1. Sources page (images ON): the PDF's text row + the image's thumbnail
# ---------------------------------------------------------------------------
def test_sources_page_lists_the_pdf_and_image_docs(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""The KB total is 2 docs / 5 chunks (the PDF: 2 page-anchored
content chunks + the phase-30 summary chunk; the image: the
description chunk + its summary chunk). The drill-down table lists
both documents: the PDF as a PLAIN text row (LOCKED A4 — it is a
text document, no thumbnail box) and the embedded image with the
FIXED 48px thumbnail (the bytes route, alt = the description)."""
login(page, app_url) # lands on /sources.html
expect(page.locator("#stat-docs")).to_have_text("2")
expect(page.locator("#stat-chunks")).to_have_text("5")
expect(page.locator("#sources-empty")).to_be_hidden()
_drill_to_source(page, SOURCE_NAME)
expect(page.locator("#docs-tbody tr")).to_have_count(2)
# The PDF — a plain text row (no thumbnail box at all). The row is
# pinned by the EXACT doc-link text (the image row's path carries
# ``report.pdf`` as a prefix — a substring match would hit both).
pdf_row = page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{DOC_PATH}")'),
)
expect(pdf_row).to_have_count(1)
expect(pdf_row.locator(".kb-doc-thumb")).to_have_count(0)
# The embedded image — the thumbnail row (the bytes route's lazy
# <img>, alt = the description — the WCAG contract).
img_row = page.locator(
"#docs-tbody tr",
has=page.locator(f'a.doc-link:text-is("{IMAGE_REL}")'),
)
expect(img_row).to_have_count(1)
img = img_row.locator(".kb-doc-thumb-img")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000) # the fetch resolves — no glyph
expect(img).to_have_attribute("src", seeded["image_url"])
expect(img).to_have_attribute("loading", "lazy")
expect(img).to_have_attribute("alt", seeded["description"])
expect(img_row.locator(".kb-doc-thumb-glyph")).to_have_count(0)
# ---------------------------------------------------------------------------
# 2. Document viewer (images ON): the extracted text renders (A4)
# ---------------------------------------------------------------------------
def test_document_viewer_renders_the_extracted_text(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""The Sources row's path link opens the same-page document modal
(phase 26): the PDF renders its EXTRACTED TEXT (LOCKED A4 — the
``---PAGE-N---`` page markers + the page texts in the plain-content
slot, never binary bytes), under the METADATA title (LOCKED A2),
with the phase-30 Summary panel above (a non-image doc with a
non-empty summary — the panel renders)."""
login(page, app_url)
_drill_to_source(page, SOURCE_NAME)
page.locator(f'#docs-tbody a.doc-link:text-is("{DOC_PATH}")').click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
# The extracted text (A4) — both page markers and both page texts.
raw = modal.locator("pre.doc-raw")
expect(raw).to_contain_text("---PAGE-1---")
expect(raw).to_contain_text(PAGE_ONE)
expect(raw).to_contain_text("---PAGE-2---")
expect(raw).to_contain_text(PAGE_TWO)
# The summary panel (a non-image doc — the panel renders; the
# verbatim-description suppression applies to image docs only).
expect(modal.locator(".doc-summary")).to_have_count(1)
# No image block — the PDF is not an image document.
expect(modal.locator(".doc-image-img")).to_have_count(0)
# ---------------------------------------------------------------------------
# 3. Document viewer (images ON): the embedded image doc renders
# ---------------------------------------------------------------------------
def test_document_viewer_renders_the_embedded_image_doc(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""The embedded image's row (the synthetic ``#page=2#img=0.png``
path) opens the modal: the PERSISTENT copy renders first (the bytes
route, alt = the description), the description follows in the
content slot, and the labeled Summary panel is SUPPRESSED for the
verbatim-description case (summary === content — the importer
invariant; the panel would duplicate the text right below the
image)."""
login(page, app_url)
_drill_to_source(page, SOURCE_NAME)
page.locator(f'#docs-tbody a.doc-link:text-is("{IMAGE_REL}")').click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
img = modal.locator(".doc-image-img")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000) # the bytes route serves the PNG
expect(img).to_have_attribute("src", seeded["image_url"])
expect(img).to_have_attribute("alt", seeded["description"])
expect(modal.locator("pre.doc-raw")).to_have_text(seeded["description"])
# The verbatim case: no duplicate Summary panel.
expect(modal.locator(".doc-summary")).to_have_count(0)
# No "Image unavailable" note — the copy is intact.
expect(modal.locator(".doc-image-unavailable")).to_have_count(0)
# ---------------------------------------------------------------------------
# 4. Chat (images ON): the answer grounds on the PDF; the embedded
# image is auto-cited with its figure
# ---------------------------------------------------------------------------
def test_chat_grounds_on_the_pdf_and_chips_the_embedded_image(
page: Page, app_url: str, seeded: dict[str, str]
) -> None:
"""A question the mock grounds on (the summary chunk's cosine
≈0.61 over the 0.30 gate; FTS corroborates), scripted to READ the
PDF (phase 119 A1): the echo carries the read result VERBATIM —
the document's FULL extracted text (LOCKED A4 — the page markers
and both page texts reach the model and land in the answer), and
the sources block carries the PDF's citation chip (the READ doc)
PLUS the embedded image's AUTO-cited chip (phase 124 — the
suggested-tier image doc) with its compact inline figure (the
bytes route, caption/alt settling to the description)."""
login(page, app_url, next="/")
expect(page.locator("#kb-banner")).to_be_hidden()
page.fill("#message-input", QUESTION)
page.click("#send-btn")
bubble = page.locator(".msg.brain .bubble").last
# The scripted read's echoed result — the read header and the FULL
# extracted text (page markers and all).
expect(bubble).to_contain_text("Here's what the read returned:", timeout=30_000)
expect(bubble).to_contain_text(f"Document {SOURCE_NAME}/{DOC_PATH}:")
expect(bubble).to_contain_text("---PAGE-1---")
expect(bubble).to_contain_text(PAGE_ONE)
expect(bubble).to_contain_text("---PAGE-2---")
expect(bubble).to_contain_text(PAGE_TWO)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send")
# The citation chips: the READ PDF first (read order), then the
# suggested image doc's auto-cite (phase 124).
chip = page.locator(".msg.brain .source-chip")
expect(chip).to_have_count(2)
expect(chip.nth(0)).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
expect(chip.nth(1)).to_have_text(f"{SOURCE_NAME}/{IMAGE_REL}")
# The inline figure on the image chip: image + caption, both
# settling to the description (the content fetch resolves the
# summary after the title).
fig = page.locator(".msg.brain .source-image")
expect(fig).to_have_count(1)
img = fig.locator(".source-image-img")
expect(img).to_be_visible(timeout=15_000)
expect(img).to_have_attribute("src", seeded["image_url"])
expect(img).to_have_attribute("alt", seeded["description"], timeout=15_000)
caption = fig.locator(".source-image-caption")
expect(caption).to_have_text(seeded["description"], timeout=15_000)
# ---------------------------------------------------------------------------
# 5. Default app (images off): the same upload + sync indexes the PDF
# text normally and skips the embedded image
# ---------------------------------------------------------------------------
def test_images_off_upload_indexes_the_pdf_and_skips_the_image(
page: Page, default_app_url: str, seeded_off: dict[str, str]
) -> None:
"""The toggle gate end-to-end (LOCKED A3): with ``BOR_IMAGES``
false the PDF's text indexes normally (1 added — the PDF only; the
embedded image is skipped, not a failure) while NO image document
exists (no row, no copy). The Sources page reads 1 doc / 3 chunks
and the drill-down table lists the PDF alone."""
login(page, default_app_url) # lands on /sources.html (admin)
expect(page.locator("#stat-docs")).to_have_text("1")
expect(page.locator("#stat-chunks")).to_have_text("3")
_drill_to_source(page, SOURCE_NAME)
row = page.locator("#docs-tbody tr", has_text=DOC_PATH)
expect(row).to_have_count(1)
expect(row.locator(".kb-doc-thumb")).to_have_count(0)
expect(page.locator("#docs-tbody tr", has_text=IMAGE_REL)).to_have_count(0)
# ---------------------------------------------------------------------------
# 6. Chat (images off): the answer grounds on the PDF; NO image chip
# ---------------------------------------------------------------------------
def test_images_off_chat_grounds_with_no_image_chip(
page: Page, default_app_url: str, seeded_off: dict[str, str]
) -> None:
"""The SAME question on the off app: the turn grounds on the PDF's
extracted text (the read echo carries the page markers + texts —
the PDF text is indexed normally, LOCKED A3's off-contract) and the
sources block carries EXACTLY the PDF's chip — no image chip, no
figure (the embedded image was never indexed)."""
try:
login(page, default_app_url, next="/")
expect(page.locator("#kb-banner")).to_be_hidden()
page.fill("#message-input", QUESTION)
page.click("#send-btn")
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text("Here's what the read returned:", timeout=30_000)
expect(bubble).to_contain_text(f"Document {SOURCE_NAME}/{DOC_PATH}:")
expect(bubble).to_contain_text(PAGE_ONE)
expect(bubble).to_contain_text(PAGE_TWO)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
chip = page.locator(".msg.brain .source-chip")
expect(chip).to_have_count(1)
expect(chip.nth(0)).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
# No image affordance at all (no chip, no figure).
expect(page.locator(".msg.brain .source-image")).to_have_count(0)
finally:
_truncate_all()
+469
View File
@@ -33,7 +33,10 @@ from __future__ import annotations
import asyncio
import base64
import hashlib
from pathlib import Path
import pymupdf as fitz # PyMuPDF — the temp-PDF fixture builder (phase 126)
from sqlalchemy import select
from app.config import Settings
@@ -444,3 +447,469 @@ def test_markdown_referencing_existing_image_keeps_null_record(db, tmp_path) ->
assert md3 is not None and md3.image_context is None
finally:
_cleanup_source(db, source_root.name)
# ---------------------------------------------------------------------------
# Phase 126 — PDF indexing through the REAL import pipeline (the real
# ``app.rag.pdf`` extractor, the page-aware chunker, the REAL
# ``_describe_or_skip`` → ``describe_image`` seam — NO seam patch).
# ---------------------------------------------------------------------------
PDF_TITLE = "Quarterly Homelab Report"
PAGE_ONE = "The core switch firmware was upgraded to the stable channel."
PAGE_TWO = "The backup schedule runs nightly at 02:00 and keeps seven rotations."
def _png_bytes(width: int, height: int) -> bytes:
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height), 0)
pix.clear_with(200)
return pix.tobytes("png")
def _build_pdf(
path, pages: list[tuple[str, list[bytes]]], title: str | None = None
) -> None:
"""One temp PDF (built with PyMuPDF — no binary fixtures): *pages* is
(page text, embedded-image bytes) per page; *title* sets the metadata
title (``None`` = leave it empty)."""
doc = fitz.open()
for text, imgs in pages:
page = doc.new_page()
if text:
page.insert_text((72, 72), text)
y = 150
for data in imgs:
pix = fitz.Pixmap(data)
page.insert_image(
fitz.Rect(72, y, 72 + pix.width, y + pix.height), stream=data
)
y += pix.height + 10
if title is not None:
doc.set_metadata({"title": title})
doc.save(path)
doc.close()
def test_pdf_becomes_document_with_text_chunks_embedding_and_summary(
db, tmp_path
) -> None:
"""The phase-126 primary scenario: a PDF in a source (the ``pdf``
import token) → sync → the document carries the extracted text
(the ``---PAGE-N---`` markers, A2 — the title is the metadata
title), page-anchored chunks with 768-dim embeddings, and the
phase-30/118 lite summary. With ``images=True`` (A3) the embedded
image joins the pipeline through the REAL seam — one vision call on
the CHAT model with the bare locked prompt (no markdown reference
exists — ``image_context`` stays NULL), the copy in ``image_dir``,
and the description as the image doc's content AND summary."""
source_root = tmp_path / "pdfkb"
source_root.mkdir()
png = _png_bytes(40, 30)
_build_pdf(
source_root / "report.pdf",
[(PAGE_ONE, []), (PAGE_TWO, [png])],
title=PDF_TITLE,
)
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
try:
summary = asyncio.run(import_sources([source_root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 2, 0)
assert summary.formats == {"pdf": 1}
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "report.pdf"
)
)
assert doc is not None and doc.is_image is False
assert doc.title == PDF_TITLE
assert doc.content == f"---PAGE-1---\n{PAGE_ONE}\n\n---PAGE-2---\n{PAGE_TWO}"
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
content_chunks = [c for c in chunks if not c.is_summary]
assert [c.content for c in content_chunks] == [
f"Page 1:\n{PAGE_ONE}",
f"Page 2:\n{PAGE_TWO}",
]
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert doc.summary is not None
assert doc.summary.endswith(f"Source: {source_root.name}/report.pdf")
# The embedded image (A3) — through the REAL describe seam.
image_rel = "report.pdf#page=2#img=0.png"
img = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == image_rel
)
)
assert img is not None and img.is_image is True
assert img.image_context is None, "no markdown reference → bare prompt"
assert img.content == DESCRIPTION and img.summary == DESCRIPTION
copy = Path(img.image_path)
assert copy.parent == Path(llm.settings.image_dir).expanduser()
assert copy.name == f"{img.id}.png"
assert copy.read_bytes() == png
# The vision call went to the CHAT model with the BARE locked
# prompt (the PDF's lite summary is the other chat call).
assert llm.chat_models == [
llm.settings.llm_summary_model, llm.settings.llm_chat_model
]
assert llm.text_parts == [DESCRIBE_PROMPT]
finally:
_cleanup_source(db, source_root.name)
def test_pdf_embedded_images_skipped_when_images_off(db, tmp_path) -> None:
"""With ``images=False`` (the DEFAULT): the PDF text is indexed
normally and the embedded images are skipped — no image documents,
no ``images_failed`` (a skip is not a failure), no copies, no vision
calls."""
source_root = tmp_path / "pdfoffkb"
source_root.mkdir()
_build_pdf(
source_root / "report.pdf",
[(PAGE_ONE, [_png_bytes(40, 30)]), (PAGE_TWO, [])],
)
llm = _image_llm(tmp_path, images=False, import_extensions="md,txt,pdf")
try:
summary = asyncio.run(import_sources([source_root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "report.pdf"
)
)
assert doc is not None and PAGE_ONE in doc.content
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
assert any(c.embedding is not None for c in chunks)
assert (
db.scalar(
select(Document).where(
Document.source == source_root.name,
Document.is_image.is_(True),
)
)
is None
), "images off: no embedded-image documents"
# No vision call at all — only the PDF's lite summary.
assert llm.chat_models == [llm.settings.llm_summary_model]
finally:
_cleanup_source(db, source_root.name)
def test_pdf_without_embedded_images_and_unchanged_rerun(db, tmp_path) -> None:
"""A PDF without embedded images → no image documents; an unchanged
re-sync → ``unchanged`` (no re-extract, no re-embed, no vision or
embed calls), and a second run never creates image documents out of
nothing."""
source_root = tmp_path / "pdfnonekb"
source_root.mkdir()
_build_pdf(
source_root / "notes.pdf",
[(PAGE_ONE, []), (PAGE_TWO, [])],
)
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
try:
s1 = asyncio.run(import_sources([source_root], llm, session=db))
assert (s1.files, s1.added, s1.unchanged) == (1, 1, 0)
assert (
db.scalar(
select(Document).where(
Document.source == source_root.name,
Document.is_image.is_(True),
)
)
is None
)
calls1, chat1 = len(llm.calls), len(llm.chat_models)
s2 = asyncio.run(import_sources([source_root], llm, session=db))
assert (s2.files, s2.added, s2.updated, s2.unchanged) == (1, 0, 0, 1)
assert len(llm.calls) == calls1 and len(llm.chat_models) == chat1
finally:
_cleanup_source(db, source_root.name)
def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
"""A PDF with no extractable text (image-only, ``images`` OFF — the
embedded image is invisible) → the document is still created, with
EMPTY content (the empty byte string's digest), NO content chunks
and NO content embedding (there is no text to embed — the only
chunk is the best-effort ``lite`` summary's ``is_summary`` row), and
the warning line is the signal."""
source_root = tmp_path / "pdfemptykb"
source_root.mkdir()
_build_pdf(source_root / "pics.pdf", [("", [_png_bytes(40, 30)])])
llm = _image_llm(tmp_path, images=False, import_extensions="md,txt,pdf")
try:
summary = asyncio.run(import_sources([source_root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
doc = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "pics.pdf"
)
)
assert doc is not None and doc.is_image is False
assert doc.content == ""
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
).all()
assert [c for c in chunks if not c.is_summary] == [], (
"no text → no content chunks (and no content embedding)"
)
# The phase-30/118 path still runs best-effort on the empty
# content — the summary is the only chunk, and it is embedded.
assert len(chunks) == 1 and chunks[0].is_summary
assert chunks[0].embedding is not None and len(chunks[0].embedding) == 768
assert doc.summary is not None
finally:
_cleanup_source(db, source_root.name)
def test_pdf_embedded_image_survives_prune_and_leaves_with_the_pdf(
db, tmp_path
) -> None:
"""Phase 126 (A3) prune parent rule — the defect the E2E surfaced:
the embedded image doc's synthetic path NEVER appears in the walk's
``seen``, so the plain prune rule destroyed it in the very sync that
(re)indexed it (the toggle-on half of the phase-122 "invisible, not
deleted" guard was missing). Now: with the parent PDF's rel still in
the walk — even UNCHANGED, so no image pass runs at all — the image
doc survives (0 pruned); when the PDF LEAVES the source, the image
doc is pruned with it (row + ``image_dir`` copy — no orphans)."""
source_root = tmp_path / "pdfprune"
source_root.mkdir()
png = _png_bytes(40, 30)
_build_pdf(
source_root / "report.pdf",
[(PAGE_ONE, []), (PAGE_TWO, [png])],
)
image_rel = "report.pdf#page=2#img=0.png"
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
try:
s1 = asyncio.run(
import_sources([source_root], llm, session=db, prune=True)
)
assert (s1.added, s1.pruned) == (2, 0)
img1 = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == image_rel
)
)
assert img1 is not None and img1.image_path is not None
copy = Path(img1.image_path)
# Unchanged re-sync (prune ON): the PDF's text digest matches →
# no image pass at all — the image doc survives anyway (the
# parent's rel is in the walk's seen set).
s2 = asyncio.run(
import_sources([source_root], llm, session=db, prune=True)
)
assert (s2.unchanged, s2.pruned) == (1, 0)
assert db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == image_rel
)
) is not None, "parent PDF present → the image doc survives the prune"
# The PDF LEAVES the source: the image doc is pruned WITH it
# (row + copy — no orphan in image_dir).
(source_root / "report.pdf").unlink()
s3 = asyncio.run(
import_sources([source_root], llm, session=db, prune=True)
)
assert (s3.pruned, s3.unchanged) == (2, 0)
assert db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == image_rel
)
) is None, "parent PDF gone → the image doc is pruned with it"
assert not copy.exists(), "the pruned image's copy leaves image_dir"
finally:
_cleanup_source(db, source_root.name)
# --- Phase 125 parity: a markdown file referencing a PDF-embedded image ---
#: A markdown file referencing the PDF's embedded image by its SYNTHETIC
#: path (``<pdf path>#page=<N>#img=<index>.<ext>`` — the doc's ``path``
#: is the upsert key, and the reference resolves to it through the shared
#: ``_resolve_ref_path`` rule) — one context paragraph before, one after.
PDF_GUIDE_MD = (
"The VLAN map diagram in the report shows the subnet layout.\n"
"\n"
"![VLAN map](report.pdf#page=2#img=0.png)\n"
"\n"
"Only expand the lab subnet after the VLAN map is reviewed.\n"
)
#: The same reference against ``alpha.pdf`` (the PDF-FIRST ordering's
#: fixture — ``alpha.pdf`` sorts before ``zguide.md`` in the walk).
PDF_ZGUIDE_MD = PDF_GUIDE_MD.replace("report.pdf", "alpha.pdf")
def _pdf_expected_block(source: str, md_name: str, pdf_name: str) -> str:
"""The task-03-shape combined string for the PDF reference (the disk
scan builds exactly this — the standalone image's shape, the
synthetic path as the image identity)."""
return (
f"Referenced in {source}/{md_name}:\n"
'- Alt text: "VLAN map"\n'
" Context: The VLAN map diagram in the report shows the subnet layout.\n"
"\n"
"Only expand the lab subnet after the VLAN map is reviewed."
)
def test_pdf_embedded_image_referenced_by_markdown_markdown_first(
db, tmp_path
) -> None:
"""The MARKDOWN-FIRST ordering (``guide.md`` sorts before
``report.pdf``): the markdown is indexed while its referenced image
has no document yet → it records the reference on its OWN row (the
``IMAGE_CONTEXT:`` record — phase 125, task 04, work 1, never
cleared); the PDF is indexed next and the task-03 disk scan (the
standalone image's parity path — the markdown is still on disk) finds
the reference, so the image doc is enriched AT INDEX TIME: the vision
call carries the context after the locked prompt, ``image_context``
is set, ``content``/``summary`` are ``<context>\\n\\n<description>``.
The post-index pass then skips the already-enriched doc (idempotent).
Final state: the record AND the enrichment coexist."""
source_root = tmp_path / "pdfmdfirst"
source_root.mkdir()
(source_root / "guide.md").write_text(PDF_GUIDE_MD, encoding="utf-8")
_build_pdf(
source_root / "report.pdf",
[(PAGE_ONE, []), (PAGE_TWO, [_png_bytes(40, 30)])],
)
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
try:
summary = asyncio.run(import_sources([source_root], llm, session=db))
# guide.md + report.pdf + the embedded image doc: all added.
assert (summary.files, summary.added, summary.images_failed) == (2, 3, 0)
expected_block = _pdf_expected_block(source_root.name, "guide.md", "report.pdf")
# Work 1 — the markdown doc STORES the reference (the record —
# line 3 is the reference, 2 context paragraphs). It is the
# post-index pass's reference index and the retrieval display's
# record; the pass does not clear it on success.
md = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "guide.md"
)
)
assert md is not None and md.is_image is False
assert md.image_context == (
"IMAGE_CONTEXT:\n"
'- [report.pdf#page=2#img=0.png] (line 3): "VLAN map"'
" — 2 paragraphs of context"
)
# The enrichment (the disk scan at PDF index time) — the image
# doc's shape is the standalone image's enriched shape.
image_rel = "report.pdf#page=2#img=0.png"
img = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == image_rel
)
)
assert img is not None and img.is_image is True
assert img.image_context == expected_block
expected_content = f"{expected_block}\n\n{DESCRIPTION}"
assert img.content == expected_content
assert img.summary == expected_content
# The vision call carried the context AFTER the locked prompt
# (the multimodal part list's text part — the describe seam's
# phase-125 extension).
assert llm.text_parts == [f"{DESCRIBE_PROMPT}{CONTEXT_PREFIX}{expected_block}"]
# Chat call order (walk order): guide.md's ``lite`` summary,
# the PDF's ``lite`` summary, then the image's CHAT-model
# description (the ONLY multimodal call).
assert llm.chat_models == [
llm.settings.llm_summary_model,
llm.settings.llm_summary_model,
llm.settings.llm_chat_model,
]
# The chunks are embedded from the ENRICHED content (the
# plain-text packer joins the enriched paragraphs with single
# newlines; the summary chunk mirrors the raw content).
packed = "\n".join(expected_content.split("\n\n"))
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == img.id).order_by(Chunk.position)
).all()
assert [c.content for c in chunks if not c.is_summary] == [packed]
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].content == expected_content
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
finally:
_cleanup_source(db, source_root.name)
def test_pdf_embedded_image_referenced_by_markdown_pdf_first(
db, tmp_path
) -> None:
"""The PDF-FIRST ordering (``alpha.pdf`` sorts before ``zguide.md``
in the walk): the PDF (and its image doc) is indexed while the
markdown is still un-indexed — the task-03 disk scan sees the
reference on disk and enriches the image AT INDEX TIME (the ordering
that, without the parity scan, would ship an un-enriched image — the
markdown's record stays NULL: at markdown index time the image doc
ALREADY exists, so the task-03 path owns the case and nothing is
recorded). Same final state as the markdown-first test — the
enrichment never depends on the walk order."""
source_root = tmp_path / "pdfpdffirst"
source_root.mkdir()
(source_root / "zguide.md").write_text(PDF_ZGUIDE_MD, encoding="utf-8")
_build_pdf(
source_root / "alpha.pdf",
[(PAGE_ONE, []), (PAGE_TWO, [_png_bytes(40, 30)])],
)
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
try:
summary = asyncio.run(import_sources([source_root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (2, 3, 0)
expected_block = _pdf_expected_block(source_root.name, "zguide.md", "alpha.pdf")
# The task-03 path owns this case: the image doc existed when
# the markdown was indexed → NO record (the field is the
# pending-enrichment index, not a reference dump).
md = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == "zguide.md"
)
)
assert md is not None and md.image_context is None
# The image doc is enriched (the disk scan at PDF index time) —
# byte-identical in shape to the markdown-first final state.
image_rel = "alpha.pdf#page=2#img=0.png"
img = db.scalar(
select(Document).where(
Document.source == source_root.name, Document.path == image_rel
)
)
assert img is not None and img.image_context == expected_block
expected_content = f"{expected_block}\n\n{DESCRIPTION}"
assert img.content == expected_content
assert img.summary == expected_content
assert llm.text_parts == [f"{DESCRIBE_PROMPT}{CONTEXT_PREFIX}{expected_block}"]
# Chat call order (walk order): the PDF's ``lite`` summary, the
# image's CHAT-model description, then zguide.md's summary.
assert llm.chat_models == [
llm.settings.llm_summary_model,
llm.settings.llm_chat_model,
llm.settings.llm_summary_model,
]
finally:
_cleanup_source(db, source_root.name)
+89
View File
@@ -419,6 +419,93 @@ def test_dispatch_still_falls_back_for_unknown_suffix() -> None:
assert chunk_document(content, "x/notes.whatever") == chunk_text(content)
# ---------------------------------------------------------------------------
# pdf (phase 126, task 03, LOCKED A2) — the page-aware chunker
# ---------------------------------------------------------------------------
def test_pdf_single_page_chunks_with_page_anchor() -> None:
content = "---PAGE-1---\nHello single page.\n"
chunks = chunk_document(content, "report.pdf")
assert chunks == ["Page 1:\nHello single page."]
assert "---PAGE-" not in "\n".join(chunks)
def test_pdf_multi_page_keeps_the_marker_page_number() -> None:
"""Blank pages carry no marker — the anchor follows the MARKER's
number, not the chunk count (page 3 stays page 3 when page 2 was
blank)."""
content = (
"---PAGE-1---\nAlpha text one.\n\n"
"---PAGE-3---\nGamma text three.\n"
)
chunks = chunk_document(content, "report.pdf")
assert chunks == ["Page 1:\nAlpha text one.", "Page 3:\nGamma text three."]
assert "Page 2:" not in "\n".join(chunks)
def test_pdf_markers_never_appear_in_chunks() -> None:
content = (
"---PAGE-1---\nAlpha text one.\n\n"
"---PAGE-2---\nBeta text two.\n\n"
"---PAGE-3---\nGamma text three.\n"
)
chunks = chunk_document(content, "deep/dir/report.pdf")
assert len(chunks) == 3
for chunk in chunks:
assert "---PAGE-" not in chunk
assert chunks[0].startswith("Page 1:")
assert chunks[1].startswith("Page 2:")
assert chunks[2].startswith("Page 3:")
def test_pdf_empty_and_markerless_content_yield_no_chunks() -> None:
assert chunk_document("", "report.pdf") == []
# No marker at all (the parser always emits one for text — belt and
# braces): nothing to anchor, no chunks (never plain-text fallback —
# a PDF chunk must carry its page).
assert chunk_document("just words, no marker", "report.pdf") == []
assert chunk_document("---PAGE-1---\n\n---PAGE-2---\n ", "report.pdf") == []
def test_pdf_long_page_sub_splits_and_every_chunk_keeps_the_anchor() -> None:
"""A page larger than the target sub-splits through the paragraph
packer — EVERY chunk of the page keeps the ``Page N:`` anchor and
the anchored chunk never exceeds the target."""
page = "\n\n".join(f"paragraph {i} " + "l" * 300 for i in range(6))
content = f"---PAGE-1---\n{page}\n\n---PAGE-2---\nshort second page\n"
chunks = chunk_document(content, "report.pdf", target_chars=800, overlap_chars=100)
assert len(chunks) >= 3 # the long page sub-splits + the short page
page1 = [c for c in chunks if c.startswith("Page 1:")]
page2 = [c for c in chunks if c.startswith("Page 2:")]
assert len(page1) >= 2 and page2 == ["Page 2:\nshort second page"]
for chunk in chunks:
assert len(chunk) <= 800, f"target exceeded: {len(chunk)}"
# All six paragraphs survive the split.
joined = "\n".join(chunks)
for i in range(6):
assert f"paragraph {i}" in joined
def test_pdf_overlap_carryover_stays_within_the_page() -> None:
"""The tail carry-over packs paragraphs INSIDE one page; a new page
never inherits the previous page's tail (the page is the boundary)."""
a = "\n\n".join(f"para-a-{i} " + "l" * 250 for i in range(3))
b = "\n\n".join(f"para-b-{i} " + "l" * 250 for i in range(3))
content = f"---PAGE-1---\n{a}\n\n---PAGE-2---\n{b}\n"
chunks = chunk_document(content, "r.pdf", target_chars=700, overlap_chars=120)
first_of_page2 = next(c for c in chunks if c.startswith("Page 2:"))
assert "para-a-" not in first_of_page2, "page 2 must not inherit page 1's tail"
assert all(len(c) <= 700 for c in chunks)
def test_pdf_dispatch_matches_direct_call() -> None:
content = "---PAGE-1---\nalpha\n\n---PAGE-2---\nbeta\n"
from app.rag.chunker import _chunk_pdf
assert chunk_document(content, "x/report.PDF") == _chunk_pdf(content)
# ---------------------------------------------------------------------------
# Hard cap across every format (aipi ~1024-token request cap)
# ---------------------------------------------------------------------------
@@ -434,6 +521,8 @@ def test_dispatch_still_falls_back_for_unknown_suffix() -> None:
("line of text\n\n" * 800, "big.txt"),
("[Unit]\n" + "key=" + "v" * 5000 + "\n", "big.container"), # phase 47
("{% for x in y %}" + "{{ x }} " * 400 + "{% endfor %}", "big.j2"), # phase 47
# phase 126: one long page (a single 12 500-char line) + one short.
("---PAGE-1---\n" + "word " * 2500 + "\n\n---PAGE-2---\nshort\n", "big.pdf"),
],
)
def test_hard_cap_holds_for_every_format(content: str, path: str) -> None:
File diff suppressed because it is too large Load Diff