phase: 126_pdf_support
All gates verified green — no code changes were needed on this pass. Final report: ## Phase 126 — PDF support: final verification pass **Verified (all tasks 01–05 already complete in working tree):** - `pymupdf` dep in `pyproject.toml`/`uv.lock`; `.env.example` documents the `pdf` token - `app/rag/pdf.py` (text + image extraction, MIME mapping), page-aware `_chunk_pdf`, importer `_index_pdf_file` + embedded-image pass, archives covered via extension walk **Test / lint / coverage results:** - `uv run pytest` → 2997 passed - `uv run pytest tests/e2e/test_pdf_support.py -v --no-cov` (isolation) → 6 passed - `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90% gate), 2997 passed - `uv run ruff check . && uv run pyright` → clean (0 errors) **Completion criteria:** 1. PDF → doc with text/chunks/embedding/summary — ✅ (integration + E2E Sources-page tests) 2. `BOR_IMAGES=true`: embedded images → standalone image docs — ✅ (integration + E2E chat-chip test) 3. `BOR_IMAGES=false`: images skipped, text indexed — ✅ (both suites) 4. PDF renders as extracted text in viewer — ✅ (E2E) 5. Test/coverage/lint gates — ✅ (above) 6. Commit + phase move — harness handles (per protocol, no `git` on this pass) **Decisions:** none new; inline `[Embedded image: …]` placeholders were a documented task-04 deviation (simplified spec wins) — not a defect. No defects found; repo left functional with changes in the working tree. **Next pending phase:** none — `todo/` is empty once 126 is moved.
This commit is contained in:
@@ -0,0 +1,644 @@
|
||||
"""Phase 126 E2E (Playwright): PDF support — the user path.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_pdf_support.py -v --no-cov
|
||||
|
||||
PDFs are first-class, retrievable documents behind the ``pdf`` token of
|
||||
``BOR_IMPORT_EXTENSIONS`` (the two module apps pin the token EXPLICITLY
|
||||
— an operator's local gitignored ``.env`` cannot leak the scope in
|
||||
either direction, and without the token the walk is blind to the file):
|
||||
|
||||
* the story app (``BOR_IMAGES=true``): the fixture PDF (two text pages
|
||||
+ one embedded PNG on page 2, built with PyMuPDF in the fixture —
|
||||
LOCKED A1) arrives through the REAL admin upload flow (a zip with a
|
||||
single PDF member — the archive unpack is content-agnostic, the
|
||||
extension walk admits it once ``pdf`` is in the token set) and is
|
||||
first-class at every surface:
|
||||
- the Sources page lists it (the extracted text's document — a plain
|
||||
text row, no thumbnail) plus the embedded image's document (the
|
||||
phase-122 pipeline, LOCKED A3 — the 48px thumbnail from the bytes
|
||||
route, alt = the mock vision description);
|
||||
- the document viewer renders the EXTRACTED TEXT (LOCKED A4 — the
|
||||
``---PAGE-N---`` page markers and the page text, never binary
|
||||
bytes) with the metadata title (LOCKED A2 — one document per PDF);
|
||||
- a grounded chat question (the mock's token-overlap cosine against
|
||||
the summary chunk ≈0.61, over the mock-calibrated gate 0.30 —
|
||||
FTS-corroborated anyway) scripted to READ it echoes the read result
|
||||
verbatim (the extracted text reaches the model AND the answer), and
|
||||
the sources block carries the PDF's citation chip PLUS the embedded
|
||||
image's auto-cited chip + compact inline figure (phase 124 — the
|
||||
suggested-tier image doc, best chunk ≈0.22 over the usefulness
|
||||
floor);
|
||||
* the default app (``BOR_IMAGES`` forced ``false``): the SAME upload +
|
||||
sync indexes the PDF's text normally — the embedded image is skipped
|
||||
(no image document, no copy, the source is a 1-document source) and
|
||||
the chat turn grounds with the PDF's chip only (no image chip, no
|
||||
figure — LOCKED A3's toggle gate end-to-end).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import zipfile
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
|
||||
import httpx
|
||||
import pymupdf as fitz # PyMuPDF — the fixture-PDF builder (LOCKED A1)
|
||||
import pytest
|
||||
from playwright.sync_api import Browser, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||
from e2e.mock_llm import IMAGE_DESCRIPTION_ANSWER
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
SOURCE_NAME = "e2e-pdf" # the archive stem (archive_source_name)
|
||||
DOC_PATH = "report.pdf"
|
||||
IMAGE_REL = "report.pdf#page=2#img=0.png" # the synthetic embedded-image path
|
||||
PDF_TITLE = "Quarterly Homelab Report" # the metadata title (LOCKED A2)
|
||||
|
||||
#: The fixture PDF's page texts (the chunker's ``Page N:`` anchors make
|
||||
#: the chunks ``Page 1:\n<text>`` / ``Page 2:\n<text>``). The question
|
||||
#: below shares its tokens with them (the mock's token-overlap cosine —
|
||||
#: the summary chunk ≈0.61 — clears the mock-calibrated gate 0.30).
|
||||
PAGE_ONE = "The quarterly homelab report covers the core switch firmware upgrade."
|
||||
PAGE_TWO = "The homelab backup schedule runs nightly at 02:00 and keeps seven rotations."
|
||||
|
||||
#: The grounded question, carrying the house scripted-read call
|
||||
#: (``SUMMARY_SEED_READ_TRIGGER`` — the phase-119 A1 convention: a
|
||||
#: zero-read grounded turn chips NOTHING, so the PDF doc must be READ to
|
||||
#: earn its citation chip): the mock emits the scripted
|
||||
#: ``read e2e-pdf/report.pdf``, then echoes the tool result VERBATIM —
|
||||
#: the read result carries the document's FULL extracted text, so the
|
||||
#: echo in the answer is the E2E lens on LOCKED A4 (the PDF served as
|
||||
#: extracted text, page markers and all).
|
||||
QUESTION = (
|
||||
"Read the suggested document: read e2e-pdf/report.pdf — "
|
||||
"what does the homelab report say about the backup schedule?"
|
||||
)
|
||||
|
||||
#: A real 1×1 transparent PNG (the unit/integration suites' fixture —
|
||||
#: the pipeline is content-agnostic; the well-formed bytes keep the
|
||||
#: extraction + copy + serve + render path honest).
|
||||
PNG_1X1 = base64.b64decode(
|
||||
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||
)
|
||||
|
||||
UPLOAD_TIMEOUT_MS = 30_000
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
SYNCED_LABEL = re.compile(r"^Synced \d{1,2}:\d{2}$")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Module apps: images ON (the story) and images forced OFF (the default
|
||||
# contract). Each owns its port + its scratch upload/source/image dirs.
|
||||
# Both pin BOR_IMPORT_EXTENSIONS WITH the ``pdf`` token explicitly
|
||||
# (phase 126 — the feature lives behind the token, and process env ranks
|
||||
# above an operator's local gitignored ``.env``).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _app_env(mock_llm: int, app_port: int, *, images: bool, scratch: Path) -> dict[str, str]:
|
||||
"""The conftest app env (leak guards included) with the phase-126
|
||||
knobs: ``BOR_IMPORT_EXTENSIONS`` pinned to a set carrying the
|
||||
``pdf`` token, ``BOR_IMAGES`` pinned EXPLICITLY (true for the story
|
||||
app, false for the default app — the contract under test cannot
|
||||
leak in either direction), and the image/upload homes in the suite's
|
||||
scratch dir (the app under test must not write copies into the
|
||||
owner's real ``~/bor-sources``)."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern) — the story's question
|
||||
# grounds on these values (see QUESTION).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget.
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# Leak guards (conftest pattern).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||
# Phase 126: the import scope WITH the ``pdf`` token (the feature is
|
||||
# gated by the token — the explicit pin keeps a local ``.env`` from
|
||||
# changing the premise) + the toggle under test + the suite-private
|
||||
# homes.
|
||||
env["BOR_IMPORT_EXTENSIONS"] = "md,markdown,txt,pdf"
|
||||
env["BOR_IMAGES"] = "true" if images else "false"
|
||||
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||
env["BOR_IMAGE_DIR"] = str(scratch / "images")
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, tmp_path_factory: pytest.TempPathFactory) -> Iterator[str]:
|
||||
"""The story app — ``BOR_IMAGES=true`` + the ``pdf`` token (the
|
||||
module env override; the conftest session app is never started in
|
||||
this isolated run, so no port clash)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_pdf_on")
|
||||
port = int(os.environ.get("E2E_APP_PORT_PDF", "8152"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, images=True, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def default_app_server(
|
||||
mock_llm: int, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> Iterator[str]:
|
||||
"""The default-contract app — ``BOR_IMAGES`` forced ``false`` (the
|
||||
toggle gate for the embedded images; only the off-app tests start
|
||||
it)."""
|
||||
scratch = tmp_path_factory.mktemp("bor_pdf_off")
|
||||
port = int(os.environ.get("E2E_APP_PORT_PDF_OFF", "8153"))
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=_app_env(mock_llm, port, images=False, scratch=scratch),
|
||||
)
|
||||
try:
|
||||
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||
yield f"http://127.0.0.1:{port}"
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def default_app_url(default_app_server: str) -> str:
|
||||
return default_app_server
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def zip_path(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The fixture archive: ONE PDF at the root (two text pages + one
|
||||
embedded PNG on page 2, built with PyMuPDF — LOCKED A1) — source
|
||||
``e2e-pdf``, document path ``report.pdf``."""
|
||||
root = tmp_path_factory.mktemp("bor_pdf_zip")
|
||||
pdf_path = root / DOC_PATH
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), PAGE_ONE)
|
||||
page2 = doc.new_page()
|
||||
page2.insert_text((72, 72), PAGE_TWO)
|
||||
page2.insert_image(fitz.Rect(72, 150, 73, 151), stream=PNG_1X1)
|
||||
doc.set_metadata({"title": PDF_TITLE})
|
||||
doc.save(pdf_path)
|
||||
doc.close()
|
||||
archive = root / f"{SOURCE_NAME}.zip"
|
||||
with zipfile.ZipFile(archive, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||
zf.write(pdf_path, arcname=DOC_PATH)
|
||||
return archive
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# DB + UI helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh KB + registry per test group (the E2E isolation pattern):
|
||||
the suites share one Postgres, and a leftover row would corrupt the
|
||||
counts. ``sources_meta`` (the KB generation counter) resets with
|
||||
the KB — the sync re-creates the row."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text(
|
||||
"TRUNCATE chunks, documents, query_log, kb_overview, "
|
||||
"git_sources, sources_meta, folder_summaries"
|
||||
))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _health_db(app_url: str) -> bool:
|
||||
try:
|
||||
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||
return False
|
||||
|
||||
|
||||
def _upload_zip(page: Page, app_url: str, archive: Path) -> None:
|
||||
"""The real admin upload flow: form login → the git-sources page →
|
||||
pick the archive → submit → the terminal result line (phase 90:
|
||||
"Uploaded <source> — press Sync sources to import it.")."""
|
||||
login(page, app_url, next="/git-sources.html")
|
||||
expect(page).to_have_url(app_url + "/git-sources.html", timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
page.set_input_files("#archive-upload-file", str(archive))
|
||||
page.click("#archive-upload-btn")
|
||||
result = page.locator("#archive-upload-result")
|
||||
expect(result).to_be_visible(timeout=UPLOAD_TIMEOUT_MS)
|
||||
expect(result).to_have_text(
|
||||
f"Uploaded {SOURCE_NAME} — press Sync sources to import it.",
|
||||
timeout=UPLOAD_TIMEOUT_MS,
|
||||
)
|
||||
|
||||
|
||||
def _sync_via_ui(page: Page, app_url: str, expected_result: str) -> None:
|
||||
"""The RAG page's "Sync sources" button (the scan): click →
|
||||
"Syncing…" → the terminal "Synced HH:MM" label + the fresh counts
|
||||
in #sync-result (the never-stale contract)."""
|
||||
page.goto(app_url + "/sources.html")
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#sync-label")).to_have_text(
|
||||
re.compile(r"^Sync sources$|^Synced \d{1,2}:\d{2}$")
|
||||
)
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||||
expect(btn).to_be_enabled()
|
||||
expect(page.locator("#sync-result")).to_have_text(expected_result)
|
||||
|
||||
|
||||
def _drill_to_source(page: Page, name: str) -> None:
|
||||
"""Top level → the source (the file table then holds its direct
|
||||
files — ours are at the source root)."""
|
||||
page.locator("#folders-tbody .folder-link").first.wait_for(state="visible")
|
||||
page.click(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||
expect(page.locator("#docs-tbody tr").first).to_be_visible(timeout=30_000)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The module seeds: upload + sync the fixture PDF through the real UI,
|
||||
# once per app (the story E2E truncate/re-import-per-module pattern).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded(
|
||||
app_url: str,
|
||||
browser: Browser,
|
||||
zip_path: Path,
|
||||
) -> Iterator[dict[str, str]]:
|
||||
"""Upload the single-PDF zip through the admin UI on the story app,
|
||||
then scan it with the RAG page's Sync button (the mock's vision
|
||||
model describes the embedded image in-process). Yields the embedded
|
||||
image doc's identity (the bytes route + the description) for the
|
||||
render assertions. The module app's ``/api/config`` flag is
|
||||
asserted first — the env override must have reached the app under
|
||||
test, or the whole premise of the suite is void."""
|
||||
if not _health_db(app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = httpx.get(f"{app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is True, "the module app must run with BOR_IMAGES=true"
|
||||
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_zip(page, app_url, zip_path)
|
||||
# The PDF + its one embedded image (A3): both count added.
|
||||
_sync_via_ui(page, app_url, expected_result="2 added")
|
||||
with SessionLocal() as db:
|
||||
pdf = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||||
)
|
||||
)
|
||||
assert pdf is not None, "the uploaded PDF must become a document"
|
||||
# LOCKED A2: one document per PDF, the extracted text with
|
||||
# the page markers; the title is the metadata title.
|
||||
assert pdf.is_image is False and pdf.image_path is None
|
||||
assert pdf.title == PDF_TITLE
|
||||
assert "---PAGE-1---" in pdf.content and PAGE_ONE in pdf.content
|
||||
assert "---PAGE-2---" in pdf.content and PAGE_TWO in pdf.content
|
||||
assert pdf.summary is not None
|
||||
img = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == IMAGE_REL
|
||||
)
|
||||
)
|
||||
assert img is not None, "the embedded image must become an image doc"
|
||||
assert img.is_image is True and img.image_path is not None
|
||||
# content == summary == the mock's description (LOCKED A3
|
||||
# fail-soft shape; no markdown reference exists — the bare
|
||||
# prompt's answer).
|
||||
assert img.content == IMAGE_DESCRIPTION_ANSWER
|
||||
assert img.summary == IMAGE_DESCRIPTION_ANSWER
|
||||
finally:
|
||||
page.close()
|
||||
yield {
|
||||
"image_url": f"/api/documents/{img.id}/image",
|
||||
"description": IMAGE_DESCRIPTION_ANSWER,
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def seeded_off(
|
||||
default_app_url: str,
|
||||
browser: Browser,
|
||||
zip_path: Path,
|
||||
) -> Iterator[dict[str, str]]:
|
||||
"""The SAME fixture zip through the default app (images off): the
|
||||
PDF's text indexes normally, the embedded image is skipped — the
|
||||
source is a 1-document source. Yields the PDF doc's identity."""
|
||||
if not _health_db(default_app_url):
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = httpx.get(f"{default_app_url}/api/config", timeout=5).json()
|
||||
assert cfg["images"] is False, "the default app must run with images off"
|
||||
|
||||
_truncate_all()
|
||||
page = browser.new_page(viewport={"width": 1280, "height": 800})
|
||||
try:
|
||||
_upload_zip(page, default_app_url, zip_path)
|
||||
_sync_via_ui(page, default_app_url, expected_result="1 added")
|
||||
with SessionLocal() as db:
|
||||
pdf = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == SOURCE_NAME, Document.path == DOC_PATH
|
||||
)
|
||||
)
|
||||
assert pdf is not None, "the uploaded PDF must become a document"
|
||||
assert PAGE_ONE in pdf.content and PAGE_TWO in pdf.content
|
||||
assert (
|
||||
db.scalar(select(Document).where(Document.is_image.is_(True)))
|
||||
is None
|
||||
), "images off: the embedded image is skipped (no image doc)"
|
||||
finally:
|
||||
page.close()
|
||||
yield {"title": PDF_TITLE}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Sources page (images ON): the PDF's text row + the image's thumbnail
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sources_page_lists_the_pdf_and_image_docs(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The KB total is 2 docs / 5 chunks (the PDF: 2 page-anchored
|
||||
content chunks + the phase-30 summary chunk; the image: the
|
||||
description chunk + its summary chunk). The drill-down table lists
|
||||
both documents: the PDF as a PLAIN text row (LOCKED A4 — it is a
|
||||
text document, no thumbnail box) and the embedded image with the
|
||||
FIXED 48px thumbnail (the bytes route, alt = the description)."""
|
||||
login(page, app_url) # lands on /sources.html
|
||||
expect(page.locator("#stat-docs")).to_have_text("2")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("5")
|
||||
expect(page.locator("#sources-empty")).to_be_hidden()
|
||||
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
expect(page.locator("#docs-tbody tr")).to_have_count(2)
|
||||
|
||||
# The PDF — a plain text row (no thumbnail box at all). The row is
|
||||
# pinned by the EXACT doc-link text (the image row's path carries
|
||||
# ``report.pdf`` as a prefix — a substring match would hit both).
|
||||
pdf_row = page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{DOC_PATH}")'),
|
||||
)
|
||||
expect(pdf_row).to_have_count(1)
|
||||
expect(pdf_row.locator(".kb-doc-thumb")).to_have_count(0)
|
||||
|
||||
# The embedded image — the thumbnail row (the bytes route's lazy
|
||||
# <img>, alt = the description — the WCAG contract).
|
||||
img_row = page.locator(
|
||||
"#docs-tbody tr",
|
||||
has=page.locator(f'a.doc-link:text-is("{IMAGE_REL}")'),
|
||||
)
|
||||
expect(img_row).to_have_count(1)
|
||||
img = img_row.locator(".kb-doc-thumb-img")
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_be_visible(timeout=15_000) # the fetch resolves — no glyph
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("loading", "lazy")
|
||||
expect(img).to_have_attribute("alt", seeded["description"])
|
||||
expect(img_row.locator(".kb-doc-thumb-glyph")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Document viewer (images ON): the extracted text renders (A4)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_document_viewer_renders_the_extracted_text(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The Sources row's path link opens the same-page document modal
|
||||
(phase 26): the PDF renders its EXTRACTED TEXT (LOCKED A4 — the
|
||||
``---PAGE-N---`` page markers + the page texts in the plain-content
|
||||
slot, never binary bytes), under the METADATA title (LOCKED A2),
|
||||
with the phase-30 Summary panel above (a non-image doc with a
|
||||
non-empty summary — the panel renders)."""
|
||||
login(page, app_url)
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
page.locator(f'#docs-tbody a.doc-link:text-is("{DOC_PATH}")').click()
|
||||
|
||||
modal = page.locator("#doc-modal")
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
expect(page.locator("#doc-modal-title")).to_have_text(PDF_TITLE)
|
||||
|
||||
# The extracted text (A4) — both page markers and both page texts.
|
||||
raw = modal.locator("pre.doc-raw")
|
||||
expect(raw).to_contain_text("---PAGE-1---")
|
||||
expect(raw).to_contain_text(PAGE_ONE)
|
||||
expect(raw).to_contain_text("---PAGE-2---")
|
||||
expect(raw).to_contain_text(PAGE_TWO)
|
||||
# The summary panel (a non-image doc — the panel renders; the
|
||||
# verbatim-description suppression applies to image docs only).
|
||||
expect(modal.locator(".doc-summary")).to_have_count(1)
|
||||
# No image block — the PDF is not an image document.
|
||||
expect(modal.locator(".doc-image-img")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Document viewer (images ON): the embedded image doc renders
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_document_viewer_renders_the_embedded_image_doc(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""The embedded image's row (the synthetic ``#page=2#img=0.png``
|
||||
path) opens the modal: the PERSISTENT copy renders first (the bytes
|
||||
route, alt = the description), the description follows in the
|
||||
content slot, and the labeled Summary panel is SUPPRESSED for the
|
||||
verbatim-description case (summary === content — the importer
|
||||
invariant; the panel would duplicate the text right below the
|
||||
image)."""
|
||||
login(page, app_url)
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
page.locator(f'#docs-tbody a.doc-link:text-is("{IMAGE_REL}")').click()
|
||||
|
||||
modal = page.locator("#doc-modal")
|
||||
expect(modal).to_be_visible(timeout=30_000)
|
||||
|
||||
img = modal.locator(".doc-image-img")
|
||||
expect(img).to_have_count(1)
|
||||
expect(img).to_be_visible(timeout=15_000) # the bytes route serves the PNG
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("alt", seeded["description"])
|
||||
|
||||
expect(modal.locator("pre.doc-raw")).to_have_text(seeded["description"])
|
||||
# The verbatim case: no duplicate Summary panel.
|
||||
expect(modal.locator(".doc-summary")).to_have_count(0)
|
||||
# No "Image unavailable" note — the copy is intact.
|
||||
expect(modal.locator(".doc-image-unavailable")).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4. Chat (images ON): the answer grounds on the PDF; the embedded
|
||||
# image is auto-cited with its figure
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_chat_grounds_on_the_pdf_and_chips_the_embedded_image(
|
||||
page: Page, app_url: str, seeded: dict[str, str]
|
||||
) -> None:
|
||||
"""A question the mock grounds on (the summary chunk's cosine
|
||||
≈0.61 over the 0.30 gate; FTS corroborates), scripted to READ the
|
||||
PDF (phase 119 A1): the echo carries the read result VERBATIM —
|
||||
the document's FULL extracted text (LOCKED A4 — the page markers
|
||||
and both page texts reach the model and land in the answer), and
|
||||
the sources block carries the PDF's citation chip (the READ doc)
|
||||
PLUS the embedded image's AUTO-cited chip (phase 124 — the
|
||||
suggested-tier image doc) with its compact inline figure (the
|
||||
bytes route, caption/alt settling to the description)."""
|
||||
login(page, app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
page.fill("#message-input", QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
# The scripted read's echoed result — the read header and the FULL
|
||||
# extracted text (page markers and all).
|
||||
expect(bubble).to_contain_text("Here's what the read returned:", timeout=30_000)
|
||||
expect(bubble).to_contain_text(f"Document {SOURCE_NAME}/{DOC_PATH}:")
|
||||
expect(bubble).to_contain_text("---PAGE-1---")
|
||||
expect(bubble).to_contain_text(PAGE_ONE)
|
||||
expect(bubble).to_contain_text("---PAGE-2---")
|
||||
expect(bubble).to_contain_text(PAGE_TWO)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
expect(page.locator("#send-label")).to_have_text("Send")
|
||||
|
||||
# The citation chips: the READ PDF first (read order), then the
|
||||
# suggested image doc's auto-cite (phase 124).
|
||||
chip = page.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(2)
|
||||
expect(chip.nth(0)).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
|
||||
expect(chip.nth(1)).to_have_text(f"{SOURCE_NAME}/{IMAGE_REL}")
|
||||
|
||||
# The inline figure on the image chip: image + caption, both
|
||||
# settling to the description (the content fetch resolves the
|
||||
# summary after the title).
|
||||
fig = page.locator(".msg.brain .source-image")
|
||||
expect(fig).to_have_count(1)
|
||||
img = fig.locator(".source-image-img")
|
||||
expect(img).to_be_visible(timeout=15_000)
|
||||
expect(img).to_have_attribute("src", seeded["image_url"])
|
||||
expect(img).to_have_attribute("alt", seeded["description"], timeout=15_000)
|
||||
caption = fig.locator(".source-image-caption")
|
||||
expect(caption).to_have_text(seeded["description"], timeout=15_000)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5. Default app (images off): the same upload + sync indexes the PDF
|
||||
# text normally and skips the embedded image
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_images_off_upload_indexes_the_pdf_and_skips_the_image(
|
||||
page: Page, default_app_url: str, seeded_off: dict[str, str]
|
||||
) -> None:
|
||||
"""The toggle gate end-to-end (LOCKED A3): with ``BOR_IMAGES``
|
||||
false the PDF's text indexes normally (1 added — the PDF only; the
|
||||
embedded image is skipped, not a failure) while NO image document
|
||||
exists (no row, no copy). The Sources page reads 1 doc / 3 chunks
|
||||
and the drill-down table lists the PDF alone."""
|
||||
login(page, default_app_url) # lands on /sources.html (admin)
|
||||
expect(page.locator("#stat-docs")).to_have_text("1")
|
||||
expect(page.locator("#stat-chunks")).to_have_text("3")
|
||||
|
||||
_drill_to_source(page, SOURCE_NAME)
|
||||
row = page.locator("#docs-tbody tr", has_text=DOC_PATH)
|
||||
expect(row).to_have_count(1)
|
||||
expect(row.locator(".kb-doc-thumb")).to_have_count(0)
|
||||
expect(page.locator("#docs-tbody tr", has_text=IMAGE_REL)).to_have_count(0)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6. Chat (images off): the answer grounds on the PDF; NO image chip
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_images_off_chat_grounds_with_no_image_chip(
|
||||
page: Page, default_app_url: str, seeded_off: dict[str, str]
|
||||
) -> None:
|
||||
"""The SAME question on the off app: the turn grounds on the PDF's
|
||||
extracted text (the read echo carries the page markers + texts —
|
||||
the PDF text is indexed normally, LOCKED A3's off-contract) and the
|
||||
sources block carries EXACTLY the PDF's chip — no image chip, no
|
||||
figure (the embedded image was never indexed)."""
|
||||
try:
|
||||
login(page, default_app_url, next="/")
|
||||
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||
|
||||
page.fill("#message-input", QUESTION)
|
||||
page.click("#send-btn")
|
||||
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
expect(bubble).to_contain_text("Here's what the read returned:", timeout=30_000)
|
||||
expect(bubble).to_contain_text(f"Document {SOURCE_NAME}/{DOC_PATH}:")
|
||||
expect(bubble).to_contain_text(PAGE_ONE)
|
||||
expect(bubble).to_contain_text(PAGE_TWO)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||
|
||||
chip = page.locator(".msg.brain .source-chip")
|
||||
expect(chip).to_have_count(1)
|
||||
expect(chip.nth(0)).to_have_text(f"{SOURCE_NAME}/{DOC_PATH}")
|
||||
# No image affordance at all (no chip, no figure).
|
||||
expect(page.locator(".msg.brain .source-image")).to_have_count(0)
|
||||
finally:
|
||||
_truncate_all()
|
||||
@@ -33,7 +33,10 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf as fitz # PyMuPDF — the temp-PDF fixture builder (phase 126)
|
||||
from sqlalchemy import select
|
||||
|
||||
from app.config import Settings
|
||||
@@ -444,3 +447,469 @@ def test_markdown_referencing_existing_image_keeps_null_record(db, tmp_path) ->
|
||||
assert md3 is not None and md3.image_context is None
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 126 — PDF indexing through the REAL import pipeline (the real
|
||||
# ``app.rag.pdf`` extractor, the page-aware chunker, the REAL
|
||||
# ``_describe_or_skip`` → ``describe_image`` seam — NO seam patch).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
PDF_TITLE = "Quarterly Homelab Report"
|
||||
|
||||
PAGE_ONE = "The core switch firmware was upgraded to the stable channel."
|
||||
PAGE_TWO = "The backup schedule runs nightly at 02:00 and keeps seven rotations."
|
||||
|
||||
|
||||
def _png_bytes(width: int, height: int) -> bytes:
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height), 0)
|
||||
pix.clear_with(200)
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
def _build_pdf(
|
||||
path, pages: list[tuple[str, list[bytes]]], title: str | None = None
|
||||
) -> None:
|
||||
"""One temp PDF (built with PyMuPDF — no binary fixtures): *pages* is
|
||||
(page text, embedded-image bytes) per page; *title* sets the metadata
|
||||
title (``None`` = leave it empty)."""
|
||||
doc = fitz.open()
|
||||
for text, imgs in pages:
|
||||
page = doc.new_page()
|
||||
if text:
|
||||
page.insert_text((72, 72), text)
|
||||
y = 150
|
||||
for data in imgs:
|
||||
pix = fitz.Pixmap(data)
|
||||
page.insert_image(
|
||||
fitz.Rect(72, y, 72 + pix.width, y + pix.height), stream=data
|
||||
)
|
||||
y += pix.height + 10
|
||||
if title is not None:
|
||||
doc.set_metadata({"title": title})
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def test_pdf_becomes_document_with_text_chunks_embedding_and_summary(
|
||||
db, tmp_path
|
||||
) -> None:
|
||||
"""The phase-126 primary scenario: a PDF in a source (the ``pdf``
|
||||
import token) → sync → the document carries the extracted text
|
||||
(the ``---PAGE-N---`` markers, A2 — the title is the metadata
|
||||
title), page-anchored chunks with 768-dim embeddings, and the
|
||||
phase-30/118 lite summary. With ``images=True`` (A3) the embedded
|
||||
image joins the pipeline through the REAL seam — one vision call on
|
||||
the CHAT model with the bare locked prompt (no markdown reference
|
||||
exists — ``image_context`` stays NULL), the copy in ``image_dir``,
|
||||
and the description as the image doc's content AND summary."""
|
||||
source_root = tmp_path / "pdfkb"
|
||||
source_root.mkdir()
|
||||
png = _png_bytes(40, 30)
|
||||
_build_pdf(
|
||||
source_root / "report.pdf",
|
||||
[(PAGE_ONE, []), (PAGE_TWO, [png])],
|
||||
title=PDF_TITLE,
|
||||
)
|
||||
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 2, 0)
|
||||
assert summary.formats == {"pdf": 1}
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "report.pdf"
|
||||
)
|
||||
)
|
||||
assert doc is not None and doc.is_image is False
|
||||
assert doc.title == PDF_TITLE
|
||||
assert doc.content == f"---PAGE-1---\n{PAGE_ONE}\n\n---PAGE-2---\n{PAGE_TWO}"
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [
|
||||
f"Page 1:\n{PAGE_ONE}",
|
||||
f"Page 2:\n{PAGE_TWO}",
|
||||
]
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||
assert doc.summary is not None
|
||||
assert doc.summary.endswith(f"Source: {source_root.name}/report.pdf")
|
||||
|
||||
# The embedded image (A3) — through the REAL describe seam.
|
||||
image_rel = "report.pdf#page=2#img=0.png"
|
||||
img = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == image_rel
|
||||
)
|
||||
)
|
||||
assert img is not None and img.is_image is True
|
||||
assert img.image_context is None, "no markdown reference → bare prompt"
|
||||
assert img.content == DESCRIPTION and img.summary == DESCRIPTION
|
||||
copy = Path(img.image_path)
|
||||
assert copy.parent == Path(llm.settings.image_dir).expanduser()
|
||||
assert copy.name == f"{img.id}.png"
|
||||
assert copy.read_bytes() == png
|
||||
# The vision call went to the CHAT model with the BARE locked
|
||||
# prompt (the PDF's lite summary is the other chat call).
|
||||
assert llm.chat_models == [
|
||||
llm.settings.llm_summary_model, llm.settings.llm_chat_model
|
||||
]
|
||||
assert llm.text_parts == [DESCRIBE_PROMPT]
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_embedded_images_skipped_when_images_off(db, tmp_path) -> None:
|
||||
"""With ``images=False`` (the DEFAULT): the PDF text is indexed
|
||||
normally and the embedded images are skipped — no image documents,
|
||||
no ``images_failed`` (a skip is not a failure), no copies, no vision
|
||||
calls."""
|
||||
source_root = tmp_path / "pdfoffkb"
|
||||
source_root.mkdir()
|
||||
_build_pdf(
|
||||
source_root / "report.pdf",
|
||||
[(PAGE_ONE, [_png_bytes(40, 30)]), (PAGE_TWO, [])],
|
||||
)
|
||||
llm = _image_llm(tmp_path, images=False, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "report.pdf"
|
||||
)
|
||||
)
|
||||
assert doc is not None and PAGE_ONE in doc.content
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert any(c.embedding is not None for c in chunks)
|
||||
assert (
|
||||
db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name,
|
||||
Document.is_image.is_(True),
|
||||
)
|
||||
)
|
||||
is None
|
||||
), "images off: no embedded-image documents"
|
||||
# No vision call at all — only the PDF's lite summary.
|
||||
assert llm.chat_models == [llm.settings.llm_summary_model]
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_without_embedded_images_and_unchanged_rerun(db, tmp_path) -> None:
|
||||
"""A PDF without embedded images → no image documents; an unchanged
|
||||
re-sync → ``unchanged`` (no re-extract, no re-embed, no vision or
|
||||
embed calls), and a second run never creates image documents out of
|
||||
nothing."""
|
||||
source_root = tmp_path / "pdfnonekb"
|
||||
source_root.mkdir()
|
||||
_build_pdf(
|
||||
source_root / "notes.pdf",
|
||||
[(PAGE_ONE, []), (PAGE_TWO, [])],
|
||||
)
|
||||
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
s1 = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (s1.files, s1.added, s1.unchanged) == (1, 1, 0)
|
||||
assert (
|
||||
db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name,
|
||||
Document.is_image.is_(True),
|
||||
)
|
||||
)
|
||||
is None
|
||||
)
|
||||
calls1, chat1 = len(llm.calls), len(llm.chat_models)
|
||||
s2 = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (s2.files, s2.added, s2.updated, s2.unchanged) == (1, 0, 0, 1)
|
||||
assert len(llm.calls) == calls1 and len(llm.chat_models) == chat1
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_without_text_becomes_an_empty_document(db, tmp_path) -> None:
|
||||
"""A PDF with no extractable text (image-only, ``images`` OFF — the
|
||||
embedded image is invisible) → the document is still created, with
|
||||
EMPTY content (the empty byte string's digest), NO content chunks
|
||||
and NO content embedding (there is no text to embed — the only
|
||||
chunk is the best-effort ``lite`` summary's ``is_summary`` row), and
|
||||
the warning line is the signal."""
|
||||
source_root = tmp_path / "pdfemptykb"
|
||||
source_root.mkdir()
|
||||
_build_pdf(source_root / "pics.pdf", [("", [_png_bytes(40, 30)])])
|
||||
llm = _image_llm(tmp_path, images=False, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "pics.pdf"
|
||||
)
|
||||
)
|
||||
assert doc is not None and doc.is_image is False
|
||||
assert doc.content == ""
|
||||
assert doc.content_hash == hashlib.sha256(b"").hexdigest()
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == doc.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c for c in chunks if not c.is_summary] == [], (
|
||||
"no text → no content chunks (and no content embedding)"
|
||||
)
|
||||
# The phase-30/118 path still runs best-effort on the empty
|
||||
# content — the summary is the only chunk, and it is embedded.
|
||||
assert len(chunks) == 1 and chunks[0].is_summary
|
||||
assert chunks[0].embedding is not None and len(chunks[0].embedding) == 768
|
||||
assert doc.summary is not None
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_embedded_image_survives_prune_and_leaves_with_the_pdf(
|
||||
db, tmp_path
|
||||
) -> None:
|
||||
"""Phase 126 (A3) prune parent rule — the defect the E2E surfaced:
|
||||
the embedded image doc's synthetic path NEVER appears in the walk's
|
||||
``seen``, so the plain prune rule destroyed it in the very sync that
|
||||
(re)indexed it (the toggle-on half of the phase-122 "invisible, not
|
||||
deleted" guard was missing). Now: with the parent PDF's rel still in
|
||||
the walk — even UNCHANGED, so no image pass runs at all — the image
|
||||
doc survives (0 pruned); when the PDF LEAVES the source, the image
|
||||
doc is pruned with it (row + ``image_dir`` copy — no orphans)."""
|
||||
source_root = tmp_path / "pdfprune"
|
||||
source_root.mkdir()
|
||||
png = _png_bytes(40, 30)
|
||||
_build_pdf(
|
||||
source_root / "report.pdf",
|
||||
[(PAGE_ONE, []), (PAGE_TWO, [png])],
|
||||
)
|
||||
image_rel = "report.pdf#page=2#img=0.png"
|
||||
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
s1 = asyncio.run(
|
||||
import_sources([source_root], llm, session=db, prune=True)
|
||||
)
|
||||
assert (s1.added, s1.pruned) == (2, 0)
|
||||
img1 = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == image_rel
|
||||
)
|
||||
)
|
||||
assert img1 is not None and img1.image_path is not None
|
||||
copy = Path(img1.image_path)
|
||||
|
||||
# Unchanged re-sync (prune ON): the PDF's text digest matches →
|
||||
# no image pass at all — the image doc survives anyway (the
|
||||
# parent's rel is in the walk's seen set).
|
||||
s2 = asyncio.run(
|
||||
import_sources([source_root], llm, session=db, prune=True)
|
||||
)
|
||||
assert (s2.unchanged, s2.pruned) == (1, 0)
|
||||
assert db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == image_rel
|
||||
)
|
||||
) is not None, "parent PDF present → the image doc survives the prune"
|
||||
|
||||
# The PDF LEAVES the source: the image doc is pruned WITH it
|
||||
# (row + copy — no orphan in image_dir).
|
||||
(source_root / "report.pdf").unlink()
|
||||
s3 = asyncio.run(
|
||||
import_sources([source_root], llm, session=db, prune=True)
|
||||
)
|
||||
assert (s3.pruned, s3.unchanged) == (2, 0)
|
||||
assert db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == image_rel
|
||||
)
|
||||
) is None, "parent PDF gone → the image doc is pruned with it"
|
||||
assert not copy.exists(), "the pruned image's copy leaves image_dir"
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
# --- Phase 125 parity: a markdown file referencing a PDF-embedded image ---
|
||||
|
||||
#: A markdown file referencing the PDF's embedded image by its SYNTHETIC
|
||||
#: path (``<pdf path>#page=<N>#img=<index>.<ext>`` — the doc's ``path``
|
||||
#: is the upsert key, and the reference resolves to it through the shared
|
||||
#: ``_resolve_ref_path`` rule) — one context paragraph before, one after.
|
||||
PDF_GUIDE_MD = (
|
||||
"The VLAN map diagram in the report shows the subnet layout.\n"
|
||||
"\n"
|
||||
"\n"
|
||||
"\n"
|
||||
"Only expand the lab subnet after the VLAN map is reviewed.\n"
|
||||
)
|
||||
|
||||
#: The same reference against ``alpha.pdf`` (the PDF-FIRST ordering's
|
||||
#: fixture — ``alpha.pdf`` sorts before ``zguide.md`` in the walk).
|
||||
PDF_ZGUIDE_MD = PDF_GUIDE_MD.replace("report.pdf", "alpha.pdf")
|
||||
|
||||
|
||||
def _pdf_expected_block(source: str, md_name: str, pdf_name: str) -> str:
|
||||
"""The task-03-shape combined string for the PDF reference (the disk
|
||||
scan builds exactly this — the standalone image's shape, the
|
||||
synthetic path as the image identity)."""
|
||||
return (
|
||||
f"Referenced in {source}/{md_name}:\n"
|
||||
'- Alt text: "VLAN map"\n'
|
||||
" Context: The VLAN map diagram in the report shows the subnet layout.\n"
|
||||
"\n"
|
||||
"Only expand the lab subnet after the VLAN map is reviewed."
|
||||
)
|
||||
|
||||
|
||||
def test_pdf_embedded_image_referenced_by_markdown_markdown_first(
|
||||
db, tmp_path
|
||||
) -> None:
|
||||
"""The MARKDOWN-FIRST ordering (``guide.md`` sorts before
|
||||
``report.pdf``): the markdown is indexed while its referenced image
|
||||
has no document yet → it records the reference on its OWN row (the
|
||||
``IMAGE_CONTEXT:`` record — phase 125, task 04, work 1, never
|
||||
cleared); the PDF is indexed next and the task-03 disk scan (the
|
||||
standalone image's parity path — the markdown is still on disk) finds
|
||||
the reference, so the image doc is enriched AT INDEX TIME: the vision
|
||||
call carries the context after the locked prompt, ``image_context``
|
||||
is set, ``content``/``summary`` are ``<context>\\n\\n<description>``.
|
||||
The post-index pass then skips the already-enriched doc (idempotent).
|
||||
Final state: the record AND the enrichment coexist."""
|
||||
source_root = tmp_path / "pdfmdfirst"
|
||||
source_root.mkdir()
|
||||
(source_root / "guide.md").write_text(PDF_GUIDE_MD, encoding="utf-8")
|
||||
_build_pdf(
|
||||
source_root / "report.pdf",
|
||||
[(PAGE_ONE, []), (PAGE_TWO, [_png_bytes(40, 30)])],
|
||||
)
|
||||
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
# guide.md + report.pdf + the embedded image doc: all added.
|
||||
assert (summary.files, summary.added, summary.images_failed) == (2, 3, 0)
|
||||
|
||||
expected_block = _pdf_expected_block(source_root.name, "guide.md", "report.pdf")
|
||||
|
||||
# Work 1 — the markdown doc STORES the reference (the record —
|
||||
# line 3 is the reference, 2 context paragraphs). It is the
|
||||
# post-index pass's reference index and the retrieval display's
|
||||
# record; the pass does not clear it on success.
|
||||
md = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "guide.md"
|
||||
)
|
||||
)
|
||||
assert md is not None and md.is_image is False
|
||||
assert md.image_context == (
|
||||
"IMAGE_CONTEXT:\n"
|
||||
'- [report.pdf#page=2#img=0.png] (line 3): "VLAN map"'
|
||||
" — 2 paragraphs of context"
|
||||
)
|
||||
|
||||
# The enrichment (the disk scan at PDF index time) — the image
|
||||
# doc's shape is the standalone image's enriched shape.
|
||||
image_rel = "report.pdf#page=2#img=0.png"
|
||||
img = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == image_rel
|
||||
)
|
||||
)
|
||||
assert img is not None and img.is_image is True
|
||||
assert img.image_context == expected_block
|
||||
expected_content = f"{expected_block}\n\n{DESCRIPTION}"
|
||||
assert img.content == expected_content
|
||||
assert img.summary == expected_content
|
||||
# The vision call carried the context AFTER the locked prompt
|
||||
# (the multimodal part list's text part — the describe seam's
|
||||
# phase-125 extension).
|
||||
assert llm.text_parts == [f"{DESCRIBE_PROMPT}{CONTEXT_PREFIX}{expected_block}"]
|
||||
# Chat call order (walk order): guide.md's ``lite`` summary,
|
||||
# the PDF's ``lite`` summary, then the image's CHAT-model
|
||||
# description (the ONLY multimodal call).
|
||||
assert llm.chat_models == [
|
||||
llm.settings.llm_summary_model,
|
||||
llm.settings.llm_summary_model,
|
||||
llm.settings.llm_chat_model,
|
||||
]
|
||||
# The chunks are embedded from the ENRICHED content (the
|
||||
# plain-text packer joins the enriched paragraphs with single
|
||||
# newlines; the summary chunk mirrors the raw content).
|
||||
packed = "\n".join(expected_content.split("\n\n"))
|
||||
chunks = db.scalars(
|
||||
select(Chunk).where(Chunk.document_id == img.id).order_by(Chunk.position)
|
||||
).all()
|
||||
assert [c.content for c in chunks if not c.is_summary] == [packed]
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].content == expected_content
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
|
||||
def test_pdf_embedded_image_referenced_by_markdown_pdf_first(
|
||||
db, tmp_path
|
||||
) -> None:
|
||||
"""The PDF-FIRST ordering (``alpha.pdf`` sorts before ``zguide.md``
|
||||
in the walk): the PDF (and its image doc) is indexed while the
|
||||
markdown is still un-indexed — the task-03 disk scan sees the
|
||||
reference on disk and enriches the image AT INDEX TIME (the ordering
|
||||
that, without the parity scan, would ship an un-enriched image — the
|
||||
markdown's record stays NULL: at markdown index time the image doc
|
||||
ALREADY exists, so the task-03 path owns the case and nothing is
|
||||
recorded). Same final state as the markdown-first test — the
|
||||
enrichment never depends on the walk order."""
|
||||
source_root = tmp_path / "pdfpdffirst"
|
||||
source_root.mkdir()
|
||||
(source_root / "zguide.md").write_text(PDF_ZGUIDE_MD, encoding="utf-8")
|
||||
_build_pdf(
|
||||
source_root / "alpha.pdf",
|
||||
[(PAGE_ONE, []), (PAGE_TWO, [_png_bytes(40, 30)])],
|
||||
)
|
||||
llm = _image_llm(tmp_path, import_extensions="md,txt,pdf")
|
||||
try:
|
||||
summary = asyncio.run(import_sources([source_root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (2, 3, 0)
|
||||
|
||||
expected_block = _pdf_expected_block(source_root.name, "zguide.md", "alpha.pdf")
|
||||
|
||||
# The task-03 path owns this case: the image doc existed when
|
||||
# the markdown was indexed → NO record (the field is the
|
||||
# pending-enrichment index, not a reference dump).
|
||||
md = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == "zguide.md"
|
||||
)
|
||||
)
|
||||
assert md is not None and md.image_context is None
|
||||
|
||||
# The image doc is enriched (the disk scan at PDF index time) —
|
||||
# byte-identical in shape to the markdown-first final state.
|
||||
image_rel = "alpha.pdf#page=2#img=0.png"
|
||||
img = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == source_root.name, Document.path == image_rel
|
||||
)
|
||||
)
|
||||
assert img is not None and img.image_context == expected_block
|
||||
expected_content = f"{expected_block}\n\n{DESCRIPTION}"
|
||||
assert img.content == expected_content
|
||||
assert img.summary == expected_content
|
||||
assert llm.text_parts == [f"{DESCRIBE_PROMPT}{CONTEXT_PREFIX}{expected_block}"]
|
||||
# Chat call order (walk order): the PDF's ``lite`` summary, the
|
||||
# image's CHAT-model description, then zguide.md's summary.
|
||||
assert llm.chat_models == [
|
||||
llm.settings.llm_summary_model,
|
||||
llm.settings.llm_chat_model,
|
||||
llm.settings.llm_summary_model,
|
||||
]
|
||||
finally:
|
||||
_cleanup_source(db, source_root.name)
|
||||
|
||||
@@ -419,6 +419,93 @@ def test_dispatch_still_falls_back_for_unknown_suffix() -> None:
|
||||
assert chunk_document(content, "x/notes.whatever") == chunk_text(content)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pdf (phase 126, task 03, LOCKED A2) — the page-aware chunker
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pdf_single_page_chunks_with_page_anchor() -> None:
|
||||
content = "---PAGE-1---\nHello single page.\n"
|
||||
chunks = chunk_document(content, "report.pdf")
|
||||
assert chunks == ["Page 1:\nHello single page."]
|
||||
assert "---PAGE-" not in "\n".join(chunks)
|
||||
|
||||
|
||||
def test_pdf_multi_page_keeps_the_marker_page_number() -> None:
|
||||
"""Blank pages carry no marker — the anchor follows the MARKER's
|
||||
number, not the chunk count (page 3 stays page 3 when page 2 was
|
||||
blank)."""
|
||||
content = (
|
||||
"---PAGE-1---\nAlpha text one.\n\n"
|
||||
"---PAGE-3---\nGamma text three.\n"
|
||||
)
|
||||
chunks = chunk_document(content, "report.pdf")
|
||||
assert chunks == ["Page 1:\nAlpha text one.", "Page 3:\nGamma text three."]
|
||||
assert "Page 2:" not in "\n".join(chunks)
|
||||
|
||||
|
||||
def test_pdf_markers_never_appear_in_chunks() -> None:
|
||||
content = (
|
||||
"---PAGE-1---\nAlpha text one.\n\n"
|
||||
"---PAGE-2---\nBeta text two.\n\n"
|
||||
"---PAGE-3---\nGamma text three.\n"
|
||||
)
|
||||
chunks = chunk_document(content, "deep/dir/report.pdf")
|
||||
assert len(chunks) == 3
|
||||
for chunk in chunks:
|
||||
assert "---PAGE-" not in chunk
|
||||
assert chunks[0].startswith("Page 1:")
|
||||
assert chunks[1].startswith("Page 2:")
|
||||
assert chunks[2].startswith("Page 3:")
|
||||
|
||||
|
||||
def test_pdf_empty_and_markerless_content_yield_no_chunks() -> None:
|
||||
assert chunk_document("", "report.pdf") == []
|
||||
# No marker at all (the parser always emits one for text — belt and
|
||||
# braces): nothing to anchor, no chunks (never plain-text fallback —
|
||||
# a PDF chunk must carry its page).
|
||||
assert chunk_document("just words, no marker", "report.pdf") == []
|
||||
assert chunk_document("---PAGE-1---\n\n---PAGE-2---\n ", "report.pdf") == []
|
||||
|
||||
|
||||
def test_pdf_long_page_sub_splits_and_every_chunk_keeps_the_anchor() -> None:
|
||||
"""A page larger than the target sub-splits through the paragraph
|
||||
packer — EVERY chunk of the page keeps the ``Page N:`` anchor and
|
||||
the anchored chunk never exceeds the target."""
|
||||
page = "\n\n".join(f"paragraph {i} " + "l" * 300 for i in range(6))
|
||||
content = f"---PAGE-1---\n{page}\n\n---PAGE-2---\nshort second page\n"
|
||||
chunks = chunk_document(content, "report.pdf", target_chars=800, overlap_chars=100)
|
||||
assert len(chunks) >= 3 # the long page sub-splits + the short page
|
||||
page1 = [c for c in chunks if c.startswith("Page 1:")]
|
||||
page2 = [c for c in chunks if c.startswith("Page 2:")]
|
||||
assert len(page1) >= 2 and page2 == ["Page 2:\nshort second page"]
|
||||
for chunk in chunks:
|
||||
assert len(chunk) <= 800, f"target exceeded: {len(chunk)}"
|
||||
# All six paragraphs survive the split.
|
||||
joined = "\n".join(chunks)
|
||||
for i in range(6):
|
||||
assert f"paragraph {i}" in joined
|
||||
|
||||
|
||||
def test_pdf_overlap_carryover_stays_within_the_page() -> None:
|
||||
"""The tail carry-over packs paragraphs INSIDE one page; a new page
|
||||
never inherits the previous page's tail (the page is the boundary)."""
|
||||
a = "\n\n".join(f"para-a-{i} " + "l" * 250 for i in range(3))
|
||||
b = "\n\n".join(f"para-b-{i} " + "l" * 250 for i in range(3))
|
||||
content = f"---PAGE-1---\n{a}\n\n---PAGE-2---\n{b}\n"
|
||||
chunks = chunk_document(content, "r.pdf", target_chars=700, overlap_chars=120)
|
||||
first_of_page2 = next(c for c in chunks if c.startswith("Page 2:"))
|
||||
assert "para-a-" not in first_of_page2, "page 2 must not inherit page 1's tail"
|
||||
assert all(len(c) <= 700 for c in chunks)
|
||||
|
||||
|
||||
def test_pdf_dispatch_matches_direct_call() -> None:
|
||||
content = "---PAGE-1---\nalpha\n\n---PAGE-2---\nbeta\n"
|
||||
from app.rag.chunker import _chunk_pdf
|
||||
|
||||
assert chunk_document(content, "x/report.PDF") == _chunk_pdf(content)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hard cap across every format (aipi ~1024-token request cap)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -434,6 +521,8 @@ def test_dispatch_still_falls_back_for_unknown_suffix() -> None:
|
||||
("line of text\n\n" * 800, "big.txt"),
|
||||
("[Unit]\n" + "key=" + "v" * 5000 + "\n", "big.container"), # phase 47
|
||||
("{% for x in y %}" + "{{ x }} " * 400 + "{% endfor %}", "big.j2"), # phase 47
|
||||
# phase 126: one long page (a single 12 500-char line) + one short.
|
||||
("---PAGE-1---\n" + "word " * 2500 + "\n\n---PAGE-2---\nshort\n", "big.pdf"),
|
||||
],
|
||||
)
|
||||
def test_hard_cap_holds_for_every_format(content: str, path: str) -> None:
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user