Files
brain-of-reese/tests/unit/test_pictorial_import.py
T
ducoterra 863519e980
Build and Push Containers / build-and-push-app (push) Successful in 2m26s
Build and Push Containers / build-and-push-db (push) Successful in 12s
phase: 147_sync_changed_only
All defaults match. Verification complete — everything is green with no defects found.

**Phase 147 final verification pass — report**

- All 3 task files found in `complete/147_sync_changed_only/` → verification pass only; no code changes needed.
- Verified: migration 0025 (`documents.file_hash` nullable String(64), no index) + model field; fast path at top of `_index_file` (streamed 1 MiB sha256 → row lookup → `_unchanged_doc`); stamping on all three branches' create/update + shared helper; pre-phase NULL-hash fallthrough; retriever/`content_hash` untouched.
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3438 passed, `app/` coverage **99%** (>90% ✓); `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` → 3 passed in isolation ✓; `uv run ruff check . && uv run pyright` → all checks passed, 0 errors ✓; prior-phase `tests/e2e/test_sync_button.py` → 3 passed (no regression) ✓.
- **Completion criteria:** all 7 pass — (1) column + round-trip ✓ (2) unchanged text/image/garbage-PDF/pictorial-PDF cost zero LLM/extraction/vision, `summary.unchanged` counted, rows byte-still ✓ (3) changed file re-indexes + re-stamps; bytes-changed/content-same stamps only ✓ (4) backfill/D1/D4 + log wordings pinned verbatim ✓ (5) `content_hash`/retriever byte-identical ✓ (6) all gates green ✓ (7) no prior-phase behavior change ✓.
- **Documentation:** full review of README.md (1,031 lines) and AGENTS.md against the code: every `.env.example`/code `BOR_*` var documented, all config defaults match `app/config.py`, quick-start/QA commands and referenced test files exist, `file://` URL + troubleshooting claims accurate; the phase's README "Updating the documents" edit is accurate. **Verified consistent — no changes needed.**
- **Notable:** no live-infrastructure changes (dev Postgres container used for the standard test gates only). Working tree left uncommitted per harness protocol.
- **Next pending phase:** `148_pdf_pages_in_sources`.
2026-10-01 14:23:33 -04:00

614 lines
26 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Phase 140, task 03 — the importer's pictorial pass (unit).
The pass itself (``app.rag.importer._pictorial_pdf_pass`` — the
IKEA assembly-instruction class, TODO L5): a PDF whose text layer
PASSES the phase-133 garbage gate but is SPARSE (mostly pictorial
pages) gets every candidate page (``pictorial_pages`` — the
deterministic gate, LOCKED A1) rendered and vision-described, and the
description appends to the page's content block as
``[Page N illustration: …]`` (LOCKED A2/A4 — no LLM judge; the digest
runs over the FINAL content).
Pinned here (the synthetic IKEA-class recipe — PyMuPDF, no binary
fixtures; the real tar is pinned in
``tests/integration/test_importer.py``):
* the happy path — every candidate page (computed IN the test from
``extract_pdf_page_texts`` + ``pictorial_pages`` — no hardcoded page
list) carries its illustration block, the text-heavy pages do NOT,
the chunks carry the canned description, the digest is over the
FINAL content, the counters + the ONE loud log line fire (PLAN §9).
* the assembly BYTE-IDENTITY pin — with NO descriptions (every vision
call fails) the stored content is ``extract_pdf_text``'s output
byte-for-byte (the page's text is never lost, never reworded).
* the branch matrix: the kill switch off (``pdf_pictorial=False`` →
byte-identical to pre-phase, zero vision calls), zero candidates
(a clean-text PDF — no render, no vision call, no log line), the
fail-soft ``None`` page (one dead page keeps its text only, the
others are described), the unreadable-page placeholder skip (the
zero-byte ``render_pdf_pages`` slot burns no vision call, the page
keeps its text only).
* the phase-147 rerun — an unchanged file re-syncs through the FAST
PATH (the raw bytes are the change signal): ``unchanged``, no
re-embed, byte-stable content + digest — and the vision calls are
NOT re-issued (the pass never runs on an unchanged file; the
pre-phase "vision re-issued before the unchanged check" no-cache
cost is gone).
The E2E surface (the real UI + the mock LLM's ``PICTORIAL_PAGE_MODE``
branch) is pinned in ``tests/e2e/test_pictorial_pdf.py`` (task 04).
"""
from __future__ import annotations
import asyncio
import base64
import hashlib
import logging
import textwrap
from pathlib import Path
from typing import Any
import pymupdf as fitz
import pytest
from sqlalchemy import select
from app.config import Settings
from app.models import Chunk, Document
from app.rag import importer
from app.rag.importer import import_sources
from app.rag.llm import LLMError
from app.rag.pdf import (
PAGE_SEPARATOR,
extract_pdf_page_texts,
extract_pdf_text,
pictorial_pages,
)
from app.rag.summarizer import PICTORIAL_PAGE_MODE, PICTORIAL_PROMPT
from tests.fakes import FakeEmbedder
#: The IKEA fixture's metadata title (the phase-140 fixture, TODO L5 —
#: Adobe InDesign's "Assembly Instructions").
TITLE = "Assembly Instructions"
#: The text-heavy pages (IKEA pages 2–5 — the multilingual warnings,
#: 3,091–7,288 chars in the real fixture): a fixed warning block well
#: over the 100-char sparseness bar, so it is NEVER a candidate.
DENSE_TEXT = (
"WARNING: Adult assembly required. Read all instructions before "
"starting. Keep this package away from children. The furniture "
"must be secured to the wall using the supplied anti-tip device."
)
assert len(DENSE_TEXT.strip()) >= 100 # the class property, not a typo
def _insert_dense(page) -> None:
"""One dense-warning page's text, inserted in 60-char wrapped
lines: a single 188-char line would be CLIPPED at the page edge
on extraction (verified — only ~104 chars survive), while the
multi-line read round-trips byte-exactly (the fixture's class
property — over the 100-char bar — must hold on the EXTRACTED
text, which the gate sees)."""
for i, line in enumerate(textwrap.wrap(DENSE_TEXT, 60)):
page.insert_text((72, 72 + i * 14), line)
def _build_ikea_like_pdf(path: Path) -> None:
"""A synthetic IKEA-class PDF (TODO L5 — the vector-drawn
assembly-instruction class): page 1 is the title page (``BESTA``
— 5 chars, as in the real fixture's ``BESTÅ``), pages 2–5 the
text-heavy warnings (≥100 chars — never candidates), pages 6–10
sparse ``Illustration N`` markers (15–16 chars — candidates)
standing in for the vector pictograms (no embedded raster images
— the phase-122 image pipeline sees nothing, as on the real
fixture). The metadata title is the fixture's."""
doc = fitz.open()
doc.new_page().insert_text((72, 72), "BESTA")
for _ in range(4):
_insert_dense(doc.new_page())
for n in range(6, 11):
doc.new_page().insert_text((72, 72), f"Illustration {n}")
doc.set_metadata({"title": TITLE})
doc.save(path)
doc.close()
def _build_dense_pdf(path: Path) -> None:
"""A clean-text PDF: every page over the 100-char bar — ZERO
pictorial candidates (the pass's no-op branch)."""
doc = fitz.open()
for _ in range(2):
_insert_dense(doc.new_page())
doc.set_metadata({"title": "Clean Reference"})
doc.save(path)
doc.close()
class _PictorialLLM(FakeEmbedder):
"""The capturing fake LLM (task 03): the MULTIMODAL describe call
(``user`` content = a part list) gets a canned reply DETERMINED BY
THE RENDERED PAGE'S BYTES (a short sha256 of the data-URL's
base64 — the same rendered page always gets the same reply,
ACROSS RUNS: the digest-stability premise holds), and the TEXT
part is recorded (``pictorial_texts`` — the ``PICTORIAL_PROMPT``
head + the ``Page N of the PDF`` block are the per-page context
witness). String (``lite``) calls keep the ``FakeEmbedder``
behaviour. The REAL ``import_sources`` → ``_pictorial_pdf_pass``
→ ``describe_pdf_page`` chain runs against it (no seam patch)."""
def __init__(self) -> None:
super().__init__()
self.pictorial_texts: list[str] = []
self.pictorial_calls = 0
@staticmethod
def reply_for(png: bytes) -> str:
"""The canned reply a vision call for *png* gets —
deterministic in the bytes (the tests build the expected
content with the same helper)."""
return (
f"Page {hashlib.sha256(png).hexdigest()[:8]} illustrated: "
"two parts are joined with a blue arrow."
)
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
text = next((p["text"] for p in user if p.get("type") == "text"), "")
assert PICTORIAL_PAGE_MODE in text, (
"a multimodal call reached the fake that is not the "
f"pictorial pass's seam: {text[:80]!r}"
)
self.pictorial_calls += 1
self.pictorial_texts.append(text)
b64 = next(p for p in user if p.get("type") == "image_url")[
"image_url"
]["url"].rsplit(",", 1)[1]
return self.reply_for(base64.b64decode(b64))
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
class _FailPagePictorialLLM(_PictorialLLM):
"""Rejects the vision call for page *fail_page* (parsed from the
text part's ``Page N of the PDF.`` block) and describes the rest —
drives the per-page fail-soft (the dead page keeps its text only,
the other candidates still get their descriptions)."""
def __init__(self, fail_page: int) -> None:
super().__init__()
self.fail_page = fail_page
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
text = next((p["text"] for p in user if p.get("type") == "text"), "")
if f"Page {self.fail_page} of the PDF." in text:
self.pictorial_calls += 1
self.pictorial_texts.append(text)
raise LLMError(
f"simulated dead vision call for page {self.fail_page} "
"(test sentinel)"
)
return await super().chat(messages, model=model)
class _DeadVisionLLM(FakeEmbedder):
"""A vision client that REJECTS every multimodal call (the dead
vision endpoint — the fail-soft's total-failure case: EVERY
candidate page keeps its text only, and the stored content is the
extraction byte-identical)."""
def __init__(self) -> None:
super().__init__()
self.pictorial_calls = 0
async def chat(
self, messages: list[dict[str, Any]], model: str | None = None
) -> str:
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
self.pictorial_calls += 1
raise LLMError("simulated dead vision endpoint (test sentinel)")
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def _settings(tmp_path: Path, **kwargs: Any) -> Settings:
"""Settings admitting the ``pdf`` import token with the phase-140
knobs at their CODE DEFAULTS (``pdf_pictorial`` ON — LOCKED A3 —
and ``pdf_pictorial_min_chars`` 100 — LOCKED A1) unless pinned
otherwise; ``images`` off (the embedded-image pass is not in play
— the fixture class is vector-drawn)."""
kwargs.setdefault("_env_file", None)
kwargs.setdefault("import_extensions", "md,txt,pdf")
kwargs.setdefault("images", False)
kwargs.setdefault("image_dir", str(tmp_path / "images"))
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
def _pdf_doc(db, source: str, rel: str) -> Document:
doc = db.scalar(
select(Document).where(Document.source == source, Document.path == rel)
)
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
return doc
def _cleanup_source(db, source: str) -> None:
for doc in db.scalars(select(Document).where(Document.source == source)).all():
db.delete(doc)
db.commit()
def _candidates_and_pngs(pdf_path: Path) -> tuple[list[int], list[bytes]]:
"""The pass's own inputs, computed IN the test (no hardcoded page
list — the fixture is stable but the test must not rot): the
deterministic gate over the per-page text map, and the candidate
render (aligned with the candidates — ascending, one slot each)."""
page_texts = extract_pdf_page_texts(pdf_path)
candidates = pictorial_pages(page_texts, min_chars=100)
pngs = importer.render_pdf_pages(pdf_path, pages=candidates)
return candidates, pngs
def _expected_content(
pdf_path: Path, fail_pages: frozenset[int] = frozenset()
) -> tuple[str, list[int]]:
"""The FINAL content the pass produces for *pdf_path* under the
stable fake — derived from the same inputs the pass uses: every
page of the extraction map as its ``PAGE_SEPARATOR`` + the page
text, the described candidates (the candidates MINUS
*fail_pages*) carrying ``[Page N illustration: <stable reply>]``
AFTER their text (LOCKED A4 — the text stays)."""
page_texts = extract_pdf_page_texts(pdf_path)
candidates, pngs = _candidates_and_pngs(pdf_path)
parts: list[str] = []
for page, text in page_texts.items():
block = text
if page in candidates and page not in fail_pages:
png = pngs[candidates.index(page)]
block += f"\n\n[Page {page} illustration: {_PictorialLLM.reply_for(png)}]"
parts.append(f"{PAGE_SEPARATOR.format(page=page)}\n{block}")
return "\n\n".join(parts), candidates
def test_sparse_pages_get_their_illustration_blocks(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""The happy path end to end (the owner's L5 rationale): every
candidate page (computed from the gate — the fixture's title page
+ its six illustration pages) carries its illustration block, the
text-heavy pages (2–5) do NOT, the chunks carry the canned
description (the embeddings hit the illustrations), the title is
the metadata title, the digest is over the FINAL content, and the
counters + the ONE loud log line fire (PLAN §9)."""
root = tmp_path / "pik"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
content, candidates = _expected_content(pdf)
assert len(candidates) == 6 # the title page + the six markers
assert set(candidates) == {1, 6, 7, 8, 9, 10}
doc = _pdf_doc(db, "pik", "ikea.pdf")
assert doc.title == TITLE # the metadata title (the fixture's)
assert doc.is_image is False
# The assembly is byte-exact: the page text STAYS, the
# description appends after it (LOCKED A4).
assert doc.content == content
# The text-heavy pages (2–5 — the multilingual warnings) carry
# NO illustration block (the gate is the ONLY selector).
for n in (2, 3, 4, 5):
assert f"[Page {n} illustration" not in doc.content
# The digest runs over the FINAL content (the phase-126/133
# convention — A4).
assert doc.content_hash == hashlib.sha256(
doc.content.encode("utf-8")
).hexdigest()
# The counters (log-only — the sync detail shape is untouched):
# one PDF with six described sparse pages.
assert summary.pdfs_pictorial == 1
assert summary.pdf_pictorial_pages == 6
assert summary.pdfs_screenshotted == 0 # the gate passed (usable)
# The chunks carry the canned description — the joined chunk
# text hits every candidate's reply (each page is its own
# ``Page N:`` chunk, the page-aware chunker).
chunks = db.scalars(
select(Chunk).where(Chunk.document_id == doc.id)
.order_by(Chunk.position)
).all()
content_chunks = [c for c in chunks if not c.is_summary]
joined = "\n".join(c.content for c in content_chunks)
_, pngs = _candidates_and_pngs(pdf)
for page in candidates:
reply = _PictorialLLM.reply_for(pngs[candidates.index(page)])
assert reply in joined, f"page {page}'s description is not indexed"
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert doc.summary is not None
# ONE vision call per candidate, in candidate order — the text
# part heads with the locked prompt and carries the page's OWN
# text as context (the page block, task 02's contract).
assert llm.pictorial_calls == len(candidates)
assert len(llm.pictorial_texts) == len(candidates)
page_texts = extract_pdf_page_texts(pdf)
for page, text_part in zip(candidates, llm.pictorial_texts, strict=True):
assert text_part.startswith(PICTORIAL_PROMPT)
assert f"Page {page} of the PDF." in text_part
assert page_texts[page] in text_part # the page's own text
# The ONE loud log line (PLAN §9) — exact, with the computed N.
lines = [
r.getMessage()
for r in caplog.records
if "pictorial pass" in r.getMessage()
]
assert lines == [
"pdf: pictorial pass source=pik path=ikea.pdf — "
"6 candidate pages, 6 described (min_chars=100)"
]
finally:
_cleanup_source(db, "pik")
def test_assembly_byte_identity_without_descriptions(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""THE byte-identity pin: with NO descriptions (the dead vision
endpoint — every candidate's call fails) the stored content is
``extract_pdf_text``'s output BYTE-FOR-BYTE (the page's text is
never lost, never reworded, the separators are untouched), the
pass still counted its candidates in the log line (0 described),
and no PDF is counted pictorial (a description is what counts)."""
root = tmp_path / "dead"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _DeadVisionLLM()
llm.settings = _settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert llm.pictorial_calls == 6 # every candidate was attempted
doc = _pdf_doc(db, "dead", "ikea.pdf")
assert doc.content == extract_pdf_text(pdf), (
"no descriptions → the stored content is the extraction "
"byte-identical (the assembly pin)"
)
assert summary.pdfs_pictorial == 0 # nothing was described
assert summary.pdf_pictorial_pages == 0
lines = [
r.getMessage()
for r in caplog.records
if "pictorial pass" in r.getMessage()
]
assert lines == [
"pdf: pictorial pass source=dead path=ikea.pdf — "
"6 candidate pages, 0 described (min_chars=100)"
]
# The dead pages' warnings (the fail-soft's signal, per page).
assert [
r
for r in caplog.records
if r.levelno == logging.WARNING
and "has no description" in r.getMessage()
]
finally:
_cleanup_source(db, "dead")
def test_feature_off_is_byte_identical_to_pre_phase(db, tmp_path: Path) -> None:
"""The kill switch (LOCKED A3, ``BOR_PDF_PICTORIAL=0``): the pass
never runs — no render, no vision call, no log line, no counters —
and the stored content is ``extract_pdf_text``'s output
byte-identical to the pre-phase-140 path."""
root = tmp_path / "off"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path, pdf_pictorial=False)
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
doc = _pdf_doc(db, "off", "ikea.pdf")
assert doc.content == extract_pdf_text(pdf)
assert llm.pictorial_calls == 0
assert (summary.pdfs_pictorial, summary.pdf_pictorial_pages) == (0, 0)
finally:
_cleanup_source(db, "off")
def test_zero_candidates_is_a_noop(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""A clean-text PDF (every page over the 100-char bar) has ZERO
candidates — the pass is a no-op: no render, NO vision call
(the pin — a clean-text PDF never burns a vision call), no log
line, no counters, and the content is the extraction untouched."""
root = tmp_path / "dense"
root.mkdir()
pdf = root / "clean.pdf"
_build_dense_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
doc = _pdf_doc(db, "dense", "clean.pdf")
assert doc.content == extract_pdf_text(pdf)
assert llm.pictorial_calls == 0
assert (summary.pdfs_pictorial, summary.pdf_pictorial_pages) == (0, 0)
assert not [
r for r in caplog.records if "pictorial pass" in r.getMessage()
]
finally:
_cleanup_source(db, "dense")
def test_fail_soft_page_keeps_its_text(
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
) -> None:
"""One dead candidate page (the vision call rejects it) keeps its
text only — a warning is the signal, the OTHER candidates still
get their descriptions, one dead page never kills the pass
(LOCKED A2 fail-soft). The described count is five of six; the
PDF still counts pictorial (≥1 described page)."""
root = tmp_path / "partfail"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _FailPagePictorialLLM(fail_page=7)
llm.settings = _settings(tmp_path)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert llm.pictorial_calls == 6 # every candidate was attempted
content, candidates = _expected_content(pdf, fail_pages=frozenset({7}))
doc = _pdf_doc(db, "partfail", "ikea.pdf")
assert doc.content == content
assert "[Page 7 illustration" not in doc.content # the dead page
assert "---PAGE-7---\nIllustration 7" in doc.content # its text stays
assert summary.pdfs_pictorial == 1
assert summary.pdf_pictorial_pages == 5
lines = [
r.getMessage()
for r in caplog.records
if "pictorial pass" in r.getMessage()
]
assert lines == [
"pdf: pictorial pass source=partfail path=ikea.pdf — "
"6 candidate pages, 5 described (min_chars=100)"
]
# The importer's per-page warning (the dead page's signal).
warnings = [
r
for r in caplog.records
if r.levelno == logging.WARNING
and "pictorial page 7" in r.getMessage()
]
assert len(warnings) == 1
finally:
_cleanup_source(db, "partfail")
def test_unreadable_page_placeholder_skips_the_vision_call(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""A candidate page that could not be RENDERED (the zero-byte
``render_pdf_pages`` placeholder — the phase-133 handling) is
skipped with a warning: it burns NO vision call, keeps its text
only, and the OTHER candidates keep their positions +
descriptions (the result's order never shifts)."""
root = tmp_path / "halfblind"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
real_render = importer.render_pdf_pages
def _render(path: Path, dpi: int = 150, pages: Any = None) -> list[bytes]:
assert pages is not None # only the pass calls it (with a selector)
pngs = real_render(path, dpi=dpi, pages=pages)
return [
b"" if n == 6 else png
for n, png in zip(pages, pngs, strict=True)
]
monkeypatch.setattr(importer, "render_pdf_pages", _render)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
try:
with caplog.at_level(logging.WARNING, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert llm.pictorial_calls == 5 # the blank page burned no call
content, candidates = _expected_content(pdf, fail_pages=frozenset({6}))
doc = _pdf_doc(db, "halfblind", "ikea.pdf")
assert doc.content == content
assert "[Page 6 illustration" not in doc.content
assert "---PAGE-6---\nIllustration 6" in doc.content
assert summary.pdfs_pictorial == 1
assert summary.pdf_pictorial_pages == 5
warnings = [
r
for r in caplog.records
if r.levelno == logging.WARNING
and "skipping pictorial page 6" in r.getMessage()
]
assert len(warnings) == 1
finally:
_cleanup_source(db, "halfblind")
def test_rerun_lands_unchanged_fast_path_no_revision(
db, tmp_path: Path
) -> None:
"""Phase 147 (task 02) — the owner's "really long sync times for
PDFs that are mostly pictoral" complaint, fixed: a re-sync of an
unchanged IKEA-class PDF (same raw bytes) takes the FAST PATH at
the top of ``_index_file`` (the file hash matches) — the
pictorial pass does NOT re-run: no re-render, the vision calls are
NOT re-issued (``pictorial_calls`` stays at run 1's 6 — the
pre-phase "vision re-issued before the unchanged check can skip
them" no-cache cost is gone), ``pdfs_pictorial`` stays 0 on run 2.
The row re-imports as UNCHANGED: no re-chunk, no re-embed, no
re-summary, no KB change (not ``updated``, no changed path), and
the content + digest are byte-stable. (A non-deterministic vision
model no longer flips the row's digest either — the raw bytes
decide.)"""
root = tmp_path / "stable"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
try:
s1 = asyncio.run(import_sources([root], llm, session=db))
assert (s1.added, s1.pdfs_pictorial, s1.pdf_pictorial_pages) == (1, 1, 6)
assert llm.pictorial_calls == 6 # one per candidate — first index
embeds1 = len(llm.calls)
doc1 = _pdf_doc(db, "stable", "ikea.pdf")
content1, digest1 = doc1.content, doc1.content_hash
s2 = asyncio.run(import_sources([root], llm, session=db))
# The fast path fired (file hash match) — the pass never ran
# on run 2: no render, no vision re-issued, no re-embed, no KB
# change.
assert (s2.added, s2.updated, s2.unchanged) == (0, 0, 1)
assert s2.pdfs_pictorial == 0 # the pass did not re-fire
assert llm.pictorial_calls == 6 # NO re-issued vision call
assert len(llm.calls) == embeds1, "unchanged: no re-embed"
assert s2.changed_paths == [], "unchanged: no KB change recorded"
doc2 = _pdf_doc(db, "stable", "ikea.pdf")
assert doc2.content == content1 # byte-stable content
assert doc2.content_hash == digest1 # byte-stable digest
finally:
_cleanup_source(db, "stable")