phase: 163_pdf_strip_additive_selfheal
All gates verified. Final report: **Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)** - Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs). - `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%) - `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation - `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed - `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins) - Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅ - **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**. - Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit. - **Next pending phase:** `164_chat_error_honesty`
This commit is contained in:
@@ -31,6 +31,16 @@ the second sync):
|
||||
* the streamed hash is transparent (a >1 MiB file hashes identically
|
||||
to ``hashlib.sha256(bytes).hexdigest()``).
|
||||
|
||||
Phase 163 (task 03) extends this file with the PDF strip RE-GATE the
|
||||
unchanged paths now run (``_regate_pdf_strip``) — the pre-phase "the
|
||||
copies persist" assumption is what broke in production (a row claiming
|
||||
``page_count = 36`` over ZERO files, every page 404ing forever): the
|
||||
re-gate re-derives the strip from the current bytes + settings (ZERO
|
||||
vision/LLM calls), heals missing copies, re-stamps a changed class,
|
||||
clears a strip that no longer qualifies (kill switch included), stays
|
||||
fail-soft, logs only when it ACTS, and costs a non-strip file zero PDF
|
||||
parsing (spy-pinned).
|
||||
|
||||
The E2E surface (the idempotent second sync through the real UI +
|
||||
mock LLM) is pinned in ``tests/e2e/test_sync_changed_only.py``
|
||||
(task 03).
|
||||
@@ -43,6 +53,7 @@ import logging
|
||||
import os
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pymupdf as fitz
|
||||
import pytest
|
||||
@@ -50,6 +61,7 @@ from sqlalchemy import select
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag import importer
|
||||
from app.rag.importer import _file_hash, import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
from tests.unit.test_importer import (
|
||||
@@ -60,6 +72,7 @@ from tests.unit.test_importer import (
|
||||
_VisionLLM,
|
||||
)
|
||||
from tests.unit.test_pictorial_import import (
|
||||
_build_dense_pdf,
|
||||
_build_ikea_like_pdf,
|
||||
_PictorialLLM,
|
||||
_settings,
|
||||
@@ -454,6 +467,520 @@ def test_pre_phase_null_file_hash_image_runs_branch_and_stamps(
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 163 (task 03): the deterministic PDF strip re-gate ----------
|
||||
#
|
||||
# The live defect (owner report, 2026-10-07): the `Letumdei` row carried
|
||||
# `page_count = 36` over ZERO files on disk (the pre-phase-163 persist
|
||||
# deleted the old copies BEFORE rendering, outside the DB transaction —
|
||||
# a later per-file rollback restored the stale count over an empty
|
||||
# directory; an external wipe of `pdf_page_dir` lands the same state).
|
||||
# The phase-147 fast path believed "the copies persist" and never
|
||||
# re-rendered, so the strip 404'd forever. `_regate_pdf_strip` now runs
|
||||
# at the tail of `_unchanged_doc`: the changed-path gate mirrored from
|
||||
# the current bytes + settings (ZERO vision/LLM calls), missing copies
|
||||
# healed, a stale strip cleared.
|
||||
|
||||
#: The PNG signature (a healed copy is a real render, not a placeholder).
|
||||
PNG_MAGIC = b"\x89PNG\r\n\x1a\n"
|
||||
|
||||
|
||||
def _regate_settings(tmp_path: Path, **kwargs: Any) -> Settings:
|
||||
"""Settings for the re-gate matrix: the ``pdf`` token admitted, the
|
||||
phase-140 knobs at their CODE DEFAULTS (``pdf_pictorial`` ON), and
|
||||
the strip's home ``pdf_page_dir`` pinned under *tmp_path* (never the
|
||||
real home)."""
|
||||
kwargs.setdefault("_env_file", None)
|
||||
kwargs.setdefault("import_extensions", "md,txt,pdf")
|
||||
kwargs.setdefault("images", False)
|
||||
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||
kwargs.setdefault("pdf_page_dir", str(tmp_path / "pages"))
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _pages_dir(tmp_path: Path) -> Path:
|
||||
return tmp_path / "pages"
|
||||
|
||||
|
||||
def _strip_copies(doc: Document, page_dir: Path, count: int) -> list[Path]:
|
||||
return [page_dir / f"{doc.id}-p{n}.png" for n in range(1, count + 1)]
|
||||
|
||||
|
||||
def test_regate_heals_deleted_copies_of_a_pictorial_strip(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""THE live-loss reproduction (heal arm): a pictorial-class PDF is
|
||||
indexed (``page_count`` set, copies written), its copies are deleted
|
||||
out from under the row (the exact live state — mode NULL on a
|
||||
pre-phase row), and an UNCHANGED re-sync restores every file with
|
||||
real PNG bytes, stamps the mode ``additive``, keeps ``page_count``
|
||||
(no drift), burns ZERO llm/vision calls, and logs the one greppable
|
||||
INFO line. The NULL-mode stamp is SILENT (no ``mode`` segment — it
|
||||
rides the line only when work followed)."""
|
||||
root = tmp_path / "rgheal"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgheal", "ikea.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
doc.page_strip_mode = None # the pre-phase row (the live shape)
|
||||
db.commit()
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
assert not list(page_dir.glob(f"{doc.id}-*.png"))
|
||||
embeds1, chats1, vision1 = len(llm.calls), len(llm.chat_calls), llm.pictorial_calls
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
# ZERO llm work of any kind (the re-gate is deterministic, A5).
|
||||
assert len(llm.calls) == embeds1, "the re-gate never re-embeds"
|
||||
assert len(llm.chat_calls) == chats1, "the re-gate never chats"
|
||||
assert llm.pictorial_calls == vision1, "the re-gate issues ZERO vision calls"
|
||||
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgheal", "ikea.pdf")
|
||||
assert doc2.page_count == 10, "no drift — the count is untouched"
|
||||
assert doc2.page_strip_mode == "additive", "the mode is stamped"
|
||||
for path in _strip_copies(doc2, page_dir, 10):
|
||||
assert path.is_file(), f"copy {path.name} was not healed"
|
||||
assert path.read_bytes()[:8] == PNG_MAGIC, f"{path.name} is not a PNG"
|
||||
assert not list(page_dir.glob("*.tmp")), "no .tmp orphan from the heal"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert (
|
||||
"import: PDF strip re-gate source=rgheal path=ikea.pdf — "
|
||||
"healed 10 pages"
|
||||
) in messages, messages
|
||||
assert not any("mode " in m for m in messages if "re-gate" in m), (
|
||||
"a NULL->stamp mode re-stamp is silent-beneficial " + str(messages)
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "rgheal")
|
||||
|
||||
|
||||
def test_regate_heals_a_fallback_strip_as_primary(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Heal arm, the FALLBACK class (phase 133's unusable text layer):
|
||||
the copies of a garbage-layer PDF's strip are restored on an
|
||||
unchanged re-sync and the mode lands ``primary`` (the strip IS the
|
||||
view) — and the phase-133 fallback vision does NOT re-fire (the
|
||||
class is re-derived from the bytes, no vision call)."""
|
||||
root = tmp_path / "rgfall"
|
||||
root.mkdir()
|
||||
_build_garbage_pdf(root / "garbage.pdf") # 2 pages
|
||||
llm = _PageVisionLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgfall", "garbage.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (2, "primary")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 2):
|
||||
path.unlink()
|
||||
vision1, chats1, embeds1 = llm.vision_calls, len(llm.chat_calls), len(llm.calls)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert second.pdfs_screenshotted == 0, "the fallback gate did not re-run"
|
||||
assert llm.vision_calls == vision1, "ZERO vision calls on the re-gate"
|
||||
assert len(llm.chat_calls) == chats1 and len(llm.calls) == embeds1
|
||||
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgfall", "garbage.pdf")
|
||||
assert doc2.page_strip_mode == "primary", "the unusable layer is primary"
|
||||
assert doc2.page_count == 2
|
||||
for path in _strip_copies(doc2, page_dir, 2):
|
||||
assert path.is_file() and path.read_bytes()[:8] == PNG_MAGIC
|
||||
assert any(
|
||||
"healed 2 pages" in m and "re-gate" in m for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgfall")
|
||||
|
||||
|
||||
def test_regate_noop_when_the_strip_is_healthy(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""No-op arm (the IKEA-class doc whose copies all stand): an
|
||||
unchanged re-sync writes NOTHING (every copy's mtime and bytes are
|
||||
untouched), renders NOTHING, and logs NO re-gate line — the INFO is
|
||||
never per-unchanged-doc — while the pre-existing ``import:
|
||||
unchanged`` line still fires."""
|
||||
root = tmp_path / "rgnoop"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgnoop", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
before = {
|
||||
p: (p.stat().st_mtime_ns, p.read_bytes())
|
||||
for p in _strip_copies(doc, page_dir, 10)
|
||||
}
|
||||
assert len(before) == 10
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgnoop", "ikea.pdf")
|
||||
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
|
||||
after = {
|
||||
p: (p.stat().st_mtime_ns, p.read_bytes())
|
||||
for p in _strip_copies(doc2, page_dir, 10)
|
||||
}
|
||||
assert after == before, "a healthy strip is not written at all"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any(m.startswith("import: unchanged ") for m in messages), messages
|
||||
assert not any("re-gate" in m for m in messages), (
|
||||
f"the re-gate logged without acting: {messages}"
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, "rgnoop")
|
||||
|
||||
|
||||
def test_regate_restamps_a_changed_mode(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""Mode re-stamp (the class flip): a row whose stamped mode no
|
||||
longer matches the gate's class (here a pictorial-class row carrying
|
||||
``primary`` — a legacy/mis-stamped row) is re-stamped to
|
||||
``additive``, and because the mode was ALREADY SET the change is
|
||||
reported (``mode primary->additive``) — the only thing the re-gate
|
||||
did, so that is the whole INFO line (no copies were missing, none
|
||||
were written)."""
|
||||
root = tmp_path / "rgmode"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgmode", "ikea.pdf")
|
||||
assert doc.page_strip_mode == "additive"
|
||||
doc.page_strip_mode = "primary" # the mis-stamp the re-gate fixes
|
||||
db.commit()
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
before = {
|
||||
p: p.stat().st_mtime_ns for p in _strip_copies(doc, page_dir, 10)
|
||||
}
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.unchanged) == (0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgmode", "ikea.pdf")
|
||||
assert doc2.page_strip_mode == "additive", "the class is re-stamped"
|
||||
assert doc2.page_count == 10
|
||||
assert {
|
||||
p: p.stat().st_mtime_ns for p in _strip_copies(doc2, page_dir, 10)
|
||||
} == before, "a mode fix writes no files"
|
||||
assert (
|
||||
"import: PDF strip re-gate source=rgmode path=ikea.pdf — "
|
||||
"mode primary->additive"
|
||||
) in [r.getMessage() for r in caplog.records], caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgmode")
|
||||
|
||||
|
||||
def test_regate_clears_a_strip_that_no_longer_qualifies(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""REPAIR arm: a row claims a strip (``page_count`` set, copies on
|
||||
disk) on a PDF whose GATE now says ineligible (a clean, dense text
|
||||
layer — zero sparse pages). The unchanged re-sync clears BOTH
|
||||
columns and deletes the copies (the strip must not linger over a
|
||||
document the viewer shows as text) + the ``cleared stale strip``
|
||||
INFO line, with zero llm calls."""
|
||||
root = tmp_path / "rgclear"
|
||||
root.mkdir()
|
||||
pdf = root / "clean.pdf"
|
||||
_build_dense_pdf(pdf) # usable layer, zero candidates — never qualifies
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgclear", "clean.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (None, None)
|
||||
# The stale state the re-gate must repair (a strip stamped by an
|
||||
# older gate/settings, copies still on disk).
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
page_dir.mkdir(parents=True, exist_ok=True)
|
||||
for path in _strip_copies(doc, page_dir, 2):
|
||||
path.write_bytes(b"STALE-COPY")
|
||||
doc.page_count = 2
|
||||
doc.page_strip_mode = "additive"
|
||||
db.commit()
|
||||
embeds1, chats1 = len(llm.calls), len(llm.chat_calls)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert len(llm.calls) == embeds1 and len(llm.chat_calls) == chats1
|
||||
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgclear", "clean.pdf")
|
||||
assert doc2.page_count is None, "the stale strip is cleared"
|
||||
assert doc2.page_strip_mode is None, "the class is cleared with it"
|
||||
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies are deleted"
|
||||
assert (
|
||||
"import: PDF strip re-gate source=rgclear path=clean.pdf — "
|
||||
"cleared stale strip"
|
||||
) in [r.getMessage() for r in caplog.records], caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgclear")
|
||||
|
||||
|
||||
def test_regate_kill_switch_clears_the_pictorial_strip(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""REPAIR arm via the kill switch (``pdf_pictorial=False``): the
|
||||
changed-path semantics mirrored — with the pass off a sparse-page
|
||||
PDF is NOT eligible, so its existing strip is cleared with its
|
||||
copies on the next (unchanged) sync. The bytes never moved: the
|
||||
setting alone decided."""
|
||||
root = tmp_path / "rgkill"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgkill", "ikea.pdf")
|
||||
assert (doc.page_count, doc.page_strip_mode) == (10, "additive")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
assert len(list(page_dir.glob(f"{doc.id}-p*.png"))) == 10
|
||||
vision1 = llm.pictorial_calls
|
||||
|
||||
llm.settings = _regate_settings(tmp_path, pdf_pictorial=False)
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert llm.pictorial_calls == vision1, "ZERO vision calls (the pass is off)"
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgkill", "ikea.pdf")
|
||||
assert doc2.page_count is None and doc2.page_strip_mode is None
|
||||
assert not list(page_dir.glob(f"{doc2.id}-p*.png")), "the copies went too"
|
||||
assert any(
|
||||
"cleared stale strip" in m for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgkill")
|
||||
|
||||
|
||||
def test_regate_render_exception_is_fail_soft(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Fail-soft: the heal's render RAISES — the sync completes
|
||||
(``unchanged``, zero errors), the row keeps its prior state
|
||||
(``page_count`` + mode untouched — the re-gate never clears a strip
|
||||
on a failure), no copies are touched, and the WARNING is the only
|
||||
signal."""
|
||||
root = tmp_path / "rgboom"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgboom", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
|
||||
def boom(*args: object, **kwargs: object) -> list[bytes]:
|
||||
raise RuntimeError("simulated render failure (test sentinel)")
|
||||
|
||||
monkeypatch.setattr(importer, "render_pdf_pages", boom)
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged, second.errors) == (
|
||||
0, 0, 1, 0,
|
||||
)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgboom", "ikea.pdf")
|
||||
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive"), (
|
||||
"the row keeps its pre-re-gate state"
|
||||
)
|
||||
assert not list(page_dir.glob(f"{doc.id}-p*.png")), "no copies were touched"
|
||||
assert any(
|
||||
"PDF strip re-gate failed" in m and "test sentinel" in m
|
||||
for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgboom")
|
||||
|
||||
|
||||
def test_regate_empty_render_keeps_the_row(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Fail-soft on the no-pages render (unopenable right after the fast
|
||||
path hashed the file — pathological): the row is kept EXACTLY as it
|
||||
was (never cleared on a transient), nothing is written, and the
|
||||
WARNING names the no-pages state."""
|
||||
root = tmp_path / "rgempty"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgempty", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
monkeypatch.setattr(importer, "render_pdf_pages", lambda *a, **k: [])
|
||||
|
||||
with caplog.at_level(logging.WARNING, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.unchanged, second.errors) == (0, 1, 0)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgempty", "ikea.pdf")
|
||||
assert (doc2.page_count, doc2.page_strip_mode) == (10, "additive")
|
||||
assert not page_dir.exists() or not list(page_dir.glob(f"{doc.id}-*"))
|
||||
assert any(
|
||||
"re-render produced no pages (state kept)" in m
|
||||
for m in caplog.messages
|
||||
), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgempty")
|
||||
|
||||
|
||||
def test_regate_reports_a_page_count_drift_on_the_heal(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Drift (pathological on unchanged bytes, so it is LOUD): the heal
|
||||
renders 6 pages for a row that claimed 10 — ``page_count`` follows
|
||||
the fresh render, the ``p7..p10`` tail beyond it is deleted, and a
|
||||
WARNING names both counts beside the ``healed`` INFO line."""
|
||||
root = tmp_path / "rgdrift"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgdrift", "ikea.pdf")
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
monkeypatch.setattr(
|
||||
importer,
|
||||
"render_pdf_pages",
|
||||
lambda *a, **k: [PNG_MAGIC + f"{i}".encode() for i in range(6)],
|
||||
)
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.unchanged) == (0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgdrift", "ikea.pdf")
|
||||
assert doc2.page_count == 6, "the count follows the fresh render"
|
||||
for n in range(1, 7):
|
||||
assert (page_dir / f"{doc2.id}-p{n}.png").is_file()
|
||||
for n in range(7, 11):
|
||||
assert not (page_dir / f"{doc2.id}-p{n}.png").exists(), f"p{n} leaked"
|
||||
messages = [r.getMessage() for r in caplog.records]
|
||||
assert any("page_count drift 10->6" in m for m in messages), messages
|
||||
assert any("healed 10 pages" in m for m in messages), messages
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
_cleanup_source(db, "rgdrift")
|
||||
|
||||
|
||||
def test_regate_fires_on_the_content_digest_hit_path(
|
||||
db, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The SECOND unchanged path (the PDF branch's content-digest hit —
|
||||
a pre-phase row with ``file_hash`` NULL): the branch re-runs its own
|
||||
gate (the byte-change's honest cost) and lands unchanged, and the
|
||||
shared helper's tail still heals the strip deleted out from under
|
||||
the row — the re-gate is not fast-path-only."""
|
||||
root = tmp_path / "rgbranch"
|
||||
root.mkdir()
|
||||
_build_ikea_like_pdf(root / "ikea.pdf")
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = _doc(db, "rgbranch", "ikea.pdf")
|
||||
doc.file_hash = None # the pre-phase row: the fast path cannot fire
|
||||
db.commit()
|
||||
page_dir = _pages_dir(tmp_path)
|
||||
for path in _strip_copies(doc, page_dir, 10):
|
||||
path.unlink()
|
||||
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
db.expire_all()
|
||||
doc2 = _doc(db, "rgbranch", "ikea.pdf")
|
||||
assert doc2.file_hash is not None, "the helper stamped the hash"
|
||||
for path in _strip_copies(doc2, page_dir, 10):
|
||||
assert path.is_file(), f"copy {path.name} was not healed"
|
||||
assert any("healed 10 pages" in m for m in caplog.messages), caplog.messages
|
||||
finally:
|
||||
_cleanup_source(db, "rgbranch")
|
||||
|
||||
|
||||
def test_regate_guard_costs_a_non_strip_file_nothing(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The GUARD (A5): a non-PDF unchanged file and a PDF whose
|
||||
``page_count`` is NULL reach the tail of ``_unchanged_doc`` and
|
||||
``extract_pdf_text`` is NEVER called for either — every non-strip
|
||||
file's fast path keeps its zero-extra-read contract (one sync of
|
||||
each class, then the spy covers the second sync only)."""
|
||||
root = tmp_path / "rgguard"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nplain text body\n")
|
||||
_build_dense_pdf(root / "clean.pdf") # usable layer, no strip
|
||||
llm = _PictorialLLM()
|
||||
llm.settings = _regate_settings(tmp_path)
|
||||
seen: list[str] = []
|
||||
real_extract = importer.extract_pdf_text
|
||||
|
||||
def spy(path: Path) -> str:
|
||||
seen.append(str(path))
|
||||
return real_extract(path)
|
||||
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
monkeypatch.setattr(importer, "extract_pdf_text", spy)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 2)
|
||||
assert seen == [], f"the guard let a PDF parse through: {seen}"
|
||||
finally:
|
||||
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
|
||||
_cleanup_source(db, "rgguard")
|
||||
|
||||
|
||||
# ---------- the preserved semantics on the fast path --------------------------
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user