Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 163_pdf_strip_additive_selfheal
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Details
Build and Push Containers / build-and-push-db (push) Successful in 21s
Details

Browse Source 
All gates verified. Final report:

**Phase 163 — final verification pass (all 5 tasks already complete; verification-only run)**

- Verified D1–D4 in code: `page_strip_mode` column + migration 0027, `DocContent` absent-never-null rule keyed off `page_count`; `_persist_pdf_pages` render → atomic tmp+`os.replace` writes → stale-tail-last + `strip_class` stamp; `_regate_pdf_strip` on both unchanged paths (mirror gate, heal, repair, fail-soft, one INFO line only when acting); viewer mode branch + Text/Pages toggle (Text default, `aria-pressed`, no innerHTML, lazy imgs).
- `uv run pytest --cov=app --cov-report=term-missing` → **3796 tests, exit 0, coverage 99%** (>90%)
- `uv run pytest tests/e2e/test_pdf_strip_additive_selfheal.py -v --no-cov` → **5 passed** in isolation
- `test_pdf_pages_in_sources + test_pictorial_pdf + test_garbage_pdf` → 10 passed; `test_document_viewer(_context) + test_image_documents` → 20 passed
- `uv run ruff check .` clean; `uv run pyright` → 0 errors; alembic **0027 downgrade→upgrade CLI round-trip clean** (+ integration pins)
- Completion criteria: additive view w/ toggle ✅ · primary DOM byte-identity ✅ · write-first order/mid-write-failure/atomic pins ✅ · heal + repair E2E ✅ · zero vision/LLM on fast path (mock-captured) ✅ · non-qualifying payload byte-identity ✅ · docs pass ✅
- **Documentation**: README.md + AGENTS.md reviewed in full — the phase's updates are accurate; all config-table defaults cross-checked against `app/config.py` (no drift), every page route, script, and E2E command referenced exists, every `.env.example` var documented — **verified consistent, no changes needed**.
- Notable: live `Letumdei` repair is owner-runtime post-deploy per D5 (next production Sync self-heals; no live changes made); the modified `.agents/screenshots/doc_modal_scrollbar_1280.png` is an E2E-regenerated artifact left for the pipeline commit.
- **Next pending phase:** `164_chat_error_honesty`
This commit is contained in:
ducoterra
2026-10-07 13:57:30 -04:00
parent 74eb2e265c
commit 6ba46e735a
52 changed files with 4729 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/unit/test_pdf_page_copies.py
+312 -18
View File
@@ -21,14 +21,20 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
layer, ≥1 sparse page) and the fallback class (unusable layer) get
``<doc-id>-p{N}.png`` for EVERY page (not just the described
candidates — the strip shows the whole manual) + ``page_count`` =
the page count + real PNG bytes; a text-heavy PDF gets neither
(no copies, NULL count, ZERO render calls, byte-identical content);
the page count + real PNG bytes + the phase-163 CLASS stamp
(``page_strip_mode`` ``additive`` / ``primary``); a text-heavy PDF
gets neither (no copies, NULL count, NULL mode, ZERO render calls,
byte-identical content);
* the phase-147 fast path — an unchanged qualifying PDF's second sync
re-renders NOTHING (zero render calls — the copies persist,
``page_count`` and the bytes untouched);
* stale replacement — a changed (shrunk) PDF's old copy set is
replaced: the new set covers the new pages, the old ``p{N}`` tail
is GONE (no orphans), ``page_count`` follows the new count;
* WRITE-FIRST replacement (phase 163, task 02, LOCKED A4) — a changed
(shrunk) PDF's new copies land atomically (tmp + ``os.replace``,
pinned) BEFORE any old copy is removed; the old ``p{N}`` tail is
GONE only after the replacement bytes are durable (no orphans),
``page_count`` follows the new count; a MID-WRITE failure leaves the
prior copy set intact (the rollback-orphan state — the row over
zero files — is structurally unrepresentable);
* qualifier loss — a qualifying PDF whose new version gained a dense
text layer loses its strip: ``page_count`` NULL + copies deleted;
* unopenable — a render that yields no pages (``render_pdf_pages`` →
@@ -40,9 +46,10 @@ builder for the negative; the copies land in a tmp ``pdf_page_dir``):
the FULL count (the gap is a display-level 404 — task 02 pins the
route's side);
* fail-soft — the render raising (warning is the signal, ``page_count``
NULL, no copies, the doc indexed fully — chunks + summary) and the
WRITER raising (the copies this call wrote are removed — no partial
orphans, ``page_count`` NULL, the doc indexed fully);
+ ``page_strip_mode`` NULL, no copies, the doc indexed fully — chunks
+ summary) and the WRITER raising (this call's ``.tmp`` files and the
finals it CREATED are removed — no partial orphans, ``page_count`` +
``page_strip_mode`` NULL, the doc indexed fully);
* prune — a qualifying doc's file removed from the source: the row
goes and its page copies go with it (no orphans in the tmp dir).
@@ -54,6 +61,7 @@ from __future__ import annotations
import asyncio
import logging
import os
import uuid
from collections.abc import Callable, Sequence
from pathlib import Path
@@ -161,6 +169,35 @@ def _render_spy(
return calls
# ---------------------------------------------------------------------------
# Phase 163 (task 01) — the strip-class column beside page_count
# ---------------------------------------------------------------------------
def test_pictorial_index_stamps_the_additive_class(db, tmp_path: Path) -> None:
"""Phase 163 (task 02, re-cut of the task-01 pin) — the STAMP: a
qualifying PDF indexed through the PICTORIAL class gets
``page_strip_mode == "additive"`` beside ``page_count`` (the text +
toggle view); the strip's copies themselves are unchanged by the
phase (task 01's shape, mode stamped exactly with the count).
"""
root = tmp_path / "pgm"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
doc = _doc(db, "pgm", "ikea.pdf")
assert doc.page_count == 10, "the strip itself is unchanged by task 02"
assert doc.page_strip_mode == "additive", (
"the pictorial class stamps the additive mode (LOCKED A1/A3)"
)
finally:
_cleanup_source(db, "pgm")
# ---------------------------------------------------------------------------
# Eligibility × new index
# ---------------------------------------------------------------------------
@@ -219,6 +256,10 @@ def test_fallback_class_garbage_pdf_writes_every_page_copy(db, tmp_path: Path) -
assert summary.pdfs_screenshotted == 1 # the fallback fired (the class)
doc = _doc(db, "pgf", "garbage.pdf")
assert doc.page_count == 2
assert doc.page_strip_mode == "primary", (
"the fallback class stamps the primary mode — the strip IS the "
"view, byte-identical to phase 148 otherwise (LOCKED A1)"
)
page_dir = _pages_dir(tmp_path)
for n in (1, 2):
path = page_dir / f"{doc.id}-p{n}.png"
@@ -253,6 +294,7 @@ def test_text_heavy_pdf_keeps_the_text_view(
assert summary.pdfs_pictorial == 0 # zero candidates — the pass no-ops
doc = _doc(db, "pgd", "clean.pdf")
assert doc.page_count is None, "a text-heavy PDF keeps the text view"
assert doc.page_strip_mode is None, "no strip — no class either"
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
assert calls == [], "zero render calls (no fallback, zero candidates)"
assert doc.content == expected, "byte-identical to the extraction"
@@ -315,9 +357,10 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
) -> None:
"""A CHANGED (shrunk, 10 → 6 pages) qualifying PDF: the re-index
replaces the copy set — the new set covers p1..p6, the old
version's ``p7..p10`` tail is GONE (the stale-replacement runs
before the write), the doc keeps its id, and ``page_count``
follows the new page count."""
version's ``p7..p10`` tail is GONE (the stale-tail delete runs
LAST, after the write-first loop — phase 163, task 02), the doc
keeps its id, ``page_count`` follows the new page count, and
``page_strip_mode`` is (re-)stamped."""
root = tmp_path / "pgc"
root.mkdir()
pdf = root / "ikea.pdf"
@@ -336,6 +379,7 @@ def test_changed_pdf_replaces_the_copy_set_without_orphans(
doc2 = _doc(db, "pgc", "ikea.pdf")
assert doc2.id == doc1.id, "the doc keeps its id (the copy path is stable)"
assert doc2.page_count == 6, "page_count follows the new page count"
assert doc2.page_strip_mode == "additive", "the class is (re-)stamped"
page_dir = _pages_dir(tmp_path)
for n in range(1, 7):
assert (page_dir / f"{doc2.id}-p{n}.png").is_file(), f"p{n} is missing"
@@ -379,6 +423,7 @@ def test_qualifier_loss_clears_page_count_and_copies(
doc2 = _doc(db, "pgl", "ikea.pdf")
assert doc2.id == doc1.id
assert doc2.page_count is None, "a doc that stopped qualifying loses its strip"
assert doc2.page_strip_mode is None, "the class is cleared with the count"
assert not list(_pages_dir(tmp_path).glob(f"{doc2.id}-p*.png")), (
"the copies must be deleted on qualifier loss"
)
@@ -411,6 +456,7 @@ def test_unopenable_pdf_page_render_is_fail_soft(
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
doc = _doc(db, "pgn", "garbage.pdf")
assert doc.page_count is None, "no pages rendered — no strip"
assert doc.page_strip_mode is None, "the class is cleared with the count"
assert not _pages_dir(tmp_path).exists(), "no copies — the dir is never created"
assert doc.content == "", "the fallback's all-failed render lands empty content"
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
@@ -493,10 +539,11 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""The PNG WRITER raises (a full disk) after the FIRST copy
landed: the copies this call wrote are removed (no partial
orphans), ``page_count`` NULL, the warning is the signal, and the
document indexes exactly as pre-phase."""
"""The PNG WRITER raises (a full disk) after the FIRST atomic copy
landed: this call's ``.tmp`` files and the final-name files it
CREATED are removed (no partial orphans — phase 163, task 02),
``page_count`` AND ``page_strip_mode`` NULL, the warning is the
signal, and the document indexes exactly as pre-phase."""
root = tmp_path / "pgw"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
@@ -507,22 +554,29 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
def boom(self: Path, data: bytes) -> int:
written_paths.append(self)
n = original_write(self, data) # the tmp file really lands first
if len(written_paths) > 1:
raise OSError("simulated disk full (test sentinel)")
return original_write(self, data)
return n
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
try:
with caplog.at_level(logging.WARNING, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.errors) == (1, 1, 0)
assert len(written_paths) == 2, "p1 landed, p2's write failed"
assert len(written_paths) == 2, "p1's tmp landed + replaced, p2's write failed"
assert all(p.name.endswith(".png.tmp") for p in written_paths), (
"the writer only ever touches .tmp paths (atomic replace)"
)
doc = _doc(db, "pgw", "ikea.pdf")
assert doc.page_count is None
assert doc.page_strip_mode is None, "the class is cleared with the count"
assert not list(_pages_dir(tmp_path).glob(f"{doc.id}-p*.png")), (
"the partial write was cleaned up — no orphans"
)
assert not written_paths[0].exists(), "the landed p1 copy was removed"
assert not list(_pages_dir(tmp_path).glob("*.tmp")), (
"the failed call's .tmp file was cleaned up too"
)
assert summary.summaries == 1, "the document is indexed exactly as pre-phase"
assert any("PDF page render failed" in m for m in caplog.messages), (
caplog.messages
@@ -531,6 +585,246 @@ def test_write_exception_is_fail_soft_and_leaves_no_orphans(
_cleanup_source(db, "pgw")
# ---------------------------------------------------------------------------
# Phase 163 (task 02) — WRITE-FIRST, atomic per file (LOCKED A4)
# ---------------------------------------------------------------------------
def test_write_first_new_bytes_are_durable_before_any_stale_delete(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE order pin (LOCKED A4 — the 2026-10-07 live loss): re-rendering
a CHANGED (shrunk, 10 → 6) PDF writes every replacement page FIRST
(tmp + ``os.replace``) and only then deletes the stale tail — the
event log shows ALL replaces of the fresh bytes before the FIRST
unlink of an old-extent file, the overwritten pages keep the NEW
bytes, the shrunk tail (p7..p10) is gone, and nothing else is
deleted."""
root = tmp_path / "pgord"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
asyncio.run(import_sources([root], llm, session=db))
doc1 = _doc(db, "pgord", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
sentinel = b"OLD-COPY-SENTINEL"
for n in range(1, 11): # make the old copies unmistakably distinguishable
(page_dir / f"{doc1.id}-p{n}.png").write_bytes(sentinel)
events: list[tuple[str, str]] = []
real_replace = os.replace
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
events.append(("replace", Path(os.fspath(dst)).name))
return real_replace(src, dst, *args, **kwargs)
real_unlink = Path.unlink
def unlink_spy(self: Path, **kwargs: Any) -> None:
if self.parent == page_dir and self.name.startswith(str(doc1.id)):
events.append(("unlink", self.name))
return real_unlink(self, **kwargs)
monkeypatch.setattr(importer.os, "replace", replace_spy)
monkeypatch.setattr(Path, "unlink", unlink_spy) # pyright: ignore[reportArgumentType]
try:
_build_ikea_shrunk(pdf) # same path, 6 pages — still qualifying
s2 = asyncio.run(import_sources([root], llm, session=db))
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
_cleanup_source(db, "pgord")
replaces = [name for kind, name in events if kind == "replace"]
unlinks = [name for kind, name in events if kind == "unlink"]
assert len(replaces) == 6, "the 6 fresh pages are written via os.replace"
assert set(unlinks) == {f"{doc1.id}-p{n}.png" for n in range(7, 11)}, (
"the stale-tail delete touches ONLY the old extent the render did "
"not rewrite — a file about to be overwritten is never deleted"
)
first_unlink = next(i for i, e in enumerate(events) if e[0] == "unlink")
last_replace = max(i for i, e in enumerate(events) if e[0] == "replace")
assert last_replace < first_unlink, (
f"write-first violated — event order: {events}"
)
for n in range(1, 7):
blob = (page_dir / f"{doc1.id}-p{n}.png").read_bytes()
assert blob != sentinel and blob[:8] == PNG_MAGIC, (
f"p{n} keeps the NEW rendered bytes"
)
for n in range(7, 11):
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"stale p{n} leaked"
assert not list(page_dir.glob("*.tmp")), "no .tmp orphans"
def test_mid_write_failure_keeps_the_prior_copies_intact(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""THE failure pin (LOCKED A4): the writer raises on the SECOND
page's tmp write of a changed re-render — the prior copy set stays
on disk INTACT (the rollback-orphan state — the row over zero
files — is structurally unrepresentable): every one of the 10 prior
files still exists (p1 with the new bytes already atomically
replaced before the failure, p2..p10 byte-for-byte the prior
copies), zero ``.tmp`` orphans from this call, and the caller's
fail-soft fires (``page_count``/``page_strip_mode`` NULL + the
warning, the doc indexed fully)."""
root = tmp_path / "pgfail"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
asyncio.run(import_sources([root], llm, session=db))
doc1 = _doc(db, "pgfail", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
prior = {
n: (page_dir / f"{doc1.id}-p{n}.png").read_bytes() for n in range(1, 11)
}
# Tag the re-render's bytes so the new p1 is recognizable, then
# blow up the SECOND tmp write (page 2).
_render_spy(
monkeypatch,
mutate_full=lambda pages: [
PNG_MAGIC + f"NEW-{i}".encode() for i in range(len(pages))
],
)
original_write = Path.write_bytes
tmp_writes: list[Path] = []
def boom(self: Path, data: bytes) -> int:
if self.suffix == ".tmp":
tmp_writes.append(self)
if len(tmp_writes) > 1:
raise OSError("simulated disk full (test sentinel)")
return original_write(self, data)
monkeypatch.setattr(Path, "write_bytes", boom) # pyright: ignore[reportArgumentType]
_build_ikea_shrunk(pdf)
try:
with caplog.at_level(logging.WARNING, logger="app.importer"):
s2 = asyncio.run(import_sources([root], llm, session=db))
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
assert len(tmp_writes) == 2, "p1's tmp wrote, p2's tmp write raised"
doc2 = _doc(db, "pgfail", "ikea.pdf")
assert doc2.page_count is None, "the caller's fail-soft nulled the count"
assert doc2.page_strip_mode is None, "the caller's fail-soft nulled the class"
assert any("PDF page render failed" in m for m in caplog.messages), caplog.messages
# The prior set is INTACT: exactly the 10 names, none lost, none
# orphaned on top.
assert {p.name for p in page_dir.iterdir()} == {
f"{doc1.id}-p{n}.png" for n in range(1, 11)
}, "files on disk exactly the prior set"
assert (
page_dir / f"{doc1.id}-p1.png"
).read_bytes() == PNG_MAGIC + b"NEW-0", (
"p1's replacement bytes were durable before the failure (atomic "
"replace) and were NOT removed by the cleanup"
)
for n in range(2, 11):
assert (
page_dir / f"{doc1.id}-p{n}.png"
).read_bytes() == prior[n], f"prior copy p{n} was disturbed"
assert not list(page_dir.glob("*.tmp")), "zero .tmp orphans from this call"
assert doc2.chunks, "the document is indexed fully (fail-soft)"
def test_page_copies_are_written_atomically_tmp_then_replace(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""THE atomic pin (LOCKED A4): the writer NEVER writes a final
``<doc-id>-p{N}.png`` path directly — every page's bytes land on
``<name>.png.tmp`` and ``os.replace`` moves them onto the final
name in order (a reader opening the final path never sees truncated
bytes), and no ``.tmp`` survives the successful call."""
root = tmp_path / "pgatom"
root.mkdir()
_build_ikea_like_pdf(root / "ikea.pdf")
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
written: list[Path] = []
original_write = Path.write_bytes
def write_spy(self: Path, data: bytes) -> int:
written.append(self)
return original_write(self, data)
replaced: list[tuple[str, str]] = []
real_replace = os.replace
def replace_spy(src, dst, *args, **kwargs): # pyright: ignore[reportUnknownParameterType]
replaced.append((os.fspath(src), os.fspath(dst)))
return real_replace(src, dst, *args, **kwargs)
monkeypatch.setattr(Path, "write_bytes", write_spy) # pyright: ignore[reportArgumentType]
monkeypatch.setattr(importer.os, "replace", replace_spy)
try:
asyncio.run(import_sources([root], llm, session=db))
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
doc = _doc(db, "pgatom", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
assert len(written) == 10 and len(replaced) == 10
assert all(p.name.endswith(".png.tmp") for p in written), (
"no final path is ever written directly"
)
for n, (src, dst) in enumerate(replaced, start=1):
assert dst == str(page_dir / f"{doc.id}-p{n}.png")
assert src == f"{dst}.tmp", "the tmp name is <final>.tmp"
assert not list(page_dir.glob("*.tmp")), "no .tmp survives the success path"
def test_stale_tail_delete_failure_is_best_effort(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""An unreadable STALE-TAIL copy (the write-first tail pass, the
``_delete_page_copies`` convention): the failure is logged, never
raised — the sync completes, the row is stamped with the fresh
count, and the reachable tail files still go."""
root = tmp_path / "pgtail"
root.mkdir()
pdf = root / "ikea.pdf"
_build_ikea_like_pdf(pdf)
llm = _PictorialLLM()
llm.settings = _settings(tmp_path)
asyncio.run(import_sources([root], llm, session=db))
doc1 = _doc(db, "pgtail", "ikea.pdf")
page_dir = _pages_dir(tmp_path)
p10 = page_dir / f"{doc1.id}-p10.png"
real_unlink = Path.unlink
def boom(self: Path, **kwargs: Any) -> None:
if self == p10:
raise OSError("simulated unreadable file (test sentinel)")
return real_unlink(self, **kwargs)
monkeypatch.setattr(Path, "unlink", boom) # pyright: ignore[reportArgumentType]
_build_ikea_shrunk(pdf)
try:
with caplog.at_level(logging.WARNING, logger="app.importer"):
s2 = asyncio.run(import_sources([root], llm, session=db))
finally:
monkeypatch.undo() # pyright: ignore[reportUnknownMemberType]
assert (s2.added, s2.updated, s2.unchanged) == (0, 1, 0)
doc2 = _doc(db, "pgtail", "ikea.pdf")
assert doc2.page_count == 6 and doc2.page_strip_mode == "additive"
assert p10.exists(), "the unreadable tail file survived (never raised)"
for n in range(7, 10):
assert not (page_dir / f"{doc1.id}-p{n}.png").exists(), f"p{n} still leaked"
assert any(
"could not delete PDF page copy" in m for m in caplog.messages
), caplog.messages
# ---------------------------------------------------------------------------
# The shared cleanup helper (the _delete_image_copy convention)
# ---------------------------------------------------------------------------