phase: 185_sync_defer_content_fast_path
All gates green. Report: **Implemented / verified (phase 185)** - `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`. - `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget). - **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`). - **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order. **Tests / lint (all 100% green)** - `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%). - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. - Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes). **Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run). **Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists). **Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified). **Next pending phase**: `186_git_log_streaming_dates`.
This commit is contained in:
@@ -0,0 +1,749 @@
|
||||
"""Integration: phase 185 — what the UNCHANGED fast path SELECTs (the probe).
|
||||
|
||||
**VERDICT: CONFIRMED (task 01) → FIXED (phase 185 task 02,
|
||||
``defer(Document.content)`` in ``app/rag/importer.py::_doc_row_stmt``).**
|
||||
The fast path used to SELECT ``content`` for every
|
||||
file it inspects. Each inspected file is looked up by
|
||||
``app/rag/importer.py::_index_file`` (``select(Document).where(
|
||||
Document.source == source, Document.path == rel)``), which is a WHOLE-ENTITY
|
||||
load: its select list names ``documents.content`` — the full stored markdown,
|
||||
**15,217** chars on average — next to the metadata the fast path actually
|
||||
reads. The three sibling loads that share the phase-147 unchanged path (the
|
||||
PDF branch, the PDF-embedded-image branch, the standalone-image branch)
|
||||
compile to the same statement, so the whole KB walk pays this for EVERY file
|
||||
it looks at, not only for the ones that changed. Nothing is released
|
||||
afterwards: ``app/db.py:30`` builds every session with
|
||||
``expire_on_commit=False``, so a row the run read — and then dirtied with the
|
||||
``file_hash`` stamp (phase 184) — stays resident with its markdown until the
|
||||
run ends.
|
||||
|
||||
**What the fix changed (measured by this file, post-fix).** The four fast-path
|
||||
row loads now share ``_doc_row_stmt``, whose projection is every ``Document``
|
||||
column EXCEPT ``content``, and the unchanged run issues **7** documents
|
||||
SELECTs for the same four files: the **4** metadata-only row loads, **1**
|
||||
primary-key follow-up (``SELECT documents.content AS documents_content …
|
||||
WHERE documents.id = …``) — paid by the ONE summary-NULL row whose phase-118
|
||||
backfill genuinely reads the stored markdown — and the same **2** per-source
|
||||
post-passes. A file that already has a summary still costs exactly **1**
|
||||
documents SELECT and zero follow-ups (``test_unchanged_file_is_one_select``).
|
||||
|
||||
**The incident (owner, 2026-10-15 — the same run phases 183/184 fixed).**
|
||||
``documents`` held **17,407** rows at ``avg(length(content)) = 15,217``
|
||||
characters = **252.6 MB of markdown**. The sync inspected ~27k files and
|
||||
re-indexed **217**; the remaining ~26.8k took the fast path and still hydrated
|
||||
their markdown to discover that nothing had changed. Phase 183 took ``content``
|
||||
out of the prune's row (six light columns), phase 184 bounded what the walk
|
||||
RETAINS (``session.expunge_all()`` every ``BOR_SYNC_SESSION_RESET_FILES``
|
||||
files); this phase takes the markdown out of what the walk READS per unchanged
|
||||
file, so a full re-sync's memory scales with row metadata instead of with the
|
||||
corpus.
|
||||
|
||||
**Measured BY THIS FILE (2026-10-15, the dev Postgres; fixture: 4 markdown
|
||||
files of ~8 KB seeded through the real ``import_sources``, then a SECOND
|
||||
``import_sources`` over the same tree — the unchanged run — captured with
|
||||
``before_cursor_execute``, the phase-183 task-01 technique).** PRE-FIX the
|
||||
unchanged run issued **6** statements over ``documents``:
|
||||
|
||||
* **4** per-file ``(source, path)`` row loads — exactly one per inspected
|
||||
file, the fast path's ONE lookup, every one of them naming
|
||||
``documents.content`` (the load-bearing pin, flipped by task 02);
|
||||
* **0** primary-key follow-up SELECTs over ``documents`` — nothing was
|
||||
deferred yet, so nothing could be re-read; every file that already has a
|
||||
summary costs exactly **1** documents SELECT in total;
|
||||
* **2** per-source post-pass walks (``_enrich_pending_images`` +
|
||||
``_backfill_image_contexts``, both ``WHERE documents.source = … AND
|
||||
documents.is_image IS true``) — twice per RUN, independent of the file
|
||||
count, and outside this phase's scope (D1 names exactly the four fast-path
|
||||
row loads);
|
||||
* the one staged row (``documents.summary = NULL``, its stale ``is_summary``
|
||||
chunk stamped so replacement is observable) cost **1** documents SELECT
|
||||
total — the row load, whose already-hydrated ``content`` the phase-118
|
||||
backfill consumed directly. Post-task-02 that row is the ONLY one that pays
|
||||
a second, single-column SELECT.
|
||||
|
||||
The captured fast-path statement, verbatim for one file **POST-FIX** — every
|
||||
column of the PRE-FIX list below EXCEPT ``documents.content`` (which sat
|
||||
between ``title`` and ``content_hash``), so no markdown travels with an
|
||||
inspected row; the fast path reads ``file_hash``, then ``summary`` /
|
||||
``created_at`` / ``page_count`` / ``path`` / ``full_path`` — never the text
|
||||
this projection drops:
|
||||
|
||||
```
|
||||
SELECT documents.id, documents.source, documents.path, documents.full_path,
|
||||
documents.title, documents.content_hash, documents.file_hash,
|
||||
documents.indexed_at, documents.created_at, documents.created_at_manual,
|
||||
documents.summary, documents.is_image, documents.image_path,
|
||||
documents.image_context, documents.page_count, documents.page_strip_mode
|
||||
FROM documents
|
||||
WHERE documents.source = %(source_1)s::VARCHAR AND documents.path =
|
||||
%(path_1)s::VARCHAR
|
||||
```
|
||||
|
||||
**The PRE-FIX shape** (task 01's evidence — the statement above WITH
|
||||
``documents.content`` between ``title`` and ``content_hash``, which is what
|
||||
put ~250 MB of markdown in front of a run that re-indexed 217 of ~27k files)
|
||||
and **the follow-up a deferred row pays when it genuinely needs the
|
||||
markdown** — the phase-118 summary-NULL backfill, ``_unchanged_doc`` step 3,
|
||||
and nothing else on the fast path (measured 2026-10-15 by putting the same
|
||||
row behind an ad-hoc ``defer(Document.content)`` load and then reading
|
||||
``doc.content``)::
|
||||
|
||||
SELECT documents.content AS documents_content
|
||||
FROM documents
|
||||
WHERE documents.id = %(pk_1)s::UUID
|
||||
|
||||
That is the whole trade (phase 185 D2 / LOCKED A2): a doc WITH a stored
|
||||
summary pays ZERO extra queries, a summary-NULL backfill pays ONE per-row
|
||||
one-time SELECT, and deliberately NO ``raiseload``/``noload`` — those would
|
||||
turn the backfill into a crash instead of a query.
|
||||
|
||||
**What this file does NOT touch:** the two per-source post-pass walks above,
|
||||
``_prune``'s light row (phase 184), the identity-map cadence (phase 184),
|
||||
``app/db.py:30``, ``iter_importable_files`` (phase 185 LOCKED A3) and the
|
||||
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any, NamedTuple
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import event, func, select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, engine
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The fixture: 4 markdown files of ~8 KB, one seed run + one unchanged run
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The probe's own source name — nothing else in the KB uses it.
|
||||
PROBE_SOURCE = "fastpath-probe"
|
||||
|
||||
#: Four files: enough that "one row load per file" is arithmetic rather than
|
||||
#: a single sample, few enough that the seed run is instant.
|
||||
FILE_COUNT = 4
|
||||
|
||||
#: The row whose stored ``summary`` is nulled before the second run — the
|
||||
#: phase-118 backfill case, the ONE fast-path branch that genuinely needs the
|
||||
#: stored markdown.
|
||||
BACKFILL_PATH = "probe-02.md"
|
||||
|
||||
#: What the pre-existing ``is_summary`` chunk of that row is stamped with, so
|
||||
#: "the backfill replaced the chunk" is observable instead of assumed (the
|
||||
#: replacement is the only thing that can make the chunk read ``expected``).
|
||||
STALE_SUMMARY = "STALE SUMMARY CHUNK — the backfill must replace this"
|
||||
|
||||
#: One repeated sentence, so every file has the same shape (and therefore the
|
||||
#: same chunk count) as every other.
|
||||
_SENTENCE = (
|
||||
"The homelab runs podman quadlets on the hnb host with a static DNS "
|
||||
"zone, nightly restic backups and a systemd timer that probes uptime. "
|
||||
)
|
||||
|
||||
#: ~8 KB per file — production ``content`` averages 15.2 KB, so a whole-entity
|
||||
#: row load is tens of kilobytes per inspected file and ``content`` is
|
||||
#: unmistakably the fat column of the row (and stays under
|
||||
#: ``BOR_SUMMARY_MAX_CHARS``, so the backfill's summary call is uncapped).
|
||||
CONTENT_CHARS_PER_FILE = 8_000
|
||||
|
||||
#: The documents SELECTs of the two per-source post-passes, measured
|
||||
#: 2026-10-15 — asserted so a future per-file extra SELECT cannot hide among
|
||||
#: them (and so a post-pass that stopped querying is noticed too).
|
||||
POST_PASS_DOC_SELECTS = 2
|
||||
|
||||
#: **0 pre-phase-185** — the whole-entity row load already carries
|
||||
#: ``content``, so the phase-118 backfill needs no follow-up query. **1
|
||||
#: **0 pre-phase-185** — the whole-entity row load already carried
|
||||
#: ``content``, so the phase-118 backfill needed no follow-up query. **1
|
||||
#: post-phase-185 (the value now)** — the row load defers ``content``, so the
|
||||
#: ONE backfill pays exactly ONE lazy ``SELECT documents.content … WHERE
|
||||
#: documents.id = …``. The arithmetic: the backfilled file costs ``1 row load
|
||||
#: + EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL`` documents SELECTs, every other
|
||||
#: unchanged file costs exactly 1 (``test_unchanged_file_is_one_select``).
|
||||
#: Everything else about the backfill is identical across the flip: the stored
|
||||
#: summary, its ``is_summary`` chunk, the run's counts and its log words
|
||||
#: (``test_unchanged_run_keeps_every_row_intact``).
|
||||
EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 1
|
||||
|
||||
|
||||
class Captured(NamedTuple):
|
||||
"""One executed statement plus its bind parameters.
|
||||
|
||||
The parameters make the probe attributable: the fast-path row load carries
|
||||
``path_1`` (the file it looked up) while the deferred column's lazy
|
||||
follow-up carries only the primary key — so "one SELECT per unchanged
|
||||
file" and "one follow-up for the backfill" are two different, separately
|
||||
asserted claims.
|
||||
"""
|
||||
|
||||
text: str
|
||||
params: dict[str, Any]
|
||||
|
||||
|
||||
class UnchangedRun(NamedTuple):
|
||||
"""The unchanged run, snapshotted at the moment it ENDED.
|
||||
|
||||
``captured`` is a copy: the listener stays installed for the rest of the
|
||||
test, so the assertions must not race the probe's own read queries.
|
||||
"""
|
||||
|
||||
captured: list[Captured]
|
||||
summary: ImportSummary
|
||||
llm: FakeEmbedder
|
||||
|
||||
@property
|
||||
def lite_prompts(self) -> list[str]:
|
||||
"""The user-message text of every ``lite`` call the run made."""
|
||||
return [
|
||||
str(message.get("content"))
|
||||
for call in self.llm.chat_calls
|
||||
for message in call
|
||||
if message.get("role") == "user"
|
||||
]
|
||||
|
||||
|
||||
def _file_body(n: int) -> str:
|
||||
"""One ~8 KB markdown document, carrying a per-file sentinel per section."""
|
||||
parts = [f"# Probe document {n}\n"]
|
||||
section = 0
|
||||
while sum(len(p) for p in parts) < CONTENT_CHARS_PER_FILE:
|
||||
parts.append(f"\n## Section {section} of probe {n} (fastpath-sentinel-{n})\n\n")
|
||||
parts.extend(_SENTENCE for _ in range(4))
|
||||
parts.append("\n")
|
||||
section += 1
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def _seed_files(tmp: Path) -> Path:
|
||||
"""``FILE_COUNT`` markdown files under a fresh source root; returns it."""
|
||||
root = tmp / PROBE_SOURCE
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
for n in range(FILE_COUNT):
|
||||
body = _file_body(n)
|
||||
assert 8_000 <= len(body) <= 9_000, len(body)
|
||||
(root / f"probe-{n:02d}.md").write_text(body)
|
||||
return root
|
||||
|
||||
|
||||
def _probe_llm() -> FakeEmbedder:
|
||||
"""The deterministic fake embedder/summarizer on the SHIPPED defaults
|
||||
(``sync_session_reset_files=200`` — no identity-map release lands inside a
|
||||
four-file walk, so nothing but the row loads is in play here)."""
|
||||
llm = FakeEmbedder()
|
||||
llm.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
return llm
|
||||
|
||||
|
||||
def _run_import(root: Path, llm: FakeEmbedder) -> ImportSummary:
|
||||
"""One real ``import_sources`` pass over *root*, in its own session."""
|
||||
session = SessionLocal()
|
||||
try:
|
||||
return asyncio.run(
|
||||
import_sources([root], llm, prune=False, session=session)
|
||||
)
|
||||
finally:
|
||||
session.close()
|
||||
|
||||
|
||||
def _doc_id(db: Session, path: str) -> Any:
|
||||
"""The id of one probe row (lazy follow-ups are keyed by primary key)."""
|
||||
doc_id = db.scalar(
|
||||
select(Document.id).where(
|
||||
Document.source == PROBE_SOURCE, Document.path == path
|
||||
)
|
||||
)
|
||||
assert doc_id is not None
|
||||
return doc_id
|
||||
|
||||
|
||||
def _stage_the_backfill(db: Session, path: str) -> None:
|
||||
"""The phase-118 state: a row whose stored ``summary`` is NULL while its
|
||||
content and ``file_hash`` are current (the direct UPDATE of
|
||||
``tests/integration/test_importer_dates.py`` — exactly what a pre-phase-118
|
||||
import left behind).
|
||||
|
||||
The stale ``is_summary`` chunk is stamped rather than deleted: after the
|
||||
run a chunk carrying the backfill text can only have got there through
|
||||
:func:`app.rag.importer._store_summary`'s replacement rule, which is what
|
||||
makes "the backfill ran" a fact rather than an assumption.
|
||||
"""
|
||||
doc_id = _doc_id(db, path)
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE documents SET summary = NULL WHERE source = :s AND path = :p"
|
||||
),
|
||||
{"s": PROBE_SOURCE, "p": path},
|
||||
)
|
||||
db.execute(
|
||||
text("UPDATE chunks SET content = :stale WHERE document_id = :d AND is_summary"),
|
||||
{"stale": STALE_SUMMARY, "d": doc_id},
|
||||
)
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
def _two_runs(db: Session, tmp_path: Path, captured: list[Captured]) -> UnchangedRun:
|
||||
"""Seed the KB, then run the UNCHANGED pass with the capture active.
|
||||
|
||||
The seed run's SQL is cleared away at the boundary: this file pins what
|
||||
the unchanged walk costs, not what first indexing costs.
|
||||
"""
|
||||
root = _seed_files(tmp_path)
|
||||
first = _run_import(root, _probe_llm())
|
||||
assert (first.files, first.added, first.unchanged, first.errors) == (
|
||||
FILE_COUNT,
|
||||
FILE_COUNT,
|
||||
0,
|
||||
0,
|
||||
)
|
||||
_stage_the_backfill(db, BACKFILL_PATH)
|
||||
|
||||
captured.clear()
|
||||
llm = _probe_llm()
|
||||
second = _run_import(root, llm)
|
||||
run = UnchangedRun(list(captured), second, llm)
|
||||
|
||||
assert (second.files, second.added, second.unchanged, second.errors) == (
|
||||
FILE_COUNT,
|
||||
0,
|
||||
FILE_COUNT,
|
||||
0,
|
||||
)
|
||||
assert second.chunks == 0 # the fast path re-embeds no content
|
||||
assert second.summary_backfilled == 1 # the staged row was backfilled
|
||||
return run
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def clean_kb(db: Session) -> Iterator[None]:
|
||||
"""``documents`` / ``chunks`` are global state: reset around every test
|
||||
(the ``test_sync_api.py`` truncate pattern) and drop whatever the session
|
||||
cached, so the next read is the database's truth."""
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
yield
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def statement_capture() -> Iterator[list[Captured]]:
|
||||
"""Every SQL the engine executes for the duration of ONE test.
|
||||
|
||||
The ``before_cursor_execute`` technique of phase 183 task 01
|
||||
(``tests/integration/test_prune_cascade_statements.py``): the column list
|
||||
a SELECT uses is invisible in the ORM's Python API and plain to see in the
|
||||
statement it compiles, so a capture is the only deterministic way to pin
|
||||
what the fast path's row actually contains. Installed on the engine and
|
||||
REMOVED in teardown, so no other test of the session ever sees the
|
||||
listener (verified by the full ``uv run pytest`` run being green with this
|
||||
file in it).
|
||||
"""
|
||||
captured: list[Captured] = []
|
||||
|
||||
def _listener(
|
||||
conn: object, # noqa: ARG001
|
||||
cursor: object, # noqa: ARG001
|
||||
statement: str,
|
||||
parameters: object,
|
||||
context: object, # noqa: ARG001
|
||||
executemany: bool, # noqa: ARG001
|
||||
) -> None:
|
||||
params = dict(parameters) if isinstance(parameters, dict) else {}
|
||||
captured.append(Captured(statement, params))
|
||||
|
||||
event.listen(engine, "before_cursor_execute", _listener)
|
||||
try:
|
||||
yield captured
|
||||
finally:
|
||||
event.remove(engine, "before_cursor_execute", _listener)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers — reading a captured statement
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: ``… FROM documents`` — case-insensitive, word-bounded.
|
||||
_FROM_DOCUMENTS_RE = re.compile(r"\bfrom\s+documents\b", re.IGNORECASE)
|
||||
|
||||
#: Where the projection ends: SQLAlchemy compiles ``FROM`` onto its own line,
|
||||
#: and everything after it is a predicate, not a column the row hydrates.
|
||||
_FROM_RE = re.compile(r"\bFROM\b")
|
||||
|
||||
#: The ``content`` COLUMN TOKEN, aliased or plain. The trailing ``\\b`` is what
|
||||
#: keeps the neighbours out: ``_`` is a WORD character, so the ``documents.content``
|
||||
#: inside ``documents.content_hash`` has no boundary after the ``t`` and never
|
||||
#: matches. That is why this is a token regex and not a substring test —
|
||||
#: ``content_hash`` is in EVERY one of these select lists, and a substring
|
||||
#: match would report a content load for every row of every run, pre- and
|
||||
#: post-fix alike.
|
||||
_CONTENT_TOKEN_RE = re.compile(r"\bdocuments\.content\b")
|
||||
|
||||
|
||||
def doc_selects(captured: list[Captured]) -> list[Captured]:
|
||||
"""The captured statements that SELECT the ``documents`` table."""
|
||||
return [
|
||||
c
|
||||
for c in captured
|
||||
if c.text.lstrip().lower().startswith("select")
|
||||
and _FROM_DOCUMENTS_RE.search(c.text)
|
||||
]
|
||||
|
||||
|
||||
def loads_content(statement: Captured | str) -> bool:
|
||||
"""``True`` when the statement's SELECT LIST names the ``content`` column.
|
||||
|
||||
Only the projection is inspected (see :data:`_FROM_RE`), and only the
|
||||
column token matches (see :data:`_CONTENT_TOKEN_RE`).
|
||||
"""
|
||||
text = statement.text if isinstance(statement, Captured) else statement
|
||||
from_at = _FROM_RE.search(text)
|
||||
projection = text[: from_at.start()] if from_at else text
|
||||
return _CONTENT_TOKEN_RE.search(projection) is not None
|
||||
|
||||
|
||||
def test_the_content_matcher_names_the_column_and_nothing_else() -> None:
|
||||
"""The matcher's own pin — every other assertion here rests on it.
|
||||
|
||||
``SELECT … documents.content, documents.content_hash …`` is a content load
|
||||
because of the FIRST token; ``documents.content_hash`` alone is not. If
|
||||
this ever stops holding, the pre-/post-fix assertion above is meaningless
|
||||
in BOTH directions, so it is asserted directly rather than trusted.
|
||||
"""
|
||||
assert loads_content(
|
||||
"SELECT documents.id, documents.content, documents.content_hash\n"
|
||||
"FROM documents\nWHERE documents.id = %(pk_1)s"
|
||||
)
|
||||
assert loads_content(
|
||||
"SELECT documents.content AS documents_content\nFROM documents\n"
|
||||
"WHERE documents.id = %(pk_1)s"
|
||||
)
|
||||
assert not loads_content(
|
||||
"SELECT documents.id, documents.content_hash, documents.file_hash\n"
|
||||
"FROM documents\nWHERE documents.source = %(source_1)s"
|
||||
)
|
||||
assert not loads_content(
|
||||
"SELECT documents.summary, documents.is_image, documents.page_count\n"
|
||||
"FROM documents\nWHERE documents.id = %(pk_1)s"
|
||||
)
|
||||
# A predicate mention is not a hydration.
|
||||
assert not loads_content(
|
||||
"SELECT documents.id\nFROM documents\nWHERE documents.content = %(c_1)s"
|
||||
)
|
||||
|
||||
|
||||
def _predicate(statement: str) -> str:
|
||||
"""Everything after the first ``FROM`` — the statement's predicate."""
|
||||
from_at = _FROM_RE.search(statement)
|
||||
return statement[from_at.end():] if from_at else ""
|
||||
|
||||
|
||||
def row_loads(captured: list[Captured]) -> list[Captured]:
|
||||
"""The ``(source, path)`` row loads — the fast path's ONE lookup per file.
|
||||
|
||||
Matched on the PREDICATE, not the projection: the four sites of phase 185
|
||||
D1 (``_index_file`` plus the PDF / PDF-embedded-image / image branch loads)
|
||||
all compile to ``WHERE documents.source = … AND documents.path = …``. The
|
||||
two per-source post-pass walks filter on ``documents.is_image`` instead
|
||||
and are therefore excluded here — they are not the fast path's row load.
|
||||
"""
|
||||
return [
|
||||
c
|
||||
for c in doc_selects(captured)
|
||||
if "documents.source" in _predicate(c.text)
|
||||
and "documents.path" in _predicate(c.text)
|
||||
]
|
||||
|
||||
|
||||
def pk_followups(captured: list[Captured]) -> list[Captured]:
|
||||
"""Documents SELECTs keyed by PRIMARY KEY — the deferred-column shape.
|
||||
|
||||
A ``defer``-red column loads as ``SELECT documents.content … WHERE
|
||||
documents.id = …``: no ``source``/``path`` predicate at all. Pre-fix this
|
||||
list is EMPTY (the row was loaded whole, so nothing is deferred); post-fix
|
||||
it holds the ONE query the phase-118 backfill pays.
|
||||
"""
|
||||
return [
|
||||
c
|
||||
for c in doc_selects(captured)
|
||||
if "documents.source" not in _predicate(c.text)
|
||||
and "documents.path" not in _predicate(c.text)
|
||||
]
|
||||
|
||||
|
||||
def _summary_chunks(db: Session, path: str) -> list[Chunk]:
|
||||
"""The ``is_summary`` chunks of one document, read by ``document_id`` so
|
||||
the parent row's ``content`` is never what this probe reads."""
|
||||
return list(
|
||||
db.scalars(
|
||||
select(Chunk)
|
||||
.where(
|
||||
Chunk.document_id == _doc_id(db, path),
|
||||
Chunk.is_summary.is_(True),
|
||||
)
|
||||
.order_by(Chunk.id)
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The probe
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unchanged_run_defers_content_post_fix(
|
||||
db: Session, tmp_path: Path, statement_capture: list[Captured]
|
||||
) -> None:
|
||||
"""THE PROBE (post-fix): no file the unchanged walk inspects SELECTs its
|
||||
markdown.
|
||||
|
||||
Four unchanged files → four ``(source, path)`` row loads, and NONE of them
|
||||
names ``documents.content`` in its projection — the fast path reads row
|
||||
metadata only (phase 185, ``_doc_row_stmt``). The only follow-up the run
|
||||
may pay is the backfill row's, attributed per primary key in
|
||||
:func:`test_summary_null_backfill_reads_stored_content`, while
|
||||
:func:`test_unchanged_file_is_one_select` pins that every
|
||||
summary-bearing file pays none. The rest of the run's documents traffic
|
||||
is the two per-source image post-passes, pinned apart so a future
|
||||
per-file extra SELECT cannot hide inside the total.
|
||||
|
||||
(Task 01's PRE-fix assertion was the inverse — ``all(loads_content(c))``
|
||||
with zero follow-ups — and flipped here exactly as planned. A deferred
|
||||
column that is genuinely READ still shows up, as a PK follow-up, so
|
||||
neither a re-appeared ``content`` nor a new lazy reader hides here.)
|
||||
"""
|
||||
run = _two_runs(db, tmp_path, statement_capture)
|
||||
|
||||
selects = doc_selects(run.captured)
|
||||
loads = row_loads(run.captured)
|
||||
assert len(loads) == FILE_COUNT, (
|
||||
f"a {FILE_COUNT}-file unchanged walk issued {len(loads)} (source, "
|
||||
"path) row loads — the fast path looks each file up exactly once"
|
||||
)
|
||||
assert len(selects) >= FILE_COUNT
|
||||
|
||||
# Post-fix (phase 185 task 02): no content in the fast-path SELECT.
|
||||
assert not any(loads_content(c) for c in loads), (
|
||||
"a fast-path row load SELECTs documents.content again — the deferral "
|
||||
"in _doc_row_stmt is gone or bypassed, which re-hydrates the corpus "
|
||||
"(252.6 MB on the incident box) for a walk that reads metadata only"
|
||||
)
|
||||
followups = pk_followups(run.captured)
|
||||
assert len(followups) == EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL, (
|
||||
f"an unchanged run over {FILE_COUNT} files (one with summary IS NULL) "
|
||||
f"paid {len(followups)} primary-key follow-up SELECTs over documents "
|
||||
f"while EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL is "
|
||||
f"{EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL} — the backfill's single "
|
||||
"deferred-column read is the only one the fast path may need"
|
||||
)
|
||||
post_passes = [c for c in selects if c not in loads and c not in followups]
|
||||
assert len(post_passes) == POST_PASS_DOC_SELECTS, (
|
||||
f"the unchanged run did {len(post_passes)} documents SELECTs beyond "
|
||||
f"the {FILE_COUNT} row loads — expected the {POST_PASS_DOC_SELECTS} "
|
||||
"per-source image post-passes and nothing else"
|
||||
)
|
||||
|
||||
|
||||
def test_unchanged_file_is_one_select(
|
||||
db: Session, tmp_path: Path, statement_capture: list[Captured]
|
||||
) -> None:
|
||||
"""One documents SELECT per unchanged file — MUST STAY GREEN POST-FIX.
|
||||
|
||||
A file WITH a stored summary reads nothing beyond what its row load
|
||||
already carries: the ``file_hash`` comparison, then ``_unchanged_doc``'s
|
||||
``summary`` / ``created_at`` / ``page_count`` bookkeeping. Pre-fix that is
|
||||
one whole-entity load; post-fix it must still be exactly ONE — this is the
|
||||
guard that ``defer(Document.content)`` does not decay into a lazy-load
|
||||
storm (``summary``, then ``created_at``, then ``page_count`` … one SELECT
|
||||
apiece), which would trade 15 KB of residency for five round trips.
|
||||
"""
|
||||
run = _two_runs(db, tmp_path, statement_capture)
|
||||
|
||||
for n in range(FILE_COUNT):
|
||||
path = f"probe-{n:02d}.md"
|
||||
if path == BACKFILL_PATH:
|
||||
continue # asserted separately, with its documented extra
|
||||
one_file = [c for c in row_loads(run.captured) if c.params.get("path_1") == path]
|
||||
assert one_file, (
|
||||
f"{path} was walked with no (source, path) row load carrying "
|
||||
"path_1 — the row-load statement shape changed, so this probe is "
|
||||
"attributing nothing"
|
||||
)
|
||||
assert len(one_file) == 1, (
|
||||
f"{path} cost {len(one_file)} documents row loads — the fast path "
|
||||
"looks a file up ONCE"
|
||||
)
|
||||
doc_id = str(_doc_id(db, path))
|
||||
followups = [
|
||||
c for c in pk_followups(run.captured) if str(c.params.get("pk_1")) == doc_id
|
||||
]
|
||||
assert followups == [], (
|
||||
f"{path} paid {len(followups)} deferred-column follow-up SELECT(s) "
|
||||
"despite having a stored summary — nothing on its fast path may "
|
||||
"read content, so a follow-up means a new reader crept in"
|
||||
)
|
||||
|
||||
|
||||
def test_summary_null_backfill_reads_stored_content(
|
||||
db: Session, tmp_path: Path, statement_capture: list[Captured]
|
||||
) -> None:
|
||||
"""The ONE branch that genuinely needs the markdown: ``summary IS NULL``.
|
||||
|
||||
``_unchanged_doc`` step 3 (phase 118, A2) backfills the summary FROM THE
|
||||
STORED ``doc.content`` — never by re-reading the file, re-extracting or
|
||||
re-describing. So the probe pins the behaviour AND its cost:
|
||||
|
||||
* the run backfills exactly one summary (``summary_backfilled=1`` while
|
||||
``summaries`` stays ``0`` — a backfill is not a KB-change signal), the
|
||||
stored summary comes back and its single ``is_summary`` chunk is the
|
||||
REPLACED one (the pre-existing chunk still carried :data:`STALE_SUMMARY`
|
||||
when the run started, and replacement is the only way it changes);
|
||||
* the text the ``lite`` call was given IS the stored markdown (~8 KB, with
|
||||
the file's own sentinel in every section) — the bytes travelled from
|
||||
``documents.content``, not from a re-read of the file;
|
||||
* the documents SELECT count for that file is ``1 +
|
||||
EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL``: the row load plus, once task
|
||||
02 defers ``content``, exactly one lazy follow-up keyed by primary key
|
||||
(D2 — a per-row, one-time, already-rare cost, accepted precisely instead
|
||||
of a ``raiseload`` that would crash this branch where it should query).
|
||||
"""
|
||||
run = _two_runs(db, tmp_path, statement_capture)
|
||||
n = int(BACKFILL_PATH.removeprefix("probe-").removesuffix(".md"))
|
||||
expected = f"Summary of #\nSource: {PROBE_SOURCE}/{BACKFILL_PATH}"
|
||||
|
||||
# The backfill happened, from the stored content.
|
||||
assert run.summary.summary_backfilled == 1
|
||||
assert run.summary.summaries == 0
|
||||
stored = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == PROBE_SOURCE, Document.path == BACKFILL_PATH
|
||||
)
|
||||
)
|
||||
assert stored is not None and len(stored) >= 8_000
|
||||
assert f"fastpath-sentinel-{n}" in stored
|
||||
|
||||
doc = db.scalar(
|
||||
select(Document).where(
|
||||
Document.source == PROBE_SOURCE, Document.path == BACKFILL_PATH
|
||||
)
|
||||
)
|
||||
assert doc is not None
|
||||
assert doc.summary == expected
|
||||
assert doc.file_hash is not None
|
||||
|
||||
chunks = _summary_chunks(db, BACKFILL_PATH)
|
||||
assert [c.content for c in chunks] == [expected], (
|
||||
"the summary chunk was not replaced by the backfill — a stale chunk "
|
||||
"survived, or more than one is_summary chunk exists"
|
||||
)
|
||||
assert chunks[0].position == -1
|
||||
assert chunks[0].embedding is not None
|
||||
|
||||
# The lite call got the STORED markdown, sentinel and all — uncapped.
|
||||
assert run.lite_prompts, "the backfill made no lite-model call"
|
||||
assert any(stored == prompt for prompt in run.lite_prompts), (
|
||||
"the summary model was not fed documents.content verbatim — the "
|
||||
"phase-118 backfill reads something other than the stored markdown"
|
||||
)
|
||||
|
||||
# And the cost of that branch, as a documented constant.
|
||||
doc_id = str(_doc_id(db, BACKFILL_PATH))
|
||||
loads = [c for c in row_loads(run.captured) if c.params.get("path_1") == BACKFILL_PATH]
|
||||
followups = [c for c in pk_followups(run.captured) if str(c.params.get("pk_1")) == doc_id]
|
||||
assert len(loads) == 1
|
||||
assert len(followups) == EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL, (
|
||||
f"the summary-NULL backfill paid {len(followups)} lazy content "
|
||||
f"SELECT(s) while EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL is "
|
||||
f"{EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL} — 0 while the row load "
|
||||
"carries content (pre task 02), 1 once content is deferred"
|
||||
)
|
||||
assert all(loads_content(c) for c in followups)
|
||||
|
||||
|
||||
def test_unchanged_run_keeps_every_row_intact(
|
||||
db: Session, tmp_path: Path, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""The observable contract the deferral must not move (phase 185 D4).
|
||||
|
||||
The reference for task 02's byte-identity claim: the unchanged run's whole
|
||||
``ImportSummary`` shape and the whole KB after it — four unchanged files,
|
||||
one backfill, no re-embed, no date refresh, every file still holding its
|
||||
full ~8 KB of content, its summary and all of its chunks. ``changed_paths``
|
||||
stays empty (the backfill is not a document change, so the folder-summary
|
||||
regeneration never sees it) and no source is pruned.
|
||||
"""
|
||||
root = _seed_files(tmp_path)
|
||||
first = _run_import(root, _probe_llm())
|
||||
assert (first.files, first.added, first.summary_backfilled) == (
|
||||
FILE_COUNT,
|
||||
FILE_COUNT,
|
||||
0,
|
||||
)
|
||||
content_chunks_after_seed = db.scalar(
|
||||
select(func.count()).select_from(Chunk).where(Chunk.is_summary.is_(False))
|
||||
)
|
||||
assert content_chunks_after_seed is not None and content_chunks_after_seed > FILE_COUNT
|
||||
_stage_the_backfill(db, BACKFILL_PATH)
|
||||
|
||||
caplog.clear()
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
second = _run_import(root, _probe_llm())
|
||||
messages = [r.getMessage() for r in caplog.records if r.name == "app.importer"]
|
||||
|
||||
# The phase-147/118 log words, verbatim (D4: task 02 may change WHICH
|
||||
# COLUMNS travel, never what the run says it did). ``import: summary
|
||||
# source=`` is the FRESH-summary line — it must not appear on an unchanged
|
||||
# run, where only the backfill's own wording is allowed.
|
||||
assert sum(m.startswith("import: unchanged ") for m in messages) == FILE_COUNT
|
||||
assert sum(m.startswith("import: summary-backfill ") for m in messages) == 1
|
||||
assert not [m for m in messages if m.startswith("import: summary source=")]
|
||||
assert not [m for m in messages if m.startswith("import: date-refreshed ")]
|
||||
assert not [m for m in messages if m.startswith("import: added ")]
|
||||
assert not [m for m in messages if m.startswith("import: updated ")]
|
||||
|
||||
assert (second.files, second.added, second.updated) == (FILE_COUNT, 0, 0)
|
||||
assert (second.unchanged, second.pruned, second.errors) == (FILE_COUNT, 0, 0)
|
||||
assert second.summary_backfilled == 1
|
||||
assert second.summaries == 0
|
||||
assert second.summary_errors == 0
|
||||
assert second.dates_updated == 0
|
||||
assert second.chunks == 0
|
||||
assert second.changed_paths == []
|
||||
assert second.formats == {"md": FILE_COUNT}
|
||||
assert second.format_counts() == "md:4"
|
||||
|
||||
# Nothing was re-embedded, nothing was lost.
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Chunk).where(Chunk.is_summary.is_(False))
|
||||
) == content_chunks_after_seed
|
||||
assert db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Document)
|
||||
.where(Document.source == PROBE_SOURCE, Document.summary.is_not(None))
|
||||
) == FILE_COUNT
|
||||
assert all(
|
||||
len(c) >= 8_000
|
||||
for c in db.scalars(
|
||||
select(Document.content).where(Document.source == PROBE_SOURCE)
|
||||
)
|
||||
)
|
||||
# Exactly one summary chunk per document (the backfill replaced, not added).
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Chunk).where(Chunk.is_summary.is_(True))
|
||||
) == FILE_COUNT
|
||||
@@ -136,6 +136,7 @@ lazy SELECT per row and fail that test, not get papered over.
|
||||
``scripts/git_sync.py`` (phase 186), ``app/db.py:30``, and the
|
||||
``doc.chunks = [...]`` re-embed path (phase 183 LOCKED A2).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
@@ -189,9 +190,7 @@ EXPECTED_SUMMARY_CHUNKS = 12
|
||||
#: Every ORM object a full ADD run creates: the 12 documents, their content
|
||||
#: chunks and their summary chunks. What the identity map would have to
|
||||
#: hold if the ADDED path retained what the phase-184 baseline describes.
|
||||
OBJECTS_CREATED_BY_ADD_RUN = (
|
||||
FILE_COUNT + EXPECTED_CONTENT_CHUNKS + EXPECTED_SUMMARY_CHUNKS
|
||||
)
|
||||
OBJECTS_CREATED_BY_ADD_RUN = FILE_COUNT + EXPECTED_CONTENT_CHUNKS + EXPECTED_SUMMARY_CHUNKS
|
||||
|
||||
#: The measured number of identity-map entries at the end of an UNCHANGED
|
||||
#: run with the cadence OFF — one per file walked, each a full ``Document``
|
||||
@@ -322,9 +321,7 @@ def _kb_snapshot(db: Session) -> dict[str, tuple[str, str | None, str, int]]:
|
||||
}
|
||||
snapshot: dict[str, tuple[str, str | None, str, int]] = {}
|
||||
for doc in db.scalars(
|
||||
select(Document)
|
||||
.where(Document.source == PROBE_SOURCE)
|
||||
.order_by(Document.path)
|
||||
select(Document).where(Document.source == PROBE_SOURCE).order_by(Document.path)
|
||||
):
|
||||
assert doc.file_hash is not None # the phase-147 stamp is always set
|
||||
snapshot[doc.path] = (
|
||||
@@ -373,9 +370,7 @@ def test_the_added_run_holds_its_objects_only_until_a_collection(
|
||||
root = _seed_files(tmp_path)
|
||||
session = SessionLocal()
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources([root], _probe_llm(), prune=False, session=session)
|
||||
)
|
||||
summary = asyncio.run(import_sources([root], _probe_llm(), prune=False, session=session))
|
||||
assert (summary.files, summary.added, summary.unchanged) == (
|
||||
FILE_COUNT,
|
||||
FILE_COUNT,
|
||||
@@ -405,9 +400,7 @@ def test_the_added_run_holds_its_objects_only_until_a_collection(
|
||||
session.close()
|
||||
|
||||
|
||||
def test_the_unchanged_run_keeps_every_document_it_walks(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
def test_the_unchanged_run_keeps_every_document_it_walks(db: Session, tmp_path: Path) -> None:
|
||||
"""THE PROBE: a repeat sync holds one full ``Document`` per file walked.
|
||||
|
||||
The production-shaped case — the second sync of the same source, every
|
||||
@@ -446,9 +439,7 @@ def test_the_unchanged_run_keeps_every_document_it_walks(
|
||||
staircase.append(_map_size(session))
|
||||
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], _probe_llm(), prune=False, session=session, progress=progress
|
||||
)
|
||||
import_sources([root], _probe_llm(), prune=False, session=session, progress=progress)
|
||||
)
|
||||
assert (summary.files, summary.added, summary.unchanged) == (
|
||||
FILE_COUNT,
|
||||
@@ -459,8 +450,7 @@ def test_the_unchanged_run_keeps_every_document_it_walks(
|
||||
# One row resident per file already walked — a staircase, never a
|
||||
# plateau and never a release.
|
||||
assert staircase == list(range(FILE_COUNT)), (
|
||||
"expected exactly one document retained per file already walked,"
|
||||
f" got {staircase}"
|
||||
f"expected exactly one document retained per file already walked, got {staircase}"
|
||||
)
|
||||
assert _map_size(session) == MEASURED_UNCHANGED_MAP_SIZE
|
||||
|
||||
@@ -643,8 +633,7 @@ def test_reset_cadence_bounds_the_unchanged_walk(db: Session, tmp_path: Path) ->
|
||||
FILE_COUNT,
|
||||
)
|
||||
assert staircase == [n % RESET_FILES for n in range(FILE_COUNT)], (
|
||||
f"the bounded walk sampled {staircase} — expected a release every"
|
||||
f" {RESET_FILES} files"
|
||||
f"the bounded walk sampled {staircase} — expected a release every {RESET_FILES} files"
|
||||
)
|
||||
assert max(staircase) < MEASURED_UNCHANGED_MAP_SIZE
|
||||
assert _map_size(session) == 0, (
|
||||
@@ -682,14 +671,10 @@ def test_reset_cadence_survives_an_embedding_error(db: Session, tmp_path: Path)
|
||||
|
||||
session = SessionLocal()
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources([root], llm, prune=False, session=session)
|
||||
)
|
||||
summary = asyncio.run(import_sources([root], llm, prune=False, session=session))
|
||||
assert (summary.files, summary.errors) == (FILE_COUNT, 1)
|
||||
assert summary.added == FILE_COUNT - 1
|
||||
assert summary.chunks == EXPECTED_CONTENT_CHUNKS - (
|
||||
EXPECTED_CONTENT_CHUNKS // FILE_COUNT
|
||||
)
|
||||
assert summary.chunks == EXPECTED_CONTENT_CHUNKS - (EXPECTED_CONTENT_CHUNKS // FILE_COUNT)
|
||||
assert summary.summaries == EXPECTED_SUMMARY_CHUNKS - 1
|
||||
assert _map_size(session) <= MAP_BOUND
|
||||
finally:
|
||||
@@ -743,9 +728,7 @@ def test_a_clean_row_leaves_the_map_and_a_dirty_one_never_does(db: Session) -> N
|
||||
assert _map_size(session) == 0, "a clean committed row must be weakly held"
|
||||
|
||||
row = session.scalar(
|
||||
select(Document).where(
|
||||
Document.source == PROBE_SOURCE, Document.path == "mechanism.md"
|
||||
)
|
||||
select(Document).where(Document.source == PROBE_SOURCE, Document.path == "mechanism.md")
|
||||
)
|
||||
assert row is not None
|
||||
row.file_hash = row.file_hash # the stamp: same value, dirty state
|
||||
@@ -765,9 +748,7 @@ def test_a_clean_row_leaves_the_map_and_a_dirty_one_never_does(db: Session) -> N
|
||||
assert _map_size(session) == 0
|
||||
assert (
|
||||
db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Document)
|
||||
.where(Document.source == PROBE_SOURCE)
|
||||
select(func.count()).select_from(Document).where(Document.source == PROBE_SOURCE)
|
||||
)
|
||||
== 1
|
||||
)
|
||||
@@ -801,9 +782,7 @@ PRUNE_BATCH = 10
|
||||
#: The SIX columns the prune loads — the ``load_only`` set of ``_prune``
|
||||
#: (docstring: the light-row contract). An eighth name here means the walk
|
||||
#: got heavier; one missing means a guard lost the column it reads.
|
||||
PRUNE_COLUMNS = frozenset(
|
||||
{"id", "source", "path", "is_image", "image_path", "page_count"}
|
||||
)
|
||||
PRUNE_COLUMNS = frozenset({"id", "source", "path", "is_image", "image_path", "page_count"})
|
||||
|
||||
#: ``… FROM documents`` — the prune's own walk AND (that is the pin) any
|
||||
#: per-document follow-up lazy load of a deferred column. The word boundary
|
||||
@@ -862,9 +841,7 @@ def _projected_columns(statement: str) -> frozenset[str]:
|
||||
return frozenset(re.findall(r"documents\.\w+", head))
|
||||
|
||||
|
||||
def _seed_prune_docs(
|
||||
session: Session, *, page_dir: Path | None = None
|
||||
) -> list[UUID]:
|
||||
def _seed_prune_docs(session: Session, *, page_dir: Path | None = None) -> list[UUID]:
|
||||
"""``PRUNE_DOC_COUNT`` indexed docs of :data:`PRUNE_SOURCE`, each with
|
||||
``PRUNE_CHUNKS_PER_DOC`` embedded chunks and a full-fat ``content``.
|
||||
|
||||
@@ -924,9 +901,7 @@ def _arm_commit_counter(session: Session) -> list[int]:
|
||||
return commits
|
||||
|
||||
|
||||
def test_prune_holds_no_content(
|
||||
db: Session, tmp_path: Path, statement_capture: list[str]
|
||||
) -> None:
|
||||
def test_prune_holds_no_content(db: Session, tmp_path: Path, statement_capture: list[str]) -> None:
|
||||
"""THE PRUNE PROBE: the walk loads SIX columns and never ``content``.
|
||||
|
||||
30 documents × 5 chunks are pruned through the importer's own ``_prune``
|
||||
@@ -971,10 +946,7 @@ def test_prune_holds_no_content(
|
||||
"prune — a deferred column is being read per row, so the light row "
|
||||
"is incomplete (add the column to load_only)"
|
||||
)
|
||||
projected = {
|
||||
col.removeprefix("documents.")
|
||||
for col in _projected_columns(selects[0])
|
||||
}
|
||||
projected = {col.removeprefix("documents.") for col in _projected_columns(selects[0])}
|
||||
assert projected == PRUNE_COLUMNS, (
|
||||
f"the prune's row is not the six-column light row: {sorted(projected)}"
|
||||
)
|
||||
@@ -985,12 +957,11 @@ def test_prune_holds_no_content(
|
||||
|
||||
# Everything really went, children included (no orphans — phase 183 D4,
|
||||
# asserted here too so a light row cannot quietly change it).
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
|
||||
) == 0
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Chunk).where(Chunk.document_id.in_(ids))
|
||||
) == 0
|
||||
assert (
|
||||
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
|
||||
== 0
|
||||
)
|
||||
assert db.scalar(select(func.count()).select_from(Chunk).where(Chunk.document_id.in_(ids))) == 0
|
||||
assert sorted(summary.changed_paths) == sorted(
|
||||
(PRUNE_SOURCE, f"docs/doc-{n:04d}.md") for n in range(PRUNE_DOC_COUNT)
|
||||
)
|
||||
@@ -1001,9 +972,7 @@ def test_prune_holds_no_content(
|
||||
assert _map_classes(db) == {}
|
||||
|
||||
|
||||
def test_prune_without_sources_touches_nothing(
|
||||
db: Session, statement_capture: list[str]
|
||||
) -> None:
|
||||
def test_prune_without_sources_touches_nothing(db: Session, statement_capture: list[str]) -> None:
|
||||
"""The empty-source guard is a real short-circuit, not a walk for naught.
|
||||
|
||||
``_prune`` with no source names returns ``0`` before the SELECT — so a
|
||||
@@ -1017,9 +986,10 @@ def test_prune_without_sources_touches_nothing(
|
||||
assert _prune(db, set(), set(), summary=summary, prune_batch=10) == 0
|
||||
assert statement_capture == []
|
||||
assert summary.changed_paths == []
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
|
||||
) == PRUNE_DOC_COUNT
|
||||
assert (
|
||||
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
|
||||
== PRUNE_DOC_COUNT
|
||||
)
|
||||
|
||||
|
||||
def test_prune_batch_commits(db: Session) -> None:
|
||||
@@ -1059,11 +1029,10 @@ def test_prune_batch_commits(db: Session) -> None:
|
||||
session.close()
|
||||
# The FK cascade did the child work in EVERY transaction the
|
||||
# prune opened, not just the last one.
|
||||
assert db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.where(Chunk.document_id.in_(ids))
|
||||
) == 0
|
||||
assert (
|
||||
db.scalar(select(func.count()).select_from(Chunk).where(Chunk.document_id.in_(ids)))
|
||||
== 0
|
||||
)
|
||||
|
||||
pruned, summary, commits = run(PRUNE_BATCH)
|
||||
assert pruned == PRUNE_DOC_COUNT
|
||||
@@ -1074,9 +1043,10 @@ def test_prune_batch_commits(db: Session) -> None:
|
||||
assert sorted(summary.changed_paths) == sorted(
|
||||
(PRUNE_SOURCE, f"docs/doc-{n:04d}.md") for n in range(PRUNE_DOC_COUNT)
|
||||
)
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
|
||||
) == 0
|
||||
assert (
|
||||
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
|
||||
== 0
|
||||
)
|
||||
|
||||
# …and the pre-phase run does the SAME work in ONE transaction.
|
||||
_truncate(db)
|
||||
@@ -1087,9 +1057,10 @@ def test_prune_batch_commits(db: Session) -> None:
|
||||
"reproduce the pre-phase single transaction exactly"
|
||||
)
|
||||
assert sorted(summary_0.changed_paths) == sorted(summary.changed_paths)
|
||||
assert db.scalar(
|
||||
select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE)
|
||||
) == 0
|
||||
assert (
|
||||
db.scalar(select(func.count()).select_from(Document).where(Document.source == PRUNE_SOURCE))
|
||||
== 0
|
||||
)
|
||||
|
||||
|
||||
def test_prune_batch_setting_reaches_the_prune(
|
||||
@@ -1137,9 +1108,7 @@ def test_prune_batch_setting_reaches_the_prune(
|
||||
monkeypatch.setattr(importer, "_prune", spy)
|
||||
second = asyncio.run(import_sources([root], llm, prune=True, session=db))
|
||||
|
||||
assert seen_kwargs == [3], (
|
||||
"import_sources did not pass Settings.sync_prune_batch to _prune"
|
||||
)
|
||||
assert seen_kwargs == [3], "import_sources did not pass Settings.sync_prune_batch to _prune"
|
||||
assert (second.files, second.pruned, second.errors) == (1, 3, 0)
|
||||
assert sorted(path for (_src, path) in second.changed_paths) == [
|
||||
"keep-0.md",
|
||||
@@ -1152,16 +1121,99 @@ def test_prune_batch_setting_reaches_the_prune(
|
||||
select(func.count()).select_from(Chunk).where(Chunk.embedding.is_not(None))
|
||||
)
|
||||
assert embedded is not None and embedded > 0
|
||||
assert db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.where(Chunk.document_id.not_in(select(Document.id)))
|
||||
) == 0
|
||||
assert (
|
||||
db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.where(Chunk.document_id.not_in(select(Document.id)))
|
||||
)
|
||||
== 0
|
||||
)
|
||||
|
||||
|
||||
def test_prune_guards_unaffected_with_light_rows(
|
||||
db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
def test_the_summary_backfill_survives_the_reset_cadence(db: Session, tmp_path: Path) -> None:
|
||||
"""The constraint that fixes WHERE the cadence may sit, in ONE file.
|
||||
|
||||
``_unchanged_doc`` (phase 118) backfills a NULL summary by calling
|
||||
:func:`app.rag.summarizer.summarize_document`, which reads
|
||||
``doc.content`` AFTER the row was loaded and its ``file_hash`` stamped.
|
||||
That local is live across statements: a
|
||||
``session.expunge_all()`` moved into ``_unchanged_doc`` — or a
|
||||
``noload``/``raiseload`` added to the row it walks — would discard or
|
||||
break the very row the backfill reads. Phase 185 task 01 chose
|
||||
``defer`` over ``noload``/``raiseload`` (LOCKED A2) for the same reason,
|
||||
and task 05 of THIS phase pinned the choice on the statement itself
|
||||
(``tests/unit/test_importer_defer_content.py``); this pins the SAME
|
||||
constraint on the run.
|
||||
|
||||
The fixture is a document whose ``is_summary`` chunk is missing while
|
||||
its ``file_hash`` matches (a re-sync with a NULL summary — exactly the
|
||||
row the backfill exists for), imported at the tightest cadence the
|
||||
setting allows that still resets inside a 12-file walk. Three
|
||||
observables, none of them a memory proxy:
|
||||
|
||||
* the backfill RUNS — one and only one ``is_summary`` chunk appears,
|
||||
carrying the fake-embedder's summary text;
|
||||
* every other file is still ``unchanged`` and the summary count is
|
||||
exactly the file count afterwards, i.e. the reset cadence destroyed
|
||||
no summary on the way through;
|
||||
* the doc's own text is still intact afterwards (``content`` was never
|
||||
rewritten by a re-index).
|
||||
"""
|
||||
root = _seed_files(tmp_path)
|
||||
seed = SessionLocal()
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], _probe_llm(), session=seed))
|
||||
assert first.added == FILE_COUNT
|
||||
finally:
|
||||
seed.close()
|
||||
|
||||
# Hand the next sync ONE document with a NULL summary: it takes
|
||||
# ``_unchanged_doc``'s backfill branch instead of the early return.
|
||||
probe_rel = "probe-00.md"
|
||||
before = _kb_snapshot(db)[probe_rel]
|
||||
target = db.execute(
|
||||
select(Document).where(Document.source == PROBE_SOURCE, Document.path == probe_rel)
|
||||
).scalar_one()
|
||||
db.query(Chunk).filter(Chunk.document_id == target.id, Chunk.is_summary.is_(True)).delete()
|
||||
target.summary = None
|
||||
db.commit()
|
||||
db.expunge_all()
|
||||
assert _kb_snapshot(db)[probe_rel][1] is None
|
||||
|
||||
session = SessionLocal()
|
||||
try:
|
||||
summary = asyncio.run(
|
||||
import_sources(
|
||||
[root], _probe_llm(reset_files=RESET_FILES), prune=False, session=session
|
||||
)
|
||||
)
|
||||
finally:
|
||||
session.close()
|
||||
# Everything stayed unchanged — the backfill never re-indexed.
|
||||
assert (summary.files, summary.added, summary.updated, summary.unchanged) == (
|
||||
FILE_COUNT,
|
||||
0,
|
||||
0,
|
||||
FILE_COUNT,
|
||||
)
|
||||
assert (summary.errors, summary.summary_backfilled) == (0, 1)
|
||||
|
||||
snap = _kb_snapshot(db)
|
||||
backfilled = snap[probe_rel]
|
||||
assert backfilled[1] is not None, "the summary-NULL backfill did not run"
|
||||
# The one doc that needed the deferred read still has its OWN text and
|
||||
# its own hash: a summary was added, the document was not rewritten.
|
||||
assert backfilled[0] == before[0]
|
||||
assert backfilled[2] == before[2]
|
||||
# And the cadence destroyed nothing on its way through: every file is
|
||||
# back to 5 content chunks + 1 summary chunk.
|
||||
assert {p: c for p, (_, _, _, c) in snap.items()} == {
|
||||
f"probe-{n:02d}.md": 6 for n in range(FILE_COUNT)
|
||||
}
|
||||
|
||||
|
||||
def test_prune_guards_unaffected_with_light_rows(db: Session, tmp_path: Path) -> None:
|
||||
"""The guards run off SIX columns — the light row is COMPLETE.
|
||||
|
||||
A column missing from ``load_only`` does not announce itself: the guard
|
||||
@@ -1216,9 +1268,7 @@ def test_prune_guards_unaffected_with_light_rows(
|
||||
seen = {(PRUNE_SOURCE, "notes/kept.md"), (PRUNE_SOURCE, "deck.pdf")}
|
||||
session = SessionLocal()
|
||||
try:
|
||||
pruned_toggle_off = _prune(
|
||||
session, {PRUNE_SOURCE}, seen, images=False, prune_batch=0
|
||||
)
|
||||
pruned_toggle_off = _prune(session, {PRUNE_SOURCE}, seen, images=False, prune_batch=0)
|
||||
finally:
|
||||
session.close()
|
||||
# Toggle OFF: BOTH image docs are invisible to the walk, not deleted —
|
||||
@@ -1228,9 +1278,7 @@ def test_prune_guards_unaffected_with_light_rows(
|
||||
|
||||
session = SessionLocal()
|
||||
try:
|
||||
pruned_toggle_on = _prune(
|
||||
session, {PRUNE_SOURCE}, seen, images=True, prune_batch=0
|
||||
)
|
||||
pruned_toggle_on = _prune(session, {PRUNE_SOURCE}, seen, images=True, prune_batch=0)
|
||||
finally:
|
||||
session.close()
|
||||
# Toggle ON: the deleted image file prunes its row AND its copy (the
|
||||
@@ -1240,8 +1288,82 @@ def test_prune_guards_unaffected_with_light_rows(
|
||||
assert not gone_copy.exists()
|
||||
assert kept_copy.exists()
|
||||
assert sorted(
|
||||
path
|
||||
for (path,) in db.execute(
|
||||
select(Document.path).where(Document.source == PRUNE_SOURCE)
|
||||
)
|
||||
path for (path,) in db.execute(select(Document.path).where(Document.source == PRUNE_SOURCE))
|
||||
) == ["deck.pdf", "deck.pdf#page=1#img=0.png", "notes/kept.md"]
|
||||
|
||||
|
||||
def test_both_prune_guards_cost_the_same_statement_budget(
|
||||
db: Session, tmp_path: Path, statement_capture: list[str]
|
||||
) -> None:
|
||||
"""The guards add NO hidden read — proved by statement count, both ways.
|
||||
|
||||
:func:`test_prune_guards_unaffected_with_light_rows` above proves the
|
||||
guards DECIDE correctly off the six light columns. This pins the other
|
||||
half of the same contract, the half that is invisible in behaviour: a
|
||||
guard reaching for a deferred attribute would still decide correctly and
|
||||
only show up as ONE MORE SELECT PER ROW. So the assertion here is a
|
||||
statement count, not an outcome:
|
||||
|
||||
* the SAME projection runs under both toggles — one candidate walk
|
||||
apiece, naming the same six columns (images OFF only skips rows
|
||||
through ``is_image``, a column the walk already reads);
|
||||
* neither walk projects ``documents.content``;
|
||||
* deleting an image row + copying its file costs no extra document
|
||||
SELECT (the ``image_path`` the copy cleanup uses is in the projection).
|
||||
|
||||
Total: exactly TWO document SELECTs for the two syncs, whatever the
|
||||
guards do.
|
||||
"""
|
||||
image_dir = tmp_path / "image_dir"
|
||||
image_dir.mkdir()
|
||||
gone_copy = image_dir / "gone.png"
|
||||
gone_copy.write_bytes(b"\x89PNG\r\n\x1a\n gone")
|
||||
|
||||
def add(path: str, *, is_image: bool, copy: Path | None = None) -> None:
|
||||
db.add(
|
||||
Document(
|
||||
source=PRUNE_SOURCE,
|
||||
path=path,
|
||||
full_path=f"/srv/brain/{PRUNE_SOURCE}/{path}",
|
||||
title=f"Budget probe {path}",
|
||||
content=PRUNE_CONTENT,
|
||||
content_hash=f"{abs(hash(path)):064x}"[:64],
|
||||
is_image=is_image,
|
||||
image_path=str(copy) if copy else None,
|
||||
)
|
||||
)
|
||||
|
||||
add("notes/kept.md", is_image=False)
|
||||
add("notes/gone.md", is_image=False)
|
||||
add("img/gone.png", is_image=True, copy=gone_copy)
|
||||
add("deck.pdf#page=1#img=0.png", is_image=True)
|
||||
db.commit()
|
||||
|
||||
seen = {(PRUNE_SOURCE, "notes/kept.md")}
|
||||
projections: list[frozenset[str]] = []
|
||||
for toggle in (False, True):
|
||||
since = len(statement_capture)
|
||||
session = SessionLocal()
|
||||
try:
|
||||
_prune(session, {PRUNE_SOURCE}, seen, images=toggle, prune_batch=0)
|
||||
finally:
|
||||
session.close()
|
||||
# Only what THIS sync emitted — the capture is cumulative.
|
||||
walks = [
|
||||
s for s in _doc_selects(statement_capture[since:]) if "WHERE documents.source" in s
|
||||
]
|
||||
assert len(walks) == 1, (
|
||||
f"the images={toggle} prune ran {len(walks)} candidate walks; "
|
||||
"one per sync is the budget (a second is a per-row lazy load)"
|
||||
)
|
||||
projections.append(_projected_columns(walks[0]))
|
||||
|
||||
assert projections[0] == projections[1], (
|
||||
"the two toggles read DIFFERENT columns — a guard grew a deferred "
|
||||
f"read:\n off: {sorted(projections[0])}\n on: {sorted(projections[1])}"
|
||||
)
|
||||
assert "documents.content" not in projections[0]
|
||||
assert len(_doc_selects(statement_capture)) == 2, (
|
||||
"two syncs must cost exactly two document SELECTs in total, guards "
|
||||
"included — any extra is a per-row read the light row must cover"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user