phase: 127_sync_incremental_folder_summaries
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Build and Push Containers / build-and-push-db (push) Successful in 15s

All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.

**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)

- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
2026-09-27 21:12:25 -04:00
parent f50d4c8bc3
commit 6e654df387
38 changed files with 2741 additions and 46 deletions
@@ -0,0 +1,619 @@
"""Phase 127 task 04 E2E (Playwright, mock-only): a one-file change
rewrites ONLY that folder's summary chain.
The dedicated story suite for ``127_sync_incremental_folder_summaries``
(A16 — one Playwright file per phase, run in isolation): the owner's
report was that "on sync brain of reese rewrites all the summaries for
every directory — so even if just one file changed this can result in
700+ directory summary rewrites in production which can take a very
long time" (``TODO.md`` L3). The fix under test: a KB-changing sync
passes the import's changed-document set
(``ImportSummary.changed_paths``) into
``generate_folder_summaries(changed_docs=…)``, so ONLY the folders a
changed document touches — each document's folder chain up to its
source root (``affected_folder_keys``) — are re-stamped; every other
stored row keeps its summary text AND ``updated_at`` byte-identical.
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
deterministic mock (the phase-94 ``FOLDER_SUMMARY_MODE`` canned
``Fixture folder summary for <folder>.`` lines): the stored rows are a
pure function of the request, so the byte-identical assertions are
exact only against the mock.
No slow-LLM proxy: the assertions are on the POST-terminal DB state
(the rows' ``summary`` AND ``updated_at``) plus the terminal status
(the keep-final-counts convention — the terminal keeps
``summaries_done == summaries_total == <affected count>``), never on
in-flight sampling, so the fast mock is dense enough by construction.
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
the same host) registered as a ``kind=local`` source (the
``test_local_directory_sources.py`` / phase-94 API-registration
pattern; no git anywhere): ``incsum/`` with FOUR ≥ 2-doc folders —
``alpha/``, ``bravo/``, ``charlie/``, ``delta/`` (2 docs each) — plus
ONE root-level doc (9 documents total). The generator's candidate set
is therefore exactly 5: the source root (``folder_path ""``) + the
four folders (the recursive-subtree ≥ 1 rule).
Autouse cleanup (the phase-96 pattern): before each test wait for no
running sync on the module app, then truncate the shared registry + KB
tables (documents/chunks/query_log/steering_notes/kb_overview/
git_sources/folder_summaries) — each test seeds its own source and
runs its own sync(s); after each test the same truncate. (The temp
tree itself persists for the module; the tests' disk mutations are
ordered by test definition and each test re-syncs the state it finds.)
Test → observable mapping (Playwright Mapping Rule):
1. ``test_first_sync_generates_all_rows`` — a first (full) sync settles
``success``; the ``folder_summaries`` table holds EXACTLY the 5
rows (source root + the four folders) with the mock's canned text
naming each folder, each ``updated_at`` non-null, and the terminal
keeps ``summaries_done == summaries_total == 5`` (the full
regeneration — a changed KB with no prior rows, the pre-127
behavior preserved end to end).
2. ``test_one_file_change_restamps_only_that_folders_chain`` — re-seed
(autouse truncated) + sync #1, capture every row's
``(summary, updated_at)``; change ONE doc inside ``alpha/`` and
sync #2 settles ``success``: the ``alpha`` AND the source-root rows
carry a NEW ``updated_at`` (strictly after the captured stamp) with
the canned text, while the ``bravo``/``charlie``/``delta`` rows are
BYTE-IDENTICAL (same ``summary`` AND same ``updated_at``); the
terminal keeps ``summaries_done == summaries_total == 2`` (the
AFFECTED count — alpha + root, NOT the table's candidate count —
the keep-final-counts convention) and the detail shows the
single-file change (1 updated, 8 unchanged, 0 added/pruned).
3. ``test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_
folder`` — re-seed + sync #1, capture; delete ``bravo/``'s last
docs (the folder VANISHES — 0 documents, no longer a candidate) and
change one doc in ``charlie/``; the sync settles ``success``: the
``bravo`` row is PRUNED (gone — the prune pass owns vanished
folders' rows), the source-root AND ``charlie`` rows are re-stamped
(new ``updated_at`` + canned text), the ``alpha``/``delta`` rows
are byte-identical, and the terminal keeps
``summaries_done == summaries_total == 2`` (the affected CANDIDATE
count — charlie + root; bravo is no longer a candidate).
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import time
from collections.abc import Iterator
from datetime import datetime
from pathlib import Path
from typing import Any
import httpx
import pytest
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import FolderSummary
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module's app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_INCSUM", "8147"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
# --------------------------------------------------------------------------
# Fixture constants (deterministic, token-controlled)
# --------------------------------------------------------------------------
#: The local source — the temp directory's basename (``kind=local`` →
#: the directory's basename is the source name, phase 38).
SOURCE = "incsum"
#: The four ≥ 2-doc folders (path order).
FOLDER_A = "alpha"
FOLDER_B = "bravo"
FOLDER_C = "charlie"
FOLDER_D = "delta"
FOLDERS = (FOLDER_A, FOLDER_B, FOLDER_C, FOLDER_D)
ROOT_NOTE = "root-note.md" # the one root-level doc
N_FILES = 9 # four folders × 2 docs + the root-level doc
N_CANDIDATES = 5 # the source root + the four folders (the ≥ 1-doc rule)
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture
#: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX``
#: tail: ``<source>`` for the root, ``<source>/<folder>`` for a folder).
SUMMARY_FOR = "Fixture folder summary for {}."
EXPECTED: dict[str, str] = {
"": SUMMARY_FOR.format(SOURCE),
FOLDER_A: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_A}"),
FOLDER_B: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_B}"),
FOLDER_C: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_C}"),
FOLDER_D: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_D}"),
}
#: The single-file change (test 2) — a revised body whose hash differs
#: from the fixture's (the importer's sha256 delta reports ``updated``)
#: and which no other fixture text contains.
ALPHA_A_REVISED = (
"Incsum alpha fixture note A, REVISED: the only file test 2 changed "
"— the incremental regeneration must touch exactly this file's chain "
"(alpha + the source root), and nothing else. (RESE- INCSUM-01)"
)
#: The charlie change (test 3) — the same shape on the other side.
CHARLIE_A_REVISED = (
"Incsum charlie fixture note A, REVISED: the one changed file of "
"test 3's sync, while bravo vanishes entirely. (RESE- INCSUM-02)"
)
def _md(title: str, body: str) -> str:
return f"# {title}\n\n{body}\n"
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def incsum_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The one-source temp tree (see the module docstring): the app
server runs on the same host, so the paths are visible to it. The
directory NAME is the source name (``kind=local``, phase 38)."""
root = tmp_path_factory.mktemp("bor_incsum")
src = root / SOURCE
for folder in FOLDERS:
(src / folder).mkdir(parents=True)
for letter in ("a", "b"):
(src / folder / f"{folder}-{letter}.md").write_text(
_md(
f"Incsum {folder.title()} {letter.upper()}",
f"Incsum {folder} fixture note {letter.upper()}: covers "
f"topic {letter.upper()} of the {SOURCE} source tree.",
),
encoding="utf-8",
)
(src / ROOT_NOTE).write_text(
_md(
"Incsum Root Note",
f"This file sits directly under the {SOURCE} source, not in "
"any folder.",
),
encoding="utf-8",
)
assert len(list(src.rglob("*.md"))) == N_FILES
return src
def _app_env(
base_url: str, tmp_path_factory: pytest.TempPathFactory
) -> dict[str, str]:
"""The per-module app env (the conftest pattern, cf.
``test_kb_tree.py`` / ``test_sync_summary_visibility.py``): the
``base_url`` is the mock LLM directly (the fast leg is the only
leg — see the module docstring), ``BOR_GIT_SOURCES`` forced empty
(the sync source is this suite's own DB-registered local row), and
the leak-guarded code defaults."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
)
# Mock-calibrated threshold (conftest pattern): this suite never
# asks the chat model anything — the gate is never on a path.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers.
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
return env
@pytest.fixture(scope="module")
def app_server(
mock_llm: int,
incsum_dir: Path,
tmp_path_factory: pytest.TempPathFactory,
) -> Iterator[str]:
"""The app under test (all three tests): ``BOR_LLM_BASE_URL`` is
the mock directly — the canned ``FOLDER_SUMMARY_MODE`` lines are
byte-stable there, and the byte-identity assertions key on that
exact text (no slow proxy: the assertions are post-terminal, see
the module docstring). ``incsum_dir`` is a dependency only for the
fixture ordering (the temp tree exists before the app boots — the
sync reads it)."""
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the candidate count and the counts the syncs and the
folder-rows assertions pin exactly."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
"""No background sync may leak across tests (the run state lives in
the app's memory, and a still-running run would keep importing into
the NEXT test's truncated KB): wait for the app's sync status to
be non-running BEFORE the truncate. Own admin session (the
endpoint is admin-only) — usually a no-op: the tests settle only
after their run's terminal state."""
with httpx.Client(base_url=base_url, timeout=5.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
if r.json()["state"] != "running":
return
time.sleep(0.2)
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
@pytest.fixture(autouse=True)
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
"""Per-test KB isolation (the phase-96 cleanup pattern): the
shared E2E Postgres is truncated before AND after every test —
each test seeds its own source and runs its own sync(s), so the
KB rows + folder_summaries rows of the temp source are exactly
this test's own. The pre-wait keeps a leaked run from the
previous test (the module app is shared) from racing the
truncate."""
_wait_sync_settled(app_url)
_truncate_all()
yield
_truncate_all()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _register_source(base_url: str, path: str) -> None:
"""Register the temp directory through the authenticated API (the
``test_local_directory_sources.py`` / phase-94 pattern — a plain
directory is an API/DB-only operation)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": path}
)
assert r.status_code == 201, r.text
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
run reaches a terminal state (the ``test_local_directory_sources``
pattern, over plain httpx — with the stale-terminal freshness gate:
the module app is shared, so the PREVIOUS run's terminal lives in
the app's memory until this run supersedes it; a terminal is
accepted only after a running tick was observed or its
``started_at`` post-dates this POST)."""
with httpx.Client(base_url=base_url, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
t0 = time.time()
r = client.post("/api/sync")
assert r.status_code == 202, r.text
saw_running = False
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
state = body["state"]
if state == "running":
saw_running = True
elif state in ("success", "failed") and (
saw_running
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
):
return body
time.sleep(0.1)
raise AssertionError(f"sync did not reach a terminal state: {body}")
def _started_at(body: dict[str, Any]) -> float:
"""The status body's ``started_at`` as a wall-clock epoch (the
server stamps it in UTC; this process runs on the same host —
same clock, the phase-64 recorder's freshness convention)."""
started = body.get("started_at")
if not started:
return 0.0
try:
return datetime.fromisoformat(str(started)).timestamp()
except ValueError:
return 0.0
def _folder_rows() -> dict[str, tuple[str, bool, datetime]]:
"""The source's stored folder summaries
``{folder_path: (summary, manually_edited, updated_at)}``
(``""`` = the source root) — the test process's direct DB access
(the E2E's other established lens)."""
with SessionLocal() as db:
rows = db.execute(
select(
FolderSummary.folder_path,
FolderSummary.summary,
FolderSummary.manually_edited,
FolderSummary.updated_at,
).where(FolderSummary.source == SOURCE)
).all()
return {
folder: (summary, manually, updated)
for folder, summary, manually, updated in rows
}
# ---------------------------------------------------------------------------
# 1. The first sync (a changed KB with no prior rows) is the FULL
# regeneration: exactly the 5 candidate rows, canned text, the
# terminal keeps the full candidate count
# ---------------------------------------------------------------------------
def test_first_sync_generates_all_rows(
app_url: str, incsum_dir: Path, db_ready: None
) -> None:
"""A first sync of the temp source settles ``success`` with all 9
docs added; the ``folder_summaries`` table holds EXACTLY the 5
candidate rows (the source root + the four folders — the
recursive-subtree ≥ 1 rule) with the mock's canned text naming
each folder, every row AI-written and freshly stamped, and the
terminal keeps ``summaries_done == summaries_total == 5`` (the
full-regeneration count — a first sync passes no prior rows, so
this is the pre-127 behavior preserved end to end)."""
_register_source(app_url, str(incsum_dir))
body = _run_sync(app_url)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == N_FILES, detail
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail
# Terminal (the keep-final-counts convention): the FULL candidate
# count — the pre-127 shape a first sync still reports.
assert body["summaries_done"] == N_CANDIDATES, body
assert body["summaries_total"] == N_CANDIDATES, body
assert body["phase"] is None, body
assert body["current_summary"] is None, body
# The DB pins the full regeneration: exactly one AI row per
# candidate, the deterministic canned line, a fresh stamp.
rows = _folder_rows()
assert set(rows) == {""} | set(FOLDERS), rows
for folder, (summary, manually, updated_at) in rows.items():
assert summary == EXPECTED[folder], (folder, summary)
assert manually is False, (folder, manually)
assert updated_at is not None, (folder, updated_at)
# ---------------------------------------------------------------------------
# 2. ONE changed file in alpha/ → exactly alpha's chain (alpha + the
# source root) is re-stamped; every other row is byte-identical
# (text AND updated_at); the terminal keeps the AFFECTED count
# ---------------------------------------------------------------------------
def test_one_file_change_restamps_only_that_folders_chain(
app_url: str, incsum_dir: Path, db_ready: None
) -> None:
"""Re-seed (the autouse truncate) + sync #1 (full), capturing every
row's ``(summary, updated_at)``; then ONE doc inside ``alpha/``
changes content and sync #2 settles ``success``: the ``alpha`` AND
the source-root rows carry a NEW ``updated_at`` (strictly after
the captured stamp) with the canned text, while the
``bravo``/``charlie``/``delta`` rows are byte-identical — same
``summary`` AND same ``updated_at`` (an unaffected folder's
documents, and hence its summary, are untouched — no re-stamp).
The terminal keeps ``summaries_done == summaries_total == 2`` (the
AFFECTED count — alpha + root, NOT the table's candidate count —
the keep-final-counts convention), and the detail shows the
single-file change."""
_register_source(app_url, str(incsum_dir))
body = _run_sync(app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == N_FILES, body["detail"]
before = _folder_rows()
assert set(before) == {""} | set(FOLDERS), before
# The single-file change: ONE alpha doc gets new content (the
# importer's sha256 delta reports ``updated`` — no add, no prune).
target = incsum_dir / FOLDER_A / f"{FOLDER_A}-a.md"
target.write_text(
_md("Incsum Alpha A (revised)", ALPHA_A_REVISED), encoding="utf-8"
)
body = _run_sync(app_url)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == 0, detail
assert detail["updated"] == 1, detail
assert detail["unchanged"] == N_FILES - 1, detail
assert detail["pruned"] == 0, detail
assert detail["overview"] is True, detail
# Terminal (the keep-final-counts convention): the AFFECTED count —
# alpha + the source root — the phase-127 observable at the wire.
assert body["summaries_done"] == 2, body
assert body["summaries_total"] == 2, body
assert body["phase"] is None, body
assert body["current_summary"] is None, body
after = _folder_rows()
assert set(after) == {""} | set(FOLDERS), after
# The chain (alpha + the source root): RE-STAMPED — a new stamp
# strictly after the captured one, the canned text, still
# AI-written (the mock labels from the folder header only, so the
# text is the same deterministic line).
for folder in ("", FOLDER_A):
summary, manually, updated_at = after[folder]
assert summary == EXPECTED[folder], (folder, summary)
assert manually is False, (folder, manually)
assert updated_at > before[folder][2], (
folder,
before[folder],
after[folder],
)
# The siblings: BYTE-IDENTICAL — same summary text AND same
# updated_at (never re-stamped, never even re-fetched for
# generation).
for folder in FOLDERS:
if folder == FOLDER_A:
continue
assert after[folder] == before[folder], (
folder,
before[folder],
after[folder],
)
# ---------------------------------------------------------------------------
# 3. A vanished folder (bravo/ loses its last docs) has its row PRUNED
# by the prune pass; the changed sibling's chain (charlie + the
# source root) is re-stamped; the other rows are byte-identical
# ---------------------------------------------------------------------------
def test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_folder(
app_url: str, incsum_dir: Path, db_ready: None
) -> None:
"""Re-seed + sync #1 (full), capturing every row; then ``bravo/``
loses its last docs (the folder VANISHES — 0 documents, no longer
a candidate) and ONE doc in ``charlie/`` changes content; the sync
settles ``success``: the ``bravo`` row is PRUNED (gone — the
prune pass owns vanished folders' rows, independent of the
generation's affected set), the source-root AND ``charlie`` rows
are re-stamped (new ``updated_at`` + canned text), the
``alpha``/``delta`` rows are byte-identical (text AND
``updated_at``), and the terminal keeps
``summaries_done == summaries_total == 2`` (the affected
CANDIDATE count — charlie + root; bravo is no longer a candidate,
so it is neither generated nor counted)."""
_register_source(app_url, str(incsum_dir))
body = _run_sync(app_url)
assert body["state"] == "success", body
assert body["detail"]["added"] == N_FILES, body["detail"]
before = _folder_rows()
assert set(before) == {""} | set(FOLDERS), before
# bravo/ loses its last docs — the folder vanishes from the
# catalogue (0 documents → below the ≥ 1 minimum → not a
# candidate → the prune pass owns its row).
for letter in ("a", "b"):
(incsum_dir / FOLDER_B / f"{FOLDER_B}-{letter}.md").unlink()
# One charlie doc gets new content (``updated``).
target = incsum_dir / FOLDER_C / f"{FOLDER_C}-a.md"
target.write_text(
_md("Incsum Charlie A (revised)", CHARLIE_A_REVISED), encoding="utf-8"
)
body = _run_sync(app_url)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == 0, detail
assert detail["updated"] == 1, detail
assert detail["unchanged"] == N_FILES - 3, detail
assert detail["pruned"] == 2, detail
assert detail["overview"] is True, detail
# Terminal (the keep-final-counts convention): the affected
# CANDIDATE count — charlie + the source root (bravo is gone from
# the candidates, so it is not in the affected set's intersection).
assert body["summaries_done"] == 2, body
assert body["summaries_total"] == 2, body
assert body["phase"] is None, body
assert body["current_summary"] is None, body
after = _folder_rows()
# The vanished folder's row is PRUNED (the prune pass — it runs in
# both modes, manual rows exempt — bravo's row is AI-written).
assert FOLDER_B not in after, after
assert set(after) == ({""} | set(FOLDERS)) - {FOLDER_B}, after
# charlie + the source root: RE-STAMPED — a new stamp strictly
# after the captured one, the canned text, still AI-written.
for folder in ("", FOLDER_C):
summary, manually, updated_at = after[folder]
assert summary == EXPECTED[folder], (folder, summary)
assert manually is False, (folder, manually)
assert updated_at > before[folder][2], (
folder,
before[folder],
after[folder],
)
# alpha + delta: BYTE-IDENTICAL — same summary text AND same
# updated_at (neither folder is in the changed set's chains).
for folder in (FOLDER_A, FOLDER_D):
assert after[folder] == before[folder], (
folder,
before[folder],
after[folder],
)
+275 -2
View File
@@ -64,6 +64,17 @@ background sync task lives on the app's event loop, so the loop must
survive across requests — exactly how the app runs under uvicorn.
(A TestClient without the context manager starts a fresh loop per
request and would cancel the task on request exit.)
Phase 127 (task 03): the changed-KB branch passes the import's change
set to the generator (``changed_docs=summary.changed_paths`` —
incremental regeneration over the changed set's folder keys only),
the ``changed=N`` log line names the change count, and the status's
``summaries_total`` is the affected count (the hook's total) rather
than the table's candidate count. The unchanged-walk gap probe is
orthogonal: it still calls the generator exactly as before
(``only_missing=True``, no change set). The fakes above record the
``changed_docs=`` kwarg (``FakeFolderSummaries.changed_docs_sets``) so
the wiring is asserted, not inferred.
"""
from __future__ import annotations
@@ -71,7 +82,7 @@ import asyncio
import logging
import threading
import time
from collections.abc import Callable, Iterator
from collections.abc import Callable, Iterator, Sequence
from datetime import UTC, datetime
from pathlib import Path
from typing import Any
@@ -85,8 +96,9 @@ from app.api import sync as sync_api
from app.config import Settings
from app.db import SessionLocal, db_available
from app.main import app as fastapi_app
from app.models import GitSource
from app.models import Document, GitSource
from app.rag import git_sources as git_sources_resolver
from app.rag.folder_summaries import affected_folder_keys
from app.rag.importer import ImportSummary
from app.rag.llm import EmbeddingError, LLMClient, ModelUnavailableError
from app.rag.sources_meta import current_sources_version
@@ -330,6 +342,11 @@ class FakeFolderSummaries:
hook when given — the same way the import's file counter is driven
through its hook, so the status's summary counters are testable
deterministically.
Phase 127 (task 03): records the ``changed_docs=`` change set the
runner passes on the CHANGED-KB branch (the import's
``changed_paths`` — None when the runner regresses to the
change-set-less full regeneration; the unchanged-walk gap-fill
branch never passes it).
"""
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
@@ -346,6 +363,7 @@ class FakeFolderSummaries:
self.skip_flags: list[bool] = []
self.only_missing_flags: list[bool] = []
self.progress_hooks: list[Callable[[int, int, str, str], None] | None] = []
self.changed_docs_sets: list[Sequence[tuple[str, str]] | None] = []
async def __call__(
self,
@@ -354,10 +372,12 @@ class FakeFolderSummaries:
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
self.skip_flags.append(skip)
self.only_missing_flags.append(only_missing)
self.changed_docs_sets.append(changed_docs)
self.progress_hooks.append(on_progress)
if skip:
return dict(self.ZERO)
@@ -663,6 +683,7 @@ class _GatedFolderSummaries:
self.steps_before = steps_before
self.steps_after = steps_after
self.hook: Callable[[int, int, str, str], None] | None = None
self.changed_docs: Sequence[tuple[str, str]] | None = None
async def __call__(
self,
@@ -671,10 +692,12 @@ class _GatedFolderSummaries:
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
assert not skip, "these runs take a generation branch, never --limit"
self.hook = on_progress
self.changed_docs = changed_docs
for done, total, source, folder_path in self.steps_before:
if on_progress is not None:
on_progress(done, total, source, folder_path)
@@ -822,6 +845,7 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
self.hook = on_progress
@@ -845,6 +869,255 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
assert failing.hook is not None # the wiring held
# --- phase 127 (task 03): the changed-KB branch's incremental wiring ----
def test_changed_sync_passes_the_change_set_to_the_generator(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Phase 127 (task 03): a changed-KB sync passes the import's
change set into the generator (``changed_docs=summary.
changed_paths`` — the incremental-regression restriction), NOT the
gap-fill flag, the success ``detail`` keeps its exact pre-phase key
set (the stats stay log-only), and the log line names the change
count (``changed=N``)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
# One updated document — the import's change set carries exactly
# its (source, path) identity (task 01's contract).
change_set = [("repo", "notes/deep.md")]
summary = ImportSummary(
files=4, added=0, updated=1, unchanged=3, changed_paths=change_set
)
fake_import = FakeImportSources(summary)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
fake_folders = FakeFolderSummaries()
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# The CHANGED-KB branch fired the generator exactly once ... with
# the change set, NOT the gap-fill flag (the two restriction
# kwargs are orthogonal — no caller combines them).
assert len(fake_folders.changed_docs_sets) == 1
assert fake_folders.changed_docs_sets == [change_set]
assert fake_folders.only_missing_flags == [False]
assert fake_folders.skip_flags == [False]
# The success detail is byte-identical in shape to pre-phase 127 —
# no new key (the stats stay log-only, the phase-98 convention).
assert set(body["detail"]) == {
"files", "added", "updated", "unchanged", "pruned", "errors",
"chunks", "summaries", "summary_errors", "dates_updated",
"overview", "sources_version",
}
assert body["detail"]["added"] == 0 and body["detail"]["updated"] == 1
assert body["detail"]["overview"] is True
assert body["detail"]["sources_version"] == 1 # changed KB → one bump
# The ample-logging line names the change count (PLAN §9).
assert any(
r.getMessage() == (
"sync: folder_summaries changed=1 "
"stats={'generated': 0, 'failed': 0, 'pruned': 0}"
)
for r in caplog.records
)
def test_unchanged_walk_gap_probe_passes_no_change_set(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
clean_documents: None,
) -> None:
"""Phase 127 (task 03): the unchanged-walk gap probe is
ORTHOGONAL to the change set — a walk that changed nothing with a
real gap (seeded catalogue, empty ``folder_summaries`` table) still
calls the generator exactly as before: ``only_missing=True`` and
NO ``changed_docs`` (the gap probe's total is the missing count,
as before)."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
# The gap: two docs under notes/ (candidates root + notes), the
# folder_summaries table empty (the autouse fixture truncated it).
db.add_all([
Document(
source="repo", path="notes/one.md", full_path="/tmp/one.md",
title="One", content="# One\nfirst\n", content_hash="a" * 64,
),
Document(
source="repo", path="notes/two.md", full_path="/tmp/two.md",
title="Two", content="# Two\nsecond\n", content_hash="b" * 64,
),
])
db.commit()
summary = ImportSummary(files=2, added=0, updated=0, unchanged=2)
fake_import = FakeImportSources(summary)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
fake_folders = FakeFolderSummaries()
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
# The gap probe fired the generator exactly once — the gap-fill
# shape, byte-identical to pre-phase 127: only_missing=True, and
# NO change set (the gap is a stored-rows gap, not a document
# change — passing changed_paths here would restrict the fill to
# nothing).
assert len(fake_folders.changed_docs_sets) == 1
assert fake_folders.changed_docs_sets == [None]
assert fake_folders.only_missing_flags == [True]
assert body["detail"]["overview"] is False # unchanged → no overview
assert body["detail"]["sources_version"] == 0 # unchanged → no bump
class _AffectedKeysFolderSummaries:
"""A folder-summary step that mirrors the REAL generator's hook
contract on a changed set (phase 127, task 02): the loop runs over
the sorted AFFECTED keys (``affected_folder_keys(changed_docs)``),
the hook fires once per key with ``total == len(keys)``, and the
step parks mid-span after the first key — the (long) phase the
user reported. The test reads the status mid-span and asserts the
status's ``summaries_total`` IS the affected count (the hook's
total), not the table's candidate count."""
def __init__(
self,
started: threading.Event,
release: threading.Event,
) -> None:
self.started = started
self.release = release
self.changed_docs: Sequence[tuple[str, str]] | None = None
self.keys: list[tuple[str, str]] = []
self.hook: Callable[[int, int, str, str], None] | None = None
async def __call__(
self,
db: Session,
llm: LLMClient,
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
assert not skip, "these runs take a generation branch, never --limit"
self.hook = on_progress
self.changed_docs = changed_docs
assert on_progress is not None, "the runner must wire the hook"
# The real generator's step 2: sorted(candidates & affected) —
# the fake takes the affected side (the candidates are a
# superset here: every affected folder holds a doc).
self.keys = sorted(affected_folder_keys(changed_docs or []))
assert self.keys, "the change set must affect at least the root"
on_progress(1, len(self.keys), self.keys[0][0], self.keys[0][1])
self.started.set()
await asyncio.to_thread(self.release.wait)
for i, (source, folder_path) in enumerate(self.keys[1:], start=2):
on_progress(i, len(self.keys), source, folder_path)
return {"generated": len(self.keys), "failed": 0, "pruned": 0}
def test_summaries_phase_reports_the_affected_count_as_total(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
) -> None:
"""Phase 127 (task 03): mid-span, the status's ``summaries_total``
is the AFFECTED count — ``len(affected_folder_keys(changed set))``
(the hook's total), not the table's candidate count — and the
terminal keeps that final count (the phase-64 convention, now over
the affected span). The change set: one nested-folder doc (a
three-key chain) + one root-level doc (shares the root key) →
exactly 3 affected keys."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
_stub_probe(monkeypatch)
change_set = [("repo", "notes/a/one.md"), ("repo", "top.md")]
fake_import = FakeImportSources(
ImportSummary(
files=6, added=1, updated=2, unchanged=3, changed_paths=change_set
)
)
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
gated_folders = _AffectedKeysFolderSummaries(
threading.Event(), threading.Event()
)
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
_login(sync_client)
assert sync_client.post("/api/sync").status_code == 202
try:
assert gated_folders.started.wait(5.0), "the run never reached the summaries"
s = sync_client.get("/api/sync/status").json()
assert s["state"] == "running"
assert s["phase"] == "summaries"
# The hook's first key: the source root (the bare source name —
# folder_path "" never gets a slash) ...
assert s["current_summary"] == "repo"
assert s["summaries_done"] == 1
# ... and the total IS the affected count (the task's pin):
# notes/a/one.md → ("repo", ""), ("repo", "notes"),
# ("repo", "notes/a"); top.md → ("repo", "") — 3 keys.
expected = sorted(affected_folder_keys(change_set))
assert expected == [("repo", ""), ("repo", "notes"), ("repo", "notes/a")]
assert s["summaries_total"] == len(expected)
assert s["summaries_total"] == 3
gated_folders.release.set()
finally:
gated_folders.release.set()
body = _poll(sync_client, "success")
# Terminal: phase + current_summary cleared, the hook's FINAL
# (affected) counts kept (the phase-64 keep-final-counts rule, now
# over the affected span).
assert body["phase"] is None and body["current_summary"] is None
assert body["summaries_done"] == 3 and body["summaries_total"] == 3
# Wiring: the runner passed the change set, not the gap-fill flag.
assert gated_folders.changed_docs == change_set
assert gated_folders.hook is not None
# --- admin: failures -------------------------------------------------------
+380
View File
@@ -29,6 +29,7 @@ from app.rag.folder_summaries import (
FOLDER_SUMMARY_MODE,
MIN_DOCS_PER_FOLDER,
SYSTEM_PROMPT,
affected_folder_keys,
build_folder_summary_prompt,
folder_of,
generate_folder_summaries,
@@ -202,6 +203,56 @@ def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
assert "top.md" not in [folder for _source, folder in groups]
# ---------- affected_folder_keys (phase 127, task 02) ----------
def test_affected_folder_keys_root_level_file_is_only_the_root() -> None:
"""A root-level file (``folder_of == ""``) affects ONLY the source
root — there is no folder prefix to walk."""
assert affected_folder_keys([("S", "a.md")]) == {("S", "")}
def test_affected_folder_keys_nested_file_walks_the_folder_chain() -> None:
"""A doc under ``a/b/`` affects ``a/b``, ``a``, and the source
root — the SAME prefix walk ``group_by_folder`` pass 1 uses (one
concept)."""
assert affected_folder_keys([("S", "a/b/c.md")]) == {
("S", ""),
("S", "a"),
("S", "a/b"),
}
def test_affected_folder_keys_multi_source_set_semantics() -> None:
"""Keys are per source (the PK is ``(source, folder_path)``), and
the result is a SET — a repeated chain adds no new keys, and an
unrelated source keeps its own root."""
keys = affected_folder_keys(
[
("S", "a/b/c.md"),
("T", "x.md"),
("S", "a/d.md"), # the same chain as the first entry — no new keys
]
)
assert keys == {("S", ""), ("S", "a"), ("S", "a/b"), ("T", "")}
def test_affected_folder_keys_fragment_paths_fold_into_their_parent_folder() -> None:
"""Embedded-image fragment paths (phase 126: ``x.pdf#page=…#img=…``)
get NO special case: the fragment never holds a ``/``, so
``folder_of`` folds them into the parent folder's chain (a
root-level fragment affects only the source root)."""
assert affected_folder_keys([("S", "dir/img.pdf#page=2#img=0")]) == {
("S", ""),
("S", "dir"),
}
assert affected_folder_keys([("S", "img.pdf#page=1#img=0")]) == {("S", "")}
def test_affected_folder_keys_empty_input_is_an_empty_set() -> None:
assert affected_folder_keys([]) == set()
# ---------- build_folder_summary_prompt: system ----------
@@ -1093,3 +1144,332 @@ def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None
("FSU", "a/b"): REPLY,
("FSU-solo", ""): REPLY,
}
# ---------- generate_folder_summaries(changed_docs=…) (phase 127, task 02) ----------
def _seed_three_sources(db: Session) -> None:
"""The restricted-mode catalogue (three sources): A is nested
(``a/`` and ``a/b/``), B is a sibling single-folder source, and C
is a single-doc root — six candidate keys total: A ×
{``""``, ``a``, ``a/b``}, B × {``""``, ``b``}, C × {``""``}.
"""
_add_doc(db, "A", "a/b/one.md", "A One", "A one lead.")
_add_doc(db, "A", "a/b/two.md", "A Two")
_add_doc(db, "A", "a/three.md", "A Three")
_add_doc(db, "B", "b/one.md", "B One")
_add_doc(db, "C", "solo.md", "C Solo")
def _stamps(
db: Session, keys: set[tuple[str, str]]
) -> dict[tuple[str, str], object]:
"""The stored rows' ``updated_at`` for *keys* (raw SQL — the
byte-identity comparisons stay honest)."""
return {
(source, folder): _updated_at(db, source, folder) for source, folder in keys
}
def _freeze(
db: Session, values: dict[tuple[str, str], str]
) -> dict[tuple[str, str], object]:
"""Overwrite the stored summaries for *values*' keys with DISTINCT
frozen text (a "regenerated" row is then distinguishable from an
"untouched" one), and return the pre-run stamps."""
for (source, folder_path), value in values.items():
db.execute(
text(
"UPDATE folder_summaries SET summary = :v"
" WHERE source = :s AND folder_path = :f"
),
{"v": value, "s": source, "f": folder_path},
)
db.commit()
return _stamps(db, set(values))
def test_changed_docs_restricted_regeneration_only_affected_keys(
db: Session, clean_tables
) -> None:
"""One changed file in A/``a/b/`` → exactly that file's folder
chain (``a/b``, ``a``, and the A root) regenerates: one lite call
per affected non-manual key, in sorted order, fresh rows. Every
OTHER stored row — same-source siblings and other sources — keeps
its summary text AND ``updated_at`` byte-identical (zero lite
calls for them)."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
frozen = {
("A", ""): "A root frozen",
("A", "a"): "A/a frozen",
("A", "a/b"): "A/a/b frozen",
("B", ""): "B root frozen",
("B", "b"): "B/b frozen",
("C", ""): "C root frozen",
}
stamps = _freeze(db, frozen)
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3, "one lite call per AFFECTED non-manual key — nothing else"
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: A",
"Folder: A/a",
"Folder: A/a/b",
], "the affected chain in sorted (source, folder_path) order"
stored = _rows(db)
assert stored[("A", "")] == REPLY
assert stored[("A", "a")] == REPLY
assert stored[("A", "a/b")] == REPLY, "the changed file's own folder regenerates"
assert stored[("B", "")] == "B root frozen"
assert stored[("B", "b")] == "B/b frozen"
assert stored[("C", "")] == "C root frozen", (
"the unaffected rows keep their stored text"
)
for key in (("B", ""), ("B", "b"), ("C", "")):
assert _updated_at(db, *key) == stamps[key], (
"the unaffected rows are never re-stamped"
)
def test_changed_docs_manual_affected_row_is_skipped(db: Session, clean_tables) -> None:
"""A manual row INSIDE the affected chain is skipped exactly as in
full mode (phase 97): ``kept_manual``, no lite call, text AND
``updated_at`` byte-identical — the restriction changes the key
set, not the manual-row rule."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
manual_text = "Owner's words about A/a."
row = db.get(FolderSummary, ("A", "a"))
assert row is not None
row.summary = manual_text
row.manually_edited = True
db.commit()
stamp_before = _updated_at(db, "A", "a")
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 2, "failed": 0, "pruned": 0, "kept_manual": 1}
assert llm.calls == 2, "the manual key burns zero lite calls"
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: A",
"Folder: A/a/b",
]
stored = _rows(db)
assert stored[("A", "a")] == manual_text, "the owner's text is never rewritten"
assert _updated_at(db, "A", "a") == stamp_before, ("never re-stamped")
assert stored[("A", "")] == REPLY and stored[("A", "a/b")] == REPLY
assert stored[("B", "")] == REPLY and stored[("B", "b")] == REPLY
assert stored[("C", "")] == REPLY, "the unaffected rows are left as they were"
def test_changed_docs_failing_affected_folder_fails_soft(db: Session, clean_tables) -> None:
"""Per-folder fail-soft under the restriction: the failing
affected key is attempted and counted, its PREVIOUS row is kept
(an old summary is better than none), the other affected keys
still land, and the unaffected rows are untouched."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
# A DISTINCT previous text on the failing folder, so "kept" is
# visible (it would otherwise be the REPLY either way).
db.execute(
text(
"UPDATE folder_summaries SET summary = 'old A/a/b'"
" WHERE (source, folder_path) = ('A', 'a/b')"
)
)
db.commit()
stamps = _stamps(db, {("B", ""), ("B", "b"), ("C", "")})
llm = _FakeLLM(fail_folders=("A/a/b",))
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 2, "failed": 1, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3, "the failing key is attempted (the hook's premise)"
stored = _rows(db)
assert stored[("A", "a/b")] == "old A/a/b", "the previous row survives"
assert stored[("A", "")] == REPLY and stored[("A", "a")] == REPLY
for key, stamp in stamps.items():
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
def test_changed_docs_multi_source_change_set(db: Session, clean_tables) -> None:
"""A change set spanning SOURCES: the affected keys are the UNION
of each document's chain (A's nested chain + B's root and ``b``),
sorted across sources; the unaffected C root row keeps its text
AND ``updated_at``."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
c_stamp = _updated_at(db, "C", "")
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(
db, llm, changed_docs=[("A", "a/b/two.md"), ("B", "b/one.md")]
)
)
assert stats == {"generated": 5, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 5
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: A",
"Folder: A/a",
"Folder: A/a/b",
"Folder: B",
"Folder: B/b",
]
stored = _rows(db)
assert stored[("C", "")] == REPLY
assert _updated_at(db, "C", "") == c_stamp
def test_changed_docs_gap_in_affected_folder_is_generated(
db: Session, clean_tables
) -> None:
"""An affected folder with NO stored row (a gap in the chain) is a
candidate and gets generated like any other affected key — while a
gap in an UNAFFECTED folder stays the unchanged-walk gap probe's
job (phase 96): the restriction never widens beyond the affected
set."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
db.execute(
text(
"DELETE FROM folder_summaries"
" WHERE (source, folder_path) IN (('A', 'a/b'), ('B', ''))"
)
)
db.commit()
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
)
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 3
stored = _rows(db)
assert stored[("A", "a/b")] == REPLY, "the gap in the affected chain is filled"
assert ("B", "") not in stored, (
"the gap in the UNAFFECTED source is left for the gap probe"
)
assert stored[("B", "b")] == REPLY
def test_changed_docs_vanished_affected_folder_is_pruned_not_generated(
db: Session, clean_tables
) -> None:
"""The ASSUMPTION pinned: an affected folder that no longer holds
any document is a PRUNE-pass job, not a generation's — the pruned
document's chain (root + its now-vanished folder) is in the change
set, only the surviving candidate (the root) generates, the
vanished folder's row is dropped, and the unaffected rows keep
their ``updated_at``."""
_add_doc(db, "A", "a/b/one.md", "A One")
_add_doc(db, "A", "a/b/two.md", "A Two")
_add_doc(db, "A", "gone/old.md", "Gone")
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
stamps = _stamps(db, {("A", "a"), ("A", "a/b")})
db.execute(text("DELETE FROM documents WHERE path = 'gone/old.md'"))
db.commit()
llm = _FakeLLM()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[("A", "gone/old.md")])
)
# affected = {("A", ""), ("A", "gone")} — the only surviving
# candidate in that set is the source root.
assert stats == {"generated": 1, "failed": 0, "pruned": 1, "kept_manual": 0}
assert llm.calls == 1
assert [user.splitlines()[0] for _s, user in llm.requests] == ["Folder: A"]
stored = _rows(db)
assert ("A", "gone") not in stored, "the vanished folder's row is pruned"
assert stored[("A", "")] == REPLY, "the root's subtree changed — regenerated"
assert stored[("A", "a")] == REPLY and stored[("A", "a/b")] == REPLY
for key, stamp in stamps.items():
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
def test_changed_docs_hook_total_is_the_affected_count(
db: Session, clean_tables
) -> None:
"""The progress hook (phase 98) under the restriction fires exactly
``len(affected)`` times with ``total == len(affected)`` — the sync
status's ``summaries_total`` then reports the affected count
through the existing hook — in sorted key order."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
llm = _FakeLLM()
events, record = _record_progress()
stats = asyncio.run(
generate_folder_summaries(
db, llm, changed_docs=[("A", "a/b/one.md")], on_progress=record
)
)
assert events == [
(1, 3, "A", ""),
(2, 3, "A", "a"),
(3, 3, "A", "a/b"),
]
assert llm.calls == 3
assert stats["generated"] == 3
def test_changed_docs_none_is_the_pre_127_full_regeneration(
db: Session, clean_tables
) -> None:
"""The default (no ``changed_docs`` kwarg) is the pre-127 FULL
regeneration over the same three-source catalogue: every
candidate key, one call each, the 4-field stats shape unchanged —
the ``scripts/import_docs.py`` CLI contract."""
_seed_three_sources(db)
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 6, "failed": 0, "pruned": 0, "kept_manual": 0}
assert llm.calls == 6, "no kwarg → every candidate burns one call"
assert set(_rows(db)) == {
("A", ""), ("A", "a"), ("A", "a/b"),
("B", ""), ("B", "b"),
("C", ""),
}
def test_changed_docs_empty_set_generates_nothing_but_still_prunes(
db: Session, clean_tables
) -> None:
"""``changed_docs=[]`` (a change set that IS given but empty)
restricts the generation to zero keys — no lite calls, no
re-stamps, the hook never fires — while the prune pass still runs
in the given mode (the sweep is mode-independent, phase 97)."""
_seed_three_sources(db)
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
db.add(FolderSummary(source="A", folder_path="gone/old", summary="stale"))
db.commit()
stamps = _stamps(db, {("A", ""), ("A", "a"), ("A", "a/b"), ("B", ""), ("C", "")})
llm = _FakeLLM()
events, record = _record_progress()
stats = asyncio.run(
generate_folder_summaries(db, llm, changed_docs=[], on_progress=record)
)
assert stats == {"generated": 0, "failed": 0, "pruned": 1, "kept_manual": 0}
assert llm.calls == 0
assert events == [], "zero keys → the hook never fires"
for key, stamp in stamps.items():
assert _updated_at(db, *key) == stamp, "no row is re-stamped"
+236
View File
@@ -15,9 +15,11 @@ from __future__ import annotations
import asyncio
import logging
import os
from datetime import UTC, datetime
from pathlib import Path
import pymupdf as fitz
import pytest
from sqlalchemy import func, select
from sqlalchemy.orm import Session
@@ -28,6 +30,7 @@ from app.models import Chunk, Document
from app.rag.importer import (
EXCLUDED_DIRS,
ImportSummary,
_prune,
_store_summary,
import_sources,
iter_importable_files,
@@ -1194,3 +1197,236 @@ def test_progress_hook_with_limit_keeps_full_total(db, tmp_path: Path) -> None:
assert events == [(root.name, "a.md", 1, 3), (root.name, "b.md", 2, 3)]
finally:
_cleanup_source(db, root.name)
# ---------- phase 127 (task 01): ImportSummary.changed_paths ----------
class _VisionLLM(FakeEmbedder):
"""The phase-122 mock VISION client (the test_pdf pattern): the
multimodal describe call (``user`` content = a part list) gets a fixed
description; text (``lite``) calls keep the ``FakeEmbedder``
behaviour."""
DESCRIPTION = "A test image: a solid swatch of color."
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
self.chat_calls.append(list(messages))
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
if isinstance(user, list):
return self.DESCRIPTION
first = user.split()
return "Summary of " + (first[0] if first else "<empty>")
def _png_bytes(width: int, height: int, rgb: tuple[float, float, float]) -> bytes:
"""One solid-color PNG built with PyMuPDF (the phase-126 no-binary-
fixtures convention)."""
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height), 0)
for y in range(height):
for x in range(width):
pix.set_pixel(x, y, rgb)
return pix.tobytes("png")
def _build_one_page_pdf(path: Path, text: str) -> None:
"""One temp single-page PDF (the phase-126 convention — no binary
fixtures)."""
doc = fitz.open()
doc.new_page().insert_text((72, 72), text)
doc.save(path)
doc.close()
def test_import_summary_changed_paths_defaults_empty() -> None:
"""Completion criterion: default construction gives an empty change
set — no existing construction site migrates."""
assert ImportSummary().changed_paths == []
def test_changed_paths_records_added_updated_and_pruned_in_order(
db, tmp_path: Path
) -> None:
"""The contract (phase 127, task 01): one ``(source, path)`` entry per
ADDED, UPDATED, or PRUNED document — added/updated in walk order, the
pruned pair last (``_prune`` runs after the walk) — and
``len(changed_paths) == added + updated + pruned``."""
root = tmp_path / "changed"
root.mkdir()
for name in ("a.md", "b.md", "c.md"):
(root / name).write_text(f"# {name}\n\nv1\n")
llm = FakeEmbedder()
try:
source = root.name
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 3 and first.pruned == 0
assert first.changed_paths == [
(source, "a.md"),
(source, "b.md"),
(source, "c.md"),
]
# Run 2: update a.md, add d.md + e.md, delete c.md (→ pruned).
(root / "a.md").write_text("# a\n\nv2 changed\n")
(root / "d.md").write_text("# d\n\nnew d\n")
(root / "e.md").write_text("# e\n\nnew e\n")
(root / "c.md").unlink()
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
assert (second.added, second.updated, second.unchanged, second.pruned) == (
2, 1, 1, 1,
)
assert second.changed_paths == [
(source, "a.md"), # updated — first in walk order
(source, "d.md"), # added
(source, "e.md"), # added
(source, "c.md"), # pruned — the _prune pass runs after the walk
]
assert len(second.changed_paths) == (
second.added + second.updated + second.pruned
)
finally:
_cleanup_source(db, root.name)
def test_changed_paths_empty_on_unchanged_rerun(db, tmp_path: Path) -> None:
"""An all-unchanged re-run (prune included) records nothing."""
root = tmp_path / "nochange"
root.mkdir()
(root / "a.md").write_text("# a\n\nstable\n")
(root / "b.md").write_text("# b\n\nstable\n")
llm = FakeEmbedder()
try:
asyncio.run(import_sources([root], llm, session=db))
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
assert second.unchanged == 2
assert second.changed_paths == []
finally:
_cleanup_source(db, root.name)
def test_changed_paths_empty_on_backfill_rerun(db, tmp_path: Path) -> None:
"""A backfill (phase 118: unchanged doc, NULL summary) is NOT a content
change: the add still records, the backfill re-run stays empty."""
root = tmp_path / "backfill"
root.mkdir()
(root / "a.yaml").write_text("alpha x\n")
try:
first = asyncio.run(import_sources([root], _FailingChatEmbedder(), session=db))
assert first.added == 1 and first.summary_errors == 1
assert first.changed_paths == [(root.name, "a.yaml")] # the ADD records
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "a.yaml")
)
assert doc is not None and doc.summary is None
second = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
assert second.unchanged == 1 and second.summary_backfilled == 1
assert second.changed_paths == []
finally:
_cleanup_source(db, root.name)
def test_changed_paths_empty_on_date_only_rerun(db, tmp_path: Path) -> None:
"""A date-only refresh (phase 106, D4: content untouched, mtime moved)
is NOT a content change: the re-run records nothing."""
root = tmp_path / "dateonly"
root.mkdir()
path = root / "a.md"
path.write_text("# a\n\nstable\n")
os.utime(path, (0, datetime(2020, 1, 1, tzinfo=UTC).timestamp()))
llm = FakeEmbedder()
try:
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 1 and first.dates_updated == 0
assert first.changed_paths == [(root.name, "a.md")]
# The content is untouched — only the mtime moved (the D4
# date-refresh on the unchanged path).
os.utime(path, (0, datetime(2021, 6, 15, tzinfo=UTC).timestamp()))
second = asyncio.run(import_sources([root], llm, session=db))
assert second.unchanged == 1 and second.dates_updated == 1
assert second.changed_paths == []
finally:
_cleanup_source(db, root.name)
def test_changed_paths_records_a_changed_image(db, tmp_path: Path) -> None:
"""The image increment site (phase 122 path): a NEW image and a
byte-CHANGED image each record their ``(source, rel)``."""
root = tmp_path / "imgsrc"
root.mkdir()
(root / "pic.png").write_bytes(_png_bytes(1, 1, (1.0, 0.0, 0.0)))
llm = _VisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
images=True,
image_dir=str(tmp_path / "images"),
)
try:
source = root.name
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 1 and first.images_failed == 0
assert first.changed_paths == [(source, "pic.png")]
# The BYTES changed (a new color) — the image digest is over the
# bytes, so this is an UPDATE.
(root / "pic.png").write_bytes(_png_bytes(1, 1, (0.0, 1.0, 1.0)))
second = asyncio.run(import_sources([root], llm, session=db))
assert second.updated == 1
assert second.changed_paths == [(source, "pic.png")]
finally:
_cleanup_source(db, root.name)
def test_changed_paths_records_a_changed_pdf(db, tmp_path: Path) -> None:
"""The PDF increment site (phase 126 path): a NEW PDF and a text-
CHANGED PDF each record their ``(source, rel)``."""
root = tmp_path / "pdfsrc"
root.mkdir()
pdf_path = root / "doc.pdf"
_build_one_page_pdf(pdf_path, "alpha page one")
llm = _VisionLLM()
llm.settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
images=True,
image_dir=str(tmp_path / "images"),
import_extensions="md,txt,pdf",
)
try:
source = root.name
first = asyncio.run(import_sources([root], llm, session=db))
assert first.added == 1
assert first.changed_paths == [(source, "doc.pdf")]
# The extracted TEXT changed — the PDF digest is over it, so this
# is an UPDATE.
_build_one_page_pdf(pdf_path, "bravo page one, revised")
second = asyncio.run(import_sources([root], llm, session=db))
assert second.updated == 1
assert second.changed_paths == [(source, "doc.pdf")]
finally:
_cleanup_source(db, root.name)
def test_prune_records_changed_paths_only_when_summary_given(db, tmp_path: Path) -> None:
"""The ``_prune`` recording seam: with NO summary (the pre-phase
count-only contract) nothing is recorded; with one, each ACTUALLY
deleted doc appends its ``(source, path)`` (skipped docs never do)."""
root = tmp_path / "prunit"
root.mkdir()
(root / "gone.md").write_text("# gone\n\nbye\n")
(root / "gone2.md").write_text("# gone2\n\nbye\n")
llm = FakeEmbedder()
try:
source = root.name
asyncio.run(import_sources([root], llm, session=db))
(root / "gone.md").unlink()
(root / "gone2.md").unlink()
# Default call (summary=None): count only, no recording.
pruned = _prune(db, {source}, {(source, "gone2.md")})
assert pruned == 1
assert db.scalar(
select(Document).where(Document.source == source, Document.path == "gone.md")
) is None
# With a summary: the deleted doc records its pair.
counters = ImportSummary()
pruned = _prune(db, {source}, set(), summary=counters)
assert pruned == 1
assert counters.changed_paths == [(source, "gone2.md")]
finally:
_cleanup_source(db, root.name)
+5 -2
View File
@@ -40,7 +40,7 @@ from __future__ import annotations
import asyncio
import re
import threading
from collections.abc import Callable, Iterator
from collections.abc import Callable, Iterator, Sequence
from datetime import datetime
from pathlib import Path
@@ -819,12 +819,15 @@ def _patch_sync_seams(
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
# Phase 98 (task 01): the runner wires the progress hook into
# the generation branches — the fake accepts it (the state-
# machine tests' canned run never fires it: zero summary
# counters, like a no-gap skip).
# counters, like a no-gap skip). Phase 127 (task 03): the
# changed-KB branch passes the change set as changed_docs= —
# the fake accepts it (these canned runs record nothing).
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)