phase: 127_sync_incremental_folder_summaries
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact. **Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass) - **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors** - **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me) - **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed - **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit) - **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
@@ -0,0 +1,619 @@
|
||||
"""Phase 127 task 04 E2E (Playwright, mock-only): a one-file change
|
||||
rewrites ONLY that folder's summary chain.
|
||||
|
||||
The dedicated story suite for ``127_sync_incremental_folder_summaries``
|
||||
(A16 — one Playwright file per phase, run in isolation): the owner's
|
||||
report was that "on sync brain of reese rewrites all the summaries for
|
||||
every directory — so even if just one file changed this can result in
|
||||
700+ directory summary rewrites in production which can take a very
|
||||
long time" (``TODO.md`` L3). The fix under test: a KB-changing sync
|
||||
passes the import's changed-document set
|
||||
(``ImportSummary.changed_paths``) into
|
||||
``generate_folder_summaries(changed_docs=…)``, so ONLY the folders a
|
||||
changed document touches — each document's folder chain up to its
|
||||
source root (``affected_folder_keys``) — are re-stamped; every other
|
||||
stored row keeps its summary text AND ``updated_at`` byte-identical.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic mock (the phase-94 ``FOLDER_SUMMARY_MODE`` canned
|
||||
``Fixture folder summary for <folder>.`` lines): the stored rows are a
|
||||
pure function of the request, so the byte-identical assertions are
|
||||
exact only against the mock.
|
||||
|
||||
No slow-LLM proxy: the assertions are on the POST-terminal DB state
|
||||
(the rows' ``summary`` AND ``updated_at``) plus the terminal status
|
||||
(the keep-final-counts convention — the terminal keeps
|
||||
``summaries_done == summaries_total == <affected count>``), never on
|
||||
in-flight sampling, so the fast mock is dense enough by construction.
|
||||
|
||||
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
|
||||
the same host) registered as a ``kind=local`` source (the
|
||||
``test_local_directory_sources.py`` / phase-94 API-registration
|
||||
pattern; no git anywhere): ``incsum/`` with FOUR ≥ 2-doc folders —
|
||||
``alpha/``, ``bravo/``, ``charlie/``, ``delta/`` (2 docs each) — plus
|
||||
ONE root-level doc (9 documents total). The generator's candidate set
|
||||
is therefore exactly 5: the source root (``folder_path ""``) + the
|
||||
four folders (the recursive-subtree ≥ 1 rule).
|
||||
|
||||
Autouse cleanup (the phase-96 pattern): before each test wait for no
|
||||
running sync on the module app, then truncate the shared registry + KB
|
||||
tables (documents/chunks/query_log/steering_notes/kb_overview/
|
||||
git_sources/folder_summaries) — each test seeds its own source and
|
||||
runs its own sync(s); after each test the same truncate. (The temp
|
||||
tree itself persists for the module; the tests' disk mutations are
|
||||
ordered by test definition and each test re-syncs the state it finds.)
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule):
|
||||
1. ``test_first_sync_generates_all_rows`` — a first (full) sync settles
|
||||
``success``; the ``folder_summaries`` table holds EXACTLY the 5
|
||||
rows (source root + the four folders) with the mock's canned text
|
||||
naming each folder, each ``updated_at`` non-null, and the terminal
|
||||
keeps ``summaries_done == summaries_total == 5`` (the full
|
||||
regeneration — a changed KB with no prior rows, the pre-127
|
||||
behavior preserved end to end).
|
||||
2. ``test_one_file_change_restamps_only_that_folders_chain`` — re-seed
|
||||
(autouse truncated) + sync #1, capture every row's
|
||||
``(summary, updated_at)``; change ONE doc inside ``alpha/`` and
|
||||
sync #2 settles ``success``: the ``alpha`` AND the source-root rows
|
||||
carry a NEW ``updated_at`` (strictly after the captured stamp) with
|
||||
the canned text, while the ``bravo``/``charlie``/``delta`` rows are
|
||||
BYTE-IDENTICAL (same ``summary`` AND same ``updated_at``); the
|
||||
terminal keeps ``summaries_done == summaries_total == 2`` (the
|
||||
AFFECTED count — alpha + root, NOT the table's candidate count —
|
||||
the keep-final-counts convention) and the detail shows the
|
||||
single-file change (1 updated, 8 unchanged, 0 added/pruned).
|
||||
3. ``test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_
|
||||
folder`` — re-seed + sync #1, capture; delete ``bravo/``'s last
|
||||
docs (the folder VANISHES — 0 documents, no longer a candidate) and
|
||||
change one doc in ``charlie/``; the sync settles ``success``: the
|
||||
``bravo`` row is PRUNED (gone — the prune pass owns vanished
|
||||
folders' rows), the source-root AND ``charlie`` rows are re-stamped
|
||||
(new ``updated_at`` + canned text), the ``alpha``/``delta`` rows
|
||||
are byte-identical, and the terminal keeps
|
||||
``summaries_done == summaries_total == 2`` (the affected CANDIDATE
|
||||
count — charlie + root; bravo is no longer a candidate).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import FolderSummary
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns its
|
||||
# port in a combined run — this module's app binds its own port instead
|
||||
# (a same-port second uvicorn dies on bind and would drive the wrong
|
||||
# server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_INCSUM", "8147"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixture constants (deterministic, token-controlled)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: The local source — the temp directory's basename (``kind=local`` →
|
||||
#: the directory's basename is the source name, phase 38).
|
||||
SOURCE = "incsum"
|
||||
|
||||
#: The four ≥ 2-doc folders (path order).
|
||||
FOLDER_A = "alpha"
|
||||
FOLDER_B = "bravo"
|
||||
FOLDER_C = "charlie"
|
||||
FOLDER_D = "delta"
|
||||
FOLDERS = (FOLDER_A, FOLDER_B, FOLDER_C, FOLDER_D)
|
||||
|
||||
ROOT_NOTE = "root-note.md" # the one root-level doc
|
||||
|
||||
N_FILES = 9 # four folders × 2 docs + the root-level doc
|
||||
N_CANDIDATES = 5 # the source root + the four folders (the ≥ 1-doc rule)
|
||||
|
||||
#: The mock's byte-stable ``FOLDER_SUMMARY_MODE`` lines for this fixture
|
||||
#: (the phase-94 template — the label is the ``FOLDER_HEADER_PREFIX``
|
||||
#: tail: ``<source>`` for the root, ``<source>/<folder>`` for a folder).
|
||||
SUMMARY_FOR = "Fixture folder summary for {}."
|
||||
EXPECTED: dict[str, str] = {
|
||||
"": SUMMARY_FOR.format(SOURCE),
|
||||
FOLDER_A: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_A}"),
|
||||
FOLDER_B: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_B}"),
|
||||
FOLDER_C: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_C}"),
|
||||
FOLDER_D: SUMMARY_FOR.format(f"{SOURCE}/{FOLDER_D}"),
|
||||
}
|
||||
|
||||
#: The single-file change (test 2) — a revised body whose hash differs
|
||||
#: from the fixture's (the importer's sha256 delta reports ``updated``)
|
||||
#: and which no other fixture text contains.
|
||||
ALPHA_A_REVISED = (
|
||||
"Incsum alpha fixture note A, REVISED: the only file test 2 changed "
|
||||
"— the incremental regeneration must touch exactly this file's chain "
|
||||
"(alpha + the source root), and nothing else. (RESE- INCSUM-01)"
|
||||
)
|
||||
|
||||
#: The charlie change (test 3) — the same shape on the other side.
|
||||
CHARLIE_A_REVISED = (
|
||||
"Incsum charlie fixture note A, REVISED: the one changed file of "
|
||||
"test 3's sync, while bravo vanishes entirely. (RESE- INCSUM-02)"
|
||||
)
|
||||
|
||||
|
||||
def _md(title: str, body: str) -> str:
|
||||
return f"# {title}\n\n{body}\n"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def incsum_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The one-source temp tree (see the module docstring): the app
|
||||
server runs on the same host, so the paths are visible to it. The
|
||||
directory NAME is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_incsum")
|
||||
src = root / SOURCE
|
||||
for folder in FOLDERS:
|
||||
(src / folder).mkdir(parents=True)
|
||||
for letter in ("a", "b"):
|
||||
(src / folder / f"{folder}-{letter}.md").write_text(
|
||||
_md(
|
||||
f"Incsum {folder.title()} {letter.upper()}",
|
||||
f"Incsum {folder} fixture note {letter.upper()}: covers "
|
||||
f"topic {letter.upper()} of the {SOURCE} source tree.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(src / ROOT_NOTE).write_text(
|
||||
_md(
|
||||
"Incsum Root Note",
|
||||
f"This file sits directly under the {SOURCE} source, not in "
|
||||
"any folder.",
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert len(list(src.rglob("*.md"))) == N_FILES
|
||||
return src
|
||||
|
||||
|
||||
def _app_env(
|
||||
base_url: str, tmp_path_factory: pytest.TempPathFactory
|
||||
) -> dict[str, str]:
|
||||
"""The per-module app env (the conftest pattern, cf.
|
||||
``test_kb_tree.py`` / ``test_sync_summary_visibility.py``): the
|
||||
``base_url`` is the mock LLM directly (the fast leg is the only
|
||||
leg — see the module docstring), ``BOR_GIT_SOURCES`` forced empty
|
||||
(the sync source is this suite's own DB-registered local row), and
|
||||
the leak-guarded code defaults."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else base_url
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern): this suite never
|
||||
# asks the chat model anything — the gate is never on a path.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the registry must hold EXACTLY the local
|
||||
# directory this suite registers.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
env["BOR_SOURCES_DIR"] = str(tmp_path_factory.mktemp("bor_checkouts"))
|
||||
return env
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(
|
||||
mock_llm: int,
|
||||
incsum_dir: Path,
|
||||
tmp_path_factory: pytest.TempPathFactory,
|
||||
) -> Iterator[str]:
|
||||
"""The app under test (all three tests): ``BOR_LLM_BASE_URL`` is
|
||||
the mock directly — the canned ``FOLDER_SUMMARY_MODE`` lines are
|
||||
byte-stable there, and the byte-identity assertions key on that
|
||||
exact text (no slow proxy: the assertions are post-terminal, see
|
||||
the module docstring). ``incsum_dir`` is a dependency only for the
|
||||
fixture ordering (the temp tree exists before the app boots — the
|
||||
sync reads it)."""
|
||||
env = _app_env(f"http://127.0.0.1:{mock_llm}/v1", tmp_path_factory)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
|
||||
share one Postgres, so a leftover git_sources row or document would
|
||||
pollute the candidate count and the counts the syncs and the
|
||||
folder-rows assertions pin exactly."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit() # without the commit the TRUNCATE rolls back (the house pattern)
|
||||
|
||||
|
||||
def _wait_sync_settled(base_url: str, timeout_s: float = 180.0) -> None:
|
||||
"""No background sync may leak across tests (the run state lives in
|
||||
the app's memory, and a still-running run would keep importing into
|
||||
the NEXT test's truncated KB): wait for the app's sync status to
|
||||
be non-running BEFORE the truncate. Own admin session (the
|
||||
endpoint is admin-only) — usually a no-op: the tests settle only
|
||||
after their run's terminal state."""
|
||||
with httpx.Client(base_url=base_url, timeout=5.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
deadline = time.monotonic() + timeout_s
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
if r.json()["state"] != "running":
|
||||
return
|
||||
time.sleep(0.2)
|
||||
raise AssertionError(f"a sync was still running at a test boundary: {base_url}")
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(app_url: str, db_ready: None) -> Iterator[None]:
|
||||
"""Per-test KB isolation (the phase-96 cleanup pattern): the
|
||||
shared E2E Postgres is truncated before AND after every test —
|
||||
each test seeds its own source and runs its own sync(s), so the
|
||||
KB rows + folder_summaries rows of the temp source are exactly
|
||||
this test's own. The pre-wait keeps a leaked run from the
|
||||
previous test (the module app is shared) from racing the
|
||||
truncate."""
|
||||
_wait_sync_settled(app_url)
|
||||
_truncate_all()
|
||||
yield
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _register_source(base_url: str, path: str) -> None:
|
||||
"""Register the temp directory through the authenticated API (the
|
||||
``test_local_directory_sources.py`` / phase-94 pattern — a plain
|
||||
directory is an API/DB-only operation)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": path}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
|
||||
|
||||
def _run_sync(base_url: str, timeout_s: float = 120.0) -> dict[str, Any]:
|
||||
"""Login + ``POST /api/sync`` + poll the status endpoint until THIS
|
||||
run reaches a terminal state (the ``test_local_directory_sources``
|
||||
pattern, over plain httpx — with the stale-terminal freshness gate:
|
||||
the module app is shared, so the PREVIOUS run's terminal lives in
|
||||
the app's memory until this run supersedes it; a terminal is
|
||||
accepted only after a running tick was observed or its
|
||||
``started_at`` post-dates this POST)."""
|
||||
with httpx.Client(base_url=base_url, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
t0 = time.time()
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
saw_running = False
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
state = body["state"]
|
||||
if state == "running":
|
||||
saw_running = True
|
||||
elif state in ("success", "failed") and (
|
||||
saw_running
|
||||
or _started_at(body) >= t0 - 1.0 # same-host clock; 1 s tolerance
|
||||
):
|
||||
return body
|
||||
time.sleep(0.1)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _started_at(body: dict[str, Any]) -> float:
|
||||
"""The status body's ``started_at`` as a wall-clock epoch (the
|
||||
server stamps it in UTC; this process runs on the same host —
|
||||
same clock, the phase-64 recorder's freshness convention)."""
|
||||
started = body.get("started_at")
|
||||
if not started:
|
||||
return 0.0
|
||||
try:
|
||||
return datetime.fromisoformat(str(started)).timestamp()
|
||||
except ValueError:
|
||||
return 0.0
|
||||
|
||||
|
||||
def _folder_rows() -> dict[str, tuple[str, bool, datetime]]:
|
||||
"""The source's stored folder summaries
|
||||
``{folder_path: (summary, manually_edited, updated_at)}``
|
||||
(``""`` = the source root) — the test process's direct DB access
|
||||
(the E2E's other established lens)."""
|
||||
with SessionLocal() as db:
|
||||
rows = db.execute(
|
||||
select(
|
||||
FolderSummary.folder_path,
|
||||
FolderSummary.summary,
|
||||
FolderSummary.manually_edited,
|
||||
FolderSummary.updated_at,
|
||||
).where(FolderSummary.source == SOURCE)
|
||||
).all()
|
||||
return {
|
||||
folder: (summary, manually, updated)
|
||||
for folder, summary, manually, updated in rows
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. The first sync (a changed KB with no prior rows) is the FULL
|
||||
# regeneration: exactly the 5 candidate rows, canned text, the
|
||||
# terminal keeps the full candidate count
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_first_sync_generates_all_rows(
|
||||
app_url: str, incsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""A first sync of the temp source settles ``success`` with all 9
|
||||
docs added; the ``folder_summaries`` table holds EXACTLY the 5
|
||||
candidate rows (the source root + the four folders — the
|
||||
recursive-subtree ≥ 1 rule) with the mock's canned text naming
|
||||
each folder, every row AI-written and freshly stamped, and the
|
||||
terminal keeps ``summaries_done == summaries_total == 5`` (the
|
||||
full-regeneration count — a first sync passes no prior rows, so
|
||||
this is the pre-127 behavior preserved end to end)."""
|
||||
_register_source(app_url, str(incsum_dir))
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == N_FILES, detail
|
||||
assert detail["updated"] == 0 and detail["unchanged"] == 0, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# Terminal (the keep-final-counts convention): the FULL candidate
|
||||
# count — the pre-127 shape a first sync still reports.
|
||||
assert body["summaries_done"] == N_CANDIDATES, body
|
||||
assert body["summaries_total"] == N_CANDIDATES, body
|
||||
assert body["phase"] is None, body
|
||||
assert body["current_summary"] is None, body
|
||||
# The DB pins the full regeneration: exactly one AI row per
|
||||
# candidate, the deterministic canned line, a fresh stamp.
|
||||
rows = _folder_rows()
|
||||
assert set(rows) == {""} | set(FOLDERS), rows
|
||||
for folder, (summary, manually, updated_at) in rows.items():
|
||||
assert summary == EXPECTED[folder], (folder, summary)
|
||||
assert manually is False, (folder, manually)
|
||||
assert updated_at is not None, (folder, updated_at)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. ONE changed file in alpha/ → exactly alpha's chain (alpha + the
|
||||
# source root) is re-stamped; every other row is byte-identical
|
||||
# (text AND updated_at); the terminal keeps the AFFECTED count
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_one_file_change_restamps_only_that_folders_chain(
|
||||
app_url: str, incsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Re-seed (the autouse truncate) + sync #1 (full), capturing every
|
||||
row's ``(summary, updated_at)``; then ONE doc inside ``alpha/``
|
||||
changes content and sync #2 settles ``success``: the ``alpha`` AND
|
||||
the source-root rows carry a NEW ``updated_at`` (strictly after
|
||||
the captured stamp) with the canned text, while the
|
||||
``bravo``/``charlie``/``delta`` rows are byte-identical — same
|
||||
``summary`` AND same ``updated_at`` (an unaffected folder's
|
||||
documents, and hence its summary, are untouched — no re-stamp).
|
||||
The terminal keeps ``summaries_done == summaries_total == 2`` (the
|
||||
AFFECTED count — alpha + root, NOT the table's candidate count —
|
||||
the keep-final-counts convention), and the detail shows the
|
||||
single-file change."""
|
||||
_register_source(app_url, str(incsum_dir))
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == N_FILES, body["detail"]
|
||||
before = _folder_rows()
|
||||
assert set(before) == {""} | set(FOLDERS), before
|
||||
|
||||
# The single-file change: ONE alpha doc gets new content (the
|
||||
# importer's sha256 delta reports ``updated`` — no add, no prune).
|
||||
target = incsum_dir / FOLDER_A / f"{FOLDER_A}-a.md"
|
||||
target.write_text(
|
||||
_md("Incsum Alpha A (revised)", ALPHA_A_REVISED), encoding="utf-8"
|
||||
)
|
||||
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 1, detail
|
||||
assert detail["unchanged"] == N_FILES - 1, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# Terminal (the keep-final-counts convention): the AFFECTED count —
|
||||
# alpha + the source root — the phase-127 observable at the wire.
|
||||
assert body["summaries_done"] == 2, body
|
||||
assert body["summaries_total"] == 2, body
|
||||
assert body["phase"] is None, body
|
||||
assert body["current_summary"] is None, body
|
||||
|
||||
after = _folder_rows()
|
||||
assert set(after) == {""} | set(FOLDERS), after
|
||||
# The chain (alpha + the source root): RE-STAMPED — a new stamp
|
||||
# strictly after the captured one, the canned text, still
|
||||
# AI-written (the mock labels from the folder header only, so the
|
||||
# text is the same deterministic line).
|
||||
for folder in ("", FOLDER_A):
|
||||
summary, manually, updated_at = after[folder]
|
||||
assert summary == EXPECTED[folder], (folder, summary)
|
||||
assert manually is False, (folder, manually)
|
||||
assert updated_at > before[folder][2], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
# The siblings: BYTE-IDENTICAL — same summary text AND same
|
||||
# updated_at (never re-stamped, never even re-fetched for
|
||||
# generation).
|
||||
for folder in FOLDERS:
|
||||
if folder == FOLDER_A:
|
||||
continue
|
||||
assert after[folder] == before[folder], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. A vanished folder (bravo/ loses its last docs) has its row PRUNED
|
||||
# by the prune pass; the changed sibling's chain (charlie + the
|
||||
# source root) is re-stamped; the other rows are byte-identical
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_pruned_file_restamps_only_its_chain_and_prunes_the_vanished_folder(
|
||||
app_url: str, incsum_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Re-seed + sync #1 (full), capturing every row; then ``bravo/``
|
||||
loses its last docs (the folder VANISHES — 0 documents, no longer
|
||||
a candidate) and ONE doc in ``charlie/`` changes content; the sync
|
||||
settles ``success``: the ``bravo`` row is PRUNED (gone — the
|
||||
prune pass owns vanished folders' rows, independent of the
|
||||
generation's affected set), the source-root AND ``charlie`` rows
|
||||
are re-stamped (new ``updated_at`` + canned text), the
|
||||
``alpha``/``delta`` rows are byte-identical (text AND
|
||||
``updated_at``), and the terminal keeps
|
||||
``summaries_done == summaries_total == 2`` (the affected
|
||||
CANDIDATE count — charlie + root; bravo is no longer a candidate,
|
||||
so it is neither generated nor counted)."""
|
||||
_register_source(app_url, str(incsum_dir))
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == N_FILES, body["detail"]
|
||||
before = _folder_rows()
|
||||
assert set(before) == {""} | set(FOLDERS), before
|
||||
|
||||
# bravo/ loses its last docs — the folder vanishes from the
|
||||
# catalogue (0 documents → below the ≥ 1 minimum → not a
|
||||
# candidate → the prune pass owns its row).
|
||||
for letter in ("a", "b"):
|
||||
(incsum_dir / FOLDER_B / f"{FOLDER_B}-{letter}.md").unlink()
|
||||
# One charlie doc gets new content (``updated``).
|
||||
target = incsum_dir / FOLDER_C / f"{FOLDER_C}-a.md"
|
||||
target.write_text(
|
||||
_md("Incsum Charlie A (revised)", CHARLIE_A_REVISED), encoding="utf-8"
|
||||
)
|
||||
|
||||
body = _run_sync(app_url)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 0, detail
|
||||
assert detail["updated"] == 1, detail
|
||||
assert detail["unchanged"] == N_FILES - 3, detail
|
||||
assert detail["pruned"] == 2, detail
|
||||
assert detail["overview"] is True, detail
|
||||
# Terminal (the keep-final-counts convention): the affected
|
||||
# CANDIDATE count — charlie + the source root (bravo is gone from
|
||||
# the candidates, so it is not in the affected set's intersection).
|
||||
assert body["summaries_done"] == 2, body
|
||||
assert body["summaries_total"] == 2, body
|
||||
assert body["phase"] is None, body
|
||||
assert body["current_summary"] is None, body
|
||||
|
||||
after = _folder_rows()
|
||||
# The vanished folder's row is PRUNED (the prune pass — it runs in
|
||||
# both modes, manual rows exempt — bravo's row is AI-written).
|
||||
assert FOLDER_B not in after, after
|
||||
assert set(after) == ({""} | set(FOLDERS)) - {FOLDER_B}, after
|
||||
# charlie + the source root: RE-STAMPED — a new stamp strictly
|
||||
# after the captured one, the canned text, still AI-written.
|
||||
for folder in ("", FOLDER_C):
|
||||
summary, manually, updated_at = after[folder]
|
||||
assert summary == EXPECTED[folder], (folder, summary)
|
||||
assert manually is False, (folder, manually)
|
||||
assert updated_at > before[folder][2], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
# alpha + delta: BYTE-IDENTICAL — same summary text AND same
|
||||
# updated_at (neither folder is in the changed set's chains).
|
||||
for folder in (FOLDER_A, FOLDER_D):
|
||||
assert after[folder] == before[folder], (
|
||||
folder,
|
||||
before[folder],
|
||||
after[folder],
|
||||
)
|
||||
@@ -64,6 +64,17 @@ background sync task lives on the app's event loop, so the loop must
|
||||
survive across requests — exactly how the app runs under uvicorn.
|
||||
(A TestClient without the context manager starts a fresh loop per
|
||||
request and would cancel the task on request exit.)
|
||||
|
||||
Phase 127 (task 03): the changed-KB branch passes the import's change
|
||||
set to the generator (``changed_docs=summary.changed_paths`` —
|
||||
incremental regeneration over the changed set's folder keys only),
|
||||
the ``changed=N`` log line names the change count, and the status's
|
||||
``summaries_total`` is the affected count (the hook's total) rather
|
||||
than the table's candidate count. The unchanged-walk gap probe is
|
||||
orthogonal: it still calls the generator exactly as before
|
||||
(``only_missing=True``, no change set). The fakes above record the
|
||||
``changed_docs=`` kwarg (``FakeFolderSummaries.changed_docs_sets``) so
|
||||
the wiring is asserted, not inferred.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -71,7 +82,7 @@ import asyncio
|
||||
import logging
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable, Iterator
|
||||
from collections.abc import Callable, Iterator, Sequence
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -85,8 +96,9 @@ from app.api import sync as sync_api
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal, db_available
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import GitSource
|
||||
from app.models import Document, GitSource
|
||||
from app.rag import git_sources as git_sources_resolver
|
||||
from app.rag.folder_summaries import affected_folder_keys
|
||||
from app.rag.importer import ImportSummary
|
||||
from app.rag.llm import EmbeddingError, LLMClient, ModelUnavailableError
|
||||
from app.rag.sources_meta import current_sources_version
|
||||
@@ -330,6 +342,11 @@ class FakeFolderSummaries:
|
||||
hook when given — the same way the import's file counter is driven
|
||||
through its hook, so the status's summary counters are testable
|
||||
deterministically.
|
||||
Phase 127 (task 03): records the ``changed_docs=`` change set the
|
||||
runner passes on the CHANGED-KB branch (the import's
|
||||
``changed_paths`` — None when the runner regresses to the
|
||||
change-set-less full regeneration; the unchanged-walk gap-fill
|
||||
branch never passes it).
|
||||
"""
|
||||
|
||||
ZERO = {"generated": 0, "failed": 0, "pruned": 0}
|
||||
@@ -346,6 +363,7 @@ class FakeFolderSummaries:
|
||||
self.skip_flags: list[bool] = []
|
||||
self.only_missing_flags: list[bool] = []
|
||||
self.progress_hooks: list[Callable[[int, int, str, str], None] | None] = []
|
||||
self.changed_docs_sets: list[Sequence[tuple[str, str]] | None] = []
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -354,10 +372,12 @@ class FakeFolderSummaries:
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
self.skip_flags.append(skip)
|
||||
self.only_missing_flags.append(only_missing)
|
||||
self.changed_docs_sets.append(changed_docs)
|
||||
self.progress_hooks.append(on_progress)
|
||||
if skip:
|
||||
return dict(self.ZERO)
|
||||
@@ -663,6 +683,7 @@ class _GatedFolderSummaries:
|
||||
self.steps_before = steps_before
|
||||
self.steps_after = steps_after
|
||||
self.hook: Callable[[int, int, str, str], None] | None = None
|
||||
self.changed_docs: Sequence[tuple[str, str]] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
@@ -671,10 +692,12 @@ class _GatedFolderSummaries:
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
assert not skip, "these runs take a generation branch, never --limit"
|
||||
self.hook = on_progress
|
||||
self.changed_docs = changed_docs
|
||||
for done, total, source, folder_path in self.steps_before:
|
||||
if on_progress is not None:
|
||||
on_progress(done, total, source, folder_path)
|
||||
@@ -822,6 +845,7 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
self.hook = on_progress
|
||||
@@ -845,6 +869,255 @@ def test_failed_terminal_after_the_summary_hook_clears_phase_keeps_counts(
|
||||
assert failing.hook is not None # the wiring held
|
||||
|
||||
|
||||
# --- phase 127 (task 03): the changed-KB branch's incremental wiring ----
|
||||
|
||||
|
||||
def test_changed_sync_passes_the_change_set_to_the_generator(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Phase 127 (task 03): a changed-KB sync passes the import's
|
||||
change set into the generator (``changed_docs=summary.
|
||||
changed_paths`` — the incremental-regression restriction), NOT the
|
||||
gap-fill flag, the success ``detail`` keeps its exact pre-phase key
|
||||
set (the stats stay log-only), and the log line names the change
|
||||
count (``changed=N``)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
# One updated document — the import's change set carries exactly
|
||||
# its (source, path) identity (task 01's contract).
|
||||
change_set = [("repo", "notes/deep.md")]
|
||||
summary = ImportSummary(
|
||||
files=4, added=0, updated=1, unchanged=3, changed_paths=change_set
|
||||
)
|
||||
fake_import = FakeImportSources(summary)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
fake_folders = FakeFolderSummaries()
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
||||
|
||||
_login(sync_client)
|
||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# The CHANGED-KB branch fired the generator exactly once ... with
|
||||
# the change set, NOT the gap-fill flag (the two restriction
|
||||
# kwargs are orthogonal — no caller combines them).
|
||||
assert len(fake_folders.changed_docs_sets) == 1
|
||||
assert fake_folders.changed_docs_sets == [change_set]
|
||||
assert fake_folders.only_missing_flags == [False]
|
||||
assert fake_folders.skip_flags == [False]
|
||||
# The success detail is byte-identical in shape to pre-phase 127 —
|
||||
# no new key (the stats stay log-only, the phase-98 convention).
|
||||
assert set(body["detail"]) == {
|
||||
"files", "added", "updated", "unchanged", "pruned", "errors",
|
||||
"chunks", "summaries", "summary_errors", "dates_updated",
|
||||
"overview", "sources_version",
|
||||
}
|
||||
assert body["detail"]["added"] == 0 and body["detail"]["updated"] == 1
|
||||
assert body["detail"]["overview"] is True
|
||||
assert body["detail"]["sources_version"] == 1 # changed KB → one bump
|
||||
# The ample-logging line names the change count (PLAN §9).
|
||||
assert any(
|
||||
r.getMessage() == (
|
||||
"sync: folder_summaries changed=1 "
|
||||
"stats={'generated': 0, 'failed': 0, 'pruned': 0}"
|
||||
)
|
||||
for r in caplog.records
|
||||
)
|
||||
|
||||
|
||||
def test_unchanged_walk_gap_probe_passes_no_change_set(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Phase 127 (task 03): the unchanged-walk gap probe is
|
||||
ORTHOGONAL to the change set — a walk that changed nothing with a
|
||||
real gap (seeded catalogue, empty ``folder_summaries`` table) still
|
||||
calls the generator exactly as before: ``only_missing=True`` and
|
||||
NO ``changed_docs`` (the gap probe's total is the missing count,
|
||||
as before)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
# The gap: two docs under notes/ (candidates root + notes), the
|
||||
# folder_summaries table empty (the autouse fixture truncated it).
|
||||
db.add_all([
|
||||
Document(
|
||||
source="repo", path="notes/one.md", full_path="/tmp/one.md",
|
||||
title="One", content="# One\nfirst\n", content_hash="a" * 64,
|
||||
),
|
||||
Document(
|
||||
source="repo", path="notes/two.md", full_path="/tmp/two.md",
|
||||
title="Two", content="# Two\nsecond\n", content_hash="b" * 64,
|
||||
),
|
||||
])
|
||||
db.commit()
|
||||
summary = ImportSummary(files=2, added=0, updated=0, unchanged=2)
|
||||
fake_import = FakeImportSources(summary)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
fake_folders = FakeFolderSummaries()
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folders)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# The gap probe fired the generator exactly once — the gap-fill
|
||||
# shape, byte-identical to pre-phase 127: only_missing=True, and
|
||||
# NO change set (the gap is a stored-rows gap, not a document
|
||||
# change — passing changed_paths here would restrict the fill to
|
||||
# nothing).
|
||||
assert len(fake_folders.changed_docs_sets) == 1
|
||||
assert fake_folders.changed_docs_sets == [None]
|
||||
assert fake_folders.only_missing_flags == [True]
|
||||
assert body["detail"]["overview"] is False # unchanged → no overview
|
||||
assert body["detail"]["sources_version"] == 0 # unchanged → no bump
|
||||
|
||||
|
||||
class _AffectedKeysFolderSummaries:
|
||||
"""A folder-summary step that mirrors the REAL generator's hook
|
||||
contract on a changed set (phase 127, task 02): the loop runs over
|
||||
the sorted AFFECTED keys (``affected_folder_keys(changed_docs)``),
|
||||
the hook fires once per key with ``total == len(keys)``, and the
|
||||
step parks mid-span after the first key — the (long) phase the
|
||||
user reported. The test reads the status mid-span and asserts the
|
||||
status's ``summaries_total`` IS the affected count (the hook's
|
||||
total), not the table's candidate count."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
started: threading.Event,
|
||||
release: threading.Event,
|
||||
) -> None:
|
||||
self.started = started
|
||||
self.release = release
|
||||
self.changed_docs: Sequence[tuple[str, str]] | None = None
|
||||
self.keys: list[tuple[str, str]] = []
|
||||
self.hook: Callable[[int, int, str, str], None] | None = None
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
db: Session,
|
||||
llm: LLMClient,
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None,
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
assert not skip, "these runs take a generation branch, never --limit"
|
||||
self.hook = on_progress
|
||||
self.changed_docs = changed_docs
|
||||
assert on_progress is not None, "the runner must wire the hook"
|
||||
# The real generator's step 2: sorted(candidates & affected) —
|
||||
# the fake takes the affected side (the candidates are a
|
||||
# superset here: every affected folder holds a doc).
|
||||
self.keys = sorted(affected_folder_keys(changed_docs or []))
|
||||
assert self.keys, "the change set must affect at least the root"
|
||||
on_progress(1, len(self.keys), self.keys[0][0], self.keys[0][1])
|
||||
self.started.set()
|
||||
await asyncio.to_thread(self.release.wait)
|
||||
for i, (source, folder_path) in enumerate(self.keys[1:], start=2):
|
||||
on_progress(i, len(self.keys), source, folder_path)
|
||||
return {"generated": len(self.keys), "failed": 0, "pruned": 0}
|
||||
|
||||
|
||||
def test_summaries_phase_reports_the_affected_count_as_total(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
"""Phase 127 (task 03): mid-span, the status's ``summaries_total``
|
||||
is the AFFECTED count — ``len(affected_folder_keys(changed set))``
|
||||
(the hook's total), not the table's candidate count — and the
|
||||
terminal keeps that final count (the phase-64 convention, now over
|
||||
the affected span). The change set: one nested-folder doc (a
|
||||
three-key chain) + one root-level doc (shares the root key) →
|
||||
exactly 3 affected keys."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
_, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_stub_probe(monkeypatch)
|
||||
change_set = [("repo", "notes/a/one.md"), ("repo", "top.md")]
|
||||
fake_import = FakeImportSources(
|
||||
ImportSummary(
|
||||
files=6, added=1, updated=2, unchanged=3, changed_paths=change_set
|
||||
)
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
|
||||
gated_folders = _AffectedKeysFolderSummaries(
|
||||
threading.Event(), threading.Event()
|
||||
)
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", gated_folders)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
try:
|
||||
assert gated_folders.started.wait(5.0), "the run never reached the summaries"
|
||||
s = sync_client.get("/api/sync/status").json()
|
||||
assert s["state"] == "running"
|
||||
assert s["phase"] == "summaries"
|
||||
# The hook's first key: the source root (the bare source name —
|
||||
# folder_path "" never gets a slash) ...
|
||||
assert s["current_summary"] == "repo"
|
||||
assert s["summaries_done"] == 1
|
||||
# ... and the total IS the affected count (the task's pin):
|
||||
# notes/a/one.md → ("repo", ""), ("repo", "notes"),
|
||||
# ("repo", "notes/a"); top.md → ("repo", "") — 3 keys.
|
||||
expected = sorted(affected_folder_keys(change_set))
|
||||
assert expected == [("repo", ""), ("repo", "notes"), ("repo", "notes/a")]
|
||||
assert s["summaries_total"] == len(expected)
|
||||
assert s["summaries_total"] == 3
|
||||
gated_folders.release.set()
|
||||
finally:
|
||||
gated_folders.release.set()
|
||||
|
||||
body = _poll(sync_client, "success")
|
||||
# Terminal: phase + current_summary cleared, the hook's FINAL
|
||||
# (affected) counts kept (the phase-64 keep-final-counts rule, now
|
||||
# over the affected span).
|
||||
assert body["phase"] is None and body["current_summary"] is None
|
||||
assert body["summaries_done"] == 3 and body["summaries_total"] == 3
|
||||
# Wiring: the runner passed the change set, not the gap-fill flag.
|
||||
assert gated_folders.changed_docs == change_set
|
||||
assert gated_folders.hook is not None
|
||||
|
||||
|
||||
# --- admin: failures -------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
@@ -29,6 +29,7 @@ from app.rag.folder_summaries import (
|
||||
FOLDER_SUMMARY_MODE,
|
||||
MIN_DOCS_PER_FOLDER,
|
||||
SYSTEM_PROMPT,
|
||||
affected_folder_keys,
|
||||
build_folder_summary_prompt,
|
||||
folder_of,
|
||||
generate_folder_summaries,
|
||||
@@ -202,6 +203,56 @@ def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
|
||||
assert "top.md" not in [folder for _source, folder in groups]
|
||||
|
||||
|
||||
# ---------- affected_folder_keys (phase 127, task 02) ----------
|
||||
|
||||
|
||||
def test_affected_folder_keys_root_level_file_is_only_the_root() -> None:
|
||||
"""A root-level file (``folder_of == ""``) affects ONLY the source
|
||||
root — there is no folder prefix to walk."""
|
||||
assert affected_folder_keys([("S", "a.md")]) == {("S", "")}
|
||||
|
||||
|
||||
def test_affected_folder_keys_nested_file_walks_the_folder_chain() -> None:
|
||||
"""A doc under ``a/b/`` affects ``a/b``, ``a``, and the source
|
||||
root — the SAME prefix walk ``group_by_folder`` pass 1 uses (one
|
||||
concept)."""
|
||||
assert affected_folder_keys([("S", "a/b/c.md")]) == {
|
||||
("S", ""),
|
||||
("S", "a"),
|
||||
("S", "a/b"),
|
||||
}
|
||||
|
||||
|
||||
def test_affected_folder_keys_multi_source_set_semantics() -> None:
|
||||
"""Keys are per source (the PK is ``(source, folder_path)``), and
|
||||
the result is a SET — a repeated chain adds no new keys, and an
|
||||
unrelated source keeps its own root."""
|
||||
keys = affected_folder_keys(
|
||||
[
|
||||
("S", "a/b/c.md"),
|
||||
("T", "x.md"),
|
||||
("S", "a/d.md"), # the same chain as the first entry — no new keys
|
||||
]
|
||||
)
|
||||
assert keys == {("S", ""), ("S", "a"), ("S", "a/b"), ("T", "")}
|
||||
|
||||
|
||||
def test_affected_folder_keys_fragment_paths_fold_into_their_parent_folder() -> None:
|
||||
"""Embedded-image fragment paths (phase 126: ``x.pdf#page=…#img=…``)
|
||||
get NO special case: the fragment never holds a ``/``, so
|
||||
``folder_of`` folds them into the parent folder's chain (a
|
||||
root-level fragment affects only the source root)."""
|
||||
assert affected_folder_keys([("S", "dir/img.pdf#page=2#img=0")]) == {
|
||||
("S", ""),
|
||||
("S", "dir"),
|
||||
}
|
||||
assert affected_folder_keys([("S", "img.pdf#page=1#img=0")]) == {("S", "")}
|
||||
|
||||
|
||||
def test_affected_folder_keys_empty_input_is_an_empty_set() -> None:
|
||||
assert affected_folder_keys([]) == set()
|
||||
|
||||
|
||||
# ---------- build_folder_summary_prompt: system ----------
|
||||
|
||||
|
||||
@@ -1093,3 +1144,332 @@ def test_on_progress_none_is_a_zero_cost_noop(db: Session, clean_tables) -> None
|
||||
("FSU", "a/b"): REPLY,
|
||||
("FSU-solo", ""): REPLY,
|
||||
}
|
||||
|
||||
|
||||
# ---------- generate_folder_summaries(changed_docs=…) (phase 127, task 02) ----------
|
||||
|
||||
|
||||
def _seed_three_sources(db: Session) -> None:
|
||||
"""The restricted-mode catalogue (three sources): A is nested
|
||||
(``a/`` and ``a/b/``), B is a sibling single-folder source, and C
|
||||
is a single-doc root — six candidate keys total: A ×
|
||||
{``""``, ``a``, ``a/b``}, B × {``""``, ``b``}, C × {``""``}.
|
||||
"""
|
||||
_add_doc(db, "A", "a/b/one.md", "A One", "A one lead.")
|
||||
_add_doc(db, "A", "a/b/two.md", "A Two")
|
||||
_add_doc(db, "A", "a/three.md", "A Three")
|
||||
_add_doc(db, "B", "b/one.md", "B One")
|
||||
_add_doc(db, "C", "solo.md", "C Solo")
|
||||
|
||||
|
||||
def _stamps(
|
||||
db: Session, keys: set[tuple[str, str]]
|
||||
) -> dict[tuple[str, str], object]:
|
||||
"""The stored rows' ``updated_at`` for *keys* (raw SQL — the
|
||||
byte-identity comparisons stay honest)."""
|
||||
return {
|
||||
(source, folder): _updated_at(db, source, folder) for source, folder in keys
|
||||
}
|
||||
|
||||
|
||||
def _freeze(
|
||||
db: Session, values: dict[tuple[str, str], str]
|
||||
) -> dict[tuple[str, str], object]:
|
||||
"""Overwrite the stored summaries for *values*' keys with DISTINCT
|
||||
frozen text (a "regenerated" row is then distinguishable from an
|
||||
"untouched" one), and return the pre-run stamps."""
|
||||
for (source, folder_path), value in values.items():
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE folder_summaries SET summary = :v"
|
||||
" WHERE source = :s AND folder_path = :f"
|
||||
),
|
||||
{"v": value, "s": source, "f": folder_path},
|
||||
)
|
||||
db.commit()
|
||||
return _stamps(db, set(values))
|
||||
|
||||
|
||||
def test_changed_docs_restricted_regeneration_only_affected_keys(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""One changed file in A/``a/b/`` → exactly that file's folder
|
||||
chain (``a/b``, ``a``, and the A root) regenerates: one lite call
|
||||
per affected non-manual key, in sorted order, fresh rows. Every
|
||||
OTHER stored row — same-source siblings and other sources — keeps
|
||||
its summary text AND ``updated_at`` byte-identical (zero lite
|
||||
calls for them)."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
frozen = {
|
||||
("A", ""): "A root frozen",
|
||||
("A", "a"): "A/a frozen",
|
||||
("A", "a/b"): "A/a/b frozen",
|
||||
("B", ""): "B root frozen",
|
||||
("B", "b"): "B/b frozen",
|
||||
("C", ""): "C root frozen",
|
||||
}
|
||||
stamps = _freeze(db, frozen)
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 3, "one lite call per AFFECTED non-manual key — nothing else"
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: A",
|
||||
"Folder: A/a",
|
||||
"Folder: A/a/b",
|
||||
], "the affected chain in sorted (source, folder_path) order"
|
||||
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "")] == REPLY
|
||||
assert stored[("A", "a")] == REPLY
|
||||
assert stored[("A", "a/b")] == REPLY, "the changed file's own folder regenerates"
|
||||
assert stored[("B", "")] == "B root frozen"
|
||||
assert stored[("B", "b")] == "B/b frozen"
|
||||
assert stored[("C", "")] == "C root frozen", (
|
||||
"the unaffected rows keep their stored text"
|
||||
)
|
||||
for key in (("B", ""), ("B", "b"), ("C", "")):
|
||||
assert _updated_at(db, *key) == stamps[key], (
|
||||
"the unaffected rows are never re-stamped"
|
||||
)
|
||||
|
||||
|
||||
def test_changed_docs_manual_affected_row_is_skipped(db: Session, clean_tables) -> None:
|
||||
"""A manual row INSIDE the affected chain is skipped exactly as in
|
||||
full mode (phase 97): ``kept_manual``, no lite call, text AND
|
||||
``updated_at`` byte-identical — the restriction changes the key
|
||||
set, not the manual-row rule."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
manual_text = "Owner's words about A/a."
|
||||
row = db.get(FolderSummary, ("A", "a"))
|
||||
assert row is not None
|
||||
row.summary = manual_text
|
||||
row.manually_edited = True
|
||||
db.commit()
|
||||
stamp_before = _updated_at(db, "A", "a")
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 2, "failed": 0, "pruned": 0, "kept_manual": 1}
|
||||
assert llm.calls == 2, "the manual key burns zero lite calls"
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: A",
|
||||
"Folder: A/a/b",
|
||||
]
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "a")] == manual_text, "the owner's text is never rewritten"
|
||||
assert _updated_at(db, "A", "a") == stamp_before, ("never re-stamped")
|
||||
assert stored[("A", "")] == REPLY and stored[("A", "a/b")] == REPLY
|
||||
assert stored[("B", "")] == REPLY and stored[("B", "b")] == REPLY
|
||||
assert stored[("C", "")] == REPLY, "the unaffected rows are left as they were"
|
||||
|
||||
|
||||
def test_changed_docs_failing_affected_folder_fails_soft(db: Session, clean_tables) -> None:
|
||||
"""Per-folder fail-soft under the restriction: the failing
|
||||
affected key is attempted and counted, its PREVIOUS row is kept
|
||||
(an old summary is better than none), the other affected keys
|
||||
still land, and the unaffected rows are untouched."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
# A DISTINCT previous text on the failing folder, so "kept" is
|
||||
# visible (it would otherwise be the REPLY either way).
|
||||
db.execute(
|
||||
text(
|
||||
"UPDATE folder_summaries SET summary = 'old A/a/b'"
|
||||
" WHERE (source, folder_path) = ('A', 'a/b')"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
stamps = _stamps(db, {("B", ""), ("B", "b"), ("C", "")})
|
||||
|
||||
llm = _FakeLLM(fail_folders=("A/a/b",))
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 2, "failed": 1, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 3, "the failing key is attempted (the hook's premise)"
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "a/b")] == "old A/a/b", "the previous row survives"
|
||||
assert stored[("A", "")] == REPLY and stored[("A", "a")] == REPLY
|
||||
for key, stamp in stamps.items():
|
||||
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
|
||||
|
||||
|
||||
def test_changed_docs_multi_source_change_set(db: Session, clean_tables) -> None:
|
||||
"""A change set spanning SOURCES: the affected keys are the UNION
|
||||
of each document's chain (A's nested chain + B's root and ``b``),
|
||||
sorted across sources; the unaffected C root row keeps its text
|
||||
AND ``updated_at``."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
c_stamp = _updated_at(db, "C", "")
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(
|
||||
db, llm, changed_docs=[("A", "a/b/two.md"), ("B", "b/one.md")]
|
||||
)
|
||||
)
|
||||
assert stats == {"generated": 5, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 5
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == [
|
||||
"Folder: A",
|
||||
"Folder: A/a",
|
||||
"Folder: A/a/b",
|
||||
"Folder: B",
|
||||
"Folder: B/b",
|
||||
]
|
||||
stored = _rows(db)
|
||||
assert stored[("C", "")] == REPLY
|
||||
assert _updated_at(db, "C", "") == c_stamp
|
||||
|
||||
|
||||
def test_changed_docs_gap_in_affected_folder_is_generated(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""An affected folder with NO stored row (a gap in the chain) is a
|
||||
candidate and gets generated like any other affected key — while a
|
||||
gap in an UNAFFECTED folder stays the unchanged-walk gap probe's
|
||||
job (phase 96): the restriction never widens beyond the affected
|
||||
set."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.execute(
|
||||
text(
|
||||
"DELETE FROM folder_summaries"
|
||||
" WHERE (source, folder_path) IN (('A', 'a/b'), ('B', ''))"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "a/b/one.md")])
|
||||
)
|
||||
assert stats == {"generated": 3, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 3
|
||||
stored = _rows(db)
|
||||
assert stored[("A", "a/b")] == REPLY, "the gap in the affected chain is filled"
|
||||
assert ("B", "") not in stored, (
|
||||
"the gap in the UNAFFECTED source is left for the gap probe"
|
||||
)
|
||||
assert stored[("B", "b")] == REPLY
|
||||
|
||||
|
||||
def test_changed_docs_vanished_affected_folder_is_pruned_not_generated(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""The ASSUMPTION pinned: an affected folder that no longer holds
|
||||
any document is a PRUNE-pass job, not a generation's — the pruned
|
||||
document's chain (root + its now-vanished folder) is in the change
|
||||
set, only the surviving candidate (the root) generates, the
|
||||
vanished folder's row is dropped, and the unaffected rows keep
|
||||
their ``updated_at``."""
|
||||
_add_doc(db, "A", "a/b/one.md", "A One")
|
||||
_add_doc(db, "A", "a/b/two.md", "A Two")
|
||||
_add_doc(db, "A", "gone/old.md", "Gone")
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
stamps = _stamps(db, {("A", "a"), ("A", "a/b")})
|
||||
|
||||
db.execute(text("DELETE FROM documents WHERE path = 'gone/old.md'"))
|
||||
db.commit()
|
||||
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[("A", "gone/old.md")])
|
||||
)
|
||||
# affected = {("A", ""), ("A", "gone")} — the only surviving
|
||||
# candidate in that set is the source root.
|
||||
assert stats == {"generated": 1, "failed": 0, "pruned": 1, "kept_manual": 0}
|
||||
assert llm.calls == 1
|
||||
assert [user.splitlines()[0] for _s, user in llm.requests] == ["Folder: A"]
|
||||
stored = _rows(db)
|
||||
assert ("A", "gone") not in stored, "the vanished folder's row is pruned"
|
||||
assert stored[("A", "")] == REPLY, "the root's subtree changed — regenerated"
|
||||
assert stored[("A", "a")] == REPLY and stored[("A", "a/b")] == REPLY
|
||||
for key, stamp in stamps.items():
|
||||
assert _updated_at(db, *key) == stamp, "the unaffected rows are untouched"
|
||||
|
||||
|
||||
def test_changed_docs_hook_total_is_the_affected_count(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""The progress hook (phase 98) under the restriction fires exactly
|
||||
``len(affected)`` times with ``total == len(affected)`` — the sync
|
||||
status's ``summaries_total`` then reports the affected count
|
||||
through the existing hook — in sorted key order."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(
|
||||
db, llm, changed_docs=[("A", "a/b/one.md")], on_progress=record
|
||||
)
|
||||
)
|
||||
assert events == [
|
||||
(1, 3, "A", ""),
|
||||
(2, 3, "A", "a"),
|
||||
(3, 3, "A", "a/b"),
|
||||
]
|
||||
assert llm.calls == 3
|
||||
assert stats["generated"] == 3
|
||||
|
||||
|
||||
def test_changed_docs_none_is_the_pre_127_full_regeneration(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""The default (no ``changed_docs`` kwarg) is the pre-127 FULL
|
||||
regeneration over the same three-source catalogue: every
|
||||
candidate key, one call each, the 4-field stats shape unchanged —
|
||||
the ``scripts/import_docs.py`` CLI contract."""
|
||||
_seed_three_sources(db)
|
||||
llm = _FakeLLM()
|
||||
stats = asyncio.run(generate_folder_summaries(db, llm))
|
||||
assert stats == {"generated": 6, "failed": 0, "pruned": 0, "kept_manual": 0}
|
||||
assert llm.calls == 6, "no kwarg → every candidate burns one call"
|
||||
assert set(_rows(db)) == {
|
||||
("A", ""), ("A", "a"), ("A", "a/b"),
|
||||
("B", ""), ("B", "b"),
|
||||
("C", ""),
|
||||
}
|
||||
|
||||
|
||||
def test_changed_docs_empty_set_generates_nothing_but_still_prunes(
|
||||
db: Session, clean_tables
|
||||
) -> None:
|
||||
"""``changed_docs=[]`` (a change set that IS given but empty)
|
||||
restricts the generation to zero keys — no lite calls, no
|
||||
re-stamps, the hook never fires — while the prune pass still runs
|
||||
in the given mode (the sweep is mode-independent, phase 97)."""
|
||||
_seed_three_sources(db)
|
||||
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
|
||||
db.commit()
|
||||
db.add(FolderSummary(source="A", folder_path="gone/old", summary="stale"))
|
||||
db.commit()
|
||||
stamps = _stamps(db, {("A", ""), ("A", "a"), ("A", "a/b"), ("B", ""), ("C", "")})
|
||||
|
||||
llm = _FakeLLM()
|
||||
events, record = _record_progress()
|
||||
stats = asyncio.run(
|
||||
generate_folder_summaries(db, llm, changed_docs=[], on_progress=record)
|
||||
)
|
||||
assert stats == {"generated": 0, "failed": 0, "pruned": 1, "kept_manual": 0}
|
||||
assert llm.calls == 0
|
||||
assert events == [], "zero keys → the hook never fires"
|
||||
for key, stamp in stamps.items():
|
||||
assert _updated_at(db, *key) == stamp, "no row is re-stamped"
|
||||
|
||||
@@ -15,9 +15,11 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pymupdf as fitz
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
@@ -28,6 +30,7 @@ from app.models import Chunk, Document
|
||||
from app.rag.importer import (
|
||||
EXCLUDED_DIRS,
|
||||
ImportSummary,
|
||||
_prune,
|
||||
_store_summary,
|
||||
import_sources,
|
||||
iter_importable_files,
|
||||
@@ -1194,3 +1197,236 @@ def test_progress_hook_with_limit_keeps_full_total(db, tmp_path: Path) -> None:
|
||||
assert events == [(root.name, "a.md", 1, 3), (root.name, "b.md", 2, 3)]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 127 (task 01): ImportSummary.changed_paths ----------
|
||||
|
||||
|
||||
class _VisionLLM(FakeEmbedder):
|
||||
"""The phase-122 mock VISION client (the test_pdf pattern): the
|
||||
multimodal describe call (``user`` content = a part list) gets a fixed
|
||||
description; text (``lite``) calls keep the ``FakeEmbedder``
|
||||
behaviour."""
|
||||
|
||||
DESCRIPTION = "A test image: a solid swatch of color."
|
||||
|
||||
async def chat(self, messages, model=None): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.chat_calls.append(list(messages))
|
||||
user = next((m["content"] for m in messages if m.get("role") == "user"), "")
|
||||
if isinstance(user, list):
|
||||
return self.DESCRIPTION
|
||||
first = user.split()
|
||||
return "Summary of " + (first[0] if first else "<empty>")
|
||||
|
||||
|
||||
def _png_bytes(width: int, height: int, rgb: tuple[float, float, float]) -> bytes:
|
||||
"""One solid-color PNG built with PyMuPDF (the phase-126 no-binary-
|
||||
fixtures convention)."""
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, width, height), 0)
|
||||
for y in range(height):
|
||||
for x in range(width):
|
||||
pix.set_pixel(x, y, rgb)
|
||||
return pix.tobytes("png")
|
||||
|
||||
|
||||
def _build_one_page_pdf(path: Path, text: str) -> None:
|
||||
"""One temp single-page PDF (the phase-126 convention — no binary
|
||||
fixtures)."""
|
||||
doc = fitz.open()
|
||||
doc.new_page().insert_text((72, 72), text)
|
||||
doc.save(path)
|
||||
doc.close()
|
||||
|
||||
|
||||
def test_import_summary_changed_paths_defaults_empty() -> None:
|
||||
"""Completion criterion: default construction gives an empty change
|
||||
set — no existing construction site migrates."""
|
||||
assert ImportSummary().changed_paths == []
|
||||
|
||||
|
||||
def test_changed_paths_records_added_updated_and_pruned_in_order(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""The contract (phase 127, task 01): one ``(source, path)`` entry per
|
||||
ADDED, UPDATED, or PRUNED document — added/updated in walk order, the
|
||||
pruned pair last (``_prune`` runs after the walk) — and
|
||||
``len(changed_paths) == added + updated + pruned``."""
|
||||
root = tmp_path / "changed"
|
||||
root.mkdir()
|
||||
for name in ("a.md", "b.md", "c.md"):
|
||||
(root / name).write_text(f"# {name}\n\nv1\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
source = root.name
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 3 and first.pruned == 0
|
||||
assert first.changed_paths == [
|
||||
(source, "a.md"),
|
||||
(source, "b.md"),
|
||||
(source, "c.md"),
|
||||
]
|
||||
# Run 2: update a.md, add d.md + e.md, delete c.md (→ pruned).
|
||||
(root / "a.md").write_text("# a\n\nv2 changed\n")
|
||||
(root / "d.md").write_text("# d\n\nnew d\n")
|
||||
(root / "e.md").write_text("# e\n\nnew e\n")
|
||||
(root / "c.md").unlink()
|
||||
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
|
||||
assert (second.added, second.updated, second.unchanged, second.pruned) == (
|
||||
2, 1, 1, 1,
|
||||
)
|
||||
assert second.changed_paths == [
|
||||
(source, "a.md"), # updated — first in walk order
|
||||
(source, "d.md"), # added
|
||||
(source, "e.md"), # added
|
||||
(source, "c.md"), # pruned — the _prune pass runs after the walk
|
||||
]
|
||||
assert len(second.changed_paths) == (
|
||||
second.added + second.updated + second.pruned
|
||||
)
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_empty_on_unchanged_rerun(db, tmp_path: Path) -> None:
|
||||
"""An all-unchanged re-run (prune included) records nothing."""
|
||||
root = tmp_path / "nochange"
|
||||
root.mkdir()
|
||||
(root / "a.md").write_text("# a\n\nstable\n")
|
||||
(root / "b.md").write_text("# b\n\nstable\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
second = asyncio.run(import_sources([root], llm, session=db, prune=True))
|
||||
assert second.unchanged == 2
|
||||
assert second.changed_paths == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_empty_on_backfill_rerun(db, tmp_path: Path) -> None:
|
||||
"""A backfill (phase 118: unchanged doc, NULL summary) is NOT a content
|
||||
change: the add still records, the backfill re-run stays empty."""
|
||||
root = tmp_path / "backfill"
|
||||
root.mkdir()
|
||||
(root / "a.yaml").write_text("alpha x\n")
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], _FailingChatEmbedder(), session=db))
|
||||
assert first.added == 1 and first.summary_errors == 1
|
||||
assert first.changed_paths == [(root.name, "a.yaml")] # the ADD records
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "a.yaml")
|
||||
)
|
||||
assert doc is not None and doc.summary is None
|
||||
second = asyncio.run(import_sources([root], FakeEmbedder(), session=db))
|
||||
assert second.unchanged == 1 and second.summary_backfilled == 1
|
||||
assert second.changed_paths == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_empty_on_date_only_rerun(db, tmp_path: Path) -> None:
|
||||
"""A date-only refresh (phase 106, D4: content untouched, mtime moved)
|
||||
is NOT a content change: the re-run records nothing."""
|
||||
root = tmp_path / "dateonly"
|
||||
root.mkdir()
|
||||
path = root / "a.md"
|
||||
path.write_text("# a\n\nstable\n")
|
||||
os.utime(path, (0, datetime(2020, 1, 1, tzinfo=UTC).timestamp()))
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 1 and first.dates_updated == 0
|
||||
assert first.changed_paths == [(root.name, "a.md")]
|
||||
# The content is untouched — only the mtime moved (the D4
|
||||
# date-refresh on the unchanged path).
|
||||
os.utime(path, (0, datetime(2021, 6, 15, tzinfo=UTC).timestamp()))
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.unchanged == 1 and second.dates_updated == 1
|
||||
assert second.changed_paths == []
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_records_a_changed_image(db, tmp_path: Path) -> None:
|
||||
"""The image increment site (phase 122 path): a NEW image and a
|
||||
byte-CHANGED image each record their ``(source, rel)``."""
|
||||
root = tmp_path / "imgsrc"
|
||||
root.mkdir()
|
||||
(root / "pic.png").write_bytes(_png_bytes(1, 1, (1.0, 0.0, 0.0)))
|
||||
llm = _VisionLLM()
|
||||
llm.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
)
|
||||
try:
|
||||
source = root.name
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 1 and first.images_failed == 0
|
||||
assert first.changed_paths == [(source, "pic.png")]
|
||||
# The BYTES changed (a new color) — the image digest is over the
|
||||
# bytes, so this is an UPDATE.
|
||||
(root / "pic.png").write_bytes(_png_bytes(1, 1, (0.0, 1.0, 1.0)))
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.updated == 1
|
||||
assert second.changed_paths == [(source, "pic.png")]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_changed_paths_records_a_changed_pdf(db, tmp_path: Path) -> None:
|
||||
"""The PDF increment site (phase 126 path): a NEW PDF and a text-
|
||||
CHANGED PDF each record their ``(source, rel)``."""
|
||||
root = tmp_path / "pdfsrc"
|
||||
root.mkdir()
|
||||
pdf_path = root / "doc.pdf"
|
||||
_build_one_page_pdf(pdf_path, "alpha page one")
|
||||
llm = _VisionLLM()
|
||||
llm.settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
images=True,
|
||||
image_dir=str(tmp_path / "images"),
|
||||
import_extensions="md,txt,pdf",
|
||||
)
|
||||
try:
|
||||
source = root.name
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert first.added == 1
|
||||
assert first.changed_paths == [(source, "doc.pdf")]
|
||||
# The extracted TEXT changed — the PDF digest is over it, so this
|
||||
# is an UPDATE.
|
||||
_build_one_page_pdf(pdf_path, "bravo page one, revised")
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.updated == 1
|
||||
assert second.changed_paths == [(source, "doc.pdf")]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_prune_records_changed_paths_only_when_summary_given(db, tmp_path: Path) -> None:
|
||||
"""The ``_prune`` recording seam: with NO summary (the pre-phase
|
||||
count-only contract) nothing is recorded; with one, each ACTUALLY
|
||||
deleted doc appends its ``(source, path)`` (skipped docs never do)."""
|
||||
root = tmp_path / "prunit"
|
||||
root.mkdir()
|
||||
(root / "gone.md").write_text("# gone\n\nbye\n")
|
||||
(root / "gone2.md").write_text("# gone2\n\nbye\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
source = root.name
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
(root / "gone.md").unlink()
|
||||
(root / "gone2.md").unlink()
|
||||
# Default call (summary=None): count only, no recording.
|
||||
pruned = _prune(db, {source}, {(source, "gone2.md")})
|
||||
assert pruned == 1
|
||||
assert db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == "gone.md")
|
||||
) is None
|
||||
# With a summary: the deleted doc records its pair.
|
||||
counters = ImportSummary()
|
||||
pruned = _prune(db, {source}, set(), summary=counters)
|
||||
assert pruned == 1
|
||||
assert counters.changed_paths == [(source, "gone2.md")]
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -40,7 +40,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import re
|
||||
import threading
|
||||
from collections.abc import Callable, Iterator
|
||||
from collections.abc import Callable, Iterator, Sequence
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
@@ -819,12 +819,15 @@ def _patch_sync_seams(
|
||||
*,
|
||||
skip: bool = False,
|
||||
only_missing: bool = False,
|
||||
changed_docs: Sequence[tuple[str, str]] | None = None, # phase 127
|
||||
on_progress: Callable[[int, int, str, str], None] | None = None,
|
||||
) -> dict[str, int]:
|
||||
# Phase 98 (task 01): the runner wires the progress hook into
|
||||
# the generation branches — the fake accepts it (the state-
|
||||
# machine tests' canned run never fires it: zero summary
|
||||
# counters, like a no-gap skip).
|
||||
# counters, like a no-gap skip). Phase 127 (task 03): the
|
||||
# changed-KB branch passes the change set as changed_docs= —
|
||||
# the fake accepts it (these canned runs record nothing).
|
||||
return {"generated": 0, "failed": 0, "pruned": 0}
|
||||
|
||||
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
|
||||
|
||||
Reference in New Issue
Block a user