phase: 127_sync_incremental_folder_summaries
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Build and Push Containers / build-and-push-db (push) Successful in 15s

All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.

**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)

- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
2026-09-27 21:12:25 -04:00
parent f50d4c8bc3
commit 6e654df387
38 changed files with 2741 additions and 46 deletions
+75 -6
View File
@@ -59,11 +59,22 @@ the stored keys — one concept, as with :func:`group_by_folder`), and
:func:`generate_folder_summaries`'s ``only_missing=True`` fills
EXACTLY those on the next sync — existing rows stay
byte-identical (text AND ``updated_at``), the prune pass still runs.
Restricted regeneration (phase 127, task 02): a changed-KB sync no
longer rewrites EVERY folder — the sync path passes the run's changed
documents (``ImportSummary.changed_paths``) as ``changed_docs=`` and
the generator then rewrites ONLY the candidate folders the change
touches — each document's folder chain up to the source root
(:func:`affected_folder_keys`, the SAME prefix walk
:func:`group_by_folder` pass 1 uses — one concept); every other
stored row keeps its text AND ``updated_at``. The
``scripts/import_docs.py`` CLI keeps the byte-identical full
regeneration (``changed_docs=None``).
"""
from __future__ import annotations
import logging
from collections.abc import Callable, Sequence
from collections.abc import Callable, Iterable, Sequence
from datetime import UTC, datetime
from typing import Protocol
@@ -209,6 +220,32 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
return groups
def affected_folder_keys(
changed: Iterable[tuple[str, str]],
) -> set[tuple[str, str]]:
"""The candidate folder keys a changed-document set invalidates.
For each ``(source, path)``: the source-root key ``(source, "")``
plus every folder prefix of *path* — :func:`folder_of` walked from
the document's nearest folder up to ``""`` — the SAME prefix walk
:func:`group_by_folder` pass 1 uses (one concept): a doc under
``a/b/`` affects ``a/b``, ``a``, and the source root. A
root-level file (``folder_of(path) == ""``) affects ONLY the
source root. Embedded-image fragment paths (``x.pdf#page=…#img=…``,
phase 126) fold into their parent's folder chain with no special
case — ``folder_of`` never sees a ``/`` inside the fragment.
Returns a set; the generator sorts it (deterministic order).
"""
keys: set[tuple[str, str]] = set()
for source, path in changed:
keys.add((source, ""))
folder = folder_of(path)
while folder:
keys.add((source, folder))
folder = folder_of(folder)
return keys
def _first_summary_line(summary: str | None) -> str:
"""First line of a stored summary, stripped; ``''`` when absent.
@@ -406,13 +443,34 @@ async def generate_folder_summaries(
*,
skip: bool = False,
only_missing: bool = False,
changed_docs: Sequence[tuple[str, str]] | None = None,
on_progress: Callable[[int, int, str, str], None] | None = None,
) -> dict[str, int]:
"""Regenerate the stored folder summaries for the current catalogue.
The sync-path orchestrator (phase 94, task 02 calls it
change-gated, like the KB overview; ``--limit`` debug runs pass
``skip=True``). Steps, in order:
``skip=True``).
Modes (phase 127, task 02 — the *changed_docs* kwarg):
``changed_docs=None`` (the DEFAULT) is the FULL regeneration —
every candidate folder, byte-identical to the pre-127 behavior;
the ``scripts/import_docs.py`` CLI stays on it. A GIVEN change set
(the sync path passes ``ImportSummary.changed_paths``) restricts
the generation to the CANDIDATE folders the changed documents
touch — :func:`affected_folder_keys`: a document's change
invalidates exactly its folder chain + source root, the folders
whose subtree summary text CAN change; every other stored row
keeps its summary text AND ``updated_at`` (an unaffected folder's
documents — and hence its summary — are untouched). An affected
folder with no stored row (a gap in the chain) is a candidate and
is generated; an affected folder that dropped below the minimum
is the prune pass's job (step 4), not a generation's. The two
restriction kwargs are ORTHOGONAL and no caller combines them:
``only_missing=True`` (the unchanged-sync gap fill, phase 96)
keeps its exact pre-127 semantics.
Steps, in order:
1. ``skip=True`` → a no-op: the zero stats dict is returned, the
LLM is never called, and no rows are touched.
@@ -461,10 +519,12 @@ async def generate_folder_summaries(
BEFORE the attempt — so an instant manual skip and a failed folder
BOTH advance the counter (the UI's position moves on either), and
``total`` is ``len(keys)`` at loop start (under ``only_missing
=True`` that is the MISSING count, not the full candidate count).
``None`` — the ``scripts/import_docs.py`` CLI path — is a
zero-cost no-op (guarded at the call site; nothing observable
changes without it).
=True`` that is the MISSING count, not the full candidate count;
under a ``changed_docs`` restriction that is the AFFECTED count —
the sync status's ``summaries_total`` reports it through this
existing hook). ``None`` — the ``scripts/import_docs.py`` CLI
path — is a zero-cost no-op (guarded at the call site; nothing
observable changes without it).
Only flushes — the CALLER commits (the phase-53
``bump_sources_version`` convention: the sync path owns the
@@ -490,6 +550,15 @@ async def generate_folder_summaries(
for row in db.execute(select(FolderSummary)).scalars()
}
keys = sorted(candidates)
if changed_docs is not None:
# Phase 127 (task 02): restricted regeneration — exactly the
# CANDIDATE folders the change set touches (each document's
# folder chain + source root). Unaffected candidates keep
# their rows byte-identical (text AND ``updated_at``); a gap
# in an affected folder is a candidate and gets generated;
# an affected folder that lost its last document is no
# candidate — the prune pass below drops its row.
keys = sorted(set(candidates) & affected_folder_keys(changed_docs))
if only_missing:
keys = [key for key in keys if key not in existing]