phase: 127_sync_incremental_folder_summaries

All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.

**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)

- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`

app/api/sync.py

@@ -56,9 +56,16 @@ decisions):
    (phase 31 trigger, best-effort inside) — and ``generate_
    folder_summaries`` (phase 94, task 02) regenerates the stored folder
    summaries (the drill-down ``ls``'s per-level descriptions): its gate
-   is the same change trigger (full regeneration) **plus**, on an
-   unchanged re-sync, a GAP probe — a candidate folder (at least 2
-   docs) with no stored row (``missing_folder_summaries``, phase 96,
+   is the same change trigger — INCREMENTAL since phase 127 (task 03):
+   the run's changed documents (``ImportSummary.changed_paths``, the
+   import's added/updated/pruned ``(source, path)`` set) are passed as
+   ``changed_docs=`` and ONLY their folder chains (each document's
+   folder up to its source root — ``affected_folder_keys``) are
+   rewritten, so a one-file change burns a handful of rows, not the
+   whole table (the ``scripts/import_docs.py`` CLI keeps the
+   byte-identical full regeneration — it passes no change set) **plus**,
+   on an unchanged re-sync, a GAP probe — a candidate folder (at least
+   1 doc) with no stored row (``missing_folder_summaries``, phase 96,
    task 03): a gap fills ONLY the missing rows (``only_missing=True``;
    the old table-empty first-sync trigger is subsumed exactly — an
    empty table leaves every candidate missing), a complete table burns
@@ -91,13 +98,16 @@ reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
 (set immediately before ``import_sources``), ``"overview"`` (set
 before ``regenerate_overview`` — a changed KB only), and
 ``"summaries"`` (set before a folder-summary generation — the
-changed-KB full regeneration or the unchanged-walk gap-fill; the
-no-gap skip sets NO phase and the run stays ``"import"``). While in
+changed-KB incremental regeneration (phase 127, task 03 — the changed
+set's folder keys only) or the unchanged-walk gap-fill; the no-gap
+skip sets NO phase and the run stays ``"import"``). While in
 ``"summaries"`` the generator's ``on_progress`` hook fills
 ``current_summary`` (the ``source`` / ``source/folder_path`` being
 summarized — the bare source name for the source-root row) plus
 ``summaries_done`` / ``summaries_total`` — the per-folder position
-through the long summary span where the file count sits still.
+through the long summary span where the file count sits still (the
+hook's ``total`` is the run's key count — the AFFECTED count on a
+changed KB, the missing count on a gap-fill).
 Terminal states (success AND failed) clear ``phase`` +
 ``current_summary`` but keep the run's final ``summaries_done`` /
 ``summaries_total`` (the phase-64 keep-final-counts convention).
@@ -161,9 +171,13 @@ class SyncStatus:
     ``"summaries"`` phase, ``current_summary`` is the source /
     ``source/folder_path`` the generator is summarizing right now and
     ``summaries_done`` / ``summaries_total`` carry the generator's
-    progress hook's position. Terminal states clear ``phase`` +
-    ``current_summary`` but keep the run's final summary counts (the
-    phase-64 keep-final-counts convention).
+    progress hook's position — the hook's ``total`` is the run's key
+    count: the AFFECTED count on a changed KB (phase 127, task 03 —
+    the changed set's folder keys, NOT the table's candidate count) or
+    the missing count on an unchanged-walk gap-fill (as before).
+    Terminal states clear ``phase`` + ``current_summary`` but keep the
+    run's final summary counts (the phase-64 keep-final-counts
+    convention).
     """
 
     state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -358,13 +372,19 @@ async def _run_sync() -> None:
         if summary.added + summary.updated > 0:
             _status.phase = "overview"  # phase 98 (task 01)
             overview = await regenerate_overview(llm)
-        # Phase 94 (task 02), phase 96 (task 03): the folder
-        # summaries — the drill-down ls's per-level descriptions. A
-        # changed KB (added + updated > 0) is a full regeneration
-        # (today's behavior, byte-identical); an unchanged re-sync
-        # takes the GAP probe instead of the old table-empty one — a
-        # candidate folder (at least 1 doc — every existing folder)
-        # with no stored row: a gap
+        # Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
+        # the folder summaries — the drill-down ls's per-level
+        # descriptions. A changed KB (added + updated > 0) is an
+        # INCREMENTAL regeneration (phase 127, task 03): the run's
+        # changed documents (summary.changed_paths — the import's
+        # added/updated/pruned set) are passed as changed_docs= and
+        # ONLY their folder chains (each document's folder up to its
+        # source root) are rewritten — a one-file change burns a
+        # handful of rows, not the whole table (the CLI keeps the
+        # byte-identical full regeneration — it passes no change set);
+        # an unchanged re-sync takes the GAP probe instead of the old
+        # table-empty one — a candidate folder (at least 1 doc — every
+        # existing folder) with no stored row: a gap
         # fills ONLY the missing rows (only_missing=True — the
         # subsumed table-empty first-sync trigger included, where
         # every candidate is missing), a complete table burns zero
@@ -378,8 +398,9 @@ async def _run_sync() -> None:
         # Phase 98 (task 01): the folder-summary progress hook — the
         # closure-captures-``_status`` convention the file's ``_hook``
         # above already uses; the generator's ``total`` (the loop-start
-        # candidate count) and the bare source name for the root row
-        # come straight from the hook's arguments (D5's shape).
+        # key count — the affected count on a changed KB, the missing
+        # count on a gap-fill) and the bare source name for the root
+        # row come straight from the hook's arguments (D5's shape).
         def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
             _status.current_summary = (
                 source if folder_path == "" else f"{source}/{folder_path}"
@@ -390,15 +411,23 @@ async def _run_sync() -> None:
         fs_db = SessionLocal()
         try:
             if summary.added + summary.updated > 0:
-                # Changed KB: full regeneration — the summaries phase
-                # starts now, and the hook reports the per-folder
-                # position through the (long) span.
+                # Changed KB: incremental regeneration (phase 127,
+                # task 03) — ONLY the changed set's folder keys
+                # (changed_docs=summary.changed_paths: each document's
+                # folder chain up to its source root); the summaries
+                # phase starts now, and the hook reports the per-folder
+                # position through the (long) span (its total is the
+                # affected count).
                 _status.phase = "summaries"
                 folder_stats = await generate_folder_summaries(
-                    fs_db, llm, on_progress=_summary_hook
+                    fs_db, llm, changed_docs=summary.changed_paths,
+                    on_progress=_summary_hook,
                 )
                 fs_db.commit()
-                logger.info("sync: folder_summaries stats=%s", folder_stats)
+                logger.info(
+                    "sync: folder_summaries changed=%d stats=%s",
+                    len(summary.changed_paths), folder_stats,
+                )
             else:
                 missing = missing_folder_summaries(fs_db)
                 if missing:

app/rag/folder_summaries.py

@@ -59,11 +59,22 @@ the stored keys — one concept, as with :func:`group_by_folder`), and
 :func:`generate_folder_summaries`'s ``only_missing=True`` fills
 EXACTLY those on the next sync — existing rows stay
 byte-identical (text AND ``updated_at``), the prune pass still runs.
+
+Restricted regeneration (phase 127, task 02): a changed-KB sync no
+longer rewrites EVERY folder — the sync path passes the run's changed
+documents (``ImportSummary.changed_paths``) as ``changed_docs=`` and
+the generator then rewrites ONLY the candidate folders the change
+touches — each document's folder chain up to the source root
+(:func:`affected_folder_keys`, the SAME prefix walk
+:func:`group_by_folder` pass 1 uses — one concept); every other
+stored row keeps its text AND ``updated_at``. The
+``scripts/import_docs.py`` CLI keeps the byte-identical full
+regeneration (``changed_docs=None``).
 """
 from __future__ import annotations
 
 import logging
-from collections.abc import Callable, Sequence
+from collections.abc import Callable, Iterable, Sequence
 from datetime import UTC, datetime
 from typing import Protocol
 
@@ -209,6 +220,32 @@ def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow
     return groups
 
 
+def affected_folder_keys(
+    changed: Iterable[tuple[str, str]],
+) -> set[tuple[str, str]]:
+    """The candidate folder keys a changed-document set invalidates.
+
+    For each ``(source, path)``: the source-root key ``(source, "")``
+    plus every folder prefix of *path* — :func:`folder_of` walked from
+    the document's nearest folder up to ``""`` — the SAME prefix walk
+    :func:`group_by_folder` pass 1 uses (one concept): a doc under
+    ``a/b/`` affects ``a/b``, ``a``, and the source root. A
+    root-level file (``folder_of(path) == ""``) affects ONLY the
+    source root. Embedded-image fragment paths (``x.pdf#page=…#img=…``,
+    phase 126) fold into their parent's folder chain with no special
+    case — ``folder_of`` never sees a ``/`` inside the fragment.
+    Returns a set; the generator sorts it (deterministic order).
+    """
+    keys: set[tuple[str, str]] = set()
+    for source, path in changed:
+        keys.add((source, ""))
+        folder = folder_of(path)
+        while folder:
+            keys.add((source, folder))
+            folder = folder_of(folder)
+    return keys
+
+
 def _first_summary_line(summary: str | None) -> str:
     """First line of a stored summary, stripped; ``''`` when absent.
 
@@ -406,13 +443,34 @@ async def generate_folder_summaries(
     *,
     skip: bool = False,
     only_missing: bool = False,
+    changed_docs: Sequence[tuple[str, str]] | None = None,
     on_progress: Callable[[int, int, str, str], None] | None = None,
 ) -> dict[str, int]:
     """Regenerate the stored folder summaries for the current catalogue.
 
     The sync-path orchestrator (phase 94, task 02 calls it
     change-gated, like the KB overview; ``--limit`` debug runs pass
-    ``skip=True``). Steps, in order:
+    ``skip=True``).
+
+    Modes (phase 127, task 02 — the *changed_docs* kwarg):
+    ``changed_docs=None`` (the DEFAULT) is the FULL regeneration —
+    every candidate folder, byte-identical to the pre-127 behavior;
+    the ``scripts/import_docs.py`` CLI stays on it. A GIVEN change set
+    (the sync path passes ``ImportSummary.changed_paths``) restricts
+    the generation to the CANDIDATE folders the changed documents
+    touch — :func:`affected_folder_keys`: a document's change
+    invalidates exactly its folder chain + source root, the folders
+    whose subtree summary text CAN change; every other stored row
+    keeps its summary text AND ``updated_at`` (an unaffected folder's
+    documents — and hence its summary — are untouched). An affected
+    folder with no stored row (a gap in the chain) is a candidate and
+    is generated; an affected folder that dropped below the minimum
+    is the prune pass's job (step 4), not a generation's. The two
+    restriction kwargs are ORTHOGONAL and no caller combines them:
+    ``only_missing=True`` (the unchanged-sync gap fill, phase 96)
+    keeps its exact pre-127 semantics.
+
+    Steps, in order:
 
     1. ``skip=True`` → a no-op: the zero stats dict is returned, the
        LLM is never called, and no rows are touched.
@@ -461,10 +519,12 @@ async def generate_folder_summaries(
     BEFORE the attempt — so an instant manual skip and a failed folder
     BOTH advance the counter (the UI's position moves on either), and
     ``total`` is ``len(keys)`` at loop start (under ``only_missing
-    =True`` that is the MISSING count, not the full candidate count).
-    ``None`` — the ``scripts/import_docs.py`` CLI path — is a
-    zero-cost no-op (guarded at the call site; nothing observable
-    changes without it).
+    =True`` that is the MISSING count, not the full candidate count;
+    under a ``changed_docs`` restriction that is the AFFECTED count —
+    the sync status's ``summaries_total`` reports it through this
+    existing hook). ``None`` — the ``scripts/import_docs.py`` CLI
+    path — is a zero-cost no-op (guarded at the call site; nothing
+    observable changes without it).
 
     Only flushes — the CALLER commits (the phase-53
     ``bump_sources_version`` convention: the sync path owns the
@@ -490,6 +550,15 @@ async def generate_folder_summaries(
         for row in db.execute(select(FolderSummary)).scalars()
     }
     keys = sorted(candidates)
+    if changed_docs is not None:
+        # Phase 127 (task 02): restricted regeneration — exactly the
+        # CANDIDATE folders the change set touches (each document's
+        # folder chain + source root). Unaffected candidates keep
+        # their rows byte-identical (text AND ``updated_at``); a gap
+        # in an affected folder is a candidate and gets generated;
+        # an affected folder that lost its last document is no
+        # candidate — the prune pass below drops its row.
+        keys = sorted(set(candidates) & affected_folder_keys(changed_docs))
     if only_missing:
         keys = [key for key in keys if key not in existing]
 

app/rag/importer.py

@@ -244,6 +244,12 @@ class ImportSummary:
     #: Fail-soft: the sync continues, this counter + the warning line
     #: are the signal.
     images_failed: int = 0
+    #: Documents this run ADDED, UPDATED, or PRUNED, as (source, path)
+    #: tuples in encounter order (added/updated in walk order, pruned in
+    #: _prune order). Drives the sync's incremental folder-summary
+    #: regeneration (phase 127): len(changed_paths) == added + updated +
+    #: pruned. Unchanged / backfilled / date-only refreshes NEVER record.
+    changed_paths: list[tuple[str, str]] = field(default_factory=list)
     #: Files walked, keyed by lowercased extension (``md``, ``yaml``, …).
     formats: dict[str, int] = field(default_factory=dict)
 
@@ -590,7 +596,8 @@ async def import_sources(
                 logger.warning("import: --prune ignored because --limit was given")
             else:
                 summary.pruned = _prune(
-                    session, source_names, seen, images=llm.settings.images
+                    session, source_names, seen, images=llm.settings.images,
+                    summary=summary,
                 )
         summary.embed_batches = llm.embed_batches
         summary.log()
@@ -794,6 +801,7 @@ async def _index_file(
         summary.added += 1
     else:
         summary.updated += 1
+    summary.changed_paths.append((source, rel))
     summary.chunks += chunks_n
     logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
 
@@ -1068,6 +1076,7 @@ async def _index_pdf_file(
         summary.added += 1
     else:
         summary.updated += 1
+    summary.changed_paths.append((source, rel))
     summary.chunks += chunks_n
     logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
 
@@ -1328,6 +1337,7 @@ async def _index_one_embedded_image(
         summary.added += 1
     else:
         summary.updated += 1
+    summary.changed_paths.append((source, image_rel))
     summary.chunks += chunks_n
     logger.info(
         "import: %s source=%s path=%s chunks=%d", verb, source, image_rel, chunks_n
@@ -1728,6 +1738,7 @@ async def _enrich_pending_images(
             )
             continue
         summary.updated += 1
+        summary.changed_paths.append((source, image_doc.path))
         summary.chunks += chunks_n
         logger.info(
             "import: image-context-enriched source=%s path=%s chunks=%d",
@@ -1994,6 +2005,7 @@ async def _index_image_file(
         summary.added += 1
     else:
         summary.updated += 1
+    summary.changed_paths.append((source, rel))
     summary.chunks += chunks_n
     logger.info("import: %s source=%s path=%s chunks=%d", verb, source, rel, chunks_n)
 
@@ -2017,6 +2029,7 @@ def _prune(
     source_names: set[str],
     seen: set[tuple[str, str]],
     images: bool = False,
+    summary: ImportSummary | None = None,
 ) -> int:
     """Delete documents of *source_names* whose file is no longer in *seen*.
 
@@ -2040,6 +2053,13 @@ def _prune(
     toggle states (toggle off, the phase-122 guard already skips every
     ``is_image`` doc; this one is the toggle-on half of the same
     "invisible, not deleted" contract).
+
+    *summary* (phase 127, task 01): when given, every document that is
+    ACTUALLY deleted (i.e. not skipped by either guard above) records
+    its ``(doc.source, doc.path)`` on ``summary.changed_paths`` —
+    appended BEFORE the ``session.delete`` — so the sync's incremental
+    folder-summary regeneration (phase 127) sees the pruned docs' folders
+    too. Callers passing nothing keep the pre-phase count-only contract.
     """
     if not source_names:
         return 0
@@ -2059,6 +2079,8 @@ def _prune(
                     # invisible to the walk, not deleted.
                     continue
             _delete_image_copy(doc.image_path)
+            if summary is not None:
+                summary.changed_paths.append((doc.source, doc.path))
             session.delete(doc)
             pruned += 1
             logger.info("import: pruned source=%s path=%s", doc.source, doc.path)