phase: 127_sync_incremental_folder_summaries
Build and Push Containers / build-and-push-app (push) Successful in 2m25s
Build and Push Containers / build-and-push-db (push) Successful in 15s

All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact.

**Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass)

- **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors**
- **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me)
- **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed
- **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit)
- **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
2026-09-27 21:12:25 -04:00
parent f50d4c8bc3
commit 6e654df387
38 changed files with 2741 additions and 46 deletions
+52 -23
View File
@@ -56,9 +56,16 @@ decisions):
(phase 31 trigger, best-effort inside) — and ``generate_
folder_summaries`` (phase 94, task 02) regenerates the stored folder
summaries (the drill-down ``ls``'s per-level descriptions): its gate
is the same change trigger (full regeneration) **plus**, on an
unchanged re-sync, a GAP probe — a candidate folder (at least 2
docs) with no stored row (``missing_folder_summaries``, phase 96,
is the same change trigger — INCREMENTAL since phase 127 (task 03):
the run's changed documents (``ImportSummary.changed_paths``, the
import's added/updated/pruned ``(source, path)`` set) are passed as
``changed_docs=`` and ONLY their folder chains (each document's
folder up to its source root — ``affected_folder_keys``) are
rewritten, so a one-file change burns a handful of rows, not the
whole table (the ``scripts/import_docs.py`` CLI keeps the
byte-identical full regeneration — it passes no change set) **plus**,
on an unchanged re-sync, a GAP probe — a candidate folder (at least
1 doc) with no stored row (``missing_folder_summaries``, phase 96,
task 03): a gap fills ONLY the missing rows (``only_missing=True``;
the old table-empty first-sync trigger is subsumed exactly — an
empty table leaves every candidate missing), a complete table burns
@@ -91,13 +98,16 @@ reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
(set immediately before ``import_sources``), ``"overview"`` (set
before ``regenerate_overview`` — a changed KB only), and
``"summaries"`` (set before a folder-summary generation — the
changed-KB full regeneration or the unchanged-walk gap-fill; the
no-gap skip sets NO phase and the run stays ``"import"``). While in
changed-KB incremental regeneration (phase 127, task 03 — the changed
set's folder keys only) or the unchanged-walk gap-fill; the no-gap
skip sets NO phase and the run stays ``"import"``). While in
``"summaries"`` the generator's ``on_progress`` hook fills
``current_summary`` (the ``source`` / ``source/folder_path`` being
summarized — the bare source name for the source-root row) plus
``summaries_done`` / ``summaries_total`` — the per-folder position
through the long summary span where the file count sits still.
through the long summary span where the file count sits still (the
hook's ``total`` is the run's key count — the AFFECTED count on a
changed KB, the missing count on a gap-fill).
Terminal states (success AND failed) clear ``phase`` +
``current_summary`` but keep the run's final ``summaries_done`` /
``summaries_total`` (the phase-64 keep-final-counts convention).
@@ -161,9 +171,13 @@ class SyncStatus:
``"summaries"`` phase, ``current_summary`` is the source /
``source/folder_path`` the generator is summarizing right now and
``summaries_done`` / ``summaries_total`` carry the generator's
progress hook's position. Terminal states clear ``phase`` +
``current_summary`` but keep the run's final summary counts (the
phase-64 keep-final-counts convention).
progress hook's position — the hook's ``total`` is the run's key
count: the AFFECTED count on a changed KB (phase 127, task 03 —
the changed set's folder keys, NOT the table's candidate count) or
the missing count on an unchanged-walk gap-fill (as before).
Terminal states clear ``phase`` + ``current_summary`` but keep the
run's final summary counts (the phase-64 keep-final-counts
convention).
"""
state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -358,13 +372,19 @@ async def _run_sync() -> None:
if summary.added + summary.updated > 0:
_status.phase = "overview" # phase 98 (task 01)
overview = await regenerate_overview(llm)
# Phase 94 (task 02), phase 96 (task 03): the folder
# summaries — the drill-down ls's per-level descriptions. A
# changed KB (added + updated > 0) is a full regeneration
# (today's behavior, byte-identical); an unchanged re-sync
# takes the GAP probe instead of the old table-empty one — a
# candidate folder (at least 1 doc — every existing folder)
# with no stored row: a gap
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
# the folder summaries — the drill-down ls's per-level
# descriptions. A changed KB (added + updated > 0) is an
# INCREMENTAL regeneration (phase 127, task 03): the run's
# changed documents (summary.changed_paths — the import's
# added/updated/pruned set) are passed as changed_docs= and
# ONLY their folder chains (each document's folder up to its
# source root) are rewritten — a one-file change burns a
# handful of rows, not the whole table (the CLI keeps the
# byte-identical full regeneration — it passes no change set);
# an unchanged re-sync takes the GAP probe instead of the old
# table-empty one — a candidate folder (at least 1 doc — every
# existing folder) with no stored row: a gap
# fills ONLY the missing rows (only_missing=True — the
# subsumed table-empty first-sync trigger included, where
# every candidate is missing), a complete table burns zero
@@ -378,8 +398,9 @@ async def _run_sync() -> None:
# Phase 98 (task 01): the folder-summary progress hook — the
# closure-captures-``_status`` convention the file's ``_hook``
# above already uses; the generator's ``total`` (the loop-start
# candidate count) and the bare source name for the root row
# come straight from the hook's arguments (D5's shape).
# key count — the affected count on a changed KB, the missing
# count on a gap-fill) and the bare source name for the root
# row come straight from the hook's arguments (D5's shape).
def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
_status.current_summary = (
source if folder_path == "" else f"{source}/{folder_path}"
@@ -390,15 +411,23 @@ async def _run_sync() -> None:
fs_db = SessionLocal()
try:
if summary.added + summary.updated > 0:
# Changed KB: full regeneration — the summaries phase
# starts now, and the hook reports the per-folder
# position through the (long) span.
# Changed KB: incremental regeneration (phase 127,
# task 03) — ONLY the changed set's folder keys
# (changed_docs=summary.changed_paths: each document's
# folder chain up to its source root); the summaries
# phase starts now, and the hook reports the per-folder
# position through the (long) span (its total is the
# affected count).
_status.phase = "summaries"
folder_stats = await generate_folder_summaries(
fs_db, llm, on_progress=_summary_hook
fs_db, llm, changed_docs=summary.changed_paths,
on_progress=_summary_hook,
)
fs_db.commit()
logger.info("sync: folder_summaries stats=%s", folder_stats)
logger.info(
"sync: folder_summaries changed=%d stats=%s",
len(summary.changed_paths), folder_stats,
)
else:
missing = missing_folder_summaries(fs_db)
if missing: