phase: 127_sync_incremental_folder_summaries
All verified green — the previous rejection was a harness interruption (no code failure), and the working tree + doc fixes are intact. **Phase 127 — final verification pass: 100% green** (all 4 tasks already complete; no code changes needed this pass) - **Re-verified implementation:** `ImportSummary.changed_paths` at all 5 increment sites + `_prune`; `affected_folder_keys` + `generate_folder_summaries(changed_docs=…)` (restricted keys, prune pass both modes, hook total = affected count); sync changed-KB branch passes `changed_docs=summary.changed_paths`; CLI keeps full-regen contract - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3022 passed**, `app/` coverage **99%**; `uv run pytest tests/e2e/test_incremental_folder_summaries.py -v --no-cov` → **3 passed in isolation**; `uv run ruff check . && uv run pyright` → clean, **0 errors** - **Completion criteria:** one-file change rewrites only its folder chain (E2E pins siblings byte-identical, text AND `updated_at`) ✅; coverage >90% ✅; isolated E2E green ✅; no completed-phase behavior change (full suite green) ✅; atomic commit — delegated to the harness per pass rules (no git commands run by me) - **Docs:** README.md re-checked — all 7 previous-pass fixes still in place (Theme page row, drill-down `ls` + capped `read`, phase-124 answer images section + `BOR_ANSWER_IMAGES`, markdown-included summaries, 17-color palette, folder summaries in `BOR_LLM_SUMMARY_MODEL`, CLI regen note); AGENTS.md verified consistent — no changes needed - **Deviations:** none; untracked owner file `NOTES.md` left untouched (rides in the harness commit) - **Next pending phase:** `128_sync_label_mobile`
This commit is contained in:
+52
-23
@@ -56,9 +56,16 @@ decisions):
|
||||
(phase 31 trigger, best-effort inside) — and ``generate_
|
||||
folder_summaries`` (phase 94, task 02) regenerates the stored folder
|
||||
summaries (the drill-down ``ls``'s per-level descriptions): its gate
|
||||
is the same change trigger (full regeneration) **plus**, on an
|
||||
unchanged re-sync, a GAP probe — a candidate folder (at least 2
|
||||
docs) with no stored row (``missing_folder_summaries``, phase 96,
|
||||
is the same change trigger — INCREMENTAL since phase 127 (task 03):
|
||||
the run's changed documents (``ImportSummary.changed_paths``, the
|
||||
import's added/updated/pruned ``(source, path)`` set) are passed as
|
||||
``changed_docs=`` and ONLY their folder chains (each document's
|
||||
folder up to its source root — ``affected_folder_keys``) are
|
||||
rewritten, so a one-file change burns a handful of rows, not the
|
||||
whole table (the ``scripts/import_docs.py`` CLI keeps the
|
||||
byte-identical full regeneration — it passes no change set) **plus**,
|
||||
on an unchanged re-sync, a GAP probe — a candidate folder (at least
|
||||
1 doc) with no stored row (``missing_folder_summaries``, phase 96,
|
||||
task 03): a gap fills ONLY the missing rows (``only_missing=True``;
|
||||
the old table-empty first-sync trigger is subsumed exactly — an
|
||||
empty table leaves every candidate missing), a complete table burns
|
||||
@@ -91,13 +98,16 @@ reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
|
||||
(set immediately before ``import_sources``), ``"overview"`` (set
|
||||
before ``regenerate_overview`` — a changed KB only), and
|
||||
``"summaries"`` (set before a folder-summary generation — the
|
||||
changed-KB full regeneration or the unchanged-walk gap-fill; the
|
||||
no-gap skip sets NO phase and the run stays ``"import"``). While in
|
||||
changed-KB incremental regeneration (phase 127, task 03 — the changed
|
||||
set's folder keys only) or the unchanged-walk gap-fill; the no-gap
|
||||
skip sets NO phase and the run stays ``"import"``). While in
|
||||
``"summaries"`` the generator's ``on_progress`` hook fills
|
||||
``current_summary`` (the ``source`` / ``source/folder_path`` being
|
||||
summarized — the bare source name for the source-root row) plus
|
||||
``summaries_done`` / ``summaries_total`` — the per-folder position
|
||||
through the long summary span where the file count sits still.
|
||||
through the long summary span where the file count sits still (the
|
||||
hook's ``total`` is the run's key count — the AFFECTED count on a
|
||||
changed KB, the missing count on a gap-fill).
|
||||
Terminal states (success AND failed) clear ``phase`` +
|
||||
``current_summary`` but keep the run's final ``summaries_done`` /
|
||||
``summaries_total`` (the phase-64 keep-final-counts convention).
|
||||
@@ -161,9 +171,13 @@ class SyncStatus:
|
||||
``"summaries"`` phase, ``current_summary`` is the source /
|
||||
``source/folder_path`` the generator is summarizing right now and
|
||||
``summaries_done`` / ``summaries_total`` carry the generator's
|
||||
progress hook's position. Terminal states clear ``phase`` +
|
||||
``current_summary`` but keep the run's final summary counts (the
|
||||
phase-64 keep-final-counts convention).
|
||||
progress hook's position — the hook's ``total`` is the run's key
|
||||
count: the AFFECTED count on a changed KB (phase 127, task 03 —
|
||||
the changed set's folder keys, NOT the table's candidate count) or
|
||||
the missing count on an unchanged-walk gap-fill (as before).
|
||||
Terminal states clear ``phase`` + ``current_summary`` but keep the
|
||||
run's final summary counts (the phase-64 keep-final-counts
|
||||
convention).
|
||||
"""
|
||||
|
||||
state: Literal["idle", "running", "success", "failed"] = "idle"
|
||||
@@ -358,13 +372,19 @@ async def _run_sync() -> None:
|
||||
if summary.added + summary.updated > 0:
|
||||
_status.phase = "overview" # phase 98 (task 01)
|
||||
overview = await regenerate_overview(llm)
|
||||
# Phase 94 (task 02), phase 96 (task 03): the folder
|
||||
# summaries — the drill-down ls's per-level descriptions. A
|
||||
# changed KB (added + updated > 0) is a full regeneration
|
||||
# (today's behavior, byte-identical); an unchanged re-sync
|
||||
# takes the GAP probe instead of the old table-empty one — a
|
||||
# candidate folder (at least 1 doc — every existing folder)
|
||||
# with no stored row: a gap
|
||||
# Phase 94 (task 02), phase 96 (task 03), phase 127 (task 03):
|
||||
# the folder summaries — the drill-down ls's per-level
|
||||
# descriptions. A changed KB (added + updated > 0) is an
|
||||
# INCREMENTAL regeneration (phase 127, task 03): the run's
|
||||
# changed documents (summary.changed_paths — the import's
|
||||
# added/updated/pruned set) are passed as changed_docs= and
|
||||
# ONLY their folder chains (each document's folder up to its
|
||||
# source root) are rewritten — a one-file change burns a
|
||||
# handful of rows, not the whole table (the CLI keeps the
|
||||
# byte-identical full regeneration — it passes no change set);
|
||||
# an unchanged re-sync takes the GAP probe instead of the old
|
||||
# table-empty one — a candidate folder (at least 1 doc — every
|
||||
# existing folder) with no stored row: a gap
|
||||
# fills ONLY the missing rows (only_missing=True — the
|
||||
# subsumed table-empty first-sync trigger included, where
|
||||
# every candidate is missing), a complete table burns zero
|
||||
@@ -378,8 +398,9 @@ async def _run_sync() -> None:
|
||||
# Phase 98 (task 01): the folder-summary progress hook — the
|
||||
# closure-captures-``_status`` convention the file's ``_hook``
|
||||
# above already uses; the generator's ``total`` (the loop-start
|
||||
# candidate count) and the bare source name for the root row
|
||||
# come straight from the hook's arguments (D5's shape).
|
||||
# key count — the affected count on a changed KB, the missing
|
||||
# count on a gap-fill) and the bare source name for the root
|
||||
# row come straight from the hook's arguments (D5's shape).
|
||||
def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
|
||||
_status.current_summary = (
|
||||
source if folder_path == "" else f"{source}/{folder_path}"
|
||||
@@ -390,15 +411,23 @@ async def _run_sync() -> None:
|
||||
fs_db = SessionLocal()
|
||||
try:
|
||||
if summary.added + summary.updated > 0:
|
||||
# Changed KB: full regeneration — the summaries phase
|
||||
# starts now, and the hook reports the per-folder
|
||||
# position through the (long) span.
|
||||
# Changed KB: incremental regeneration (phase 127,
|
||||
# task 03) — ONLY the changed set's folder keys
|
||||
# (changed_docs=summary.changed_paths: each document's
|
||||
# folder chain up to its source root); the summaries
|
||||
# phase starts now, and the hook reports the per-folder
|
||||
# position through the (long) span (its total is the
|
||||
# affected count).
|
||||
_status.phase = "summaries"
|
||||
folder_stats = await generate_folder_summaries(
|
||||
fs_db, llm, on_progress=_summary_hook
|
||||
fs_db, llm, changed_docs=summary.changed_paths,
|
||||
on_progress=_summary_hook,
|
||||
)
|
||||
fs_db.commit()
|
||||
logger.info("sync: folder_summaries stats=%s", folder_stats)
|
||||
logger.info(
|
||||
"sync: folder_summaries changed=%d stats=%s",
|
||||
len(summary.changed_paths), folder_stats,
|
||||
)
|
||||
else:
|
||||
missing = missing_folder_summaries(fs_db)
|
||||
if missing:
|
||||
|
||||
Reference in New Issue
Block a user