phase: 98_sync_summary_visibility

All verification complete. Final report:

**Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed)

- **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5.
- **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0)
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%)
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal)
- **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓
- **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run).
- **Decisions/deviations:** none — no fixes were required this pass.
- **Next pending phase:** `99_kb_tree_table_and_back_nav`.

app/api/docs.py

@@ -40,7 +40,7 @@ from app.core.auth import require_admin, require_user
 from app.db import get_db
 from app.models import Chunk, Document, FolderSummary
 from app.rag.agent import list_source_names
-from app.rag.folder_summaries import folder_of
+from app.rag.folder_summaries import MIN_DOCS_PER_FOLDER, folder_of
 from app.rag.importer import match_extension
 from app.rag.llm import EmbeddingError, LLMClient
 from app.schemas import (
@@ -376,6 +376,11 @@ def _level_children(
     :func:`app.rag.agent.group_folder_listing` level-for-level; the
     file list is NOT capped (the ``ls`` 50-line cap is a model-context
     budget — the UI is for humans). Recurses one level per call.
+
+    Each folder node also carries the phase-98 D3 ``summary_pending``
+    flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
+    stored ``folder_summaries`` row for ``(source, sub)`` — the same
+    rule the source node applies (see :func:`build_kb_tree`).
     """
     children: list[KbTreeFolder | KbTreeFile] = []
     for sub in sorted(g for g in folders if folder_of(g) == folder):
@@ -384,6 +389,8 @@ def _level_children(
                 path=sub,
                 documents=counts[sub],
                 summary=summaries.get((source, sub)),
+                summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
+                and (source, sub) not in summaries,
                 children=_level_children(source, sub, folders, counts, rows, summaries),
             )
         )
@@ -431,7 +438,22 @@ def build_kb_tree(
       shape.
     * **File nodes** — direct files only, in input (catalog) order;
       ``path`` source-relative; ``title`` / ``chunks`` / ``indexed_at``
-      verbatim from the catalogue row.
+      verbatim from the catalogue row. File nodes carry NO pending
+      flag (the file table has no description column).
+    * **Pending** — ``summary_pending`` on the SOURCE and every FOLDER
+      node (phase 98, decision D3 — ONE concept): true iff the node's
+      recursive ``documents`` count ≥
+      :data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (2) AND it
+      has NO stored ``folder_summaries`` row (AI or manual — any row;
+      the builder sees stored rows only). That is EXACTLY
+      :func:`app.rag.folder_summaries.missing_folder_summaries`'s
+      candidate set (phase 96's gap-fill regenerates precisely those
+      keys on the next sync — the marker is honest: "waiting to
+      generate", and the integration cross-check pins the tree's
+      pending set to that function so the marker can never drift from
+      the gap-fill). A < 2-document folder is NEVER pending (it never
+      gets a summary — its one file line IS its description), and a
+      registered 0-document source never is.
 
     ONE concept end to end: the builder reuses
     :func:`app.rag.folder_summaries.folder_of` and the phase-94
@@ -471,13 +493,19 @@ def _source_node(
     count (every one of its documents, the set its stored
     ``(source, "")`` summary describes). A source with no rows lists
     ``documents: 0`` and no children (the registered 0-document source
-    — the phase-70/72 invariant, extended by the superset rule).
+    — the phase-70/72 invariant, extended by the superset rule) and
+    is never ``summary_pending`` (0 < the minimum).
+
+    ``summary_pending`` (phase 98, D3): the whole-source count ≥
+    :data:`MIN_DOCS_PER_FOLDER` AND no stored ``(source, "")`` row —
+    the source-root arm of the rule :func:`build_kb_tree` documents.
     """
     folders, counts = _folder_counts(rows)
     return KbTreeSource(
         name=source,
         documents=len(rows),
         summary=summaries.get((source, "")),
+        summary_pending=len(rows) >= MIN_DOCS_PER_FOLDER and (source, "") not in summaries,
         children=_level_children(source, "", folders, counts, rows, summaries),
     )
 

app/api/sync.py

@@ -75,6 +75,23 @@ carries the phase-64 per-file progress — ``current_file`` (the
 (clone/pull reports no file yet) and in terminal states, which clear
 ``current_file`` but keep the run's final counts.
 
+Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the
+pipeline stage the run is in: the model-check + clone/pull prelude
+reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
+(set immediately before ``import_sources``), ``"overview"`` (set
+before ``regenerate_overview`` — a changed KB only), and
+``"summaries"`` (set before a folder-summary generation — the
+changed-KB full regeneration or the unchanged-walk gap-fill; the
+no-gap skip sets NO phase and the run stays ``"import"``). While in
+``"summaries"`` the generator's ``on_progress`` hook fills
+``current_summary`` (the ``source`` / ``source/folder_path`` being
+summarized — the bare source name for the source-root row) plus
+``summaries_done`` / ``summaries_total`` — the per-folder position
+through the long summary span where the file count sits still.
+Terminal states (success AND failed) clear ``phase`` +
+``current_summary`` but keep the run's final ``summaries_done`` /
+``summaries_total`` (the phase-64 keep-final-counts convention).
+
 The ``failed`` state's ``error`` string is masked by the shared
 sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
 (imported here under the private name ``_sanitize_error``).
@@ -127,6 +144,16 @@ class SyncStatus:
     after); ``files_done`` / ``files_total`` carry the hook's
     done/total position and survive a terminal state (the run's last
     position is useful context next to the error).
+
+    Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage
+    (null in the model-check + clone/pull prelude and in terminal
+    states — the module docstring's phase machine); while in the
+    ``"summaries"`` phase, ``current_summary`` is the source /
+    ``source/folder_path`` the generator is summarizing right now and
+    ``summaries_done`` / ``summaries_total`` carry the generator's
+    progress hook's position. Terminal states clear ``phase`` +
+    ``current_summary`` but keep the run's final summary counts (the
+    phase-64 keep-final-counts convention).
     """
 
     state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -139,6 +166,13 @@ class SyncStatus:
     current_file: str | None = None
     files_done: int = 0
     files_total: int = 0
+    # Phase 98 (task 01): the phase machine — the pipeline stage the
+    # run is in and, while in the folder-summary phase, the folder
+    # being summarized plus the hook's done/total position.
+    phase: Literal["import", "overview", "summaries"] | None = None
+    current_summary: str | None = None
+    summaries_done: int = 0
+    summaries_total: int = 0
 
 
 _status = SyncStatus()
@@ -153,7 +187,11 @@ def sync_status() -> dict[str, Any]:
     ``current_file`` (phase 64) is the ``source/relative/path`` the
     import is processing right now — null during the clone/pull phase
     and in terminal states; ``files_done`` / ``files_total`` carry the
-    hook's position (0/0 idle).
+    hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline
+    stage — null in the prelude and terminal states; while in
+    ``"summaries"``, ``current_summary`` + ``summaries_done`` /
+    ``summaries_total`` carry the generator's position (the terminal
+    keeps the final summary counts).
     """
     return {
         "state": _status.state,
@@ -164,6 +202,13 @@ def sync_status() -> dict[str, Any]:
         "current_file": _status.current_file,
         "files_done": _status.files_done,
         "files_total": _status.files_total,
+        # Phase 98 (task 01): the phase machine — null/0/0 idle (the
+        # dataclass defaults) and in terminal states (which clear
+        # phase + current_summary but keep the final summary counts).
+        "phase": _status.phase,
+        "current_summary": _status.current_summary,
+        "summaries_done": _status.summaries_done,
+        "summaries_total": _status.summaries_total,
     }
 
 
@@ -202,6 +247,13 @@ async def _run_sync() -> None:
     _status.current_file = None
     _status.files_done = 0
     _status.files_total = 0
+    # Phase 98 (task 01): the phase fields reset with the run — no
+    # phase until the import starts (the model-check + clone/pull
+    # prelude reports null), no folder until the summary span starts.
+    _status.phase = None
+    _status.current_summary = None
+    _status.summaries_done = 0
+    _status.summaries_total = 0
     try:
         settings = get_settings()
         # Step 1 (phase 41): fail fast — verify both models the sync
@@ -270,12 +322,17 @@ async def _run_sync() -> None:
             _status.files_done = done
             _status.files_total = total
 
+        # Phase 98 (task 01): the phase machine — the import phase
+        # starts NOW (the model-check + clone/pull prelude above
+        # reported ``phase: null``).
+        _status.phase = "import"
         summary: ImportSummary = await import_sources(
             sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
             include_hidden_by_root=include_hidden_by_root,
         )
         overview = False
         if summary.added + summary.updated > 0:
+            _status.phase = "overview"  # phase 98 (task 01)
             overview = await regenerate_overview(llm)
         # Phase 94 (task 02), phase 96 (task 03): the folder
         # summaries — the drill-down ls's per-level descriptions. A
@@ -290,19 +347,42 @@ async def _run_sync() -> None:
         # outage never flips the run to failed) and only flushes: this
         # run's own short-lived session commits (the phase-53
         # convention), so the step-6 bump stays change-gated on the KB,
-        # not on the summaries. No status-surface change: the stats
-        # are log-only (the detail shape is untouched).
+        # not on the summaries. The status surface is the PHASE
+        # machine (phase 98, task 01): the stats dict itself stays
+        # log-only (the detail shape is untouched).
+        # Phase 98 (task 01): the folder-summary progress hook — the
+        # closure-captures-``_status`` convention the file's ``_hook``
+        # above already uses; the generator's ``total`` (the loop-start
+        # candidate count) and the bare source name for the root row
+        # come straight from the hook's arguments (D5's shape).
+        def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
+            _status.current_summary = (
+                source if folder_path == "" else f"{source}/{folder_path}"
+            )
+            _status.summaries_done = done
+            _status.summaries_total = total
+
         fs_db = SessionLocal()
         try:
             if summary.added + summary.updated > 0:
-                folder_stats = await generate_folder_summaries(fs_db, llm)
+                # Changed KB: full regeneration — the summaries phase
+                # starts now, and the hook reports the per-folder
+                # position through the (long) span.
+                _status.phase = "summaries"
+                folder_stats = await generate_folder_summaries(
+                    fs_db, llm, on_progress=_summary_hook
+                )
                 fs_db.commit()
                 logger.info("sync: folder_summaries stats=%s", folder_stats)
             else:
                 missing = missing_folder_summaries(fs_db)
                 if missing:
+                    # Unchanged-walk gap-fill: the SAME summaries
+                    # phase + hook (the fill's total is the missing
+                    # count — the hook's loop-start ``total``).
+                    _status.phase = "summaries"
                     folder_stats = await generate_folder_summaries(
-                        fs_db, llm, only_missing=True
+                        fs_db, llm, only_missing=True, on_progress=_summary_hook
                     )
                     fs_db.commit()
                     logger.info(
@@ -310,6 +390,9 @@ async def _run_sync() -> None:
                         len(missing), folder_stats,
                     )
                 else:
+                    # The no-gap skip sets NO phase — the run stays
+                    # "import" through to the terminal (the phase-64
+                    # bare-label pin holds for this shape).
                     logger.info("sync: folder_summaries skipped (KB unchanged)")
         finally:
             fs_db.close()
@@ -336,6 +419,8 @@ async def _run_sync() -> None:
         _status.state = "success"
         _status.finished_at = datetime.now(UTC)
         _status.current_file = None  # phase 64: keep the final counts
+        _status.phase = None  # phase 98: keep the final summary counts
+        _status.current_summary = None
         _status.detail = {
             "files": summary.files,
             "added": summary.added,
@@ -356,3 +441,5 @@ async def _run_sync() -> None:
         _status.finished_at = datetime.now(UTC)
         _status.error = _sanitize_error(str(e))
         _status.current_file = None  # phase 64: keep the final counts
+        _status.phase = None  # phase 98: keep the final summary counts
+        _status.current_summary = None

app/rag/folder_summaries.py

@@ -42,7 +42,12 @@ Chat turns never generate folder summaries — the agent's ``ls`` output
 (phase 94, task 03) only reads the stored rows. Generation is the
 caller's job at sync time (phase 94, task 02), and :func:`generate_
 folder_summaries` only flushes — the sync path owns the transaction
-(the phase-53 ``bump_sources_version`` convention).
+(the phase-53 ``bump_sources_version`` convention). The sync path
+additionally passes the optional ``on_progress`` hook (phase 98,
+task 01) so the sync status can report the folder being summarized;
+the ``scripts/import_docs.py`` CLI passes none — the hook defaults to
+``None`` and is a zero-cost no-op, leaving the CLI's log-only stats
+contract untouched.
 
 Self-heal (phase 96): an exhausted one-shot retry can still leave a
 candidate folder without a row. :func:`missing_folder_summaries`
@@ -55,7 +60,7 @@ byte-identical (text AND ``updated_at``), the prune pass still runs.
 from __future__ import annotations
 
 import logging
-from collections.abc import Sequence
+from collections.abc import Callable, Sequence
 from datetime import UTC, datetime
 from typing import Protocol
 
@@ -390,6 +395,7 @@ async def generate_folder_summaries(
     *,
     skip: bool = False,
     only_missing: bool = False,
+    on_progress: Callable[[int, int, str, str], None] | None = None,
 ) -> dict[str, int]:
     """Regenerate the stored folder summaries for the current catalogue.
 
@@ -436,6 +442,17 @@ async def generate_folder_summaries(
        unchanged catalogue it is a no-op (the invariant kept), and it
        still drops rows whose folder fell below the minimum.
 
+    Progress (phase 98, task 01): when *on_progress* is given, it is
+    called once per candidate with ``(done, total, source,
+    folder_path)`` in the same sorted ``(source, folder_path)`` order,
+    BEFORE the attempt — so an instant manual skip and a failed folder
+    BOTH advance the counter (the UI's position moves on either), and
+    ``total`` is ``len(keys)`` at loop start (under ``only_missing
+    =True`` that is the MISSING count, not the full candidate count).
+    ``None`` — the ``scripts/import_docs.py`` CLI path — is a
+    zero-cost no-op (guarded at the call site; nothing observable
+    changes without it).
+
     Only flushes — the CALLER commits (the phase-53
     ``bump_sources_version`` convention: the sync path owns the
     transaction, so a failed sync rolls the summaries back with it).
@@ -463,8 +480,15 @@ async def generate_folder_summaries(
     if only_missing:
         keys = [key for key in keys if key not in existing]
 
-    for key in keys:
+    for i, key in enumerate(keys):
         source, folder_path = key
+        # Phase 98 (task 01): the optional progress hook fires BEFORE
+        # the attempt — instant manual skips and failed folders both
+        # advance the counter (D5), and ``total`` is the loop-start
+        # count (the missing count under ``only_missing``). ``None``
+        # (the CLI path) is a zero-cost no-op.
+        if on_progress is not None:
+            on_progress(i + 1, len(keys), source, folder_path)
         stored = existing.get(key)
         if stored is not None and stored.manually_edited:
             # Owner-edited description (phase 97, task 01): NEVER

app/schemas.py

@@ -274,12 +274,21 @@ class KbTreeFolder(BaseModel):
     (path order) followed by the direct files (catalog order) — the
     recursive union (Pydantic v2 resolves it with
     ``from __future__ import annotations``).
+
+    ``summary_pending`` (phase 98, D3 — ONE concept): true iff this
+    folder's recursive count ≥ ``MIN_DOCS_PER_FOLDER`` (2) AND it has
+    NO stored ``folder_summaries`` row (AI or manual — any row) —
+    exactly the candidate the sync-time gap-fill regenerates (the
+    :func:`app.rag.folder_summaries.missing_folder_summaries` set).
+    A < 2-document folder is never pending (it never gets a summary —
+    its one file line IS its description).
     """
 
     kind: Literal["folder"] = "folder"
     path: str
     documents: int = Field(ge=0)
     summary: str | None = None
+    summary_pending: bool = False
     children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
 
 
@@ -294,11 +303,21 @@ class KbTreeSource(BaseModel):
     ``(source, "")`` source-root row or null; ``children`` are the
     source's direct subfolders + direct files (same shape as a folder
     node's).
+
+    ``summary_pending`` (phase 98, D3 — ONE concept): true iff the
+    source's recursive ``documents`` count ≥ ``MIN_DOCS_PER_FOLDER``
+    (2) AND no stored ``(source, "")`` row (AI or manual — any row) —
+    exactly the source-root candidate the sync-time gap-fill
+    regenerates (the
+    :func:`app.rag.folder_summaries.missing_folder_summaries` set).
+    A registered 0-document source is never pending (there is nothing
+    to summarize).
     """
 
     name: str
     documents: int = Field(ge=0)
     summary: str | None = None
+    summary_pending: bool = False
     children: list[KbTreeFolder | KbTreeFile] = Field(default_factory=list)
 
 
@@ -308,6 +327,13 @@ class KbTree(BaseModel):
     The FULL recursive tree in ONE fetch — the RAG view (admin) drills
     client-side, zero per-level fetches (the ``00_phase.md`` "The tree
     endpoint" contract).
+
+    Every SOURCE and FOLDER node carries ``summary_pending`` (phase
+    98, D3): true iff its recursive document count ≥
+    ``MIN_DOCS_PER_FOLDER`` (2) AND it has no stored ``folder_summaries``
+    row — exactly ``missing_folder_summaries``'s candidate set (the
+    marker never drifts from the gap-fill); FILE nodes carry no flag
+    (the file table has no description column).
     """
 
     sources: list[KbTreeSource]