phase: 98_sync_summary_visibility

All verification complete. Final report:

**Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed)

- **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5.
- **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0)
- **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%)
- **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal)
- **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓
- **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run).
- **Decisions/deviations:** none — no fixes were required this pass.
- **Next pending phase:** `99_kb_tree_table_and_back_nav`.

app/api/docs.py

@@ -40,7 +40,7 @@ from app.core.auth import require_admin, require_user
 from app.db import get_db
 from app.models import Chunk, Document, FolderSummary
 from app.rag.agent import list_source_names
-from app.rag.folder_summaries import folder_of
+from app.rag.folder_summaries import MIN_DOCS_PER_FOLDER, folder_of
 from app.rag.importer import match_extension
 from app.rag.llm import EmbeddingError, LLMClient
 from app.schemas import (
@@ -376,6 +376,11 @@ def _level_children(
     :func:`app.rag.agent.group_folder_listing` level-for-level; the
     file list is NOT capped (the ``ls`` 50-line cap is a model-context
     budget — the UI is for humans). Recurses one level per call.
+
+    Each folder node also carries the phase-98 D3 ``summary_pending``
+    flag: the recursive count ≥ :data:`MIN_DOCS_PER_FOLDER` AND no
+    stored ``folder_summaries`` row for ``(source, sub)`` — the same
+    rule the source node applies (see :func:`build_kb_tree`).
     """
     children: list[KbTreeFolder | KbTreeFile] = []
     for sub in sorted(g for g in folders if folder_of(g) == folder):
@@ -384,6 +389,8 @@ def _level_children(
                 path=sub,
                 documents=counts[sub],
                 summary=summaries.get((source, sub)),
+                summary_pending=counts[sub] >= MIN_DOCS_PER_FOLDER
+                and (source, sub) not in summaries,
                 children=_level_children(source, sub, folders, counts, rows, summaries),
             )
         )
@@ -431,7 +438,22 @@ def build_kb_tree(
       shape.
     * **File nodes** — direct files only, in input (catalog) order;
       ``path`` source-relative; ``title`` / ``chunks`` / ``indexed_at``
-      verbatim from the catalogue row.
+      verbatim from the catalogue row. File nodes carry NO pending
+      flag (the file table has no description column).
+    * **Pending** — ``summary_pending`` on the SOURCE and every FOLDER
+      node (phase 98, decision D3 — ONE concept): true iff the node's
+      recursive ``documents`` count ≥
+      :data:`app.rag.folder_summaries.MIN_DOCS_PER_FOLDER` (2) AND it
+      has NO stored ``folder_summaries`` row (AI or manual — any row;
+      the builder sees stored rows only). That is EXACTLY
+      :func:`app.rag.folder_summaries.missing_folder_summaries`'s
+      candidate set (phase 96's gap-fill regenerates precisely those
+      keys on the next sync — the marker is honest: "waiting to
+      generate", and the integration cross-check pins the tree's
+      pending set to that function so the marker can never drift from
+      the gap-fill). A < 2-document folder is NEVER pending (it never
+      gets a summary — its one file line IS its description), and a
+      registered 0-document source never is.
 
     ONE concept end to end: the builder reuses
     :func:`app.rag.folder_summaries.folder_of` and the phase-94
@@ -471,13 +493,19 @@ def _source_node(
     count (every one of its documents, the set its stored
     ``(source, "")`` summary describes). A source with no rows lists
     ``documents: 0`` and no children (the registered 0-document source
-    — the phase-70/72 invariant, extended by the superset rule).
+    — the phase-70/72 invariant, extended by the superset rule) and
+    is never ``summary_pending`` (0 < the minimum).
+
+    ``summary_pending`` (phase 98, D3): the whole-source count ≥
+    :data:`MIN_DOCS_PER_FOLDER` AND no stored ``(source, "")`` row —
+    the source-root arm of the rule :func:`build_kb_tree` documents.
     """
     folders, counts = _folder_counts(rows)
     return KbTreeSource(
         name=source,
         documents=len(rows),
         summary=summaries.get((source, "")),
+        summary_pending=len(rows) >= MIN_DOCS_PER_FOLDER and (source, "") not in summaries,
         children=_level_children(source, "", folders, counts, rows, summaries),
     )
 

app/api/sync.py

@@ -75,6 +75,23 @@ carries the phase-64 per-file progress — ``current_file`` (the
 (clone/pull reports no file yet) and in terminal states, which clear
 ``current_file`` but keep the run's final counts.
 
+Phase 98 (task 01) adds the run's PHASE machine — ``phase`` is the
+pipeline stage the run is in: the model-check + clone/pull prelude
+reports ``null`` (the bare "Syncing…" label stays), then ``"import"``
+(set immediately before ``import_sources``), ``"overview"`` (set
+before ``regenerate_overview`` — a changed KB only), and
+``"summaries"`` (set before a folder-summary generation — the
+changed-KB full regeneration or the unchanged-walk gap-fill; the
+no-gap skip sets NO phase and the run stays ``"import"``). While in
+``"summaries"`` the generator's ``on_progress`` hook fills
+``current_summary`` (the ``source`` / ``source/folder_path`` being
+summarized — the bare source name for the source-root row) plus
+``summaries_done`` / ``summaries_total`` — the per-folder position
+through the long summary span where the file count sits still.
+Terminal states (success AND failed) clear ``phase`` +
+``current_summary`` but keep the run's final ``summaries_done`` /
+``summaries_total`` (the phase-64 keep-final-counts convention).
+
 The ``failed`` state's ``error`` string is masked by the shared
 sanitizer — the ``user:pass@`` masker now lives in :mod:`app.core.errors`
 (imported here under the private name ``_sanitize_error``).
@@ -127,6 +144,16 @@ class SyncStatus:
     after); ``files_done`` / ``files_total`` carry the hook's
     done/total position and survive a terminal state (the run's last
     position is useful context next to the error).
+
+    Phase 98 (task 01) phase fields: ``phase`` is the pipeline stage
+    (null in the model-check + clone/pull prelude and in terminal
+    states — the module docstring's phase machine); while in the
+    ``"summaries"`` phase, ``current_summary`` is the source /
+    ``source/folder_path`` the generator is summarizing right now and
+    ``summaries_done`` / ``summaries_total`` carry the generator's
+    progress hook's position. Terminal states clear ``phase`` +
+    ``current_summary`` but keep the run's final summary counts (the
+    phase-64 keep-final-counts convention).
     """
 
     state: Literal["idle", "running", "success", "failed"] = "idle"
@@ -139,6 +166,13 @@ class SyncStatus:
     current_file: str | None = None
     files_done: int = 0
     files_total: int = 0
+    # Phase 98 (task 01): the phase machine — the pipeline stage the
+    # run is in and, while in the folder-summary phase, the folder
+    # being summarized plus the hook's done/total position.
+    phase: Literal["import", "overview", "summaries"] | None = None
+    current_summary: str | None = None
+    summaries_done: int = 0
+    summaries_total: int = 0
 
 
 _status = SyncStatus()
@@ -153,7 +187,11 @@ def sync_status() -> dict[str, Any]:
     ``current_file`` (phase 64) is the ``source/relative/path`` the
     import is processing right now — null during the clone/pull phase
     and in terminal states; ``files_done`` / ``files_total`` carry the
-    hook's position (0/0 idle).
+    hook's position (0/0 idle). ``phase`` (phase 98) is the pipeline
+    stage — null in the prelude and terminal states; while in
+    ``"summaries"``, ``current_summary`` + ``summaries_done`` /
+    ``summaries_total`` carry the generator's position (the terminal
+    keeps the final summary counts).
     """
     return {
         "state": _status.state,
@@ -164,6 +202,13 @@ def sync_status() -> dict[str, Any]:
         "current_file": _status.current_file,
         "files_done": _status.files_done,
         "files_total": _status.files_total,
+        # Phase 98 (task 01): the phase machine — null/0/0 idle (the
+        # dataclass defaults) and in terminal states (which clear
+        # phase + current_summary but keep the final summary counts).
+        "phase": _status.phase,
+        "current_summary": _status.current_summary,
+        "summaries_done": _status.summaries_done,
+        "summaries_total": _status.summaries_total,
     }
 
 
@@ -202,6 +247,13 @@ async def _run_sync() -> None:
     _status.current_file = None
     _status.files_done = 0
     _status.files_total = 0
+    # Phase 98 (task 01): the phase fields reset with the run — no
+    # phase until the import starts (the model-check + clone/pull
+    # prelude reports null), no folder until the summary span starts.
+    _status.phase = None
+    _status.current_summary = None
+    _status.summaries_done = 0
+    _status.summaries_total = 0
     try:
         settings = get_settings()
         # Step 1 (phase 41): fail fast — verify both models the sync
@@ -270,12 +322,17 @@ async def _run_sync() -> None:
             _status.files_done = done
             _status.files_total = total
 
+        # Phase 98 (task 01): the phase machine — the import phase
+        # starts NOW (the model-check + clone/pull prelude above
+        # reported ``phase: null``).
+        _status.phase = "import"
         summary: ImportSummary = await import_sources(
             sources, llm, prune=True, progress=_hook, ignore_by_root=ignore_by_root,
             include_hidden_by_root=include_hidden_by_root,
         )
         overview = False
         if summary.added + summary.updated > 0:
+            _status.phase = "overview"  # phase 98 (task 01)
             overview = await regenerate_overview(llm)
         # Phase 94 (task 02), phase 96 (task 03): the folder
         # summaries — the drill-down ls's per-level descriptions. A
@@ -290,19 +347,42 @@ async def _run_sync() -> None:
         # outage never flips the run to failed) and only flushes: this
         # run's own short-lived session commits (the phase-53
         # convention), so the step-6 bump stays change-gated on the KB,
-        # not on the summaries. No status-surface change: the stats
-        # are log-only (the detail shape is untouched).
+        # not on the summaries. The status surface is the PHASE
+        # machine (phase 98, task 01): the stats dict itself stays
+        # log-only (the detail shape is untouched).
+        # Phase 98 (task 01): the folder-summary progress hook — the
+        # closure-captures-``_status`` convention the file's ``_hook``
+        # above already uses; the generator's ``total`` (the loop-start
+        # candidate count) and the bare source name for the root row
+        # come straight from the hook's arguments (D5's shape).
+        def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
+            _status.current_summary = (
+                source if folder_path == "" else f"{source}/{folder_path}"
+            )
+            _status.summaries_done = done
+            _status.summaries_total = total
+
         fs_db = SessionLocal()
         try:
             if summary.added + summary.updated > 0:
-                folder_stats = await generate_folder_summaries(fs_db, llm)
+                # Changed KB: full regeneration — the summaries phase
+                # starts now, and the hook reports the per-folder
+                # position through the (long) span.
+                _status.phase = "summaries"
+                folder_stats = await generate_folder_summaries(
+                    fs_db, llm, on_progress=_summary_hook
+                )
                 fs_db.commit()
                 logger.info("sync: folder_summaries stats=%s", folder_stats)
             else:
                 missing = missing_folder_summaries(fs_db)
                 if missing:
+                    # Unchanged-walk gap-fill: the SAME summaries
+                    # phase + hook (the fill's total is the missing
+                    # count — the hook's loop-start ``total``).
+                    _status.phase = "summaries"
                     folder_stats = await generate_folder_summaries(
-                        fs_db, llm, only_missing=True
+                        fs_db, llm, only_missing=True, on_progress=_summary_hook
                     )
                     fs_db.commit()
                     logger.info(
@@ -310,6 +390,9 @@ async def _run_sync() -> None:
                         len(missing), folder_stats,
                     )
                 else:
+                    # The no-gap skip sets NO phase — the run stays
+                    # "import" through to the terminal (the phase-64
+                    # bare-label pin holds for this shape).
                     logger.info("sync: folder_summaries skipped (KB unchanged)")
         finally:
             fs_db.close()
@@ -336,6 +419,8 @@ async def _run_sync() -> None:
         _status.state = "success"
         _status.finished_at = datetime.now(UTC)
         _status.current_file = None  # phase 64: keep the final counts
+        _status.phase = None  # phase 98: keep the final summary counts
+        _status.current_summary = None
         _status.detail = {
             "files": summary.files,
             "added": summary.added,
@@ -356,3 +441,5 @@ async def _run_sync() -> None:
         _status.finished_at = datetime.now(UTC)
         _status.error = _sanitize_error(str(e))
         _status.current_file = None  # phase 64: keep the final counts
+        _status.phase = None  # phase 98: keep the final summary counts
+        _status.current_summary = None