brain-of-reese/.agents/phases/todo/98_sync_summary_visibility/00_phase.md

Phase 98 — Sync makes summary generation visible: status phases + "summary pending" in the catalog

Source: Owner request (chat, 2026-09-12) — "When syncing sources, the user has no idea when summaries (document or directory) are happening, they just see the number pause for a really long time. Also before the summaries generate it looks like those directories/files were missed, the UI should tell the user those summaries are waiting to generate." Story: n/a (owner request — the sync progress surface on 64_sync_upload_progress; the folder-summary generator + gap-fill on 94_ls_tree_drilldown / 96_oneshot_resilience; the drill-down catalog tree on 97_kb_tree_catalog). Context: POST /api/sync runs in-process as one background task (app/api/sync.py _run_sync): model check → clone/pull → import_sources (the ONLY phase with per-file progress — the phase-64 hook feeds SyncStatus.current_file / files_done / files_total, which the RAG-page sync button polls every 2 s and renders as "Syncing… <file> (n/m)") → regenerate_overview (ONE lite call, change-gated) → generate_folder_summaries (one lite call PER candidate folder — up to hundreds; the phase-94 generator, per-folder fail-soft, only_missing gap-fill since phase 96) → the sources_meta bump. After the import finishes the count sits at its final value for the ENTIRE overview + folder-summary span (minutes on a large KB) — the user's "number pauses for a really long time": nothing on the wire says what is happening. GET /api/sync/status today returns state / started_at / finished_at / detail / error / current_file / files_done / files_total (null/0 idle; terminal states clear current_file but keep the final counts). The catalog tree (GET /api/docs/tree, the pure build_kb_tree in app/api/docs.py + the KbTree* schemas in app/schemas.py) renders a Description cell per source/folder from the stored folder_summaries rows — a folder with NO stored row shows an EMPTY cell (the agent's ls shows the count only, the phase-94 rule) — after a fail-soft miss or a cleared manual description it looks like the folder was missed, when in fact the next sync's gap-fill (missing_folder_summaries, phase 96) will generate it. generate_folder_summaries(db, llm, *, skip=False, only_missing=False) iterates sorted(candidates) (candidate = recursive subtree ≥ MIN_DOCS_PER_FOLDER = 2 docs, app/rag/folder_summaries.py), skipping manual rows (kept_manual) and failing soft per folder (failed). The RAG view's sync machinery lives in frontend/assets/sources.js (fmtSyncLabel, enterSyncRunningState, the two-job poll decision tree — the upload job's status has no summary phases and stays bare "Importing…"). E2E conventions: the phase-64 tests/e2e/test_sync_upload_progress.py tight-poll recorder (a daemon thread polling the status endpoint at ~100 ms while the UI's 2 s poll drives the label) + the tests/e2e/slow_llm.py proxy (per-request delay, SLOW_DELAY_S) that stretches a run past the poll cadence; the phase-96 tests/e2e/test_oneshot_llm_retry.py direct-DB row deletion + re-sync pattern; the mock LLM's deterministic FOLDER_SUMMARY_MODE one-liner (Fixture folder summary for <folder>.).

Objective

While a sync runs, the status endpoint and the sync button tell the user exactly which phase the run is in — the import phase keeps its byte-identical file label, the KB-overview phase is named, and the (long) folder-summary phase reports the folder being summarized plus a done/total count — and after a sync, every source/folder whose summary is due but missing shows an explicit "Summary pending" marker in the catalog (row Description cell AND the level block) instead of an empty cell that reads as "missed". The pending set is exactly the candidate set the phase-96 gap-fill regenerates on the next sync — the marker's copy says so.

Dependencies

• 97_kb_tree_catalog (complete) — the tree endpoint / build_kb_tree / KbTree* schemas / the RAG-view tree UI (makeDescCell, the level block, renderLevel).
• 96_oneshot_resilience (complete) — missing_folder_summaries + the only_missing gap-fill (the pending marker's honest "waiting" semantics) + the E2E row-deletion pattern.
• 94_ls_tree_drilldown (complete) — the folder-summary generator, MIN_DOCS_PER_FOLDER, the per-folder fail-soft loop this phase instruments.
• 64_sync_upload_progress (complete) — SyncStatus's progress fields, the 2 s poll, the live-label contract this phase extends.

Decisions recorded here (owner review — PLAN.md is being redone by the owner)

• D1 — additive status fields, existing contract byte-identical: SyncStatus / the status JSON gain exactly four keys — phase ("import" | "overview" | "summaries" | null), current_summary (the source / source/folder being summarized, or null), summaries_done, summaries_total (ints, 0 idle/terminal-reset). The model-check and clone/pull prelude keeps phase: null (the bare "Syncing…" label stays — the phase-64 pins hold); terminal states clear phase + current_summary and KEEP the final summaries_done / summaries_total (the phase-64 keep-final-counts convention). current_file / files_done / files_total / detail are untouched.
• D2 — the labels (UI): the sync button's running label for the sync job becomes: phase "overview" → Writing KB overview…; phase "summaries" → Summarizing folders… <current_summary> (n/m) (the folder part omitted when current_summary is null — the first poll of the phase); anything else → today's Syncing… <file> (n/m) logic, byte-identical (import + prelude). The UPLOAD job's label is untouched (its status endpoint has no phase). The untruncated label still rides the button title + #sync-result (the aria-live announcer); CSS ellipsizes the label span only (phase-64 A4 contract).
• D3 — the pending rule (ONE concept): a source or folder node is summary_pending ⟺ its recursive document count ≥ MIN_DOCS_PER_FOLDER (2) AND it has NO stored folder_summaries row (AI or manual — the builder sees stored rows only). That is exactly missing_folder_summaries's candidate set (phase 96) — the marker is honest: the next sync's gap-fill (or the changed-KB regeneration) will generate it. A < 2-document folder is NEVER pending (it never gets a summary — its one file line IS its description). FILE nodes carry no pending flag — the file table has no description column and the pending concept is the folder-summary one (the owner's "directories/files" is served by the source + folder rows, which are the catalog's description-bearing rows).
• D4 — the marker's surfaces: the row Description cell shows the muted text Summary pending (a title carries "No stored description yet — the next sync will generate one.") with the ALWAYS-present Edit button kept (a manual save creates the row and clears the marker in place); the level block (#kb-level) shows when the current level has a stored description (as today) or is pending (title + a pending note line + the Edit button — write one manually right now); neither → hidden (today's behavior). State is text + color, never color alone (B5); the muted ink-soft pair is AA on the surface (no new hue — the phase-92 monochrome invariant).
• D5 — the generator gets an optional progress hook: generate_folder_summaries gains on_progress: Callable[[int, int, str, str], None] | None (done, total, source, folder_path) — called once per sorted candidate BEFORE its attempt (manual skips advance the counter — they are instant). None (the scripts/import_docs.py CLI path) is a no-op: the CLI's log-only stats contract is unchanged. No new endpoint, no new env var, no SSE event, no migration.

Design (shared by all tasks — the executor reads this, not the chat)

The sync status phases (task 01)

• app/api/sync.py — SyncStatus dataclass + the /status response gain the four D1 fields (defaults null/0; reset with the run at the top of _run_sync, exactly where current_file resets). _run_sync sets _status.phase = "import" immediately before import_sources; wraps regenerate_overview with phase = "overview" (and back — to "summaries" when the folder branch runs, else the run proceeds to the bump/terminal); the folder-summary branch (changed-KB full regeneration or the gap-fill) sets phase = "summaries" and passes the hook:

  def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
      _status.current_summary = source if folder_path == "" else f"{source}/{folder_path}"
      _status.summaries_done = done
      _status.summaries_total = total
  

  (the closure-captures-_status convention the file's _hook already uses). The no-gap skip branch sets no phase (stays "import"). Terminal paths (success AND the except block) set phase = None, current_summary = None — keep summaries_done / summaries_total (final counts, D1).
• app/rag/folder_summaries.py — generate_folder_summaries(..., on_progress: Callable[[int, int, str, str], None] | None = None): inside the for loop over keys, index i (enumerate), call on_progress(i + 1, len(keys), source, folder_path) before the manual-skip check (the counter advances for instant skips, D5). The stats dict, the fail-soft behavior, the logger line, and the flush-only transaction contract are untouched. Docstrings updated (the module rule).
• app/api/sync.py's module docstring: the status paragraph names the four new fields + the phase machine (the house "docstrings carry the contracts" rule).

The sync button (task 02)

• frontend/assets/sources.js — the sync-job running label: replace the single fmtSyncLabel(kind, currentFile, done, total) call sites for the SYNC job with a phase-aware builder (the upload job keeps the bare label — its status has no phase):
  • phase === "overview" → Writing KB overview…
  • phase === "summaries" → Summarizing folders… + (current_summary ? ${current_summary} : ``) + (${summaries_done}/${summaries_total})
  • otherwise → today's Syncing… [current_file] [(done/total)] (byte-identical — the phase-64 pins). enterSyncRunningState (and the poll's enterSyncRunningState("sync", …) call + initSyncButton's re-attach) thread the phase fields through; the button title + #sync-result mirror the full untruncated label (A4). The load-time re-attach (initSyncButton) re-enters a RUNNING run with whatever phase the status reports (a mid-summaries reload shows the summaries label — the never-stale contract).
• tests/unit/test_frontend_sync_upload.py — the label-builder pins gain the three phase cases (+ the upload-unchanged negative case).

The tree pending flag (task 03)

• app/schemas.py — KbTreeFolder and KbTreeSource gain summary_pending: bool = False (wire-additive; KbTreeFile untouched).
• app/api/docs.py — build_kb_tree: import MIN_DOCS_PER_FOLDER from app.rag.folder_summaries (the builder already imports folder_of from there — no new dependency edge). A source node: pending ⟺ its recursive document count (the builder already computes per-source rows) ≥ MIN_DOCS_PER_FOLDER AND (source, "") not in summaries. A folder node in _level_children: pending ⟺ counts[sub] >= MIN_DOCS_PER_FOLDER AND (source, sub) not in summaries. The endpoint's fetches are unchanged (the summaries mapping already carries ALL stored rows).
• Unit (tests/unit/test_kb_tree_builder.py): the pending matrix — a ≥2-doc folder with no row → true; with a stored row (any) → false; a 1-doc folder with no row → false (never pending); the source root with ≥2 docs and no (source, "") row → true on the source node; a registered 0-document source → false; multi-source independence.
• Integration (tests/integration/test_docs_api.py): the tree endpoint returns summary_pending — and the CROSS-CHECK (D3, one concept): for a seeded dataset with a partial summary table, the set of (source, folder_path) flagged pending in the tree (root = "") equals missing_folder_summaries(db) (phase 96's public function) — the marker can never drift from the gap-fill.

The pending UI (task 04)

• frontend/assets/sources.js:
  • makeDescCell — when node.summary is empty (or absent) AND node.summary_pending → the text span carries the marker: class kb-summary-pending, text Summary pending, title = No stored description yet — the next sync will generate one. (textContent only — the house rule). The Edit button is unaffected (always present — a manual save CREATES the row). The editor's success path (node.summary = data.summary in wireDescriptionEdit) additionally clears the flag in place: node.summary_pending = false (a created description is no longer pending — no re-fetch).
  • renderLevel — the level block shows when node.summary (today) OR node.summary_pending (new): title as today (the full source-relative path); the summary <p> shows the stored text, or the pending note No description stored yet — the next sync will generate one. (You can write one yourself.) when pending; the block stays hidden for a non-pending level with no stored description (the ls rule, unchanged).
• frontend/assets/styles.css — .kb-summary-pending { color: var(--ink-soft); } (5.1:1 on --surface, AA; no italic, no new hue) — the row-cell font-size already applies (the class sits on the existing text span).
• Source pins (tests/unit/test_kb_tree_ui.py + the styles.css pins there): the marker copy + title, the pending branch in makeDescCell/renderLevel, the in-place flag clear on save, the class name in the CSS.

The E2E (task 05)

tests/e2e/test_sync_summary_visibility.py (new; app_server + mock_llm + db_ready fixtures; admin login via tests/e2e/auth_helpers.py; the temp-local-source seeding + POST /api/sync pattern from test_ls_tree_drilldown.py / test_oneshot_llm_retry.py; the sync leg of the summary-phase test runs against the slow_llm proxy — the test_sync_upload_progress.py fixture, SLOW_DELAY_S sized so the overview + 3 folder calls outlive the recorder's 100 ms cadence by ~15×):

1. Endpoint — the phase machine: seed a temp local source with TWO ≥ 2-doc folders (candidates: the source root + 2 folders = 3); start a KB-changing sync; the tight-poll recorder asserts: some running tick has phase == "overview"; some running ticks have phase == "summaries" with summaries_total == 3, current_summary non-null (starts with the source name; the root call is the bare source name), summaries_done monotonically increasing up to 3; every phase == "summaries" tick keeps files_done == files_total (the import finished — the pause the user reported); the terminal tick has phase null, current_summary null, and summaries_done == summaries_total == 3.
2. UI — the label: a fresh admin page on /sources.html starts the sync (button click) and the page's own 2 s poll renders a label matching /Summarizing folders/ with (n/3) at some point (generous timeout, the phase-64 UI pattern); after settle the button reads the terminal label (the existing Synced HH:MM contract) and #sync-result carries the counts.
3. Tree — the pending marker: after a successful (mock-LLM) sync, DELETE one folder's folder_summaries row directly (the phase-96 pattern) + DELETE the source-root row; reload the RAG view (nav re-show → the phase-77 refresh re-fetch) → the two affected rows' Description cells show Summary pending (with the title), the intact folder's cell shows its stored Fixture folder summary for … line and NO marker; clicking an affected folder shows the level block with the pending note; saving a manual description from the row's Edit button clears the marker in place (the cell shows the text) — and a second unchanged sync's gap-fill regenerates the OTHER deleted row (its marker goes away, its cell carries the deterministic mock line).

Tasks

1. 01_summary_phase_status.md — SyncStatus + the status JSON gain phase / current_summary / summaries_done / summaries_total; the generator's on_progress hook; _run_sync sets the phases
2. 02_sync_label_summary_phases.md — the sync button's phase-aware labels (overview / summaries) + the label-builder unit pins
3. 03_tree_pending_flag.md — summary_pending on the tree schemas + the build_kb_tree rule + the missing_folder_summaries cross-check
4. 04_tree_pending_ui.md — the row-cell + level-block pending markers + the in-place clear + the CSS class + source pins
5. 05_e2e_summary_visibility.md — tests/e2e/test_sync_summary_visibility.py + the regression sweep + the atomic commit

Testing & Quality

• Unit: the on_progress call matrix in the generator's suite (order/values, manual-skip advance, None no-op, skip=True unchanged); the status shape in tests/unit/test_sync_button.py (idle nulls/zeros; the four new keys); the label builders in tests/unit/test_frontend_sync_upload.py (D2's three cases + upload negative); the builder's pending matrix (tests/unit/test_kb_tree_builder.py); the source pins (tests/unit/test_kb_tree_ui.py, the styles.css class).
• Integration: tests/integration/test_sync_api.py (the status field contract across the state machine); tests/integration/test_sync_folder_summaries.py (the hook fires on the changed-KB AND gap-fill branches, never on the skip branch); tests/integration/test_docs_api.py (the pending shape + the D3 cross-check).
• E2E (mandatory, A16): uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov in isolation; regressions green in isolation: test_kb_tree.py, test_ls_tree_drilldown.py, test_sync_button.py, test_sync_upload_progress.py, test_oneshot_llm_retry.py, test_local_directory_sources.py.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing).
• Lint/types: uv run ruff check . && uv run pyright.

Completion Criteria

• while a sync runs, GET /api/sync/status reports the phase: the import keeps current_file/counts with phase "import", the overview shows phase "overview", the folder span shows phase "summaries" with current_summary + a done/total that climbs to the candidate count; terminal states clear the phase + folder but keep the final summary counts
• the RAG-page sync button reads Writing KB overview… / Summarizing folders… <folder> (n/m) in those phases (aria-live + button title carry the untruncated text) and is byte-identical to today in the prelude/import/upload cases
• a source/folder with ≥ 2 docs and no stored summary row shows Summary pending in its tree row AND its level block (the marker set equals missing_folder_summaries — integration-pinned); a manual save or the next sync's gap-fill makes the marker go away; < 2-doc folders and file rows never show it
• the CLI import path and its log lines are byte-identical (the hook is optional, the stats contract untouched); the agent's ls is byte-identical (the tree reads the same rows)
• uv run pytest green; coverage >90%; ruff + pyright clean
• uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov green in isolation (DB up: podman compose up -d db); the regression suites green in isolation
• one atomic Conventional Commit, --no-gpg-sign (e.g. feat(sync): surface the summary phases of a sync and mark folders whose summaries are pending)