phase: 98_sync_summary_visibility
All verification complete. Final report: **Phase 98 — Sync summary visibility: final verification pass** (all 5 tasks already complete; implementation verified against the design, no defects found, no code changes needed) - **Implementation checked:** `SyncStatus` phase machine (4 new keys, terminal-keep counts), `on_progress` hook in `generate_folder_summaries`, `summary_pending` on `KbTreeSource`/`KbTreeFolder` + D3 rule in `build_kb_tree`, phase-aware sync labels + pending UI in `sources.js`, `.kb-summary-pending` CSS — all match decisions D1–D5. - **Unit + integration:** `uv run pytest` → 2184 tests, 0 failed/errors (exit 0) - **Coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **99%** on `app/` (criterion >90% ✓; `app/api/sync.py` and `app/rag/folder_summaries.py` at 100%) - **Lint/types:** `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings - **Phase E2E (isolation):** `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` → **3 passed** (phase machine, live label, pending markers + gap-fill self-heal) - **Regression suites (each isolated, `--no-cov`):** test_kb_tree ✓, test_ls_tree_drilldown 3 ✓, test_sync_button 3 ✓, test_sync_upload_progress 4 ✓, test_oneshot_llm_retry 2 ✓, test_local_directory_sources 3 ✓ - **Completion criteria:** all 7 verified green — status phase fields + terminal semantics; `Writing KB overview…`/`Summarizing folders… (n/m)` labels (title + aria-live); pending set == `missing_folder_summaries` (integration cross-check pinned at `test_docs_api.py:428`); CLI/`ls` byte-identity (no changes to those paths, pins green); suite/coverage/lint gates; dedicated + regression E2E. Commit left to the harness per protocol (no `git add`/`commit` run). - **Decisions/deviations:** none — no fixes were required this pass. - **Next pending phase:** `99_kb_tree_table_and_back_nav`.
This commit is contained in:
@@ -0,0 +1,85 @@
|
||||
# Phase 98 — Sync makes summary generation visible: status phases + "summary pending" in the catalog
|
||||
|
||||
**Source:** Owner request (chat, 2026-09-12) — "When syncing sources, the user has no idea when summaries (document or directory) are happening, they just see the number pause for a really long time. Also before the summaries generate it looks like those directories/files were missed, the UI should tell the user those summaries are waiting to generate."
|
||||
**Story:** n/a (owner request — the sync progress surface on `64_sync_upload_progress`; the folder-summary generator + gap-fill on `94_ls_tree_drilldown` / `96_oneshot_resilience`; the drill-down catalog tree on `97_kb_tree_catalog`).
|
||||
**Context:** `POST /api/sync` runs in-process as one background task (`app/api/sync.py` `_run_sync`): model check → clone/pull → `import_sources` (the ONLY phase with per-file progress — the phase-64 hook feeds `SyncStatus.current_file` / `files_done` / `files_total`, which the RAG-page sync button polls every 2 s and renders as "Syncing… \<file\> (n/m)") → `regenerate_overview` (ONE `lite` call, change-gated) → `generate_folder_summaries` (one `lite` call PER candidate folder — up to hundreds; the phase-94 generator, per-folder fail-soft, `only_missing` gap-fill since phase 96) → the `sources_meta` bump. After the import finishes the count sits at its final value for the ENTIRE overview + folder-summary span (minutes on a large KB) — the user's "number pauses for a really long time": nothing on the wire says what is happening. `GET /api/sync/status` today returns `state` / `started_at` / `finished_at` / `detail` / `error` / `current_file` / `files_done` / `files_total` (null/0 idle; terminal states clear `current_file` but keep the final counts). The catalog tree (`GET /api/docs/tree`, the pure `build_kb_tree` in `app/api/docs.py` + the `KbTree*` schemas in `app/schemas.py`) renders a Description cell per source/folder from the stored `folder_summaries` rows — a folder with NO stored row shows an EMPTY cell (the agent's `ls` shows the count only, the phase-94 rule) — after a fail-soft miss or a cleared manual description it looks like the folder was missed, when in fact the next sync's gap-fill (`missing_folder_summaries`, phase 96) will generate it. `generate_folder_summaries(db, llm, *, skip=False, only_missing=False)` iterates `sorted(candidates)` (candidate = recursive subtree ≥ `MIN_DOCS_PER_FOLDER = 2` docs, `app/rag/folder_summaries.py`), skipping manual rows (`kept_manual`) and failing soft per folder (`failed`). The RAG view's sync machinery lives in `frontend/assets/sources.js` (`fmtSyncLabel`, `enterSyncRunningState`, the two-job poll decision tree — the upload job's status has no summary phases and stays bare "Importing…"). E2E conventions: the phase-64 `tests/e2e/test_sync_upload_progress.py` tight-poll recorder (a daemon thread polling the status endpoint at ~100 ms while the UI's 2 s poll drives the label) + the `tests/e2e/slow_llm.py` proxy (per-request delay, `SLOW_DELAY_S`) that stretches a run past the poll cadence; the phase-96 `tests/e2e/test_oneshot_llm_retry.py` direct-DB row deletion + re-sync pattern; the mock LLM's deterministic `FOLDER_SUMMARY_MODE` one-liner (`Fixture folder summary for <folder>.`).
|
||||
|
||||
## Objective
|
||||
While a sync runs, the status endpoint and the sync button tell the user exactly which phase the run is in — the import phase keeps its byte-identical file label, the KB-overview phase is named, and the (long) folder-summary phase reports the folder being summarized plus a done/total count — and after a sync, every source/folder whose summary is due but missing shows an explicit "Summary pending" marker in the catalog (row Description cell AND the level block) instead of an empty cell that reads as "missed". The pending set is exactly the candidate set the phase-96 gap-fill regenerates on the next sync — the marker's copy says so.
|
||||
|
||||
## Dependencies
|
||||
- `97_kb_tree_catalog` (complete) — the tree endpoint / `build_kb_tree` / `KbTree*` schemas / the RAG-view tree UI (`makeDescCell`, the level block, `renderLevel`).
|
||||
- `96_oneshot_resilience` (complete) — `missing_folder_summaries` + the `only_missing` gap-fill (the pending marker's honest "waiting" semantics) + the E2E row-deletion pattern.
|
||||
- `94_ls_tree_drilldown` (complete) — the folder-summary generator, `MIN_DOCS_PER_FOLDER`, the per-folder fail-soft loop this phase instruments.
|
||||
- `64_sync_upload_progress` (complete) — `SyncStatus`'s progress fields, the 2 s poll, the live-label contract this phase extends.
|
||||
|
||||
## Decisions recorded here (owner review — PLAN.md is being redone by the owner)
|
||||
- **D1 — additive status fields, existing contract byte-identical:** `SyncStatus` / the status JSON gain exactly four keys — `phase` (`"import"` | `"overview"` | `"summaries"` | null), `current_summary` (the `source` / `source/folder` being summarized, or null), `summaries_done`, `summaries_total` (ints, 0 idle/terminal-reset). The model-check and clone/pull prelude keeps `phase: null` (the bare "Syncing…" label stays — the phase-64 pins hold); terminal states clear `phase` + `current_summary` and KEEP the final `summaries_done` / `summaries_total` (the phase-64 keep-final-counts convention). `current_file` / `files_done` / `files_total` / `detail` are untouched.
|
||||
- **D2 — the labels (UI):** the sync button's running label for the sync job becomes: `phase "overview"` → `Writing KB overview…`; `phase "summaries"` → `Summarizing folders… <current_summary> (n/m)` (the folder part omitted when `current_summary` is null — the first poll of the phase); anything else → today's `Syncing… <file> (n/m)` logic, byte-identical (import + prelude). The UPLOAD job's label is untouched (its status endpoint has no phase). The untruncated label still rides the button `title` + `#sync-result` (the aria-live announcer); CSS ellipsizes the label span only (phase-64 A4 contract).
|
||||
- **D3 — the pending rule (ONE concept):** a source or folder node is `summary_pending` ⟺ its recursive document count ≥ `MIN_DOCS_PER_FOLDER` (2) AND it has NO stored `folder_summaries` row (AI or manual — the builder sees stored rows only). That is exactly `missing_folder_summaries`'s candidate set (phase 96) — the marker is honest: the next sync's gap-fill (or the changed-KB regeneration) will generate it. A < 2-document folder is NEVER pending (it never gets a summary — its one file line IS its description). FILE nodes carry no pending flag — the file table has no description column and the pending concept is the folder-summary one (the owner's "directories/files" is served by the source + folder rows, which are the catalog's description-bearing rows).
|
||||
- **D4 — the marker's surfaces:** the row Description cell shows the muted text `Summary pending` (a `title` carries "No stored description yet — the next sync will generate one.") with the ALWAYS-present Edit button kept (a manual save creates the row and clears the marker in place); the level block (`#kb-level`) shows when the current level has a stored description (as today) **or** is pending (title + a pending note line + the Edit button — write one manually right now); neither → hidden (today's behavior). State is text + color, never color alone (B5); the muted ink-soft pair is AA on the surface (no new hue — the phase-92 monochrome invariant).
|
||||
- **D5 — the generator gets an optional progress hook:** `generate_folder_summaries` gains `on_progress: Callable[[int, int, str, str], None] | None` (done, total, source, folder_path) — called once per sorted candidate BEFORE its attempt (manual skips advance the counter — they are instant). `None` (the `scripts/import_docs.py` CLI path) is a no-op: the CLI's log-only stats contract is unchanged. No new endpoint, no new env var, no SSE event, no migration.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
### The sync status phases (task 01)
|
||||
- `app/api/sync.py` — `SyncStatus` dataclass + the `/status` response gain the four D1 fields (defaults null/0; reset with the run at the top of `_run_sync`, exactly where `current_file` resets). `_run_sync` sets `_status.phase = "import"` immediately before `import_sources`; wraps `regenerate_overview` with `phase = "overview"` (and back — to `"summaries"` when the folder branch runs, else the run proceeds to the bump/terminal); the folder-summary branch (changed-KB full regeneration **or** the gap-fill) sets `phase = "summaries"` and passes the hook:
|
||||
```python
|
||||
def _summary_hook(done: int, total: int, source: str, folder_path: str) -> None:
|
||||
_status.current_summary = source if folder_path == "" else f"{source}/{folder_path}"
|
||||
_status.summaries_done = done
|
||||
_status.summaries_total = total
|
||||
```
|
||||
(the closure-captures-`_status` convention the file's `_hook` already uses). The no-gap skip branch sets no phase (stays `"import"`). Terminal paths (success AND the except block) set `phase = None`, `current_summary = None` — keep `summaries_done` / `summaries_total` (final counts, D1).
|
||||
- `app/rag/folder_summaries.py` — `generate_folder_summaries(..., on_progress: Callable[[int, int, str, str], None] | None = None)`: inside the `for` loop over `keys`, index `i` (enumerate), call `on_progress(i + 1, len(keys), source, folder_path)` before the manual-skip check (the counter advances for instant skips, D5). The stats dict, the fail-soft behavior, the logger line, and the flush-only transaction contract are untouched. Docstrings updated (the module rule).
|
||||
- `app/api/sync.py`'s module docstring: the status paragraph names the four new fields + the phase machine (the house "docstrings carry the contracts" rule).
|
||||
|
||||
### The sync button (task 02)
|
||||
- `frontend/assets/sources.js` — the sync-job running label: replace the single `fmtSyncLabel(kind, currentFile, done, total)` call sites for the SYNC job with a phase-aware builder (the upload job keeps the bare label — its status has no phase):
|
||||
- `phase === "overview"` → `Writing KB overview…`
|
||||
- `phase === "summaries"` → `Summarizing folders… ` + (current_summary ? `${current_summary} ` : ``) + `(${summaries_done}/${summaries_total})`
|
||||
- otherwise → today's `Syncing… [current_file] [(done/total)]` (byte-identical — the phase-64 pins).
|
||||
`enterSyncRunningState` (and the poll's `enterSyncRunningState("sync", …)` call + `initSyncButton`'s re-attach) thread the phase fields through; the button `title` + `#sync-result` mirror the full untruncated label (A4). The load-time re-attach (`initSyncButton`) re-enters a RUNNING run with whatever phase the status reports (a mid-summaries reload shows the summaries label — the never-stale contract).
|
||||
- `tests/unit/test_frontend_sync_upload.py` — the label-builder pins gain the three phase cases (+ the upload-unchanged negative case).
|
||||
|
||||
### The tree pending flag (task 03)
|
||||
- `app/schemas.py` — `KbTreeFolder` and `KbTreeSource` gain `summary_pending: bool = False` (wire-additive; `KbTreeFile` untouched).
|
||||
- `app/api/docs.py` — `build_kb_tree`: import `MIN_DOCS_PER_FOLDER` from `app.rag.folder_summaries` (the builder already imports `folder_of` from there — no new dependency edge). A source node: pending ⟺ its recursive document count (the builder already computes per-source rows) ≥ `MIN_DOCS_PER_FOLDER` AND `(source, "")` not in `summaries`. A folder node in `_level_children`: pending ⟺ `counts[sub] >= MIN_DOCS_PER_FOLDER` AND `(source, sub)` not in `summaries`. The endpoint's fetches are unchanged (the `summaries` mapping already carries ALL stored rows).
|
||||
- Unit (`tests/unit/test_kb_tree_builder.py`): the pending matrix — a ≥2-doc folder with no row → true; with a stored row (any) → false; a 1-doc folder with no row → false (never pending); the source root with ≥2 docs and no `(source, "")` row → true on the source node; a registered 0-document source → false; multi-source independence.
|
||||
- Integration (`tests/integration/test_docs_api.py`): the tree endpoint returns `summary_pending` — and the CROSS-CHECK (D3, one concept): for a seeded dataset with a partial summary table, the set of `(source, folder_path)` flagged pending in the tree (root = `""`) equals `missing_folder_summaries(db)` (phase 96's public function) — the marker can never drift from the gap-fill.
|
||||
|
||||
### The pending UI (task 04)
|
||||
- `frontend/assets/sources.js`:
|
||||
- `makeDescCell` — when `node.summary` is empty (or absent) AND `node.summary_pending` → the text span carries the marker: class `kb-summary-pending`, text `Summary pending`, `title` = `No stored description yet — the next sync will generate one.` (textContent only — the house rule). The Edit button is unaffected (always present — a manual save CREATES the row). The editor's success path (`node.summary = data.summary` in `wireDescriptionEdit`) additionally clears the flag in place: `node.summary_pending = false` (a created description is no longer pending — no re-fetch).
|
||||
- `renderLevel` — the level block shows when `node.summary` (today) OR `node.summary_pending` (new): title as today (the full source-relative path); the summary `<p>` shows the stored text, or the pending note `No description stored yet — the next sync will generate one. (You can write one yourself.)` when pending; the block stays hidden for a non-pending level with no stored description (the ls rule, unchanged).
|
||||
- `frontend/assets/styles.css` — `.kb-summary-pending { color: var(--ink-soft); }` (5.1:1 on `--surface`, AA; no italic, no new hue) — the row-cell font-size already applies (the class sits on the existing text span).
|
||||
- Source pins (`tests/unit/test_kb_tree_ui.py` + the styles.css pins there): the marker copy + title, the pending branch in `makeDescCell`/`renderLevel`, the in-place flag clear on save, the class name in the CSS.
|
||||
|
||||
### The E2E (task 05)
|
||||
`tests/e2e/test_sync_summary_visibility.py` (new; `app_server` + `mock_llm` + `db_ready` fixtures; admin login via `tests/e2e/auth_helpers.py`; the temp-local-source seeding + `POST /api/sync` pattern from `test_ls_tree_drilldown.py` / `test_oneshot_llm_retry.py`; the sync leg of the summary-phase test runs against the `slow_llm` proxy — the `test_sync_upload_progress.py` fixture, `SLOW_DELAY_S` sized so the overview + 3 folder calls outlive the recorder's 100 ms cadence by ~15×):
|
||||
1. **Endpoint — the phase machine:** seed a temp local source with TWO ≥ 2-doc folders (candidates: the source root + 2 folders = 3); start a KB-changing sync; the tight-poll recorder asserts: some running tick has `phase == "overview"`; some running ticks have `phase == "summaries"` with `summaries_total == 3`, `current_summary` non-null (starts with the source name; the root call is the bare source name), `summaries_done` monotonically increasing up to 3; every `phase == "summaries"` tick keeps `files_done == files_total` (the import finished — the pause the user reported); the terminal tick has `phase` null, `current_summary` null, and `summaries_done == summaries_total == 3`.
|
||||
2. **UI — the label:** a fresh admin page on `/sources.html` starts the sync (button click) and the page's own 2 s poll renders a label matching `/Summarizing folders/ ` with `(n/3)` at some point (generous timeout, the phase-64 UI pattern); after settle the button reads the terminal label (the existing `Synced HH:MM` contract) and `#sync-result` carries the counts.
|
||||
3. **Tree — the pending marker:** after a successful (mock-LLM) sync, DELETE one folder's `folder_summaries` row directly (the phase-96 pattern) + DELETE the source-root row; reload the RAG view (nav re-show → the phase-77 refresh re-fetch) → the two affected rows' Description cells show `Summary pending` (with the title), the intact folder's cell shows its stored `Fixture folder summary for …` line and NO marker; clicking an affected folder shows the level block with the pending note; saving a manual description from the row's Edit button clears the marker in place (the cell shows the text) — and a second unchanged sync's gap-fill regenerates the OTHER deleted row (its marker goes away, its cell carries the deterministic mock line).
|
||||
|
||||
## Tasks
|
||||
1. `01_summary_phase_status.md` — `SyncStatus` + the status JSON gain `phase` / `current_summary` / `summaries_done` / `summaries_total`; the generator's `on_progress` hook; `_run_sync` sets the phases
|
||||
2. `02_sync_label_summary_phases.md` — the sync button's phase-aware labels (overview / summaries) + the label-builder unit pins
|
||||
3. `03_tree_pending_flag.md` — `summary_pending` on the tree schemas + the `build_kb_tree` rule + the `missing_folder_summaries` cross-check
|
||||
4. `04_tree_pending_ui.md` — the row-cell + level-block pending markers + the in-place clear + the CSS class + source pins
|
||||
5. `05_e2e_summary_visibility.md` — `tests/e2e/test_sync_summary_visibility.py` + the regression sweep + the atomic commit
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the `on_progress` call matrix in the generator's suite (order/values, manual-skip advance, `None` no-op, `skip=True` unchanged); the status shape in `tests/unit/test_sync_button.py` (idle nulls/zeros; the four new keys); the label builders in `tests/unit/test_frontend_sync_upload.py` (D2's three cases + upload negative); the builder's pending matrix (`tests/unit/test_kb_tree_builder.py`); the source pins (`tests/unit/test_kb_tree_ui.py`, the styles.css class).
|
||||
- Integration: `tests/integration/test_sync_api.py` (the status field contract across the state machine); `tests/integration/test_sync_folder_summaries.py` (the hook fires on the changed-KB AND gap-fill branches, never on the skip branch); `tests/integration/test_docs_api.py` (the pending shape + the D3 cross-check).
|
||||
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` in isolation; regressions green in isolation: `test_kb_tree.py`, `test_ls_tree_drilldown.py`, `test_sync_button.py`, `test_sync_upload_progress.py`, `test_oneshot_llm_retry.py`, `test_local_directory_sources.py`.
|
||||
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] while a sync runs, `GET /api/sync/status` reports the phase: the import keeps `current_file`/counts with `phase "import"`, the overview shows `phase "overview"`, the folder span shows `phase "summaries"` with `current_summary` + a done/total that climbs to the candidate count; terminal states clear the phase + folder but keep the final summary counts
|
||||
- [ ] the RAG-page sync button reads `Writing KB overview…` / `Summarizing folders… <folder> (n/m)` in those phases (aria-live + button title carry the untruncated text) and is byte-identical to today in the prelude/import/upload cases
|
||||
- [ ] a source/folder with ≥ 2 docs and no stored summary row shows `Summary pending` in its tree row AND its level block (the marker set equals `missing_folder_summaries` — integration-pinned); a manual save or the next sync's gap-fill makes the marker go away; < 2-doc folders and file rows never show it
|
||||
- [ ] the CLI import path and its log lines are byte-identical (the hook is optional, the stats contract untouched); the agent's `ls` is byte-identical (the tree reads the same rows)
|
||||
- [ ] `uv run pytest` green; coverage >90%; ruff + pyright clean
|
||||
- [ ] `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` green in isolation (DB up: `podman compose up -d db`); the regression suites green in isolation
|
||||
- [ ] one atomic Conventional Commit, `--no-gpg-sign` (e.g. `feat(sync): surface the summary phases of a sync and mark folders whose summaries are pending`)
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 01 — Sync status phases: `phase` / `current_summary` / `summaries_done` / `summaries_total` + the generator progress hook
|
||||
|
||||
**Phase:** `98_sync_summary_visibility` · **Story:** n/a (owner request)
|
||||
|
||||
## Objective
|
||||
`GET /api/sync/status` reports which phase a running sync is in (import / overview / folder summaries) with per-folder summary progress, so the UI (task 02) can tell the user exactly what is happening while the file count sits still.
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py` — `SyncStatus` gains four fields (D1, `00_phase.md`): `phase: Literal["import", "overview", "summaries"] | None = None`, `current_summary: str | None = None`, `summaries_done: int = 0`, `summaries_total: int = 0`. The `/status` response dict returns them (null/0 idle — the dataclass defaults cover the never-run state). `_run_sync` resets all four at run start (next to the existing `current_file` / `files_done` / `files_total` reset) and drives them:
|
||||
- `phase = "import"` immediately before `import_sources` (the model-check + clone/pull prelude stays `None` — D1).
|
||||
- `phase = "overview"` before `await regenerate_overview(llm)` (inside the existing `added + updated > 0` gate).
|
||||
- the folder-summary branch: when it CALLS the generator (changed-KB full regeneration OR the `only_missing` gap-fill) set `phase = "summaries"` first and pass `on_progress=_summary_hook`; the no-gap skip branch sets no phase (stays `"import"`).
|
||||
- the `_summary_hook(done, total, source, folder_path)` closure (D5's shape — mirrors the existing `_hook` convention): `current_summary = source if folder_path == "" else f"{source}/{folder_path}"`, plus the done/total.
|
||||
- BOTH terminal paths (the success block and the `except` block) set `phase = None` and `current_summary = None` — and keep `summaries_done` / `summaries_total` (the run's final counts, the phase-64 keep-final-counts convention).
|
||||
- The module docstring's status paragraph names the four fields + the phase machine (the docstring-carries-the-contract rule).
|
||||
2. `app/rag/folder_summaries.py` — `generate_folder_summaries(db, llm, *, skip=False, only_missing=False, on_progress: Callable[[int, int, str, str], None] | None = None)`: `enumerate` the `keys` loop and call `on_progress(i + 1, len(keys), source, folder_path)` BEFORE the manual-skip check (instant skips advance the counter — D5). Guard with `if on_progress is not None` at the call site (or an early `noop` default — either, but `None` from the CLI must be a zero-cost no-op). Docstrings: the generator's docstring notes the hook (done counts processed keys including instant manual skips; `total = len(keys)` at loop start) and the module docstring's generation paragraph mentions it. No change to the stats dict, the fail-soft per-folder behavior, the flush-only contract, or the logger line.
|
||||
3. `scripts/import_docs.py` — UNCHANGED (it calls the generator without the hook — the log-only stats contract is untouched; verify by reading the call sites: no keyword to add).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/test_folder_summaries.py` extensions): the hook fires once per candidate in `keys` order with the right `(done, total, source, folder_path)` (done climbs 1..total, total = the candidate count); manual-skip keys still advance the counter; a failed (LLMError) key still advances; `only_missing=True` → total = the missing count; `skip=True` → zero calls; `on_progress=None` → the generator behaves byte-identically to today (the fake LLM call log unchanged).
|
||||
- Unit (`tests/unit/test_sync_button.py` — the status-shape pins): the `/status` dict carries the four new keys — null/0 in the idle state; extend any running-state shape assertions.
|
||||
- Integration (`tests/integration/test_sync_api.py` + `tests/integration/test_sync_folder_summaries.py`): the status contract across the state machine (idle → running carries `phase "import"` with the file hook as today; terminal success/failed clear `phase` + `current_summary` and keep the final summary counts); the hook fires on BOTH generation branches (changed-KB regeneration, unchanged-KB gap-fill) and NEVER on the no-gap skip branch (a zero-`FOLDER_SUMMARY_MODE`-call unchanged re-sync leaves `phase` at `"import"` through to the terminal).
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/sync/status` (idle) returns `phase: null`, `current_summary: null`, `summaries_done: 0`, `summaries_total: 0` alongside the unchanged existing keys
|
||||
- [ ] the generator's `on_progress` call matrix is unit-pinned (order, values, skip/fail advance, `None` no-op); the CLI import path is untouched
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work (the phase-64/94/96 sync + folder-summary suites green)
|
||||
@@ -0,0 +1,25 @@
|
||||
# Task 02 — The sync button names the summary phases (label builder + pins)
|
||||
|
||||
**Phase:** `98_sync_summary_visibility` · **Story:** n/a (owner request)
|
||||
|
||||
## Objective
|
||||
The RAG-page sync button (and its aria-live line) shows the user what the running sync is doing in the post-import span: `Writing KB overview…` and `Summarizing folders… <folder> (n/m)` — while the prelude, import, and upload labels stay byte-identical (the phase-64 contract).
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/sources.js` — the sync-job running label (D2, `00_phase.md`):
|
||||
- Replace/augment the label path: the SYNC job's running label is now phase-aware — `status.phase === "overview"` → `Writing KB overview…`; `status.phase === "summaries"` → `Summarizing folders… ` + (status.current_summary ? `${status.current_summary} ` : ``) + `(${status.summaries_done}/${status.summaries_total})`; anything else (null phase prelude, `"import"`, and every field the old label used) → today's `Syncing… [current_file] [(done/total)]` logic, byte-identical.
|
||||
- Thread the fields: `enterSyncRunningState` gains the phase fields for the sync job (extend its signature or pass the status object — keep the upload call sites on the bare label: the upload status has no phase, D2). Update the poll's `enterSyncRunningState("sync", …)` call and `initSyncButton`'s running re-attach (a reload mid-summaries must re-enter with the summaries label — the never-stale contract).
|
||||
- The A4 contract carries over: the untruncated label rides the button `title` (set for every running label, not just file labels — remove/adjust the "title only when there is a current file" rule for the phase labels) and `#sync-result` (the aria-live announcer); CSS ellipsizes the label span only (no CSS change expected — verify the existing `.sync-label` ellipsis handles the longer text; the mobile `max-width: none` override already lifts the cap).
|
||||
- The module docstring's sync section: update the decision-tree comments (branch 1 now names the three label forms).
|
||||
2. `tests/unit/test_frontend_sync_upload.py` — label-builder pins: the overview label (exact string, no counts); the summaries label WITH `current_summary` (exact: `Summarizing folders… <src>/<folder> (1/3)` shape) and without (bare `Summarizing folders… (1/3)`); the fall-through — a null-phase and an `"import"`-phase status produce today's byte-identical labels (the existing pins keep passing); the upload job's label is untouched (negative case).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the label-builder matrix above (this task's core — the `00_phase.md` D2 strings are the contract, pinned to the exact copy).
|
||||
- Coverage: **>90%** on this task's new/modified code (frontend source pins are the house pattern for JS — the unit file above IS the coverage for this task; `app/` unchanged).
|
||||
- Note: the E2E proof of the live label lands in task 05 (the UI leg) — do NOT add an E2E file here.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the three sync-job label forms are pinned to the exact D2 strings (overview / summaries±folder / today's fall-through byte-identical); the upload label is pinned untouched
|
||||
- [ ] a mid-summaries page reload re-enters the running state with the summaries label (the `initSyncButton` path, source-pinned)
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work (the phase-64 sync-label pins green)
|
||||
@@ -0,0 +1,26 @@
|
||||
# Task 03 — `summary_pending` on the tree: the schemas + `build_kb_tree` + the gap-fill cross-check
|
||||
|
||||
**Phase:** `98_sync_summary_visibility` · **Story:** n/a (owner request)
|
||||
|
||||
## Objective
|
||||
`GET /api/docs/tree` flags every source/folder whose summary is due but missing (`summary_pending`) — exactly the candidate set the phase-96 gap-fill regenerates — so the UI (task 04) can say "waiting to generate" instead of showing an empty cell.
|
||||
|
||||
## Work
|
||||
1. `app/schemas.py` — `KbTreeFolder` and `KbTreeSource` gain `summary_pending: bool = False` (wire-additive; the `KbTree` docstring + the two node docstrings note the D3 rule from `00_phase.md`: pending ⟺ recursive count ≥ `MIN_DOCS_PER_FOLDER` AND no stored row; `KbTreeFile` stays untouched).
|
||||
2. `app/api/docs.py` — `build_kb_tree` (and `_level_children` where folder nodes are built):
|
||||
- Import `MIN_DOCS_PER_FOLDER` from `app.rag.folder_summaries` (next to the existing `folder_of` import — no new dependency edge; `app.rag.folder_summaries` already depends on nothing in `app.api`).
|
||||
- Folder node in `_level_children`: `summary_pending = counts[sub] >= MIN_DOCS_PER_FOLDER and (source, sub) not in summaries`.
|
||||
- Source node in `build_kb_tree`: pending ⟺ the source's recursive document count (the builder already groups `doc_rows` per source) ≥ `MIN_DOCS_PER_FOLDER` AND `(source, "") not in summaries`.
|
||||
- `build_kb_tree`'s docstring: the pending rule (D3) + the pointer that it is `missing_folder_summaries`'s candidate set (one concept — the cross-check below pins it).
|
||||
3. Endpoint: NO fetch change (the `summaries` mapping the endpoint already builds carries ALL stored rows — AI and manual).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (`tests/unit/test_kb_tree_builder.py` extensions) — the pending matrix: a ≥ 2-doc folder with no stored row → `summary_pending` true; the same folder WITH a stored row (the builder cannot tell AI from manual — any row) → false; a 1-doc folder with no row → false (a single-document folder never gets a summary — never pending); the name-sharing edge (the phase-94 count rule): documents `one/a` AND `one/a/b` → folder `one/a` exists and its recursive count is 2 (the document whose path EQUALS the folder name counts) → pending true with no stored row — pending follows the RECURSIVE count, not the number of direct children; the source root: a source with ≥ 2 docs and no `(source, "")` row → the SOURCE node is pending; a registered 0-document source → never pending; two sources pending independently (one with a root row, one without).
|
||||
- Integration (`tests/integration/test_docs_api.py` extensions): the endpoint returns `summary_pending` on source + folder nodes (both values, a nested case); **the D3 cross-check (one concept end to end):** seed a multi-folder dataset, DELETE some `folder_summaries` rows (the phase-96 pattern), then assert `set of (source, folder_path) flagged pending in the fetched tree (root = "")` == `set(missing_folder_summaries(db))` — the marker can never drift from the gap-fill.
|
||||
- Coverage: **>90%** on this task's new/modified code (full gate: `app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the tree JSON carries `summary_pending` on source + folder nodes only, computed by the D3 rule (unit matrix green)
|
||||
- [ ] the integration cross-check: pending set == `missing_folder_summaries(db)` on a dataset with partial summaries
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work (the existing tree shape assertions in `test_kb_tree_builder.py` / `test_docs_api.py` still pass — the field is additive)
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 04 — The "Summary pending" markers: row cell + level block + the in-place clear
|
||||
|
||||
**Phase:** `98_sync_summary_visibility` · **Story:** n/a (owner request)
|
||||
|
||||
## Objective
|
||||
The catalog tells the owner a summary is waiting: a pending source/folder row shows `Summary pending` in its Description cell (Edit button kept) and the level block shows a pending note when drilled into — writing a manual description clears the marker in place.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/sources.js`:
|
||||
- `makeDescCell` (D4, `00_phase.md`): when `node.summary` is empty/absent AND `node.summary_pending` → the text span carries the marker — class `kb-summary-pending`, textContent `Summary pending`, `title` = `No stored description yet — the next sync will generate one.` (textContent only — the house rule; the span already exists, this only changes its content/class/attribute in the pending case). The Edit button is UNCHANGED (always present — a manual save creates the row).
|
||||
- `wireDescriptionEdit`'s success path: after `node.summary = data.summary`, also set `node.summary_pending = false` (a created/updated description is no longer pending — the in-place clear, no re-fetch).
|
||||
- `renderLevel`'s level-block branch: show `#kb-level` when `node.summary` (as today) OR `node.summary_pending` — title as today (the full source-relative path); `#kb-level-summary` shows the stored text, or — when pending — the note `No description stored yet — the next sync will generate one. (You can write one yourself.)`; a level that is neither stored nor pending stays hidden (the ls rule, unchanged). The block's Edit button (write one manually now) already ships in the static markup.
|
||||
- Module docstring: the phase-98 note (the marker's surfaces + the in-place clear).
|
||||
2. `frontend/assets/styles.css` — `.kb-summary-pending { color: var(--ink-soft); }` (5.1:1 on `--surface` — AA; text + color, never color alone (B5); no italic, no new hue — the phase-92 monochrome invariant). The class sits on the existing text span, so the row cell's existing font-size/line-height apply — the marker must not change row height.
|
||||
3. Source pins (`tests/unit/test_kb_tree_ui.py` + its styles.css pins): the marker copy + `title` (exact strings from D4), the pending branch condition in `makeDescCell` (pending + no summary → marker; a stored summary → the text, never the marker; non-pending + no summary → the empty cell as today), the in-place `summary_pending = false` on save, the `renderLevel` pending-note branch (exact note string), and the `.kb-summary-pending` class in the CSS.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the source pins above are this task's test layer (the house pattern for frontend logic — `app/` untouched).
|
||||
- Coverage: **>90%** on this task's new/modified code (frontend pins cover the JS; `app/` unchanged).
|
||||
- Note: the E2E proof of the markers (incl. the gap-fill recovery) lands in task 05.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a pending row's Description cell shows `Summary pending` (with the D4 title) + the always-present Edit button; a stored-description row and a non-pending empty row are unchanged
|
||||
- [ ] drilling into a pending level shows the level block with the pending note + Edit; a stored level and a neither-stored-nor-pending level behave exactly as today
|
||||
- [ ] saving a manual description clears the marker in place (source-pinned; no re-fetch)
|
||||
- [ ] full test suite green, coverage >90%
|
||||
- [ ] no behavior change in completed work (the phase-97 tree/editor pins green)
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 05 — The dedicated E2E: phase machine live, labels live, pending markers + gap-fill recovery
|
||||
|
||||
**Phase:** `98_sync_summary_visibility` · **Story:** n/a (owner request)
|
||||
|
||||
## Objective
|
||||
Pin the owner-visible contract in a browser: the sync's summary phases are visible live (endpoint + button label), and missing folder summaries read "waiting to generate" — then the next sync's gap-fill makes the marker go away.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_sync_summary_visibility.py` (new — the phase's dedicated A16 suite, run in isolation). Fixtures: `app_server` + `mock_llm` + `db_ready` (the `conftest` pattern); admin login via `tests/e2e/auth_helpers.py`; the temp-local-source seeding + `POST /api/sync` pattern from `test_ls_tree_drilldown.py` / `test_oneshot_llm_retry.py`; for test 1–2 the `slow_llm` proxy fixture from `tests/e2e/test_sync_upload_progress.py` (`SLOW_DELAY_S` sized so ONE LLM call outlives the recorder's ~100 ms cadence by ~15× — the phase-64 sizing rationale; the mock LLM stays on its port for the fast tests). An autouse cleanup deletes the temp source's KB rows + `folder_summaries` rows (the phase-96 cleanup pattern).
|
||||
- **`test_sync_status_reports_the_summary_phases`** (endpoint — the slow-LLM leg): seed a temp local source with TWO ≥ 2-doc folders (candidate set: the source root + 2 folders = 3 — the root is a candidate too, `folder_path ""`); make the KB change-gate fire (first sync of the source); a daemon-thread recorder tight-polls `GET /api/sync/status` (~100 ms) from the 202 until the terminal state (the phase-64 recorder pattern). Assert: some running tick has `phase == "overview"`; some running ticks have `phase == "summaries"` with `summaries_total == 3`, non-null `current_summary` (starts with the source name; one tick is the bare source name — the root call), `summaries_done` strictly increasing across the summaries ticks up to 3; every `phase == "summaries"` tick has `files_done == files_total` (the import finished — the reported pause is now labeled); the terminal tick: `phase` null, `current_summary` null, `summaries_done == summaries_total == 3`, `state "success"`.
|
||||
- **`test_sync_button_names_the_summary_phase`** (UI — the slow-LLM leg): a fresh admin page on `/sources.html`; click the sync button; within a generous timeout (the phase-64 UI pattern) the page's own 2 s poll renders `#sync-label` matching `/Summarizing folders/` with a `(n/3)` suffix (the label is built by the page JS — this test asserts the RENDERED text, not the endpoint); on settle the button shows the terminal `Synced …` label (the phase-32 contract) and `#sync-result` carries the counts line.
|
||||
- **`test_missing_folder_summaries_read_as_pending_and_self_heal`** (tree — the fast mock LLM): seed a source with two ≥ 2-doc folders + a root, sync (mock `FOLDER_SUMMARY_MODE` → every candidate gets the deterministic `Fixture folder summary for <folder>.` line); open the RAG view: no `Summary pending` anywhere. DELETE one folder's row AND the source-root row directly (`db_ready` — the phase-96 pattern); trigger the phase-77 re-fetch (re-click the active RAG nav link, or `page.reload()`): the source row (top level) + the affected folder's row show `Summary pending` (+ the D4 title attribute), the intact folder's cell shows its stored line and NO marker; click the affected folder → the level block is visible with the pending note (exact D4 string); use the row's Edit → Save a manual description → the marker is gone in place (the cell shows the saved text — no reload); then run a second (unchanged-KB) sync → the OTHER deleted row (the source root) is gap-filled: its marker is gone, its cell/level shows the deterministic mock line.
|
||||
2. `tests/e2e/mock_llm.py` — NO change expected (the `FOLDER_SUMMARY_MODE` branch + the phase-96 failure injection already cover this suite's needs); if a test needs a deterministically SLOWED folder call the slow proxy is the seam (it delays every request it proxies) — do not add new mock branches unless a test genuinely requires one.
|
||||
3. Regression sweep (each in isolation, `--no-cov`, DB up): `test_kb_tree.py`, `test_ls_tree_drilldown.py`, `test_sync_button.py`, `test_sync_upload_progress.py`, `test_oneshot_llm_retry.py`, `test_local_directory_sources.py`. Fix ONLY pins broken by the additive changes (new status keys, the new schema field, the marker) — asserted behavior changes nowhere else.
|
||||
4. Full gates + commit: `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean; move `98_sync_summary_visibility` → `.agents/phases/complete/`; one atomic commit:
|
||||
```bash
|
||||
git add -A .agents/ app/ frontend/ tests/ && git commit --no-gpg-sign -m "feat(sync): surface the summary phases of a sync and mark folders whose summaries are pending"
|
||||
```
|
||||
|
||||
## Testing & Quality
|
||||
- E2E (mandatory, A16): `uv run pytest tests/e2e/test_sync_summary_visibility.py -v --no-cov` green in isolation.
|
||||
- Coverage: **>90%** on `app/` (the full suite gate).
|
||||
- Lint/types: `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the three E2E tests above pass in isolation (the phase machine + the live label + the pending markers + the gap-fill recovery)
|
||||
- [ ] the regression suites pass in isolation
|
||||
- [ ] full suite green, coverage >90%, ruff + pyright clean
|
||||
- [ ] phase dir moved to `complete/`, one atomic `--no-gpg-sign` Conventional Commit
|
||||
Reference in New Issue
Block a user