feat(chat): invalidate saved chats on sources sync — versioned stamps, stale marker, Regenerate against the new index
This commit is contained in:
@@ -1,27 +0,0 @@
|
||||
# Task 01 — Sources Version Table + `saved_chats.sources_version`
|
||||
|
||||
**Phase:** `53_stale_saved_chats` · **Source:** `TODO.md:4` — "Make sure the saved chats are invalidated if the docs are synced, that way it generates a new answer with new data"
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
A durable record of "which generation of the KB was this chat saved against": a single-row `sources_meta` version counter plus a `sources_version` stamp on every `saved_chats` row.
|
||||
|
||||
## Work
|
||||
1. `app/models.py` — add `SourcesMeta` (single row, the `KbOverview` id=1 precedent): `id` Integer PK `server_default="1"`, `version` Integer NOT NULL `server_default="0"`, `updated_at` timestamptz (`server_default=func.now()`, `onupdate=func.now()`). Add `SavedChat.sources_version: Mapped[int]` — Integer, NOT NULL, `server_default="0"`, with the model docstring noting: existing rows stamp 0 = "the pre-counter KB" and become stale on the first bump.
|
||||
2. `alembic/versions/0010_sources_version.py` — ONE migration for the feature: `op.create_table("sources_meta", …)` + the seed row (id 1, version 0) + `op.add_column("saved_chats", sa.Column("sources_version", sa.Integer(), nullable=False, server_default="0"))`. Downgrade reverses in the safe order (drop column, drop table). `down_revision` = the 0009 revision (verify against `alembic/versions/0009_saved_chat_share_token.py`).
|
||||
3. `app/rag/sources_meta.py` (new module) — `current_sources_version(db: Session) -> int` (PK read of the seed row; returns 0 when the row is absent — defensive, never raises) and `bump_sources_version(db: Session) -> int` (upsert the row, `version += 1`, `db.flush()` — the CALLER commits, because the two sync paths each own their session; returns the new version).
|
||||
4. `tests/integration/test_migration_0010.py` — the house migration pattern (model from `tests/integration/test_migration_0009.py`): upgrade head → the seed row exists (version 0), `saved_chats.sources_version` is NOT NULL with default 0, an inserted row round-trips; downgrade → column + table gone.
|
||||
5. `tests/unit/test_sources_meta.py` — the helpers on a real/fixture session (house DB-test pattern): absent row → `current` 0; first bump → 1; second bump → 2; `current` after a bump reflects it; two bumps in two sessions don't race to the same value (flush-order check is out of scope — one writer at a time is the deployment reality).
|
||||
|
||||
- ASSUMPTION: both schema changes ship in ONE migration (0010) — one feature, one atomic schema change; the seed row is inserted in the migration (not lazily on first bump), so `current_sources_version` is a plain PK read.
|
||||
- ASSUMPTION: the helper module lives at `app/rag/sources_meta.py` (rag domain — both sync paths already import from `app.rag.*`); no new setting, no new dependency.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: `tests/integration/test_migration_0010.py` green.
|
||||
- Unit: `tests/unit/test_sources_meta.py` green.
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run alembic upgrade head` applies 0010 cleanly on a fresh DB (and on a DB already at 0009); the seed row is present with version 0.
|
||||
- [ ] `uv run pytest tests/integration/test_migration_0010.py tests/unit/test_sources_meta.py -v` green.
|
||||
- [ ] `uv run pytest` green (no existing test broken by the new NOT NULL column); `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -1,26 +0,0 @@
|
||||
# Task 02 — Bump the Version on Every KB-Changing Sync
|
||||
|
||||
**Phase:** `53_stale_saved_chats` · **Source:** `TODO.md:4` — "Make sure the saved chats are invalidated if the docs are synced, that way it generates a new answer with new data" (this task wires the bump into the two canonical sync paths)
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
Every sync path that changed the knowledge base bumps `sources_meta.version` exactly once — the in-app Sync button and the CLI — so the staleness marker tracks the index, not the click.
|
||||
|
||||
## Work
|
||||
1. `app/api/sync.py` — in `_run_sync`, after `summary = await import_sources(sources, llm, prune=True)` succeeds and the overview decision is made: when `summary.added + summary.updated + summary.pruned > 0`, open a short-lived `SessionLocal()` (the run's status is in-memory; the `effective_sources` call above sets the exact pattern — open, use, close in `finally`) and run `bump_sources_version(db)` + commit. Add `"sources_version": <new version>` to `_status.detail` (the never-stale status object gains the new generation number). Any exception before this point (model check, source resolution, clone/pull, import failure) aborts the run in the `failed` state — a FAILED sync never bumps, so it never invalidates chats.
|
||||
- Note the deliberate gate difference: the overview regenerates on `added + updated > 0`, but the version bump also covers `pruned > 0` — a pruned document can invalidate a saved answer that cited it. The two gates stay separate on purpose.
|
||||
2. `scripts/import_docs.py` — in `_run` (the `asyncio.run` coroutine), the same bump with the same gate: after the `--limit`/unchanged early-returns, when `summary.added + summary.updated + summary.pruned > 0`, bump via a short `SessionLocal()` (the file already opens sessions for `_overview_row_exists`). `--limit` debug runs NEVER bump (an incomplete walk is debug-only — mirrors the existing `--limit` overview skip); an unchanged re-run (`added + updated + pruned == 0`) NEVER bumps. Add `sources_version=<value or "skipped">` to the summary `print` line (the cron/quadlet log surface).
|
||||
3. Tests:
|
||||
- `tests/integration/test_sync_api.py` — extend (fakes pattern from `tests/fakes.py`): a sync whose import changed the KB bumps the version exactly once AND the `/api/sync/status` detail carries `sources_version`; an unchanged re-sync (all `unchanged`) does NOT bump; a failing sync (e.g. the phase-41 model-down path) does NOT bump.
|
||||
- `tests/integration/test_import_docs_overview.py` (or the existing `import_docs` gate test if the file that pins `--limit`/unchanged gating lives elsewhere — match the file that already asserts the overview gate) — extend with the bump asserts: changed run → bump + `sources_version` in the print line; `--limit` run → no bump, `skipped`; unchanged run → no bump, `skipped`.
|
||||
- ASSUMPTION: the bump commits independently of the best-effort overview regeneration — a failed `lite` overview never rolls back the version (the index did change), and the bump happens in its own short session so it survives an overview exception.
|
||||
- ASSUMPTION: `scripts/import_docs.py --source <manual dir>` runs DO bump when they change the KB — a manual import changes the index exactly like a git sync (the TODO says "if the docs are synced"; the CLI is the other canonical sync).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: the extended sync + import_docs suites green (`uv run pytest tests/integration/test_sync_api.py tests/integration/test_import_docs_overview.py -v`).
|
||||
- Coverage: **>90%** on `app/` (the sync-path bump is in `app/api/sync.py`; the CLI bump is in `scripts/`, covered by its own tests).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] A changed sync (both paths) bumps exactly once; unchanged / `--limit` / failed syncs never bump.
|
||||
- [ ] No behavior change in completed phases (the sync pipeline's steps 1–5 and exit codes are untouched apart from the new bump + detail field / print token).
|
||||
@@ -1,23 +0,0 @@
|
||||
# Task 03 — Stamp-on-Save + `stale` on the Chats API
|
||||
|
||||
**Phase:** `53_stale_saved_chats` · **Source:** `TODO.md:4` — "Make sure the saved chats are invalidated if the docs are synced, that way it generates a new answer with new data" (this task exposes staleness on the admin chats API)
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
A saved chat knows the KB generation it was saved against, and the API tells the client whether it is stale — no client-side staleness math.
|
||||
|
||||
## Work
|
||||
1. `app/api/chats.py` — `create_chat`: stamp `row.sources_version = current_sources_version(db)` on the PENDING row (ships in the same INSERT, the `share_token` precedent). `update_chat` (the Re-Save upsert): re-stamp `sources_version` to the current value — a Re-Save is the owner affirming this content against the current KB. Share/unshare are UNCHANGED (raw SQL touching only `share_token` — the version, like `updated_at`, is untouched).
|
||||
2. `app/schemas.py` — add `stale: bool` to `SavedChatRow` (list shape) and `SavedChatOut` (detail shape). `SharedChatOut` is UNCHANGED — the public snapshot is frozen by design (phase 51; the staleness surface is admin-only).
|
||||
3. `app/api/chats.py` — `list_chats` + `get_chat`: call `current_sources_version(db)` ONCE per request and compute `stale = row.sources_version < current` inside `_to_row` / `_to_out` (pass the current version in — the module functions stay pure).
|
||||
4. `tests/integration/test_chats_api.py` — extend: create stamps the current version (assert via the helper read); bump (`bump_sources_version` in the test) → `GET /api/chats` rows and `GET /api/chats/<id>` both report `stale: true`; a Re-Save (`PUT`) re-stamps → `stale: false` again; share/unshare leave `sources_version` untouched; the 403/anonymous contract is unchanged.
|
||||
- ASSUMPTION: Re-Save ALWAYS re-stamps to the current version (not "only when stale") — uniform semantics, one fewer code path; a re-Save is also the manual escape hatch for a false-positive stale row.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: `tests/integration/test_chats_api.py` green (extended).
|
||||
- Coverage: **>90%** on `app/` (validate.sh gate).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `POST` / `PUT /api/chats` stamp `sources_version`; list + detail carry `stale`; anonymous 403s unchanged; `/api/shared/<token>` body unchanged.
|
||||
- [ ] `uv run pytest` green; coverage TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed phases (existing `test_chats_api.py` / `test_share_chat.py` assertions still hold).
|
||||
@@ -1,23 +0,0 @@
|
||||
# Task 04 — Stale Column on the History Table
|
||||
|
||||
**Phase:** `53_stale_saved_chats` · **Source:** `TODO.md:4` — "Make sure the saved chats are invalidated if the docs are synced, that way it generates a new answer with new data" (this task surfaces staleness on the History page)
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
The History page shows at a glance which saved chats predate the last KB-changing sync.
|
||||
|
||||
## Work
|
||||
1. `frontend/history.html` — add a `Stale` `<th>` to the full-width table, between the `Updated` and `Actions` columns (house rule 5: a column in the full-width table — no skinny side list).
|
||||
2. `frontend/assets/history.js` — render the cell from `row.stale` (the phase-50 row shape, extended in task 03): a rose `Stale` pill when true, an em-dash when false; the pill carries `title="Sources have changed since this chat was saved — open the chat to Regenerate"` and the `<td>` an `aria-label` so the marker is conveyed without the visual (WCAG 2.1 AA). Non-stale rows render exactly as before.
|
||||
3. `frontend/assets/styles.css` — the `.stale-pill` treatment: rose family (the Stop-treatment tokens, so "stale" reads in the same visual language as the in-flight control), theme-token based (both themes), contrast ≥4.5:1 on `--surface`.
|
||||
4. Source pins (`tests/unit/test_history_page.py` extended, house pin style): the `history.js` stale-cell branch (pill class + em-dash fallback + the `aria-label`), the `Stale` `<th>` in `history.html`, and the `.stale-pill` rule in `styles.css`.
|
||||
- ASSUMPTION: the marker is a READ-ONLY badge — opening the row shows the chat-page banner (task 05) which carries the Regenerate action; no bulk "regenerate all" in v1, no per-row action button (the table's Actions column keeps its Open/Share/Delete trio).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the extended `tests/unit/test_history_page.py` pins green.
|
||||
- Coverage: **>90%** — no `app/` change; the gate stays green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The History table renders the Stale column; stale rows show the pill, fresh rows the em-dash; the table stays full-width (AGENTS.md rule 5).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed phases (the Open/Share/Delete flows and the cache-busting contract are untouched).
|
||||
@@ -1,30 +0,0 @@
|
||||
# Task 05 — Stale Banner + Regenerate on the Chat Page
|
||||
|
||||
**Phase:** `53_stale_saved_chats` · **Source:** `TODO.md:4` — "Make sure the saved chats are invalidated if the docs are synced, that way it generates a new answer with new data" (this task delivers the regeneration on the chat page)
|
||||
**Story:** n/a (TODO-derived)
|
||||
|
||||
## Objective
|
||||
Opening a saved chat that predates the current KB shows a banner; **Regenerate** re-asks the last question against the new index, in place, and re-saves the row so the stored answer is current again.
|
||||
|
||||
## Work
|
||||
1. `frontend/index.html` — add a `#stale-banner` section directly after `#kb-banner` (the chat-shell top-of-column position, `role="status"`, `hidden` by default): the text "The sources have been updated since this chat was saved." + a `#stale-regenerate` button labeled "Regenerate" (the brand-pill treatment, matched to the Save/Share pair; icon = the redo glyph, reused from the phase-49 Retry asset in `app.js`).
|
||||
2. `frontend/assets/app.js`:
|
||||
- `retryLastTurn(wrap)` (phase 49, ~L1381): make it RETURN the `runTurn(text, { reask: true })` promise (today it is `void runTurn(...)`) — the existing Retry click handler ignores the return value, so phase-49 behavior is byte-identical; the Regenerate path needs the turn's completion to know when to persist.
|
||||
- The `?chat=<id>` boot-load block (the phase-50 code around `fetch(\`/api/chats/${chatId}\`)` → `currentChatId = chatId` → `saveConversation()` → `history.replaceState`): when the fetched payload has `stale: true`, reveal `#stale-banner` (remove `hidden`).
|
||||
- New handler for `#stale-regenerate`: call `retryLastTurn(wrap)` on the LAST brain bubble's rendered wrap (the phase-49 targeting — reuse the `markLastRetryable` / last-`.msg.brain`-wrap resolution; the guard `wrap !== lastBrainWrap` already protects against a stale click), AWAIT the returned turn promise; on completion WITHOUT an error banner, persist the linked row through the existing upsert path (linked → `PUT /api/chats/<id>` — the server re-stamps `sources_version`, see task 03), then hide the banner and announce the outcome in the existing `#send-status` live region (PLAN §7.4 never-stale). A 404 PUT (row deleted from History meanwhile) follows the `saveCurrentChat` stale-link rule: unlink + recreate — the owner is never left with an unsaved conversation.
|
||||
- Guard: a stale conversation with NO brain record (user-only / deflection-only) — the banner shows WITHOUT the Regenerate button (text only); `retryLastTurn` is never called in that state.
|
||||
3. `frontend/assets/styles.css` — `#stale-banner` styling (the kb-banner family, distinct redo icon) + the Regenerate button; theme tokens only, contrast ≥4.5:1 in both themes; banner stacks correctly above `#kb-banner` when both are visible (e.g. empty-KB banner + stale chat — kb-banner wins the top slot, stale-banner directly below).
|
||||
4. Source pins (house style, extend `tests/unit/test_save_chat_ui.py`): the banner reveal branch on `payload.stale`, the Regenerate → `retryLastTurn` wiring, the `retryLastTurn` return-promise change, the post-regenerate linked PUT, and the no-brain-record guard.
|
||||
- ASSUMPTION: Regenerate = the phase-49 redo-in-place of the LAST brain bubble ONLY — the full conversation context is kept, earlier answers are not re-run (re-asking every question is out of scope for v1; the TODO's "a new answer" is satisfied by the answer the user was reading going stale).
|
||||
- ASSUMPTION: the auto re-save after a successful regenerate is the linked `PUT` (the row's `updated_at` bumps — it IS a content edit, so the History "latest activity" order follows it); a regenerate that errors mid-stream (the error banner shows) leaves the row untouched — stale stays true; a regenerate STOPPED mid-stream (phase 48) persists the stopped partial via the same PUT (the owner engaged with the new index).
|
||||
- ASSUMPTION: localStorage-only (unsaved) conversations are never stale — staleness is a property of the saved row; the banner appears only on the `?chat=<id>` boot path, never on the phase-14 local restore.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the extended `tests/unit/test_save_chat_ui.py` pins green.
|
||||
- Coverage: **>90%** — no `app/` change in this task; the gate stays green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] Opening a stale `?chat=<id>` shows the banner; Regenerate streams the fresh answer in place and clears the banner after the linked row is re-saved.
|
||||
- [ ] The phase-49 Retry button behaves exactly as before (pin-verified: the return-promise change is behavior-neutral for the existing click handler).
|
||||
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] No behavior change in completed phases (phase-14 local restore, phase-48 stop, phase-50 save/load flows).
|
||||
Reference in New Issue
Block a user