phase: 189_chat_save_long_turns
**Phase 189 final verification — all green**
- **Verified (all 4 tasks already implemented):** A1 caps in `app/schemas.py` — `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; `text`/`thinking` 600 000, `error`/`image` 500, `answer_text` 50, `messages` 500, 8 M budget all UNCHANGED; rationale docstrings rewritten to `merge_cited_docs`/`agent_max_rounds`. A2 `shareCurrentChat` linked branch upserts (PUT, 404→create+re-link) before `POST /{id}/share`, aborts on failure (banner, no token); unlinked branch byte-identical. A3 `savePending` + `finally` re-run. A4 `saveFailedSticky` + one 5 s `setTimeout` retry on network/5xx only, 4xx never retries.
- `uv run pytest --cov=app --cov-report=term-missing` → **4609 passed, 1 warning, EXIT=0**, `TOTAL 99%`, `app/schemas.py 100%` (>90%).
- `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` → **4 passed, EXIT=0** (isolation).
- Regression green & **unedited** (`git status` clean): `test_chat_save_payload_limits.py` 2, `test_chat_history.py` 5, `test_share_chat.py` 4. Extra blast-radius sweep (share/save/persist/segments surface, all EXIT=0): `test_save_share_ux`, `test_stale_saved_chats`, `test_history_pagination`, `test_history_copy`, `test_bottom_chat_actions`, `test_reasoning_bubble`, `test_tool_budget_telemetry`, `test_verbatim_show_text`, `test_smoke`. No leaked `saved_chats` rows (`save-long-%` = 0).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- **Completion criteria:** caps ✓ · incident payload 22/33/51 → 201/200 anonymously + 201 tools/451 segments/101 suggestions → 422 naming the offender, row unchanged ✓ · share-saves-first + abort + unlinked byte-identical ✓ · overlap re-run + sticky note + one-shot retry ✓ · phase E2E in isolation + 3 regression suites unedited ✓ · full suite / coverage / lint / types ✓ · `git diff --stat` = `app/schemas.py`, `frontend/assets/app.js`, 3 test files, `README.md` + new E2E & phase files (no `app/api/chats.py`, no migration, no new dependency) ✓ · commit + `todo/`→`complete/` move left to the harness (rules forbid git index writes).
- **Docs:** full README review (all 2 258 lines) + machine cross-check — all 85 settings documented in the Configuration reference, all 34 `.env.example` vars documented, documented defaults match `app/config.py`. Fixed 2 inconsistencies: (1) the 117-char prose line task 04 left when reflowing the Saved-chats pager paragraph; (2) the phase-189 sentence implying the 8 M total bounds the widened arrays — now states precisely that the total counts `text`/`thinking`/verbatim-block text only, and that the arrays are bounded by fixed item COUNT (matches `_require_within_total_budget`). AGENTS.md verified consistent — it enumerates no save contract or list cap (rule 5's phase-166 in-flight-turn clause untouched), so **no AGENTS.md change** per the phase mandate.
- **Notable/deviation:** the wider caps raise the *theoretical* anonymous-POST worst case (arrays are outside the 8 M char budget); left as designed (A1 fixes every other bound), documented honestly in README — realistic payloads stay ~10 MB, so the existing `client_max_body_size ≥ 16 MB` guidance still holds. No live-infra changes were made (data-repair of the two stale shared rows stays owner action).
- **Next pending phase:** none — `189_chat_save_long_turns` is the last entry in `.agents/phases/todo/`.
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# Phase 189 — Long agent turns save (and share) the whole conversation
|
||||
|
||||
**Source:** owner incident 2026-10-16 — the shared chat `0101d2ca-b24c-4d13-9f65-c0837e993c50` (row `8b345ab5-aab3-4412-9642-0f869b3cec4d`) shows ONLY the user's question although the LLM answered it completely; the shared row `be92e464-7918-4907-9186-09d8d98ff2c1` carries the same signature.
|
||||
**Story:** n/a (incident-derived, owner directive 2026-10-16)
|
||||
**Depends on:** none functionally (builds on the completed phase-55/83/158/164 saved-chat surface).
|
||||
|
||||
## Evidence (diagnosed 2026-10-16, production DB + app logs — read this, not the chat)
|
||||
|
||||
1. The row behind the shared token: `n_msgs = 1`, `who0 = user`, `created_at == updated_at` — the row was written EXACTLY ONCE (the send-time auto-save `POST /api/chats`) and never updated. 84 other single-write user-only rows exist (2 shared, incl. this one); the 2-message rows (279) were written >1 time — the pattern is specific to the *last* save point of a conversation.
|
||||
2. The app log for this row shows the whole sequence: the turn's per-turn line (`thinking=xhigh thinking_chars=23937 tool_calls=33 rounds=34 cap_hit=0 total_ms=400173`, `POST /api/chat 200 stream=1` — the client consumed the FULL 400 s stream, so the done frame landed) followed by **`PUT /api/chats/8b345ab5… 422 Unprocessable Entity`** — the done-time auto-save REJECTED — then `GET /api/chats/… 200`, `POST …/share 200` (×2), `GET 200`. The share only minted a token on the stale row.
|
||||
3. WHY the 422: the turn's done frame carried **22 sources** (counted from the per-turn log line's `sources=[…]`), but `ChatMessage.sources` is capped at **20** (`app/schemas.py` L1460 — phase 83 A1 "top-N docs + agent reads"). The cited tier is the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587/L1289 — retrieval top-N ∪ the agent's read docs, NOT bounded by `top_n_docs`), and production runs `BOR_AGENT_MAX_ROUNDS=50` (the app container's env; the config default is 10) — so a long research turn cites 22+ docs routinely and the done-time auto-save 422s. The same turn had **33 tool calls** (`tools` cap 50 — at the production round cap with ZERO headroom) and `segments` entries = 33 tool entries + one thinking entry per contiguous thinking run (`buildSegments`, `app.js` L1775) under `xhigh` thinking over 34 rounds — the `segments` cap of 50 (phase 158, docstring sized "the round cap is 10 → ≤ 20 entries") is exceeded by the same class of turn. `suggestions` (cap 50) and `related` (cap 20) are likewise sized against settings (`suggested_docs`, `related_max_docs`) whose validators are unbounded above.
|
||||
4. WHY the 422 lost the answer: the phase-55 A2 quiet contract swallows a failed auto-save — a one-line `#send-status` note ("Couldn't save automatically — will try on the next message."), no error banner — and NOTHING retries: the retry is the NEXT save point (next user send / next brain-done / the pagehide partial), which never came (this was the last turn). The row stayed at its pre-turn state.
|
||||
5. WHY the share froze the loss: `shareCurrentChat`'s LINKED branch (`app.js` L3345) only calls `POST /api/chats/<id>/share` (token mint, `app/api/chats.py` `share_chat`) — it never re-saves the current conversation, so the shared snapshot is whatever the row last held.
|
||||
6. SECOND loss path (same symptom class): `persistConversation`'s `persisting` double-fire guard (`app.js` L3216) DROPS a save point that arrives while an upsert is in flight ("the next save point retries" — which, per 4., may never come). A done-time save dropped by a still-in-flight send-time save (a 400 s turn on a slow line) leaves the same user-only row.
|
||||
|
||||
## Objective
|
||||
A long agent turn (production `BOR_AGENT_MAX_ROUNDS=50`) saves cleanly at every save point: the saved-chat schema caps move from the round-cap-10 era to the real configuration (A1); a Share never publishes a row the current conversation has outgrown — it saves first (A2); a dropped or failed save self-heals (overlap re-run, one transient retry) and a failed save STAYS visible until the next success (A3/A4).
|
||||
|
||||
## Context
|
||||
- `app/schemas.py` — `ChatMessage` (L1381; `extra="forbid"`, L1458): `text`/`thinking` `max_length=600_000` (phase 164), `sources` `max_length=20` (L1459), `related` `max_length=20`, `suggestions` `max_length=50` (items `_Chip` ≤ 200), `tools` `max_length=50`, `segments` `max_length=50` (L1479), `error` ≤ 500, `image` ≤ 500, `answer_image` `SourceRef`, `answer_text` `max_length=50`, `budget_exhausted`; `SourceRef` (L1260-ish: `source` ≤ 120 / `path` ≤ 1000 / `title` ≤ 500 / `image_url`), `ToolCall` (L1209: `name` ≤ 100, `argument` ≤ 2000, `chars_shown`/`chars_total`/`offset`, the phase-139 `range` + phase-146 `path` additive keys), `SavedChatCreate`/`SavedChatUpdate` (`messages` `min_length=1, max_items=500` — phase 164; the `_require_within_total_budget` validator), `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` (L1611). The phase-83 docstrings on the list fields still say "top-N docs + agent reads — the UI shows a handful" — that assumption is RETIRED (see evidence 3).
|
||||
- `app/api/chat.py` — `merge_cited_docs` (L587), the done-event build (L1402–1447: `cited_refs` = the merged read docs, `related_refs` capped by `related_max_docs`), `app/rag/retriever.py` (the tiering), `app/config.py` (`agent_max_rounds` L279, validator L—: `>= 0`, NO upper bound; `suggested_docs` / `related_max_docs` validators likewise unbounded above; production env `BOR_AGENT_MAX_ROUNDS=50`).
|
||||
- `app/api/chats.py` — the public write surface (phase 55 A1), `share_chat` (token-only, L379-ish), the 422 is the framework's standard validation response (phase 83 A2: boundary-only, no route changes).
|
||||
- `frontend/assets/app.js` — `persistConversation` (L3218: the `persisting` guard L3216, the 404→recreate fallback, the A2 quiet contract on failure), `rememberBrainTurn` (L3484 — save point 2, rides L3496), the send save point (L1371), `shareCurrentChat` (L3345: linked → `POST /{id}/share` only; unlinked → `POST /api/chats {messages, share: true}`), `regenerateStaleChat` (L3419 — its own upsert copy), the pagehide partial (L4821, rides `rememberBrainTurn`), `buildSegments` (L1775), `sendStatus` (the `#send-status` live region).
|
||||
- `tests/unit/test_schemas.py` — the boundary-constant block (L47–75: `SOURCES_CAP = 20`, `SUGGESTIONS_CAP = 50`, `TOOLS_CAP = 50`, `TEXT_CAP = 600_000`, …) and the at-cap/one-over pins per field; `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862), the oversized-422 pins, the full brain-record pin; `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — text caps UNCHANGED, this suite is unaffected); `tests/unit/test_save_chat_ui.py` (the JS marker pins for the save/share contract — the A2 quiet-contract pin names the status-note text and the `persisting` guard); `tests/e2e/mock_llm.py` (the deterministic mock turn: single-read tool flow, `slow_llm.py` for timing).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
- **A1 — the caps move to the real configuration (task 01; `app/schemas.py` ONLY, boundary-only — the phase-83 A2 style: NO route change, the 422 stays the framework's standard response).** The five list caps change, every other bound (per-message `text`/`thinking` 600 000, `error` 500, `image` 500, the `SourceRef` field caps, the `ToolCall` field caps, `answer_text` 50, the `max_items=500` messages cap, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8 000 000`) is UNCHANGED:
|
||||
- `sources` 20 → **200** — the cited tier is top-N ∪ the agent's READ docs (`merge_cited_docs`), so its size is bounded by `agent_max_rounds` (production 50, validator-unbounded) × calls-per-round, not by `top_n_docs`; 200 = 4× the production round cap (each round typically carries one `read`);
|
||||
- `related` 20 → **200** — `related_max_docs` is owner-tunable with no upper bound; one `SourceRef`-family cap with `sources`;
|
||||
- `tools` 50 → **200** — one-or-more calls per round are legal (a round is one model request, not one call); 4× the production round cap;
|
||||
- `segments` 50 → **450** — ≤ `tools` (200) tool entries + ≤ ~50 thinking runs (one per round, the cap-50 worst case) + margin;
|
||||
- `suggestions` 50 → **100** — `suggested_docs` is owner-tunable with no upper bound; 20× the default (5); the item cap (200 chars) is unchanged.
|
||||
The SEC-05 intent survives: the worst-case list bytes are orders of magnitude inside the 8 M TOTAL budget (`200 SourceRefs ≈ 360 KB`, `200 ToolCalls ≈ 440 KB`, `450 segments ≈ 1 MB`), and the fixed boundary still bounds item COUNT for an operator who raises `BOR_AGENT_MAX_ROUNDS` far beyond 50. The `ChatMessage`/`SourceRef` docstrings are updated: the "top-N + a handful" sizing rationale is replaced by the `merge_cited_docs` / `agent_max_rounds` reality (evidence 3), the phase-83 history KEPT (the 32 000 mirror is phase 164's 600 000), and a note that the 8 M total budget is the byte bound. The unit constants (`SOURCES_CAP`, `TOOLS_CAP`, `SUGGESTIONS_CAP`) and the integration pins move with the values.
|
||||
- **A2 — Share saves first (task 02; `shareCurrentChat`, `app.js` L3345).** The LINKED branch upserts the CURRENT conversation BEFORE minting the token — the SAME upsert semantics as `persistConversation` (PUT; a 404 unlinks and retries as a create), then `POST /api/chats/<id>/share`. An upsert FAILURE (non-ok or network) ABORTS the share with the existing neutral error banner ("Couldn't share the conversation — try again.") — NO token, NO toast, `shareBtn` released in the `finally` (the never-stale contract, PLAN §7.4). The UNLINKED branch is unchanged (its `POST {messages, share: true}` already saves the current conversation in one commit). Docstring: sharing a conversation that the row has not yet caught up would publish a truncated snapshot (the incident) — the upsert-first ordering makes the shared page the current conversation.
|
||||
- **A3 — a dropped save re-runs (task 03; `persistConversation`, `app.js` L3216–3260).** `persisting` stays the one-upsert-at-a-time guard, but a save point that arrives while an upsert is in flight sets a module-level `savePending = true` instead of vanishing; the in-flight upsert's `finally` (after `persisting = false`) re-runs `persistConversation()` once when `savePending` is set (the flag is cleared first; the re-run is a fresh, serialized call — never concurrent with the upsert it trails). The re-run carries the LATEST `conversation` (the array is read at call time), so the dropped done-time save lands with the brain record. No new save points are added — the existing ones (send, `rememberBrainTurn`/pagehide, regenerate) are unchanged.
|
||||
- **A4 — a failed save stays visible and retries once (task 03).** On failure (non-ok or network), the existing one-line note is kept AND a module flag `saveFailedSticky = true` set — the note persists until the NEXT SUCCESSFUL save (a success clears the flag and resets `#send-status` to empty when the flag was set; other flows may overwrite the note in the meantime — best-effort visibility, the flag + the next save point are the guarantee). A TRANSIENT failure (network error or 5xx status) additionally schedules ONE retry after 5 000 ms (a `setTimeout` + a one-shot `retryScheduled` guard so overlapping failures never stack retries; the retry is a plain `persistConversation()` — the LATEST conversation, the same guards). A 4xx NEVER retries (deterministic — the cap fix in task 01 is the remedy; the retry would 422 again). The A2 contract's shape is unchanged: still a one-line `#send-status` note, still NO error banner, the turn never blocks — only the note's lifetime and the one transient retry are added. `regenerateStaleChat`'s own upsert copy is LEFT AS IS (its failure path already errors visibly through the turn's banner state, and A1 removes the 422 class it could hit).
|
||||
|
||||
## Tasks
|
||||
1. `01_schema_cap_realign.md` — the five list-cap moves + docstring rationale (A1), the unit boundary pins at the NEW values, the incident-payload integration pin (22 sources / 51 segments / 201 tools now 201/200), the oversized-422 pins at the new boundaries.
|
||||
2. `02_share_saves_first.md` — `shareCurrentChat`'s linked branch upserts before minting the token, aborts the share on an upsert failure; the JS marker pins.
|
||||
3. `03_persist_rerun_sticky.md` — the `savePending` overlap re-run, the sticky failure note, the one transient retry; the JS marker pins.
|
||||
4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the incident regression at the wire + the overlap / share-saves-first behavior through the real UI), the full gates, the docs pass, the atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
- **Unit:** `tests/unit/test_schemas.py` — the NEW boundary pairs per field (200/201 `sources`, 200/201 `related`, 200/201 `tools`, 100/101 `suggestions`, 450/451 `segments`), the realistic-payload round-trip pin EXTENDED (a 22-source / 34-tool / 40-segment brain record validates and round-trips byte-identically), every pre-phase pin green. `tests/unit/test_save_chat_ui.py` — the A2/A3/A4 marker pins (the upsert-before-share ordering in `shareCurrentChat`'s linked branch, the abort-on-failure path, `savePending` + the `finally` re-run, the sticky flag + the 5 s one-shot retry, the 4xx-no-retry distinction; the existing A2 quiet-contract pins stay green — the note text and the "no error banner" shape are unchanged).
|
||||
- **Integration:** `tests/integration/test_chats_api.py` — anonymous `POST`/`PUT` with the INCIDENT payload shape (22 `sources`, 33 `tools`, 51 `segments` incl. thinking runs) → **201/200** (pre-fix: 422); the over-cap pins move to the new boundaries (201 `sources` → 422, no row; 201 `tools` → 422; 451 `segments` → 422; 101 `suggestions` → 422); the full brain-record pin + the stale/share pins stay green.
|
||||
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_chat_save_long_turns.py`, run **in isolation** — (a) the incident regression at the wire through `page.request` (22-source brain record → 201, pre-fix 422); (b) a real UI mock turn's done-time auto-save lands (row: 2 messages); (c) the overlap re-run (the send-time save's PUT delayed 3 s via a Playwright route, the mock turn settles in flight → the dropped done-time save re-runs → final row 2 messages); (d) share-saves-first (with the auto-saves still delayed, the Share click → the shared page renders the BRAIN message, not just the question).
|
||||
- **Regression:** `tests/e2e/test_chat_save_payload_limits.py`, `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py` (the phase-51/55 share contract) green unedited.
|
||||
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the changed file is `app/schemas.py` (declarative — exercised by every unit/integration pin); the frontend change is marker-pinned per the house JS convention.
|
||||
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Documentation & Consistency
|
||||
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
|
||||
- `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap), Share saves the current conversation before minting the token, a failed auto-save stays visible until the next success and retries once on a transient failure (task 04). `AGENTS.md`: nothing to change unless the final pass finds the phase-55 save contract described elsewhere (the save points are not enumerated in AGENTS.md — the UI-Structure rule's "in-flight turn" clause is untouched; verify, do not add).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `ChatMessage` list caps: `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; every other bound unchanged; the docstring rationale updated (A1).
|
||||
- [ ] The incident payload (22 sources / 33 tools / 51 segments) saves 201/200 anonymously; over-cap variants 422 with no row stored (unit + integration pins).
|
||||
- [ ] A linked Share upserts the current conversation before minting the token and aborts (banner, no token) on an upsert failure; the unlinked branch is byte-identical (A2, marker pins).
|
||||
- [ ] A save point dropped by an in-flight upsert re-runs once after it settles (A3, marker pin); a failed save's note persists until the next success, one 5 s retry on network/5xx only (A4, marker pins).
|
||||
- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** (the four scenarios); `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green unedited.
|
||||
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `git diff --stat` limited to `app/schemas.py`, `frontend/assets/app.js`, the test files, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency).
|
||||
- [ ] One atomic `--no-gpg-sign` commit (e.g. `fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run`); phase dir moved to `.agents/phases/complete/`.
|
||||
|
||||
## Owner action (data repair — NOT code, recorded here so the phase's commit message can cite it)
|
||||
The two affected shared rows (`8b345ab5…`, `be92e464…`) still hold the user-only snapshot. Their brain answers were never persisted (the SSE bytes are not stored server-side). If the owner's browser still carries the conversations (`bor.chat.v1` localStorage restore), opening the chat page (which re-links to the row) and letting the FIXED client's next save point (or a Share — now save-first) fire restores the full content through the existing upsert; otherwise the answers are unrecoverable and the rows may be deleted from History.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 01 — The saved-chat list caps move to the real agent configuration (A1)
|
||||
|
||||
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Raise the five `ChatMessage` list caps from the round-cap-10 era to the production `BOR_AGENT_MAX_ROUNDS=50` reality (the incident turn's 22-source done frame 422'd at the old 20 cap), update the sizing rationale in the docstrings, and move the boundary pins — so a long research turn's done-time auto-save lands instead of 422ing into the phase-55 quiet-failure, while the SEC-05 boundedness (fixed item-count boundary + the 8 M total budget) survives.
|
||||
|
||||
## Work
|
||||
1. `app/schemas.py` — `ChatMessage` (L1381; the fields at L1458–1481):
|
||||
- `sources: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20)
|
||||
- `related: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20)
|
||||
- `suggestions: list[_Chip] | None = Field(default=None, max_length=100)` (was 50; the `_Chip` 200-char item cap is unchanged)
|
||||
- `tools: list[ToolCall] | None = Field(default=None, max_length=200)` (was 50)
|
||||
- `segments: list[ChatSegment] | None = Field(default=None, max_length=450)` (was 50; the comment's "max_length=50 (the phase-83 voice): far past any realistic turn — the round cap is 10 → ≤ 20 entries; 50 matches the tools cap" is REPLACED with the phase-189 sizing: ≤ 200 tool entries (4× the production round cap of 50 — a round is one model request, one-or-more calls per round are legal) + ≤ ~50 thinking runs (one per round) + margin)
|
||||
- **UNCHANGED, pinned as-is:** `text`/`thinking` 600 000, `error` 500, `image` 500, `answer_text` 50, the `SourceRef` field caps (120/1000/500 + `image_url`), the `ToolCall` field caps (name 100 / argument 2000 / the additive keys), `SavedChatCreate`/`SavedChatUpdate` `max_items=500`, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` and `_require_within_total_budget` (the verbatim-block accounting untouched).
|
||||
- The `ChatMessage` class docstring: the phase-83 paragraph's "(sources ≤ 20 — top-N docs + agent reads; related ≤ 20 — the related-doc tier, same ref shape; suggestions ≤ 50 chips of ≤ 200 chars; tools ≤ 50 — one entry per tool call, the round cap is 10)" is UPDATED — the cited tier has been the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587): retrieval top-N ∪ the agent's read docs, bounded by `agent_max_rounds` (production 50 via `BOR_AGENT_MAX_ROUNDS`, the validator L— unbounded above), so the caps are `sources`/`related`/`tools` 200 (4× the production round cap) and `suggestions` 100 (20× the default 5; `suggested_docs` is tunable, unbounded above), `segments` 450; the phase-83 history (the 32 000 mirror raised to 600 000 with phase 164) is KEPT; one sentence added: the fixed boundary bounds item COUNT (an operator raising `BOR_AGENT_MAX_ROUNDS` far past 50 still hits a boundary) while `SAVED_CHAT_TOTAL_CHARS_BUDGET` is the byte bound (worst-case list bytes ≈ 360 KB / 440 KB / 1 MB — orders of magnitude inside 8 M).
|
||||
2. `tests/unit/test_schemas.py` — the boundary-constant block (L47–75): `SOURCES_CAP = 200`, `TOOLS_CAP = 200`, `SUGGESTIONS_CAP = 100` (a `RELATED_CAP = 200` and `SEGMENTS_CAP = 450` constant added in the same style); the existing at-cap/one-over pins (L145–216 and the `segments` pins at L922–1009) now bind the NEW values with no edits (they read the constants); the realistic-payload round-trip pin (around L563) is EXTENDED with the incident shape: a brain record carrying 22 `sources`, 33 `tools`, and 40 `segments` (33 tool entries + 7 thinking entries) validates and `model_dump()`s byte-identically (the `_drop_image_when_absent` omission rules untouched — a record with all the keys present keeps them).
|
||||
3. `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862) is RENAMED/retargeted to the new boundary (`test_post_rejects_sources_over_200_and_stores_nothing`: 201 `SourceRef` items → 422, no row); NEW pins: (a) `test_post_accepts_the_incident_payload_shape` — an anonymous `POST /api/chats` with a 2-message conversation (user + a brain record carrying 22 `sources`, 33 `tools`, 51 `segments` = 33 tool entries + 18 thinking entries — the incident turn's shape) → **201**, the row's `messages` round-trips all three lists intact; (b) the same shape through `PUT /api/chats/<id>` → **200**, the row's message count bumps to the new array; (c) `test_put_rejects_over_cap_lists_and_leaves_row_unchanged` — 201 `tools` / 451 `segments` / 101 `suggestions` each → **422** and the row's stored messages are byte-identical (the 422 is the framework's standard body — the phase-83 A2 shape, no custom error copy). Every pre-existing pin (the full brain-record pin, the stale/share pins, the 500-item messages cap, the total-budget pins) stays green unedited.
|
||||
4. Verify: `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the new boundary pairs (200/201 `sources`, 200/201 `related`, 100/101 `suggestions`, 200/201 `tools`, 450/451 `segments`) pinned at BOTH boundaries; the extended realistic round-trip pin.
|
||||
- Integration: the incident payload shape 201/200s anonymously (pre-fix: 422); the over-cap 422 pins at the new boundaries with no row stored / row unchanged; the pre-existing pins green unedited.
|
||||
- Coverage: **>90%** on `app/` (the changed file is `app/schemas.py` — declarative, exercised by every pin).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `ChatMessage` list caps are 200/200/100/200/450 (sources/related/suggestions/tools/segments); every other bound unchanged; the docstring rationale is the `merge_cited_docs`/`agent_max_rounds` one with the 8 M-budget sentence.
|
||||
- [ ] `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v` green — the incident shape 201/200s, the over-cap variants 422 with no row stored, all pre-existing pins green unedited.
|
||||
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
|
||||
- [ ] no behavior change in completed work (the SSE `done` event is untouched — `ChatDoneEvent.sources` has no list cap; the `ChatMessage` caps bind the saved-chat boundary only)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 02 — Share saves first: a linked Share never publishes a row the conversation has outgrown (A2)
|
||||
|
||||
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Make `shareCurrentChat`'s linked branch upsert the CURRENT conversation (the same upsert semantics as `persistConversation`) BEFORE minting the share token — so a Share can never freeze a stale row (the incident: the row held only the question, the Share minted a token on it, and the shared page showed no answer) — and abort the share with the existing neutral error banner when the upsert fails. The unlinked branch (save-AND-share in one `POST`) is byte-identical.
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — `shareCurrentChat` (L3345), the linked branch (the `if (currentChatId)` arm, ~L3353–3363) becomes, in order:
|
||||
1. the upsert: `const body = JSON.stringify({ messages: conversation }); const headers = { "Content-Type": "application/json" };` → `res = await fetch(`/api/chats/${currentChatId}`, { method: "PUT", headers, body });` with the SAME 404→recreate fallback as `persistConversation` (L3227–3234): on a 404, `currentChatId = null; res = await fetch("/api/chats", { method: "POST", headers, body });` and, on the 201, re-link (`currentChatId = String((await res.json()).id)`);
|
||||
2. an upsert FAILURE (non-ok `res` after the fallback, or a network `throw`) → `showErrorBanner("Couldn't share the conversation — try again.")` and `return` (NO token minted, NO toast — the existing failure UI, the existing `finally` still releases `shareBtn.disabled`);
|
||||
3. only then the existing `POST /api/chats/<id>/share` mint (unchanged) and the existing success paths (clipboard copy → the inline-field fallback, the `#send-status` line, the task-04 toast on BOTH success branches) — byte-identical.
|
||||
The function docstring gains the phase-189 paragraph: the linked branch upserts first because a Share that minted a token on a row the conversation had outgrown published a truncated snapshot (the incident — the done-time auto-save had 422'd into the quiet-failure and the row held only the question); an upsert failure aborts the share (the owner must never be handed a link to content that is not the current conversation). The unlinked branch's docstring ("one action saves AND shares") is unchanged.
|
||||
2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention, the module's existing style — no browser):
|
||||
- `test_share_linked_branch_upserts_before_the_share_mint`: the linked branch of `shareCurrentChat` contains the PUT to `` `/api/chats/${currentChatId}` `` (the `persistConversation`-shaped upsert) positioned BEFORE the `` `/api/chats/${currentChatId}/share` `` fetch in the function body (a string-ordering pin: the PUT line's index < the share-mint line's index), and carries the 404→recreate fallback markers (the `status === 404` arm + the `POST` to `"/api/chats"` with the same `body`/`headers`);
|
||||
- `test_share_aborts_on_upsert_failure`: the failure arm between the upsert and the mint sets the EXACT neutral banner text `"Couldn't share the conversation — try again."` and `return`s (no `showToast` call between the upsert's failure check and the banner — the toast is success-only, the existing pin stays green);
|
||||
- `test_share_unlinked_branch_unchanged`: the unlinked branch still POSTs `"/api/chats"` with `share: true` in the body and re-links on the 201 (the existing pins, green unedited);
|
||||
- the existing phase-55 pins (the always-visible `#share-chat-btn` markup, the neutral error copy, the `showToast` success-only contract, the double-click guard released in the `finally`) stay green unedited.
|
||||
3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (JS markers): the upsert-before-mint ordering, the 404→recreate fallback in the share, the abort-on-failure path (banner, no token, no toast), the unlinked branch unchanged; every pre-existing pin green.
|
||||
- Coverage: the frontend change is marker-pinned per the house JS convention (no JS runtime in the unit gate); the behavior is E2E-pinned by task 04.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `shareCurrentChat`'s linked branch: PUT upsert (with the 404→recreate fallback) → failure abort (neutral banner, no token, no toast) → share mint → the existing success paths byte-identical; the docstring says why.
|
||||
- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin.
|
||||
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
|
||||
- [ ] no behavior change in completed work (the unlinked branch, the toast/banner contract, the `share_chat` endpoint — token-only, `app/api/chats.py` UNTOUCHED)
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 03 — Dropped saves re-run; a failed save stays visible and retries once (A3/A4)
|
||||
|
||||
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Close the two silent-loss paths in `persistConversation`: a save point dropped by the `persisting` double-fire guard (evidence 6 — a done-time save dropped by an in-flight send-time save vanishes until the next save point, which may never come) re-runs once after the in-flight upsert settles; a failed save's status note STAYS until the next success and a transient failure (network / 5xx) retries once after 5 s — while the phase-55 A2 contract's shape is unchanged (one-line `#send-status` note, NO error banner, the turn never blocks, silent success).
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/app.js` — `persistConversation` (L3216–3260) and its guard (L3216):
|
||||
- `let persisting = false;` stays (the one-upsert-at-a-time guard); add `let savePending = false;` (A3) and `let saveFailedSticky = false; let retryScheduled = false;` (A4) at the same module scope, each with a one-line comment naming the phase-189 decision.
|
||||
- **A3 — the overlap re-run:** the top guard becomes `if (persisting) { savePending = true; return; }` (a save point that arrives mid-upsert is PENDING, not dropped). At the top of the body, after the guard: `savePending = false;` (a fresh run consumes the flag — the pending re-run is this run when it was set by a re-run). In the `finally` (after `persisting = false;`): `if (savePending) { savePending = false; void persistConversation(); }` — the re-run is a fresh serialized call (it starts only after the in-flight upsert fully settled; never concurrent with it). The re-run reads `conversation` at call time, so the dropped done-time save lands with the brain record. No new save points, no other function touched.
|
||||
- **A4 — sticky note + one transient retry:** in the failure paths (the `!res.ok` arm and the `catch` arm), after setting the existing note text, set `saveFailedSticky = true;` and — transient only — `if (!retryScheduled && (res.status >= 500 || <network-failure-arm>)) { retryScheduled = true; setTimeout(() => { retryScheduled = false; void persistConversation(); }, 5000); }` (the `catch` arm IS the network-failure arm; the 4xx — every other non-ok — NEVER schedules: a 422 is deterministic, the phase-189 task-01 cap fix is the remedy, and a retry would 422 again; the one-shot `retryScheduled` guard keeps overlapping failures from stacking retries). In the success path (the `staleBanner` clearing block), add: `if (saveFailedSticky) { saveFailedSticky = false; sendStatus.textContent = ""; }` (a success clears the stuck note — the note persists until the next success; other flows may overwrite the note in the meantime, best-effort visibility, the flag + the next save point are the guarantee).
|
||||
- The `persistConversation` docstring: the "double-fire guard" paragraph is updated — a dropped save point is now PENDING and re-runs once after the in-flight upsert settles (A3); the A2 quiet-contract paragraph gains the two phase-189 lines (the note persists until the next success; one 5 s retry on network/5xx, never on 4xx — the note text and the no-banner shape are UNCHANGED).
|
||||
- `regenerateStaleChat` (L4439–4460, its own upsert copy) is LEFT AS IS (its failure surfaces through the turn's banner state; A1 removes the 422 class it could hit).
|
||||
2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention):
|
||||
- `test_persisting_overlap_sets_pending_and_reruns`: `persistConversation` contains the `savePending = true` assignment inside the `if (persisting)` arm, the `savePending = false` reset at the top of the body, and the `finally` block re-runs `persistConversation()` guarded by `savePending` (the ordering: `persisting = false` before the `savePending` check in the `finally`);
|
||||
- `test_failed_save_note_is_sticky_until_success`: the failure arms set `saveFailedSticky = true`, and the success path clears the flag + resets `sendStatus` (the pin: the success block contains the `saveFailedSticky` guard);
|
||||
- `test_transient_failure_retries_once_after_5s`: the retry scheduling is present with the 5000 ms delay, the `>= 500` status condition, the `catch`-arm scheduling, and the one-shot `retryScheduled` guard (set before the `setTimeout`, reset inside it);
|
||||
- `test_4xx_never_retries`: the retry condition is `res.status >= 500` (a 4xx non-ok schedules nothing — the pin: no other `setTimeout` scheduling in the failure arms);
|
||||
- the existing phase-55 A2 pins (the one-line note text `"Couldn't save automatically — will try on the next message."`, no error-banner call in `persistConversation`, the silent-success shape, the `persisting` guard) stay green unedited.
|
||||
3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit (JS markers): the pending re-run (set/reset/finally-re-run ordering), the sticky-note lifetime, the one-shot 5 s transient retry, the 4xx exclusion; every pre-existing pin green.
|
||||
- Coverage: the frontend change is marker-pinned per the house JS convention; the behavior is E2E-pinned by task 04 (the overlap scenario).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `persistConversation`: an overlapping save point sets `savePending`; the `finally` re-runs once after `persisting = false`; a fresh run consumes the flag.
|
||||
- [ ] A failed save's note persists until the next success; a network/5xx failure schedules ONE 5 s retry (one-shot guard); a 4xx never retries; the A2 contract's shape (note text, no banner, silent success) is unchanged.
|
||||
- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin.
|
||||
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
|
||||
- [ ] no behavior change in completed work (`regenerateStaleChat` untouched; no new save points; `app/api/chats.py` untouched)
|
||||
@@ -0,0 +1,57 @@
|
||||
# Task 04 — The E2E suite, the full gates, the docs pass, the commit
|
||||
|
||||
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
Pin the owner-visible contract of the whole phase through the real UI: the incident regression at the wire (a 22-source brain record saves instead of 422ing), a mock turn's done-time auto-save landing, the overlap re-run (a dropped save lands after the in-flight upsert settles), and share-saves-first (a Share with the auto-saves still in flight publishes the BRAIN message, not just the question) — then the full gate, the docs pass, and the atomic commit.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_chat_save_long_turns.py` — the phase's dedicated Playwright suite (the house `conftest` app + `mock_llm.py` pattern, run **in isolation** per AGENTS rule 4; a fresh saved-chat row per test — the house `saved_chats` cleanup helper pattern from `test_chat_history.py`):
|
||||
- `test_incident_payload_shape_saves_at_the_wire`: through `page.request` (the `test_chat_save_payload_limits.py` pattern — the app's own origin), an anonymous `POST /api/chats` with the INCIDENT shape (a user message + a brain record carrying 22 `sources` (`SourceRef` dicts at realistic column-length values), 33 `tools` (`ToolCall` dicts), 51 `segments` = 33 `{kind: "tool", …}` + 18 `{kind: "thinking", text: …}`) → **201**; a follow-up `PUT` replacing the brain record with the same shape → **200** and the row's `messages` round-trips all three lists (a `GET /api/chats/<id>` admin assertion). (Pre-fix: both 422 — the incident.)
|
||||
- `test_done_time_auto_save_lands_for_a_mock_turn`: the real UI — send a mock question (the `mock_llm.py` no-tools flow, the `test_chat_history.py` send pattern), wait for the done bubble (the house `wait_for_answer` helper), then `GET /api/chats/<id>` (the row id from the created-response `#…` — the house pattern reads `currentChatId` via `page.evaluate`): the row has **2** messages and the brain record's `text` equals the mock answer.
|
||||
- `test_dropped_done_time_save_reruns_after_the_inflight_upsert_settles`: `page.route`-delay the FIRST PUT to `/api/chats/**` (the send-time auto-save) by 3 000 ms (a `page.route("**/api/chats/**", handler)` that resolves on `route.fulfill` after a `wait_for_timeout` ONLY for the first matched PUT — a module-level counter; later PUTs pass through untouched), then send the mock question and let the mock turn settle (the done bubble appears in flight — the mock is fast, the upsert is still parked at 3 s). Wait to ~5 s; `GET /api/chats/<id>`: the row has **2** messages (the dropped done-time save re-ran after the parked upsert settled — pre-fix: the row held 1).
|
||||
- `test_share_saves_first_publishes_the_brain_message`: the same 3 s delay on the first PUT; send the mock question; while the send-time save is still parked (the done bubble has appeared), click `#share-chat-btn` (the house click helper, wait for the toast). The share's own upsert PUT passes the route filter UNdelayed (the counter already consumed by the parked send-time save), mints the token, and the toast lands; open `share_url` (the `test_share_chat.py` pattern — the `/shared/<token>` page, the `#shared-title` visible) and assert the shared page renders the BRAIN answer text (a `.msg .bubble` containing the mock answer string) — pre-fix: the shared page showed only the question. Assert the row (via admin `GET`) has 2 messages.
|
||||
- Module docstring: the incident's evidence summary (the 422 PUT log line, the 22-source done frame, `created_at == updated_at` single-write row) so a future reader sees WHY these four tests exist.
|
||||
2. The regression suites (green UNEDITED — do not touch them): `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — the text caps are unchanged), `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py`.
|
||||
3. The full gate, in order: `uv run pytest -v` (unit + integration 100% green); `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/`); `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` (**in isolation**); `uv run ruff check . && uv run pyright` clean.
|
||||
4. The docs pass (the phase's Documentation & Consistency mandate): review `README.md` and `AGENTS.md` in their entirety against the post-phase app; fix every inconsistency found, pre-existing ones included. `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap — 200/100/450), a Share saves the current conversation before minting the token (a failed share upsert aborts — no link to a truncated snapshot), a failed auto-save stays visible until the next success and retries once after 5 s on a transient failure (a 4xx never retries). `AGENTS.md`: verify the phase-55 save contract is not described anywhere in it (the save points are not enumerated — the UI-Structure rule's "in-flight turn" clause is untouched); if the final pass finds any stale save-contract sentence, fix it in place (no new rule added).
|
||||
5. The atomic commit (AGENTS rule 8; the git-commit skill's `--no-gpg-sign` + `LLM-Generated: true` trailer): `git add -A && git commit --no-gpg-sign -m "fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run
|
||||
|
||||
The done-time auto-save of a long turn (production BOR_AGENT_MAX_ROUNDS=50;
|
||||
the incident turn cited 22 docs, 33 tool calls, 34 rounds) 422'd at the
|
||||
phase-83 list caps sized for the round-cap-10 era (sources ≤ 20, tools ≤ 50,
|
||||
segments ≤ 50), and the phase-55 quiet-failure swallowed it: nothing retried,
|
||||
so the row kept its pre-turn state — and a Share minted a token on that stale
|
||||
row, publishing a conversation without its answer (incident 2026-10-16,
|
||||
shared 0101d2ca… / row 8b345ab5…).
|
||||
|
||||
- A1: ChatMessage list caps follow the real configuration — sources/related/
|
||||
tools 200 (4× the production round cap; the cited tier is the read docs,
|
||||
merge_cited_docs), suggestions 100, segments 450; every other bound and the
|
||||
8 M total budget unchanged (SEC-05 boundedness survives).
|
||||
- A2: a linked Share upserts the current conversation before minting the
|
||||
token and aborts on an upsert failure — never a link to a truncated
|
||||
snapshot.
|
||||
- A3: a save point dropped by the in-flight guard re-runs once after the
|
||||
upsert settles.
|
||||
- A4: a failed save's note persists until the next success; one 5 s retry on
|
||||
network/5xx (a 4xx never retries).
|
||||
|
||||
LLM-Generated: true"`.
|
||||
6. Move the phase directory: `mv .agents/phases/todo/189_chat_save_long_turns .agents/phases/complete/` and `git add .agents && git commit --no-gpg-sign -m "chore(agents): complete phase 189 (chat save long turns)"` — OR fold the phase move into the atomic commit per the house convention (check the last 3 phase commits' shape and match it; the AGENTS rule 8 says one atomic commit per phase including the phase files — if the house convention is one commit, `git add` the phase dir into step 5's commit and skip step 6's commit, keeping the `mv`).
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the four scenarios above, in isolation.
|
||||
- Regression: the three listed suites green unedited.
|
||||
- Coverage / lint / types: the full gate (step 3).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** — the incident shape 201/200 at the wire, the mock turn's done-time save lands, the dropped save re-runs (final row 2 messages), the share with in-flight auto-saves publishes the brain message on the shared page.
|
||||
- [ ] `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green UNEDITED.
|
||||
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] README.md / AGENTS.md reviewed in full; the saved-chat ops note in place; no stale save-contract sentence left.
|
||||
- [ ] `git diff --stat` (before the commit) limited to `app/schemas.py`, `frontend/assets/app.js`, `tests/unit/test_schemas.py`, `tests/unit/test_save_chat_ui.py`, `tests/integration/test_chats_api.py`, `tests/e2e/test_chat_save_long_turns.py`, `README.md`, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency).
|
||||
- [ ] the atomic `--no-gpg-sign` commit exists (message above); the phase dir is in `.agents/phases/complete/`.
|
||||
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
|
||||
- [ ] no behavior change in completed phases (the SSE `done` event, the `share_chat` endpoint, the unlinked share branch, the phase-55 A2 contract's shape — all unchanged)
|
||||
Reference in New Issue
Block a user