phase: 187_sync_peak_memory_line
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 187 — final verification pass: all criteria met, repo 100% green.**

Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.

Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)

Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.

Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).

Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.

Next pending phase: `188_chunks_hnsw_index`.
This commit is contained in:
2026-10-16 15:22:51 -04:00
parent 644f7de696
commit c054b47df5
72 changed files with 3826 additions and 96 deletions
@@ -1,54 +0,0 @@
# Phase 187 — Make the sync's memory a logged, visible number
**Source:** owner incident 2026-10-15 — reconstructing "how much RAM did the sync actually use?" meant reading `systemd` unit-teardown lines from three boots (`brain-of-code-app.service: … 3.3G memory peak`), and boot −1 (the run that killed the box) had NO teardown record at all because the VM never shut down cleanly.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; lands after 183–186 so the numbers it reports describe the fixed sync.
**Context:** `.agents/PLAN.md` §9 Observability (the log-format authority — PLAN is never edited by an agent, so the new fields are documented in `README.md` + `AGENTS.md` rule 10 and the owner reconciles PLAN); `AGENTS.md` rule 10 ("Amply log, visibly feed back"); `app/core/response_budget.py` (the house precedent this phase mirrors: a pure-stdlib `app/core/` module, one greppable logger name — `grep ' :: api '`, `BOR_LOG_LARGE_RESPONSE_BYTES`, and `scripts/api_payload_audit.py` as the offline audit); `app/api/sync.py` — the `SyncStatus` dataclass (**L276**, phase-64 "keep the final counts in the terminal" convention), `sync_status()` (**L385**, its additive-keys contract "Additive only — every pre-existing key keeps its name, type, and meaning"), the `sync: started repos=%d origin=%s git=%d local=%d` line (**L591–594**), the phase transitions (`_status.phase = "import"` **L718**, `"overview"` **L743**, `"summaries"`), `sync: done detail=%s` (**L903**), the cancel terminal's `sync: cancelled by user state=cancelled files_done=%d files_total=%d` (**L933–936**), the failed terminal (`logger.exception("sync: failed")` **L938**); `app/rag/importer.py` — `ImportSummary.format_counts` (**L513**) + `ImportSummary.log()` (**L520**, the cron-safe greppable summary line PLAN §9 names); `frontend/assets/sources.js` — the sync progress box + result line (`setSyncProgress(…)` around **L1325**, the label/counts helper at **L1080–1100** and **L1378**, the aria-live announcements), `frontend/index.html` (the progress box markup), `tests/unit/test_sync_api*.py` / `tests/integration/test_sync_api.py` (the status-shape pins), `tests/e2e/test_sync_summary_visibility.py`, `tests/e2e/test_sync_progress_box.py`, `tests/e2e/test_sync_status_freshness.py`, `tests/e2e/test_sync_cancel.py`.
## Objective
Make "how much memory did that sync use?" answerable from one log line and visible on the page that started it: a small `app/core/memory.py` reading current + peak RSS, appended to every sync lifecycle line and the importer's summary line, plus an additive `rss_peak_mb` key on `GET /api/sync/status` and a `peak <N> MB` clause on the Sources page's sync result line.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `app/core/memory.py` never raises and never lies.** Three pure helpers: `rss_mb() -> int | None` (Linux: parse `VmRSS` from `/proc/self/status`; non-Linux/failure: `resource.getrusage(RUSAGE_SELF).ru_maxrss // 1024` as the best available proxy), `rss_peak_mb() -> int | None` (`ru_maxrss // 1024` — the process high-water mark), and `cgroup_peak_mb() -> int | None` (read `/sys/fs/cgroup/memory.peak` when present, else `None`). Docstring states the honest nuance the incident turned on: **inside a container the process high-water and the host's cgroup `memory.peak` are different numbers, and the cgroup number includes page cache** (the incident box's app cgroup showed 2.16 GB of `file` against 211 MB of `anon` while idle) — so the app logs the PROCESS number and the operator's dashboard may show a larger one; neither is wrong, and the log says which it is.
- **D2 — the format is append-only, in the §9 style (task 02).** `sync: started … rss_mb=N`, and every terminal (`sync: done detail=… rss_mb=N rss_peak_mb=M`, `sync: cancelled by user … rss_mb=N rss_peak_mb=M`, and the failed funnel) gains the same two trailing fields; `ImportSummary.log()` gains a trailing ` rss_mb=N`. A `None` renders as `rss_mb=?` — a helper `fmt_mb(v: int | None) -> str` keeps that in one place. No existing field is renamed, reordered or re-typed (the phase-167/182 append-only convention), so every existing `grep` keeps working.
- **D3 — sampled, not polled per file (task 03).** `_status` gains `rss_mb: int | None` and `rss_peak_mb: int | None`, reset at run start; sampled at the run start, at each `_status.phase` transition (`import` L718, `overview` L743, `summaries`), and in EVERY terminal (which keep the final value next to the counts — the phase-64 convention). The per-file progress hook samples at most once per 5 s via a `time.monotonic()` guard, so a 27k-file walk costs ~600 `/proc` reads, not 27k, and the operator watching the page sees the number move (PLAN §7.4's never-stale contract).
- **D4 — the status key and the UI clause are additive and admin-only (task 03).** `sync_status()` returns two new keys (`rss_mb`, `rss_peak_mb`, `int | null`); `sources.js` appends ` · peak 512 MB` to the existing result/counts line — a clause on text that already exists, never a new control, never replacing a state string, and the aria-live announcement keeps its current wording (a screen-reader user gets the same announcement plus the clause; the peak is supplementary, so it must not become the announcement's only content). The Sources view is admin-only (PLAN B5), which is the exposure guard; no memory number appears on any anonymous surface (chat, shared chat, health).
- **D5 — an offline reader for the log (task 02, small).** Document `grep ' :: ' | grep ' rss_peak_mb='` in `README.md`'s operations section. NO new audit script: `scripts/api_payload_audit.py` exists for bytes; a memory equivalent would be a second parser for one number. Explicitly out of scope (A3).
- **NOT touched:** `PLAN.md` (never), the sync pipeline's sequence/errors/cancel semantics, the response-budget logger, the chat's per-turn line (this phase is about the sync), the container/quadlet memory limits (owner's own task), the retrieval path.
## Dependencies
— (none; after 183–186 in the roadmap so the reported numbers describe the bounded sync)
## Tasks
1. `01_memory_helper.md` — `app/core/memory.py` (the three helpers + `fmt_mb`), unit-tested with `/proc` and `resource` stubbed (Linux path, missing-file path, garbage-file path, non-Linux fallback).
2. `02_sync_log_lines.md` — the four sync lifecycle lines + `ImportSummary.log()` gain `rss_mb=` / `rss_peak_mb=`, the `fmt_mb` single-sourcing, the log-format assertions in the integration tests, the README operations note.
3. `03_status_key_and_ui.md` — the `SyncStatus` fields + sampling (start, phase transitions, terminals, the 5 s-throttled progress sample), the two additive `sync_status()` keys, the `sources.js` clause + aria-live behaviour, the shape pins.
4. `04_e2e_peak_memory.md` — `tests/e2e/test_sync_peak_memory_visible.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_core_memory.py` — every helper branch (real `/proc` on Linux + stubbed failure modes, the `None` rendering, the kB→MB arithmetic), `tests/unit/test_settings*` untouched.
- **Integration:** `tests/integration/test_sync_api.py` — the lifecycle lines carry the fields (caplog assertions on the `sync: started` / `sync: done` / `sync: cancelled` / `sync: failed` shapes with `rss_mb=?` accepted on a platform without `/proc`), the status payload's two new keys with their types and the terminal-keeps-final-value convention, every pre-existing key byte-identical.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_sync_peak_memory_visible.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `AGENTS.md` rule 10: the required sync lines now carry `rss_mb=` / `rss_peak_mb=` (extend rule 10's parenthetical, do not rewrite it). `README.md`: the operations grep, what each number means, and the container-vs-cgroup caveat (page cache included in the host's number) written in one short paragraph. PLAN §9 is the owner's to reconcile — flag it in the phase report, do not edit it.
## Completion Criteria
- [ ] `app/core/memory.py` exists with the four documented functions, every branch unit-tested, and never raises.
- [ ] All four sync lifecycle lines + `import: summary …` carry `rss_mb=` (and the terminals `rss_peak_mb=`), appended after the existing fields; no existing field changed.
- [ ] `GET /api/sync/status` exposes `rss_mb` / `rss_peak_mb` (`int | null`), keeps the final value in every terminal, and every pre-existing key is unchanged.
- [ ] The Sources page's sync result line shows the ` · peak <N> MB` clause with no new control and no regression in the existing state strings.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the PLAN §9 reconciliation flagged, not performed).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(observability): log and show the sync's RSS so memory is a number, not a forensic exercise`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — the app logs PROCESS RSS, not the cgroup number**, with the caveat documented (`anon` vs `file` differ by an order of magnitude and the cgroup figure includes page cache — demonstrated during the incident investigation).
- **A2 — append-only to the existing lines and keys.** No new log line type, no renamed field, no separate metrics endpoint (A12: no aux services), no Prometheus.
- **A3 — no memory audit script** this phase; the grep is documented instead.
- **A4 — sampling is throttled (5 s) and never per-file**, and the UI clause is text on a line that already exists on an admin-only view.
@@ -1,26 +0,0 @@
# Task 01 — `app/core/memory.py`
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
One pure-stdlib module that answers "how much RAM is this process using, and what was its high-water mark" without ever raising, mirroring `app/core/response_budget.py`'s shape (no FastAPI, no imports from `app.api`, logger only if needed).
## Work
1. `app/core/memory.py`:
- `rss_mb() -> int | None` — parse `VmRSS` from `/proc/self/status` (kB → MB, integer, floor); any `OSError`/`KeyError`/parse failure returns `None`. Non-Linux fallback: `resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024` (documented as a high-water proxy, not a current value).
- `rss_peak_mb() -> int | None` — `resource.getrusage(RUSAGE_SELF).ru_maxrss // 1024`, `None` only if `resource` is unavailable (wrap in a guarded import for platforms without it).
- `cgroup_peak_mb() -> int | None` — read `/sys/fs/cgroup/memory.peak` (bytes → MB) when it exists and parses, else `None`. Docstring: inside a container this file is usually absent (the host reads it), and the host's number counts PAGE CACHE too — the incident box's app cgroup showed ~2.16 GB `file` against ~211 MB `anon` while idle, and `systemd` reported `3.3G memory peak` for the unit while the uvicorn process was a fraction of that. This module therefore reports PROCESS numbers and this distinction is written here once, so no future reader conflates them.
- `fmt_mb(v: int | None) -> str` — `"?"` for `None`, else `str(v)` (so log lines never print `rss_mb=None`).
- Module docstring: the incident (one paragraph), the three helpers, the never-raise contract, and the container/cgroup caveat above.
2. `tests/unit/test_core_memory.py` — every branch, with the filesystem and `resource` stubbed: a real read (Linux runner: `/proc/self/status` has `VmRSS`, so `rss_mb()` is a positive int), a missing file, an unparseable file, a file without a `VmRSS` line, the non-Linux `resource` fallback path (patch the `/proc` open to raise + patch `getrusage`), `rss_peak_mb()`'s arithmetic, `cgroup_peak_mb()` present/absent/garbage, `fmt_mb(None) == "?"` / `fmt_mb(0) == "0"` (0 is a value, not missing).
3. Verify: `uv run pytest tests/unit/test_core_memory.py -v`; `uv run ruff check . && uv run pyright` clean; `uv run pytest` green.
## Testing & Quality
- Unit: 100% branch coverage of the new module (well above the >90% gate).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `app/core/memory.py` with the four functions, the never-raise contract, and the container/cgroup caveat documented.
- [ ] `uv run pytest tests/unit/test_core_memory.py -v` green, every branch covered.
- [ ] No import cycle (`app.core.memory` imports stdlib only — prove it in the module docstring's "pure stdlib" line and by the absence of `app.` imports).
@@ -1,28 +0,0 @@
# Task 02 — The lifecycle lines carry the numbers
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Append `rss_mb=` to the sync's start line, `rss_mb=`/`rss_peak_mb=` to all three terminals, and `rss_mb=` to the importer's cron-safe summary line — append-only, so every existing grep and every existing field keeps working.
## Work
1. `app/api/sync.py` — the four lifecycle log sites, using `app.core.memory.rss_mb/rss_peak_mb/fmt_mb` (imported at module level, matching the file's import style):
- **L591–594** `sync: started repos=%d origin=%s git=%d local=%d` → append ` rss_mb=%s` (`fmt_mb(rss_mb())`).
- **L903** `sync: done detail=%s` → append ` rss_mb=%s rss_peak_mb=%s`.
- **L933–936** `sync: cancelled by user state=cancelled files_done=%d files_total=%d` → append the same two fields.
- **L938** the failed funnel: `logger.exception("sync: failed")` stays as-is (the traceback), and the FAILED line gains a dedicated `logger.error("sync: failed state=failed files_done=%d files_total=%d rss_mb=%s rss_peak_mb=%s error=%s", …)` immediately after the `_status.error = _sanitize_error(str(e))` assignment (sanitized exactly like the status field — AGENTS/security: the raw exception string never reaches the log un-sanitized). Keep `logger.exception` so the traceback survives.
2. `app/rag/importer.py` — `ImportSummary.log()` (**L520**): append ` rss_mb=%s` to the existing `import: summary …` line (the `format_counts()` helper at L513 stays untouched — the memory field belongs to the LOG line, not the counts formatter, so the CLI's other consumers of `format_counts()` are byte-identical).
3. `tests/integration/test_sync_api.py` — caplog assertions (the house caplog pattern): after a run to `success`, the captured records include a `sync: started … rss_mb=` line and a `sync: done … rss_mb=… rss_peak_mb=…` line whose values are `?` or a positive integer; the same for the cancelled terminal (reuse the existing cancel test's fixture) and the failed terminal (reuse the existing failure test — `sync: model check failed` or a stubbed raise). Assert the PRE-existing prefixes are unchanged (`line.startswith("sync: started repos=")` etc.) — the append-only pin.
4. `tests/integration/test_importer_e2e.py` (or the importer family's log-pin home): the `import: summary …` line ends with ` rss_mb=<int|?>` and its earlier fields are unchanged.
5. `README.md` — the operations grep (`grep ' :: ' … | grep ' rss_peak_mb='`), the field meanings, and the process-vs-cgroup caveat in one short paragraph (D1/A1). `AGENTS.md` rule 10: extend the parenthetical to mention the sync's new fields (do not rewrite the rule).
6. Verify: `uv run pytest tests/integration/test_sync_api.py tests/integration/test_importer_e2e.py -v`; `uv run pytest`; ruff + pyright.
## Testing & Quality
- Integration: the four lifecycle terminals + the importer summary line pin the fields and the unchanged prefixes.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All four sync lifecycle lines and `import: summary …` carry the new fields, appended after the existing ones; the failed terminal gains the sanitized one-line summary while `logger.exception` still writes the traceback.
- [ ] `uv run pytest tests/integration/test_sync_api.py -v` green (fields present, pre-existing prefixes byte-identical).
- [ ] `README.md` + `AGENTS.md` rule 10 updated; `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -1,30 +0,0 @@
# Task 03 — The status key and the Sources-page clause
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Put the number where the operator is: two additive keys on `GET /api/sync/status` (sampled at the run start, each phase transition, every terminal, and at most every 5 s during the walk) and a ` · peak <N> MB` clause on the Sources page's existing sync result line.
## Work
1. `app/api/sync.py`:
- `SyncStatus` (**L276**): two new fields `rss_mb: int | None = None`, `rss_peak_mb: int | None = None`, documented as phase-187 additive state with the phase-64 "terminals keep the final value" convention and the run-start reset.
- A single private sampler on the module: `_sample_rss(force: bool = False) -> None` — reads `app.core.memory.rss_mb()` / `rss_peak_mb()`, stores both, keeps a `time.monotonic()` guard so a non-forced call is a no-op within 5 s (D3/A4); forced at the run start, each `_status.phase = "import"|"overview"|"summaries"` transition (**L718** and friends), and every terminal (success L903 path, cancelled L933 path, failed L938 path) so the final value is exact.
- `sync_status()` (**L385**): append `"rss_mb"` and `"rss_peak_mb"` to the dict (the file's own words: "Additive only — every pre-existing key keeps its name, type, and meaning") and document the two keys + the terminal convention in its docstring.
- The per-file `_hook` (L677) calls `_sample_rss()` (unforced → throttled).
2. `frontend/assets/sources.js`: on the terminal render path, append the clause to the EXISTING result/counts text — ` · peak 512 MB` — sourced from `status.rss_peak_mb ?? status.rss_mb`, and render NOTHING when both are null (never `peak ? MB`). Touch the result-line helper used at **L1080–1100** / **L1378** and the terminal/announce path around **L1314–1330**; the existing state strings ("Sync cancelled", the counts line, the error copy) keep their exact wording with the clause appended, and the aria-live announcement keeps announcing what it announces today plus the clause (no new announcement). No new DOM control, no layout shift on the 375px viewport (the clause wraps inside the existing line's `.cell-clip`-free text; verify visually in the E2E at mobile width).
3. Pins:
- `tests/integration/test_sync_api.py`: the payload carries both keys with `int | None`; after a run the terminal payload's `rss_peak_mb` is non-null on Linux and `>= rss_mb` when both are set; idle (never-run) reports `null`/`null`; every pre-existing key/value shape asserted unchanged (the additive pin); anonymous `GET /api/sync/status` still 403 (no new anonymous surface — A4).
- `tests/unit/test_sync_status_shape.py` (or this file's existing home for the status-shape matrix): the key set is exactly the old set ∪ {`rss_mb`, `rss_peak_mb`}.
- `frontend`: the clause appears in the terminal line and NOT in `running`; a null peak renders no clause.
4. Verify: `uv run pytest tests/integration/test_sync_api.py -v`; the three sync E2E neighbours in isolation (`test_sync_summary_visibility.py`, `test_sync_progress_box.py`, `test_sync_status_freshness.py`) still green (they assert the existing strings — the clause must not break them; if one asserts an exact full-line equality, update it to the clause-appended form and say so in the task note); `uv run pytest`; ruff + pyright.
## Testing & Quality
- Integration: the shape + additive + admin-gating pins above.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `SyncStatus` + `sync_status()` carry the two additive keys; sampling happens at start / phase transitions / every terminal and is throttled to 5 s in the per-file hook.
- [ ] The Sources page shows ` · peak <N> MB` on the terminal line, renders nothing when the value is null, and adds no new control.
- [ ] The three existing sync E2E families are green (any exact-string update documented).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -1,34 +0,0 @@
# Task 04 — E2E: the operator sees the peak (the phase's dedicated Playwright suite)
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the browser that the number is there and honest: a real sync from the Sources page ends with a `peak <N> MB` clause on the result line, the status payload's `rss_peak_mb` is a positive integer that survives the terminal, the running state shows no clause, and nothing anonymous can read it. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_sync_peak_memory_visible.py` — a NEW module, mock-only, the `tests/e2e/test_sync_progress_box.py` / `test_sync_summary_visibility.py` house shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`).
- **Test 1 `test_result_line_shows_the_peak`:** sync a small fixture → after the `success` terminal, the progress box's result line matches `/peak \d+ MB/`, the existing wording ("Sync complete"/the counts line) is still present, and the aria-live region contains the same clause.
- **Test 2 `test_status_payload_reports_rss`:** during `running` and after the terminal, `GET /api/sync/status` via `page.request` → `rss_peak_mb` is a positive int in the terminal, `>= rss_mb` when both are set, and pre-existing keys are unchanged (the additive pin end to end). During `running` the result line shows NO `peak` clause (the clause is terminal-only in the UI).
- **Test 3 `test_anonymous_cannot_read_memory`:** logged-out `GET /api/sync/status` → 403 (the A4 exposure guard, unchanged).
- **Test 4 `test_cancelled_terminal_keeps_the_peak`:** cancel a run (the `test_sync_cancel.py` shape) → the neutral "Sync cancelled" state still renders, with the peak clause present (terminals keep the value).
- **Mobile width:** assert the clause at 375 px does not overflow the progress box (the PLAN §7 container rule).
- Module docstring: the incident reference, the fields + the process-vs-cgroup caveat pointer to `README.md`, the test→observable mapping, the run-in-isolation command, prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` in full against the post-phase app, every inconsistency fixed (pre-existing included); confirm rule 10's extension and the operations grep landed in task 02; flag the PLAN §9 reconciliation in the phase report (never edit PLAN.md).
3. **The full gates:**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov`, `tests/e2e/test_sync_status_freshness.py -v --no-cov`, `tests/e2e/test_sync_cancel.py -v --no-cov` — green in isolation (the surfaces the clause touches).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `feat(observability): log and show the sync's RSS so memory is a number, not a forensic exercise` — covering `app/core/memory.py`, `app/api/sync.py`, `app/rag/importer.py`, `frontend/assets/sources.js`, the tests, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/187_sync_peak_memory_line/`.
## Testing & Quality
- E2E: the dedicated suite + the three sync UI families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` green **in isolation** (clause on both terminals, absent while running, payload types, anonymous 403, mobile width clean).
- [ ] The three regression E2E families green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project; the PLAN §9 reconciliation flagged for the owner.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,68 @@
# Phase 189 — Long agent turns save (and share) the whole conversation
**Source:** owner incident 2026-10-16 — the shared chat `0101d2ca-b24c-4d13-9f65-c0837e993c50` (row `8b345ab5-aab3-4412-9642-0f869b3cec4d`) shows ONLY the user's question although the LLM answered it completely; the shared row `be92e464-7918-4907-9186-09d8d98ff2c1` carries the same signature.
**Story:** n/a (incident-derived, owner directive 2026-10-16)
**Depends on:** none functionally (builds on the completed phase-55/83/158/164 saved-chat surface).
## Evidence (diagnosed 2026-10-16, production DB + app logs — read this, not the chat)
1. The row behind the shared token: `n_msgs = 1`, `who0 = user`, `created_at == updated_at` — the row was written EXACTLY ONCE (the send-time auto-save `POST /api/chats`) and never updated. 84 other single-write user-only rows exist (2 shared, incl. this one); the 2-message rows (279) were written >1 time — the pattern is specific to the *last* save point of a conversation.
2. The app log for this row shows the whole sequence: the turn's per-turn line (`thinking=xhigh thinking_chars=23937 tool_calls=33 rounds=34 cap_hit=0 total_ms=400173`, `POST /api/chat 200 stream=1` — the client consumed the FULL 400 s stream, so the done frame landed) followed by **`PUT /api/chats/8b345ab5… 422 Unprocessable Entity`** — the done-time auto-save REJECTED — then `GET /api/chats/… 200`, `POST …/share 200` (×2), `GET 200`. The share only minted a token on the stale row.
3. WHY the 422: the turn's done frame carried **22 sources** (counted from the per-turn log line's `sources=[…]`), but `ChatMessage.sources` is capped at **20** (`app/schemas.py` L1460 — phase 83 A1 "top-N docs + agent reads"). The cited tier is the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587/L1289 — retrieval top-N ∪ the agent's read docs, NOT bounded by `top_n_docs`), and production runs `BOR_AGENT_MAX_ROUNDS=50` (the app container's env; the config default is 10) — so a long research turn cites 22+ docs routinely and the done-time auto-save 422s. The same turn had **33 tool calls** (`tools` cap 50 — at the production round cap with ZERO headroom) and `segments` entries = 33 tool entries + one thinking entry per contiguous thinking run (`buildSegments`, `app.js` L1775) under `xhigh` thinking over 34 rounds — the `segments` cap of 50 (phase 158, docstring sized "the round cap is 10 → ≤ 20 entries") is exceeded by the same class of turn. `suggestions` (cap 50) and `related` (cap 20) are likewise sized against settings (`suggested_docs`, `related_max_docs`) whose validators are unbounded above.
4. WHY the 422 lost the answer: the phase-55 A2 quiet contract swallows a failed auto-save — a one-line `#send-status` note ("Couldn't save automatically — will try on the next message."), no error banner — and NOTHING retries: the retry is the NEXT save point (next user send / next brain-done / the pagehide partial), which never came (this was the last turn). The row stayed at its pre-turn state.
5. WHY the share froze the loss: `shareCurrentChat`'s LINKED branch (`app.js` L3345) only calls `POST /api/chats/<id>/share` (token mint, `app/api/chats.py` `share_chat`) — it never re-saves the current conversation, so the shared snapshot is whatever the row last held.
6. SECOND loss path (same symptom class): `persistConversation`'s `persisting` double-fire guard (`app.js` L3216) DROPS a save point that arrives while an upsert is in flight ("the next save point retries" — which, per 4., may never come). A done-time save dropped by a still-in-flight send-time save (a 400 s turn on a slow line) leaves the same user-only row.
## Objective
A long agent turn (production `BOR_AGENT_MAX_ROUNDS=50`) saves cleanly at every save point: the saved-chat schema caps move from the round-cap-10 era to the real configuration (A1); a Share never publishes a row the current conversation has outgrown — it saves first (A2); a dropped or failed save self-heals (overlap re-run, one transient retry) and a failed save STAYS visible until the next success (A3/A4).
## Context
- `app/schemas.py` — `ChatMessage` (L1381; `extra="forbid"`, L1458): `text`/`thinking` `max_length=600_000` (phase 164), `sources` `max_length=20` (L1459), `related` `max_length=20`, `suggestions` `max_length=50` (items `_Chip` ≤ 200), `tools` `max_length=50`, `segments` `max_length=50` (L1479), `error` ≤ 500, `image` ≤ 500, `answer_image` `SourceRef`, `answer_text` `max_length=50`, `budget_exhausted`; `SourceRef` (L1260-ish: `source` ≤ 120 / `path` ≤ 1000 / `title` ≤ 500 / `image_url`), `ToolCall` (L1209: `name` ≤ 100, `argument` ≤ 2000, `chars_shown`/`chars_total`/`offset`, the phase-139 `range` + phase-146 `path` additive keys), `SavedChatCreate`/`SavedChatUpdate` (`messages` `min_length=1, max_items=500` — phase 164; the `_require_within_total_budget` validator), `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` (L1611). The phase-83 docstrings on the list fields still say "top-N docs + agent reads — the UI shows a handful" — that assumption is RETIRED (see evidence 3).
- `app/api/chat.py` — `merge_cited_docs` (L587), the done-event build (L1402–1447: `cited_refs` = the merged read docs, `related_refs` capped by `related_max_docs`), `app/rag/retriever.py` (the tiering), `app/config.py` (`agent_max_rounds` L279, validator L—: `>= 0`, NO upper bound; `suggested_docs` / `related_max_docs` validators likewise unbounded above; production env `BOR_AGENT_MAX_ROUNDS=50`).
- `app/api/chats.py` — the public write surface (phase 55 A1), `share_chat` (token-only, L379-ish), the 422 is the framework's standard validation response (phase 83 A2: boundary-only, no route changes).
- `frontend/assets/app.js` — `persistConversation` (L3218: the `persisting` guard L3216, the 404→recreate fallback, the A2 quiet contract on failure), `rememberBrainTurn` (L3484 — save point 2, rides L3496), the send save point (L1371), `shareCurrentChat` (L3345: linked → `POST /{id}/share` only; unlinked → `POST /api/chats {messages, share: true}`), `regenerateStaleChat` (L3419 — its own upsert copy), the pagehide partial (L4821, rides `rememberBrainTurn`), `buildSegments` (L1775), `sendStatus` (the `#send-status` live region).
- `tests/unit/test_schemas.py` — the boundary-constant block (L47–75: `SOURCES_CAP = 20`, `SUGGESTIONS_CAP = 50`, `TOOLS_CAP = 50`, `TEXT_CAP = 600_000`, …) and the at-cap/one-over pins per field; `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862), the oversized-422 pins, the full brain-record pin; `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — text caps UNCHANGED, this suite is unaffected); `tests/unit/test_save_chat_ui.py` (the JS marker pins for the save/share contract — the A2 quiet-contract pin names the status-note text and the `persisting` guard); `tests/e2e/mock_llm.py` (the deterministic mock turn: single-read tool flow, `slow_llm.py` for timing).
## Design (shared by all tasks — the executor reads this, not the chat)
- **A1 — the caps move to the real configuration (task 01; `app/schemas.py` ONLY, boundary-only — the phase-83 A2 style: NO route change, the 422 stays the framework's standard response).** The five list caps change, every other bound (per-message `text`/`thinking` 600 000, `error` 500, `image` 500, the `SourceRef` field caps, the `ToolCall` field caps, `answer_text` 50, the `max_items=500` messages cap, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8 000 000`) is UNCHANGED:
- `sources` 20 → **200** — the cited tier is top-N ∪ the agent's READ docs (`merge_cited_docs`), so its size is bounded by `agent_max_rounds` (production 50, validator-unbounded) × calls-per-round, not by `top_n_docs`; 200 = 4× the production round cap (each round typically carries one `read`);
- `related` 20 → **200** — `related_max_docs` is owner-tunable with no upper bound; one `SourceRef`-family cap with `sources`;
- `tools` 50 → **200** — one-or-more calls per round are legal (a round is one model request, not one call); 4× the production round cap;
- `segments` 50 → **450** — ≤ `tools` (200) tool entries + ≤ ~50 thinking runs (one per round, the cap-50 worst case) + margin;
- `suggestions` 50 → **100** — `suggested_docs` is owner-tunable with no upper bound; 20× the default (5); the item cap (200 chars) is unchanged.
The SEC-05 intent survives: the worst-case list bytes are orders of magnitude inside the 8 M TOTAL budget (`200 SourceRefs ≈ 360 KB`, `200 ToolCalls ≈ 440 KB`, `450 segments ≈ 1 MB`), and the fixed boundary still bounds item COUNT for an operator who raises `BOR_AGENT_MAX_ROUNDS` far beyond 50. The `ChatMessage`/`SourceRef` docstrings are updated: the "top-N + a handful" sizing rationale is replaced by the `merge_cited_docs` / `agent_max_rounds` reality (evidence 3), the phase-83 history KEPT (the 32 000 mirror is phase 164's 600 000), and a note that the 8 M total budget is the byte bound. The unit constants (`SOURCES_CAP`, `TOOLS_CAP`, `SUGGESTIONS_CAP`) and the integration pins move with the values.
- **A2 — Share saves first (task 02; `shareCurrentChat`, `app.js` L3345).** The LINKED branch upserts the CURRENT conversation BEFORE minting the token — the SAME upsert semantics as `persistConversation` (PUT; a 404 unlinks and retries as a create), then `POST /api/chats/<id>/share`. An upsert FAILURE (non-ok or network) ABORTS the share with the existing neutral error banner ("Couldn't share the conversation — try again.") — NO token, NO toast, `shareBtn` released in the `finally` (the never-stale contract, PLAN §7.4). The UNLINKED branch is unchanged (its `POST {messages, share: true}` already saves the current conversation in one commit). Docstring: sharing a conversation that the row has not yet caught up would publish a truncated snapshot (the incident) — the upsert-first ordering makes the shared page the current conversation.
- **A3 — a dropped save re-runs (task 03; `persistConversation`, `app.js` L3216–3260).** `persisting` stays the one-upsert-at-a-time guard, but a save point that arrives while an upsert is in flight sets a module-level `savePending = true` instead of vanishing; the in-flight upsert's `finally` (after `persisting = false`) re-runs `persistConversation()` once when `savePending` is set (the flag is cleared first; the re-run is a fresh, serialized call — never concurrent with the upsert it trails). The re-run carries the LATEST `conversation` (the array is read at call time), so the dropped done-time save lands with the brain record. No new save points are added — the existing ones (send, `rememberBrainTurn`/pagehide, regenerate) are unchanged.
- **A4 — a failed save stays visible and retries once (task 03).** On failure (non-ok or network), the existing one-line note is kept AND a module flag `saveFailedSticky = true` set — the note persists until the NEXT SUCCESSFUL save (a success clears the flag and resets `#send-status` to empty when the flag was set; other flows may overwrite the note in the meantime — best-effort visibility, the flag + the next save point are the guarantee). A TRANSIENT failure (network error or 5xx status) additionally schedules ONE retry after 5 000 ms (a `setTimeout` + a one-shot `retryScheduled` guard so overlapping failures never stack retries; the retry is a plain `persistConversation()` — the LATEST conversation, the same guards). A 4xx NEVER retries (deterministic — the cap fix in task 01 is the remedy; the retry would 422 again). The A2 contract's shape is unchanged: still a one-line `#send-status` note, still NO error banner, the turn never blocks — only the note's lifetime and the one transient retry are added. `regenerateStaleChat`'s own upsert copy is LEFT AS IS (its failure path already errors visibly through the turn's banner state, and A1 removes the 422 class it could hit).
## Tasks
1. `01_schema_cap_realign.md` — the five list-cap moves + docstring rationale (A1), the unit boundary pins at the NEW values, the incident-payload integration pin (22 sources / 51 segments / 201 tools now 201/200), the oversized-422 pins at the new boundaries.
2. `02_share_saves_first.md` — `shareCurrentChat`'s linked branch upserts before minting the token, aborts the share on an upsert failure; the JS marker pins.
3. `03_persist_rerun_sticky.md` — the `savePending` overlap re-run, the sticky failure note, the one transient retry; the JS marker pins.
4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the incident regression at the wire + the overlap / share-saves-first behavior through the real UI), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_schemas.py` — the NEW boundary pairs per field (200/201 `sources`, 200/201 `related`, 200/201 `tools`, 100/101 `suggestions`, 450/451 `segments`), the realistic-payload round-trip pin EXTENDED (a 22-source / 34-tool / 40-segment brain record validates and round-trips byte-identically), every pre-phase pin green. `tests/unit/test_save_chat_ui.py` — the A2/A3/A4 marker pins (the upsert-before-share ordering in `shareCurrentChat`'s linked branch, the abort-on-failure path, `savePending` + the `finally` re-run, the sticky flag + the 5 s one-shot retry, the 4xx-no-retry distinction; the existing A2 quiet-contract pins stay green — the note text and the "no error banner" shape are unchanged).
- **Integration:** `tests/integration/test_chats_api.py` — anonymous `POST`/`PUT` with the INCIDENT payload shape (22 `sources`, 33 `tools`, 51 `segments` incl. thinking runs) → **201/200** (pre-fix: 422); the over-cap pins move to the new boundaries (201 `sources` → 422, no row; 201 `tools` → 422; 451 `segments` → 422; 101 `suggestions` → 422); the full brain-record pin + the stale/share pins stay green.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_chat_save_long_turns.py`, run **in isolation** — (a) the incident regression at the wire through `page.request` (22-source brain record → 201, pre-fix 422); (b) a real UI mock turn's done-time auto-save lands (row: 2 messages); (c) the overlap re-run (the send-time save's PUT delayed 3 s via a Playwright route, the mock turn settles in flight → the dropped done-time save re-runs → final row 2 messages); (d) share-saves-first (with the auto-saves still delayed, the Share click → the shared page renders the BRAIN message, not just the question).
- **Regression:** `tests/e2e/test_chat_save_payload_limits.py`, `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py` (the phase-51/55 share contract) green unedited.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the changed file is `app/schemas.py` (declarative — exercised by every unit/integration pin); the frontend change is marker-pinned per the house JS convention.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap), Share saves the current conversation before minting the token, a failed auto-save stays visible until the next success and retries once on a transient failure (task 04). `AGENTS.md`: nothing to change unless the final pass finds the phase-55 save contract described elsewhere (the save points are not enumerated in AGENTS.md — the UI-Structure rule's "in-flight turn" clause is untouched; verify, do not add).
## Completion Criteria
- [ ] `ChatMessage` list caps: `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; every other bound unchanged; the docstring rationale updated (A1).
- [ ] The incident payload (22 sources / 33 tools / 51 segments) saves 201/200 anonymously; over-cap variants 422 with no row stored (unit + integration pins).
- [ ] A linked Share upserts the current conversation before minting the token and aborts (banner, no token) on an upsert failure; the unlinked branch is byte-identical (A2, marker pins).
- [ ] A save point dropped by an in-flight upsert re-runs once after it settles (A3, marker pin); a failed save's note persists until the next success, one 5 s retry on network/5xx only (A4, marker pins).
- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** (the four scenarios); `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green unedited.
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `git diff --stat` limited to `app/schemas.py`, `frontend/assets/app.js`, the test files, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency).
- [ ] One atomic `--no-gpg-sign` commit (e.g. `fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run`); phase dir moved to `.agents/phases/complete/`.
## Owner action (data repair — NOT code, recorded here so the phase's commit message can cite it)
The two affected shared rows (`8b345ab5…`, `be92e464…`) still hold the user-only snapshot. Their brain answers were never persisted (the SSE bytes are not stored server-side). If the owner's browser still carries the conversations (`bor.chat.v1` localStorage restore), opening the chat page (which re-links to the row) and letting the FIXED client's next save point (or a Share — now save-first) fire restores the full content through the existing upsert; otherwise the answers are unrecoverable and the rows may be deleted from History.
@@ -0,0 +1,32 @@
# Task 01 — The saved-chat list caps move to the real agent configuration (A1)
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Raise the five `ChatMessage` list caps from the round-cap-10 era to the production `BOR_AGENT_MAX_ROUNDS=50` reality (the incident turn's 22-source done frame 422'd at the old 20 cap), update the sizing rationale in the docstrings, and move the boundary pins — so a long research turn's done-time auto-save lands instead of 422ing into the phase-55 quiet-failure, while the SEC-05 boundedness (fixed item-count boundary + the 8 M total budget) survives.
## Work
1. `app/schemas.py` — `ChatMessage` (L1381; the fields at L1458–1481):
- `sources: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20)
- `related: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20)
- `suggestions: list[_Chip] | None = Field(default=None, max_length=100)` (was 50; the `_Chip` 200-char item cap is unchanged)
- `tools: list[ToolCall] | None = Field(default=None, max_length=200)` (was 50)
- `segments: list[ChatSegment] | None = Field(default=None, max_length=450)` (was 50; the comment's "max_length=50 (the phase-83 voice): far past any realistic turn — the round cap is 10 → ≤ 20 entries; 50 matches the tools cap" is REPLACED with the phase-189 sizing: ≤ 200 tool entries (4× the production round cap of 50 — a round is one model request, one-or-more calls per round are legal) + ≤ ~50 thinking runs (one per round) + margin)
- **UNCHANGED, pinned as-is:** `text`/`thinking` 600 000, `error` 500, `image` 500, `answer_text` 50, the `SourceRef` field caps (120/1000/500 + `image_url`), the `ToolCall` field caps (name 100 / argument 2000 / the additive keys), `SavedChatCreate`/`SavedChatUpdate` `max_items=500`, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` and `_require_within_total_budget` (the verbatim-block accounting untouched).
- The `ChatMessage` class docstring: the phase-83 paragraph's "(sources ≤ 20 — top-N docs + agent reads; related ≤ 20 — the related-doc tier, same ref shape; suggestions ≤ 50 chips of ≤ 200 chars; tools ≤ 50 — one entry per tool call, the round cap is 10)" is UPDATED — the cited tier has been the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587): retrieval top-N ∪ the agent's read docs, bounded by `agent_max_rounds` (production 50 via `BOR_AGENT_MAX_ROUNDS`, the validator L— unbounded above), so the caps are `sources`/`related`/`tools` 200 (4× the production round cap) and `suggestions` 100 (20× the default 5; `suggested_docs` is tunable, unbounded above), `segments` 450; the phase-83 history (the 32 000 mirror raised to 600 000 with phase 164) is KEPT; one sentence added: the fixed boundary bounds item COUNT (an operator raising `BOR_AGENT_MAX_ROUNDS` far past 50 still hits a boundary) while `SAVED_CHAT_TOTAL_CHARS_BUDGET` is the byte bound (worst-case list bytes ≈ 360 KB / 440 KB / 1 MB — orders of magnitude inside 8 M).
2. `tests/unit/test_schemas.py` — the boundary-constant block (L47–75): `SOURCES_CAP = 200`, `TOOLS_CAP = 200`, `SUGGESTIONS_CAP = 100` (a `RELATED_CAP = 200` and `SEGMENTS_CAP = 450` constant added in the same style); the existing at-cap/one-over pins (L145–216 and the `segments` pins at L922–1009) now bind the NEW values with no edits (they read the constants); the realistic-payload round-trip pin (around L563) is EXTENDED with the incident shape: a brain record carrying 22 `sources`, 33 `tools`, and 40 `segments` (33 tool entries + 7 thinking entries) validates and `model_dump()`s byte-identically (the `_drop_image_when_absent` omission rules untouched — a record with all the keys present keeps them).
3. `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862) is RENAMED/retargeted to the new boundary (`test_post_rejects_sources_over_200_and_stores_nothing`: 201 `SourceRef` items → 422, no row); NEW pins: (a) `test_post_accepts_the_incident_payload_shape` — an anonymous `POST /api/chats` with a 2-message conversation (user + a brain record carrying 22 `sources`, 33 `tools`, 51 `segments` = 33 tool entries + 18 thinking entries — the incident turn's shape) → **201**, the row's `messages` round-trips all three lists intact; (b) the same shape through `PUT /api/chats/<id>` → **200**, the row's message count bumps to the new array; (c) `test_put_rejects_over_cap_lists_and_leaves_row_unchanged` — 201 `tools` / 451 `segments` / 101 `suggestions` each → **422** and the row's stored messages are byte-identical (the 422 is the framework's standard body — the phase-83 A2 shape, no custom error copy). Every pre-existing pin (the full brain-record pin, the stale/share pins, the 500-item messages cap, the total-budget pins) stays green unedited.
4. Verify: `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the new boundary pairs (200/201 `sources`, 200/201 `related`, 100/101 `suggestions`, 200/201 `tools`, 450/451 `segments`) pinned at BOTH boundaries; the extended realistic round-trip pin.
- Integration: the incident payload shape 201/200s anonymously (pre-fix: 422); the over-cap 422 pins at the new boundaries with no row stored / row unchanged; the pre-existing pins green unedited.
- Coverage: **>90%** on `app/` (the changed file is `app/schemas.py` — declarative, exercised by every pin).
## Completion Criteria
- [ ] `ChatMessage` list caps are 200/200/100/200/450 (sources/related/suggestions/tools/segments); every other bound unchanged; the docstring rationale is the `merge_cited_docs`/`agent_max_rounds` one with the 8 M-budget sentence.
- [ ] `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v` green — the incident shape 201/200s, the over-cap variants 422 with no row stored, all pre-existing pins green unedited.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
- [ ] no behavior change in completed work (the SSE `done` event is untouched — `ChatDoneEvent.sources` has no list cap; the `ChatMessage` caps bind the saved-chat boundary only)
@@ -0,0 +1,31 @@
# Task 02 — Share saves first: a linked Share never publishes a row the conversation has outgrown (A2)
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Make `shareCurrentChat`'s linked branch upsert the CURRENT conversation (the same upsert semantics as `persistConversation`) BEFORE minting the share token — so a Share can never freeze a stale row (the incident: the row held only the question, the Share minted a token on it, and the shared page showed no answer) — and abort the share with the existing neutral error banner when the upsert fails. The unlinked branch (save-AND-share in one `POST`) is byte-identical.
## Work
1. `frontend/assets/app.js` — `shareCurrentChat` (L3345), the linked branch (the `if (currentChatId)` arm, ~L3353–3363) becomes, in order:
1. the upsert: `const body = JSON.stringify({ messages: conversation }); const headers = { "Content-Type": "application/json" };` → `res = await fetch(`/api/chats/${currentChatId}`, { method: "PUT", headers, body });` with the SAME 404→recreate fallback as `persistConversation` (L3227–3234): on a 404, `currentChatId = null; res = await fetch("/api/chats", { method: "POST", headers, body });` and, on the 201, re-link (`currentChatId = String((await res.json()).id)`);
2. an upsert FAILURE (non-ok `res` after the fallback, or a network `throw`) → `showErrorBanner("Couldn't share the conversation — try again.")` and `return` (NO token minted, NO toast — the existing failure UI, the existing `finally` still releases `shareBtn.disabled`);
3. only then the existing `POST /api/chats/<id>/share` mint (unchanged) and the existing success paths (clipboard copy → the inline-field fallback, the `#send-status` line, the task-04 toast on BOTH success branches) — byte-identical.
The function docstring gains the phase-189 paragraph: the linked branch upserts first because a Share that minted a token on a row the conversation had outgrown published a truncated snapshot (the incident — the done-time auto-save had 422'd into the quiet-failure and the row held only the question); an upsert failure aborts the share (the owner must never be handed a link to content that is not the current conversation). The unlinked branch's docstring ("one action saves AND shares") is unchanged.
2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention, the module's existing style — no browser):
- `test_share_linked_branch_upserts_before_the_share_mint`: the linked branch of `shareCurrentChat` contains the PUT to `` `/api/chats/${currentChatId}` `` (the `persistConversation`-shaped upsert) positioned BEFORE the `` `/api/chats/${currentChatId}/share` `` fetch in the function body (a string-ordering pin: the PUT line's index < the share-mint line's index), and carries the 404→recreate fallback markers (the `status === 404` arm + the `POST` to `"/api/chats"` with the same `body`/`headers`);
- `test_share_aborts_on_upsert_failure`: the failure arm between the upsert and the mint sets the EXACT neutral banner text `"Couldn't share the conversation — try again."` and `return`s (no `showToast` call between the upsert's failure check and the banner — the toast is success-only, the existing pin stays green);
- `test_share_unlinked_branch_unchanged`: the unlinked branch still POSTs `"/api/chats"` with `share: true` in the body and re-links on the 201 (the existing pins, green unedited);
- the existing phase-55 pins (the always-visible `#share-chat-btn` markup, the neutral error copy, the `showToast` success-only contract, the double-click guard released in the `finally`) stay green unedited.
3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green.
## Testing & Quality
- Unit (JS markers): the upsert-before-mint ordering, the 404→recreate fallback in the share, the abort-on-failure path (banner, no token, no toast), the unlinked branch unchanged; every pre-existing pin green.
- Coverage: the frontend change is marker-pinned per the house JS convention (no JS runtime in the unit gate); the behavior is E2E-pinned by task 04.
## Completion Criteria
- [ ] `shareCurrentChat`'s linked branch: PUT upsert (with the 404→recreate fallback) → failure abort (neutral banner, no token, no toast) → share mint → the existing success paths byte-identical; the docstring says why.
- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
- [ ] no behavior change in completed work (the unlinked branch, the toast/banner contract, the `share_chat` endpoint — token-only, `app/api/chats.py` UNTOUCHED)
@@ -0,0 +1,34 @@
# Task 03 — Dropped saves re-run; a failed save stays visible and retries once (A3/A4)
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Close the two silent-loss paths in `persistConversation`: a save point dropped by the `persisting` double-fire guard (evidence 6 — a done-time save dropped by an in-flight send-time save vanishes until the next save point, which may never come) re-runs once after the in-flight upsert settles; a failed save's status note STAYS until the next success and a transient failure (network / 5xx) retries once after 5 s — while the phase-55 A2 contract's shape is unchanged (one-line `#send-status` note, NO error banner, the turn never blocks, silent success).
## Work
1. `frontend/assets/app.js` — `persistConversation` (L3216–3260) and its guard (L3216):
- `let persisting = false;` stays (the one-upsert-at-a-time guard); add `let savePending = false;` (A3) and `let saveFailedSticky = false; let retryScheduled = false;` (A4) at the same module scope, each with a one-line comment naming the phase-189 decision.
- **A3 — the overlap re-run:** the top guard becomes `if (persisting) { savePending = true; return; }` (a save point that arrives mid-upsert is PENDING, not dropped). At the top of the body, after the guard: `savePending = false;` (a fresh run consumes the flag — the pending re-run is this run when it was set by a re-run). In the `finally` (after `persisting = false;`): `if (savePending) { savePending = false; void persistConversation(); }` — the re-run is a fresh serialized call (it starts only after the in-flight upsert fully settled; never concurrent with it). The re-run reads `conversation` at call time, so the dropped done-time save lands with the brain record. No new save points, no other function touched.
- **A4 — sticky note + one transient retry:** in the failure paths (the `!res.ok` arm and the `catch` arm), after setting the existing note text, set `saveFailedSticky = true;` and — transient only — `if (!retryScheduled && (res.status >= 500 || <network-failure-arm>)) { retryScheduled = true; setTimeout(() => { retryScheduled = false; void persistConversation(); }, 5000); }` (the `catch` arm IS the network-failure arm; the 4xx — every other non-ok — NEVER schedules: a 422 is deterministic, the phase-189 task-01 cap fix is the remedy, and a retry would 422 again; the one-shot `retryScheduled` guard keeps overlapping failures from stacking retries). In the success path (the `staleBanner` clearing block), add: `if (saveFailedSticky) { saveFailedSticky = false; sendStatus.textContent = ""; }` (a success clears the stuck note — the note persists until the next success; other flows may overwrite the note in the meantime, best-effort visibility, the flag + the next save point are the guarantee).
- The `persistConversation` docstring: the "double-fire guard" paragraph is updated — a dropped save point is now PENDING and re-runs once after the in-flight upsert settles (A3); the A2 quiet-contract paragraph gains the two phase-189 lines (the note persists until the next success; one 5 s retry on network/5xx, never on 4xx — the note text and the no-banner shape are UNCHANGED).
- `regenerateStaleChat` (L4439–4460, its own upsert copy) is LEFT AS IS (its failure surfaces through the turn's banner state; A1 removes the 422 class it could hit).
2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention):
- `test_persisting_overlap_sets_pending_and_reruns`: `persistConversation` contains the `savePending = true` assignment inside the `if (persisting)` arm, the `savePending = false` reset at the top of the body, and the `finally` block re-runs `persistConversation()` guarded by `savePending` (the ordering: `persisting = false` before the `savePending` check in the `finally`);
- `test_failed_save_note_is_sticky_until_success`: the failure arms set `saveFailedSticky = true`, and the success path clears the flag + resets `sendStatus` (the pin: the success block contains the `saveFailedSticky` guard);
- `test_transient_failure_retries_once_after_5s`: the retry scheduling is present with the 5000 ms delay, the `>= 500` status condition, the `catch`-arm scheduling, and the one-shot `retryScheduled` guard (set before the `setTimeout`, reset inside it);
- `test_4xx_never_retries`: the retry condition is `res.status >= 500` (a 4xx non-ok schedules nothing — the pin: no other `setTimeout` scheduling in the failure arms);
- the existing phase-55 A2 pins (the one-line note text `"Couldn't save automatically — will try on the next message."`, no error-banner call in `persistConversation`, the silent-success shape, the `persisting` guard) stay green unedited.
3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green.
## Testing & Quality
- Unit (JS markers): the pending re-run (set/reset/finally-re-run ordering), the sticky-note lifetime, the one-shot 5 s transient retry, the 4xx exclusion; every pre-existing pin green.
- Coverage: the frontend change is marker-pinned per the house JS convention; the behavior is E2E-pinned by task 04 (the overlap scenario).
## Completion Criteria
- [ ] `persistConversation`: an overlapping save point sets `savePending`; the `finally` re-runs once after `persisting = false`; a fresh run consumes the flag.
- [ ] A failed save's note persists until the next success; a network/5xx failure schedules ONE 5 s retry (one-shot guard); a 4xx never retries; the A2 contract's shape (note text, no banner, silent success) is unchanged.
- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
- [ ] no behavior change in completed work (`regenerateStaleChat` untouched; no new save points; `app/api/chats.py` untouched)
@@ -0,0 +1,57 @@
# Task 04 — The E2E suite, the full gates, the docs pass, the commit
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Pin the owner-visible contract of the whole phase through the real UI: the incident regression at the wire (a 22-source brain record saves instead of 422ing), a mock turn's done-time auto-save landing, the overlap re-run (a dropped save lands after the in-flight upsert settles), and share-saves-first (a Share with the auto-saves still in flight publishes the BRAIN message, not just the question) — then the full gate, the docs pass, and the atomic commit.
## Work
1. `tests/e2e/test_chat_save_long_turns.py` — the phase's dedicated Playwright suite (the house `conftest` app + `mock_llm.py` pattern, run **in isolation** per AGENTS rule 4; a fresh saved-chat row per test — the house `saved_chats` cleanup helper pattern from `test_chat_history.py`):
- `test_incident_payload_shape_saves_at_the_wire`: through `page.request` (the `test_chat_save_payload_limits.py` pattern — the app's own origin), an anonymous `POST /api/chats` with the INCIDENT shape (a user message + a brain record carrying 22 `sources` (`SourceRef` dicts at realistic column-length values), 33 `tools` (`ToolCall` dicts), 51 `segments` = 33 `{kind: "tool", …}` + 18 `{kind: "thinking", text: …}`) → **201**; a follow-up `PUT` replacing the brain record with the same shape → **200** and the row's `messages` round-trips all three lists (a `GET /api/chats/<id>` admin assertion). (Pre-fix: both 422 — the incident.)
- `test_done_time_auto_save_lands_for_a_mock_turn`: the real UI — send a mock question (the `mock_llm.py` no-tools flow, the `test_chat_history.py` send pattern), wait for the done bubble (the house `wait_for_answer` helper), then `GET /api/chats/<id>` (the row id from the created-response `#…` — the house pattern reads `currentChatId` via `page.evaluate`): the row has **2** messages and the brain record's `text` equals the mock answer.
- `test_dropped_done_time_save_reruns_after_the_inflight_upsert_settles`: `page.route`-delay the FIRST PUT to `/api/chats/**` (the send-time auto-save) by 3 000 ms (a `page.route("**/api/chats/**", handler)` that resolves on `route.fulfill` after a `wait_for_timeout` ONLY for the first matched PUT — a module-level counter; later PUTs pass through untouched), then send the mock question and let the mock turn settle (the done bubble appears in flight — the mock is fast, the upsert is still parked at 3 s). Wait to ~5 s; `GET /api/chats/<id>`: the row has **2** messages (the dropped done-time save re-ran after the parked upsert settled — pre-fix: the row held 1).
- `test_share_saves_first_publishes_the_brain_message`: the same 3 s delay on the first PUT; send the mock question; while the send-time save is still parked (the done bubble has appeared), click `#share-chat-btn` (the house click helper, wait for the toast). The share's own upsert PUT passes the route filter UNdelayed (the counter already consumed by the parked send-time save), mints the token, and the toast lands; open `share_url` (the `test_share_chat.py` pattern — the `/shared/<token>` page, the `#shared-title` visible) and assert the shared page renders the BRAIN answer text (a `.msg .bubble` containing the mock answer string) — pre-fix: the shared page showed only the question. Assert the row (via admin `GET`) has 2 messages.
- Module docstring: the incident's evidence summary (the 422 PUT log line, the 22-source done frame, `created_at == updated_at` single-write row) so a future reader sees WHY these four tests exist.
2. The regression suites (green UNEDITED — do not touch them): `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — the text caps are unchanged), `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py`.
3. The full gate, in order: `uv run pytest -v` (unit + integration 100% green); `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/`); `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` (**in isolation**); `uv run ruff check . && uv run pyright` clean.
4. The docs pass (the phase's Documentation & Consistency mandate): review `README.md` and `AGENTS.md` in their entirety against the post-phase app; fix every inconsistency found, pre-existing ones included. `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap — 200/100/450), a Share saves the current conversation before minting the token (a failed share upsert aborts — no link to a truncated snapshot), a failed auto-save stays visible until the next success and retries once after 5 s on a transient failure (a 4xx never retries). `AGENTS.md`: verify the phase-55 save contract is not described anywhere in it (the save points are not enumerated — the UI-Structure rule's "in-flight turn" clause is untouched); if the final pass finds any stale save-contract sentence, fix it in place (no new rule added).
5. The atomic commit (AGENTS rule 8; the git-commit skill's `--no-gpg-sign` + `LLM-Generated: true` trailer): `git add -A && git commit --no-gpg-sign -m "fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run
The done-time auto-save of a long turn (production BOR_AGENT_MAX_ROUNDS=50;
the incident turn cited 22 docs, 33 tool calls, 34 rounds) 422'd at the
phase-83 list caps sized for the round-cap-10 era (sources ≤ 20, tools ≤ 50,
segments ≤ 50), and the phase-55 quiet-failure swallowed it: nothing retried,
so the row kept its pre-turn state — and a Share minted a token on that stale
row, publishing a conversation without its answer (incident 2026-10-16,
shared 0101d2ca… / row 8b345ab5…).
- A1: ChatMessage list caps follow the real configuration — sources/related/
tools 200 (4× the production round cap; the cited tier is the read docs,
merge_cited_docs), suggestions 100, segments 450; every other bound and the
8 M total budget unchanged (SEC-05 boundedness survives).
- A2: a linked Share upserts the current conversation before minting the
token and aborts on an upsert failure — never a link to a truncated
snapshot.
- A3: a save point dropped by the in-flight guard re-runs once after the
upsert settles.
- A4: a failed save's note persists until the next success; one 5 s retry on
network/5xx (a 4xx never retries).
LLM-Generated: true"`.
6. Move the phase directory: `mv .agents/phases/todo/189_chat_save_long_turns .agents/phases/complete/` and `git add .agents && git commit --no-gpg-sign -m "chore(agents): complete phase 189 (chat save long turns)"` — OR fold the phase move into the atomic commit per the house convention (check the last 3 phase commits' shape and match it; the AGENTS rule 8 says one atomic commit per phase including the phase files — if the house convention is one commit, `git add` the phase dir into step 5's commit and skip step 6's commit, keeping the `mv`).
## Testing & Quality
- E2E: the four scenarios above, in isolation.
- Regression: the three listed suites green unedited.
- Coverage / lint / types: the full gate (step 3).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** — the incident shape 201/200 at the wire, the mock turn's done-time save lands, the dropped save re-runs (final row 2 messages), the share with in-flight auto-saves publishes the brain message on the shared page.
- [ ] `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green UNEDITED.
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README.md / AGENTS.md reviewed in full; the saved-chat ops note in place; no stale save-contract sentence left.
- [ ] `git diff --stat` (before the commit) limited to `app/schemas.py`, `frontend/assets/app.js`, `tests/unit/test_schemas.py`, `tests/unit/test_save_chat_ui.py`, `tests/integration/test_chats_api.py`, `tests/e2e/test_chat_save_long_turns.py`, `README.md`, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency).
- [ ] the atomic `--no-gpg-sign` commit exists (message above); the phase dir is in `.agents/phases/complete/`.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] no behavior change in completed phases (the SSE `done` event, the `share_chat` endpoint, the unlinked share branch, the phase-55 A2 contract's shape — all unchanged)