From c054b47df568692eb55e9972ed54071cfd9d8a35 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Fri, 16 Oct 2026 15:22:51 -0400 Subject: [PATCH] phase: 187_sync_peak_memory_line MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Phase 187 — final verification pass: all criteria met, repo 100% green.** Verified (all four tasks already in `complete/`) - `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests. - Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical. - `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403. - `sources.js` `fmtSyncPeakClause`: ` · peak MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region. Gates (exact commands) - `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%** - `uv run ruff check .` → clean · `uv run pyright` → **0 errors** - `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice) - All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …) Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness. Documentation - README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed. - `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`). Notable - Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made. - **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched. Next pending phase: `188_chunks_hnsw_index`. --- .../187_sync_peak_memory_line/00_phase.md | 0 .../01_memory_helper.md | 0 .../02_sync_log_lines.md | 0 .../03_status_key_and_ui.md | 0 .../04_e2e_peak_memory.md | 0 .../todo/189_chat_save_long_turns/00_phase.md | 68 ++ .../01_schema_cap_realign.md | 32 + .../02_share_saves_first.md | 31 + .../03_persist_rerun_sticky.md | 34 + .../04_e2e_and_gates.md | 57 ++ ...187_sync_peak_memory_line__00_phase.a1.err | 0 .../187_sync_peak_memory_line__00_phase.a1.md | 25 + ...nc_peak_memory_line__00_phase.a1.md.status | 1 + ...ync_peak_memory_line__00_phase.a1.validate | 145 +++ ..._peak_memory_line__01_memory_helper.a1.err | 0 ...c_peak_memory_line__01_memory_helper.a1.md | 16 + ...memory_line__01_memory_helper.a1.md.status | 1 + ..._memory_line__01_memory_helper.a1.validate | 145 +++ ...peak_memory_line__02_sync_log_lines.a1.err | 0 ..._peak_memory_line__02_sync_log_lines.a1.md | 16 + ...emory_line__02_sync_log_lines.a1.md.status | 1 + ...memory_line__02_sync_log_lines.a1.validate | 145 +++ ...k_memory_line__03_status_key_and_ui.a1.err | 0 ...ak_memory_line__03_status_key_and_ui.a1.md | 14 + ...ry_line__03_status_key_and_ui.a1.md.status | 1 + ...ory_line__03_status_key_and_ui.a1.validate | 145 +++ ...eak_memory_line__04_e2e_peak_memory.a1.err | 0 ...peak_memory_line__04_e2e_peak_memory.a1.md | 18 + ...mory_line__04_e2e_peak_memory.a1.md.status | 1 + ...emory_line__04_e2e_peak_memory.a1.validate | 145 +++ AGENTS.md | 45 +- README.md | 111 ++- TODO.md | 2 + app/api/sync.py | 202 ++++- app/core/memory.py | 155 ++++ app/rag/importer.py | 11 +- frontend/assets/sources.js | 58 +- tests/e2e/sync_helpers.py | 61 ++ tests/e2e/test_answer_images.py | 3 +- tests/e2e/test_fast_path_deferred_content.py | 3 +- tests/e2e/test_force_push_resync.py | 7 +- tests/e2e/test_git_log_stream_dates.py | 3 +- tests/e2e/test_git_source_branch.py | 9 +- tests/e2e/test_git_source_dates.py | 3 +- tests/e2e/test_image_anchor_context.py | 3 +- tests/e2e/test_image_documents.py | 3 +- tests/e2e/test_pdf_image_context.py | 3 +- tests/e2e/test_pdf_image_extraction.py | 3 +- tests/e2e/test_pdf_support.py | 3 +- tests/e2e/test_pictorial_pdf.py | 3 +- tests/e2e/test_prune_cascade_cleanup.py | 3 +- tests/e2e/test_sync_button.py | 5 +- tests/e2e/test_sync_cancel.py | 8 +- tests/e2e/test_sync_changed_only.py | 3 +- tests/e2e/test_sync_chunk_progress.py | 3 +- tests/e2e/test_sync_git_freeze.py | 10 +- tests/e2e/test_sync_image_progress.py | 11 +- tests/e2e/test_sync_label_mobile.py | 3 +- tests/e2e/test_sync_model_down.py | 5 +- tests/e2e/test_sync_peak_memory_visible.py | 844 ++++++++++++++++++ tests/e2e/test_sync_progress_box.py | 13 +- tests/e2e/test_sync_session_reset.py | 13 +- tests/e2e/test_sync_status_freshness.py | 3 +- tests/e2e/test_sync_summary_visibility.py | 3 +- tests/e2e/test_sync_upload_progress.py | 13 +- tests/e2e/test_upload_no_scan.py | 5 +- tests/integration/test_importer_e2e.py | 45 +- tests/integration/test_sync_api.py | 391 +++++++- tests/unit/test_core_memory.py | 285 ++++++ tests/unit/test_frontend_sync_upload.py | 7 +- tests/unit/test_importer.py | 21 +- tests/unit/test_sync_button.py | 496 +++++++++- 72 files changed, 3826 insertions(+), 96 deletions(-) rename .agents/phases/{todo => complete}/187_sync_peak_memory_line/00_phase.md (100%) rename .agents/phases/{todo => complete}/187_sync_peak_memory_line/01_memory_helper.md (100%) rename .agents/phases/{todo => complete}/187_sync_peak_memory_line/02_sync_log_lines.md (100%) rename .agents/phases/{todo => complete}/187_sync_peak_memory_line/03_status_key_and_ui.md (100%) rename .agents/phases/{todo => complete}/187_sync_peak_memory_line/04_e2e_peak_memory.md (100%) create mode 100644 .agents/phases/todo/189_chat_save_long_turns/00_phase.md create mode 100644 .agents/phases/todo/189_chat_save_long_turns/01_schema_cap_realign.md create mode 100644 .agents/phases/todo/189_chat_save_long_turns/02_share_saves_first.md create mode 100644 .agents/phases/todo/189_chat_save_long_turns/03_persist_rerun_sticky.md create mode 100644 .agents/phases/todo/189_chat_save_long_turns/04_e2e_and_gates.md create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.err create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md.status create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.validate create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.err create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md.status create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.validate create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.err create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md.status create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.validate create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.err create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md.status create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.validate create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.err create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md.status create mode 100644 .agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.validate create mode 100644 app/core/memory.py create mode 100644 tests/e2e/sync_helpers.py create mode 100644 tests/e2e/test_sync_peak_memory_visible.py create mode 100644 tests/unit/test_core_memory.py diff --git a/.agents/phases/todo/187_sync_peak_memory_line/00_phase.md b/.agents/phases/complete/187_sync_peak_memory_line/00_phase.md similarity index 100% rename from .agents/phases/todo/187_sync_peak_memory_line/00_phase.md rename to .agents/phases/complete/187_sync_peak_memory_line/00_phase.md diff --git a/.agents/phases/todo/187_sync_peak_memory_line/01_memory_helper.md b/.agents/phases/complete/187_sync_peak_memory_line/01_memory_helper.md similarity index 100% rename from .agents/phases/todo/187_sync_peak_memory_line/01_memory_helper.md rename to .agents/phases/complete/187_sync_peak_memory_line/01_memory_helper.md diff --git a/.agents/phases/todo/187_sync_peak_memory_line/02_sync_log_lines.md b/.agents/phases/complete/187_sync_peak_memory_line/02_sync_log_lines.md similarity index 100% rename from .agents/phases/todo/187_sync_peak_memory_line/02_sync_log_lines.md rename to .agents/phases/complete/187_sync_peak_memory_line/02_sync_log_lines.md diff --git a/.agents/phases/todo/187_sync_peak_memory_line/03_status_key_and_ui.md b/.agents/phases/complete/187_sync_peak_memory_line/03_status_key_and_ui.md similarity index 100% rename from .agents/phases/todo/187_sync_peak_memory_line/03_status_key_and_ui.md rename to .agents/phases/complete/187_sync_peak_memory_line/03_status_key_and_ui.md diff --git a/.agents/phases/todo/187_sync_peak_memory_line/04_e2e_peak_memory.md b/.agents/phases/complete/187_sync_peak_memory_line/04_e2e_peak_memory.md similarity index 100% rename from .agents/phases/todo/187_sync_peak_memory_line/04_e2e_peak_memory.md rename to .agents/phases/complete/187_sync_peak_memory_line/04_e2e_peak_memory.md diff --git a/.agents/phases/todo/189_chat_save_long_turns/00_phase.md b/.agents/phases/todo/189_chat_save_long_turns/00_phase.md new file mode 100644 index 0000000..74f2549 --- /dev/null +++ b/.agents/phases/todo/189_chat_save_long_turns/00_phase.md @@ -0,0 +1,68 @@ +# Phase 189 — Long agent turns save (and share) the whole conversation + +**Source:** owner incident 2026-10-16 — the shared chat `0101d2ca-b24c-4d13-9f65-c0837e993c50` (row `8b345ab5-aab3-4412-9642-0f869b3cec4d`) shows ONLY the user's question although the LLM answered it completely; the shared row `be92e464-7918-4907-9186-09d8d98ff2c1` carries the same signature. +**Story:** n/a (incident-derived, owner directive 2026-10-16) +**Depends on:** none functionally (builds on the completed phase-55/83/158/164 saved-chat surface). + +## Evidence (diagnosed 2026-10-16, production DB + app logs — read this, not the chat) + +1. The row behind the shared token: `n_msgs = 1`, `who0 = user`, `created_at == updated_at` — the row was written EXACTLY ONCE (the send-time auto-save `POST /api/chats`) and never updated. 84 other single-write user-only rows exist (2 shared, incl. this one); the 2-message rows (279) were written >1 time — the pattern is specific to the *last* save point of a conversation. +2. The app log for this row shows the whole sequence: the turn's per-turn line (`thinking=xhigh thinking_chars=23937 tool_calls=33 rounds=34 cap_hit=0 total_ms=400173`, `POST /api/chat 200 stream=1` — the client consumed the FULL 400 s stream, so the done frame landed) followed by **`PUT /api/chats/8b345ab5… 422 Unprocessable Entity`** — the done-time auto-save REJECTED — then `GET /api/chats/… 200`, `POST …/share 200` (×2), `GET 200`. The share only minted a token on the stale row. +3. WHY the 422: the turn's done frame carried **22 sources** (counted from the per-turn log line's `sources=[…]`), but `ChatMessage.sources` is capped at **20** (`app/schemas.py` L1460 — phase 83 A1 "top-N docs + agent reads"). The cited tier is the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587/L1289 — retrieval top-N ∪ the agent's read docs, NOT bounded by `top_n_docs`), and production runs `BOR_AGENT_MAX_ROUNDS=50` (the app container's env; the config default is 10) — so a long research turn cites 22+ docs routinely and the done-time auto-save 422s. The same turn had **33 tool calls** (`tools` cap 50 — at the production round cap with ZERO headroom) and `segments` entries = 33 tool entries + one thinking entry per contiguous thinking run (`buildSegments`, `app.js` L1775) under `xhigh` thinking over 34 rounds — the `segments` cap of 50 (phase 158, docstring sized "the round cap is 10 → ≤ 20 entries") is exceeded by the same class of turn. `suggestions` (cap 50) and `related` (cap 20) are likewise sized against settings (`suggested_docs`, `related_max_docs`) whose validators are unbounded above. +4. WHY the 422 lost the answer: the phase-55 A2 quiet contract swallows a failed auto-save — a one-line `#send-status` note ("Couldn't save automatically — will try on the next message."), no error banner — and NOTHING retries: the retry is the NEXT save point (next user send / next brain-done / the pagehide partial), which never came (this was the last turn). The row stayed at its pre-turn state. +5. WHY the share froze the loss: `shareCurrentChat`'s LINKED branch (`app.js` L3345) only calls `POST /api/chats//share` (token mint, `app/api/chats.py` `share_chat`) — it never re-saves the current conversation, so the shared snapshot is whatever the row last held. +6. SECOND loss path (same symptom class): `persistConversation`'s `persisting` double-fire guard (`app.js` L3216) DROPS a save point that arrives while an upsert is in flight ("the next save point retries" — which, per 4., may never come). A done-time save dropped by a still-in-flight send-time save (a 400 s turn on a slow line) leaves the same user-only row. + +## Objective +A long agent turn (production `BOR_AGENT_MAX_ROUNDS=50`) saves cleanly at every save point: the saved-chat schema caps move from the round-cap-10 era to the real configuration (A1); a Share never publishes a row the current conversation has outgrown — it saves first (A2); a dropped or failed save self-heals (overlap re-run, one transient retry) and a failed save STAYS visible until the next success (A3/A4). + +## Context +- `app/schemas.py` — `ChatMessage` (L1381; `extra="forbid"`, L1458): `text`/`thinking` `max_length=600_000` (phase 164), `sources` `max_length=20` (L1459), `related` `max_length=20`, `suggestions` `max_length=50` (items `_Chip` ≤ 200), `tools` `max_length=50`, `segments` `max_length=50` (L1479), `error` ≤ 500, `image` ≤ 500, `answer_image` `SourceRef`, `answer_text` `max_length=50`, `budget_exhausted`; `SourceRef` (L1260-ish: `source` ≤ 120 / `path` ≤ 1000 / `title` ≤ 500 / `image_url`), `ToolCall` (L1209: `name` ≤ 100, `argument` ≤ 2000, `chars_shown`/`chars_total`/`offset`, the phase-139 `range` + phase-146 `path` additive keys), `SavedChatCreate`/`SavedChatUpdate` (`messages` `min_length=1, max_items=500` — phase 164; the `_require_within_total_budget` validator), `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` (L1611). The phase-83 docstrings on the list fields still say "top-N docs + agent reads — the UI shows a handful" — that assumption is RETIRED (see evidence 3). +- `app/api/chat.py` — `merge_cited_docs` (L587), the done-event build (L1402–1447: `cited_refs` = the merged read docs, `related_refs` capped by `related_max_docs`), `app/rag/retriever.py` (the tiering), `app/config.py` (`agent_max_rounds` L279, validator L—: `>= 0`, NO upper bound; `suggested_docs` / `related_max_docs` validators likewise unbounded above; production env `BOR_AGENT_MAX_ROUNDS=50`). +- `app/api/chats.py` — the public write surface (phase 55 A1), `share_chat` (token-only, L379-ish), the 422 is the framework's standard validation response (phase 83 A2: boundary-only, no route changes). +- `frontend/assets/app.js` — `persistConversation` (L3218: the `persisting` guard L3216, the 404→recreate fallback, the A2 quiet contract on failure), `rememberBrainTurn` (L3484 — save point 2, rides L3496), the send save point (L1371), `shareCurrentChat` (L3345: linked → `POST /{id}/share` only; unlinked → `POST /api/chats {messages, share: true}`), `regenerateStaleChat` (L3419 — its own upsert copy), the pagehide partial (L4821, rides `rememberBrainTurn`), `buildSegments` (L1775), `sendStatus` (the `#send-status` live region). +- `tests/unit/test_schemas.py` — the boundary-constant block (L47–75: `SOURCES_CAP = 20`, `SUGGESTIONS_CAP = 50`, `TOOLS_CAP = 50`, `TEXT_CAP = 600_000`, …) and the at-cap/one-over pins per field; `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862), the oversized-422 pins, the full brain-record pin; `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — text caps UNCHANGED, this suite is unaffected); `tests/unit/test_save_chat_ui.py` (the JS marker pins for the save/share contract — the A2 quiet-contract pin names the status-note text and the `persisting` guard); `tests/e2e/mock_llm.py` (the deterministic mock turn: single-read tool flow, `slow_llm.py` for timing). + +## Design (shared by all tasks — the executor reads this, not the chat) + +- **A1 — the caps move to the real configuration (task 01; `app/schemas.py` ONLY, boundary-only — the phase-83 A2 style: NO route change, the 422 stays the framework's standard response).** The five list caps change, every other bound (per-message `text`/`thinking` 600 000, `error` 500, `image` 500, the `SourceRef` field caps, the `ToolCall` field caps, `answer_text` 50, the `max_items=500` messages cap, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8 000 000`) is UNCHANGED: + - `sources` 20 → **200** — the cited tier is top-N ∪ the agent's READ docs (`merge_cited_docs`), so its size is bounded by `agent_max_rounds` (production 50, validator-unbounded) × calls-per-round, not by `top_n_docs`; 200 = 4× the production round cap (each round typically carries one `read`); + - `related` 20 → **200** — `related_max_docs` is owner-tunable with no upper bound; one `SourceRef`-family cap with `sources`; + - `tools` 50 → **200** — one-or-more calls per round are legal (a round is one model request, not one call); 4× the production round cap; + - `segments` 50 → **450** — ≤ `tools` (200) tool entries + ≤ ~50 thinking runs (one per round, the cap-50 worst case) + margin; + - `suggestions` 50 → **100** — `suggested_docs` is owner-tunable with no upper bound; 20× the default (5); the item cap (200 chars) is unchanged. + The SEC-05 intent survives: the worst-case list bytes are orders of magnitude inside the 8 M TOTAL budget (`200 SourceRefs ≈ 360 KB`, `200 ToolCalls ≈ 440 KB`, `450 segments ≈ 1 MB`), and the fixed boundary still bounds item COUNT for an operator who raises `BOR_AGENT_MAX_ROUNDS` far beyond 50. The `ChatMessage`/`SourceRef` docstrings are updated: the "top-N + a handful" sizing rationale is replaced by the `merge_cited_docs` / `agent_max_rounds` reality (evidence 3), the phase-83 history KEPT (the 32 000 mirror is phase 164's 600 000), and a note that the 8 M total budget is the byte bound. The unit constants (`SOURCES_CAP`, `TOOLS_CAP`, `SUGGESTIONS_CAP`) and the integration pins move with the values. +- **A2 — Share saves first (task 02; `shareCurrentChat`, `app.js` L3345).** The LINKED branch upserts the CURRENT conversation BEFORE minting the token — the SAME upsert semantics as `persistConversation` (PUT; a 404 unlinks and retries as a create), then `POST /api/chats//share`. An upsert FAILURE (non-ok or network) ABORTS the share with the existing neutral error banner ("Couldn't share the conversation — try again.") — NO token, NO toast, `shareBtn` released in the `finally` (the never-stale contract, PLAN §7.4). The UNLINKED branch is unchanged (its `POST {messages, share: true}` already saves the current conversation in one commit). Docstring: sharing a conversation that the row has not yet caught up would publish a truncated snapshot (the incident) — the upsert-first ordering makes the shared page the current conversation. +- **A3 — a dropped save re-runs (task 03; `persistConversation`, `app.js` L3216–3260).** `persisting` stays the one-upsert-at-a-time guard, but a save point that arrives while an upsert is in flight sets a module-level `savePending = true` instead of vanishing; the in-flight upsert's `finally` (after `persisting = false`) re-runs `persistConversation()` once when `savePending` is set (the flag is cleared first; the re-run is a fresh, serialized call — never concurrent with the upsert it trails). The re-run carries the LATEST `conversation` (the array is read at call time), so the dropped done-time save lands with the brain record. No new save points are added — the existing ones (send, `rememberBrainTurn`/pagehide, regenerate) are unchanged. +- **A4 — a failed save stays visible and retries once (task 03).** On failure (non-ok or network), the existing one-line note is kept AND a module flag `saveFailedSticky = true` set — the note persists until the NEXT SUCCESSFUL save (a success clears the flag and resets `#send-status` to empty when the flag was set; other flows may overwrite the note in the meantime — best-effort visibility, the flag + the next save point are the guarantee). A TRANSIENT failure (network error or 5xx status) additionally schedules ONE retry after 5 000 ms (a `setTimeout` + a one-shot `retryScheduled` guard so overlapping failures never stack retries; the retry is a plain `persistConversation()` — the LATEST conversation, the same guards). A 4xx NEVER retries (deterministic — the cap fix in task 01 is the remedy; the retry would 422 again). The A2 contract's shape is unchanged: still a one-line `#send-status` note, still NO error banner, the turn never blocks — only the note's lifetime and the one transient retry are added. `regenerateStaleChat`'s own upsert copy is LEFT AS IS (its failure path already errors visibly through the turn's banner state, and A1 removes the 422 class it could hit). + +## Tasks +1. `01_schema_cap_realign.md` — the five list-cap moves + docstring rationale (A1), the unit boundary pins at the NEW values, the incident-payload integration pin (22 sources / 51 segments / 201 tools now 201/200), the oversized-422 pins at the new boundaries. +2. `02_share_saves_first.md` — `shareCurrentChat`'s linked branch upserts before minting the token, aborts the share on an upsert failure; the JS marker pins. +3. `03_persist_rerun_sticky.md` — the `savePending` overlap re-run, the sticky failure note, the one transient retry; the JS marker pins. +4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the incident regression at the wire + the overlap / share-saves-first behavior through the real UI), the full gates, the docs pass, the atomic commit. + +## Testing & Quality +- **Unit:** `tests/unit/test_schemas.py` — the NEW boundary pairs per field (200/201 `sources`, 200/201 `related`, 200/201 `tools`, 100/101 `suggestions`, 450/451 `segments`), the realistic-payload round-trip pin EXTENDED (a 22-source / 34-tool / 40-segment brain record validates and round-trips byte-identically), every pre-phase pin green. `tests/unit/test_save_chat_ui.py` — the A2/A3/A4 marker pins (the upsert-before-share ordering in `shareCurrentChat`'s linked branch, the abort-on-failure path, `savePending` + the `finally` re-run, the sticky flag + the 5 s one-shot retry, the 4xx-no-retry distinction; the existing A2 quiet-contract pins stay green — the note text and the "no error banner" shape are unchanged). +- **Integration:** `tests/integration/test_chats_api.py` — anonymous `POST`/`PUT` with the INCIDENT payload shape (22 `sources`, 33 `tools`, 51 `segments` incl. thinking runs) → **201/200** (pre-fix: 422); the over-cap pins move to the new boundaries (201 `sources` → 422, no row; 201 `tools` → 422; 451 `segments` → 422; 101 `suggestions` → 422); the full brain-record pin + the stale/share pins stay green. +- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_chat_save_long_turns.py`, run **in isolation** — (a) the incident regression at the wire through `page.request` (22-source brain record → 201, pre-fix 422); (b) a real UI mock turn's done-time auto-save lands (row: 2 messages); (c) the overlap re-run (the send-time save's PUT delayed 3 s via a Playwright route, the mock turn settles in flight → the dropped done-time save re-runs → final row 2 messages); (d) share-saves-first (with the auto-saves still delayed, the Share click → the shared page renders the BRAIN message, not just the question). +- **Regression:** `tests/e2e/test_chat_save_payload_limits.py`, `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py` (the phase-51/55 share contract) green unedited. +- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the changed file is `app/schemas.py` (declarative — exercised by every unit/integration pin); the frontend change is marker-pinned per the house JS convention. +- **Lint/types:** `uv run ruff check . && uv run pyright`. + +## Documentation & Consistency +- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included. +- `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap), Share saves the current conversation before minting the token, a failed auto-save stays visible until the next success and retries once on a transient failure (task 04). `AGENTS.md`: nothing to change unless the final pass finds the phase-55 save contract described elsewhere (the save points are not enumerated in AGENTS.md — the UI-Structure rule's "in-flight turn" clause is untouched; verify, do not add). + +## Completion Criteria +- [ ] `ChatMessage` list caps: `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; every other bound unchanged; the docstring rationale updated (A1). +- [ ] The incident payload (22 sources / 33 tools / 51 segments) saves 201/200 anonymously; over-cap variants 422 with no row stored (unit + integration pins). +- [ ] A linked Share upserts the current conversation before minting the token and aborts (banner, no token) on an upsert failure; the unlinked branch is byte-identical (A2, marker pins). +- [ ] A save point dropped by an in-flight upsert re-runs once after it settles (A3, marker pin); a failed save's note persists until the next success, one 5 s retry on network/5xx only (A4, marker pins). +- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** (the four scenarios); `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green unedited. +- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean. +- [ ] `git diff --stat` limited to `app/schemas.py`, `frontend/assets/app.js`, the test files, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency). +- [ ] One atomic `--no-gpg-sign` commit (e.g. `fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run`); phase dir moved to `.agents/phases/complete/`. + +## Owner action (data repair — NOT code, recorded here so the phase's commit message can cite it) +The two affected shared rows (`8b345ab5…`, `be92e464…`) still hold the user-only snapshot. Their brain answers were never persisted (the SSE bytes are not stored server-side). If the owner's browser still carries the conversations (`bor.chat.v1` localStorage restore), opening the chat page (which re-links to the row) and letting the FIXED client's next save point (or a Share — now save-first) fire restores the full content through the existing upsert; otherwise the answers are unrecoverable and the rows may be deleted from History. diff --git a/.agents/phases/todo/189_chat_save_long_turns/01_schema_cap_realign.md b/.agents/phases/todo/189_chat_save_long_turns/01_schema_cap_realign.md new file mode 100644 index 0000000..2e74bae --- /dev/null +++ b/.agents/phases/todo/189_chat_save_long_turns/01_schema_cap_realign.md @@ -0,0 +1,32 @@ +# Task 01 — The saved-chat list caps move to the real agent configuration (A1) + +**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16 +**Story:** n/a + +## Objective +Raise the five `ChatMessage` list caps from the round-cap-10 era to the production `BOR_AGENT_MAX_ROUNDS=50` reality (the incident turn's 22-source done frame 422'd at the old 20 cap), update the sizing rationale in the docstrings, and move the boundary pins — so a long research turn's done-time auto-save lands instead of 422ing into the phase-55 quiet-failure, while the SEC-05 boundedness (fixed item-count boundary + the 8 M total budget) survives. + +## Work +1. `app/schemas.py` — `ChatMessage` (L1381; the fields at L1458–1481): + - `sources: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20) + - `related: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20) + - `suggestions: list[_Chip] | None = Field(default=None, max_length=100)` (was 50; the `_Chip` 200-char item cap is unchanged) + - `tools: list[ToolCall] | None = Field(default=None, max_length=200)` (was 50) + - `segments: list[ChatSegment] | None = Field(default=None, max_length=450)` (was 50; the comment's "max_length=50 (the phase-83 voice): far past any realistic turn — the round cap is 10 → ≤ 20 entries; 50 matches the tools cap" is REPLACED with the phase-189 sizing: ≤ 200 tool entries (4× the production round cap of 50 — a round is one model request, one-or-more calls per round are legal) + ≤ ~50 thinking runs (one per round) + margin) + - **UNCHANGED, pinned as-is:** `text`/`thinking` 600 000, `error` 500, `image` 500, `answer_text` 50, the `SourceRef` field caps (120/1000/500 + `image_url`), the `ToolCall` field caps (name 100 / argument 2000 / the additive keys), `SavedChatCreate`/`SavedChatUpdate` `max_items=500`, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` and `_require_within_total_budget` (the verbatim-block accounting untouched). + - The `ChatMessage` class docstring: the phase-83 paragraph's "(sources ≤ 20 — top-N docs + agent reads; related ≤ 20 — the related-doc tier, same ref shape; suggestions ≤ 50 chips of ≤ 200 chars; tools ≤ 50 — one entry per tool call, the round cap is 10)" is UPDATED — the cited tier has been the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587): retrieval top-N ∪ the agent's read docs, bounded by `agent_max_rounds` (production 50 via `BOR_AGENT_MAX_ROUNDS`, the validator L— unbounded above), so the caps are `sources`/`related`/`tools` 200 (4× the production round cap) and `suggestions` 100 (20× the default 5; `suggested_docs` is tunable, unbounded above), `segments` 450; the phase-83 history (the 32 000 mirror raised to 600 000 with phase 164) is KEPT; one sentence added: the fixed boundary bounds item COUNT (an operator raising `BOR_AGENT_MAX_ROUNDS` far past 50 still hits a boundary) while `SAVED_CHAT_TOTAL_CHARS_BUDGET` is the byte bound (worst-case list bytes ≈ 360 KB / 440 KB / 1 MB — orders of magnitude inside 8 M). +2. `tests/unit/test_schemas.py` — the boundary-constant block (L47–75): `SOURCES_CAP = 200`, `TOOLS_CAP = 200`, `SUGGESTIONS_CAP = 100` (a `RELATED_CAP = 200` and `SEGMENTS_CAP = 450` constant added in the same style); the existing at-cap/one-over pins (L145–216 and the `segments` pins at L922–1009) now bind the NEW values with no edits (they read the constants); the realistic-payload round-trip pin (around L563) is EXTENDED with the incident shape: a brain record carrying 22 `sources`, 33 `tools`, and 40 `segments` (33 tool entries + 7 thinking entries) validates and `model_dump()`s byte-identically (the `_drop_image_when_absent` omission rules untouched — a record with all the keys present keeps them). +3. `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862) is RENAMED/retargeted to the new boundary (`test_post_rejects_sources_over_200_and_stores_nothing`: 201 `SourceRef` items → 422, no row); NEW pins: (a) `test_post_accepts_the_incident_payload_shape` — an anonymous `POST /api/chats` with a 2-message conversation (user + a brain record carrying 22 `sources`, 33 `tools`, 51 `segments` = 33 tool entries + 18 thinking entries — the incident turn's shape) → **201**, the row's `messages` round-trips all three lists intact; (b) the same shape through `PUT /api/chats/` → **200**, the row's message count bumps to the new array; (c) `test_put_rejects_over_cap_lists_and_leaves_row_unchanged` — 201 `tools` / 451 `segments` / 101 `suggestions` each → **422** and the row's stored messages are byte-identical (the 422 is the framework's standard body — the phase-83 A2 shape, no custom error copy). Every pre-existing pin (the full brain-record pin, the stale/share pins, the 500-item messages cap, the total-budget pins) stays green unedited. +4. Verify: `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean. + +## Testing & Quality +- Unit: the new boundary pairs (200/201 `sources`, 200/201 `related`, 100/101 `suggestions`, 200/201 `tools`, 450/451 `segments`) pinned at BOTH boundaries; the extended realistic round-trip pin. +- Integration: the incident payload shape 201/200s anonymously (pre-fix: 422); the over-cap 422 pins at the new boundaries with no row stored / row unchanged; the pre-existing pins green unedited. +- Coverage: **>90%** on `app/` (the changed file is `app/schemas.py` — declarative, exercised by every pin). + +## Completion Criteria +- [ ] `ChatMessage` list caps are 200/200/100/200/450 (sources/related/suggestions/tools/segments); every other bound unchanged; the docstring rationale is the `merge_cited_docs`/`agent_max_rounds` one with the 8 M-budget sentence. +- [ ] `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v` green — the incident shape 201/200s, the over-cap variants 422 with no row stored, all pre-existing pins green unedited. +- [ ] full test suite **100% passing** (every test in the project, not just this task's) +- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide) +- [ ] no behavior change in completed work (the SSE `done` event is untouched — `ChatDoneEvent.sources` has no list cap; the `ChatMessage` caps bind the saved-chat boundary only) diff --git a/.agents/phases/todo/189_chat_save_long_turns/02_share_saves_first.md b/.agents/phases/todo/189_chat_save_long_turns/02_share_saves_first.md new file mode 100644 index 0000000..463f1b0 --- /dev/null +++ b/.agents/phases/todo/189_chat_save_long_turns/02_share_saves_first.md @@ -0,0 +1,31 @@ +# Task 02 — Share saves first: a linked Share never publishes a row the conversation has outgrown (A2) + +**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16 +**Story:** n/a + +## Objective +Make `shareCurrentChat`'s linked branch upsert the CURRENT conversation (the same upsert semantics as `persistConversation`) BEFORE minting the share token — so a Share can never freeze a stale row (the incident: the row held only the question, the Share minted a token on it, and the shared page showed no answer) — and abort the share with the existing neutral error banner when the upsert fails. The unlinked branch (save-AND-share in one `POST`) is byte-identical. + +## Work +1. `frontend/assets/app.js` — `shareCurrentChat` (L3345), the linked branch (the `if (currentChatId)` arm, ~L3353–3363) becomes, in order: + 1. the upsert: `const body = JSON.stringify({ messages: conversation }); const headers = { "Content-Type": "application/json" };` → `res = await fetch(`/api/chats/${currentChatId}`, { method: "PUT", headers, body });` with the SAME 404→recreate fallback as `persistConversation` (L3227–3234): on a 404, `currentChatId = null; res = await fetch("/api/chats", { method: "POST", headers, body });` and, on the 201, re-link (`currentChatId = String((await res.json()).id)`); + 2. an upsert FAILURE (non-ok `res` after the fallback, or a network `throw`) → `showErrorBanner("Couldn't share the conversation — try again.")` and `return` (NO token minted, NO toast — the existing failure UI, the existing `finally` still releases `shareBtn.disabled`); + 3. only then the existing `POST /api/chats//share` mint (unchanged) and the existing success paths (clipboard copy → the inline-field fallback, the `#send-status` line, the task-04 toast on BOTH success branches) — byte-identical. + The function docstring gains the phase-189 paragraph: the linked branch upserts first because a Share that minted a token on a row the conversation had outgrown published a truncated snapshot (the incident — the done-time auto-save had 422'd into the quiet-failure and the row held only the question); an upsert failure aborts the share (the owner must never be handed a link to content that is not the current conversation). The unlinked branch's docstring ("one action saves AND shares") is unchanged. +2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention, the module's existing style — no browser): + - `test_share_linked_branch_upserts_before_the_share_mint`: the linked branch of `shareCurrentChat` contains the PUT to `` `/api/chats/${currentChatId}` `` (the `persistConversation`-shaped upsert) positioned BEFORE the `` `/api/chats/${currentChatId}/share` `` fetch in the function body (a string-ordering pin: the PUT line's index < the share-mint line's index), and carries the 404→recreate fallback markers (the `status === 404` arm + the `POST` to `"/api/chats"` with the same `body`/`headers`); + - `test_share_aborts_on_upsert_failure`: the failure arm between the upsert and the mint sets the EXACT neutral banner text `"Couldn't share the conversation — try again."` and `return`s (no `showToast` call between the upsert's failure check and the banner — the toast is success-only, the existing pin stays green); + - `test_share_unlinked_branch_unchanged`: the unlinked branch still POSTs `"/api/chats"` with `share: true` in the body and re-links on the 201 (the existing pins, green unedited); + - the existing phase-55 pins (the always-visible `#share-chat-btn` markup, the neutral error copy, the `showToast` success-only contract, the double-click guard released in the `finally`) stay green unedited. +3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green. + +## Testing & Quality +- Unit (JS markers): the upsert-before-mint ordering, the 404→recreate fallback in the share, the abort-on-failure path (banner, no token, no toast), the unlinked branch unchanged; every pre-existing pin green. +- Coverage: the frontend change is marker-pinned per the house JS convention (no JS runtime in the unit gate); the behavior is E2E-pinned by task 04. + +## Completion Criteria +- [ ] `shareCurrentChat`'s linked branch: PUT upsert (with the 404→recreate fallback) → failure abort (neutral banner, no token, no toast) → share mint → the existing success paths byte-identical; the docstring says why. +- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin. +- [ ] full test suite **100% passing** (every test in the project, not just this task's) +- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide) +- [ ] no behavior change in completed work (the unlinked branch, the toast/banner contract, the `share_chat` endpoint — token-only, `app/api/chats.py` UNTOUCHED) diff --git a/.agents/phases/todo/189_chat_save_long_turns/03_persist_rerun_sticky.md b/.agents/phases/todo/189_chat_save_long_turns/03_persist_rerun_sticky.md new file mode 100644 index 0000000..d470d3f --- /dev/null +++ b/.agents/phases/todo/189_chat_save_long_turns/03_persist_rerun_sticky.md @@ -0,0 +1,34 @@ +# Task 03 — Dropped saves re-run; a failed save stays visible and retries once (A3/A4) + +**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16 +**Story:** n/a + +## Objective +Close the two silent-loss paths in `persistConversation`: a save point dropped by the `persisting` double-fire guard (evidence 6 — a done-time save dropped by an in-flight send-time save vanishes until the next save point, which may never come) re-runs once after the in-flight upsert settles; a failed save's status note STAYS until the next success and a transient failure (network / 5xx) retries once after 5 s — while the phase-55 A2 contract's shape is unchanged (one-line `#send-status` note, NO error banner, the turn never blocks, silent success). + +## Work +1. `frontend/assets/app.js` — `persistConversation` (L3216–3260) and its guard (L3216): + - `let persisting = false;` stays (the one-upsert-at-a-time guard); add `let savePending = false;` (A3) and `let saveFailedSticky = false; let retryScheduled = false;` (A4) at the same module scope, each with a one-line comment naming the phase-189 decision. + - **A3 — the overlap re-run:** the top guard becomes `if (persisting) { savePending = true; return; }` (a save point that arrives mid-upsert is PENDING, not dropped). At the top of the body, after the guard: `savePending = false;` (a fresh run consumes the flag — the pending re-run is this run when it was set by a re-run). In the `finally` (after `persisting = false;`): `if (savePending) { savePending = false; void persistConversation(); }` — the re-run is a fresh serialized call (it starts only after the in-flight upsert fully settled; never concurrent with it). The re-run reads `conversation` at call time, so the dropped done-time save lands with the brain record. No new save points, no other function touched. + - **A4 — sticky note + one transient retry:** in the failure paths (the `!res.ok` arm and the `catch` arm), after setting the existing note text, set `saveFailedSticky = true;` and — transient only — `if (!retryScheduled && (res.status >= 500 || )) { retryScheduled = true; setTimeout(() => { retryScheduled = false; void persistConversation(); }, 5000); }` (the `catch` arm IS the network-failure arm; the 4xx — every other non-ok — NEVER schedules: a 422 is deterministic, the phase-189 task-01 cap fix is the remedy, and a retry would 422 again; the one-shot `retryScheduled` guard keeps overlapping failures from stacking retries). In the success path (the `staleBanner` clearing block), add: `if (saveFailedSticky) { saveFailedSticky = false; sendStatus.textContent = ""; }` (a success clears the stuck note — the note persists until the next success; other flows may overwrite the note in the meantime, best-effort visibility, the flag + the next save point are the guarantee). + - The `persistConversation` docstring: the "double-fire guard" paragraph is updated — a dropped save point is now PENDING and re-runs once after the in-flight upsert settles (A3); the A2 quiet-contract paragraph gains the two phase-189 lines (the note persists until the next success; one 5 s retry on network/5xx, never on 4xx — the note text and the no-banner shape are UNCHANGED). + - `regenerateStaleChat` (L4439–4460, its own upsert copy) is LEFT AS IS (its failure surfaces through the turn's banner state; A1 removes the 422 class it could hit). +2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention): + - `test_persisting_overlap_sets_pending_and_reruns`: `persistConversation` contains the `savePending = true` assignment inside the `if (persisting)` arm, the `savePending = false` reset at the top of the body, and the `finally` block re-runs `persistConversation()` guarded by `savePending` (the ordering: `persisting = false` before the `savePending` check in the `finally`); + - `test_failed_save_note_is_sticky_until_success`: the failure arms set `saveFailedSticky = true`, and the success path clears the flag + resets `sendStatus` (the pin: the success block contains the `saveFailedSticky` guard); + - `test_transient_failure_retries_once_after_5s`: the retry scheduling is present with the 5000 ms delay, the `>= 500` status condition, the `catch`-arm scheduling, and the one-shot `retryScheduled` guard (set before the `setTimeout`, reset inside it); + - `test_4xx_never_retries`: the retry condition is `res.status >= 500` (a 4xx non-ok schedules nothing — the pin: no other `setTimeout` scheduling in the failure arms); + - the existing phase-55 A2 pins (the one-line note text `"Couldn't save automatically — will try on the next message."`, no error-banner call in `persistConversation`, the silent-success shape, the `persisting` guard) stay green unedited. +3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green. + +## Testing & Quality +- Unit (JS markers): the pending re-run (set/reset/finally-re-run ordering), the sticky-note lifetime, the one-shot 5 s transient retry, the 4xx exclusion; every pre-existing pin green. +- Coverage: the frontend change is marker-pinned per the house JS convention; the behavior is E2E-pinned by task 04 (the overlap scenario). + +## Completion Criteria +- [ ] `persistConversation`: an overlapping save point sets `savePending`; the `finally` re-runs once after `persisting = false`; a fresh run consumes the flag. +- [ ] A failed save's note persists until the next success; a network/5xx failure schedules ONE 5 s retry (one-shot guard); a 4xx never retries; the A2 contract's shape (note text, no banner, silent success) is unchanged. +- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin. +- [ ] full test suite **100% passing** (every test in the project, not just this task's) +- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide) +- [ ] no behavior change in completed work (`regenerateStaleChat` untouched; no new save points; `app/api/chats.py` untouched) diff --git a/.agents/phases/todo/189_chat_save_long_turns/04_e2e_and_gates.md b/.agents/phases/todo/189_chat_save_long_turns/04_e2e_and_gates.md new file mode 100644 index 0000000..fab37c5 --- /dev/null +++ b/.agents/phases/todo/189_chat_save_long_turns/04_e2e_and_gates.md @@ -0,0 +1,57 @@ +# Task 04 — The E2E suite, the full gates, the docs pass, the commit + +**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16 +**Story:** n/a + +## Objective +Pin the owner-visible contract of the whole phase through the real UI: the incident regression at the wire (a 22-source brain record saves instead of 422ing), a mock turn's done-time auto-save landing, the overlap re-run (a dropped save lands after the in-flight upsert settles), and share-saves-first (a Share with the auto-saves still in flight publishes the BRAIN message, not just the question) — then the full gate, the docs pass, and the atomic commit. + +## Work +1. `tests/e2e/test_chat_save_long_turns.py` — the phase's dedicated Playwright suite (the house `conftest` app + `mock_llm.py` pattern, run **in isolation** per AGENTS rule 4; a fresh saved-chat row per test — the house `saved_chats` cleanup helper pattern from `test_chat_history.py`): + - `test_incident_payload_shape_saves_at_the_wire`: through `page.request` (the `test_chat_save_payload_limits.py` pattern — the app's own origin), an anonymous `POST /api/chats` with the INCIDENT shape (a user message + a brain record carrying 22 `sources` (`SourceRef` dicts at realistic column-length values), 33 `tools` (`ToolCall` dicts), 51 `segments` = 33 `{kind: "tool", …}` + 18 `{kind: "thinking", text: …}`) → **201**; a follow-up `PUT` replacing the brain record with the same shape → **200** and the row's `messages` round-trips all three lists (a `GET /api/chats/` admin assertion). (Pre-fix: both 422 — the incident.) + - `test_done_time_auto_save_lands_for_a_mock_turn`: the real UI — send a mock question (the `mock_llm.py` no-tools flow, the `test_chat_history.py` send pattern), wait for the done bubble (the house `wait_for_answer` helper), then `GET /api/chats/` (the row id from the created-response `#…` — the house pattern reads `currentChatId` via `page.evaluate`): the row has **2** messages and the brain record's `text` equals the mock answer. + - `test_dropped_done_time_save_reruns_after_the_inflight_upsert_settles`: `page.route`-delay the FIRST PUT to `/api/chats/**` (the send-time auto-save) by 3 000 ms (a `page.route("**/api/chats/**", handler)` that resolves on `route.fulfill` after a `wait_for_timeout` ONLY for the first matched PUT — a module-level counter; later PUTs pass through untouched), then send the mock question and let the mock turn settle (the done bubble appears in flight — the mock is fast, the upsert is still parked at 3 s). Wait to ~5 s; `GET /api/chats/`: the row has **2** messages (the dropped done-time save re-ran after the parked upsert settled — pre-fix: the row held 1). + - `test_share_saves_first_publishes_the_brain_message`: the same 3 s delay on the first PUT; send the mock question; while the send-time save is still parked (the done bubble has appeared), click `#share-chat-btn` (the house click helper, wait for the toast). The share's own upsert PUT passes the route filter UNdelayed (the counter already consumed by the parked send-time save), mints the token, and the toast lands; open `share_url` (the `test_share_chat.py` pattern — the `/shared/` page, the `#shared-title` visible) and assert the shared page renders the BRAIN answer text (a `.msg .bubble` containing the mock answer string) — pre-fix: the shared page showed only the question. Assert the row (via admin `GET`) has 2 messages. + - Module docstring: the incident's evidence summary (the 422 PUT log line, the 22-source done frame, `created_at == updated_at` single-write row) so a future reader sees WHY these four tests exist. +2. The regression suites (green UNEDITED — do not touch them): `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — the text caps are unchanged), `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py`. +3. The full gate, in order: `uv run pytest -v` (unit + integration 100% green); `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/`); `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` (**in isolation**); `uv run ruff check . && uv run pyright` clean. +4. The docs pass (the phase's Documentation & Consistency mandate): review `README.md` and `AGENTS.md` in their entirety against the post-phase app; fix every inconsistency found, pre-existing ones included. `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap — 200/100/450), a Share saves the current conversation before minting the token (a failed share upsert aborts — no link to a truncated snapshot), a failed auto-save stays visible until the next success and retries once after 5 s on a transient failure (a 4xx never retries). `AGENTS.md`: verify the phase-55 save contract is not described anywhere in it (the save points are not enumerated — the UI-Structure rule's "in-flight turn" clause is untouched); if the final pass finds any stale save-contract sentence, fix it in place (no new rule added). +5. The atomic commit (AGENTS rule 8; the git-commit skill's `--no-gpg-sign` + `LLM-Generated: true` trailer): `git add -A && git commit --no-gpg-sign -m "fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run + +The done-time auto-save of a long turn (production BOR_AGENT_MAX_ROUNDS=50; +the incident turn cited 22 docs, 33 tool calls, 34 rounds) 422'd at the +phase-83 list caps sized for the round-cap-10 era (sources ≤ 20, tools ≤ 50, +segments ≤ 50), and the phase-55 quiet-failure swallowed it: nothing retried, +so the row kept its pre-turn state — and a Share minted a token on that stale +row, publishing a conversation without its answer (incident 2026-10-16, +shared 0101d2ca… / row 8b345ab5…). + +- A1: ChatMessage list caps follow the real configuration — sources/related/ + tools 200 (4× the production round cap; the cited tier is the read docs, + merge_cited_docs), suggestions 100, segments 450; every other bound and the + 8 M total budget unchanged (SEC-05 boundedness survives). +- A2: a linked Share upserts the current conversation before minting the + token and aborts on an upsert failure — never a link to a truncated + snapshot. +- A3: a save point dropped by the in-flight guard re-runs once after the + upsert settles. +- A4: a failed save's note persists until the next success; one 5 s retry on + network/5xx (a 4xx never retries). + +LLM-Generated: true"`. +6. Move the phase directory: `mv .agents/phases/todo/189_chat_save_long_turns .agents/phases/complete/` and `git add .agents && git commit --no-gpg-sign -m "chore(agents): complete phase 189 (chat save long turns)"` — OR fold the phase move into the atomic commit per the house convention (check the last 3 phase commits' shape and match it; the AGENTS rule 8 says one atomic commit per phase including the phase files — if the house convention is one commit, `git add` the phase dir into step 5's commit and skip step 6's commit, keeping the `mv`). + +## Testing & Quality +- E2E: the four scenarios above, in isolation. +- Regression: the three listed suites green unedited. +- Coverage / lint / types: the full gate (step 3). + +## Completion Criteria +- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** — the incident shape 201/200 at the wire, the mock turn's done-time save lands, the dropped save re-runs (final row 2 messages), the share with in-flight auto-saves publishes the brain message on the shared page. +- [ ] `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green UNEDITED. +- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean. +- [ ] README.md / AGENTS.md reviewed in full; the saved-chat ops note in place; no stale save-contract sentence left. +- [ ] `git diff --stat` (before the commit) limited to `app/schemas.py`, `frontend/assets/app.js`, `tests/unit/test_schemas.py`, `tests/unit/test_save_chat_ui.py`, `tests/integration/test_chats_api.py`, `tests/e2e/test_chat_save_long_turns.py`, `README.md`, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency). +- [ ] the atomic `--no-gpg-sign` commit exists (message above); the phase dir is in `.agents/phases/complete/`. +- [ ] full test suite **100% passing** (every test in the project, not just this task's) +- [ ] no behavior change in completed phases (the SSE `done` event, the `share_chat` endpoint, the unlinked share branch, the phase-55 A2 contract's shape — all unchanged) diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.err b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md new file mode 100644 index 0000000..4c5a35f --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md @@ -0,0 +1,25 @@ +**Phase 187 — final verification pass: all criteria met, repo 100% green.** + +Verified (all four tasks already in `complete/`) +- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests. +- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical. +- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403. +- `sources.js` `fmtSyncPeakClause`: ` · peak MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region. + +Gates (exact commands) +- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%** +- `uv run ruff check .` → clean · `uv run pyright` → **0 errors** +- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice) +- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …) + +Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness. + +Documentation +- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed. +- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`). + +Notable +- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made. +- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched. + +Next pending phase: `188_chunks_hnsw_index`. diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md.status b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.validate b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.validate new file mode 100644 index 0000000..1cec5c9 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__00_phase.a1.validate @@ -0,0 +1,145 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 89%] +........................................................................ [ 91%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 97%] +........................................................................ [ 99%] +.................................. [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/admin_settings.py 114 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 292 1 99% +app/api/chat_images.py 52 0 100% +app/api/chats.py 111 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 239 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 231 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 318 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 43 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/memory.py 32 0 100% +app/core/rate_limit.py 44 0 100% +app/core/response_budget.py 63 0 100% +app/core/security_headers.py 20 0 100% +app/core/settings_catalog.py 18 0 100% +app/core/settings_migrate.py 32 0 100% +app/core/settings_overlay.py 111 0 100% +app/core/theming.py 40 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 105 0 100% +app/models.py 141 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 494 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 865 3 99% +app/rag/llm.py 253 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 162 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/rag/svg_text.py 111 0 100% +app/rag/verbatim.py 160 0 100% +app/schemas.py 549 0 100% +-------------------------------------------------- +TOTAL 6932 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.err b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md new file mode 100644 index 0000000..acdb140 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md @@ -0,0 +1,16 @@ +**Report — phase 187 / task 01 (`app/core/memory.py`)** + +- **Implemented:** `app/core/memory.py` — pure-stdlib (no `app.` imports), mirroring `response_budget.py`'s shape: + - `rss_mb()` — parses `VmRSS` (kB→MB floor) from `/proc/self/status`; any `OSError`/`KeyError`/`ValueError`/`IndexError` or missing line falls back to the `getrusage` high-water proxy; `None` only when both are unavailable. + - `rss_peak_mb()` — `ru_maxrss // 1024`, guarded `import resource` (`None` on platforms without it), never-raise wrapper around `getrusage`. + - `cgroup_peak_mb()` — `/sys/fs/cgroup/memory.peak` bytes→MB, `None` otherwise; docstring carries the one-time container/cgroup caveat (`file` vs `anon`, systemd `3.3G` anecdote, LOCKED A1). + - `fmt_mb()` — `"?"` for `None`, `"0"` for 0; module docstring documents the incident, helpers, never-raise contract, "Pure stdlib" line. +- **Tests:** `tests/unit/test_core_memory.py` — 24 tests: real Linux `/proc` read (fallback pinned to a sentinel to prove the parse path), kB→MB floor, missing/unparseable/value-less/no-`VmRSS`-file fallbacks, resource stubs, guarded-import branch covered via `importlib.reload` with `sys.modules["resource"]=None` (no pragma — house style), cgroup present/absent/garbage, `fmt_mb`, AST purity pin. +- **Gates (all green):** + - `uv run pytest tests/unit/test_core_memory.py -v` → 24 passed; `--cov=app.core.memory --cov-branch` → **100%** (0 missed stmts/branches). + - `uv run pytest --cov=app --cov-report=term-missing` → **4555 passed**, `app/` TOTAL **99%** (>90% gate). + - `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. + - `uv run pytest tests/e2e/test_smoke.py --no-cov` → 3 passed (sanity). +- **Docs:** no doc impact — nothing user-visible yet (log lines / status keys / README grep land in tasks 02–03, docs pass in task 04). +- **Decisions:** guarded `resource` import typed as `ModuleType | None`; path constants (`PROC_STATUS_PATH`, `CGROUP_PEAK_PATH`) patched via `monkeypatch` instead of global `open` mocks; `KeyError` kept in the suppress list per spec with a docstring note that the loop form can't raise it. +- **Next pending task:** `187_sync_peak_memory_line/02_sync_log_lines.md`. diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md.status b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.validate b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.validate new file mode 100644 index 0000000..c3c05ad --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__01_memory_helper.a1.validate @@ -0,0 +1,145 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 41%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 49%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 60%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 79%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 98%] +........................................................................ [ 99%] +................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/admin_settings.py 114 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 292 1 99% +app/api/chat_images.py 52 0 100% +app/api/chats.py 111 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 239 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 204 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 318 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 43 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/memory.py 32 0 100% +app/core/rate_limit.py 44 0 100% +app/core/response_budget.py 63 0 100% +app/core/security_headers.py 20 0 100% +app/core/settings_catalog.py 18 0 100% +app/core/settings_migrate.py 32 0 100% +app/core/settings_overlay.py 111 0 100% +app/core/theming.py 40 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 105 0 100% +app/models.py 141 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 494 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 864 3 99% +app/rag/llm.py 253 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 162 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/rag/svg_text.py 111 0 100% +app/rag/verbatim.py 160 0 100% +app/schemas.py 549 0 100% +-------------------------------------------------- +TOTAL 6904 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.err b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md new file mode 100644 index 0000000..6aaacf0 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md @@ -0,0 +1,16 @@ +**Implemented (task 02 — `187_sync_peak_memory_line`)** +- `app/api/sync.py`: `sync: started … rss_mb=N`; `sync: done detail=… rss_mb=N rss_peak_mb=M`; `sync: cancelled by user … rss_mb=N rss_peak_mb=M`; failed funnel keeps `logger.exception` (traceback) and gains `logger.error("sync: failed state=failed files_done=%d files_total=%d rss_mb=%s rss_peak_mb=%s error=%s")` using the **already-sanitized** `_status.error`. +- `app/rag/importer.py`: `ImportSummary.log()` line gains trailing ` rss_mb=%s`; `format_counts()` untouched (CLI consumers byte-identical). +- All rendering single-sourced through `app.core.memory.fmt_mb` (`?` for unknown); strictly append-only (LOCKED A2) — module docstrings updated. +- Tests: new `test_success_lifecycle_lines_carry_memory_fields`; pins in `test_sync_api.py` (started/done/cancel/failed prefixes byte-identical + `rss_mb=(\?|int)` tails, traceback survives, `error=` masked), `test_importer_e2e.py` (both runs end in ` rss_mb=`, earlier fields intact), `tests/unit/test_importer.py` (line prefix pin + `formats=` no longer last), `tests/unit/test_sync_button.py` (cancel-line fullmatch). +- Also documented a real nuance found while verifying: on Linux `rss_mb` can print ~1 MB **above** `rss_peak_mb` (kernel high-water accounting + MB flooring) — noted in `app/core/memory.py` + README so nobody "fixes" a correct log line. + +**Gates (all 100% green)** +- `uv run pytest` → **4556 passed** +- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%**, `app/api/sync.py` 100%, `app/core/memory.py` 100% (gate >90%) +- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors +- Regression insurance: `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov` → 8 passed (no UI touched; phase E2E belongs to task 04) + +**Docs**: `README.md` — new “Operational notes: the sync’s memory (phase 187)” (greps, field meanings, process-vs-cgroup/page-cache caveat, append-only note) + a bullet in *Sync from the UI*. `AGENTS.md` rule 10 — parenthetical extended with the sync memory fields + `grep ' :: ' | grep ' rss_peak_mb='`. `PLAN.md` §9 (`import: summary … formats=…`) now trails `rss_mb=` — **flagged for the owner to reconcile, not edited**. + +**Next pending task**: `.agents/phases/todo/187_sync_peak_memory_line/03_status_key_and_ui.md` (`SyncStatus` fields + sampling, `rss_mb`/`rss_peak_mb` on `/api/sync/status`, `sources.js` ` · peak MB` clause). diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md.status b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.validate b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.validate new file mode 100644 index 0000000..db85b65 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__02_sync_log_lines.a1.validate @@ -0,0 +1,145 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 30%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 41%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 60%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 71%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 79%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 90%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 97%] +........................................................................ [ 99%] +.................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/admin_settings.py 114 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 292 1 99% +app/api/chat_images.py 52 0 100% +app/api/chats.py 111 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 239 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 206 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 318 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 43 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/memory.py 32 0 100% +app/core/rate_limit.py 44 0 100% +app/core/response_budget.py 63 0 100% +app/core/security_headers.py 20 0 100% +app/core/settings_catalog.py 18 0 100% +app/core/settings_migrate.py 32 0 100% +app/core/settings_overlay.py 111 0 100% +app/core/theming.py 40 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 105 0 100% +app/models.py 141 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 494 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 865 3 99% +app/rag/llm.py 253 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 162 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/rag/svg_text.py 111 0 100% +app/rag/verbatim.py 160 0 100% +app/schemas.py 549 0 100% +-------------------------------------------------- +TOTAL 6907 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.err b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md new file mode 100644 index 0000000..6abacdf --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md @@ -0,0 +1,14 @@ +All gates are green. Final state: + +**Backend** (`app/api/sync.py`) +- `SyncStatus.rss_mb` / `rss_peak_mb` (`int | None`, default `None`, reset with each run) written only through the ONE sampler `_sample_rss(force=False)`: `time.monotonic()` guard makes an unforced call a no-op inside `_RSS_SAMPLE_INTERVAL_S = 5.0`; forced at run start, all three `phase` transitions and every terminal, throttled in the per-file hook. +- `GET /api/sync/status` returns both keys additively (null = "no reading", never 0); success / cancelled / failed terminals log exactly the pair they report, so `sync: done …` and the payload are one instant. + +**Frontend** (`frontend/assets/sources.js`) +- ONE builder `fmtSyncPeakClause(status)` (`status.rss_peak_mb ?? status.rss_mb`, non-finite/negative → `""`), appended by `setSyncProgress` to the two result strings that already exist (`fmtSyncResult(detail)`, `"Sync cancelled"`) — no new control/element/live region, no clause in running labels, `applySyncFailure`, `applySyncIdle` or the upload settles. + +**Tests** — unit: `_sample_rss` sampling/throttle/force/null, start+phase+terminal sampling, terminal log==payload, key-set pin (old ∪ {`rss_mb`,`rss_peak_mb`}), idle pin, source pins + Node render of the clause. Integration: additive-keys/null-before-run, success + failed terminal pairs (`peak >= rss` on Linux), cancelled pair, anonymous `403` guard. E2E: added `tests/e2e/sync_helpers.py` (`PEAK_CLAUSE`, `counts_line()`, `strip_peak_clause()`) and updated 27 sync-facing suites that asserted exact result-line equality — **documented exact-string update**: the clause is optional trailing text there, and `test_sync_session_reset.py` strips it before its cross-app byte-identity comparison. + +**Gates** — `uv run pytest --cov=app` → EXIT 0, TOTAL 99%, `app/api/sync.py` 100%, `app/core/memory.py` 100%; `ruff check .` clean, `pyright` 0 errors; E2E in isolation (`--no-cov`): `test_sync_summary_visibility` 3 passed, `test_sync_progress_box` 8 passed, `test_sync_status_freshness` 4 passed, `test_sync_button` 3 passed, and the other 23 patched suites all EXIT 0 (390px layout pin still holds in `test_sync_label_mobile`). + +**Docs** — `README.md` (status keys, the ` · peak N MB` clause, sampling/throttle, null renders nothing) and `AGENTS.md` rule 10 + rule 5 UI-structure clause. No git index/history changes; `03_status_key_and_ui.md` left in `todo/` for the harness. diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md.status b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.validate b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.validate new file mode 100644 index 0000000..02e4066 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__03_status_key_and_ui.a1.validate @@ -0,0 +1,145 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 89%] +........................................................................ [ 91%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 97%] +........................................................................ [ 99%] +................................ [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/admin_settings.py 114 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 292 1 99% +app/api/chat_images.py 52 0 100% +app/api/chats.py 111 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 239 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 227 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 318 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 43 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/memory.py 32 0 100% +app/core/rate_limit.py 44 0 100% +app/core/response_budget.py 63 0 100% +app/core/security_headers.py 20 0 100% +app/core/settings_catalog.py 18 0 100% +app/core/settings_migrate.py 32 0 100% +app/core/settings_overlay.py 111 0 100% +app/core/theming.py 40 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 105 0 100% +app/models.py 141 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 494 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 865 3 99% +app/rag/llm.py 253 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 162 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/rag/svg_text.py 111 0 100% +app/rag/verbatim.py 160 0 100% +app/schemas.py 549 0 100% +-------------------------------------------------- +TOTAL 6928 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.err b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md new file mode 100644 index 0000000..35b8bea --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md @@ -0,0 +1,18 @@ +All gates are green. Final state: + +**Implemented (task 04)** +- `tests/e2e/test_sync_peak_memory_visible.py` — new mock-only module (own app port 8204 / slow-LLM proxy 8910, dedicated `BOR_SOURCES_DIR`, DB truncated before **and** after each test, admin form login, JS console/page-error guard). 5 tests: success result line `2 added · peak N MB` (wording intact, inside the sole `role=status`/`aria-live` region, no new element, DOM number == payload `rss_peak_mb`, survives reload); payload additive (pre-187 key set + typed `rss_mb`/`rss_peak_mb`, `peak >= current`, kept in the terminal, **no clause while running**); anonymous `GET /api/sync/status` → `403 {"detail":"admin only"}` + no memory text in the shell; cancelled terminal stays neutral `Sync cancelled · peak N MB`; clause fits 375 px with zero horizontal overflow. +- Fixture renamed to `memline-fixture/notes/mem-*.md` + a documented naming rule: nothing on the page may contain "peak" except the clause, so the "no clause while running" pin can't be satisfied by a file name. + +**Defect fixed (found by this phase's gates, in task 03's code)** `tests/integration/test_sync_api.py` intermittently failed: `ru_maxrss` refreshes lazily, so a raw peak could sit **below** the current reading (`164` vs `162`). `app/api/sync.py::_sample_rss` (the ONE sampling point) now lifts a lagging peak to the current reading — never fabricating an unknown one — so every surface satisfies `peak >= current`; slack in the integration/E2E pins went 1 MB → 0, two new unit tests pin both branches. + +**Gates** `pytest --cov=app`: **4570 passed**, `TOTAL 99%`, `app/api/sync.py` and `app/core/memory.py` **100%** · `ruff check .` clean · `pyright` 0 errors · target E2E **5 passed in isolation** (+2 repeat runs) · `test_sync_progress_box`, `test_sync_status_freshness`, `test_sync_cancel`, `test_sync_summary_visibility`, `test_sync_button` green in isolation · mutation-checked (drop success clause → tests 1&5 fail; drop cancel clause → test 4 fails; leak clause into the running label → all 5 fail), `sources.js` verified byte-identical to the pre-mutation state. + +**Docs** `README.md` (sync bullet, "States" settle line, *Operational notes: the sync's memory* invariant) and `AGENTS.md` rule 10 now state the `peak >= current` invariant, the terminal-only clause and the new suite; fixed pre-existing page-naming drift (3× "Sources page" → "RAG page (`/sources.html`)"); README/AGENTS path + `BOR_*` default checks report 0 issues. + +**Deviations / flags** +1. Per the harness protocol I did **not** commit and did **not** move `todo/187_sync_peak_memory_line/` → `complete/` (tasks 01–03 are already in `complete/`; `00_phase.md` + `04_e2e_peak_memory.md` remain). Everything is staged-free in the working tree for the harness's single commit — suggested message: `feat(observability): report the sync's RSS and peak memory in the log, the status payload and the result line`. +2. **PLAN §9 reconciliation (owner action, PLAN.md untouched):** §9 still lists only the per-turn line and the `import: summary` line; phase 187 adds the sync lifecycle memory fields (`rss_mb` on `sync: started` + summary, `rss_mb`/`rss_peak_mb` on all three terminals, plus the two additive `/api/sync/status` keys). §9 needs a matching bullet, or AGENTS rule 10 stays the only place the sync lines are specified. +3. **Pre-existing, not from this phase:** running an E2E module and `tests/unit/test_sync_button.py` in the *same* pytest invocation fails (`asyncio.run() cannot be called from a running event loop`) because pytest-playwright's session-scoped loop leaves the main thread marked as running; reproduced with HEAD's unmodified unit file + `test_sync_cancel.py`. The documented gates never mix those paths (`testpaths` excludes `tests/e2e`), so I left it alone rather than touch shared fixtures. + +**Next pending task:** `.agents/phases/todo/188_chunks_hnsw_index/01_hnsw_migration.md`. diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md.status b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.validate b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.validate new file mode 100644 index 0000000..1cec5c9 --- /dev/null +++ b/.agents/reports/187_sync_peak_memory_line/187_sync_peak_memory_line__04_e2e_peak_memory.a1.validate @@ -0,0 +1,145 @@ +........................................................................ [ 1%] +........................................................................ [ 3%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 7%] +........................................................................ [ 9%] +........................................................................ [ 11%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 15%] +........................................................................ [ 17%] +........................................................................ [ 18%] +........................................................................ [ 20%] +........................................................................ [ 22%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 26%] +........................................................................ [ 28%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 37%] +........................................................................ [ 39%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 45%] +........................................................................ [ 47%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 56%] +........................................................................ [ 58%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 64%] +........................................................................ [ 66%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 75%] +........................................................................ [ 77%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 81%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 89%] +........................................................................ [ 91%] +........................................................................ [ 92%] +........................................................................ [ 94%] +........................................................................ [ 96%] +........................................................................ [ 97%] +........................................................................ [ 99%] +.................................. [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/admin_settings.py 114 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 292 1 99% +app/api/chat_images.py 52 0 100% +app/api/chats.py 111 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 239 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 231 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 318 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/compression.py 59 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 43 0 100% +app/core/errors.py 5 0 100% +app/core/http_cache.py 22 0 100% +app/core/logging.py 13 0 100% +app/core/memory.py 32 0 100% +app/core/rate_limit.py 44 0 100% +app/core/response_budget.py 63 0 100% +app/core/security_headers.py 20 0 100% +app/core/settings_catalog.py 18 0 100% +app/core/settings_migrate.py 32 0 100% +app/core/settings_overlay.py 111 0 100% +app/core/theming.py 40 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 105 0 100% +app/models.py 141 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 494 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 865 3 99% +app/rag/llm.py 253 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 78 0 100% +app/rag/pdf.py 192 0 100% +app/rag/prompts.py 162 0 100% +app/rag/retriever.py 252 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/rag/svg_text.py 111 0 100% +app/rag/verbatim.py 160 0 100% +app/schemas.py 549 0 100% +-------------------------------------------------- +TOTAL 6932 16 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/AGENTS.md b/AGENTS.md index e1cbdb9..c508703 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -71,14 +71,23 @@ settle it), the Copy button's clipboard content equal to the server string exactly, and the shared page's panel Copy button + plain `/document.html` link the ONE owner-directed exception to that page's - zero-controls rule) — phase 182 task 07) and - meets WCAG 2.1 AA basics (semantic + zero-controls rule) — phase 182 task 07), and the sync's peak-memory + clause is TRAILING TEXT on the result line that already exists + (` · peak MB` on the RAG page's `#sync-progress` counts line + or "Sync cancelled", written by the ONE `fmtSyncPeakClause` off + `rss_peak_mb ?? rss_mb` — no new control, no new live region, no new + element, no clause in a running label or in the failed terminal's + empty box, and NOTHING rendered when the pair is null — phase 187 + task 03) and meets WCAG 2.1 AA basics (semantic landmarks, labels, contrast ≥4.5:1, focus-visible, aria-live for the stream). 6. **"No CDN Rule":** all CSS, JS, fonts, and images must be served statically from within the FastAPI application. No `