Compare commits

...
18 Commits
Author SHA1 Message Date
ducoterra a45f744604 phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
2026-10-20 06:36:21 -04:00
ducoterra ae080ac7f3 phase: 193_sync_remaining_count
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Successful in 16s
**Phase 193 — final verification pass: all gates green.**

- **Verified (all 3 tasks already complete in `complete/193_sync_remaining_count/`):** `fmtSyncLabel`'s count part is `total > 0 && total - done > 0 ? \`${total - done} remaining\` : ""` (plain int, hidden at `N==0`, `total===0` → no part); docstrings updated; chunk/image/summaries parts + `counts_line` terminal + `app/` byte-untouched (`git diff --stat`: no `app/` files).
- **Completion criteria:** `N remaining` shape + marker pins ✅ · upload shares the builder (E2E upload arm) ✅ · chunk/image/summaries pins green ✅ · terminal `counts_line` green unedited + A4 title/`#sync-progress` pin ✅ · suite green, coverage 99% ✅ · lint/types clean ✅ · commit + `todo/→complete/` left to the harness (no git index/history commands run).
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, junit: **4792 tests, 0 failures/errors/skipped**, `TOTAL 7181 16 → 99%` (>90%)
- `uv run pytest tests/e2e/test_sync_remaining_count.py -v --no-cov` → **3 passed** (isolated, 53 s)
- Isolated `--no-cov` sweep of the 14 sync-lineage suites (label_mobile, upload_progress, chunk_progress, image_progress, progress_box, summary_visibility, archive_upload_sources, status_freshness, upload_no_scan, peak_memory_visible, session_reset, fast_path_deferred_content, git_freeze, smoke) → **14/14 exit 0**
- `uv run ruff check .` → All checks passed · `uv run pyright` → **0 errors**
- **Docs:** README.md reviewed in full (all 2408 lines) + AGENTS.md. Fixed 2 inconsistencies: the sync running-label clause claimed the upload label shows the count — now states it shares the ONE builder/shape but reports no counts since phase 90 (renders bare); phase-190's "the other ~72 rows" → "the other 83 rows of the panel's 86-row catalogue" (mechanically: 86 CATALOG rows, 84 overridable). `N remaining` already in the sync paragraph; task-01's `(n/m)` examples gone. AGENTS.md verified consistent (no clause names the running count part → unchanged per the phase's "do not add if absent"). Mechanically re-verified: all 86 `BOR_*` settings documented + defaults match, `.env.example` 88 vars covered, 0 broken README anchors, all cited file paths exist, 17 palette colors, migrations head `0030`, pages/shell routes, ports 8123/8901, E2E port allocations (new suite's 8211/8914 unique).
- **Notable:** task 02's documented deviation stands — `files_done` is the importer's 1-based *current* file, so the phase doc's `[1-4]` range is unreachable; count-bearing frames are `1 … N−1` (pinned). Pre-existing, untouched: `E2E_APP_PORT_TOOLBUDGET`/`SAVEDOC` both 8129 and `SYNCUP`/`SYCNCEL` both 8133 — harmless under the one-suite-at-a-time rule.
- **Next pending phase:** none — `193_sync_remaining_count` is the last entry in `.agents/phases/todo/`.
2026-10-19 17:30:03 -04:00
ducoterra fb17d15fbe phase: 192_verbatim_inline_midturn
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Phase 192 — final verification pass complete.

- **Verified (all 4 tasks already complete):** additive `verbatim_block` SSE frame (`VerbatimBlockPiece` → `ChatVerbatimBlockEvent` → chat dispatch, after the round's `tool`/`tool_result`, before the next round); `hydrateVerbatimNow` matched-only mid-turn pass + app.js branch; `dropPlacedTwin` per-locator done/restore dedupe; the dedicated E2E suite.
- **Root cause of the rejected attempt:** my own concurrent background `pytest` run deadlocked the harness validation run on the shared Postgres (`psycopg.errors.DeadlockDetected`). No code defect — re-ran every gate **exclusively**.
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, `TOTAL 7181 16 99%` (>90%)
- `uv run pytest tests/e2e/test_verbatim_inline_midturn.py --no-cov` → **6 passed** (isolated)
- `uv run pytest tests/e2e/test_verbatim_show_text.py tests/e2e/test_chat_history.py --no-cov` → **18 passed** (`test_chat_history.py` byte-unedited; the phase-182 suite's "no eighth type" pin was tightened, not loosened, to seven + `verbatim_block` with frame-order asserts)
- `uv run ruff check .` → All checks passed (exit 0) · `uv run pyright` → **0 errors**
- **Completion criteria:** single frame per success + zero per refusal ✅ · `BOR_ANSWER_TEXT=0` wire byte-identical (integration spelled baseline + E2E App B) ✅ · fence hydrates in place / unfenced lands at the call moment, before `done` ✅ · append+fence → 1 panel, two ranges → 2 panels ✅ · restore 1 panel/locator, `done.answer_text` + record key + `verbatim_blocks=N` unchanged ✅ · suite green, coverage 99%, lint/types clean ✅ · no-show_text turn byte-identical ✅ · commit → left to the harness (no git index/history commands run).
- **Docs:** README reviewed line-by-line (2399 lines) + AGENTS.md in full; mechanically verified all 86 `BOR_*` settings, every `.env.example` var, table defaults, internal anchors (0 broken), ports 8123/8901, tree depth 1–8, pagination 25/1–100, 17 palette colors, `0029`/`0030` migrations, and every cited test file. Fixed 1 inconsistency: the verbatim "Parity" bullet claimed the live turn/reload/shared page share "the same three globals" — now names the settle-path trio and states that `hydrateVerbatimNow` is the ONE chat-only addition (a shared page has no stream).
- **Notable:** `run_agent`'s four repeated piece unions were collapsed into the ONE exported `AgentPiece` alias, so a future piece type is spelled in one place; `test_frontend_edit_message.py`'s `runTurn` byte-pin was regenerated, not weakened.
- **Next pending phase:** `193_sync_remaining_count`.
2026-10-19 12:19:51 -04:00
ducoterra 4f20c256d7 phase: 191_source_name_labels_tags
Build and Push Containers / build-and-push-app (push) Successful in 7m24s
Build and Push Containers / build-and-push-db (push) Successful in 36s
All gates re-verified against a stable tree. Report:

**Verified (all 6 tasks in `complete/`; this was the final pass)**
- `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`.
- **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned).
- **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code.
- **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria.
- **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
2026-10-19 06:13:32 -04:00
ducoterra c6ffa050a6 phase: 190_app_name_settings_fix
Build and Push Containers / build-and-push-app (push) Successful in 2m9s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates re-verified green. Report:

- **Verified (all 4 tasks already complete; this was the final pass):** `effective_settings` resolves the 3 brand strings panel-row → tab-row → env (`applied_override_keys`, presence-not-value); `PUT /api/ui-settings` ignores string body fields; Theme-tab form is colors-only (17 pickers, no `#theme-app-name`/`#theme-placeholder`/`#theme-footer`); the ONE shared `applyBrandFromConfig` re-applies title/`.brand-text`/footer/placeholder after a panel save-reset and a tab save-reset; boot path byte-identical (single `/api/config`).
- **Completion criteria:**
  - strings order panel→tab→env, both branches — ✅ `tests/unit/test_theming.py::test_effective_app_name_prefers_settings_panel_row_over_ui_settings_row`, `..._fall_back_to_ui_settings_row_without_panel_row`, `..._per_field_panel_then_tab_then_env`, `..._ignore_panel_rows_when_the_overlay_is_off`
  - PUT ignores string fields + no 3 inputs in the tab — ✅ `tests/integration/test_ui_settings_api.py::test_put_ignores_string_fields_in_body` (+2) and the E2E `test_theme_tab_dom_is_colors_only`
  - live apply without reload + incident regression — ✅ `tests/e2e/test_app_name_settings_fix.py` tests 1–4
  - byte-identical boot, no double-apply — ✅ `tests/unit/test_brand_live.py` markers + E2E "exactly one /api/config"
  - suite green, coverage >90%, ruff+pyright clean — ✅
  - no behavior change in prior phases — ✅ (palette, phase-168 panel, env fallback suites all green)
  - atomic commit — **left to the harness** (pipeline rule overrides per-phase committing; no git write run)
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **4630 passed, TOTAL 6941 16 99%**, exit 0 · `uv run pytest tests/e2e/test_app_name_settings_fix.py -v --no-cov` → **6 passed** in isolation · regression E2E (isolated, one process each, all exit 0): `test_smoke` `test_configurable_brand` `test_admin_theme_tab` `test_theme_save_and_coverage` `test_theme_semantic_completion` `test_themed_favicon` `test_dark_tech_theme` `test_admin_settings_panel` `test_admin_settings_layout` `test_settings_mobile_value_width` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Documentation:** README — fixed "saving one repaints *every* open page" (only the open document repaints; other pages on next load) and a broken paragraph wrap in *Customizing the look*; AGENTS.md rule 16 — `test_chunks_hnsw_recall.py` now described accurately (index paths forced with `enable_seqscan`/`indexonlyscan`/`bitmapscan` off, unforced plan asserted only on a pristine relation); `tests/unit/test_theme_frontend.py` — two stale section headers ("20 form fields"/"20 inputs" → 17 color). No AGENTS.md settings-overlay/resolution-order clause exists, so per the phase none was added. Full env-var cross-check (`BOR_*` in `app/config.py` vs README) → 0 gaps; 0 missing file references.
- **Notable:** the working tree also carries this phase's flake hardening of the phase-188 `tests/integration/test_chunks_hnsw_recall.py` (verified legit and green). Live infra untouched; shared DB left clean (0 `app_settings`, 0 `ui_settings`). **Flagged, not changed:** `settings_overlay.applied_override_keys` lets a `ProgrammingError` (missing `app_settings`) propagate from public `GET /api/config`, where `load_overrides` deliberately fails soft — only reachable on an un-migrated (0027) DB, outside this phase's pinned A1 semantics, so left for the owner to rule on rather than silently deviating.
- **Next pending phase:** `191_source_name_labels_tags`.
2026-10-18 02:59:19 -04:00
ducoterra 47456a44df chore(agent): phase roadmap from TODO.md, 4 phases (190-193)
Convert the four open TODO items into executable phases (append mode,
starting at 190 after the completed 189):

  190_app_name_settings_fix  - the Settings-panel app name wins over the
      Theme tab; the three brand strings stripped from the Theme tab
      (colors-only); live brand re-apply on save, no reload
  191_source_name_labels_tags - source_labels table + admin API + Sources
      page editor; the separate bounded source_name_hit_bonus retrieval
      boost; label surfacing in the <document> blocks and KB overview
  192_verbatim_inline_midturn - an additive verbatim_block SSE frame at
      the show_text success moment; in-place mid-turn fence hydration;
      the done/restore per-locator dedupe; the done frame, record key,
      restore and shared page stay byte-identical
  193_sync_remaining_count - the running sync/upload label counts down:
      "N remaining" (N = total - done), hidden at the final file; the
      chunk/image/summaries/terminal parts untouched

TODO.md is cleared (its items now live in .agents/phases/todo/).

LLM-Generated: true
2026-10-17 19:35:55 -04:00
ducoterra 140a343b92 phase: 189_chat_save_long_turns
Build and Push Containers / build-and-push-app (push) Successful in 3m4s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 189 final verification — all green**

- **Verified (all 4 tasks already implemented):** A1 caps in `app/schemas.py` — `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; `text`/`thinking` 600 000, `error`/`image` 500, `answer_text` 50, `messages` 500, 8 M budget all UNCHANGED; rationale docstrings rewritten to `merge_cited_docs`/`agent_max_rounds`. A2 `shareCurrentChat` linked branch upserts (PUT, 404→create+re-link) before `POST /{id}/share`, aborts on failure (banner, no token); unlinked branch byte-identical. A3 `savePending` + `finally` re-run. A4 `saveFailedSticky` + one 5 s `setTimeout` retry on network/5xx only, 4xx never retries.
- `uv run pytest --cov=app --cov-report=term-missing` → **4609 passed, 1 warning, EXIT=0**, `TOTAL 99%`, `app/schemas.py 100%` (>90%).
- `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` → **4 passed, EXIT=0** (isolation).
- Regression green & **unedited** (`git status` clean): `test_chat_save_payload_limits.py` 2, `test_chat_history.py` 5, `test_share_chat.py` 4. Extra blast-radius sweep (share/save/persist/segments surface, all EXIT=0): `test_save_share_ux`, `test_stale_saved_chats`, `test_history_pagination`, `test_history_copy`, `test_bottom_chat_actions`, `test_reasoning_bubble`, `test_tool_budget_telemetry`, `test_verbatim_show_text`, `test_smoke`. No leaked `saved_chats` rows (`save-long-%` = 0).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- **Completion criteria:** caps ✓ · incident payload 22/33/51 → 201/200 anonymously + 201 tools/451 segments/101 suggestions → 422 naming the offender, row unchanged ✓ · share-saves-first + abort + unlinked byte-identical ✓ · overlap re-run + sticky note + one-shot retry ✓ · phase E2E in isolation + 3 regression suites unedited ✓ · full suite / coverage / lint / types ✓ · `git diff --stat` = `app/schemas.py`, `frontend/assets/app.js`, 3 test files, `README.md` + new E2E & phase files (no `app/api/chats.py`, no migration, no new dependency) ✓ · commit + `todo/`→`complete/` move left to the harness (rules forbid git index writes).
- **Docs:** full README review (all 2 258 lines) + machine cross-check — all 85 settings documented in the Configuration reference, all 34 `.env.example` vars documented, documented defaults match `app/config.py`. Fixed 2 inconsistencies: (1) the 117-char prose line task 04 left when reflowing the Saved-chats pager paragraph; (2) the phase-189 sentence implying the 8 M total bounds the widened arrays — now states precisely that the total counts `text`/`thinking`/verbatim-block text only, and that the arrays are bounded by fixed item COUNT (matches `_require_within_total_budget`). AGENTS.md verified consistent — it enumerates no save contract or list cap (rule 5's phase-166 in-flight-turn clause untouched), so **no AGENTS.md change** per the phase mandate.
- **Notable/deviation:** the wider caps raise the *theoretical* anonymous-POST worst case (arrays are outside the 8 M char budget); left as designed (A1 fixes every other bound), documented honestly in README — realistic payloads stay ~10 MB, so the existing `client_max_body_size ≥ 16 MB` guidance still holds. No live-infra changes were made (data-repair of the two stale shared rows stays owner action).
- **Next pending phase:** none — `189_chat_save_long_turns` is the last entry in `.agents/phases/todo/`.
2026-10-17 07:36:28 -04:00
ducoterra f20d40e4fe phase: 188_chunks_hnsw_index
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
2026-10-17 03:18:23 -04:00
ducoterra c054b47df5 phase: 187_sync_peak_memory_line
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 187 — final verification pass: all criteria met, repo 100% green.**

Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.

Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)

Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.

Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).

Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.

Next pending phase: `188_chunks_hnsw_index`.
2026-10-16 15:22:51 -04:00
ducoterra 644f7de696 phase: 186_git_log_streaming_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m42s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Final report — phase 186 `git_log_streaming_dates`

**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.

### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |

Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.

### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.

Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.

⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).

Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
2026-10-16 07:50:16 -04:00
ducoterra 632fb88e43 phase: 185_sync_defer_content_fast_path
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All gates green. Report:

**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.

**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).

**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).

**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).

**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).

**Next pending phase**: `186_git_log_streaming_dates`.
2026-10-16 04:54:01 -04:00
ducoterra 065304b52f phase: 184_sync_bounded_session
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure.

**Verified / implemented (phase 184 final pass)**
- All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries.
- `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched.
- `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact.

**Tests / lint / coverage (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%).
- `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**.
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**.
- Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed.
- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria**
1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green
2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅
3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅
4. Every existing importer/sync/removal test green with unchanged expectations → ✅
5. Full suite green, coverage >90%, ruff + pyright clean → ✅
6. Phase E2E green in isolation → ✅
7. README + AGENTS accurate/consistent → ✅
8. Atomic commit — left to the harness (I ran no git write commands).

**Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written).

**Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched.

**Next pending phase:** `185_sync_defer_content_fast_path`.
2026-10-15 23:19:27 -04:00
ducoterra 7b5ea7c2ee phase: 183_sync_passive_delete_cascade
Build and Push Containers / build-and-push-app (push) Successful in 2m7s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green. Final state verified.

**Implemented / verified**
- Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after.
- Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings.
- **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`).

**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%)
- `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors

**Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands).

**Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit).

**Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify.

**Next pending phase**: `184_sync_bounded_session`.
2026-10-15 16:00:28 -04:00
ducoterra b34b8ca672 phase: 182_verbatim_show_text
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**

- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
2026-10-15 08:33:12 -04:00
ducoterra 814a400e6d phase: 181_svg_document_preview
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Successful in 15s
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`![alt](/api/documents/<id>/image)` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
2026-10-14 22:54:43 -04:00
ducoterra 6880bd0866 phase: 180_markdown_inline_links_images
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All verification complete. Final state summary:

**Report — phase 180 final verification pass**
- Verified (all already implemented in working tree): `inline()` chain = code→strong→em→linked-image→image→link→headings→lists in `frontend/assets/markdown.js`; `allowLinkUrl`/`allowImageUrl` allowlists (A5), shared `mdAnchorOpen` (target/rel only for absolute http(s)); `####`→`<h5>`, `#####`/`######`→`<h6>`, `#`/`##`→`<h3>`/`###`→`<h4>` unchanged; `.md-link`/`.md-img`/h5-h6 scale in `styles.css` (palette vars only). All surfaces (chat, thinking, restore, viewer, modal) route through the one renderer (pinned in `app.js`/`document.js`).
- Completion criteria:
  - `[text](url)`→`a.md-link`, `javascript:`/`data:` inert — ✅ (node probe + unit + E2E)
  - `![alt](src)`→`img.md-img loading="lazy"` with `alt`; bad src inert — ✅
  - `####`/`#####`/`######` ladder; shallow rungs unchanged — ✅
  - Fenced link/image/`####` stay code; cell-level link renders in `.md-table` — ✅ (node probe)
  - Byte-identity fixture + phase-44/138 suites pass unedited (git-verified untouched; 6 + 3 E2E pass in isolation) — ✅
  - `uv run pytest --cov=app --cov-report=term-missing`: **4198 passed, TOTAL 99%**; `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov`: **4 passed**; `uv run ruff check .`: clean; `uv run pyright`: **0 errors** — ✅
- Docs: full README/AGENTS.md review vs the whole project — env-example↔README table complete, page routes match `app/main.py`, AGENTS quick-ref artifacts exist. Fixed 1 inconsistency: README understated the image allowlist (impl per task 02 accepts first segments ≤2 chars, `^.{0,2}/`, not only `./`/`../`); phase-180 paragraphs otherwise accurate.
- Notable: image allowlist `.{0,2}/` is broader than the overview's `^\.{1,2}/` but is what task 02 explicitly specifies and the docstring documents — not a deviation from the tasks; SVG sizing deliberately deferred to phase 181.
- Next pending phase: `181_svg_document_preview`.
2026-10-14 16:03:36 -04:00
ducoterra 3d82a72b6f phase: 179_git_clone_server_freeze
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green; working tree holds exactly the phase's changes for the harness to commit.

## Phase 179 — Final verification report

- **Verified (all 3 tasks already complete in `complete/`)**: repro + CONFIRMED verdict, `asyncio.to_thread` fix, dedicated E2E suite.
- `app/api/sync.py::_run_sync`: both git ops (`clone_or_pull`, `file_commit_dates`) now `await asyncio.to_thread(...)`; phase-134 boundary note + module docstring document worker-thread-runs-to-completion and the A3 DB-calls-stay-on-loop note.
- **Tests / lint / coverage** (full suite 100% green):
  - `uv run pytest tests/integration/test_sync_git_freeze.py -v` → **4 passed** (post-fix probe <1.5 s during 3 s stub)
  - `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` → **2 passed in 25.65 s in isolation** (wall guard 120 s holds)
  - `uv run pytest --cov=app --cov-report=term-missing` → **4173 passed**, `app/` coverage **99%** (`app/api/sync.py` 100%)
  - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
- **Completion criteria**: ① integration probe green + verdict (CONFIRMED, mechanism, measured 3.04 s→0.003 s) in docstring — **met**; ② off-loop git ops + boundary/A3 notes — **met**; ③ full suite green, coverage 99% >90%, ruff+pyright clean — **met**; ④ E2E green in isolation (responsive-poll + cancel-during-clone contracts) — **met**; ⑤ atomic commit + move to `complete/` — **harness-owned** (pipeline rule overrides per-phase self-commit; all changes staged-ready in the working tree).
- **Documentation**: full README (1526 l) + AGENTS.md review — phase-179 bullets present and accurate (verified against fixture constants, `.env.example` fully covered by the config reference, pages table matches `app/main.py` shell routes, quick-reference commands/artifacts exist); **consistent — no further changes needed**.
- **Notable**: task-03 pipeline sessions died on context overflow but their harness validation passed; this pass independently re-ran every gate to confirm. No live infra touched; no git index commands run.
- **Next pending phase**: `180_markdown_inline_links_images`.
2026-10-14 13:06:39 -04:00
ducoterra 2b57180998 chore(repo): remove stray image.png screenshot artifact
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
The root-level image.png was committed accidentally during phase
160 (tool budget telemetry) as a bug-report screenshot. It is not
served or referenced as an asset by the app; the remaining
"image.png" mentions in tests and comments only describe the
owner's screenshot in prose, so removing the file is safe.

LLM-Generated: true
2026-10-14 10:41:04 -04:00
697 changed files with 83333 additions and 1423 deletions
+35 -3
View File
@@ -84,6 +84,25 @@ A15 — `thinking`, `tool`, unlimited tool rounds under
never-truncated contract is unchanged; the `read`-tool cap is
the only exception, owner-permitted 2026-09-10.
Phase 182 (verbatim text display, owner chat 2026-10-14) adds
**`show_text(path, offset?, limit?)` as the fifth agent tool** — offered
only when `BOR_ANSWER_TEXT=1` (the default), gated exactly like
`show_image`, with its own `<tools>` clause and its own kill switch — and
the **additive `done.answer_text`** field above. The new settings are
`BOR_ANSWER_TEXT` (master switch, ON), `BOR_SHOW_TEXT_MAX_CHARS` (per-block
budget, 12 000) and `BOR_SHOW_TEXT_MAX_BLOCKS` (per-answer budget, 5), all
following the `agent_max_rounds` pattern (`0` inert, negative refuses
startup). The principle in one sentence: **the model points, the server
supplies the bytes** — a ` ```bor source/path 12-40 ` fence or a
`show_text` call names a location (canonical locator `source/path` or
`source/path#start-end`, 1-based and inclusive), and the displayed text is
sliced from `documents.content` server-side, so what the user copies is the
knowledge base, never a re-typed approximation. **A1–A6 and A15 are
otherwise
UNCHANGED:** no new SSE event type, no new table or column, no new model,
endpoint or transport, no change to retrieval, the honesty gate, the
persona text or the embedding dimension.
---
## 3. High-Level Architecture
@@ -129,7 +148,9 @@ POST /api/chat {message, history?} (auth: require_user)
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
│ + <tools> → agent loop (ls/read/grep; round-capped
│ + <tools> → agent loop (ls/read/grep, plus show_image when
│ BOR_ANSWER_IMAGES=1 and show_text when BOR_ANSWER_TEXT=1 — the two
│ "show the user an artifact" tools, gated independently; round-capped
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
byte-identical direct-stream path) → turbo streamed
@@ -139,7 +160,15 @@ POST /api/chat {message, history?} (auth: require_user)
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
suggestions[], budget_exhausted?}` (terminal). Failure: `error` (terminal — no `done`, no
suggestions[], budget_exhausted?, answer_text[]?}` (terminal). **Phase 182
extends the `done` frame ADDITIVELY** — `answer_text` is the list of
server-resolved verbatim blocks (`locator`, `source`, `path`, `start`,
`end`, `total_lines`, `text`, `format`, `truncated`) the answer displayed;
there is **no eighth event type**, the key is OMITTED when no block was
displayed, and with `BOR_ANSWER_TEXT=0` it is absent on every turn (the
frame is then byte-identical to pre-phase). The same array persists on the
saved chat record, so restore and the anonymous shared page re-render the
panels from the server's bytes. Failure: `error` (terminal — no `done`, no
`query_log` row; a pre-stream DB outage is a plain 503 JSON). A
`tools=None` request never emits a `tool` frame; a round-capped turn
whose final reply carries no visible content answers with the budget
@@ -273,8 +302,11 @@ scripts.…`, and tests all honor it.
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
retries=N scaffold_stripped=N
retries=N scaffold_stripped=N verbatim_blocks=N
```
(`verbatim_blocks=` — phase 182 — is the count of verbatim display blocks
actually SENT on the `done` frame, `0` on every turn without them,
deflected included.)
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
executed only; a cancelled turn writes neither line nor `query_log`
row.)
@@ -0,0 +1,57 @@
# Phase 183 — Mass prune must not load the embeddings it is deleting
**Source:** owner incident 2026-10-15 — "brain-of-code hit 4 GB of RAM while running the sync" (the sync `started repos=4` at 11:19:23, the run reached `_prune` at 12:14:14, pruned **3,008 litellm docs**, and the VM locked up hard enough that boot −1 has no shutdown record at all).
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Context:** `app/models.py` — `Document.chunks` (**L227–229**, `relationship(back_populates="document", cascade="all, delete-orphan")` — **no `passive_deletes`**), `Chunk` (L232), `Chunk.embedding` (L241, `Vector(EMBEDDING_DIM)` = 768), `Chunk.document` (L247); `app/rag/importer.py` — `_prune` (**L3860**, the one-shot `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` at **L3909**, the per-doc `session.delete(doc)` at **L3931**, the single `if pruned: session.commit()` at **L3934–3935**), `_replace_content_chunks` (L992, the `doc.chunks = [...]` ORPHAN-removal path at L1033), `_store_summary` (L1595, the `for old in [c for c in doc.chunks if c.is_summary]` loop at L1648); `app/api/git_sources.py` — `delete_git_source` (L855, the `db.delete(doc)` loop at **L926**, the docstring's "chunks + embeddings via the `all, delete-orphan` cascade" line at **L863**, `docs_pruned` at L919); `app/api/docs.py` — the document-delete endpoint's `db.delete(row)` (L700); `alembic/versions/0001_initial_schema.py` — **L54 `sa.ForeignKey("documents.id", ondelete="CASCADE")`** (the DB already cascades — production confirms: `chunks_document_id_fkey`, `confdeltype='c'`); `tests/unit/test_importer.py` + `tests/integration/test_importer_e2e.py` (the importer families), `tests/integration/test_source_removal_api.py` + `tests/e2e/test_source_removal_cleanup.py` (the removal families), `tests/e2e/test_force_push_resync.py` (the "upstream deleted the file, re-sync prunes it" shape).
## Measured baseline (the numbers this phase is written against; recorded 2026-10-15)
- A loaded 768-dim `Chunk.embedding` costs **24.8 KiB of Python per chunk** (list header + 768 float objects) against **3 KiB on disk** — the ORM round-trip is an 8× expansion, and pgvector hands back a `list[float]`, never a buffer.
- `session.delete(doc)` on a relationship **without** `passive_deletes` makes SQLAlchemy load the child collection first (it must know which children to orphan/delete). Measured against the dev DB with production-shaped rows (400 docs × 25 chunks, ~745-char chunk content): **RSS grows linearly at ~0.24 MB per document deleted** = 25 chunks × ~9.7 KiB resident, +241 MB over 400 deletes, and NOTHING is released before the single commit because `app/db.py:30` builds sessions with `expire_on_commit=False`.
- Applied to the incident run: 3,008 docs × ~24 chunks ≈ **720 MB of float lists materialised to delete 72k rows**, on top of the ~300 MB the same walk already retained — on a 3,904 MB VM with **no swap**, sharing RAM with a 3.1 GB Postgres.
## Objective
Make a mass prune cost O(documents) instead of O(chunks × 24.8 KiB): `Document.chunks` declares `passive_deletes=True` so the DB's already-present `ON DELETE CASCADE` removes the chunk rows without the ORM ever hydrating them, on both mass-delete surfaces (the importer's prune and the admin's source removal), with every user-visible behaviour (counts, log lines, image/page-copy cleanup, the phase-122/126 prune guards, the `changed_paths` feed into the folder summaries) byte-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the pin is statement count, not RSS (task 01).** Memory assertions are flaky; the SQLAlchemy `before_cursor_execute` event stream is not. A new `tests/integration/test_prune_cascade_statements.py` counts, for one prune, every SELECT whose SQL targets the `chunks` table. PRE-fix the count is **exactly one per pruned document** (the lazy child collection load) and that test is GREEN before any code change — it IS the demonstration, exactly as phase 179 task 01 pinned the freeze with a latency probe. The measured RSS delta (the ~0.24 MB/doc figure above) is recorded in the module docstring as the corroborating number, asserted only as a loose ceiling (`< 40 MB` for a 200-doc fixture, where pre-fix measures ≈ 124 MB) so a noisy CI box cannot fail the suite on allocator behaviour.
- **D2 — the fix is one keyword, and it is a semantic claim about the schema (task 02).** `app/models.py:227–229` becomes `relationship(back_populates="document", cascade="all, delete-orphan", passive_deletes=True)`. That is only correct because the FK really cascades — so task 02 also adds a **schema-drift guard test** that reads the live constraint (`pg_constraint` where `conrelid='chunks'::regclass and contype='f'` → `confdeltype == 'c'`) and fails loudly if a future migration ever drops `ON DELETE CASCADE`, because at that point `passive_deletes=True` would silently orphan chunk rows. **No new alembic revision** — 0001 L54 already declares it, and the incident box's live schema matches (A13 stays satisfied because nothing changes).
- **D3 — both mass-delete surfaces, one keyword's blast radius (task 02).** With `passive_deletes=True` the ORM stops hydrating children for EVERY `session.delete(document)`: the importer's `_prune` (L3931), `delete_git_source`'s `db.delete(doc)` loop (`app/api/git_sources.py:926` — the endpoint that took 47 s for 2,303 immich docs on 2026-10-15 11:18:30), and `app/api/docs.py:700`. Audit-and-note, not change: none of the three reads `doc.chunks` AFTER deleting it (verified: `_prune` only touches `image_path`/`page_count`/`source`/`path`; the git-sources endpoint only counts; the docs endpoint only logs). The `doc.chunks = [...]` reassignment inside `_replace_content_chunks` (L1033) is the **orphan-removal** path, which `passive_deletes` does NOT short-circuit — the collection still loads there (a re-embed needs the old rows gone and the new ones built); it runs only for CHANGED files (217 in the incident run), and it is explicitly OUT OF SCOPE (A2) — the docstring says so in one line so the next reader does not "discover" it.
- **D4 — the cascade contract is verified at the row level, not assumed (tasks 01 + 03).** Every prune test asserts `select(count()).select_from(Chunk).where(Chunk.document_id.in_(deleted ids)) == 0` — no orphan chunk rows, before and after the change — plus the existing `documents`-level assertions. This is the phase's non-negotiable: a delete that stops hydrating children must still delete the children.
- **D5 — the E2E pins the owner-visible prune (task 03).** `tests/e2e/test_prune_cascade_cleanup.py` (this phase's dedicated Playwright file, AGENTS rule 4), the `test_force_push_resync.py` + `test_source_removal_cleanup.py` house shapes (dedicated module app, mock LLM, `BOR_SOURCES_DIR`/`BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated before AND after each test): a fixture source of N files is synced to `success` from the Sources page, a subset is deleted from disk upstream, the re-sync's counts line reports `pruned=N-subset` and those documents vanish from the KB (the ls/tree row is gone, `GET /api/docs` 404s the path, the Documents list total drops), while the chunk rows are gone too (the test's own DB query — the same D4 assertion, end to end) and the surviving documents still answer a chat question (a partial prune never damages the RAG). `E2E_REAL_LLM=1` skipped.
- **NOT touched:** the `_prune` batching (phase 184), the fast-path column set (phase 185), `expire_on_commit` (deliberately left `False` — every request path relies on reading attributes after commit), `Chunk.embedding`'s type, the retrieval query, the frontend, and any LLM/vision path.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; phases 184 and 185 build on this one, in that order)
## Tasks
1. `01_prune_cascade_probe.md` — the deterministic statement-count probe (one `chunks` SELECT per pruned doc, PRE-fix, green before the fix) + the row-level orphan assertions + the measured RSS delta recorded in the module docstring.
2. `02_passive_deletes.md` — the fix (`passive_deletes=True`), the live-FK drift guard, the three delete-site audit, the probe flipped to the post-fix assertion (`0` chunk SELECTs), the git-sources endpoint docstring correction.
3. `03_e2e_prune_cascade.md` — `tests/e2e/test_prune_cascade_cleanup.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Integration:** `tests/integration/test_prune_cascade_statements.py` (the statement-count probe, the orphan-row assertions, the loose RSS ceiling + the recorded measurement).
- **Unit:** the existing importer prune pins stay green as written — the phase-122 images-off prune guard, the phase-126 parent-PDF rule, `changed_paths` population, the `summary.pruned` count (`tests/unit/test_importer.py`, `tests/unit/test_importer_ignore.py`); add a unit pin that `Document.__mapper__.relationships["chunks"].passive_deletes is True`.
- **Regression:** `tests/integration/test_importer.py`, `test_importer_e2e.py`, `test_source_removal_api.py`, `test_sync_api.py`, `test_git_sources_api.py` — all green, byte-identical counts and log lines.
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_prune_cascade_cleanup.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the operations note that a prune deletes chunk rows in the DATABASE (FK cascade) and that the operator-visible cost of a mass prune is bounded by documents, not chunks (the number the incident was missing).
- The `_prune` docstring (`app/rag/importer.py:3860`) gains the one-paragraph "why `passive_deletes` + the FK cascade, and why the re-embed path is different" note; `app/models.py:227` gains the one-line "do not remove without re-checking the FK's ON DELETE CASCADE (pinned by tests/integration/test_prune_cascade_statements.py)" warning.
## Completion Criteria
- [ ] `app/models.py::Document.chunks` declares `passive_deletes=True`; a unit pin asserts it.
- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with the POST-fix assertion (zero `chunks` SELECTs during a prune of N docs, `0` orphan chunk rows, RSS delta under the pinned ceiling) and the pre/post measurements + the incident reference in the module docstring.
- [ ] The live-FK drift guard is green and would fail if `ON DELETE CASCADE` were absent (proven by the test's own negative fixture or a documented, executed one-off check).
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` green **in isolation** (the counts line, the vanished documents, zero orphan chunks, the surviving KB still answers).
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `fix(models): prune with passive_deletes so deleting documents never hydrates their embeddings`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — mechanism: `passive_deletes=True`, not a bulk-`delete()` rewrite.** The relationship keyword keeps `session.delete(doc)` as the one delete shape every caller already uses (importer prune, source removal, doc delete), so there is no new control flow to get wrong; the bulk-rewrite variant belongs to phase 184's batching, which builds on this.
- **A2 — out of scope: the `doc.chunks = [...]` orphan path** (`_replace_content_chunks` L1033). `passive_deletes` does not cover collection replacement, and a re-embed genuinely needs the loaded collection; it runs for changed files only. Documented as a known, bounded, deliberate non-fix.
- **A3 — no migration.** The FK's `ON DELETE CASCADE` ships in 0001 L54 and is verified live; this phase adds a drift guard instead of a schema change.
- **A4 — the probe is statement-count based.** Deterministic, CI-safe, and it names the exact waste; the RSS figure is documentation plus a loose ceiling, never the assertion that can flake a build.
@@ -0,0 +1,29 @@
# Task 01 — Pin what a prune actually loads (the probe, green PRE-fix)
**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
**Story:** n/a
## Objective
A deterministic integration probe that shows — before any code changes — that deleting documents hydrates their chunk rows (and therefore every 768-dim embedding) into Python, one SELECT per document. The probe is GREEN pre-fix; task 02 flips it. The measured cost is recorded in the module docstring so the next reader never has to re-derive it.
## Work
1. `tests/integration/test_prune_cascade_statements.py` — a NEW file, house patterns:
- Session/DB fixtures per `tests/integration/test_importer.py` (the shared test DB, the per-test truncate, `SessionLocal` from `app.db`).
- A `seed_docs` helper: **200** `Document` rows, `source="cascade-probe"`, each with **25** `Chunk` rows whose `content` is ~745 chars of fixed text and whose `embedding` is a deterministic 768-float vector (`[i / 768 for i in range(768)]` rotated by the doc index — deterministic, and L2-normalised enough not to matter here). Commit.
- **The statement capture:** a `sqlalchemy.event` listener on the engine's `before_cursor_execute` that appends `(statement, params)` tuples for the duration of the test; a helper `chunk_selects(captured)` returns the statements that are `SELECT` and reference `"FROM chunks"` (case-insensitive, `chunks` word-boundary so `documents` never matches). Install in the fixture, remove in teardown (`event.remove`) so no other test in the session sees the listener.
- **The probe test** `test_prune_loads_every_chunk_collection`: build the prune inputs the way `_prune` receives them (`source_names={"cascade-probe"}`, `seen=set()` → everything prunes), call `app.rag.importer._prune(session, source_names, seen, images=False, summary=ImportSummary(), page_dir=None)` (the real signature, `app/rag/importer.py:3860`), then assert:
- `chunk_selects(captured) == 200` — **exactly one child-collection SELECT per deleted document.** This is the PRE-fix assertion and it is GREEN now. Mark it so the flip is unmissable: a module-level `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC` constant with the comment `# 1 pre-phase-183 (the lazy child load); 0 post-phase-183 — task 02 flips this`.
- every `Chunk` row of the 200 documents is gone (`select(count()).select_from(Chunk).where(Chunk.document_id.in_(ids)) == 0`) — the D4 no-orphan contract, asserted before the fix so post-fix it means "unchanged, and not by accident".
- the `documents` rows are gone, `_prune`'s return value is 200, and `ImportSummary.changed_paths` holds the 200 `(source, path)` pairs.
- **The loose memory ceiling** `test_prune_memory_delta_has_a_ceiling`: the same 200-doc prune with `VmRSS` read from `/proc/self/status` before and after (a tiny local helper inside the test module — no new `app/` code in this task), asserting the delta is **< 40 MB**, with the MEASURED value recorded in the docstring. Pre-fix this measures ≈ **48 MB** for 200 docs (200 × 25 × 24.8 KiB ≈ 124 MB of float lists, of which the allocator keeps what it can reuse — the point is the order, not the exact figure), so the pinned ceiling must be picked from the ACTUAL measured pre-fix number: set it to roughly **one third** of what the pre-fix run measures, i.e. a ceiling the pre-fix run VIOLATES if it can be made to (if the pre-fix delta comes in under 40 MB on this box, LOWER the ceiling to a value the post-fix run clears with room and document both numbers — the assertion's job is to catch the hydrate-the-embeddings behaviour, never to fail on allocator noise). If the delta cannot be made discriminating on this hardware, keep the test but assert only `< 40 MB` and state in the docstring that the statement count is the load-bearing pin.
2. **The module docstring (the measurement record):** the incident in one paragraph (the 2026-10-15 sync pruned 3,008 litellm docs; the app cgroup peaked at 3.3 GB twice that week; the VM has 3,904 MB, no swap, and shares RAM with a 3.1 GB Postgres); the mechanism (`app/models.py:227–229` has no `passive_deletes`, so `session.delete(doc)` must load the children; `app/db.py:30` sets `expire_on_commit=False`, so nothing is released before the single commit at `importer.py:3934`); the measured unit costs (**24.8 KiB** of Python per 768-dim embedding vs 3 KiB on disk; **~0.24 MB resident per document deleted** at 25 chunks/doc, measured 2026-10-15 against the dev DB with production-shaped rows); this test's own measured numbers (fill them in from the run); and the verdict line **CONFIRMED — one chunk SELECT per pruned document** (if the probe comes back with 0 selects, STOP and flag it in the docstring and the phase report instead of proceeding).
3. Verify: `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green; `uv run pytest` green (the event listener is scoped to the fixture, nothing else sees it).
## Testing & Quality
- Integration: the probe family above (statement count, orphan-row check, return value, `changed_paths`, the loose RSS ceiling).
- Coverage: **>90%** on `app/` — test-only task, `app/` untouched, so the gate must simply stay green.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 1` (the pre-fix assertion) and the 0-orphan-chunks assertion.
- [ ] The module docstring records the incident, the mechanism, the 24.8 KiB / 0.24 MB-per-doc unit costs, and this run's measured statement count + RSS delta, plus the **CONFIRMED** verdict.
- [ ] `uv run pytest` green; no `app/` change in this task.
@@ -0,0 +1,52 @@
# Task 02 — The fix: `passive_deletes=True`, with the FK contract pinned
**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
**Story:** n/a
## Objective
Stop the ORM from hydrating chunk rows (and their 768-dim embeddings) when a `Document` is deleted: declare `passive_deletes=True` on `Document.chunks` so the database's existing `ON DELETE CASCADE` does the work, prove the schema really cascades with a drift guard, and flip task 01's probe to the post-fix assertion.
## Work
1. `app/models.py` — `Document.chunks` (**L227–229**):
```python
chunks: Mapped[list[Chunk]] = relationship(
back_populates="document",
cascade="all, delete-orphan",
# Phase 183: deleting a Document removes its chunks IN THE DATABASE
# (chunks_document_id_fkey is ON DELETE CASCADE — alembic 0001 L54,
# pinned live by tests/integration/test_prune_cascade_statements.py).
# WITHOUT this keyword the ORM loads every child before deleting it,
# which hydrates every 768-dim embedding (~24.8 KiB of Python each):
# a 3,008-document prune cost ~720 MB and took the 2026-10-15 VM
# (3,904 MB, no swap) down. Do not remove this without re-checking
# the FK — the drift guard fails the suite if the cascade is gone.
passive_deletes=True,
)
```
Nothing else in `models.py` changes (`Chunk.document` L247, `Chunk.embedding` L241 untouched).
2. `app/rag/importer.py` — the `_prune` docstring (L3860): append the phase-183 paragraph — the children go through the FK cascade, so no chunk row is ever SELECTed here; the `changed_paths` / image-copy / page-copy cleanup is unchanged; and the **A2 note**: the re-embed path `_replace_content_chunks` (L992, `doc.chunks = [...]` at L1033) is collection REPLACEMENT, which `passive_deletes` does not cover and which legitimately needs the loaded collection — a known bounded non-fix.
3. `app/api/git_sources.py` — `delete_git_source`'s docstring (**L863**, "documents (chunks + embeddings via the `all, delete-orphan` cascade)") → correct it to name the FK cascade (phase 183): the rows go in the database, the app never loads them. No logic change in the endpoint (verified: the loop at L919–927 only counts `docs_pruned`, it never reads `doc.chunks`).
4. `tests/integration/test_prune_cascade_statements.py` — flip the probe: `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, comment updated (`0 post-phase-183 — the FK cascade; 1 pre-phase-183 was the lazy child load`). Keep every other assertion (0 orphan chunks, 0 documents, the return value, `changed_paths`, the RSS ceiling — now clearing it by a wide margin) and extend the docstring: `CONFIRMED → FIXED (phase 183, passive_deletes=True)` with the post-fix measured statement count and RSS delta.
5. **The drift guard** — same file, `test_chunks_fk_cascades_at_database_level`: query the live schema and assert the cascade exists, so `passive_deletes=True` can never silently orphan rows.
```python
row = session.execute(
text("select confdeltype from pg_constraint "
"where conrelid = 'chunks'::regclass and contype = 'f'")
).all()
assert row and all(r[0] == "c" for r in row) # 'c' = ON DELETE CASCADE
```
Prove the guard is real (not a tautology) in the task's own verification: run it once against a scratch DB where the constraint is recreated `ON DELETE NO ACTION` (a throwaway database the task creates and drops — never the shared test DB) and record in the test's comment that the guard fails there, plus a unit pin `Document.__mapper__.relationships["chunks"].passive_deletes is True` in `tests/unit/test_importer.py`.
6. **Audit note (no code change):** grep `session.delete` / `db.delete` for the `Document` deletes (`app/rag/importer.py:3931`, `app/api/git_sources.py:926`, `app/api/docs.py:700`) and confirm none reads `doc.chunks` after the delete; record the confirmation in the task's completion note. If any site DOES read the collection after deleting, stop and report it rather than papering over it.
7. Verify: `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green post-fix; then the families the keyword touches — `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/integration/test_sync_api.py tests/integration/test_git_sources_api.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the flipped probe + the live-schema drift guard + the five families above, byte-identical counts and log lines.
- Unit: the `passive_deletes` mapper pin in `tests/unit/test_importer.py`.
- Coverage: **>90%** on `app/` (the `app/` change is one keyword + docstrings).
## Completion Criteria
- [ ] `app/models.py::Document.chunks` has `passive_deletes=True` with the explanatory comment; the unit mapper pin is green.
- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, zero orphan chunks, and the docstring's CONFIRMED → FIXED record with both measured numbers.
- [ ] The drift guard is green on the test DB and documented as failing on a scratch DB whose FK lacks `ON DELETE CASCADE`.
- [ ] `_prune`'s and `delete_git_source`'s docstrings match the new behaviour (the A2 re-embed note included); the three-delete-site audit is recorded.
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,34 @@
# Task 03 — E2E: a mass prune leaves nothing behind (the phase's dedicated Playwright suite)
**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
**Story:** n/a
## Objective
Pin the fix against the real pipeline the owner actually drives: upstream deletes files from a synced source, the admin re-syncs from the Sources page, and the prune removes exactly the vanished documents, leaves **zero orphan chunk rows**, keeps the surviving KB answerable, and reports the same counts line it always did. Then run the full gates, do the docs pass, and commit the phase.
## Work
1. `tests/e2e/test_prune_cascade_cleanup.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), following the `tests/e2e/test_force_push_resync.py` + `tests/e2e/test_source_removal_cleanup.py` house shapes point for point:
- **Isolation:** a per-module app on a dedicated port (the conftest session app is never started in the isolated run; `tests/e2e/conftest.py` — `APP_PORT` L32, `app_server` L106, `page` L219), the mock LLM on its conftest port, a dedicated `BOR_SOURCES_DIR` and `BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated **before AND after** every test, admin login via `tests/e2e/auth_helpers.py`.
- **Fixture:** a `prune_source` fixture dir with **9** importable `.md` files across two folders (`notes/keep-1..3.md`, `notes/gone-1..6.md`), one of the `keep-*` files carrying the sentinel phrase the chat test asks about.
- **Test 1 `test_deleted_files_prune_without_orphans`:** sync → assert the run reaches `success` and the Sources counts line reports `added=9` / `pruned=0` → delete the six `gone-*.md` files from disk → re-sync → assert the counts line reports `pruned=6` and `added=0` (the unchanged 3 stay unchanged) → the six documents are gone from the KB (the ls/tree row for `notes/gone-1.md` is absent, `GET /api/docs` for the path 404s via `page.request`, the Documents total is 3) → **zero orphan chunk rows** for the deleted document ids (the test's own `psycopg`/SQLAlchemy query against the E2E DB, the D4 assertion) and `select(count()).select_from(Chunk)` equals the sum of the 3 survivors' chunk counts → the run leaves no `documents` row whose `source` lost its file.
- **Test 2 `test_surviving_kb_still_answers_after_a_partial_prune`:** with the same pruned state, ask the chat the sentinel question (the `test_chat_rag.py` shape) and assert the answer cites the surviving sentinel document and the turn completes (no error state, no empty answer) — the prune did not damage retrieval.
- **Test 3 `test_source_removal_prunes_every_document_of_the_source`:** the admin path (`delete_git_source`, `app/api/git_sources.py:855`) through the Git sources UI or `page.request` `DELETE /api/git-sources/{id}` → every document AND every chunk row of that source is gone (the same orphan assertion), `docs_pruned` matches the seeded count via the log/response contract the endpoint already exposes, and the remaining source is intact.
- The module docstring: the incident reference, the fix (`passive_deletes=True` + the FK cascade), the test→observable mapping, the run-in-isolation command, the prerequisites (DB up: `podman compose up -d db`).
2. **Docs pass (the phase's final documentation review):** `README.md` + `AGENTS.md` reviewed in full against the post-phase app and every inconsistency fixed (pre-existing included). Add to `README.md`'s operations/sync section: a prune deletes chunk rows through the database cascade, so the cost of a mass prune scales with documents, not embeddings — and the `chunks` FK's `ON DELETE CASCADE` is load-bearing (pinned by `tests/integration/test_prune_cascade_statements.py`), so it must never be dropped.
3. **The full gates:**
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_source_removal_cleanup.py -v --no-cov` and `uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov` — green in isolation (the two prune/removal E2E families this phase must not have broken; run them AFTER the new suite so the DB truncate ordering stays clean).
- `uv run pytest --cov=app --cov-report=term-missing` — the whole suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/models.py`, the two docstring updates, the new integration + E2E files, the unit pin and the `.agents/` tree — e.g. `fix(models): prune with passive_deletes so deleting documents never hydrates their embeddings`, body naming the 2026-10-15 incident and the measured 24.8 KiB/embedding, ~0.24 MB/doc. Move the phase dir to `.agents/phases/complete/183_sync_passive_delete_cascade/`.
## Testing & Quality
- E2E: the dedicated suite above + the two regression E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` green **in isolation** (the counts line, the vanished documents, zero orphan chunks, the surviving KB answers, the source-removal variant clean).
- [ ] `tests/e2e/test_source_removal_cleanup.py` and `tests/e2e/test_force_push_resync.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the cascade-is-load-bearing note included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,59 @@
# Phase 184 — The sync's session stops growing with the knowledge base
**Source:** owner incident 2026-10-15 (as phase 183) — the import run held ONE session (`app/db.py:30` sets `expire_on_commit=False`) for the whole walk + prune, so every row it touched stayed resident until the run ended.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** `183_sync_passive_delete_cascade` (the batched prune deletes documents with the FK cascade doing the child work — it is only cheap because of `passive_deletes=True`).
**Context:** `app/db.py` — **L30 `SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False, future=True)`** (left EXACTLY as it is — every request path reads attributes after commit); `app/rag/importer.py` — `import_sources` (**L708**, `owns_session = session is None` at **L878**, `session = SessionLocal()` at **L880**, the `total` pre-walk at L884–899, the per-source loop at L901, the per-file loop at **L916** with `progress(source, rel, summary.files, total)` at **L943** and `await _index_file(...)` at **L945**, the `EmbeddingError` → `session.rollback()` handler at L951–958, the per-source post-passes `_enrich_pending_images` (**L965**) and `_backfill_image_contexts` (**L972**), `_prune(...)` (**L979**), `session.close()` in the `finally` at L984–985), `_prune` (**L3860**, `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` at **L3909**, `session.delete(doc)` at **L3931**, the single commit at **L3934–3935**), `_delete_image_copy` (takes `image_path: str | None`), `_delete_page_copies(doc, page_dir)` (needs only `doc.id` + `doc.page_count`); `app/api/sync.py` — `import_sources(...)` called with **no `session=`** at **L725** (the importer owns it — this phase never changes that contract); `app/config.py` (the `BOR_`-prefixed `Settings` pattern, `chunk_target_chars` at L228 as the shape), `app/core/settings_catalog.py` — `GROUPS` at **L72** (the `"Import & sources"` group) and the `CATALOG` entry shape at L226, pinned complete by `tests/unit/test_settings_catalog.py::test_catalog_covers_exactly_model_fields` (**`set(CATALOG) == set(Settings.model_fields)`** — a new `Settings` field without a catalogue row FAILS the build); `tests/unit/test_importer.py`, `tests/integration/test_importer_e2e.py`, `tests/integration/test_sync_api.py`, `tests/e2e/test_sync_changed_only.py`, `tests/e2e/test_sync_summary_visibility.py`.
## Measured baseline (2026-10-15, this repo's dev DB, production-shaped rows)
- A `Document`'s `content` averages **15.2 KB** (17,407 docs, 252.6 MB of markdown in the incident DB), and the identity map keeps every one the walk touched — see phase 185 for the column-level fix.
- Objects are released by NOTHING during the run: `expire_on_commit=False` means a commit does not drop attributes, and the importer never expunges. The identity map is therefore the run's memory profile, and it is unbounded: it scales with (files walked) + (chunks created) + (chunks hydrateable per prune — phase 183).
## Objective
Bound the import run's resident set independently of the size of the knowledge base: `session.expunge_all()` on a fixed cadence at the one point where no previously-loaded object is reachable, and `_prune` released in committed batches instead of one transaction at the end of the whole import — with every count, log line, guard, cancel boundary and summary number byte-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the pin is the identity map, not RSS (task 01).** `len(session.identity_map)` is exact, deterministic and it IS the retention surface. `tests/integration/test_import_session_retention.py` runs an import of N files through `import_sources` (the real entry point, the `test_importer_e2e.py` fixture shapes + the mock-embedder pattern from `tests/unit/test_importer.py`) and asserts the map size **at the end of the run**: PRE-fix it is `≈ docs + chunks + documents-touched` (record the measured number), and the test pins `>= N` to prove accumulation. The post-fix assertion (task 02) is `<= reset_files + slack`. RSS is recorded in the docstring only, never asserted (phase 183 A4's rule, inherited).
- **D2 — one new setting per new cadence, catalogue-compliant (tasks 02 + 03).** `sync_session_reset_files: int = 200` (`BOR_SYNC_SESSION_RESET_FILES`) and `sync_prune_batch: int = 500` (`BOR_SYNC_PRUNE_BATCH`). **`0` means "never reset / one transaction" = the exact pre-phase behaviour**, the operator's rollback knob without a re-deploy (the `BOR_LOG_LARGE_RESPONSE_BYTES` precedent). Both get `SettingMeta` entries in the `"Import & sources"` group (the completeness pin makes this mandatory, not optional), both documented in `.env.example` and `README.md`, neither `restart_required` (the importer reads `llm.settings` per run) and neither `secret`.
- **D3 — the reset point is the top of the per-file loop, and nowhere else (task 02).** Inside `import_sources`, at the top of the L916 loop body — after the PREVIOUS file's `_index_file` has returned, before this file's `progress(...)` hook at L943 — call `session.expunge_all()` every `sync_session_reset_files` files (`summary.files % reset == 0`), plus once at each source boundary before the two post-passes (L965/L972) so a source's walk residue never crosses into the next source. Proof of safety that the task must state in the comment: no object from a finished file is reachable afterwards (`_index_file` returns `None`, the progress/chunk/image hooks only assign fields on `_status`, `dates_map`/`ignore` are plain data, and `total` was computed before the loop), and the `EmbeddingError` handler's `session.rollback()` (L951–958) is safe with detached objects because it touches none. The two post-passes select their own rows fresh (`L3241`, `L3391`), so they are unaffected.
- **D4 — the prune is batched on light rows, not streamed cursors (task 03).** `_prune` replaces `select(Document).where(...).all()` with `select(Document).where(...).options(load_only(Document.id, Document.source, Document.path, Document.is_image, Document.image_path, Document.page_count)).order_by(Document.id)` — the exact column set the guards and the copy cleanup need (`is_image` + `path` for the phase-122 toggle guard and the phase-126 `_EMBEDDED_IMAGE_RE` parent rule, `image_path` for `_delete_image_copy`, `id` + `page_count` for `_delete_page_copies`) — and commits every `sync_prune_batch` deletes instead of once at the end (`sync_session_reset_files`'s sibling knob, `0` = the pre-phase single commit). Deletes stay `session.delete(doc)` (the phase-183 `passive_deletes` makes them child-load-free; a bulk `delete()` is NOT used, so `changed_paths`, the log line and the guards keep their exact per-document shape). `yield_per`/server-side cursors are explicitly rejected: a flush inside an open streaming result is a trap, and the light rows are ~1 KB each.
- **D5 — batch commits change the prune's atomicity, deliberately (A2).** Pre-phase a crash mid-prune rolled the whole prune back; post-phase a crash leaves a PREFIX pruned. That is safe here and must be stated in the docstring: `seen` is recomputed from the filesystem on every run, so the next sync converges; the import already commits per file throughout the walk (L1176–1190 and friends), so the run was never one transaction. The `summary.pruned` count is the count of documents actually deleted, unchanged in meaning.
- **D6 — the E2E pins "nothing changed, at scale" (task 04).** `tests/e2e/test_sync_session_reset.py`: a fixture source of **more files than the reset cadence** (the module pins `BOR_SYNC_SESSION_RESET_FILES=3` with 10 files, so the cadence fires repeatedly mid-run), synced from the Sources page → every document present exactly once, the counts line identical to a `BOR_SYNC_SESSION_RESET_FILES=0` run of the same fixture (the byte-identical assertion the phase lives on), a second sync reports all-`unchanged` with `pruned=0`, no 500s and no network-error state on the page, and the KB totals in the Sources stats match the DB. Plus the prune path with the batch pin at 2 (so a multi-batch prune is exercised through the UI).
- **NOT touched:** `app/db.py:30` (the global `expire_on_commit=False` — changing it would re-fetch rows across every request path for no benefit here), the importer's `owns_session` contract, the per-file index/summary/vision logic, the walk's column set (phase 185), `scripts/git_sync.py` (phase 186), the retrieval path, the frontend.
## Dependencies
- `183_sync_passive_delete_cascade` — required: batched `session.delete(doc)` without child hydration is the whole point of the batch cadence.
## Tasks
1. `01_identity_map_probe.md` — the deterministic identity-map probe (an `import_sources` run pinned to accumulate; the measurement recorded) plus the `sync_session_reset_files` / `sync_prune_batch` settings + catalogue + `.env.example` groundwork with the completeness pin green.
2. `02_session_reset_cadence.md` — the `expunge_all` cadence at the per-file loop top + source boundaries, the probe flipped to the bounded assertion, the safety-argument comment.
3. `03_batched_prune.md` — `_prune`'s light `load_only` select + per-batch commits + the atomicity note, the prune-side retention and statement-count pins.
4. `04_e2e_bounded_sync.md` — `tests/e2e/test_sync_session_reset.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the settings defaults + `0`-means-disabled semantics; `tests/unit/test_settings_catalog.py` completeness pin green with the two new fields in `"Import & sources"`; the existing prune guard pins (`images`-off skip, phase-126 parent rule, `changed_paths`) green as written.
- **Integration:** `tests/integration/test_import_session_retention.py` (the identity-map probe, both cadences), `tests/integration/test_importer*.py`, `test_sync_api.py`, `test_source_removal_api.py` — byte-identical counts and log lines.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_sync_session_reset.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md` + `.env.example`: the two new settings (default, `0` = pre-phase behaviour, what each bounds); one operations paragraph stating that the sync's memory is now bounded per-run rather than per-KB, and where the operator sets the ceiling.
## Completion Criteria
- [ ] `sync_session_reset_files=200` and `sync_prune_batch=500` exist on `Settings`, appear in `CATALOG` under `"Import & sources"`, in `.env.example`, in `README.md`; `uv run pytest tests/unit/test_settings_catalog.py -v` green.
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green with the POST-fix bound (identity-map size ≤ `reset_files + slack` with the cadence on, unbounded with it `0`) and both measured numbers in the module docstring.
- [ ] `import_sources` expunges only at the two proven-safe points (documented in the code comment); `_prune` deletes light rows in `sync_prune_batch` commits with the phase-122/126 guards and `changed_paths` intact.
- [ ] Every existing importer/sync/removal test is green with unchanged expectations (the byte-identical contract).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(import): bound the sync session with a reset cadence and batched prune commits`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — mechanism: periodic `expunge_all` + batched prune commits.** Chosen over "new session per source" (would re-derive the importer's `owns_session`/rollback semantics for a smaller win) and over "flip `expire_on_commit=True` for the sync session" (re-reads rows on every commit across a 27k-file walk and relies on implicit expiry instead of an explicit release).
- **A2 — the prune's atomicity changes to per-batch, accepted.** `seen` is recomputed from disk each run so a partial prune converges on the next sync; the import already commits per file, so no run-level transaction existed to lose.
- **A3 — both knobs default ON, `0` restores pre-phase behaviour exactly.** The operator keeps a no-redeploy rollback (the phase-167 `BOR_LOG_LARGE_RESPONSE_BYTES` precedent).
- **A4 — no server-side cursors / `yield_per`.** A flush inside an open streaming result is a correctness trap; the light prune rows cost ~1 KB each, so materialising them is not the problem this phase is fixing.
@@ -0,0 +1,33 @@
# Task 01 — Pin the run's retention (identity map) and lay the two knobs
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15 — the sync's memory grew with the KB, not the work
**Story:** n/a
## Objective
A deterministic integration probe on the real `import_sources` entry point showing that the run's SQLAlchemy identity map grows with every file touched and is never released (PRE-fix, green before the fix), plus the two new settings (`sync_session_reset_files`, `sync_prune_batch`) with their catalogue rows so the cadence tasks have somewhere to read their numbers from.
## Work
1. `app/config.py` — two new `Settings` fields, the `chunk_target_chars` (L228) shape, `BOR_` prefix from `env_prefix` (L38):
- `sync_session_reset_files: int = 200` — the number of files between identity-map releases inside `import_sources`; `0` = never (the pre-phase behaviour). Docstring names the phase and the `0` semantics.
- `sync_prune_batch: int = 500` — the number of pruned documents per commit inside `_prune`; `0` = one commit at the end (the pre-phase behaviour).
Both `int`, both validated to reject negatives (the `show_text_max_*` "negative refuses startup" precedent — follow whatever those fields do; if they validate in a `model_validator`, add these to the same validator).
2. `app/core/settings_catalog.py` — `CATALOG` entries in the **`"Import & sources"`** group (`GROUPS` L72, the group `chunk_target_chars` lives in is `"Chunking & embedding"` — use `"Import & sources"`, these are sync-run knobs): `"Sync session reset files"`, `"Sync prune batch"`. Neither `secret`, neither `restart_required`, both `overridable`. Verify `uv run pytest tests/unit/test_settings_catalog.py -v` — the `set(CATALOG) == set(Settings.model_fields)` pin fails until both entries exist, so this step is proven by that test going green.
3. `.env.example` — both keys with their defaults and a one-line comment each (`0` = pre-phase behaviour).
4. `tests/integration/test_import_session_retention.py` — a NEW file, house patterns (`tests/integration/test_importer_e2e.py` for the fixture-KB + mock-embedder shape, `tests/unit/test_importer.py` for the fake `LLMClient`):
- Fixture: a `sources` tmp dir with **12** `.md` files (each ~4 KB so each produces ≥2 chunks), a `Settings` with `sync_session_reset_files=0` and `sync_prune_batch=0` (the PRE-phase behaviour, explicitly), an isolated truncate of `documents`/`chunks` before and after.
- `test_import_accumulates_everything_it_touched`: run `await import_sources([root], llm, prune=False, session=session)` on a **fresh** session, then assert `len(session.identity_map) >= 12 * 2` (the 12 documents plus their chunks — the assertion that proves nothing is released; pin the exact measured figure in a module constant with the measured value in a comment). Record in the docstring: the map size, the chunk count, and `sum(1 for s in session.identity_map.all_states() if s.class_ is Document)`.
- `test_reset_cadence_is_configurable_off`: with `sync_session_reset_files=0` the map is exactly as the previous test measured (the `0` = pre-phase pin, which stays true after task 02 ships — it is the rollback guarantee, not a pre-fix only assertion).
- A module-level `_map_size(session)` helper (`len(session.identity_map)`) so tasks 02/03 assert through one door.
- Docstring: the incident paragraph, `app/db.py:30`'s `expire_on_commit=False` (which every request path depends on and this phase does NOT change), the measured numbers, the verdict **CONFIRMED — the run's identity map grows monotonically and nothing releases it**.
5. Verify: `uv run pytest tests/integration/test_import_session_retention.py tests/unit/test_settings_catalog.py -v` green; `uv run pytest` green.
## Testing & Quality
- Unit: the two settings' defaults, negative rejection, and the catalogue completeness pin.
- Integration: the retention probe (both cadence-off assertions).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] Both settings exist with the pinned defaults, `0`-means-disabled semantics, catalogue rows in `"Import & sources"`, `.env.example` entries.
- [ ] `uv run pytest tests/unit/test_settings_catalog.py -v` green (the completeness pin proves the catalogue work).
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green with the accumulation assertion and the docstring's measured numbers + **CONFIRMED** verdict.
- [ ] `uv run pytest` green; no behaviour change yet (the settings are read by nothing until tasks 02/03).
@@ -0,0 +1,45 @@
# Task 02 — The fix: release the identity map on a cadence at the safe point
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Bound the import run's resident set: `session.expunge_all()` every `sync_session_reset_files` files at the one point in `import_sources` where no previously-loaded object is reachable, plus once at each source boundary — with every count, log line and error path byte-identical, and task 01's probe flipped to the bounded assertion.
## Work
1. `app/rag/importer.py` — `import_sources` (L708):
- Read the knob once before the source loop: `reset_every = llm.settings.sync_session_reset_files` (`0` → disabled; resolve it ONCE so a mid-run settings overlay cannot change cadence between files).
- **The per-file reset** — at the TOP of the L916 loop body, before `summary.files += 1` / the `progress(...)` call at L943:
```python
# Phase 184 (task 02): the identity-map release. expire_on_commit=False
# (app/db.py:30) means a commit drops NOTHING, so without this the run
# holds every Document and Chunk it touched until the END of the import
# — the 2026-10-15 incident's second driver. This is the ONLY safe
# point: the previous file's _index_file has already returned (it
# returns None), the progress/chunk/image hooks only assign fields on
# the sync's own status object, dates_map/ignore are plain data, and
# `total` was computed before the loop. The EmbeddingError handler
# below calls session.rollback() and touches no object of its own, so
# detached state is irrelevant to it. reset_every == 0 → never
# (the exact pre-phase behaviour, the operator's rollback knob).
if reset_every and summary.files and summary.files % reset_every == 0:
session.expunge_all()
```
- **The source-boundary reset** — immediately before `await _enrich_pending_images(...)` (L965), the same guard (`if reset_every: session.expunge_all()`), with a one-line comment: the two post-passes (L965, L972) select their own rows fresh (L3241, L3391), so no object survives the reset.
- Do NOT reset inside `_index_file`, `_unchanged_doc`, `_store_summary`, or any post-pass — their locals are live across statements.
2. `tests/integration/test_import_session_retention.py` — extend, then flip:
- `test_reset_cadence_bounds_the_map`: the same 12-file fixture with `sync_session_reset_files=3` → after the run `assert _map_size(session) <= 3 + SLACK` (pick `SLACK` from the measurement — it must cover the last partial batch plus whatever a single file's own chunks contribute; document the arithmetic in the comment) and the import still reports `added=12`, the same chunk total, and the same documents as the cadence-`0` run (the byte-identical comparison the phase lives on — compare the two runs' `ImportSummary` fields and the DB's per-path chunk counts).
- Keep `test_import_accumulates_everything_it_touched` and `test_reset_cadence_is_configurable_off` (cadence `0` still accumulates → the rollback guarantee).
- `test_reset_cadence_survives_an_embedding_error`: a stub embedder that raises `EmbeddingError` on file #4 (with `sync_session_reset_files=2`, so a reset has already happened) → the run records `errors=1`, keeps going, and finishes with the other 11 files indexed; asserts nothing blows up on detached state (the handler's `session.rollback()` path).
- Docstring: `CONFIRMED → FIXED (phase 184)`, both measured map sizes (cadence 0 vs cadence 3), and the post-fix bound.
3. Verify: `uv run pytest tests/integration/test_import_session_retention.py -v`; then the families — `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_importer_dates.py tests/integration/test_sync_api.py tests/unit/test_importer.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the bounded-map test, the cadence-`0` guarantee, the error-path test, the byte-identical summary comparison.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `import_sources` calls `session.expunge_all()` at exactly the two documented points, guarded by `reset_every`, with the safety argument in the comment.
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green (bounded with the cadence on, unbounded with it `0`, error path clean, identical summaries).
- [ ] No existing importer/sync test needed an expectation change.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,51 @@
# Task 03 — The prune stops holding 17k full documents and releases in batches
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15 (the prune phase was where the box died)
**Story:** n/a
## Objective
Make `_prune`'s resident set independent of the size of the knowledge base: select only the columns its guards and cleanup need (not `content`, avg 15.2 KB × 17,407 rows), and commit every `sync_prune_batch` deletes instead of one transaction after the whole loop. The per-document semantics — the phase-122 images-off guard, the phase-126 parent-PDF rule, `changed_paths`, the `import: pruned …` log line, the returned count — stay byte-identical.
## Work
1. `app/rag/importer.py` — `_prune` (L3860):
- Replace **L3909** `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` with a light-column, deterministically ordered select:
```python
docs = session.scalars(
select(Document)
.options(
load_only(
Document.id, Document.source, Document.path,
Document.is_image, Document.image_path, Document.page_count,
)
)
.where(Document.source.in_(source_names))
.order_by(Document.id)
).all()
```
with `from sqlalchemy.orm import load_only` added to the module's SQLAlchemy imports, and a comment naming exactly why those six columns (the phase-122 toggle guard reads `is_image`; the phase-126 `_EMBEDDED_IMAGE_RE` parent rule reads `path`; `_delete_image_copy` takes `image_path`; `_delete_page_copies` needs `id` + `page_count`; the `seen` tuple and `changed_paths` need `source` + `path`) and why `content` is not among them (252 MB of markdown per full KB walk; phase 185 does the same job for the fast path).
- Add the batch commit. After `session.delete(doc)` / `pruned += 1` / the `logger.info("import: pruned …")` line, commit on the batch boundary and once at the end:
```python
if prune_batch and pruned and pruned % prune_batch == 0:
session.commit() # phase 184: a long prune no longer holds a huge transaction
...
if pruned:
session.commit()
```
`prune_batch = llm-free`: `_prune` has no `llm` argument, so pass the cadence in as a keyword — `prune_batch: int = 500` on the signature, and the caller at L979 passes `prune_batch=llm.settings.sync_prune_batch`. Keep `prune_batch=0` meaning "single commit at the end" (the pre-phase behaviour, and the default for the standalone `scripts/import_docs.py` path if it calls `_prune` directly — check and keep its call byte-identical by leaving the keyword default at the value that preserves its behaviour).
- Update the `_prune` docstring: the light-row contract, the batch-commit contract, and the **A2 atomicity note** (a crash mid-prune leaves a prefix pruned and the next sync converges, because `seen` is recomputed from the filesystem; the import already commits per file, so no run-level transaction existed).
2. `tests/integration/test_import_session_retention.py` — add the prune side:
- `test_prune_holds_no_content`: seed 30 docs × 5 chunks, prune them all, and assert (a) via a `before_cursor_execute` capture that the documents SELECT issued by `_prune` does NOT list `documents.content` in its column list (the same deterministic technique phase 183 task 01 uses for chunks — assert on the captured statement text), and (b) 0 orphan chunks afterwards.
- `test_prune_batch_commits`: `sync_prune_batch=10` over 30 prunable docs → assert the commit count (a `after_commit`/`after_transaction_end` event counter, or spy `Session.commit` with `monkeypatch`) is 3 (+ whatever the caller adds), that all 30 documents and 150 chunks are gone, and `summary.pruned == 30`; with `sync_prune_batch=0` the commit count is 1 and every other number is identical.
- `test_prune_guards_unaffected_with_light_rows`: the phase-122 images-off guard and the phase-126 parent-PDF rule keep working with `load_only` rows (an `is_image` doc with `images=False` survives; an embedded-image doc whose parent PDF rel is in `seen` survives) — this is the assertion that the six columns really are enough.
3. **Check the deferred-attribute trap explicitly:** anything inside the prune loop that touches a non-loaded attribute would emit a per-row lazy SELECT — the statement capture in `test_prune_holds_no_content` must assert **no per-document follow-up SELECT** (the captured count of `FROM documents` statements stays at 1 for the initial load). If a lazy load appears, the missing column goes into `load_only` (and the comment updates), it does not get papered over with `raiseload`.
4. Verify: `uv run pytest tests/integration/test_import_session_retention.py -v`; then `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/unit/test_importer.py tests/unit/test_importer_ignore.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the no-`content` statement pin, the batch-commit pins (both cadences), the guard pins with light rows, the orphan-row pins.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `_prune` selects through `load_only(id, source, path, is_image, image_path, page_count)` ordered by `Document.id`, commits every `sync_prune_batch` (default 500, `0` = single commit), and the docstring records the column rationale + the A2 atomicity note.
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green (no `content` in the prune SELECT, no per-row lazy load, the batch counts, the guards, zero orphans).
- [ ] `scripts/import_docs.py`'s prune path (if any) keeps its pre-phase behaviour.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,35 @@
# Task 04 — E2E: the bounded sync still syncs everything (the phase's dedicated Playwright suite)
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the UI that the phase's two cadences change the run's memory profile and NOTHING else: the same fixture produces the same counts line with the cadence on and off, every document lands exactly once, a re-sync is all-`unchanged`, a multi-batch prune behaves, and the Sources page never stalls or errors. Then the full gates, the docs pass, and the atomic commit.
## Work
1. `tests/e2e/test_sync_session_reset.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_sync_changed_only.py` + `test_sync_summary_visibility.py` house shapes:
- **Isolation:** a dedicated module app + port (`tests/e2e/conftest.py` — `APP_PORT` L32, `app_server` L106, `page` L219), the mock LLM, dedicated `BOR_SOURCES_DIR` / `BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated before AND after every test, admin login via `tests/e2e/auth_helpers.py`.
- **Fixture:** 10 `.md` files across two folders (`a/1..5.md`, `b/1..5.md`), one carrying the sentinel phrase.
- **Test 1 `test_reset_cadence_produces_the_identical_run`:** module sets `BOR_SYNC_SESSION_RESET_FILES=3` (so the cadence fires 3× through a 10-file walk). Sync from the Sources page → `success`; the counts line reports the 10 added (the phase-98 counts-line contract, `test_sync_summary_visibility.py` shape); the KB totals match `select(count())` on `documents` (10) and on `chunks` (>10); no network-error state and no 500 during the run's polls.
- **Test 2 `test_resync_is_unchanged_with_the_cadence_on`:** a second sync from the same fixture (nothing changed on disk) → `unchanged=10`, `added=0`, `pruned=0`, `success`, and the DB's chunk total is unchanged (no silent re-embed — the cadence must not be losing rows and re-adding them).
- **Test 3 `test_multi_batch_prune_from_the_ui`:** `BOR_SYNC_PRUNE_BATCH=2` (module-level env), delete 5 of the 10 files from disk, re-sync → the counts line reports `pruned=5`, the 5 documents are gone from the KB and from the ls/tree rows, the survivors' chunk total is unchanged, and 0 orphan chunk rows exist (the test's own DB query). The batch cadence therefore commits 3 times through the UI's own run.
- **Test 4 `test_cadence_zero_matches_cadence_three` (the byte-identical cross-check):** re-seed the fixture, run with `BOR_SYNC_SESSION_RESET_FILES=0` and compare the resulting `documents`/`chunks` per-path counts against test 1's recorded numbers (the test records its numbers into a module-level fixture or asserts against a fixed expected table — whichever is deterministic in this suite's house style; the assertion must be a real comparison, not a tautology). The two runs must agree on every path, every chunk count, and the counts line.
- Module docstring: the incident reference, the two knobs + their defaults + `0` semantics, the test→observable mapping, the run-in-isolation command, the prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` reviewed in full against the post-phase app; every inconsistency fixed, pre-existing included. `README.md` gains the operations paragraph (the sync's memory is now bounded per run; the two knobs; `0` restores the old behaviour) and `.env.example` matches the shipped defaults.
3. **The full gates:**
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` and `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` — green in isolation (the sync-suite neighbours this phase must not disturb; run after the new suite).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/config.py`, `app/core/settings_catalog.py`, `app/rag/importer.py`, the integration + E2E files, `.env.example`, the docs and the `.agents/` tree — e.g. `perf(import): bound the sync session with a reset cadence and batched prune commits`. Move the phase dir to `.agents/phases/complete/184_sync_bounded_session/`.
## Testing & Quality
- E2E: the dedicated suite above + the two regression E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` green **in isolation** (identical run with the cadence on, unchanged re-sync, multi-batch prune, the cadence-0 vs cadence-3 cross-check).
- [ ] `tests/e2e/test_sync_changed_only.py` and `tests/e2e/test_sync_git_freeze.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md`, `AGENTS.md` and `.env.example` accurate and consistent with the whole project.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,55 @@
# Phase 185 — The unchanged fast path stops fetching 15 KB of markdown per file
**Source:** owner incident 2026-10-15 (as phases 183/184) — every one of the ~27k files the walk inspected hydrated its `Document` row, `content` included, and `expire_on_commit=False` kept it resident for the whole run.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; sequenced after `183`/`184` because it touches the same walk and reuses their statement-capture technique.
**Context:** `app/rag/importer.py` — `_index_file` (**L1384**; the row load **`doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))` at L1452** — the ONLY place the fast path gets its row, `_file_hash` streamed digest at L1451, the fast-path guard at L1460–1469, the three sibling loads at **L1914** (PDF), **L2675** (embedded image), **L3674** (image) which the phase-147 fast path shares), `_row_kind_mismatch` (**L1352** — reads `doc.is_image` only), `_unchanged_doc` (**L1061** — `doc.file_hash = file_digest` at L1137, the **phase-118 summary-NULL backfill `if doc.summary is None: await _store_summary(..., content=doc.content ...)` at L1139–1142**, `_regate_strip` at L1145–1157 which reads `doc.page_count` + `doc.path` + `doc.full_path` for a `.pdf` with a strip, the D1 manual-date lock at L1163, the D4 date refresh at L1172), `_store_summary` (L1595), `ImportSummary.log()` (L520); `app/db.py:30` (`expire_on_commit=False`, unchanged); `tests/unit/test_importer_unchanged_fast_path.py` (the phase-147 matrix, whose docstring already pins "the phase-118 backfill on the fast path runs from the STORED `doc.content`"), `tests/unit/test_importer.py`, `tests/integration/test_importer.py`, `tests/integration/test_importer_e2e.py`, `tests/e2e/test_document_viewer.py` + `tests/e2e/test_document_summaries.py` (the surfaces that render stored `content` and summaries), `tests/e2e/test_sync_changed_only.py`.
## Measured baseline (2026-10-15, incident box)
`documents`: 17,407 rows, `avg(length(content)) = 15,217` chars, **252.6 MB of markdown total**. `_index_file` loads the whole row for every file it inspects — the incident run inspected ~27k files and re-indexed **217**, so the run paid ~250 MB of markdown resident (`expire_on_commit=False`) to discover that 99% of the KB had not changed.
## Objective
Keep the phase-147 unchanged fast path's behaviour exactly as it is and make it stop hydrating `Document.content`: the row load defers the one column the fast path does not read, so a full walk's memory scales with row metadata instead of markdown — while the phase-118 summary-NULL backfill (the one fast-path branch that genuinely needs the stored content) keeps working from the stored text.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `defer(Document.content)`, not a `load_only` allowlist (task 02).** The surgical form: `select(Document).options(defer(Document.content)).where(...)` at the row loads. A `load_only` allowlist would defer every column not named — including columns added by a future phase, which is how a "memory fix" turns into a mysterious N+1 six phases later. `defer()` names exactly what is dropped, so everything else keeps today's semantics, and pyright/the suite stay honest. Applies to the shared row load at **L1452** and the three sibling loads that share the phase-147 fast path — **L1914** (PDF), **L2675** (PDF-embedded image), **L3674** (image) — via ONE module-level helper so the four sites cannot drift: `_DOC_ROW_OPTS: tuple[Callable[[str], Any], ...] = (defer(Document.content),)` (or a tiny `_doc_row_stmt(source, rel)` helper returning the prepared `select` — the executor picks one, and a unit pin asserts every fast-path load uses it).
- **D2 — NO `raiseload`.** `noload`/`raiseload` would break the phase-118 backfill (L1139–1142) and any future legitimate reader. With plain `defer`, reading `doc.content` emits one lazy SELECT for that row and behaves exactly as today. The trade is explicit: unchanged files with a stored summary pay ZERO extra queries; a summary-NULL backfill pays ONE extra SELECT, which is a per-row, one-time, already-rare cost. The docstring says so.
- **D3 — the pin is the statement's column list, deterministic (task 01).** `tests/integration/test_import_fast_path_columns.py` captures SQL through `before_cursor_execute` (the phase-183 task 01 technique) and asserts, for an unchanged re-sync of a seeded KB: (a) the documents SELECT issued by `_index_file` does **not** mention `documents.content` in its select list — PRE-fix that assertion FAILS, so task 01 writes it as the *inverse* assertion (content IS present, green pre-fix) and task 02 flips it, exactly the 179 pattern; (b) exactly ONE documents SELECT per file (no lazy follow-up) for a doc WITH a summary; (c) for a doc with `summary IS NULL`, the backfill still runs and emits exactly ONE extra documents SELECT (the lazy `content` load) and produces the same summary chunk as today.
- **D4 — observable parity is non-negotiable.** Every count (`files/added/updated/unchanged/dates_updated`), every log word (`import: unchanged …`, `import: date-refreshed …`, `import: summary …`), the D1 manual-date lock, the phase-163 PDF strip re-gate and the phase-118 backfill result stay byte-identical — this phase changes WHICH COLUMNS travel, never what the run does. `tests/unit/test_importer_unchanged_fast_path.py` must go green with **no expectation edits** (if an expectation must change, the design is wrong — stop and flag).
- **D5 — the E2E pins the two surfaces `content` feeds (task 03).** `tests/e2e/test_fast_path_deferred_content.py`: (1) a doc whose `summary` is NULL in the DB (the test nulls it directly, the way phase 118's tests do) is re-synced from the Sources page and its summary comes back from the STORED content (the document page's summary renders + the summary chunk exists + `dates_updated`/counts stay honest); (2) a normal unchanged doc's viewer still shows the FULL stored text (no truncation, no blank) after a sync that ran with the deferral; (3) a changed file re-indexes and its viewer shows the NEW text. `E2E_REAL_LLM=1` skipped.
- **NOT touched:** `app/db.py:30`, the walk itself (`iter_importable_files`, L624 — its `sorted(root.rglob("*"))` materialisation is a separate, much smaller cost and stays), `_unchanged_doc`'s logic, `_store_summary`, the index/summary/vision pipelines, the retrieval path, the frontend, phases 183/184's files beyond the shared row-load line.
## Dependencies
— (none functionally; run after `183_sync_passive_delete_cascade` and `184_sync_bounded_session` so the walk's three memory fixes land in one coherent story)
## Tasks
1. `01_fast_path_statement_probe.md` — the statement-capture probe pinning that the fast-path row load hydrates `content` (PRE-fix, green now), the zero-extra-SELECT pin for a doc with a summary, and the backfill's lazy-load pin.
2. `02_defer_content.md` — the fix: the shared `_doc_row_stmt` / `defer(Document.content)` helper wired into the four fast-path row loads, the docstrings, the probe flipped.
3. `03_e2e_deferred_content.md` — `tests/e2e/test_fast_path_deferred_content.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_importer_unchanged_fast_path.py` green with **no expectation edits** (the phase-147 matrix: zero-LLM-call, D1/D4 date rules, the phase-118 backfill from stored content); `tests/unit/test_importer.py` green.
- **Integration:** `tests/integration/test_import_fast_path_columns.py` (the column-list + statement-count pins), `tests/integration/test_importer.py`, `test_importer_e2e.py`, `test_importer_dates.py`, `test_sync_api.py`.
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_fast_path_deferred_content.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md` (sync/import operations): one sentence — the unchanged fast path reads row metadata only, so a full-resync walk's cost is metadata-bound; the rare summary-NULL backfill re-reads the stored content (one query) and that is intentional.
- `app/rag/importer.py`: the module docstring's phase-147 fast-path paragraph and `_index_file`'s docstring gain the deferral note (with the D2 no-`raiseload` rationale), so nobody "optimises" it into a lazy-load landmine.
## Completion Criteria
- [ ] The four fast-path row loads share ONE deferral helper; `Document.content` is deferred and nothing else is.
- [ ] `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green with the POST-fix column-list assertion (no `documents.content` in the fast-path SELECT), the one-SELECT-per-file pin, and the backfill's single lazy-load pin, all measured numbers in the module docstring.
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py -v` green with zero edited expectations.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` green **in isolation** (backfill from stored content, full text in the viewer, changed file re-indexes).
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(import): defer Document.content on the unchanged fast path so a full walk no longer hydrates the KB`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — `defer(Document.content)` only.** Not a `load_only` allowlist (future columns would silently become lazy), not `undefer_group`s, not a DTO/read-model rewrite of the importer (too large a change to keep phase-147 parity provable).
- **A2 — no `raiseload`/`noload`.** The phase-118 backfill must keep reading the stored content; failing loudly there would be a behaviour regression, and the extra SELECT is per-row and one-time.
- **A3 — the walk itself (`iter_importable_files`) is out of scope** this phase — its `sorted(root.rglob("*"))` list is a one-off per source (tens of thousands of `Path`s, ~10 MB), orders of magnitude below the markdown this phase removes.
@@ -0,0 +1,27 @@
# Task 01 — Pin that the fast path hydrates the markdown it never reads
**Phase:** `185_sync_defer_content_fast_path` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
A deterministic, SQL-level probe proving that the phase-147 unchanged fast path SELECTs `documents.content` for every file it inspects (green PRE-fix, task 02 flips it), that it issues exactly one documents SELECT per unchanged file, and that the phase-118 summary-NULL backfill path behaves as documented.
## Work
1. `tests/integration/test_import_fast_path_columns.py` — a NEW file, house patterns (`tests/integration/test_importer_e2e.py`'s fixture-KB + fake-embedder shape; the statement capture is phase 183 task 01's: a `sqlalchemy.event` `before_cursor_execute` listener installed and removed in a fixture).
- Fixture: 4 `.md` files (each ~8 KB so `content` is unmistakably large), seeded through the real `import_sources` so rows look production-shaped, then a SECOND `import_sources` run over the same tree (the unchanged run) with the capture active. One of the four rows gets `documents.summary = NULL` committed before the second run (the phase-118 backfill case).
- Helpers: `doc_selects(captured)` → statements that are SELECTs `FROM documents`; `loads_content(stmt_text)` → True when the statement's select list names `content` (match on the aliased/plain column token, and state the matcher in a comment so it cannot match `content_hash`).
- `test_unchanged_run_hydrates_content_pre_fix`: assert `len(doc_selects(captured)) >= 4` and `all(loads_content(s) for s in doc_selects(captured))` — **the PRE-fix assertion**, with the marker comment `# task 02 flips this: no content in the fast-path SELECT`. Record the captured statement text for ONE file in the docstring (the evidence).
- `test_unchanged_file_is_one_select`: for a doc WITH a summary, the per-file documents statement count is exactly **1** (this pin must hold before AND after the fix — post-fix it is the guard against the deferral turning into a per-row lazy-load storm).
- `test_summary_null_backfill_reads_stored_content`: for the summary-NULL row, assert the second run produced a summary chunk (`is_summary` chunk exists, its content matches the stored markdown) and, before the fix, no extra documents SELECT is needed; structure the assertion so task 02's post-fix expectation is a documented constant: `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 0` → task 02 sets it to `1` (the single lazy `content` load) with the arithmetic in the comment.
2. Docstring: the measured baseline (17,407 rows, avg 15,217 chars, 252.6 MB), the mechanism (`_index_file` L1452 loads the whole row; `app/db.py:30` `expire_on_commit=False` keeps it), the captured statement evidence, and the verdict **CONFIRMED — the fast path SELECTs `content` for every file inspected**.
3. Verify: `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green; `uv run pytest` green.
## Testing & Quality
- Integration: the three pins above.
- Coverage: **>90%** on `app/` (test-only task).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green with the PRE-fix content-in-SELECT assertion and `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 0`.
- [ ] The one-SELECT-per-file pin is green and marked as must-stay-green post-fix.
- [ ] The docstring carries the baseline numbers, the captured statement, and the **CONFIRMED** verdict.
- [ ] `uv run pytest` green; no `app/` change in this task.
@@ -0,0 +1,53 @@
# Task 02 — The fix: defer `Document.content` on the fast-path row loads
**Phase:** `185_sync_defer_content_fast_path` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Make the unchanged fast path read row metadata instead of markdown: one shared statement helper that defers `Document.content` (and nothing else), wired into the four row loads that share the phase-147 fast path, with task 01's probe flipped and zero behaviour change.
## Work
1. `app/rag/importer.py` — add ONE module-level helper next to `_file_hash` (L687):
```python
def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
"""The importer's single Document-row statement (phase 185).
``defer(Document.content)`` — the unchanged fast path (phase 147) reads
file_hash/is_image/page_count/summary/dates, never the stored markdown,
and app/db.py builds sessions with expire_on_commit=False, so hydrating
content for EVERY inspected file is what took the 2026-10-15 run to
~250 MB on a 17k-document KB (avg 15,217 chars/row). ONE column is
deferred on purpose: a load_only allowlist would silently defer every
column a later phase adds, and the deferred read would then surface as
an unexplained N+1. No raiseload/noload: the phase-118 summary-NULL
backfill (``_unchanged_doc``) genuinely reads ``doc.content`` and must
keep working — it costs one extra SELECT for that one row.
"""
return (
select(Document)
.options(defer(Document.content))
.where(Document.source == source, Document.path == rel)
)
```
with `from sqlalchemy import select` already present and `from sqlalchemy.orm import defer` added (import style: match the module's existing import block).
2. Replace the four identical fast-path row loads with `session.scalar(_doc_row_stmt(source, rel))`:
- `_index_file` **L1452** (the text/branch dispatch),
- `_index_pdf_file` **L1914**,
- `_index_one_embedded_image` **L2675** (the `image_doc` variant — same statement shape, different variable name),
- `_index_image_file` **L3674**.
Keep every surrounding line untouched (`file_digest` computed first, the guard, the branch bodies). Grep for any OTHER `select(Document).where(Document.source ==` in the importer to confirm no fifth site exists; if one does (e.g. the post-passes at L3241/L3391 select differently shaped rows), leave it and say so in the task note — those are NOT fast-path loads.
3. Docstrings: the module docstring's phase-147 paragraph and `_index_file`'s docstring (L1440–1470 region) gain the deferral sentence; `_unchanged_doc`'s step-3 comment (the phase-118 backfill) gains "…which is the one fast-path read that pulls the deferred `content` back — one SELECT, one time, by design".
4. `tests/integration/test_import_fast_path_columns.py` — flip: `EXPECTED_CONTENT_IN_FAST_PATH_SELECT = False` (assert NO `content` token in the fast-path documents SELECT list) and `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 1`; keep the one-SELECT-per-file pin green; docstring `CONFIRMED → FIXED (phase 185, defer(Document.content))` with the post-fix captured statement text (the evidence that markdown no longer travels).
5. `tests/unit/test_importer_unchanged_fast_path.py` — must go green **without editing an expectation**. If a unit test stubs the session (no real lazy load possible) and fails on the deferral, fix the TEST HARNESS honestly (e.g. the fake returns a real `Document` instance, which already has `content` set) rather than weakening a pin — and say in the task note which harness detail was adjusted.
6. Verify: `uv run pytest tests/integration/test_import_fast_path_columns.py tests/unit/test_importer_unchanged_fast_path.py -v`; then `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_importer_dates.py tests/integration/test_import_extensions_env.py tests/integration/test_sync_api.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the flipped column pin, the one-SELECT pin, the single-lazy-load backfill pin.
- Unit: the phase-147 matrix green, zero edited expectations.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `_doc_row_stmt` exists and all four fast-path loads use it (grep evidence in the task note); only `Document.content` is deferred.
- [ ] `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green post-fix with both flipped constants and the post-fix captured statement in the docstring.
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py -v` green with zero edited expectations (any harness adjustment documented).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,32 @@
# Task 03 — E2E: the deferred column still shows up where the owner reads it (the phase's dedicated Playwright suite)
**Phase:** `185_sync_defer_content_fast_path` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the UI that the deferral is invisible to the owner: the phase-118 summary-NULL backfill still produces a summary from the STORED content, an unchanged document's viewer still renders its full stored text, and a changed file re-indexes to the new text. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_fast_path_deferred_content.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_document_summaries.py` / `test_document_viewer.py` / `test_sync_changed_only.py` house shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, the shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`).
- **Test 1 `test_summary_null_backfill_survives_the_deferral`:** seed+sync a fixture with 2 files, then set `documents.summary = NULL` for one row directly (the way phase 118's tests do), re-sync from the Sources page → the run reports `unchanged` (not `added`/`updated`) and the document page shows a non-empty summary; the `is_summary` chunk exists for that document; exactly one summary chunk (no duplicate).
- **Test 2 `test_unchanged_document_viewer_shows_full_stored_text`:** re-sync without touching the files → open the document view and assert the rendered text equals the stored `content` (the sentinel phrase at the START and the END of the file both present, no truncation, no empty body) — the deferral must not have made the viewer's source row thinner.
- **Test 3 `test_changed_file_reindexes_under_the_deferral`:** rewrite one file on disk (new sentinel + changed length), re-sync → the run reports `updated=1`, the viewer shows the NEW text and no stale fragment, the chunk count changes, and a third sync reports all `unchanged` (the fast path still fires after a deferral-mediated write).
- Module docstring: the incident reference, the deferral (`defer(Document.content)`, one column, no `raiseload` — A1/A2), the test→observable mapping, the run-in-isolation command, prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` reviewed in full against the post-phase app; every inconsistency fixed, pre-existing included. `README.md` gains the one-sentence operations note (a full re-sync walk is metadata-bound; the summary-NULL backfill intentionally re-reads the stored content once).
3. **The full gates:**
- `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` and `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` — green in isolation (the fast path's UI neighbours).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `perf(import): defer Document.content on the unchanged fast path so a full walk no longer hydrates the KB` — covering `app/rag/importer.py`, the integration + E2E files, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/185_sync_defer_content_fast_path/`.
## Testing & Quality
- E2E: the dedicated suite + the two regression E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` green **in isolation** (backfill, full text, changed-file re-index).
- [ ] `tests/e2e/test_document_dates.py` and `tests/e2e/test_sync_changed_only.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,68 @@
# Phase 186 — The commit-date walk streams instead of buffering the whole `git log`
**Source:** owner incident 2026-10-15 (as phases 183–185) — `_run_sync` builds `doc_dates_by_root` from a full-history `git log --name-only` captured ENTIRELY in memory, per source, and keeps the result for the whole run.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; sequenced after 183–185 so the sync's memory work lands together.
**Context:** `scripts/git_sync.py` — the module docstring's **A11 "every git command goes through `run_git`" claim at L31**, `clone_or_pull` (L96), `_unshallow_if_shallow` (L221), **`_parse_commit_dates(output: str)` (L240)** — `for line in output.splitlines()` at L245-ish, the first-sighting-wins `dates.setdefault(path, commit)`, the `ValueError` paths; **`file_commit_dates` (L270)** — `output = run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)` (L298), the two fail-soft `except` arms returning `{}` with the `file_commit_dates: …` warnings, `run_git` (**L320**, `subprocess.run(argv, cwd=cwd, capture_output=True, text=True, env=env)` at **L342**, `GitSyncError` with `proc.stderr` on non-zero); `app/api/sync.py` — `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)` (**L646**, phase 179), the dict's consumer `dates_map = (doc_dates_by_root or {}).get(str(root), {})` in `app/rag/importer.py` (**L913**), `raw_date=dates_map.get(rel)` passed to `_index_file` (**L945–948**), the mtime fallback `file_mtime_datetime` (`app/rag/doc_dates.py` L57) and `normalize_doc_date`; `app/rag/doc_dates.py` (pure, unchanged); `tests/unit/test_git_sync.py` (the `_parse_commit_dates` + argv pins, which call it with **strings**), `tests/integration/test_git_file_dates.py`, `tests/integration/test_importer_dates.py`, `tests/e2e/test_git_source_dates.py`, `tests/e2e/test_document_dates.py`.
## Measured baseline (2026-10-15, the incident box's own checkouts, read-only `git` inside the app container)
| checkout | commits | `git log --name-only --format=@@%cI` lines | bytes |
|---|---|---|---|
| litellm | 54,502 | 346,829 | 15,077,351 |
| llama.cpp | 11,659 | 69,922 | 1,867,046 |
| vllm-radiance | 192 | 4,855 | 387,672 |
| gufo | 589 | 9,155 | 327,032 |
`run_git` returns that as ONE `str`; `_parse_commit_dates` then calls `.splitlines()`, materialising a second copy as a list of 346,829 `str` objects — tens of MB of transient peak per source on a 3,904 MB box, before the retained dict. The source removed that morning (`server`, nextcloud) is the case that shows how unbounded this is: the walk scales with the repository's ENTIRE history, not with its working tree.
## Objective
Stream the walk: parse the `git log` output line by line from the subprocess pipe so the run never holds the whole output, keep the phase-106 D2 contract byte-for-byte (first-sighting-wins last-commit dates, the same fail-soft arms, the same warning lines, mtime fallback), and amend the A11 single-site claim so the streaming variant is a documented, tested sibling rather than an exception hiding in the module.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `stream_git_lines(argv, cwd)` is the second git site, and A11 says so (task 02).** A generator in `scripts/git_sync.py`:
```python
def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
"""Stream one git command's stdout, line by line (phase 186, A11 sibling)."""
```
`subprocess.Popen(argv, cwd=cwd, stdout=PIPE, stderr=PIPE, text=True)`; stdout is iterated directly (`for line in proc.stdout`); **stderr is drained by a daemon thread into a list** so a chatty stderr can never deadlock the pipe while stdout is being consumed (the classic `Popen` + two-pipes trap — this is the reason not to `communicate()` at the end and not to `stderr=STDOUT`, which would pollute the parse). On `proc.wait() != 0` it raises `GitSyncError(f"git {' '.join(argv[1:])} failed (exit …): {stderr}")` — the SAME message shape as `run_git` (L344–347), so every caller's fail-soft handling is unchanged. The module docstring's L31 claim becomes: `run_git` is the buffered site, `stream_git_lines` the streaming one, and **both** are the only places a `subprocess` is spawned (a unit pin greps `subprocess.` occurrences in the file — the phase-179 AGENTS rule 11 "one git-invocation site" intent survives as "one for each shape, both owned here").
- **D2 — the parser takes any iterable; strings keep working (task 01 first).** `_parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]`: `if isinstance(lines, str): lines = lines.splitlines()` and everything else — the `@@%cI` commit-header rule, the whitespace path split, the `\`→`/` normalisation, the leading-`/` strip, `setdefault` first-sighting-wins, both `ValueError` arms — stays byte-identical, so `tests/unit/test_git_sync.py`'s existing string pins need no edits.
- **D3 — `file_commit_dates` consumes the stream, fail-soft unchanged (task 02).** `return _parse_commit_dates(stream_git_lines([...], cwd=dest))` inside the existing `try`, the existing `except GitSyncError` → the same warning + `{}`, the existing `except ValueError` → the same warning + `{}`. Two consequences documented in the docstring, not hidden: a mid-stream failure now surfaces AFTER some lines were parsed, and the method still returns `{}` because the caller falls back to mtimes (the phase-106 D2 fail-soft contract is about the RESULT, and the result is unchanged); and the returned dict — one entry per path in the repository's history — remains retained by `_run_sync` in `doc_dates_by_root` for the run (A3: out of scope; the dict is ~10% of the buffered cost this phase removes, and bounding it changes dates, not just memory).
- **D4 — the pins are comparative and deterministic (tasks 01 + 02).** Unit: a one-shot generator input proves single-pass consumption (a generator that raises on a second `iter()`), and the lazy-vs-buffered assertion measures `tracemalloc` peak for the SAME synthetic corpus (≈300k lines) through (a) one big `str` + `splitlines()` and (b) a generator, asserting `streamed < buffered * 0.6` with both numbers printed in the failure message. Integration: `tests/integration/test_git_file_dates.py` extended with a generated two-commit repo proving the post-fix dates equal the pre-fix dates for (i) a file only in commit 1, (ii) a file modified in commit 2 (its date is commit 2's), (iii) a path in a subdirectory, (iv) a file absent from history entirely (mtime fallback).
- **D5 — the E2E pins the owner-visible date (task 03).** `tests/e2e/test_git_log_stream_dates.py`: a local `file://` fixture repo with two dated commits (the `test_git_source_dates.py` `_git` helper shape, `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` pinned so the dates are exact), synced from the Sources page; the document date shown for the modified file equals commit 2's date and the untouched file shows commit 1's date (the phase-106 contract, now through the stream), a file git never committed falls back to its mtime, and the D1 manual-date lock still survives a re-sync. Whatever the walk currently does with a quoted non-ASCII path is recorded in the module docstring as-is and pinned as UNCHANGED behaviour (this phase does not change the parser's quoting semantics — it only changes how the bytes arrive).
- **NOT touched:** the `git log` ARGV (`--name-only --format=@@%cI` stays, pinned by the existing argv unit tests), `clone_or_pull` / `_unshallow_if_shallow` (they stay on `run_git`), `app/rag/doc_dates.py` (pure), the importer's `dates_map.get(rel)` lookup, the mtime fallback, the sync's cancel semantics (phase 134/179), the frontend.
## Dependencies
— (none functionally; run after `183`/`184`/`185`)
## Tasks
1. `01_parser_accepts_lines.md` — `_parse_commit_dates` takes `str | Iterable[str]` with byte-identical semantics, the one-shot-generator + comparative-memory unit pins, the existing string pins untouched.
2. `02_stream_git_log.md` — `stream_git_lines` (Popen + stderr thread + the `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream, the A11 docstring amendment, the grep pin, the integration date-parity tests.
3. `03_e2e_streamed_dates.md` — `tests/e2e/test_git_log_stream_dates.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_git_sync.py` — existing string pins green unedited; the new iterable/one-shot/lazy-memory pins; the `subprocess.` occurrence grep pin.
- **Integration:** `tests/integration/test_git_file_dates.py` (date parity on a generated repo, the fail-soft arms still returning `{}` with the same warnings), `tests/integration/test_importer_dates.py`, `test_import_docs_git.py`, `test_sync_api.py` (the `file_commit_dates` monkeypatch points keep working — `app.api.sync` imports the name, phase 179's probe still patches it).
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_git_log_stream_dates.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` plus full coverage of `scripts/git_sync.py`'s new function (the existing `tests/unit/test_git_sync.py` family covers this module today — keep it covered, not just `app/`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `AGENTS.md` rule 11 (the "never block the event loop" rule already cites `run_git`): one clause noting the streaming sibling exists and that BOTH are off-loop via `asyncio.to_thread` (phase 179) — do not rewrite the rule, extend the reference. `README.md`: nothing user-facing changes; add the ops note that the commit-date walk streams (so a huge repo no longer needs the whole `git log` in RAM) and that the per-source date dict is still held for the run (A3).
## Completion Criteria
- [ ] `_parse_commit_dates` accepts `str | Iterable[str]` with byte-identical parse semantics; the existing string-based unit pins are green with no edits.
- [ ] `stream_git_lines` exists, drains stderr on a thread, raises the `run_git`-shaped `GitSyncError`; `file_commit_dates` streams; the module docstring's A11 claim names both sites and the grep pin enforces it.
- [ ] The lazy-vs-buffered memory comparison and the one-shot generator pins are green with their measured numbers in the test docstrings.
- [ ] `uv run pytest tests/integration/test_git_file_dates.py tests/unit/test_git_sync.py -v` green (date parity across all four cases, both fail-soft arms unchanged).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(sync): stream the git commit-date walk instead of buffering git log in RAM`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — stream, do not shrink the argv.** No `--since`, no `--max-count`, no per-file `git log -1`: every one of those changes WHICH dates the KB stores, and the phase-106 D2 last-commit-date contract is a data contract, not a performance knob.
- **A2 — stderr on a worker thread.** The deadlock-free way to stream stdout while still reporting git's stderr; `stderr=STDOUT` would corrupt the parse, `stderr=DEVNULL` would lose the error text `GitSyncError` carries today.
- **A3 — the retained `doc_dates_by_root` dict is out of scope.** Bounding it (by age, by working-tree membership, or by streaming it to a temp file) changes document dates or adds a new subsystem; this phase removes the transient whole-`git log` copy, which is the unbounded piece.
- **A4 — `run_git` keeps its shape.** Every other caller (`clone_or_pull`, `_unshallow_if_shallow`, `app/core/docs_push.py`) keeps the buffered site; migrating them is churn without a measured cost.
@@ -0,0 +1,26 @@
# Task 01 — The parser takes a stream (semantics untouched)
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Widen `_parse_commit_dates` to accept any iterable of lines (keeping `str` working so every existing pin is untouched), and pin in tests that it consumes its input exactly once and that streaming allocates materially less than the buffered path for the same corpus.
## Work
1. `scripts/git_sync.py` — `_parse_commit_dates` (**L240**): signature `def _parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]:` with the single line `if isinstance(lines, str): lines = lines.splitlines()` at the top; rename the loop variable to match; `from collections.abc import Iterable` added. Every parsing rule stays byte-identical: the `@@`-prefixed `%cI` header (`datetime.fromisoformat(line[2:])`), the blank-line skip, `line.split()` for multiple paths per line, `raw_path.replace("\\", "/").lstrip("/")`, `dates.setdefault(path, commit)` (newest-first, first sighting wins), and BOTH `ValueError` arms (a malformed date, a path line before any commit header). The docstring gains: "*lines* may be a string or any iterable of lines (phase 186 — `stream_git_lines` feeds it a live pipe); a string input behaves exactly as before."
2. `tests/unit/test_git_sync.py` — extend (existing string-based tests untouched):
- `test_parser_accepts_a_line_iterable`: the same corpus the existing tests use, fed as `iter([...])`, produces the identical dict.
- `test_parser_consumes_exactly_once`: a one-shot generator wrapper that raises `AssertionError("re-iterated")` if `iter()` is called twice on it; parse succeeds → proves the parser does not materialise-then-rescan (and rules out an accidental `list(lines)` inside the loop).
- `test_streamed_parse_alloculates_less_than_buffered`: build ONE synthetic corpus of ~300,000 lines in the exact real shape (3,000 commits × 100 paths, `@@2026-01-01T00:00:00+00:00` headers), then measure `tracemalloc` peak over (a) `_parse_commit_dates(one_big_str)` (the buffered path's own `splitlines()`) and (b) `_parse_commit_dates(iter_of_lines_from_a_generator)` (each line built lazily from a fixed seed), asserting `streamed_peak < buffered_peak * 0.6` with both numbers in the assertion message. Record both numbers + the machine in the docstring; if the real ratio on this box differs, keep the 0.6 factor only if the pre/post split genuinely clears it and document the measured ratio (never weaken to `<= buffered_peak`, which is a tautology).
- Corpus justification in the module docstring (not a test): the incident box's measured sizes — litellm 54,502 commits / 346,829 lines / 15 MB; llama.cpp 69,922 lines; vllm-radiance 4,855; gufo 9,155 — which is why the synthetic corpus is ~300k lines.
3. Verify: `uv run pytest tests/unit/test_git_sync.py -v`; `uv run pytest` green.
## Testing & Quality
- Unit: the three new pins + every existing pin green unedited.
- Coverage: **>90%** on `app/` and `scripts/git_sync.py`'s parser stays fully covered.
## Completion Criteria
- [ ] `_parse_commit_dates` accepts `str | Iterable[str]`; no parsing rule changed; the docstring says so.
- [ ] The one-shot-generator pin is green (no re-iteration, no hidden `list()`).
- [ ] The streamed-vs-buffered memory comparison is green with both measured peaks recorded in the docstring.
- [ ] `uv run pytest tests/unit/test_git_sync.py -v` and `uv run pytest` green.
@@ -0,0 +1,37 @@
# Task 02 — The fix: `stream_git_lines` + `file_commit_dates` off the buffer
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Add the streaming git site and put the commit-date walk on it, so no `git log` output is ever held as one string — with `run_git`'s error contract, the phase-106 D2 fail-soft result, and the A11 single-site rule kept intact (as two documented siblings).
## Work
1. `scripts/git_sync.py` — new function next to `run_git` (L320):
```python
def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
```
- `proc = subprocess.Popen(argv, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=None)` (the same env semantics as `run_git`: `None` inherits — do NOT introduce the env-replacement trap the module docstring warns about).
- stderr drained by a daemon thread appending to a list, started before stdout is consumed (A2: the deadlock-free shape — never `communicate()` after iterating stdout, never `stderr=STDOUT`, never `stderr=DEVNULL`).
- `for line in proc.stdout or (): yield line` (guard the `text=True` optional type for pyright), then `proc.stdout.close()`, `err_thread.join()`, `rc = proc.wait()`; `if rc != 0: raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {rc}): " + "".join(stderr_lines).strip())` — the message shape copied from `run_git` L344–347 so every caller's fail-soft text is unchanged.
- `finally`-safe: if the generator is closed early (`GeneratorExit`), terminate the process (`proc.terminate()` guarded by try/except) so a partial consume never leaks a `git` child.
- Module docstring: amend the **L31** claim → `run_git` (buffered) and `stream_git_lines` (streaming, phase 186) are the ONLY two places a git subprocess is spawned; both own the argv→`GitSyncError` contract; both are reached from async code only through `asyncio.to_thread` (phase 179 / AGENTS rule 11).
2. `file_commit_dates` (**L270**): `output = run_git([...])` (L298) → `return _parse_commit_dates(stream_git_lines(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest))` inside the existing `try`, keeping BOTH `except` arms and their exact warning strings. Add the docstring notes: the walk streams (no whole-`git log` in RAM — the incident's transient peak); a mid-stream failure still yields `{}` + the mtime fallback (the D2 contract is about the RESULT); and the returned per-source dict stays retained by `_run_sync` for the whole run — **A3, out of scope, named**.
3. `tests/unit/test_git_sync.py` — add:
- `test_stream_git_lines_yields_lines_and_raises_run_git_shaped_error` — run it against a real throwaway git repo (the house `_git` helper): success path yields the expected lines; `git log` in a non-repo dir raises `GitSyncError` whose message matches `^git log failed \(exit \d+\): ` (the `run_git` shape).
- `test_stream_git_lines_reports_stderr_without_deadlocking` — a command that writes >100 KB to stderr (e.g. `git -c core.quotepath=bogus …` is contrived; use `git log` in a repo with a corrupt arg that spews usage, or `git` with `--help` on stderr via a bad flag) and assert the call raises with a non-empty stderr excerpt and completes in a bounded time (a wall-clock guard, e.g. `< 10 s`).
- `test_only_two_git_spawn_sites` — read `scripts/git_sync.py` and assert `subprocess.` occurrences are exactly inside `run_git` and `stream_git_lines` (the A11 grep pin), and that no other module in `app/` or `scripts/` spawns `subprocess` for git (`grep`-style assertion over the repo; if the docs-push path legitimately uses `run_git`, it passes by construction).
- Keep every existing `run_git`/argv pin green (the `git log` argv is unchanged — A1).
4. `tests/integration/test_git_file_dates.py` — extend with the date-parity fixture (a generated repo, `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` pinned per commit): file only in commit 1 → commit 1's date; file modified in commit 2 → commit 2's date; a file in a subdirectory → its own last-commit date; a file that exists but was never committed → `{}` entry, so the importer's mtime fallback applies (assert the empty lookup, not the resulting date, to keep this test about `file_commit_dates`). Also `test_file_commit_dates_fail_soft_unchanged`: a non-directory and a non-repo directory both return `{}` with the existing warning lines (wording pinned).
5. Verify: `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v`; then `uv run pytest`.
## Testing & Quality
- Unit: the streaming site (success, error shape, stderr safety, spawn-site grep).
- Integration: date parity + both fail-soft arms + the phase-179 `app.api.sync.file_commit_dates` monkeypatch point still exercised by `test_sync_git_freeze.py`.
- Coverage: **>90%** on `app/`; `scripts/git_sync.py` fully covered by the unit family.
## Completion Criteria
- [ ] `stream_git_lines` exists with the stderr-thread + terminate-on-close shape and the `run_git`-shaped `GitSyncError`; the module docstring names both spawn sites and the grep pin enforces it.
- [ ] `file_commit_dates` streams, returns byte-identical dicts for the parity fixture, and keeps both fail-soft arms and warnings.
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` green.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,34 @@
# Task 03 — E2E: the dates the owner reads come through the stream (the phase's dedicated Playwright suite)
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Pin the phase-106 D2 promise through the browser with the streaming walk: after syncing a real two-commit `file://` repo, the dates shown for each document are that file's LAST-COMMIT dates (not the sync date, not the clone date), uncommitted files fall back to mtime, and a manual date correction survives a re-sync. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_git_log_stream_dates.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_git_source_dates.py` + `test_document_dates.py` shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, `BOR_GIT_SOURCES=file://<fixture repo>`, the shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`, the `_git` helper with `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` set per commit so dates are exact and stable).
- Fixture repo: commit A (dated 2026-01-05) adds `notes/old.md` + `notes/changed.md`; commit B (dated 2026-03-20) modifies `notes/changed.md` only; plus an untracked `notes/untracked.md` written directly into the CHECKOUT after cloning (or, if the sync re-clones, into the source dir the run walks — whichever the house fixture model uses; state which in the docstring).
- **Test 1 `test_document_dates_are_the_last_commit_dates`:** sync from the Sources page → `notes/old.md`'s document date is 2026-01-05 and `notes/changed.md`'s is 2026-03-20 (assert via `GET /api/docs/{id}` through `page.request` and the document page's date surface), neither equal to the run's own timestamp.
- **Test 2 `test_uncommitted_file_falls_back_to_mtime`:** the untracked file's document date equals its file mtime (the fail-soft contract through the streaming path) and the run reaches `success` with no error banner.
- **Test 3 `test_manual_date_survives_a_resync` (the D1 lock):** set a document's date through the existing date-edit surface, re-sync, and the manual date is unchanged while the other documents refresh from the stream.
- **Test 4 `test_resync_is_unchanged_and_dates_hold`:** a second sync with nothing changed reports all `unchanged` and identical dates (the stream re-parses, it does not drift).
- Non-ASCII/quoted path behaviour: record what the walk does TODAY with a `core.quotepath`-quoted path in the module docstring and pin it as UNCHANGED behaviour (whatever the run shows — this phase changes transport, not parsing; if the behaviour is wrong, note it for a future phase rather than fixing it here).
2. **Docs pass:** `README.md` + `AGENTS.md` in full, every inconsistency fixed (pre-existing included). `AGENTS.md` rule 11 gains the one-clause reference to `stream_git_lines` (both sites off-loop via `asyncio.to_thread`); `README.md` gains the ops note (the commit-date walk streams; the per-source date dict is still held for the run — phase 186 A3).
3. **The full gates:**
- `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` and `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` — green in isolation (the date families this phase must not disturb).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `perf(sync): stream the git commit-date walk instead of buffering git log in RAM` — covering `scripts/git_sync.py`, the unit + integration + E2E files, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/186_git_log_streaming_dates/`.
## Testing & Quality
- E2E: the dedicated suite + the two date E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` green **in isolation** (last-commit dates, mtime fallback, the D1 manual lock, the unchanged re-sync).
- [ ] `tests/e2e/test_git_source_dates.py` and `tests/e2e/test_document_dates.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the rule-11 clause included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,54 @@
# Phase 187 — Make the sync's memory a logged, visible number
**Source:** owner incident 2026-10-15 — reconstructing "how much RAM did the sync actually use?" meant reading `systemd` unit-teardown lines from three boots (`brain-of-code-app.service: … 3.3G memory peak`), and boot −1 (the run that killed the box) had NO teardown record at all because the VM never shut down cleanly.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; lands after 183–186 so the numbers it reports describe the fixed sync.
**Context:** `.agents/PLAN.md` §9 Observability (the log-format authority — PLAN is never edited by an agent, so the new fields are documented in `README.md` + `AGENTS.md` rule 10 and the owner reconciles PLAN); `AGENTS.md` rule 10 ("Amply log, visibly feed back"); `app/core/response_budget.py` (the house precedent this phase mirrors: a pure-stdlib `app/core/` module, one greppable logger name — `grep ' :: api '`, `BOR_LOG_LARGE_RESPONSE_BYTES`, and `scripts/api_payload_audit.py` as the offline audit); `app/api/sync.py` — the `SyncStatus` dataclass (**L276**, phase-64 "keep the final counts in the terminal" convention), `sync_status()` (**L385**, its additive-keys contract "Additive only — every pre-existing key keeps its name, type, and meaning"), the `sync: started repos=%d origin=%s git=%d local=%d` line (**L591–594**), the phase transitions (`_status.phase = "import"` **L718**, `"overview"` **L743**, `"summaries"`), `sync: done detail=%s` (**L903**), the cancel terminal's `sync: cancelled by user state=cancelled files_done=%d files_total=%d` (**L933–936**), the failed terminal (`logger.exception("sync: failed")` **L938**); `app/rag/importer.py` — `ImportSummary.format_counts` (**L513**) + `ImportSummary.log()` (**L520**, the cron-safe greppable summary line PLAN §9 names); `frontend/assets/sources.js` — the sync progress box + result line (`setSyncProgress(…)` around **L1325**, the label/counts helper at **L1080–1100** and **L1378**, the aria-live announcements), `frontend/index.html` (the progress box markup), `tests/unit/test_sync_api*.py` / `tests/integration/test_sync_api.py` (the status-shape pins), `tests/e2e/test_sync_summary_visibility.py`, `tests/e2e/test_sync_progress_box.py`, `tests/e2e/test_sync_status_freshness.py`, `tests/e2e/test_sync_cancel.py`.
## Objective
Make "how much memory did that sync use?" answerable from one log line and visible on the page that started it: a small `app/core/memory.py` reading current + peak RSS, appended to every sync lifecycle line and the importer's summary line, plus an additive `rss_peak_mb` key on `GET /api/sync/status` and a `peak <N> MB` clause on the Sources page's sync result line.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `app/core/memory.py` never raises and never lies.** Three pure helpers: `rss_mb() -> int | None` (Linux: parse `VmRSS` from `/proc/self/status`; non-Linux/failure: `resource.getrusage(RUSAGE_SELF).ru_maxrss // 1024` as the best available proxy), `rss_peak_mb() -> int | None` (`ru_maxrss // 1024` — the process high-water mark), and `cgroup_peak_mb() -> int | None` (read `/sys/fs/cgroup/memory.peak` when present, else `None`). Docstring states the honest nuance the incident turned on: **inside a container the process high-water and the host's cgroup `memory.peak` are different numbers, and the cgroup number includes page cache** (the incident box's app cgroup showed 2.16 GB of `file` against 211 MB of `anon` while idle) — so the app logs the PROCESS number and the operator's dashboard may show a larger one; neither is wrong, and the log says which it is.
- **D2 — the format is append-only, in the §9 style (task 02).** `sync: started … rss_mb=N`, and every terminal (`sync: done detail=… rss_mb=N rss_peak_mb=M`, `sync: cancelled by user … rss_mb=N rss_peak_mb=M`, and the failed funnel) gains the same two trailing fields; `ImportSummary.log()` gains a trailing ` rss_mb=N`. A `None` renders as `rss_mb=?` — a helper `fmt_mb(v: int | None) -> str` keeps that in one place. No existing field is renamed, reordered or re-typed (the phase-167/182 append-only convention), so every existing `grep` keeps working.
- **D3 — sampled, not polled per file (task 03).** `_status` gains `rss_mb: int | None` and `rss_peak_mb: int | None`, reset at run start; sampled at the run start, at each `_status.phase` transition (`import` L718, `overview` L743, `summaries`), and in EVERY terminal (which keep the final value next to the counts — the phase-64 convention). The per-file progress hook samples at most once per 5 s via a `time.monotonic()` guard, so a 27k-file walk costs ~600 `/proc` reads, not 27k, and the operator watching the page sees the number move (PLAN §7.4's never-stale contract).
- **D4 — the status key and the UI clause are additive and admin-only (task 03).** `sync_status()` returns two new keys (`rss_mb`, `rss_peak_mb`, `int | null`); `sources.js` appends ` · peak 512 MB` to the existing result/counts line — a clause on text that already exists, never a new control, never replacing a state string, and the aria-live announcement keeps its current wording (a screen-reader user gets the same announcement plus the clause; the peak is supplementary, so it must not become the announcement's only content). The Sources view is admin-only (PLAN B5), which is the exposure guard; no memory number appears on any anonymous surface (chat, shared chat, health).
- **D5 — an offline reader for the log (task 02, small).** Document `grep ' :: ' | grep ' rss_peak_mb='` in `README.md`'s operations section. NO new audit script: `scripts/api_payload_audit.py` exists for bytes; a memory equivalent would be a second parser for one number. Explicitly out of scope (A3).
- **NOT touched:** `PLAN.md` (never), the sync pipeline's sequence/errors/cancel semantics, the response-budget logger, the chat's per-turn line (this phase is about the sync), the container/quadlet memory limits (owner's own task), the retrieval path.
## Dependencies
— (none; after 183–186 in the roadmap so the reported numbers describe the bounded sync)
## Tasks
1. `01_memory_helper.md` — `app/core/memory.py` (the three helpers + `fmt_mb`), unit-tested with `/proc` and `resource` stubbed (Linux path, missing-file path, garbage-file path, non-Linux fallback).
2. `02_sync_log_lines.md` — the four sync lifecycle lines + `ImportSummary.log()` gain `rss_mb=` / `rss_peak_mb=`, the `fmt_mb` single-sourcing, the log-format assertions in the integration tests, the README operations note.
3. `03_status_key_and_ui.md` — the `SyncStatus` fields + sampling (start, phase transitions, terminals, the 5 s-throttled progress sample), the two additive `sync_status()` keys, the `sources.js` clause + aria-live behaviour, the shape pins.
4. `04_e2e_peak_memory.md` — `tests/e2e/test_sync_peak_memory_visible.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_core_memory.py` — every helper branch (real `/proc` on Linux + stubbed failure modes, the `None` rendering, the kB→MB arithmetic), `tests/unit/test_settings*` untouched.
- **Integration:** `tests/integration/test_sync_api.py` — the lifecycle lines carry the fields (caplog assertions on the `sync: started` / `sync: done` / `sync: cancelled` / `sync: failed` shapes with `rss_mb=?` accepted on a platform without `/proc`), the status payload's two new keys with their types and the terminal-keeps-final-value convention, every pre-existing key byte-identical.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_sync_peak_memory_visible.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `AGENTS.md` rule 10: the required sync lines now carry `rss_mb=` / `rss_peak_mb=` (extend rule 10's parenthetical, do not rewrite it). `README.md`: the operations grep, what each number means, and the container-vs-cgroup caveat (page cache included in the host's number) written in one short paragraph. PLAN §9 is the owner's to reconcile — flag it in the phase report, do not edit it.
## Completion Criteria
- [ ] `app/core/memory.py` exists with the four documented functions, every branch unit-tested, and never raises.
- [ ] All four sync lifecycle lines + `import: summary …` carry `rss_mb=` (and the terminals `rss_peak_mb=`), appended after the existing fields; no existing field changed.
- [ ] `GET /api/sync/status` exposes `rss_mb` / `rss_peak_mb` (`int | null`), keeps the final value in every terminal, and every pre-existing key is unchanged.
- [ ] The Sources page's sync result line shows the ` · peak <N> MB` clause with no new control and no regression in the existing state strings.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the PLAN §9 reconciliation flagged, not performed).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(observability): log and show the sync's RSS so memory is a number, not a forensic exercise`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — the app logs PROCESS RSS, not the cgroup number**, with the caveat documented (`anon` vs `file` differ by an order of magnitude and the cgroup figure includes page cache — demonstrated during the incident investigation).
- **A2 — append-only to the existing lines and keys.** No new log line type, no renamed field, no separate metrics endpoint (A12: no aux services), no Prometheus.
- **A3 — no memory audit script** this phase; the grep is documented instead.
- **A4 — sampling is throttled (5 s) and never per-file**, and the UI clause is text on a line that already exists on an admin-only view.
@@ -0,0 +1,26 @@
# Task 01 — `app/core/memory.py`
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
One pure-stdlib module that answers "how much RAM is this process using, and what was its high-water mark" without ever raising, mirroring `app/core/response_budget.py`'s shape (no FastAPI, no imports from `app.api`, logger only if needed).
## Work
1. `app/core/memory.py`:
- `rss_mb() -> int | None` — parse `VmRSS` from `/proc/self/status` (kB → MB, integer, floor); any `OSError`/`KeyError`/parse failure returns `None`. Non-Linux fallback: `resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024` (documented as a high-water proxy, not a current value).
- `rss_peak_mb() -> int | None` — `resource.getrusage(RUSAGE_SELF).ru_maxrss // 1024`, `None` only if `resource` is unavailable (wrap in a guarded import for platforms without it).
- `cgroup_peak_mb() -> int | None` — read `/sys/fs/cgroup/memory.peak` (bytes → MB) when it exists and parses, else `None`. Docstring: inside a container this file is usually absent (the host reads it), and the host's number counts PAGE CACHE too — the incident box's app cgroup showed ~2.16 GB `file` against ~211 MB `anon` while idle, and `systemd` reported `3.3G memory peak` for the unit while the uvicorn process was a fraction of that. This module therefore reports PROCESS numbers and this distinction is written here once, so no future reader conflates them.
- `fmt_mb(v: int | None) -> str` — `"?"` for `None`, else `str(v)` (so log lines never print `rss_mb=None`).
- Module docstring: the incident (one paragraph), the three helpers, the never-raise contract, and the container/cgroup caveat above.
2. `tests/unit/test_core_memory.py` — every branch, with the filesystem and `resource` stubbed: a real read (Linux runner: `/proc/self/status` has `VmRSS`, so `rss_mb()` is a positive int), a missing file, an unparseable file, a file without a `VmRSS` line, the non-Linux `resource` fallback path (patch the `/proc` open to raise + patch `getrusage`), `rss_peak_mb()`'s arithmetic, `cgroup_peak_mb()` present/absent/garbage, `fmt_mb(None) == "?"` / `fmt_mb(0) == "0"` (0 is a value, not missing).
3. Verify: `uv run pytest tests/unit/test_core_memory.py -v`; `uv run ruff check . && uv run pyright` clean; `uv run pytest` green.
## Testing & Quality
- Unit: 100% branch coverage of the new module (well above the >90% gate).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `app/core/memory.py` with the four functions, the never-raise contract, and the container/cgroup caveat documented.
- [ ] `uv run pytest tests/unit/test_core_memory.py -v` green, every branch covered.
- [ ] No import cycle (`app.core.memory` imports stdlib only — prove it in the module docstring's "pure stdlib" line and by the absence of `app.` imports).
@@ -0,0 +1,28 @@
# Task 02 — The lifecycle lines carry the numbers
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Append `rss_mb=` to the sync's start line, `rss_mb=`/`rss_peak_mb=` to all three terminals, and `rss_mb=` to the importer's cron-safe summary line — append-only, so every existing grep and every existing field keeps working.
## Work
1. `app/api/sync.py` — the four lifecycle log sites, using `app.core.memory.rss_mb/rss_peak_mb/fmt_mb` (imported at module level, matching the file's import style):
- **L591–594** `sync: started repos=%d origin=%s git=%d local=%d` → append ` rss_mb=%s` (`fmt_mb(rss_mb())`).
- **L903** `sync: done detail=%s` → append ` rss_mb=%s rss_peak_mb=%s`.
- **L933–936** `sync: cancelled by user state=cancelled files_done=%d files_total=%d` → append the same two fields.
- **L938** the failed funnel: `logger.exception("sync: failed")` stays as-is (the traceback), and the FAILED line gains a dedicated `logger.error("sync: failed state=failed files_done=%d files_total=%d rss_mb=%s rss_peak_mb=%s error=%s", …)` immediately after the `_status.error = _sanitize_error(str(e))` assignment (sanitized exactly like the status field — AGENTS/security: the raw exception string never reaches the log un-sanitized). Keep `logger.exception` so the traceback survives.
2. `app/rag/importer.py` — `ImportSummary.log()` (**L520**): append ` rss_mb=%s` to the existing `import: summary …` line (the `format_counts()` helper at L513 stays untouched — the memory field belongs to the LOG line, not the counts formatter, so the CLI's other consumers of `format_counts()` are byte-identical).
3. `tests/integration/test_sync_api.py` — caplog assertions (the house caplog pattern): after a run to `success`, the captured records include a `sync: started … rss_mb=` line and a `sync: done … rss_mb=… rss_peak_mb=…` line whose values are `?` or a positive integer; the same for the cancelled terminal (reuse the existing cancel test's fixture) and the failed terminal (reuse the existing failure test — `sync: model check failed` or a stubbed raise). Assert the PRE-existing prefixes are unchanged (`line.startswith("sync: started repos=")` etc.) — the append-only pin.
4. `tests/integration/test_importer_e2e.py` (or the importer family's log-pin home): the `import: summary …` line ends with ` rss_mb=<int|?>` and its earlier fields are unchanged.
5. `README.md` — the operations grep (`grep ' :: ' … | grep ' rss_peak_mb='`), the field meanings, and the process-vs-cgroup caveat in one short paragraph (D1/A1). `AGENTS.md` rule 10: extend the parenthetical to mention the sync's new fields (do not rewrite the rule).
6. Verify: `uv run pytest tests/integration/test_sync_api.py tests/integration/test_importer_e2e.py -v`; `uv run pytest`; ruff + pyright.
## Testing & Quality
- Integration: the four lifecycle terminals + the importer summary line pin the fields and the unchanged prefixes.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All four sync lifecycle lines and `import: summary …` carry the new fields, appended after the existing ones; the failed terminal gains the sanitized one-line summary while `logger.exception` still writes the traceback.
- [ ] `uv run pytest tests/integration/test_sync_api.py -v` green (fields present, pre-existing prefixes byte-identical).
- [ ] `README.md` + `AGENTS.md` rule 10 updated; `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,30 @@
# Task 03 — The status key and the Sources-page clause
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Put the number where the operator is: two additive keys on `GET /api/sync/status` (sampled at the run start, each phase transition, every terminal, and at most every 5 s during the walk) and a ` · peak <N> MB` clause on the Sources page's existing sync result line.
## Work
1. `app/api/sync.py`:
- `SyncStatus` (**L276**): two new fields `rss_mb: int | None = None`, `rss_peak_mb: int | None = None`, documented as phase-187 additive state with the phase-64 "terminals keep the final value" convention and the run-start reset.
- A single private sampler on the module: `_sample_rss(force: bool = False) -> None` — reads `app.core.memory.rss_mb()` / `rss_peak_mb()`, stores both, keeps a `time.monotonic()` guard so a non-forced call is a no-op within 5 s (D3/A4); forced at the run start, each `_status.phase = "import"|"overview"|"summaries"` transition (**L718** and friends), and every terminal (success L903 path, cancelled L933 path, failed L938 path) so the final value is exact.
- `sync_status()` (**L385**): append `"rss_mb"` and `"rss_peak_mb"` to the dict (the file's own words: "Additive only — every pre-existing key keeps its name, type, and meaning") and document the two keys + the terminal convention in its docstring.
- The per-file `_hook` (L677) calls `_sample_rss()` (unforced → throttled).
2. `frontend/assets/sources.js`: on the terminal render path, append the clause to the EXISTING result/counts text — ` · peak 512 MB` — sourced from `status.rss_peak_mb ?? status.rss_mb`, and render NOTHING when both are null (never `peak ? MB`). Touch the result-line helper used at **L1080–1100** / **L1378** and the terminal/announce path around **L1314–1330**; the existing state strings ("Sync cancelled", the counts line, the error copy) keep their exact wording with the clause appended, and the aria-live announcement keeps announcing what it announces today plus the clause (no new announcement). No new DOM control, no layout shift on the 375px viewport (the clause wraps inside the existing line's `.cell-clip`-free text; verify visually in the E2E at mobile width).
3. Pins:
- `tests/integration/test_sync_api.py`: the payload carries both keys with `int | None`; after a run the terminal payload's `rss_peak_mb` is non-null on Linux and `>= rss_mb` when both are set; idle (never-run) reports `null`/`null`; every pre-existing key/value shape asserted unchanged (the additive pin); anonymous `GET /api/sync/status` still 403 (no new anonymous surface — A4).
- `tests/unit/test_sync_status_shape.py` (or this file's existing home for the status-shape matrix): the key set is exactly the old set ∪ {`rss_mb`, `rss_peak_mb`}.
- `frontend`: the clause appears in the terminal line and NOT in `running`; a null peak renders no clause.
4. Verify: `uv run pytest tests/integration/test_sync_api.py -v`; the three sync E2E neighbours in isolation (`test_sync_summary_visibility.py`, `test_sync_progress_box.py`, `test_sync_status_freshness.py`) still green (they assert the existing strings — the clause must not break them; if one asserts an exact full-line equality, update it to the clause-appended form and say so in the task note); `uv run pytest`; ruff + pyright.
## Testing & Quality
- Integration: the shape + additive + admin-gating pins above.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `SyncStatus` + `sync_status()` carry the two additive keys; sampling happens at start / phase transitions / every terminal and is throttled to 5 s in the per-file hook.
- [ ] The Sources page shows ` · peak <N> MB` on the terminal line, renders nothing when the value is null, and adds no new control.
- [ ] The three existing sync E2E families are green (any exact-string update documented).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,34 @@
# Task 04 — E2E: the operator sees the peak (the phase's dedicated Playwright suite)
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the browser that the number is there and honest: a real sync from the Sources page ends with a `peak <N> MB` clause on the result line, the status payload's `rss_peak_mb` is a positive integer that survives the terminal, the running state shows no clause, and nothing anonymous can read it. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_sync_peak_memory_visible.py` — a NEW module, mock-only, the `tests/e2e/test_sync_progress_box.py` / `test_sync_summary_visibility.py` house shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`).
- **Test 1 `test_result_line_shows_the_peak`:** sync a small fixture → after the `success` terminal, the progress box's result line matches `/peak \d+ MB/`, the existing wording ("Sync complete"/the counts line) is still present, and the aria-live region contains the same clause.
- **Test 2 `test_status_payload_reports_rss`:** during `running` and after the terminal, `GET /api/sync/status` via `page.request` → `rss_peak_mb` is a positive int in the terminal, `>= rss_mb` when both are set, and pre-existing keys are unchanged (the additive pin end to end). During `running` the result line shows NO `peak` clause (the clause is terminal-only in the UI).
- **Test 3 `test_anonymous_cannot_read_memory`:** logged-out `GET /api/sync/status` → 403 (the A4 exposure guard, unchanged).
- **Test 4 `test_cancelled_terminal_keeps_the_peak`:** cancel a run (the `test_sync_cancel.py` shape) → the neutral "Sync cancelled" state still renders, with the peak clause present (terminals keep the value).
- **Mobile width:** assert the clause at 375 px does not overflow the progress box (the PLAN §7 container rule).
- Module docstring: the incident reference, the fields + the process-vs-cgroup caveat pointer to `README.md`, the test→observable mapping, the run-in-isolation command, prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` in full against the post-phase app, every inconsistency fixed (pre-existing included); confirm rule 10's extension and the operations grep landed in task 02; flag the PLAN §9 reconciliation in the phase report (never edit PLAN.md).
3. **The full gates:**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov`, `tests/e2e/test_sync_status_freshness.py -v --no-cov`, `tests/e2e/test_sync_cancel.py -v --no-cov` — green in isolation (the surfaces the clause touches).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `feat(observability): log and show the sync's RSS so memory is a number, not a forensic exercise` — covering `app/core/memory.py`, `app/api/sync.py`, `app/rag/importer.py`, `frontend/assets/sources.js`, the tests, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/187_sync_peak_memory_line/`.
## Testing & Quality
- E2E: the dedicated suite + the three sync UI families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` green **in isolation** (clause on both terminals, absent while running, payload types, anonymous 403, mobile width clean).
- [ ] The three regression E2E families green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project; the PLAN §9 reconciliation flagged for the owner.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,62 @@
# Phase 188 — The vector search stops full-scanning 414,220 embeddings
**Source:** found while investigating the 2026-10-15 memory incident — `pg_indexes` for `chunks` lists `chunks_pkey`, `ix_chunks_document_id`, `ix_chunks_tsv` (GIN) and **no index on `embedding`**, so every chat turn runs `ORDER BY embedding <=> :vec LIMIT 100` (A7's cosine tier) as a sequential scan of a 3 GB relation: per-turn CPU on the interactive path, plus 3 GB of page-cache churn on a 3,904 MB box that the sync also needs.
**Story:** n/a (incident-adjacent, owner directive 2026-10-15)
**Depends on:** none functionally; LAST of the six incident phases — it is the only one that touches the schema and the retrieval quality contract, so it lands after the sync's memory work is in.
**Context:** `.agents/PLAN.md` §2 — **A3** (PostgreSQL 17 + pgvector, cosine `<=>`, one system), **A6** (dim **768**, fixed at table creation, a mismatch fails loudly), **A7** (hybrid: **cosine top-100** ∪ FTS top-30 → RRF k=60 → full text of the top-N=2 documents), **A8** (the honesty gate: deflect only when best cosine < `BOR_RELEVANCE_THRESHOLD` 0.62 AND (zero FTS hits OR best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` 0.35)), **A13** (every migration ships a TESTED downgrade), **A16** (per-phase unit + integration + >90% + one dedicated E2E file), **A21** (LLM calls are free; the interactive path's latency is what matters); `app/models.py` — `Chunk.embedding: Mapped[list[float] | None] = mapped_column(Vector(EMBEDDING_DIM))` (**L241**); `app/rag/retriever.py` — **`_vector_candidates(db, question_embedding, limit)` (L431)** building `distance = Chunk.embedding.cosine_distance(question_embedding)` (**L440**) `.where(Chunk.embedding.is_not(None))` (**L444**) `.order_by(distance).limit(limit)` (**L446**), the fusion caller at **L690–711** (`vector = _vector_candidates(db, question_embedding, v_n)`), `_lexical_candidates` (L632); `app/config.py` (the `BOR_` `Settings` pattern, `relevance_threshold` / `lexical_support_floor` / `suggested_docs` as the retrieval-knob neighbours); `app/core/settings_catalog.py` — `GROUPS` (**L72**, the `"Retrieval & honesty"` group) + the `set(CATALOG) == set(Settings.model_fields)` completeness pin; `alembic/versions/` — head is **`0028`** (`uv run alembic heads`), so this phase adds **`0029`**; `db/Containerfile` — **`ARG PGVECTOR_VERSION=v0.8.0`** (HNSW + `halfvec` available; verified 0.8.0 on both the dev and the production instance); `scripts/eval_retrieval.py` (+ `tests/fixtures/retrieval_battery.txt`, the phase-119/162 regression battery — `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`, `AIPI_KEY` required); `benchmarks/model_benchmarks.csv` (the recorded-measurement home); `tests/e2e/test_retrieval_quality.py`, `tests/e2e/test_search_recall_small_sources.py`, `tests/e2e/test_chat_rag.py`, `tests/unit/test_retriever.py`, `tests/unit/test_retriever_recency.py`.
## Production numbers (read 2026-10-15, read-only)
414,220 chunks, **all 414,220 with a non-null `embedding`**, `chunks` relation ≈ **3,000 MB**, database 3,147 MB, `shared_buffers=128MB`, `work_mem=4MB`, host 3,904 MB with **no swap**. pgvector **0.8.0**.
## Objective
Add an HNSW cosine index on `chunks.embedding` through a reversible migration built `CONCURRENTLY` (A3's one-system rule respected — no new service, no new dependency), give the retrieval tier an explicit `hnsw.ef_search` knob that can never be set below the A7 top-100 candidate count, and prove with a deterministic recall pin plus the live retrieval battery that the approximate index does not move the top hits, the `top_score`, or the honesty-gate verdicts.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the index is on the existing `vector(768)` column, `vector_cosine_ops` (task 01).** `CREATE INDEX CONCURRENTLY … USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)` — the pgvector defaults' neighbourhood, which keeps the graph buildable on a 4 GB box. `halfvec(768)` (half the index size) was considered and REJECTED for this phase (A2): it needs `embedding::halfvec(768)` in `_vector_candidates` AND a quantisation-recall study, i.e. two quality risks in one phase. The index is smaller-with-halfvec; if the prod build does not fit, that is a follow-up phase, not a silent compromise here.
- **D2 — `CONCURRENTLY` needs autocommit, and an interrupted build leaves an INVALID index (task 01).** Migration `alembic/versions/0029_chunks_embedding_hnsw.py` runs its DDL inside `with op.get_context().autocommit_block():` and, before creating, drops any existing INVALID index of that name (`select indisvalid from pg_index join pg_class …` → `DROP INDEX CONCURRENTLY IF EXISTS ix_chunks_embedding_hnsw` when `indisvalid` is false), then re-checks `indisvalid` afterwards and raises if the build did not validate — because a `CONCURRENTLY` build that dies (a 3,904 MB box is exactly where it dies) leaves an index that costs insert time and buys nothing, silently. Upgrade and downgrade both run in the autocommit block; the downgrade is `DROP INDEX CONCURRENTLY IF EXISTS …` and is TESTED (A13) by `alembic downgrade -1` / `upgrade head` in the task's verification.
- **D3 — `ef_search` is explicit and clamped (task 02).** `retrieval_ef_search: int = 100` (`BOR_RETRIEVAL_EF_SEARCH`, `"Retrieval & honesty"` catalogue group): `_vector_candidates` issues `SET LOCAL hnsw.ef_search = :ef` (per-transaction — no leakage between requests, no global GUC change; A12 keeps state where it belongs) immediately before the SELECT, and a value below the A7 candidate count is CLAMPED UP to it with one WARNING line (`retrieval: ef_search=%d raised to %d (A7 candidate count)`) rather than silently degrading recall. Negative values refuse startup the way the phase-182 `show_text_max_*` knobs do.
- **D4 — recall is pinned twice: deterministic in CI, live against the battery (tasks 03 + 04).**
- **CI (deterministic, no LLM):** seed ~5,000 chunks with fixed-seed 768-dim vectors, compute the EXACT cosine top-100 in Python (`numpy`-free is fine — a pure-Python dot product over the seeded corpus, or reuse the existing helper if `tests/` already has one), then assert the index-backed query's top-100 set overlaps the exact set by **≥ 0.99** at `ef_search=100` and that `1 - distance` (`top_score`) matches the exact value to 6 decimals for the returned rows. Plus the plan pin: `EXPLAIN (FORMAT JSON)` for the real `_vector_candidates` statement names `ix_chunks_embedding_hnsw` (and does NOT show `Seq Scan on chunks`).
- **Live (A8's honesty contract):** the retrieval battery, index OFF vs ON, on the 1,000-document live-replica KB (the `restore-test-db` skill) with `AIPI_KEY` set — every question's `top_score`, top-5 document set and deflection verdict must be identical (a difference means RAISING `ef_search`, never accepting it). The before/after numbers are recorded in `benchmarks/` and the phase report; the build's wall time and the box's free RAM before/after are recorded too (they are the deploy-runbook numbers).
- **D5 — the build is an operator step with a runbook (task 01 + README).** The migration is correct, but building HNSW over 414k × 768 on the production 3,904 MB box needs a documented pre-flight: free-RAM check, `maintenance_work_mem` set explicitly for the session (e.g. 512 MB, never left at the 4 MB `work_mem` default), the build run against the postgres container directly (`podman exec … psql`) if the app container is memory-limited (the owner's separate task), `autovacuum` left running afterwards, and the INVALID-index recovery path. `README.md` gains that runbook section; the phase's own CI build runs against the small test corpus, not 414k rows.
- **NOT touched:** the embedding dim (A6), the `<=>` operator and the hybrid fusion (A7), the thresholds themselves (A8 — unchanged unless the battery forces an `ef_search` raise), the `tsv` GIN index, `chunks`/`documents` columns, the importer/sync (phases 183–186), the frontend, any new dependency or service (A3/A12).
## Dependencies
— (none; deliberately the LAST of the six incident phases — schema + retrieval-quality risk after the sync's memory work)
## Tasks
1. `01_hnsw_migration.md` — migration `0029` (autocommit block, `CONCURRENTLY`, the INVALID-index pre-drop + post-validate), the tested downgrade, the migration tests.
2. `02_ef_search_setting.md` — `retrieval_ef_search` on `Settings` + the catalogue group + `.env.example`, `SET LOCAL` in `_vector_candidates`, the clamp-and-warn rule, the negative-refuses-startup pin.
3. `03_plan_and_recall_pins.md` — the `EXPLAIN` plan pin (index used, no seq scan) and the deterministic ≥0.99 overlap / exact-`top_score` recall pin on a fixed-seed corpus.
4. `04_recall_battery_gate.md` — the live battery index-off vs index-on on the 1,000-doc replica KB; the recorded numbers in `benchmarks/` + the phase report; the `ef_search` raise loop if anything moved.
5. `05_e2e_index_chat_and_runbook.md` — `tests/e2e/test_vector_index_recall.py` (the phase's dedicated Playwright suite), the `README.md` build runbook, the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the `retrieval_ef_search` default/clamp/negative rules; the migration module's helpers (the INVALID-index decision function is a pure function over a query result, so it unit-tests without a DB).
- **Integration:** `tests/integration/test_chunks_hnsw_index.py` — upgrade/downgrade round-trip on the test DB, `pg_index.indisvalid` asserted, the INVALID-index recovery path exercised by creating an INVALID index deliberately (`… WHERE false` won't produce INVALID; use the documented `UPDATE pg_index SET indisvalid=false` superuser trick or a deliberately interrupted build — pick the one the test DB allows and document it), the `EXPLAIN` plan pin, the deterministic recall pin. `tests/unit/test_retriever.py` + `test_retriever_recency.py` + `tests/integration/test_chat_db_sessions.py` green with `SET LOCAL` in play.
- **Live gate:** the retrieval battery off/on, identical verdicts, recorded.
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_vector_index_recall.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the `0029` deploy section (the D5 build runbook: pre-flight free RAM, `maintenance_work_mem`, `CONCURRENTLY` semantics, the INVALID-index recovery, expected build time from task 04's measurement) and the `BOR_RETRIEVAL_EF_SEARCH` knob (default 100, the clamp, "raise it if recall matters"). `AGENTS.md`: the quick-reference gains nothing new unless a command changes — the retrieval battery line already exists; keep it accurate.
## Completion Criteria
- [ ] `alembic/versions/0029_…` creates `ix_chunks_embedding_hnsw` `CONCURRENTLY` inside an autocommit block, pre-drops an INVALID index of that name, post-validates `indisvalid`, and its downgrade is tested (`upgrade head` → `downgrade -1` → `upgrade head` green on the test DB, A13).
- [ ] `retrieval_ef_search` (default 100) is applied with `SET LOCAL` in `_vector_candidates`, clamped up with a WARNING below the A7 candidate count, negative refuses startup, and it appears in `CATALOG` (`"Retrieval & honesty"`) + `.env.example` + `README.md`.
- [ ] The `EXPLAIN` pin shows the index used and no seq scan; the deterministic recall pin shows ≥0.99 top-100 overlap and exact `top_score`s.
- [ ] The live battery ran index-off and index-on over the 1,000-doc replica with IDENTICAL top scores, top-5 sets and deflection verdicts, and the numbers (plus build time + free-RAM readings) are recorded in `benchmarks/` and the phase report.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the runbook included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(retrieval): index chunks.embedding with HNSW and pin recall so cosine stops full-scanning 414k vectors`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — HNSW on `vector(768)` with `vector_cosine_ops`, built `CONCURRENTLY`.** No query rewrite, no new dependency, `<=>` uses it directly (A3).
- **A2 — `halfvec` quantisation is NOT part of this phase** (it needs a statement cast + its own recall study); revisit only if the production build does not fit in RAM.
- **A3 — recall is a gate, not a hope.** The honesty gate (A8) reads `top_score`; an approximate index that changes a top hit can flip a deflect. The deterministic ≥0.99 pin AND the live battery must pass, and the remedy is a higher `ef_search`, never a relaxed assertion.
- **A4 — `SET LOCAL`, per transaction.** No global GUC mutation, no connection-pool-wide state (A12: no aux services, restart clears state).
- **A5 — the production build is an operator action with a runbook**, executed after the owner's container-memory work lands; the phase's CI never builds a 414k-row graph.
@@ -0,0 +1,44 @@
# Task 01 — Migration `0029`: the HNSW index, built concurrently, reversible
**Phase:** `188_chunks_hnsw_index` · **Source:** the 2026-10-15 incident investigation (`pg_indexes` shows no index on `chunks.embedding`)
**Story:** n/a
## Objective
Add `ix_chunks_embedding_hnsw` on `chunks.embedding` with `vector_cosine_ops` through alembic `0029`, built `CONCURRENTLY` inside an autocommit block, self-healing an interrupted build's INVALID index, with a tested downgrade (A13).
## Work
1. `alembic/versions/0029_chunks_embedding_hnsw.py` — `revision = "0029"`, `down_revision = "0028"` (verify with `uv run alembic heads` before writing), the house docstring shape (why / what / reversibility):
- Pure helpers (unit-testable, no I/O): `INDEX_NAME = "ix_chunks_embedding_hnsw"`, and `def should_drop_invalid(indisvalid: bool | None) -> bool` (True when the row exists and `indisvalid is False`), plus `def is_valid(indisvalid: bool | None) -> bool`.
- `upgrade()`:
```python
with op.get_context().autocommit_block():
if should_drop_invalid(_lookup_index_validity(op.get_bind())):
op.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {INDEX_NAME}")
op.execute(
f"CREATE INDEX CONCURRENTLY {INDEX_NAME} ON chunks "
"USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)"
)
if not is_valid(_lookup_index_validity(op.get_bind())):
raise RuntimeError(f"{INDEX_NAME} did not validate — drop it and re-run")
```
`_lookup_index_validity(bind)` runs `select i.indisvalid from pg_index i join pg_class c on c.oid = i.indexrelid where c.relname = :name` and returns `None` when absent. The docstring names the failure mode this guards (a `CONCURRENTLY` build killed mid-way leaves an INVALID index that costs inserts and serves nothing — on a 3,904 MB production box that is the expected failure, not the exotic one).
- `downgrade()`: the same autocommit block with `DROP INDEX CONCURRENTLY IF EXISTS ix_chunks_embedding_hnsw`.
- No `chunk`/`document` column change, no data rewrite (so the migration is cheap on a healthy box and interruptible).
2. `tests/integration/test_chunks_hnsw_index.py` — the migration family:
- `test_upgrade_creates_a_valid_hnsw_index`: run the migration's `upgrade()` against the test DB (the house alembic-in-test pattern — find the existing migration test's approach first and reuse it), then assert `pg_indexes` has the index, `pg_index.indisvalid` is true, and `pg_get_indexdef` contains `USING hnsw` + `vector_cosine_ops` + `ef_construction=64`.
- `test_downgrade_removes_it` then `test_upgrade_head_again` — the A13 round-trip.
- `test_invalid_index_is_dropped_and_rebuilt`: force an INVALID state (either `UPDATE pg_index SET indisvalid = false WHERE indexrelid = '…'::regclass` as the test DB's superuser, or a documented alternative) → run `upgrade()` → the index is valid again (no `relation already exists` error, no leftover INVALID index).
- `test_should_drop_invalid_matrix` (pure): `None` → False, `True` → False, `False` → True.
- Skip-guard: these tests need the `vector` extension and DDL rights — follow the existing migration tests' skip/fixture conventions (do NOT invent a new one).
3. Verify: `uv run alembic upgrade head && uv run alembic downgrade -1 && uv run alembic upgrade head` on the dev DB (record the output in the task note); `uv run pytest tests/integration/test_chunks_hnsw_index.py -v`; `uv run pytest`.
## Testing & Quality
- Integration: the four migration pins above (create / downgrade / re-upgrade / INVALID self-heal).
- Unit: the two pure helpers' matrices.
- Coverage: **>90%** on `app/` (the migration module is covered by its own tests).
## Completion Criteria
- [ ] `alembic/versions/0029_chunks_embedding_hnsw.py` exists with the autocommit block, `CONCURRENTLY`, the INVALID self-heal, the post-validate raise, and the `CONCURRENTLY` downgrade.
- [ ] The `upgrade head → downgrade -1 → upgrade head` round-trip is green on the dev DB (output recorded) and in the integration tests.
- [ ] The forced-INVALID test is green and documents the trick it used.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,37 @@
# Task 02 — `BOR_RETRIEVAL_EF_SEARCH`: explicit, per-transaction, clamped
**Phase:** `188_chunks_hnsw_index` · **Source:** the 2026-10-15 incident investigation
**Story:** n/a
## Objective
Give the HNSW tier an operator-visible `ef_search` that defaults to the A7 candidate count and can never be set below it, applied per transaction with `SET LOCAL` so no request leaks state into another.
## Work
1. `app/config.py` — `retrieval_ef_search: int = 100`, the `hybrid_vector_candidates` (L351) docstring style, `BOR_RETRIEVAL_EF_SEARCH` via the `env_prefix`. A negative value refuses startup exactly like the phase-182 `show_text_max_*` knobs (follow their validator, don't invent one); `0` is rejected the same way (there is no sensible `ef_search=0` — the clamp would silently rewrite it, which is exactly what a `0`-means-off operator would not expect).
2. `app/core/settings_catalog.py` — `SettingMeta("retrieval_ef_search", "Retrieval & honesty", "Vector ef_search")`; `uv run pytest tests/unit/test_settings_catalog.py -v` proves the completeness pin.
3. `app/rag/retriever.py` — `_vector_candidates` (**L430–446**): before the SELECT,
```python
ef = settings.retrieval_ef_search # settings passed in / read the way this module already reads them
if ef < limit:
logger.warning("retrieval: ef_search=%d raised to %d (A7 candidate count)", ef, limit)
ef = limit
db.execute(text(f"SET LOCAL hnsw.ef_search = {int(ef)}")) # int()-coerced: no user string reaches SQL
```
Notes the code must carry: `SET LOCAL` is transaction-scoped (A4 — the session's transaction commits/rolls back with the request; no global GUC mutation, nothing surviving a pool recycle); the value is `int()`-coerced so nothing but a number is interpolated; the clamp exists because `ef_search < LIMIT` returns FEWER candidates than the A7 tier promises, i.e. a silent recall regression.
Threading: `_vector_candidates` currently takes `(db, question_embedding, limit)` — read `Settings` the same way the surrounding functions do (find the module's existing `get_settings()`/passed-settings convention and match it; no new plumbing if one exists).
4. Pins:
- `tests/unit/test_retriever.py`: the clamp (ef 10 with limit 100 → warning + 100 used), the pass-through (ef 200 → 200), the default (unset → 100), the negative-refusal at settings construction; assert the emitted statement text through a stub session that records `db.execute` calls (no DB needed).
- `tests/integration/test_chunks_hnsw_index.py`: after a real `SET LOCAL`, `show hnsw.ef_search` in the same transaction reports the clamped value and a NEW transaction reports the server default again (the scoping pin).
- `tests/integration/test_chat_db_sessions.py` + `tests/unit/test_retriever_recency.py` green (the extra statement in the turn's transaction breaks nothing).
5. Verify: `uv run pytest tests/unit/test_retriever.py tests/unit/test_settings_catalog.py tests/integration/test_chunks_hnsw_index.py tests/integration/test_chat_db_sessions.py -v`; `uv run pytest`.
## Testing & Quality
- Unit: clamp / pass-through / default / negative refusal / statement text.
- Integration: `SET LOCAL` scoping (same-transaction vs new).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `retrieval_ef_search` exists (default 100, negative refused), is catalogued under `"Retrieval & honesty"`, and is in `.env.example` + `README.md`.
- [ ] `_vector_candidates` issues `SET LOCAL hnsw.ef_search` with the clamp-and-warn rule, int-coerced.
- [ ] The scoping pin proves a new transaction sees the server default again.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,27 @@
# Task 03 — Prove the planner uses it and that recall did not move (deterministic)
**Phase:** `188_chunks_hnsw_index` · **Source:** the 2026-10-15 incident investigation
**Story:** n/a
## Objective
Two CI pins: the cosine tier's statement uses `ix_chunks_embedding_hnsw` (no sequential scan), and the index-backed top-100 matches the exact top-100 on a fixed-seed corpus, with `top_score` values exact to 6 decimals.
## Work
1. `tests/integration/test_chunks_hnsw_recall.py` — a NEW file:
- **The corpus fixture:** ~5,000 `Chunk` rows with 768-dim vectors from a FIXED seed (a documented `random.Random(20261015)` scheme: 40 cluster centroids + noise around them, so nearest neighbours are meaningful rather than uniformly random), all L2-normalised (cosine == dot product, so the exact answer is trivial to compute), committed, then the HNSW index built for this corpus via the `0029` migration's `upgrade()` (task 01) — never `CREATE INDEX` inline (the migration is the single definition).
- `test_plan_uses_the_hnsw_index`: `EXPLAIN (FORMAT JSON)` for the exact statement `_vector_candidates` builds (the same SQLAlchemy statement compiled with literal binds, so the planner sees a real plan), assert the node names `ix_chunks_embedding_hnsw` / an index scan node and that no `Seq Scan on chunks` node appears. Assert the same statement's plan WITHOUT the index (a `DROP INDEX`/re-create around a sub-fixture, or a `SET enable_indexscan = off`-style comparison — pick one, document it) DOES seq-scan, so the pin cannot pass vacuously.
- `test_top100_recall_at_ef_search_100`: for 30 fixed-seed query vectors, compute the EXACT cosine top-100 (pure Python over the fixture corpus) and compare against the index-backed `_vector_candidates(db, q, 100)` result: assert `|overlap| / 100 >= 0.99` for every query (mean reported in the docstring) and that each returned row's `1 - distance` equals the exact cosine within 1e-6. Print the per-query overlap in the failure message (so a future raise of `ef_search` has evidence).
- `test_recall_never_below_the_pinned_floor_when_ef_search_raised`: the same assertion at `ef_search = 200` must be ≥ the `ef_search = 100` mean overlap (monotonicity sanity: raising `ef` cannot make recall worse).
- Module docstring: the production numbers (414,220 chunks / 3,000 MB / no index found / pgvector 0.8.0), why 5,000 rows is enough for a plan pin but not a proxy for prod (the live battery in task 04 is), the measured overlaps, and the `ef_search` values used.
2. Runtime guard: the fixture corpus build + index creation must stay fast (target < ~60 s); if it drifts, reduce the corpus (never the assertions) and record the new size — the phase-179 A4 discipline ("cut the fixture, never the guard").
3. Verify: `uv run pytest tests/integration/test_chunks_hnsw_recall.py -v`; `uv run pytest`.
## Testing & Quality
- Integration: the plan pin (with its vacuity guard) + the two recall pins.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_chunks_hnsw_recall.py -v` green: index used, no seq scan, mean top-100 overlap ≥ 0.99, `top_score`s exact to 1e-6, monotonic at `ef_search=200`.
- [ ] The vacuity guard (seq scan when the index is unavailable) is present and green.
- [ ] The docstring records the measured overlaps and the prod context numbers.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,35 @@
# Task 04 — The live gate: the retrieval battery, index off vs on
**Phase:** `188_chunks_hnsw_index` · **Source:** the 2026-10-15 incident investigation; PLAN A8
**Story:** n/a
## Objective
Prove on the real embedding endpoint and the 1,000-document live-replica KB that the approximate index does not move a single verdict — same `top_score`, same top-5 documents, same deflection decisions — and record the numbers plus the production build cost.
## Why this is a gate and not a note
PLAN **A8** derives the honesty decision from `best_cosine` against `BOR_RELEVANCE_THRESHOLD` (0.62) and `BOR_LEXICAL_SUPPORT_FLOOR` (0.35). An approximate index that swaps which chunk is nearest can flip a HIGH into a deflect, or cite a different document — the exact class of silent regression the phase-119/162 battery exists to catch.
## Prerequisites (document them in the phase report before starting)
`AIPI_KEY` in the environment (the same convention as `scripts/llm_probe.py` / `scripts/eval_retrieval.py`), the 1,000-document live-replica KB restored (the `restore-test-db` skill), and `uv run alembic upgrade head` applied. If `AIPI_KEY` or the replica KB is unavailable, STOP and report it — do not commit the migration as phase-complete on the strength of task 03's synthetic pin alone.
## Work
1. **Baseline (index OFF):** drop the index (`DROP INDEX CONCURRENTLY ix_chunks_embedding_hnsw` — the migration's downgrade), then
`uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt > /tmp/battery_off.txt`.
Capture per question: the effective/top score, the top-5 document list, the honesty-gate verdict column.
2. **Candidate (index ON):** re-apply (`uv run alembic upgrade head`), same command to `/tmp/battery_on.txt`.
3. **The diff:** compare question by question. Required: identical deflection verdicts, identical top-5 document sets, and `top_score` deltas ≤ 1e-4. Any difference → raise `BOR_RETRIEVAL_EF_SEARCH` (150 → 200 → 400) and re-run BOTH sides until the verdicts match; the `ef_search` that restores parity becomes the shipped default (update `Settings` + `.env.example` + `README.md` + the task-02 tests, and say so in the commit body). Never widen the tolerance.
4. **Record** (the repo's measurement homes, not chat):
- `benchmarks/` — a row/section per run: pgvector version, index definition (`m`, `ef_construction`), `ef_search`, corpus size, the battery's question count, verdict-agreement counts, the max `top_score` delta, the battery's wall time off vs on, and the per-question cosine-tier latency if the script reports it.
- the phase report (`.agents/reports/`, house location) — the production BUILD numbers captured while building on a box of the deployment's size class where possible: build wall time, `maintenance_work_mem` used, free RAM before/after (`free -m`), index size (`select pg_size_pretty(pg_relation_size('ix_chunks_embedding_hnsw'))`), and the seq-scan → index-scan latency change on a real chat turn.
5. **Timing sanity:** the interactive path got faster, not slower (A21's "latency binds the interactive path"). Record the before/after; if the index makes a turn slower at the shipped `ef_search`, that is a finding for the phase report, not a silent acceptance.
6. Re-run the app's own retrieval regression expectations if the battery script's output is asserted anywhere in the suite (`grep -rn "retrieval_battery" tests/ scripts/`), and keep those green.
## Testing & Quality
- Live gate: the off/on battery comparison with identical verdicts (the remedy is `ef_search`, never the tolerance).
- Recorded artifacts: `benchmarks/` entry + the phase report numbers (both referenced by path in the commit body).
## Completion Criteria
- [ ] `/tmp/battery_off.txt` and `/tmp/battery_on.txt` (or their committed equivalents) exist, and the comparison shows identical deflection verdicts, identical top-5 sets, max score delta ≤ 1e-4.
- [ ] The shipped `retrieval_ef_search` default is the value that achieved parity (documented whether it stayed 100 or was raised).
- [ ] `benchmarks/` records the index definition, sizes, build time, free-RAM readings and the off/on timings; the phase report repeats the production build numbers.
- [ ] Any test that reads the battery fixture stays green.
@@ -0,0 +1,36 @@
# Task 05 — E2E: chat over the indexed KB (the phase's dedicated Playwright suite)
**Phase:** `188_chunks_hnsw_index` · **Source:** the 2026-10-15 incident investigation
**Story:** n/a
## Objective
Prove through the browser that retrieval behaves identically over the indexed KB — the same document is cited, the answer is not deflected, and the turn completes — plus the `README.md` build runbook, the full gates, the docs pass, and the atomic commit.
## Work
1. `tests/e2e/test_vector_index_recall.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped; the E2E mock's deterministic L2-normalised token-overlap embeddings make the cosine tier reproducible, PLAN A16), the `tests/e2e/test_retrieval_quality.py` / `test_search_recall_small_sources.py` / `test_chat_rag.py` shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`).
- Fixture: a small KB (8–12 docs) imported through the real importer so chunks + embeddings exist, with the `0029` migration's index present in the test DB (import-time inserts into an HNSW-indexed table are exercised by this fixture too — inserts are the phase's insert-cost surface, so keep them in the path rather than importing then indexing).
- **Test 1 `test_answered_question_cites_the_same_document_indexed_or_not`:** the sentinel question is asked twice — once with the index dropped (test-local DDL, restored after) and once with it present — and the cited source chip / cited document is the SAME in both, with no deflection either way. This is the A7/A8 contract expressed in the UI, and it is a real before/after, not a tautology.
- **Test 2 `test_index_exists_and_is_valid_after_import`:** after the fixture import, the test's own DB query asserts `pg_index.indisvalid` is true for `ix_chunks_embedding_hnsw` (an index that died during the fixture's inserts would be caught here — the silent-INVALID failure mode from task 01, at the app layer).
- **Test 3 `test_near_miss_question_still_deflects`:** a question with no KB support is deflected with the index present (A8's LOW path is unchanged — the index must not manufacture a high enough `top_score` to flip a deflect).
- **Test 4 `test_ef_search_setting_is_honoured`:** with `BOR_RETRIEVAL_EF_SEARCH=150` on the module app, the turn still answers normally (a knob change cannot break a chat turn), and the clamp warning appears when set to a value below the A7 tier (caplog or the status/log surface the test harness can read).
- Module docstring: the incident/investigation reference, the index definition + `ef_search` default, the test→observable mapping, the run-in-isolation command, prerequisites (`podman compose up -d db`).
2. **README runbook (D5):** a `Deploying migration 0029 (HNSW)` section — the pre-flight (`free -m`, `shared_buffers`, expected peak), building with an explicit `maintenance_work_mem` (never the 4 MB default), `CREATE INDEX CONCURRENTLY` semantics (no lock, longer build, may leave an INVALID index if killed), the recovery path (drop and re-run the migration), the measured numbers from task 04 (build time, index size), and the pointer that the owner's container-memory work should land before building on a 4 GB-class box.
3. **Docs pass:** `README.md` + `AGENTS.md` reviewed in full against the post-phase app; every inconsistency fixed, pre-existing included; `AGENTS.md`'s quick-reference stays accurate (the retrieval-battery line already exists and is now load-bearing for this phase — confirm its command line is still correct).
4. **The full gates:**
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov`, `tests/e2e/test_search_recall_small_sources.py -v --no-cov`, `tests/e2e/test_chat_rag.py -v --no-cov` — green in isolation (the retrieval/chat families the index touches; run after the new suite).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
- `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` — task 04's parity verdict still holds at the shipped `ef_search`.
5. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `perf(retrieval): index chunks.embedding with HNSW and pin recall so cosine stops full-scanning 414k vectors` — covering `alembic/versions/0029_…`, `app/config.py`, `app/core/settings_catalog.py`, `app/rag/retriever.py`, the tests, `benchmarks/`, `README.md`, `.env.example` and `.agents/`. Move the phase dir to `.agents/phases/complete/188_chunks_hnsw_index/`.
## Testing & Quality
- E2E: the dedicated suite + the three retrieval/chat E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` green **in isolation** (same citation indexed vs not, index valid after import, near-miss still deflects, `ef_search` change harmless).
- [ ] The three regression E2E families green in isolation; the battery parity verdict still holds.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` (runbook + knob) and `AGENTS.md` accurate and consistent with the whole project.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,68 @@
# Phase 189 — Long agent turns save (and share) the whole conversation
**Source:** owner incident 2026-10-16 — the shared chat `0101d2ca-b24c-4d13-9f65-c0837e993c50` (row `8b345ab5-aab3-4412-9642-0f869b3cec4d`) shows ONLY the user's question although the LLM answered it completely; the shared row `be92e464-7918-4907-9186-09d8d98ff2c1` carries the same signature.
**Story:** n/a (incident-derived, owner directive 2026-10-16)
**Depends on:** none functionally (builds on the completed phase-55/83/158/164 saved-chat surface).
## Evidence (diagnosed 2026-10-16, production DB + app logs — read this, not the chat)
1. The row behind the shared token: `n_msgs = 1`, `who0 = user`, `created_at == updated_at` — the row was written EXACTLY ONCE (the send-time auto-save `POST /api/chats`) and never updated. 84 other single-write user-only rows exist (2 shared, incl. this one); the 2-message rows (279) were written >1 time — the pattern is specific to the *last* save point of a conversation.
2. The app log for this row shows the whole sequence: the turn's per-turn line (`thinking=xhigh thinking_chars=23937 tool_calls=33 rounds=34 cap_hit=0 total_ms=400173`, `POST /api/chat 200 stream=1` — the client consumed the FULL 400 s stream, so the done frame landed) followed by **`PUT /api/chats/8b345ab5… 422 Unprocessable Entity`** — the done-time auto-save REJECTED — then `GET /api/chats/… 200`, `POST …/share 200` (×2), `GET 200`. The share only minted a token on the stale row.
3. WHY the 422: the turn's done frame carried **22 sources** (counted from the per-turn log line's `sources=[…]`), but `ChatMessage.sources` is capped at **20** (`app/schemas.py` L1460 — phase 83 A1 "top-N docs + agent reads"). The cited tier is the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587/L1289 — retrieval top-N ∪ the agent's read docs, NOT bounded by `top_n_docs`), and production runs `BOR_AGENT_MAX_ROUNDS=50` (the app container's env; the config default is 10) — so a long research turn cites 22+ docs routinely and the done-time auto-save 422s. The same turn had **33 tool calls** (`tools` cap 50 — at the production round cap with ZERO headroom) and `segments` entries = 33 tool entries + one thinking entry per contiguous thinking run (`buildSegments`, `app.js` L1775) under `xhigh` thinking over 34 rounds — the `segments` cap of 50 (phase 158, docstring sized "the round cap is 10 → ≤ 20 entries") is exceeded by the same class of turn. `suggestions` (cap 50) and `related` (cap 20) are likewise sized against settings (`suggested_docs`, `related_max_docs`) whose validators are unbounded above.
4. WHY the 422 lost the answer: the phase-55 A2 quiet contract swallows a failed auto-save — a one-line `#send-status` note ("Couldn't save automatically — will try on the next message."), no error banner — and NOTHING retries: the retry is the NEXT save point (next user send / next brain-done / the pagehide partial), which never came (this was the last turn). The row stayed at its pre-turn state.
5. WHY the share froze the loss: `shareCurrentChat`'s LINKED branch (`app.js` L3345) only calls `POST /api/chats/<id>/share` (token mint, `app/api/chats.py` `share_chat`) — it never re-saves the current conversation, so the shared snapshot is whatever the row last held.
6. SECOND loss path (same symptom class): `persistConversation`'s `persisting` double-fire guard (`app.js` L3216) DROPS a save point that arrives while an upsert is in flight ("the next save point retries" — which, per 4., may never come). A done-time save dropped by a still-in-flight send-time save (a 400 s turn on a slow line) leaves the same user-only row.
## Objective
A long agent turn (production `BOR_AGENT_MAX_ROUNDS=50`) saves cleanly at every save point: the saved-chat schema caps move from the round-cap-10 era to the real configuration (A1); a Share never publishes a row the current conversation has outgrown — it saves first (A2); a dropped or failed save self-heals (overlap re-run, one transient retry) and a failed save STAYS visible until the next success (A3/A4).
## Context
- `app/schemas.py` — `ChatMessage` (L1381; `extra="forbid"`, L1458): `text`/`thinking` `max_length=600_000` (phase 164), `sources` `max_length=20` (L1459), `related` `max_length=20`, `suggestions` `max_length=50` (items `_Chip` ≤ 200), `tools` `max_length=50`, `segments` `max_length=50` (L1479), `error` ≤ 500, `image` ≤ 500, `answer_image` `SourceRef`, `answer_text` `max_length=50`, `budget_exhausted`; `SourceRef` (L1260-ish: `source` ≤ 120 / `path` ≤ 1000 / `title` ≤ 500 / `image_url`), `ToolCall` (L1209: `name` ≤ 100, `argument` ≤ 2000, `chars_shown`/`chars_total`/`offset`, the phase-139 `range` + phase-146 `path` additive keys), `SavedChatCreate`/`SavedChatUpdate` (`messages` `min_length=1, max_items=500` — phase 164; the `_require_within_total_budget` validator), `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` (L1611). The phase-83 docstrings on the list fields still say "top-N docs + agent reads — the UI shows a handful" — that assumption is RETIRED (see evidence 3).
- `app/api/chat.py` — `merge_cited_docs` (L587), the done-event build (L1402–1447: `cited_refs` = the merged read docs, `related_refs` capped by `related_max_docs`), `app/rag/retriever.py` (the tiering), `app/config.py` (`agent_max_rounds` L279, validator L—: `>= 0`, NO upper bound; `suggested_docs` / `related_max_docs` validators likewise unbounded above; production env `BOR_AGENT_MAX_ROUNDS=50`).
- `app/api/chats.py` — the public write surface (phase 55 A1), `share_chat` (token-only, L379-ish), the 422 is the framework's standard validation response (phase 83 A2: boundary-only, no route changes).
- `frontend/assets/app.js` — `persistConversation` (L3218: the `persisting` guard L3216, the 404→recreate fallback, the A2 quiet contract on failure), `rememberBrainTurn` (L3484 — save point 2, rides L3496), the send save point (L1371), `shareCurrentChat` (L3345: linked → `POST /{id}/share` only; unlinked → `POST /api/chats {messages, share: true}`), `regenerateStaleChat` (L3419 — its own upsert copy), the pagehide partial (L4821, rides `rememberBrainTurn`), `buildSegments` (L1775), `sendStatus` (the `#send-status` live region).
- `tests/unit/test_schemas.py` — the boundary-constant block (L47–75: `SOURCES_CAP = 20`, `SUGGESTIONS_CAP = 50`, `TOOLS_CAP = 50`, `TEXT_CAP = 600_000`, …) and the at-cap/one-over pins per field; `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862), the oversized-422 pins, the full brain-record pin; `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — text caps UNCHANGED, this suite is unaffected); `tests/unit/test_save_chat_ui.py` (the JS marker pins for the save/share contract — the A2 quiet-contract pin names the status-note text and the `persisting` guard); `tests/e2e/mock_llm.py` (the deterministic mock turn: single-read tool flow, `slow_llm.py` for timing).
## Design (shared by all tasks — the executor reads this, not the chat)
- **A1 — the caps move to the real configuration (task 01; `app/schemas.py` ONLY, boundary-only — the phase-83 A2 style: NO route change, the 422 stays the framework's standard response).** The five list caps change, every other bound (per-message `text`/`thinking` 600 000, `error` 500, `image` 500, the `SourceRef` field caps, the `ToolCall` field caps, `answer_text` 50, the `max_items=500` messages cap, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8 000 000`) is UNCHANGED:
- `sources` 20 → **200** — the cited tier is top-N ∪ the agent's READ docs (`merge_cited_docs`), so its size is bounded by `agent_max_rounds` (production 50, validator-unbounded) × calls-per-round, not by `top_n_docs`; 200 = 4× the production round cap (each round typically carries one `read`);
- `related` 20 → **200** — `related_max_docs` is owner-tunable with no upper bound; one `SourceRef`-family cap with `sources`;
- `tools` 50 → **200** — one-or-more calls per round are legal (a round is one model request, not one call); 4× the production round cap;
- `segments` 50 → **450** — ≤ `tools` (200) tool entries + ≤ ~50 thinking runs (one per round, the cap-50 worst case) + margin;
- `suggestions` 50 → **100** — `suggested_docs` is owner-tunable with no upper bound; 20× the default (5); the item cap (200 chars) is unchanged.
The SEC-05 intent survives: the worst-case list bytes are orders of magnitude inside the 8 M TOTAL budget (`200 SourceRefs ≈ 360 KB`, `200 ToolCalls ≈ 440 KB`, `450 segments ≈ 1 MB`), and the fixed boundary still bounds item COUNT for an operator who raises `BOR_AGENT_MAX_ROUNDS` far beyond 50. The `ChatMessage`/`SourceRef` docstrings are updated: the "top-N + a handful" sizing rationale is replaced by the `merge_cited_docs` / `agent_max_rounds` reality (evidence 3), the phase-83 history KEPT (the 32 000 mirror is phase 164's 600 000), and a note that the 8 M total budget is the byte bound. The unit constants (`SOURCES_CAP`, `TOOLS_CAP`, `SUGGESTIONS_CAP`) and the integration pins move with the values.
- **A2 — Share saves first (task 02; `shareCurrentChat`, `app.js` L3345).** The LINKED branch upserts the CURRENT conversation BEFORE minting the token — the SAME upsert semantics as `persistConversation` (PUT; a 404 unlinks and retries as a create), then `POST /api/chats/<id>/share`. An upsert FAILURE (non-ok or network) ABORTS the share with the existing neutral error banner ("Couldn't share the conversation — try again.") — NO token, NO toast, `shareBtn` released in the `finally` (the never-stale contract, PLAN §7.4). The UNLINKED branch is unchanged (its `POST {messages, share: true}` already saves the current conversation in one commit). Docstring: sharing a conversation that the row has not yet caught up would publish a truncated snapshot (the incident) — the upsert-first ordering makes the shared page the current conversation.
- **A3 — a dropped save re-runs (task 03; `persistConversation`, `app.js` L3216–3260).** `persisting` stays the one-upsert-at-a-time guard, but a save point that arrives while an upsert is in flight sets a module-level `savePending = true` instead of vanishing; the in-flight upsert's `finally` (after `persisting = false`) re-runs `persistConversation()` once when `savePending` is set (the flag is cleared first; the re-run is a fresh, serialized call — never concurrent with the upsert it trails). The re-run carries the LATEST `conversation` (the array is read at call time), so the dropped done-time save lands with the brain record. No new save points are added — the existing ones (send, `rememberBrainTurn`/pagehide, regenerate) are unchanged.
- **A4 — a failed save stays visible and retries once (task 03).** On failure (non-ok or network), the existing one-line note is kept AND a module flag `saveFailedSticky = true` set — the note persists until the NEXT SUCCESSFUL save (a success clears the flag and resets `#send-status` to empty when the flag was set; other flows may overwrite the note in the meantime — best-effort visibility, the flag + the next save point are the guarantee). A TRANSIENT failure (network error or 5xx status) additionally schedules ONE retry after 5 000 ms (a `setTimeout` + a one-shot `retryScheduled` guard so overlapping failures never stack retries; the retry is a plain `persistConversation()` — the LATEST conversation, the same guards). A 4xx NEVER retries (deterministic — the cap fix in task 01 is the remedy; the retry would 422 again). The A2 contract's shape is unchanged: still a one-line `#send-status` note, still NO error banner, the turn never blocks — only the note's lifetime and the one transient retry are added. `regenerateStaleChat`'s own upsert copy is LEFT AS IS (its failure path already errors visibly through the turn's banner state, and A1 removes the 422 class it could hit).
## Tasks
1. `01_schema_cap_realign.md` — the five list-cap moves + docstring rationale (A1), the unit boundary pins at the NEW values, the incident-payload integration pin (22 sources / 51 segments / 201 tools now 201/200), the oversized-422 pins at the new boundaries.
2. `02_share_saves_first.md` — `shareCurrentChat`'s linked branch upserts before minting the token, aborts the share on an upsert failure; the JS marker pins.
3. `03_persist_rerun_sticky.md` — the `savePending` overlap re-run, the sticky failure note, the one transient retry; the JS marker pins.
4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the incident regression at the wire + the overlap / share-saves-first behavior through the real UI), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_schemas.py` — the NEW boundary pairs per field (200/201 `sources`, 200/201 `related`, 200/201 `tools`, 100/101 `suggestions`, 450/451 `segments`), the realistic-payload round-trip pin EXTENDED (a 22-source / 34-tool / 40-segment brain record validates and round-trips byte-identically), every pre-phase pin green. `tests/unit/test_save_chat_ui.py` — the A2/A3/A4 marker pins (the upsert-before-share ordering in `shareCurrentChat`'s linked branch, the abort-on-failure path, `savePending` + the `finally` re-run, the sticky flag + the 5 s one-shot retry, the 4xx-no-retry distinction; the existing A2 quiet-contract pins stay green — the note text and the "no error banner" shape are unchanged).
- **Integration:** `tests/integration/test_chats_api.py` — anonymous `POST`/`PUT` with the INCIDENT payload shape (22 `sources`, 33 `tools`, 51 `segments` incl. thinking runs) → **201/200** (pre-fix: 422); the over-cap pins move to the new boundaries (201 `sources` → 422, no row; 201 `tools` → 422; 451 `segments` → 422; 101 `suggestions` → 422); the full brain-record pin + the stale/share pins stay green.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_chat_save_long_turns.py`, run **in isolation** — (a) the incident regression at the wire through `page.request` (22-source brain record → 201, pre-fix 422); (b) a real UI mock turn's done-time auto-save lands (row: 2 messages); (c) the overlap re-run (the send-time save's PUT delayed 3 s via a Playwright route, the mock turn settles in flight → the dropped done-time save re-runs → final row 2 messages); (d) share-saves-first (with the auto-saves still delayed, the Share click → the shared page renders the BRAIN message, not just the question).
- **Regression:** `tests/e2e/test_chat_save_payload_limits.py`, `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py` (the phase-51/55 share contract) green unedited.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the changed file is `app/schemas.py` (declarative — exercised by every unit/integration pin); the frontend change is marker-pinned per the house JS convention.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap), Share saves the current conversation before minting the token, a failed auto-save stays visible until the next success and retries once on a transient failure (task 04). `AGENTS.md`: nothing to change unless the final pass finds the phase-55 save contract described elsewhere (the save points are not enumerated in AGENTS.md — the UI-Structure rule's "in-flight turn" clause is untouched; verify, do not add).
## Completion Criteria
- [ ] `ChatMessage` list caps: `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; every other bound unchanged; the docstring rationale updated (A1).
- [ ] The incident payload (22 sources / 33 tools / 51 segments) saves 201/200 anonymously; over-cap variants 422 with no row stored (unit + integration pins).
- [ ] A linked Share upserts the current conversation before minting the token and aborts (banner, no token) on an upsert failure; the unlinked branch is byte-identical (A2, marker pins).
- [ ] A save point dropped by an in-flight upsert re-runs once after it settles (A3, marker pin); a failed save's note persists until the next success, one 5 s retry on network/5xx only (A4, marker pins).
- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** (the four scenarios); `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green unedited.
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `git diff --stat` limited to `app/schemas.py`, `frontend/assets/app.js`, the test files, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency).
- [ ] One atomic `--no-gpg-sign` commit (e.g. `fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run`); phase dir moved to `.agents/phases/complete/`.
## Owner action (data repair — NOT code, recorded here so the phase's commit message can cite it)
The two affected shared rows (`8b345ab5…`, `be92e464…`) still hold the user-only snapshot. Their brain answers were never persisted (the SSE bytes are not stored server-side). If the owner's browser still carries the conversations (`bor.chat.v1` localStorage restore), opening the chat page (which re-links to the row) and letting the FIXED client's next save point (or a Share — now save-first) fire restores the full content through the existing upsert; otherwise the answers are unrecoverable and the rows may be deleted from History.
@@ -0,0 +1,32 @@
# Task 01 — The saved-chat list caps move to the real agent configuration (A1)
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Raise the five `ChatMessage` list caps from the round-cap-10 era to the production `BOR_AGENT_MAX_ROUNDS=50` reality (the incident turn's 22-source done frame 422'd at the old 20 cap), update the sizing rationale in the docstrings, and move the boundary pins — so a long research turn's done-time auto-save lands instead of 422ing into the phase-55 quiet-failure, while the SEC-05 boundedness (fixed item-count boundary + the 8 M total budget) survives.
## Work
1. `app/schemas.py` — `ChatMessage` (L1381; the fields at L1458–1481):
- `sources: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20)
- `related: list[SourceRef] | None = Field(default=None, max_length=200)` (was 20)
- `suggestions: list[_Chip] | None = Field(default=None, max_length=100)` (was 50; the `_Chip` 200-char item cap is unchanged)
- `tools: list[ToolCall] | None = Field(default=None, max_length=200)` (was 50)
- `segments: list[ChatSegment] | None = Field(default=None, max_length=450)` (was 50; the comment's "max_length=50 (the phase-83 voice): far past any realistic turn — the round cap is 10 → ≤ 20 entries; 50 matches the tools cap" is REPLACED with the phase-189 sizing: ≤ 200 tool entries (4× the production round cap of 50 — a round is one model request, one-or-more calls per round are legal) + ≤ ~50 thinking runs (one per round) + margin)
- **UNCHANGED, pinned as-is:** `text`/`thinking` 600 000, `error` 500, `image` 500, `answer_text` 50, the `SourceRef` field caps (120/1000/500 + `image_url`), the `ToolCall` field caps (name 100 / argument 2000 / the additive keys), `SavedChatCreate`/`SavedChatUpdate` `max_items=500`, `SAVED_CHAT_TOTAL_CHARS_BUDGET = 8_000_000` and `_require_within_total_budget` (the verbatim-block accounting untouched).
- The `ChatMessage` class docstring: the phase-83 paragraph's "(sources ≤ 20 — top-N docs + agent reads; related ≤ 20 — the related-doc tier, same ref shape; suggestions ≤ 50 chips of ≤ 200 chars; tools ≤ 50 — one entry per tool call, the round cap is 10)" is UPDATED — the cited tier has been the READ DOCS since phase 119 (`merge_cited_docs`, `app/api/chat.py` L587): retrieval top-N ∪ the agent's read docs, bounded by `agent_max_rounds` (production 50 via `BOR_AGENT_MAX_ROUNDS`, the validator L— unbounded above), so the caps are `sources`/`related`/`tools` 200 (4× the production round cap) and `suggestions` 100 (20× the default 5; `suggested_docs` is tunable, unbounded above), `segments` 450; the phase-83 history (the 32 000 mirror raised to 600 000 with phase 164) is KEPT; one sentence added: the fixed boundary bounds item COUNT (an operator raising `BOR_AGENT_MAX_ROUNDS` far past 50 still hits a boundary) while `SAVED_CHAT_TOTAL_CHARS_BUDGET` is the byte bound (worst-case list bytes ≈ 360 KB / 440 KB / 1 MB — orders of magnitude inside 8 M).
2. `tests/unit/test_schemas.py` — the boundary-constant block (L47–75): `SOURCES_CAP = 200`, `TOOLS_CAP = 200`, `SUGGESTIONS_CAP = 100` (a `RELATED_CAP = 200` and `SEGMENTS_CAP = 450` constant added in the same style); the existing at-cap/one-over pins (L145–216 and the `segments` pins at L922–1009) now bind the NEW values with no edits (they read the constants); the realistic-payload round-trip pin (around L563) is EXTENDED with the incident shape: a brain record carrying 22 `sources`, 33 `tools`, and 40 `segments` (33 tool entries + 7 thinking entries) validates and `model_dump()`s byte-identically (the `_drop_image_when_absent` omission rules untouched — a record with all the keys present keeps them).
3. `tests/integration/test_chats_api.py` — `test_post_rejects_sources_over_20_and_stores_nothing` (L1862) is RENAMED/retargeted to the new boundary (`test_post_rejects_sources_over_200_and_stores_nothing`: 201 `SourceRef` items → 422, no row); NEW pins: (a) `test_post_accepts_the_incident_payload_shape` — an anonymous `POST /api/chats` with a 2-message conversation (user + a brain record carrying 22 `sources`, 33 `tools`, 51 `segments` = 33 tool entries + 18 thinking entries — the incident turn's shape) → **201**, the row's `messages` round-trips all three lists intact; (b) the same shape through `PUT /api/chats/<id>` → **200**, the row's message count bumps to the new array; (c) `test_put_rejects_over_cap_lists_and_leaves_row_unchanged` — 201 `tools` / 451 `segments` / 101 `suggestions` each → **422** and the row's stored messages are byte-identical (the 422 is the framework's standard body — the phase-83 A2 shape, no custom error copy). Every pre-existing pin (the full brain-record pin, the stale/share pins, the 500-item messages cap, the total-budget pins) stays green unedited.
4. Verify: `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v`; `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the new boundary pairs (200/201 `sources`, 200/201 `related`, 100/101 `suggestions`, 200/201 `tools`, 450/451 `segments`) pinned at BOTH boundaries; the extended realistic round-trip pin.
- Integration: the incident payload shape 201/200s anonymously (pre-fix: 422); the over-cap 422 pins at the new boundaries with no row stored / row unchanged; the pre-existing pins green unedited.
- Coverage: **>90%** on `app/` (the changed file is `app/schemas.py` — declarative, exercised by every pin).
## Completion Criteria
- [ ] `ChatMessage` list caps are 200/200/100/200/450 (sources/related/suggestions/tools/segments); every other bound unchanged; the docstring rationale is the `merge_cited_docs`/`agent_max_rounds` one with the 8 M-budget sentence.
- [ ] `uv run pytest tests/unit/test_schemas.py tests/integration/test_chats_api.py -v` green — the incident shape 201/200s, the over-cap variants 422 with no row stored, all pre-existing pins green unedited.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
- [ ] no behavior change in completed work (the SSE `done` event is untouched — `ChatDoneEvent.sources` has no list cap; the `ChatMessage` caps bind the saved-chat boundary only)
@@ -0,0 +1,31 @@
# Task 02 — Share saves first: a linked Share never publishes a row the conversation has outgrown (A2)
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Make `shareCurrentChat`'s linked branch upsert the CURRENT conversation (the same upsert semantics as `persistConversation`) BEFORE minting the share token — so a Share can never freeze a stale row (the incident: the row held only the question, the Share minted a token on it, and the shared page showed no answer) — and abort the share with the existing neutral error banner when the upsert fails. The unlinked branch (save-AND-share in one `POST`) is byte-identical.
## Work
1. `frontend/assets/app.js` — `shareCurrentChat` (L3345), the linked branch (the `if (currentChatId)` arm, ~L3353–3363) becomes, in order:
1. the upsert: `const body = JSON.stringify({ messages: conversation }); const headers = { "Content-Type": "application/json" };` → `res = await fetch(`/api/chats/${currentChatId}`, { method: "PUT", headers, body });` with the SAME 404→recreate fallback as `persistConversation` (L3227–3234): on a 404, `currentChatId = null; res = await fetch("/api/chats", { method: "POST", headers, body });` and, on the 201, re-link (`currentChatId = String((await res.json()).id)`);
2. an upsert FAILURE (non-ok `res` after the fallback, or a network `throw`) → `showErrorBanner("Couldn't share the conversation — try again.")` and `return` (NO token minted, NO toast — the existing failure UI, the existing `finally` still releases `shareBtn.disabled`);
3. only then the existing `POST /api/chats/<id>/share` mint (unchanged) and the existing success paths (clipboard copy → the inline-field fallback, the `#send-status` line, the task-04 toast on BOTH success branches) — byte-identical.
The function docstring gains the phase-189 paragraph: the linked branch upserts first because a Share that minted a token on a row the conversation had outgrown published a truncated snapshot (the incident — the done-time auto-save had 422'd into the quiet-failure and the row held only the question); an upsert failure aborts the share (the owner must never be handed a link to content that is not the current conversation). The unlinked branch's docstring ("one action saves AND shares") is unchanged.
2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention, the module's existing style — no browser):
- `test_share_linked_branch_upserts_before_the_share_mint`: the linked branch of `shareCurrentChat` contains the PUT to `` `/api/chats/${currentChatId}` `` (the `persistConversation`-shaped upsert) positioned BEFORE the `` `/api/chats/${currentChatId}/share` `` fetch in the function body (a string-ordering pin: the PUT line's index < the share-mint line's index), and carries the 404→recreate fallback markers (the `status === 404` arm + the `POST` to `"/api/chats"` with the same `body`/`headers`);
- `test_share_aborts_on_upsert_failure`: the failure arm between the upsert and the mint sets the EXACT neutral banner text `"Couldn't share the conversation — try again."` and `return`s (no `showToast` call between the upsert's failure check and the banner — the toast is success-only, the existing pin stays green);
- `test_share_unlinked_branch_unchanged`: the unlinked branch still POSTs `"/api/chats"` with `share: true` in the body and re-links on the 201 (the existing pins, green unedited);
- the existing phase-55 pins (the always-visible `#share-chat-btn` markup, the neutral error copy, the `showToast` success-only contract, the double-click guard released in the `finally`) stay green unedited.
3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green.
## Testing & Quality
- Unit (JS markers): the upsert-before-mint ordering, the 404→recreate fallback in the share, the abort-on-failure path (banner, no token, no toast), the unlinked branch unchanged; every pre-existing pin green.
- Coverage: the frontend change is marker-pinned per the house JS convention (no JS runtime in the unit gate); the behavior is E2E-pinned by task 04.
## Completion Criteria
- [ ] `shareCurrentChat`'s linked branch: PUT upsert (with the 404→recreate fallback) → failure abort (neutral banner, no token, no toast) → share mint → the existing success paths byte-identical; the docstring says why.
- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
- [ ] no behavior change in completed work (the unlinked branch, the toast/banner contract, the `share_chat` endpoint — token-only, `app/api/chats.py` UNTOUCHED)
@@ -0,0 +1,34 @@
# Task 03 — Dropped saves re-run; a failed save stays visible and retries once (A3/A4)
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Close the two silent-loss paths in `persistConversation`: a save point dropped by the `persisting` double-fire guard (evidence 6 — a done-time save dropped by an in-flight send-time save vanishes until the next save point, which may never come) re-runs once after the in-flight upsert settles; a failed save's status note STAYS until the next success and a transient failure (network / 5xx) retries once after 5 s — while the phase-55 A2 contract's shape is unchanged (one-line `#send-status` note, NO error banner, the turn never blocks, silent success).
## Work
1. `frontend/assets/app.js` — `persistConversation` (L3216–3260) and its guard (L3216):
- `let persisting = false;` stays (the one-upsert-at-a-time guard); add `let savePending = false;` (A3) and `let saveFailedSticky = false; let retryScheduled = false;` (A4) at the same module scope, each with a one-line comment naming the phase-189 decision.
- **A3 — the overlap re-run:** the top guard becomes `if (persisting) { savePending = true; return; }` (a save point that arrives mid-upsert is PENDING, not dropped). At the top of the body, after the guard: `savePending = false;` (a fresh run consumes the flag — the pending re-run is this run when it was set by a re-run). In the `finally` (after `persisting = false;`): `if (savePending) { savePending = false; void persistConversation(); }` — the re-run is a fresh serialized call (it starts only after the in-flight upsert fully settled; never concurrent with it). The re-run reads `conversation` at call time, so the dropped done-time save lands with the brain record. No new save points, no other function touched.
- **A4 — sticky note + one transient retry:** in the failure paths (the `!res.ok` arm and the `catch` arm), after setting the existing note text, set `saveFailedSticky = true;` and — transient only — `if (!retryScheduled && (res.status >= 500 || <network-failure-arm>)) { retryScheduled = true; setTimeout(() => { retryScheduled = false; void persistConversation(); }, 5000); }` (the `catch` arm IS the network-failure arm; the 4xx — every other non-ok — NEVER schedules: a 422 is deterministic, the phase-189 task-01 cap fix is the remedy, and a retry would 422 again; the one-shot `retryScheduled` guard keeps overlapping failures from stacking retries). In the success path (the `staleBanner` clearing block), add: `if (saveFailedSticky) { saveFailedSticky = false; sendStatus.textContent = ""; }` (a success clears the stuck note — the note persists until the next success; other flows may overwrite the note in the meantime, best-effort visibility, the flag + the next save point are the guarantee).
- The `persistConversation` docstring: the "double-fire guard" paragraph is updated — a dropped save point is now PENDING and re-runs once after the in-flight upsert settles (A3); the A2 quiet-contract paragraph gains the two phase-189 lines (the note persists until the next success; one 5 s retry on network/5xx, never on 4xx — the note text and the no-banner shape are UNCHANGED).
- `regenerateStaleChat` (L4439–4460, its own upsert copy) is LEFT AS IS (its failure surfaces through the turn's banner state; A1 removes the 422 class it could hit).
2. `tests/unit/test_save_chat_ui.py` — marker pins (the house JS-marker convention):
- `test_persisting_overlap_sets_pending_and_reruns`: `persistConversation` contains the `savePending = true` assignment inside the `if (persisting)` arm, the `savePending = false` reset at the top of the body, and the `finally` block re-runs `persistConversation()` guarded by `savePending` (the ordering: `persisting = false` before the `savePending` check in the `finally`);
- `test_failed_save_note_is_sticky_until_success`: the failure arms set `saveFailedSticky = true`, and the success path clears the flag + resets `sendStatus` (the pin: the success block contains the `saveFailedSticky` guard);
- `test_transient_failure_retries_once_after_5s`: the retry scheduling is present with the 5000 ms delay, the `>= 500` status condition, the `catch`-arm scheduling, and the one-shot `retryScheduled` guard (set before the `setTimeout`, reset inside it);
- `test_4xx_never_retries`: the retry condition is `res.status >= 500` (a 4xx non-ok schedules nothing — the pin: no other `setTimeout` scheduling in the failure arms);
- the existing phase-55 A2 pins (the one-line note text `"Couldn't save automatically — will try on the next message."`, no error-banner call in `persistConversation`, the silent-success shape, the `persisting` guard) stay green unedited.
3. Verify: `uv run pytest tests/unit/test_save_chat_ui.py -v`; `uv run pytest` green.
## Testing & Quality
- Unit (JS markers): the pending re-run (set/reset/finally-re-run ordering), the sticky-note lifetime, the one-shot 5 s transient retry, the 4xx exclusion; every pre-existing pin green.
- Coverage: the frontend change is marker-pinned per the house JS convention; the behavior is E2E-pinned by task 04 (the overlap scenario).
## Completion Criteria
- [ ] `persistConversation`: an overlapping save point sets `savePending`; the `finally` re-runs once after `persisting = false`; a fresh run consumes the flag.
- [ ] A failed save's note persists until the next success; a network/5xx failure schedules ONE 5 s retry (one-shot guard); a 4xx never retries; the A2 contract's shape (note text, no banner, silent success) is unchanged.
- [ ] `uv run pytest tests/unit/test_save_chat_ui.py -v` green — the new marker pins + every pre-existing pin.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the phase's final pass re-checks both project-wide)
- [ ] no behavior change in completed work (`regenerateStaleChat` untouched; no new save points; `app/api/chats.py` untouched)
@@ -0,0 +1,57 @@
# Task 04 — The E2E suite, the full gates, the docs pass, the commit
**Phase:** `189_chat_save_long_turns` · **Source:** owner incident 2026-10-16
**Story:** n/a
## Objective
Pin the owner-visible contract of the whole phase through the real UI: the incident regression at the wire (a 22-source brain record saves instead of 422ing), a mock turn's done-time auto-save landing, the overlap re-run (a dropped save lands after the in-flight upsert settles), and share-saves-first (a Share with the auto-saves still in flight publishes the BRAIN message, not just the question) — then the full gate, the docs pass, and the atomic commit.
## Work
1. `tests/e2e/test_chat_save_long_turns.py` — the phase's dedicated Playwright suite (the house `conftest` app + `mock_llm.py` pattern, run **in isolation** per AGENTS rule 4; a fresh saved-chat row per test — the house `saved_chats` cleanup helper pattern from `test_chat_history.py`):
- `test_incident_payload_shape_saves_at_the_wire`: through `page.request` (the `test_chat_save_payload_limits.py` pattern — the app's own origin), an anonymous `POST /api/chats` with the INCIDENT shape (a user message + a brain record carrying 22 `sources` (`SourceRef` dicts at realistic column-length values), 33 `tools` (`ToolCall` dicts), 51 `segments` = 33 `{kind: "tool", …}` + 18 `{kind: "thinking", text: …}`) → **201**; a follow-up `PUT` replacing the brain record with the same shape → **200** and the row's `messages` round-trips all three lists (a `GET /api/chats/<id>` admin assertion). (Pre-fix: both 422 — the incident.)
- `test_done_time_auto_save_lands_for_a_mock_turn`: the real UI — send a mock question (the `mock_llm.py` no-tools flow, the `test_chat_history.py` send pattern), wait for the done bubble (the house `wait_for_answer` helper), then `GET /api/chats/<id>` (the row id from the created-response `#…` — the house pattern reads `currentChatId` via `page.evaluate`): the row has **2** messages and the brain record's `text` equals the mock answer.
- `test_dropped_done_time_save_reruns_after_the_inflight_upsert_settles`: `page.route`-delay the FIRST PUT to `/api/chats/**` (the send-time auto-save) by 3 000 ms (a `page.route("**/api/chats/**", handler)` that resolves on `route.fulfill` after a `wait_for_timeout` ONLY for the first matched PUT — a module-level counter; later PUTs pass through untouched), then send the mock question and let the mock turn settle (the done bubble appears in flight — the mock is fast, the upsert is still parked at 3 s). Wait to ~5 s; `GET /api/chats/<id>`: the row has **2** messages (the dropped done-time save re-ran after the parked upsert settled — pre-fix: the row held 1).
- `test_share_saves_first_publishes_the_brain_message`: the same 3 s delay on the first PUT; send the mock question; while the send-time save is still parked (the done bubble has appeared), click `#share-chat-btn` (the house click helper, wait for the toast). The share's own upsert PUT passes the route filter UNdelayed (the counter already consumed by the parked send-time save), mints the token, and the toast lands; open `share_url` (the `test_share_chat.py` pattern — the `/shared/<token>` page, the `#shared-title` visible) and assert the shared page renders the BRAIN answer text (a `.msg .bubble` containing the mock answer string) — pre-fix: the shared page showed only the question. Assert the row (via admin `GET`) has 2 messages.
- Module docstring: the incident's evidence summary (the 422 PUT log line, the 22-source done frame, `created_at == updated_at` single-write row) so a future reader sees WHY these four tests exist.
2. The regression suites (green UNEDITED — do not touch them): `tests/e2e/test_chat_save_payload_limits.py` (oversized-TEXT 422 + small-save 201 — the text caps are unchanged), `tests/e2e/test_chat_history.py`, `tests/e2e/test_share_chat.py`.
3. The full gate, in order: `uv run pytest -v` (unit + integration 100% green); `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/`); `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` (**in isolation**); `uv run ruff check . && uv run pyright` clean.
4. The docs pass (the phase's Documentation & Consistency mandate): review `README.md` and `AGENTS.md` in their entirety against the post-phase app; fix every inconsistency found, pre-existing ones included. `README.md`: the saved-chat section gains the ops note — a long turn (many cited docs / tool rounds) saves cleanly (the caps now follow the agent's real round cap — 200/100/450), a Share saves the current conversation before minting the token (a failed share upsert aborts — no link to a truncated snapshot), a failed auto-save stays visible until the next success and retries once after 5 s on a transient failure (a 4xx never retries). `AGENTS.md`: verify the phase-55 save contract is not described anywhere in it (the save points are not enumerated — the UI-Structure rule's "in-flight turn" clause is untouched); if the final pass finds any stale save-contract sentence, fix it in place (no new rule added).
5. The atomic commit (AGENTS rule 8; the git-commit skill's `--no-gpg-sign` + `LLM-Generated: true` trailer): `git add -A && git commit --no-gpg-sign -m "fix(chats): save long agent turns — realign schema caps, share saves first, dropped saves re-run
The done-time auto-save of a long turn (production BOR_AGENT_MAX_ROUNDS=50;
the incident turn cited 22 docs, 33 tool calls, 34 rounds) 422'd at the
phase-83 list caps sized for the round-cap-10 era (sources ≤ 20, tools ≤ 50,
segments ≤ 50), and the phase-55 quiet-failure swallowed it: nothing retried,
so the row kept its pre-turn state — and a Share minted a token on that stale
row, publishing a conversation without its answer (incident 2026-10-16,
shared 0101d2ca… / row 8b345ab5…).
- A1: ChatMessage list caps follow the real configuration — sources/related/
tools 200 (4× the production round cap; the cited tier is the read docs,
merge_cited_docs), suggestions 100, segments 450; every other bound and the
8 M total budget unchanged (SEC-05 boundedness survives).
- A2: a linked Share upserts the current conversation before minting the
token and aborts on an upsert failure — never a link to a truncated
snapshot.
- A3: a save point dropped by the in-flight guard re-runs once after the
upsert settles.
- A4: a failed save's note persists until the next success; one 5 s retry on
network/5xx (a 4xx never retries).
LLM-Generated: true"`.
6. Move the phase directory: `mv .agents/phases/todo/189_chat_save_long_turns .agents/phases/complete/` and `git add .agents && git commit --no-gpg-sign -m "chore(agents): complete phase 189 (chat save long turns)"` — OR fold the phase move into the atomic commit per the house convention (check the last 3 phase commits' shape and match it; the AGENTS rule 8 says one atomic commit per phase including the phase files — if the house convention is one commit, `git add` the phase dir into step 5's commit and skip step 6's commit, keeping the `mv`).
## Testing & Quality
- E2E: the four scenarios above, in isolation.
- Regression: the three listed suites green unedited.
- Coverage / lint / types: the full gate (step 3).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` green **in isolation** — the incident shape 201/200 at the wire, the mock turn's done-time save lands, the dropped save re-runs (final row 2 messages), the share with in-flight auto-saves publishes the brain message on the shared page.
- [ ] `test_chat_save_payload_limits.py`, `test_chat_history.py`, `test_share_chat.py` green UNEDITED.
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] README.md / AGENTS.md reviewed in full; the saved-chat ops note in place; no stale save-contract sentence left.
- [ ] `git diff --stat` (before the commit) limited to `app/schemas.py`, `frontend/assets/app.js`, `tests/unit/test_schemas.py`, `tests/unit/test_save_chat_ui.py`, `tests/integration/test_chats_api.py`, `tests/e2e/test_chat_save_long_turns.py`, `README.md`, the phase files (NO `app/api/chats.py` diff, no migration, no new dependency).
- [ ] the atomic `--no-gpg-sign` commit exists (message above); the phase dir is in `.agents/phases/complete/`.
- [ ] full test suite **100% passing** (every test in the project, not just this task's)
- [ ] no behavior change in completed phases (the SSE `done` event, the `share_chat` endpoint, the unlinked share branch, the phase-55 A2 contract's shape — all unchanged)
@@ -0,0 +1,51 @@
# Phase 190 — The Settings-panel app name is the app name again (and saves apply live)
**Source:** `TODO.md` L3 — "changing the App name in settings broke, it no longer updates the name."
**Story:** n/a (owner report)
**Depends on:** none functionally (builds on the completed phase-168 admin Settings panel and phase-91/92 admin Theme tab).
## Evidence (diagnosed from the code, read this not the chat)
1. **Two writers for one setting.** `app_name` (and `input_placeholder` / `footer_text`) is now writable from TWO admin surfaces: the Settings panel (`/api/admin/settings/app_name` → `app_settings` table → the `app.config` DB-over-env overlay, phase 168) and the Theme tab (`/api/ui-settings` PUT → the single `ui_settings` row, phase 91).
2. **The Theme tab row WINS.** `app.core.theming.effective_settings` (L144–177) resolves each string as: the `ui_settings` row value when it is a non-empty string, **else** `settings.<field>` (the env value, which — via the phase-168 overlay — is the Settings-panel row value when one exists). So the order is Theme-tab row → Settings-panel row → env.
3. **A Theme-tab save bakes the strings in.** `theme.js` `collectBody()` (L526) collects ALL 20 fields — the 3 string inputs are pre-populated with the EFFECTIVE value on load — and `PUT /api/ui-settings` writes every non-empty string into the row. So the moment an admin saves ANY color from the Theme tab, the current app name is baked into `ui_settings.app_name`, and from then on **every later Settings-panel app-name change is shadowed** by that baked row. This is exactly "it no longer updates the name" — the Settings panel shows the new value with its `db` badge, but `/api/config` (which serves `effective_settings`) reports the Theme tab's baked name.
4. **No live brand update, on either surface.** `settings.js` re-GETs the catalogue after a save but never re-applies the brand strings to the open document; `theme.js` after a save runs `applyServedTheme(settings)` which reconciles the COLOR palette (phase 92) but never touches `document.title`, `.brand-text`, the footer or the composer placeholder. `brand.js` runs the brand path once at boot from `/api/config`. So even when the server-side value is right, the open page keeps showing the old name until a manual reload.
## Objective
The Settings panel is the single writable owner of the three brand strings and always wins; the Theme tab becomes colors-only (its PUT no longer writes the strings, its form drops them — for legacy baked-in rows the panel still wins); and a save in either surface re-applies the brand strings to the open document immediately, so the change is visible the moment "save" is clicked (the phase-168 contract extended to the display).
## ASSUMPTIONS (owner-confirmed 2026-10-18)
- **A1 — Panel wins, strings stripped from the Theme tab.** The Settings-panel row beats the Theme-tab row for `app_name` / `input_placeholder` / `footer_text` (no data migration — existing baked-in rows simply stop shadowing the panel; with no panel row they still apply as the documented fallback). The Theme tab's form drops the three string inputs and its PUT body no longer carries them (the server ignores string fields in the PUT body defensively — an old client's save can no longer re-bake a string). The Theme tab's GET keeps reporting the effective strings (read-only, old-client compat).
- **A2 — Live brand on save.** After a successful Settings-panel save/clear or Theme-tab save/reset, the open document's `document.title`, `.brand-text`, the footer and the composer placeholder are re-applied from a fresh `/api/config` — no reload. The boot path (`brand.js`) is byte-identical.
## Context
- `app/core/theming.py` — `effective_settings` (L144–177: the string resolution to change), `STRING_FIELDS`, `COLOR_FIELDS`, `BUILTIN_COLORS`; `app/api/config.py` — `GET /api/config` (serves `effective_settings` + `source` per string); `app/api/ui_settings.py` — the phase-91 admin API (GET reports 20 fields, PUT writes 3 strings + 17 colors, DELETE resets the row); `app/core/settings_overlay.py` + `app/config.py` `invalidate_settings` (L1129) — the phase-168 DB-over-env overlay (the panel rows; `settings.app_name` already resolves through it); `app/api/admin_settings.py` (phase 168, the panel surface).
- `frontend/assets/brand.js` — the boot-time brand path (title / `.brand-text` / footer / placeholder from `/api/config`); `frontend/assets/theme.js` — `FIELDS` (L183–185 the 3 string fields), `collectBody` (L526), `saveTheme` (L580, `applyServedTheme` L589), `resetTheme` (L597); `frontend/assets/settings.js` — the phase-168 panel (its header names the phase-168 contract "the change should take effect on clicking 'save'"); `frontend/index.html` — the Theme-tab markup (`#theme-app-name` L1354 et al., the three string rows to remove).
- `tests/unit/test_theming.py` (the `effective_settings` pins), `tests/integration/test_ui_settings_api.py` (the phase-91 API pins), `tests/e2e/test_admin_theme_tab.py` (fills `#theme-app-name` — must move with the strip), `tests/e2e/test_settings_panel.py` (the phase-168 pins), `tests/unit/test_save_chat_ui.py` (JS marker-pin convention).
## Tasks
1. `01_reproduce_shadow_and_staleness.md` — failing tests pinning both defects (the shadowing through `effective_settings` / `/api/config`, the Theme-tab PUT string bake, the no-live-update behavior).
2. `02_precedence_and_strip_theme_tab.md` — A1: the resolution order flips to panel → tab → env; the Theme tab drops the 3 string inputs and its PUT stops writing them; the pinned theme-tab suites move.
3. `03_live_brand_on_save.md` — A2: one shared `applyBrandFromConfig` pass (the `brand.js` DOM path extracted, boot behavior byte-identical) re-applied after a Settings-panel save/clear and a Theme-tab save/reset.
4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the incident regression through the real UI), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_theming.py` — the NEW resolution order pins (panel row beats a conflicting `ui_settings` row; with no panel row the tab row still applies — the A1 fallback); the pre-phase pins that do not involve a panel row stay green.
- **Integration:** `tests/integration/test_ui_settings_api.py` — a Theme-tab PUT with string fields in the body **ignores** them (the stored row's string columns are untouched / the response no longer round-trips a changed string); `tests/integration/` Settings-panel pins (phase 168) green — the panel write + `invalidate_settings` path is untouched.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_app_name_settings_fix.py`, run in isolation — (a) the incident regression: set the name in the Settings panel → save a color in the Theme tab → the name still shows the panel's value (pre-fix: the baked tab value); (b) live apply: change the app name in the Settings panel → `document.title`, `.brand-text`, the footer and the composer placeholder update WITHOUT a reload; (c) a Theme-tab save no longer bakes the name (a later panel change is visible on reload); (d) a panel reset (DELETE) falls back to env live.
- **Regression:** `tests/e2e/test_settings_panel.py`, `tests/e2e/test_admin_theme_tab.py` (moved in task 02), `tests/e2e/test_theme_save_and_coverage.py` green.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the frontend change is marker-pinned per the house JS convention.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the settings/theme section notes the single-writer ownership (the Settings panel owns the brand strings; the Theme tab is colors-only) and the live-apply behavior (task 04). `AGENTS.md`: the phase-168 "settings overlay" description gains the resolution-order clause if it names one (verify, do not add if absent).
## Completion Criteria
- [ ] `effective_settings` resolves the 3 strings panel-row → tab-row → env (unit pins, both branches).
- [ ] `PUT /api/ui-settings` with string fields in the body does not write them (integration pin); the Theme-tab form no longer renders the 3 inputs (E2E pin).
- [ ] A Settings-panel app-name change is visible in `document.title`, `.brand-text`, the footer and the placeholder without a reload (E2E); the Theme-tab color save no longer shadows it (E2E regression of the incident).
- [ ] the `brand.js` boot path is byte-identical (marker pin) — no double-apply at boot.
- [ ] test suite green, coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed phases (the 17-color palette, the phase-168 panel, the env fallback all behave as before).
- [ ] one atomic Conventional Commits commit with `--no-gpg-sign`.
@@ -0,0 +1,31 @@
# Task 01 — Reproduce and pin the two defects (shadowing + no live update)
**Phase:** `190_app_name_settings_fix` · **Source:** `TODO.md:4 — "changing the App name in settings broke, it no longer updates the name."`
**Story:** n/a
## Objective
Before touching behavior: pin BOTH defects with failing tests — (a) the `ui_settings` row shadows the Settings-panel row in `effective_settings` and therefore in `GET /api/config`, and a Theme-tab PUT bakes the effective strings into the row (the mechanism of "it no longer updates"); (b) a Settings-panel save never re-applies the brand strings to the open document (marker pin on the ABSENT behavior, inverted in task 03).
## Work
1. `tests/unit/test_theming.py` — NEW pins (failing until task 02):
- `test_effective_app_name_prefers_settings_panel_row_over_ui_settings_row` — an `app_settings` row (via the phase-168 overlay / a `Settings` with `app_name="Panel Name"`) + a `ui_settings` row with `app_name="Tab Name"` → `effective_settings["app_name"] == "Panel Name"` (today: `"Tab Name"` — the documented A1 order). Same pair for `input_placeholder` and `footer_text`.
- `test_effective_strings_fall_back_to_ui_settings_row_without_panel_row` — `ui_settings` row set, no panel row (the `Settings` at the env default) → the tab value applies (the A1 fallback this pin protects — it must stay GREEN pre-phase and post-phase).
- The pre-existing pins that assume tab-over-env with NO panel row stay green unedited.
2. `tests/integration/test_ui_settings_api.py` — NEW pins (failing until task 02):
- `test_put_ignores_string_fields_in_body` — a `PUT /api/ui-settings` whose body carries `app_name` / `input_placeholder` / `footer_text` DIFFERENT from the stored row's values and from the effective value → 200; the stored row's string columns are UNCHANGED (re-GET shows the old values); the 17 colors in the same body ARE stored (the colors half of the contract survives).
- `test_get_reports_effective_strings_read_only` — with a conflicting panel row, `GET /api/ui-settings` reports the EFFECTIVE (panel) value for the strings (read-only view; the A1 order reaches the API).
3. `tests/unit/test_brand_live.py` (NEW module) — marker pins on the CURRENT (absent) behavior, documented as "inverted in task 03":
- `settings.js` contains NO call that re-applies `document.title` / `.brand-text` after a save (assert the marker is ABSENT pre-phase — the test asserts the file's save path does not reference the brand DOM ids; task 03 inverts the expectation);
- `theme.js` `saveTheme`'s success path calls `applyServedTheme` (colors only) and does NOT touch the brand DOM (marker pin — stays green through the phase; the live-apply added in task 03 is asserted there, not here).
4. Verify: `uv run pytest tests/unit/test_theming.py tests/integration/test_ui_settings_api.py tests/unit/test_brand_live.py -v` — the NEW pins FAIL (documented), everything else green.
## Testing & Quality
- Unit/integration: the pins above — the shadowing mechanism, the string-bake, the fallback (green pre-phase), the absent-live-behavior markers.
- Coverage: **>90%** on `app/` (no app code changes this task — the pins exercise existing code).
## Completion Criteria
- [ ] the 3 `effective_settings` order pins FAIL pre-fix (documented in the test docstrings: "fixed by task 02").
- [ ] the string-bake integration pin FAILS pre-fix (documented); the colors-half pin is GREEN pre-fix.
- [ ] the live-behavior marker pins pass (asserting the ABSENT behavior, inverted in task 03).
- [ ] full test suite **100% passing** except the documented failing pins (the failing pins are the task's artifact — the phase's E2E gate runs them post-fix).
- [ ] no behavior change in completed work (app/ untouched).
@@ -0,0 +1,45 @@
# Task 02 — The panel wins; the Theme tab becomes colors-only (A1)
**Phase:** `190_app_name_settings_fix` · **Source:** `TODO.md:4 — "changing the App name in settings broke, it no longer updates the name."`
**Story:** n/a
## Objective
Flip the string resolution order to Settings-panel row → Theme-tab row → env (no data migration — existing baked-in rows stop shadowing the panel), and strip the three string inputs from the Theme tab: the form drops them, `collectBody` stops collecting them, and the server's `PUT /api/ui-settings` persists ONLY the 17 colors (string fields in the body are ignored defensively — an old client's save can no longer re-bake a string; string columns are left untouched, not cleared, so the A1 fallback survives).
## Work
1. `app/core/theming.py` — `effective_settings` (L170–173), the string loop becomes a three-way resolution per field:
- **panel row** — `getattr(settings, field)` (the `Settings` passed by the route; the phase-168 overlay makes it the panel value when an `app_settings` row exists, else env) is used when a panel row exists for that key. The panel presence is read through the SAME overlay source the `Settings` object carries: the overlay's `app_settings` row map (`app.core.settings_overlay.load_overrides()` — the one query the overlay already caches per process; the route passes the settled `Settings`, so the check is `field in the override map` — implement as `has_override(field)` on the overlay module, one cached read, no second query per field);
- **else the tab row** — the `ui_settings` row value when it is a non-empty string (the phase-91 fallback, now second);
- **else the env value** — `getattr(settings, field)` (identical expression to today's fallback).
- The 17-color half is UNTOUCHED (row → `BUILTIN_COLORS`). The docstring's resolution table is rewritten (the old "DB value when non-empty, else env" becomes the three-way table).
- `app/api/config.py` — the per-string `source` badge (`db` / `env`): `db` now means the PANEL row; the tab row yields the existing non-`db` value (verify the current `source` vocabulary — if it is a two-value `db`/`env` pair, `db` = any stored row wins, unchanged; do NOT grow the vocabulary).
2. `app/api/ui_settings.py` — the PUT handler:
- persist ONLY the 17 `COLOR_FIELDS` from the body; the three string columns are NEVER written (the request model keeps accepting the fields — old-client compat — the handler drops them before persisting); the stored row's string values are left exactly as stored (a legacy baked-in row keeps its value; the A1 order decides visibility);
- the DELETE (reset) path: UNCHANGED (it nulls the whole row — including the strings; a reset is a user decision, and the A1 fallback then falls through to env, which is the documented reset behavior).
- the GET: unchanged (reports the effective strings — with the new order, a conflicting panel value is what the tab page pre-populates its (now removed) inputs with in an old client; the read-only report stays).
3. `frontend/assets/theme.js`:
- `FIELDS` (L183–185): the three `kind: "string"` entries are REMOVED — `FIELDS` becomes the 17 color fields only; everything FIELDS-driven (live preview, `collectBody`, `applyServedTheme`'s tag content) stays in `COLOR_FIELDS` order (the comment at L180–181 already says so — it becomes true);
- `collectBody` (L526): the body is the 17 colors only (no string keys at all);
- the string-specific docstring lines (L183–185 region + any "20 fields" counts in the module header) become "17 colors."
4. `frontend/index.html` — the Theme-tab markup: the three string rows (`#theme-app-name` L1354 et al., the placeholder + footer rows) are REMOVED from the form; the section count / any "3 strings + 17 colors" comment becomes colors-only.
5. `tests/unit/test_theming.py` — the task-01 failing pins now PASS unedited (they name the A1 order); the no-panel-row fallback pin stays green.
6. `tests/integration/test_ui_settings_api.py` — the task-01 string-bake pin now PASSES (body strings ignored, colors stored, string columns untouched).
7. The pinned theme-tab E2E suites that fill the removed inputs move:
- `tests/e2e/test_admin_theme_tab.py` — every step that fills `#theme-app-name` / `#theme-placeholder` / `#theme-footer` (or asserts their presence) is RETARGETED: the tab renders 17 color inputs only (a new pin: the three string inputs are ABSENT from the tab DOM — `page.locator("#theme-app-name")` count 0); a save still round-trips the 17 colors through the GET pre-population.
- `tests/e2e/test_theme_save_and_coverage.py` / any other suite that seeds a string through the tab: the string is now seeded through the Settings panel (the `test_settings_panel.py` helper) or the API (the integration test covers the wire; the E2E keeps its color assertions).
8. `tests/unit/test_theme_ui.py` (or the module that marker-pins `theme.js` — verify the name in `tests/unit/`) — marker pins: `FIELDS` has no `kind: "string"` entry; `collectBody`'s body has no string keys (the source-marker convention); the `saveTheme` success path still calls `applyServedTheme`.
9. Verify: `uv run pytest tests/unit/test_theming.py tests/integration/test_ui_settings_api.py -v` green; `uv run pytest tests/e2e/test_admin_theme_tab.py tests/e2e/test_settings_panel.py -v --no-cov` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the A1 order pins (both branches) + the marker pins (tab has no string fields).
- Integration: string-ignored PUT pin + the phase-91 color pins green unedited.
- E2E (regression, run in isolation): `test_admin_theme_tab.py` (moved) + `test_settings_panel.py` green.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `effective_settings` resolves the 3 strings panel → tab → env (both-branch unit pins green).
- [ ] `PUT /api/ui-settings` with string fields in the body stores no string change and stores the colors (integration pin green).
- [ ] the Theme-tab DOM renders 17 color inputs only; the 3 string inputs are absent (E2E pin).
- [ ] the phase-91 color contract (GET pre-population, save, reset) is green unedited.
- [ ] full test suite **100% passing**.
- [ ] no behavior change in completed work (the 17 colors, the phase-168 panel, the env fallback, the DELETE-reset behavior all as before).
@@ -0,0 +1,37 @@
# Task 03 — The save is visible without a reload (A2)
**Phase:** `190_app_name_settings_fix` · **Source:** `TODO.md:4 — "changing the App name in settings broke, it no longer updates the name."` (the phase-168 header contract: "the change should take effect on clicking 'save'")
**Story:** n/a
## Objective
A successful save in the Settings panel (or a per-row reset) and a save/reset in the Theme tab re-applies the brand strings to the OPEN document immediately — `document.title`, `.brand-text`, the footer and the composer placeholder — from a fresh `GET /api/config`, with no reload. The boot path in `brand.js` is extracted into the shared pass byte-identically, so boot and live-apply are ONE function and the never-stale contract (PLAN §7.4) covers the strings the way it already covers the palette (phase 92 `applyServedTheme`).
## Work
1. `frontend/assets/brand.js` — extract the DOM path (title / `.brand-text` / footer / placeholder assignment from the `/api/config` payload) into ONE named function `applyBrandFromConfig(cfg)` (same file, same module scope — no new file, no new CDN; `brand.js` stays the single owner of the brand DOM):
- the BOOT call site (`initBrand` / the boot fetch-then-apply) calls `applyBrandFromConfig(cfg)` — the observable boot behavior is byte-identical (same request, same assignments, same order);
- the function is exported to the module's existing export surface (the `window` / module pattern `brand.js` already uses — verify the convention; `theme.js` and `settings.js` import from the same pattern).
2. `frontend/assets/settings.js` — the save path:
- after a SUCCESSFUL per-key PUT (`PUT /api/admin/settings/<key>`, the phase-168 success note) and after a SUCCESSFUL per-row reset (DELETE) and the reset-all: `await fetch("/api/config")` → `applyBrandFromConfig(cfg)` — ONLY when the saved/removed key is one of the three brand strings (`app_name` / `input_placeholder` / `footer_text`); a save of any other key does NOT re-apply the brand (no wasted request, no DOM churn on 72 other rows);
- the §7.4 guard: while the re-apply fetch is in flight, a second save does not stack a second fetch (the existing `saveInFlight` guard's scope — extend, do not fork);
- the existing post-save re-GET of the catalogue is UNTOUCHED (it stays the catalogue's own refresh; the brand re-apply is additive and independent).
3. `frontend/assets/theme.js` — the save/reset path:
- `saveTheme` success (after the existing `applyServedTheme(settings)` at L589) and `resetTheme` success: `await fetch("/api/config")` → `applyBrandFromConfig(cfg)`. (The tab no longer edits the strings, but a RESET nulls the stored row — with a legacy baked-in string that reset makes the effective string FALL BACK (panel row → env), so the open document must re-read it; the save path keeps the same call — one code path for both, the body change is the colors.)
- the fetch-failure path: the brand keeps its last applied state (never a blanked title/footer — the phase-168 "never a blanked panel" precedent, applied to the brand: a failed re-apply leaves the DOM as-is; the save's own error note is the visibility).
4. `tests/unit/test_brand_live.py` — the task-01 absent-behavior marker pins are INVERTED:
- `settings.js`'s save path references `applyBrandFromConfig` (the marker is now PRESENT) and gates the re-apply on the three string keys (the source marker names the key check);
- `theme.js`'s `saveTheme`/`resetTheme` success paths reference `applyBrandFromConfig` (marker present);
- `brand.js` exports `applyBrandFromConfig` and the boot call site calls it (marker — the extraction, not a duplication: `initBrand`/boot contains no second copy of the four DOM assignments — they exist only in the shared function).
5. Verify: `uv run pytest tests/unit/test_brand_live.py -v` green; `uv run pytest tests/e2e/test_settings_panel.py -v --no-cov` green (the phase-168 suite — its save flow now also triggers the brand re-apply; its pins must stay green — the re-apply is additive); `uv run pytest` green.
## Testing & Quality
- Unit: the inverted marker pins (shared function exists; both surfaces call it; the boot path is the extraction — single copy).
- Integration: none (pure frontend — the wire contract is task 02's; `/api/config` is unchanged).
- E2E: covered by task 04 (the owner-visible arm).
- Coverage: **>90%** on `app/` (unchanged — the task is frontend; the app-side gate stays green).
## Completion Criteria
- [ ] `applyBrandFromConfig` is ONE function in `brand.js`; the boot call site uses it; the four DOM assignments exist in exactly one place (marker pins).
- [ ] `settings.js` re-applies the brand after a save/reset of a brand-string key only; `theme.js` re-applies after save and reset (marker pins).
- [ ] a failed re-apply leaves the DOM at its last applied state (marker pin on the failure path).
- [ ] `test_settings_panel.py` green unedited.
- [ ] no behavior change in completed work (the boot path, the phase-168 catalogue refresh, the phase-92 palette sync all as before).
@@ -0,0 +1,35 @@
# Task 04 — The phase E2E suite, the full gates, the docs pass, the commit
**Phase:** `190_app_name_settings_fix` · **Source:** `TODO.md:4`
**Story:** n/a
## Objective
The dedicated Playwright suite proves the incident regression and the live-apply contract through the real UI (run in isolation per AGENTS rule 4); the full gates pass; `README.md` / `AGENTS.md` are consistent with the post-phase app; one atomic commit.
## Work
1. `tests/e2e/test_app_name_settings_fix.py` (NEW — the module docstring names the incident: "a Theme-tab save used to bake the effective name into `ui_settings` and shadow every later Settings-panel change") — the app under test is the standard module-app pattern (the phase-168/91 suites' fixture — verify the convention in `tests/e2e/conftest.py` / the sibling suites):
- **(a) the incident regression (server-side, at the wire):** with an env default name, the admin sets the app name via the Settings panel UI (type + save, the phase-168 flow) → `GET /api/config` (via `page.request`) reports the panel value with the `db` source; then the admin opens the Theme tab, changes ONE color and saves → `GET /api/config` STILL reports the panel value (pre-fix: the baked tab value); the stored `ui_settings` row's string columns are unchanged (the API-level assertion — `page.request` GET `/api/ui-settings`).
- **(b) live apply, panel:** change the app name in the Settings panel (a brand-string row) → after the save note, WITHOUT a reload: `document.title`, `.brand-text` (the header), the footer text and the composer placeholder all carry the new name (the four assertions in one test — the A2 contract).
- **(c) live apply, tab reset:** with a legacy `ui_settings` row pre-seeded (the API seeds the row directly — a baked `app_name`), a Theme-tab Reset → the open document's brand falls back (panel row or env — seed NO panel row so the fallback is env) WITHOUT a reload.
- **(d) the boot path is byte-identical:** a fresh page load (no save) shows the env/panel name exactly as the pre-phase `brand.js` did (the `test_admin_theme_tab.py` boot pin, moved here if the moved suite no longer covers it).
- **(e) the tab DOM is colors-only:** the Theme tab renders 17 color inputs and NO `#theme-app-name` / placeholder / footer string inputs (the task-02 pin, asserted in the isolated suite as well).
2. Regression, run in isolation: `tests/e2e/test_admin_theme_tab.py` (moved in task 02), `tests/e2e/test_settings_panel.py`, `tests/e2e/test_theme_save_and_coverage.py` — all green unedited after the moves.
3. The full gates, in order:
- `uv run pytest` (unit + integration) — 100% passing;
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on `app/`;
- `uv run pytest tests/e2e/test_app_name_settings_fix.py -v --no-cov` — green in isolation;
- `uv run ruff check . && uv run pyright` — clean.
4. Documentation pass (the phase file's Documentation section): `README.md` + `AGENTS.md` reviewed in their entirety against the post-phase app; the settings/theme section notes the single-writer ownership (Settings panel owns the 3 brand strings, panel-row → tab-row → env order) and the live-apply behavior; `AGENTS.md` corrected only where it names the old order.
5. Commit: one atomic, professional commit (Conventional Commits, `--no-gpg-sign`), e.g.
`fix(settings): the Settings-panel app name wins over the Theme tab and saves apply live (phase 190)` — the body names the two defects (the `ui_settings` shadow bake, the absent live re-apply), the A1/A2 decisions, and the moved theme-tab suites. `.agents/phases/todo/190_app_name_settings_fix/` moves to `.agents/phases/complete/` in the SAME commit (AGENTS rule 8).
## Testing & Quality
- E2E: the suite above, isolated; the three regression suites green.
- Coverage: **>90%** on `app/`.
- Lint/types: clean.
## Completion Criteria
- [ ] `tests/e2e/test_app_name_settings_fix.py` green in isolation — (a) the incident regression (the panel value survives a Theme-tab color save), (b) the four-DOM live apply without reload, (c) the tab-reset fallback live, (d) the byte-identical boot, (e) the colors-only tab DOM.
- [ ] `uv run pytest` 100% passing; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `README.md` / `AGENTS.md` consistent (the single-writer + live-apply clauses).
- [ ] the phase directory is in `.agents/phases/complete/`; one atomic commit with `--no-gpg-sign`.
@@ -0,0 +1,72 @@
# Phase 191 — Sources carry a name, a label and tags — and the name focuses retrieval
**Source:** `TODO.md` L4 — "Sources need a name, a label, tag(s) so you can focus the model when prompting if you have a project name or a standard name you reference things by."
**Story:** n/a (owner directive)
**Depends on:** none functionally (builds on the completed phase-119 name-signal selection bonus, the phase-31/142 KB overview, the phase-76/35/38/49/90 Sources page).
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the data (task 01).** A new single-purpose table `source_labels` (alembic **0030**, additive, no backfill — a source without a row has no name/label/tags and behaves exactly as pre-phase):
- `source` — `String(120)` **PK**: the source name EXACTLY as `Document.source` carries it (the derived basename — the clone dir / local dir / upload file stem); the join key to the documents and to the `git_sources` row (via `_url_basename` / the local path basename);
- `display_name` — `Text`, nullable, ≤ 120 chars (the trimmed value; the owner's name for the source — "the git server");
- `label` — `Text`, nullable, ≤ 200 chars (the short one-line description/category — "self-hosted forge + Gitea");
- `tags` — `JSONB`, `NOT NULL`, server default `'[]'` (the focus tokens, each ≤ 32 chars, ≤ 10 tags — the owner's "tag(s)");
- `updated_at` — `DateTime(timezone=True)`, server default now, onupdate now.
- **The display name is an ALIAS ONLY** (owner-confirmed assumption): `Document.source`, the `source/path` document identity, the read/grep/`show_text` locators, the citations and the KB tree are ALL untouched by a label row — renaming a source never invalidates a saved chat, a shared page or a verbatim locator. The derived name stays the wire identity; the display name rides beside it.
- The ORM model `SourceLabel` (the `SourcesMeta` single-column style, `app/models.py`) + `app/rag/source_labels.py` (the new module — `app/rag/sources_meta.py` is the phase-53 version counter, NOT this feature; no name collision): `get_label(db, source)`, `set_label(db, source, display_name, label, tags)` (upsert, flush-only — the caller commits, the phase-53 session convention), `all_labels(db)`, `label_for(db, source)` → a frozen `SourceLabelView` (display_name / label / tags, `None` when absent), `question_name_hits(labels, question)` → the matching rule below.
- **D2 — the matching rule (task 04).** A source is a SOURCE-NAME HIT when at least one `name_hit_tokens(question)` candidate (the phase-119 class-agnostic token list, reused UNCHANGED — `app/rag/retriever.py` L191) equals, case-insensitively (the `_normalize_name` form), a token of the source's `display_name`, `label` or any `tags` entry. The phase-119 digit-class rule does NOT apply to user-given names (the owner's names are not paths — "gitea" must hit even though it is digitless; the phase-119 digit rule exists to keep prose tokens from matching PATHS, and user names have no prose-precision problem: a 4-char minimum token length (`NAME_TOKEN_MIN_LEN`, reused) is the only filter, applied to the SOURCE side too — a 2-char tag never matches). One query per turn: the label rows (a handful) are loaded once and the match is in Python (the phase-119 one-query convention).
- **D3 — the boost (task 04).** A NEW bounded selection-time bonus, the phase-119 `name_hit_bonus` pattern end to end, a SEPARATE knob (the two signals stay independently kill-able):
- `source_name_hit_bonus: float = 0.005` (default = the phase-119 starting point; validator `>= 0`, `0` = off; `BOR_SOURCE_NAME_HIT_BONUS` env; the `settings_catalog` metadata entry — the catalogue-completeness pin is extended, NOT waived);
- `RetrievedChunk` gains the additive field `source_name_hit: bool = False` (the `name_hit` docstring pattern: default `False` keeps every ordinary construction valid; `fuse`'s `replace()` copies it; the double-hit merge ORs it in — a vector row that is also the source-name hit keeps the flag);
- the flag is set at the retrieval boundary in `app/api/chat.py` (the call sites where `select_suggested` / `select_related` / `weak_hit_titles` run — L537–572 and L875 — the question is in scope): for every chunk whose `document.source` is a D2 hit, `rc.source_name_hit = True` (a loop over the few dozen retrieved chunks against the handful of hit sources — never a new query);
- `_selection_order` (L754) gains the second bonus parameter: `effective_score = best_fused + (bonus if any name_hit) + (source_bonus if any source_name_hit)` — each bonus per DOCUMENT, applied ONCE, at most two bonuses per document (bounded: 0.010 on the fused scale, the same scale phase 119 proved); `source_bonus == 0` (or no flagged chunk) → the walk is BYTE-IDENTICAL to the pre-phase walk (the phase-119 kill-switch contract, LOCKED A3 — a new unit pin mirrors the phase-119 one);
- the three application sites (`select_suggested` L537/L875, `select_related` L538, `weak_hit_titles` L572) pass `get_settings().source_name_hit_bonus`; the bonus lives in the SELECTION layer only — `score` / `cosine` / `fts_hit` / `query_log.top_score` are untouched (the phase-119 LOCKED A3 clause, mirrored).
- **D4 — the prompt surfacing (task 05).** The model learns the name→docs mapping from the prompt, not from retrieval alone:
- the KB overview's INPUT lines (`app/rag/overview.py` `build_overview_prompt`, L148 — `source — path — title — summary`) render `display_name — path — …` when set (the raw source name is REPLACED in the lite model's input only — the stored overview text is the lite model's prose, which then naturally uses the owner's name; the overview row's `source` field in the DB is untouched);
- the suggested-tier `<documents>` block (the per-doc line in `app/rag/prompts.py`, the phase-31 section builder) renders `display_name (label)` before the path when set — e.g. `the git server (self-hosted forge) — docs/gitea.md — …` (the parenthetical only when BOTH are set; one of the two → just the present one; neither → the pre-phase line, byte-identical).
- **D5 — the API (task 02).** `app/api/source_labels.py` (NEW router, the phase-91 admin gate — `require_admin` on every route; registered in `app/main.py` alongside the other admin routers):
- `GET /api/source-labels` → `{"labels": [ {source, display_name, label, tags} … ]}` — the FULL source catalogue: the union of (a) every `git_sources` row's derived source name (git + local, the `effective_sources` rows — an env-only deployment lists the env source names) and (b) every `SELECT DISTINCT source FROM documents` name NOT covered by (a) (the uploads) — one query for (b), the handful of `git_sources` rows for (a); each entry carries its label row or nulls;
- `PUT /api/source-labels/{source}` (path param ≤ 120) body `{display_name?, label?, tags?}` → the upsert; validation (the phase-121 boundary style — the framework's standard 422): `display_name` trimmed ≤ 120 (empty string / whitespace-only = NULL), `label` trimmed ≤ 200 (same), `tags` ≤ 10 entries, each a trimmed ≤ 32 non-empty token, de-duplicated case-insensitively preserving first case, `[]` legal (clears); an explicit `null` clears that field (an omitted field leaves it — the phase-168 partial-update shape); a `source` that matches NO git row and NO document → **404** (no orphan label rows — a label for a source that does not exist would be a dead boost input);
- `DELETE /api/source-labels/{source}` → row removed (404 when absent) — the reset.
- **D6 — the UI (task 03).** The Sources page (`#view-git-sources`, `git-sources.js`) — per-source name/label/tags editor for EVERY source kind (git, local, upload — the upload rows are the same table's extra section, phase 49/90):
- one expandable row per source (the house pattern — the phase-121 token field is the closest precedent): three fields (Name / Label / Tags, the last a comma-or-Enter token input, the phase-89 ignore-paths token input is the precedent) + Save / Reset, the §7.4 never-stale save state ("Saving…" → "saved" / the one-line error note, the phase-168 shape);
- the saved state renders back from the GET (the pre-population — the phase-91 convention); the derived source name is the row's identity (read-only, the `git-sources.js` name cell — the display name is a SEPARATE field, D1);
- the chat page's source cluster chips and the RAG page's source tree keep the DERIVED name (D1 — no identity change anywhere); the display name surfaces in the chat only through the prompt (D4) and the verbatim panel caption is untouched (it names `source/path` — the identity).
## ASSUMPTIONS (owner-confirmed 2026-10-18)
- **A1 (D1):** the display name is an alias — the `Document.source` / `source/path` identity is untouched.
- **A2 (D2/D3):** the focus mechanism is the new bounded `source_name_hit_bonus` selection boost (default 0.005, `0` = off) + the KB-overview / suggested-tier surfacing (D4).
- **A3 (D5):** admin-gated like the phase-91/168 admin surfaces; labels for non-existent sources are rejected (404).
## Context
- `app/models.py` (the `SourcesMeta` L— single-row precedent; the `GitSource` row — the `url`/`path`/`kind` columns the derived names come from), `app/rag/git_sources.py` (`_url_basename` L89, `effective_sources` L285 — the git/local source-name derivation), alembic `0029` (the last migration — 0030 is next), `app/rag/retriever.py` (`RetrievedChunk` L276+, `name_hit_tokens` L191, `_name_hit_chunks` L576, `_selection_order` L754, the three bonus application sites L838/L987/L1022), `app/api/chat.py` (the selection call sites L537–572, L875), `app/config.py` (`name_hit_bonus` L431 + validator L964 — the knob pattern), `app/core/settings_catalog.py` (L225 — the metadata entry pattern), `app/rag/overview.py` (`build_overview_prompt` L148 — the input lines), `app/rag/prompts.py` (the `<knowledge_base>` section L561, the suggested-tier doc lines), `app/api/git_sources.py` (the router pattern — admin gate, the upload status), `frontend/assets/git-sources.js` (the Sources page), `tests/fixtures/retrieval_battery.txt` + `scripts/eval_retrieval.py` (the battery), `tests/integration/` (the phase-119 pins — the kill-switch pattern to mirror).
## Tasks
1. `01_source_labels_table.md` — the `SourceLabel` model, alembic 0030, the `app/rag/source_labels.py` helpers (D1/D2 pure half).
2. `02_source_labels_api.md` — the admin-gated router (D5): the catalogue GET, the validated PUT upsert, the 404s, the DELETE.
3. `03_source_labels_ui.md` — the per-source name/label/tags editor on the Sources page (D6), all three source kinds.
4. `04_source_name_boost.md` — the `source_name_hit_bonus` knob + the `source_name_hit` flag + the `_selection_order` second bonus (D3), the kill-switch pins, the battery run.
5. `05_prompt_surfacing.md` — the KB-overview input lines + the suggested-tier doc lines carry the display name / label (D4).
6. `06_e2e_and_gates.md` — the dedicated Playwright suite (the owner-visible arm: a named source's document is suggested when the question uses the name, the alias never breaks a locator), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_source_labels.py` (NEW) — the D2 matching rule (name/label/tag hits, the case-insensitive `_normalize_name` form, the 4-char minimum on both sides, the no-hit baseline), the D1 helpers (upsert, the partial-update shape, the 404 shape at the helper level); `tests/unit/test_retriever.py` — the phase-119 mirror pins for the SECOND bonus (the kill-switch byte-identity, the bounded per-document application, the OR-merge through `fuse`).
- **Integration:** `tests/integration/test_source_labels_api.py` (NEW) — the admin gate (401/403 the phase-91 shape), the catalogue GET (git + local + upload union, no dupes), the PUT validation 422s (121-char name, 201-char label, 11 tags, a 33-char tag, a whitespace tag), the partial-update shape (an omitted field is untouched, an explicit null clears), the 404s (no such source; DELETE of an absent row), the round-trip.
- **E2E (AGENTS rule 4, task 06):** `tests/e2e/test_source_name_labels.py`, run in isolation — the owner-visible arm through the real UI (set a name + tag on a fixture source → a chat question using the name surfaces that source's document in the suggested tier via the mock/real battery — the phase-119 E2E is the precedent), the alias contract (a read/grep locator for the renamed source still resolves — the derived name is untouched), the 404 arm (a label for a non-existent source is rejected in the UI with the one-line note).
- **Battery:** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` — the pre/post run at the default bonus (0.005) and at `0` (byte-identical to pre-phase) is recorded in the phase file (the phase-119 convention).
- **Coverage:** **>90%** on `app/`; the frontend change is marker-pinned per the house JS convention.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the sources section gains the naming feature (the name/label/tags editor, the alias-only contract, the new knob `BOR_SOURCE_NAME_HIT_BONUS` with its `0` = off meaning) (task 06). `AGENTS.md`: the phase-119 name-signal clause gains the parallel source-name clause if it names the bonus (verify, do not add if absent).
## Completion Criteria
- [ ] alembic 0030 applied cleanly; `SourceLabel` round-trips (unit); the alias contract holds (a `Document.source` / locator is untouched by a label — the E2E arm).
- [ ] `GET /api/source-labels` lists git + local + upload sources with their labels (integration + E2E); the PUT validation 422s and 404s pinned; the DELETE resets.
- [ ] the Sources page edits name/label/tags for all three source kinds (E2E), save state per the §7.4 contract.
- [ ] `source_name_hit_bonus = 0` → the selection walk byte-identical to pre-phase (unit pin); at 0.005 the battery shows the named source surfacing (battery record); the three application sites pass the knob (unit pins).
- [ ] the KB-overview input lines and the suggested-tier doc lines carry `display_name (label)` when set, byte-identical otherwise (unit pins on both builders).
- [ ] test suite green, coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed phases (no label row → every surface byte-identical to pre-phase).
- [ ] one atomic Conventional Commits commit with `--no-gpg-sign`.
@@ -0,0 +1,41 @@
# Task 01 — The `source_labels` table and helpers (D1/D2 pure half)
**Phase:** `191_source_name_labels_tags` · **Source:** `TODO.md:5 — "Sources need a name, a label, tag(s) …"`
**Story:** n/a
## Objective
The additive data layer: the `SourceLabel` ORM model + alembic 0030 (no backfill) + `app/rag/source_labels.py` with the upsert/get helpers and the D2 `question_name_hits` matching rule — so a source without a row is byte-identical to pre-phase everywhere.
## Work
1. `app/models.py` — NEW `SourceLabel(Base)` (the `SourcesMeta` style, tabulated docstring — the D1 column spec from the phase file):
- `source: Mapped[str] = mapped_column(String(120), primary_key=True)` (the `Document.source` name — the join key);
- `display_name: Mapped[str | None] = mapped_column(Text, default=None)` (≤ 120 trimmed — enforced at the API, the column is plain Text, the phase-121 pattern);
- `label: Mapped[str | None] = mapped_column(Text, default=None)` (≤ 200);
- `tags: Mapped[list] = mapped_column(JSONB, default=list, server_default=text("'[]'"), nullable=False)`;
- `updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())`.
2. `alembic/versions/0030_source_labels.py` — NEW, revision `0030`, down_revision `0029` (verify the 0029 rev id — it is the current head): `CREATE TABLE source_labels` (the columns above, the PK, the JSONB server default), no backfill (an empty table — every source's pre-phase behavior is "no row"); the downgrade `DROP TABLE` (the house migration style — `0028` is the shape to copy).
3. `app/rag/source_labels.py` (NEW module — the docstring names the feature and the D1 alias contract):
- `@dataclass(frozen=True) SourceLabelView` — `source, display_name, label, tags` (tags: `tuple[str, …]`); `label_for(db, source) -> SourceLabelView | None` (one PK read; `None` when absent);
- `get_label(db, source) -> SourceLabel | None` (one PK read);
- `set_label(db, source, *, display_name, label, tags) -> SourceLabel` — upsert (the `SourcesMeta` upsert shape): an existing row is updated in place (only the provided non-`None` fields are overwritten — the partial-update shape; an explicit `None` clears), a new row is created; **flush-only — the caller commits** (the phase-53 session convention);
- `all_labels(db) -> dict[str, SourceLabelView]` (one query, keyed by `source`);
- `delete_label(db, source) -> bool` (one PK delete; `False` when absent; flush-only);
- `normalize_tags(raw: list[str]) -> list[str]` (pure: trim, drop empties, ≤ 32-char guard raises `ValueError`, ≤ 10 guard raises `ValueError`, case-insensitive de-dupe preserving first case — the API's 422 comes from these raises);
- `question_name_hits(view: SourceLabelView | None, question: str) -> bool` (the D2 rule, pure): `None` → `False`; else any `name_hit_tokens(question)` candidate (imported from `app.rag.retriever` — the phase-119 list, UNCHANGED) equals, in `_normalize_name` form, a token of `display_name` / `label` / any `tags` entry — the SOURCE side tokenized the same way (whitespace-split, `_normalize_name`, `NAME_TOKEN_MIN_LEN` (4) minimum on BOTH sides);
- `question_name_hit_sources(db, question: str) -> set[str]` (one `all_labels` read + the pure rule per row — a handful of rows; the per-turn entry point task 04 uses).
4. `tests/unit/test_source_labels.py` (NEW):
- the model round-trip (create via `set_label`, the `updated_at` stamp, the partial-update shape — an omitted field untouched, an explicit `None` clears);
- the D2 rule: a digitless name hit (`"gitea"` question token vs `display_name "Gitea"` — case-insensitive via `_normalize_name`), a label hit, a tag hit, a 3-char source-side token NEVER matches (the 4-char minimum), a 3-char QUESTION token never matches (the phase-119 `NAME_TOKEN_MIN_LEN` side), `None` view → `False`, a no-overlap question → `False`;
- `normalize_tags` (the 11-tag ValueError, the 33-char ValueError, the whitespace drop, the case-insensitive de-dupe keeping first case, `[]` legal);
- the migration 0030 up/down (the `tests/unit` or `tests/integration` migration-pin convention — verify where the phase-168 0028 pin lives and mirror it).
5. Verify: `uv run alembic upgrade head` clean on the dev DB; `uv run pytest tests/unit/test_source_labels.py -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the pins above (the D2 rule is the heart of the phase — pin every branch).
- Coverage: **>90%** on `app/` (the new module is fully exercised by its unit module).
## Completion Criteria
- [ ] alembic 0030 applies and reverses cleanly; the `SourceLabel` model matches the D1 column spec.
- [ ] `tests/unit/test_source_labels.py` green — the D2 rule, the partial-update shape, the `normalize_tags` guards.
- [ ] full test suite **100% passing** (the empty table changes nothing).
- [ ] no behavior change in completed work (no row → no surface reads it yet).
@@ -0,0 +1,35 @@
# Task 02 — The admin `source-labels` API (D5)
**Phase:** `191_source_name_labels_tags` · **Source:** `TODO.md:5 — "Sources need a name, a label, tag(s) …"`
**Story:** n/a
## Objective
The wire surface: `app/api/source_labels.py` — the catalogue `GET` (every source, all three kinds, with its label or nulls), the validated `PUT` upsert (the framework's standard 422; a 404 for a source that exists nowhere), the `DELETE` reset — admin-gated like the phase-91/168 admin surfaces, registered in `app/main.py`.
## Work
1. `app/schemas.py` — NEW shapes (the phase-91 `extra="forbid"` style):
- `SourceLabelOut` — `source: str` (≤ 120), `display_name: str | None` (≤ 120), `label: str | None` (≤ 200), `tags: list[str]` (≤ 10 items, each ≤ 32);
- `SourceLabelList` — `labels: list[SourceLabelOut]`;
- `SourceLabelUpdate` — `display_name: str | None = Field(default=None, max_length=120)`, `label: str | None = Field(default=None, max_length=200)`, `tags: list[str] | None = Field(default=None, max_length=10)` (each item ≤ 32 — an item-level `Field(max_length=32)` or a model validator; verify the house item-cap convention in `SavedChatCreate`'s messages and mirror it) — `extra="forbid"`; the OMITTED-vs-NULL distinction is the partial-update shape (D5): omitted → untouched, `null` → cleared, a value → set (the validator trims; a whitespace-only `display_name`/`label` → `None` at the validator boundary).
2. `app/api/source_labels.py` (NEW router, `APIRouter(prefix="/api/source-labels", tags=["source-labels"], dependencies=[Depends(require_admin)])` — the phase-91 gate line):
- `GET ""` → `SourceLabelList`: (a) the `git_sources` rows — `effective_sources(db)` (the phase-35/38 helper, L285): each row's derived source name (git: `_url_basename(row.url)`; local: the stored path's basename — the SAME expression the importer uses for `Document.source` — verify the importer's local-name expression and reuse it, do not re-derive it); an env-only deployment (no DB rows, the `env` source state) lists the env source names (the `effective_sources` `source` return is `"db"`/`"env"` — for `env`, parse the env list the phase-35 fallback uses); (b) `select(Document.source).distinct()` (ONE query, no `content` — the phase-126 load discipline) minus the names from (a) (the uploads); de-duplicated, sorted by `source` (stable order — the UI and the E2E rely on it); each entry = its label row via `all_labels(db)` (one query) or nulls;
- `PUT "/{source}"` (path param, the router's `max_length` via the path conversion — a >120 path → the framework's 422) → the D5 404 arm FIRST: `source` matches no `git_sources` row name AND is not in the distinct document sources (the SAME two queries the GET runs — no third) → 404 (the standard body, the phase-91 shape); else `normalize_tags` (422 on its raises — a `ValueError` → the framework's 422 via the validator, NOT a 500), `set_label` (flush + `db.commit()` — the router owns the session here, the phase-91 commit convention), 200 `SourceLabelOut`;
- `DELETE "/{source}"` → the 404 arm (no label row → 404), else `delete_label` + commit, 200 (the phase-91 DELETE shape — verify its body: an empty object or a message; mirror the sibling admin router).
3. `app/main.py` — `app.include_router(source_labels.router)` (the admin routers' block — verify the exact registration list and the order convention).
4. `tests/integration/test_source_labels_api.py` (NEW — the phase-91 `test_ui_settings_api.py` fixture pattern):
- the gate: an anonymous GET/PUT/DELETE → the phase-91 gate shape (403 / the redirect-to-login — verify the phase-91 gate pin and mirror it EXACTLY);
- the catalogue GET: a module app with one git row (a `file://` URL — the phase-35 fixture), one local row, one upload (a document row with a source name no row derives) + a label on the upload → the response lists all three (sorted, no dupes), the upload carries its label, the others nulls;
- the PUT: a valid upsert round-trips (set name+label+tags → GET shows them; a second PUT with only `label` → the name/tags untouched; a PUT with explicit `null` `display_name` → cleared, the label kept); the 422s (a 121-char `display_name`, a 201-char `label`, 11 `tags`, a 33-char tag, a whitespace-only tag — each with the framework's standard body and NO row change: a pre-seeded row is byte-identical after the 422); the 404s (a `source` that is neither a row name nor a document source — the row table unchanged);
- the DELETE: a labelled source → 200, the GET shows nulls; a second DELETE → 404.
5. Verify: `uv run pytest tests/integration/test_source_labels_api.py -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Integration: the pins above — the gate, the union catalogue, the validation 422s (no partial write), the 404s, the round-trip.
- Coverage: **>90%** on `app/` (the router is fully exercised by its integration module).
## Completion Criteria
- [ ] `GET /api/source-labels` lists git + local + upload sources (integration pin) with their labels; admin-gated.
- [ ] `PUT /api/source-labels/{source}` upserts with the D5 partial-update shape; every 422 leaves the row unchanged; a 404 for a non-existent source creates nothing.
- [ ] `DELETE` resets; a second DELETE 404s.
- [ ] full test suite **100% passing**.
- [ ] no behavior change in completed work (the `git_sources` API, the `documents` API, the phase-91/168 admin surfaces all green unedited).
@@ -0,0 +1,35 @@
# Task 03 — The per-source name/label/tags editor on the Sources page (D6)
**Phase:** `191_source_name_labels_tags` · **Source:** `TODO.md:5 — "Sources need a name, a label, tag(s) …"`
**Story:** n/a
## Objective
The owner-visible surface: every source on the Sources page (git, local, upload — the catalogue `GET /api/source-labels` is the single source of truth for the row list, so uploads appear without a second API) gets an inline name/label/tags editor with the §7.4 never-stale save state — the phase-121 token field and the phase-89 ignore-paths token input are the house precedents.
## Work
1. `frontend/index.html` — `#view-git-sources` (L646+): the sources table area (the `#git-sources-table-wrap` region, L792) gains the per-row expandable label block markup — ONE shared template (a `<template id="source-label-row-template">`) the JS clones per row:
- three fields in one row (the house grid): **Name** (`input`, ≤ 120, `maxlength="120"`), **Label** (`input`, ≤ 200, `maxlength="200"`), **Tags** (the token input — the phase-89 ignore-paths input's exact widget: type + Enter/comma adds a chip, Backspace deletes the last, ≤ 10 chips, each ≤ 32 chars — reuse its CSS classes, no new styles beyond the phase-89 ones);
- **Save** / **Reset** buttons (the phase-168 button shapes) + the save-state line (`role="status"` — "Saving…" → "saved" / the one-line error note; Reset visible only when the row has a stored label, the phase-168 per-row reset shape);
- the row's IDENTITY cell (the derived source name) is UNCHANGED and read-only (D1/A1 — the editor sits beside it, never replaces it).
2. `frontend/assets/git-sources.js`:
- the load path: after the existing catalogue load, `GET /api/source-labels` (one extra fetch — the gate is the SAME as the page's: the anonymous gate hides the whole page, the admin sees it; the fetch runs inside the existing admin-gated content reveal); merge the labels into the row data (by `source` name — the GET is the union, so the row list for the label editor comes from the GET, not from the `git-sources` GET — the upload rows have no `git-sources` row);
- the editor state per row: the three fields pre-populated from the label (nulls → empty); dirty tracking (a field change enables Save — the phase-168 `.is-dirty` + "unsaved changes" hint shape, scoped per row);
- **Save**: `PUT /api/source-labels/<source>` (the trimmed values; the tags array from the chips) → success: the "saved" state, the row's label re-read from the response (no second GET — the response is the new truth); failure: the one-line error note (the framework's detail — the 422/404 arm: a 404 shows "that source no longer exists" — the note text names the 404 arm, the phase-168 error-note shape); the §7.4 guard: one in-flight save per row (a second click while saving is ignored, the phase-168 `saveInFlight` pattern scoped per row);
- **Reset**: `DELETE /api/source-labels/<source>` → the "saved" state, the fields cleared, the row's stored label nulled;
- the page's existing sync / upload / git-row flows are UNTOUCHED (the editor is additive — the row's existing action buttons keep their behavior).
3. `tests/unit/` — the JS marker pins (the module that marker-pins `git-sources.js` — verify the name; the `test_save_chat_ui.py` convention):
- the editor's save path references `PUT /api/source-labels/` and the response (not a second GET) as the new truth (source marker);
- the per-row `saveInFlight` guard marker; the 404 note text marker;
- the token input reuses the phase-89 widget classes (a marker naming the class reuse — no new tag-input CSS).
4. Verify: `uv run pytest tests/unit/<js-pin-module> -v` green; `uv run playwright`-free local check: the page renders the editor for a seeded label row (the E2E covers this in task 06 — a manual `uv run uvicorn` smoke is optional, the E2E is the gate).
## Testing & Quality
- Unit: the marker pins above.
- E2E: task 06 (the owner-visible arm).
- Coverage: **>90%** on `app/` (unchanged — the task is frontend; the app-side gate stays green).
## Completion Criteria
- [ ] every source row (git, local, upload) renders the name/label/tags editor; the derived name cell is read-only and untouched (D6/A1).
- [ ] Save/Reset hit the phase-02 endpoints; the save state follows the §7.4 contract (never a stuck "Saving…", the 404 note names the arm) — marker pins.
- [ ] the existing sources-page flows (add git/local source, upload, sync) are green unedited.
- [ ] no behavior change in completed work.
@@ -0,0 +1,42 @@
# Task 04 — The `source_name_hit_bonus` selection boost (D2/D3)
**Phase:** `191_source_name_labels_tags` · **Source:** `TODO.md:5 — "… so you can focus the model when prompting …"`
**Story:** n/a
## Objective
The focus mechanism: a NEW bounded, kill-able selection-time bonus that lifts a document when the question names its source (D2 rule) — the phase-119 `name_hit_bonus` pattern end to end, a separate knob, the SAME shared walk, `0` = byte-identical pre-phase.
## Work
1. `app/config.py` — the knob (the `name_hit_bonus` L431 pattern, directly below it):
- `source_name_hit_bonus: float = 0.005` (the docstring: the D3 spec — default = the phase-119 starting point; `0` = off, byte-identical walk; bounded per-document, at most two bonuses on the fused scale);
- the `field_validator` (the L964 pattern): `>= 0`, the message `"source_name_hit_bonus must be >= 0 (0 = off)"`.
2. `app/core/settings_catalog.py` — the `SettingMeta` entry (the L225 `name_hit_bonus` pattern): `"source_name_hit_bonus"` in the **"Retrieval & honesty"** section, label "Source name hit bonus", the restart-required flag FALSE (the phase-168 runtime knob shape — verify the `name_hit_bonus` entry's flags and mirror them). Extend the catalogue-completeness pin (the unit test asserting every `Settings` field has a `SettingMeta` entry — find the module; it starts failing when the field lands, the entry is the fix).
3. `app/rag/retriever.py`:
- `RetrievedChunk` (L276+): the additive field `source_name_hit: bool = False` + the docstring line (the `name_hit` line pattern — default `False` keeps every ordinary construction valid; `fuse`'s `replace()` copies it automatically — the frozen dataclass; the double-hit merge at L342–358 gains `source_name_hit=existing.source_name_hit or rc.source_name_hit` beside the existing `name_hit` OR);
- `_selection_order` (L754): signature `_selection_order(chunks, bonus, source_bonus: float = 0.0)` — the walk gains `any_source_hit[doc.id]` (the `any_name_hit` pattern, L792/L799/L804); the effective-score dict: `best_score[doc.id] + (bonus if any_name_hit[doc.id] else 0.0) + (source_bonus if any_source_hit[doc.id] else 0.0)`; the re-sort condition: `(bonus > 0.0 and any_name_hit…) or (source_bonus > 0.0 and any_source_hit…)` (the pre-phase condition extended — when neither bonus is active, the walk is byte-identical: no re-sort, the same order list, the same `+ 0.0`); the docstring's D2/LOCKED-A3 paragraph is extended for the second bonus (each bonus per DOCUMENT, applied ONCE, at most two per document — bounded 0.010; the bonus lives in the SELECTION layer only — `score` / `cosine` / `fts_hit` / `query_log.top_score` untouched);
- `weak_hit_titles` (L825), `select_suggested` (L952), `select_related` (L1000): a `source_bonus: float | None = None` parameter each; `if source_bonus is None: source_bonus = get_settings().source_name_hit_bonus`; passed to `_selection_order`.
4. `app/api/chat.py` — the flag-set at the retrieval boundary (the call sites L537–572 and L875 — the question is in scope, the session is in scope):
- ONE call per turn: `hits = question_name_hit_sources(db, question)` (task 01's per-turn entry point — the single `all_labels` read) BEFORE the first selection call;
- the flag loop: `for rc in chunks: if rc.document.source in hits: rc.source_name_hit = True` (the few dozen retrieved chunks against the handful of hit sources — never a per-chunk query); the selection calls keep their existing arguments (the functions pull the knob from settings themselves — the call sites gain NO new positional arguments);
- the per-turn log line (PLAN §9) is UNCHANGED (the `verbatim_blocks=` / `scaffold_stripped=` fields are untouched — the boost is invisible to the log; no new log field).
5. `tests/unit/test_retriever.py` (or the phase-119 pin module — verify the name):
- the KILL-SWITCH pin (the phase-119 mirror): `source_name_hit_bonus = 0` (and no flagged chunks) → the selection order is byte-identical to the pre-phase walk (the same fixture, the same expectation list);
- the boost pin: a document whose source is a D2 hit (a `source_name_hit=True` chunk) ranks ABOVE an equal-fused-score non-hit when the bonus is on (the 0.005 head start), and a document carrying BOTH flags gets exactly +0.010 (the bounded two-bonus pin — no amplification);
- the `fuse` merge pin: a vector row merged with a flagged source-name row keeps `source_name_hit=True`;
- the settings pins: the validator (`-0.1` → the ValueError), the env name (`BOR_SOURCE_NAME_HIT_BONUS`), the catalogue-completeness pin extended.
6. The battery (the phase-119 convention): `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` — run it TWICE and record both in the phase file: (a) pre-change (the baseline), (b) post-change at `source_name_hit_bonus = 0` (must match (a) line-for-line — the kill switch) and at the default 0.005 (no label rows exist in the battery app → also identical to (a) — the no-row baseline).
7. Verify: `uv run pytest tests/unit/test_retriever.py tests/unit/test_settings*.py -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the kill-switch byte-identity, the bounded two-bonus application, the `fuse` OR-merge, the settings/catalogue pins.
- Integration: the E2E task's app fixture exercises the flag-set (no separate integration module — the flag loop is covered by the E2E arm + the unit pins).
- Battery: the three-run record above.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `source_name_hit_bonus = 0` → the selection walk byte-identical to pre-phase (the unit pin + the battery match).
- [ ] at the default 0.005, a D2-hit document gets exactly the 0.005 head start (0.010 with both flags) — the bounded pin; no chunk field is modified (the untouched-fields pin).
- [ ] the three selection helpers pull the knob from settings by default (the unit pins with a settings override).
- [ ] `tests/fixtures/retrieval_battery.txt` battery: the kill-switch run matches the baseline line-for-line (recorded in the phase file).
- [ ] full test suite **100% passing**; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed work (the phase-119 bonus, the A8 gate inputs, the per-turn log line all untouched).
@@ -0,0 +1,37 @@
# Task 05 — The prompt learns the name: the `<document>` block + the KB overview (D4)
**Phase:** `191_source_name_labels_tags` · **Source:** `TODO.md:5 — "… so you can focus the model when prompting …"`
**Story:** n/a
## Objective
The model learns the name→docs mapping from the prompt: a suggested `<document>` block carries the source's `name` / `label` as additive attributes (when set), and the KB overview's INPUT lines carry the display name — both byte-identical for a source without a label row. ONE `all_labels` read per turn feeds the task-04 boost flags AND these surfaces.
## Work
1. `app/rag/prompts.py` — `build_high_prompt` (L828):
- a new optional parameter `names: Mapping[str, "SourceLabelView"] | None = None` (the source-name → view map; `None` = pre-phase, byte-identical — the `show_image`/`show_text` optional-parameter pattern);
- the block builder (L934–939): when `names.get(doc.source)` has a `display_name`, the block head gains ` name="{display_name}"` immediately after the `source="…"` attribute; when it has a `label`, it gains ` label="{label}"` after the `name` (the attribute order `source` → `name?` → `label?` → `path` → `title` → `date`; an unset field adds NOTHING — a no-label document's block is byte-identical to pre-phase, and a label-only document gets exactly the one `label` attribute);
- the values are XML-escaped with the SAME helper the block already uses for `source`/`path`/`title` (verify the escaping at L934 — a `"` in a display name must not break the attribute; the phase-106 date formatting is untouched).
2. `app/rag/overview.py` — `build_overview_prompt` (L148):
- a new optional parameter `names: Mapping[str, str] | None = None` (source → display_name); the line builder (L173): `line = f"{names.get(source, source)} — {path} — {title}"` (the display name REPLACES the raw source name in the lite model's input only — the stored overview row and the DB are untouched); no `names` / no entry → the pre-phase line, byte-identical.
- `regenerate_overview` (the caller — verify the signature): it already holds a session; it gains the ONE `all_labels(db)` read (task 01's helper) and passes (a) the display-name map to `build_overview_prompt` and (b) the same read is NOT re-queried — the map is derived from the views in the same loop.
3. `app/api/chat.py` — the task-04 call site is REFACTORED to the shared read (the "one query per turn" contract):
- `labels = all_labels(db)` (the single read, before the first selection call — replacing task 04's `question_name_hit_sources` call, whose internals it inlines);
- `hits = {v.source for v in labels.values() if question_name_hits(v, question)}` (the pure D2 rule per view — the flag loop of task 04 is unchanged, `hits` is now local);
- the `build_high_prompt` call (the grounded plan — verify the exact call site in the module) passes `names=labels` (the `SourceLabelView` map — `build_high_prompt` accepts the views directly; the overview's `str` map is the overview's own derivation, a different surface, no conversion shared).
4. `tests/unit/test_prompts.py` (or the prompt-lock module — verify the name):
- the KILL pin: `names=None` and `names={}` and a map with no entry for the doc's source → the built prompt is byte-identical to the pre-phase build (the existing pre-task pins stay green UNEDITED — they are the kill pin);
- the attribute pins: a `display_name`-only view → exactly ` name="…"` after `source="…"` (the attribute-order pin); a `label`-only view → exactly the `label` attribute (no `name`); both → both attributes in order; the XML-escaping pin (a `"` in a display name renders as the escaped entity in the attribute);
- the overview pins: `names=None` → the pre-phase lines byte-identical; a map entry → the line uses the display name (the `path` / `title` / summary halves untouched); a missing entry → the raw source name.
5. Verify: `uv run pytest tests/unit/test_prompts.py tests/unit/test_overview.py -v` green (verify the overview test module name); `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the byte-identical kill pins (both builders), the attribute-order + escaping pins, the overview line-swap pins.
- Integration: the E2E task's arm covers the wire (the system prompt of a turn with a named source — the mock LLM's captured prompt, the phase-119 E2E precedent).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] a named source's suggested document renders `name="…"` / `label="…"` (the unit pins); an unnamed source's block and overview line are byte-identical to pre-phase (the unedited pre-task pins + the kill pins).
- [ ] ONE `all_labels` read per turn feeds the boost flags and the prompt surfaces (the marker pin on the chat.py call site — no second label query in the turn).
- [ ] `regenerate_overview` uses the same single read (the marker pin — no second query).
- [ ] full test suite **100% passing**; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed work (the `<document>` block for label-less sources, the KB overview text for label-less sources, the phase-106 date attribute all byte-identical).
@@ -0,0 +1,37 @@
# Task 06 — The phase E2E suite, the full gates, the docs pass, the commit
**Phase:** `191_source_name_labels_tags` · **Source:** `TODO.md:5`
**Story:** n/a
## Objective
The dedicated Playwright suite proves the owner-visible arm through the real UI — a named source's document surfaces when the question uses the name, the alias never breaks a locator, the 404 arm is visible — run in isolation per AGENTS rule 4; the full gates pass; the docs are consistent; one atomic commit.
## Work
1. `tests/e2e/test_source_name_labels.py` (NEW — the module docstring names the TODO item and the D1 alias contract) — the module-app fixture pattern (the phase-119/168 E2E convention — verify in `tests/e2e/conftest.py` / the sibling suites; the mock LLM from `tests/e2e/mock_llm.py` for the prompt-capture arm):
- the fixture: the standard small KB (the phase-119 fixture convention — a handful of documents in a couple of sources) + one uploaded document (the phase-49/90 upload flow or a direct document-row seed — verify the sibling suites' seeding helper).
- **(a) the editor, all three kinds:** as the admin, open the Sources page; set a Name ("the git server"), a Label ("self-hosted forge") and a tag ("gitea") on a git source row, a Name on a local source row, a Name + tag on the upload row → each save shows "saved" (the §7.4 state) and a reload pre-populates the saved values (the phase-91 pre-population convention); the derived-name cells are unchanged (the A1 pin — the row identity cell text is the pre-save text).
- **(b) the focus, owner-visible:** a chat question that uses the NAME ("what does the git server run?") with the mock LLM → the suggested tier (the `done` frame's `sources` / the UI's source cluster) includes the named source's document (the D3 boost's owner-visible arm — the phase-119 E2E is the precedent; the mock's captured system prompt asserts the `<document name="the git server" label="self-hosted forge" …>` attributes — the D4 arm at the wire); the same question phrased WITHOUT the name (the pre-baseline question from the battery) shows the pre-phase suggested set (the no-signal control — the boost is the only difference).
- **(c) the alias contract:** after the rename, a verbatim-fenced answer (or a direct `page.request` read of the document API) for `source/path` (the DERIVED name) still resolves — the locator is untouched (the A1 pin through the wire); the KB tree / RAG page still lists the derived name (the D1 surface pin).
- **(d) the 404 arm:** the editor's PUT for a source that no longer exists (delete the underlying git row via the API mid-test, then save a label through the UI) → the one-line error note names the 404 arm (the task-03 pin, owner-visible).
- **(e) the reset:** the editor's Reset on the named source → "saved", the fields cleared, a reload shows empty, and the chat's suggested set for the named question reverts to the control (the label's absence is the pre-phase state end to end).
2. Regression, run in isolation: `tests/e2e/test_name_signal.py` (the phase-119 suite — verify the exact name in `tests/e2e/`), `tests/e2e/test_settings_panel.py`, `tests/e2e/test_git_sources.py` (the phase-76 suite — verify the name) — green unedited.
3. The full gates, in order:
- `uv run pytest` (unit + integration) — 100% passing;
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on `app/`;
- `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` — green in isolation;
- `uv run ruff check . && uv run pyright` — clean.
4. Documentation pass (the phase file's Documentation section): `README.md` + `AGENTS.md` reviewed in their entirety against the post-phase app; the sources section gains the naming feature (the editor, the alias-only contract, `BOR_SOURCE_NAME_HIT_BONUS` with its `0` = off meaning); `AGENTS.md` corrected only where it names the phase-119 bonus as the ONLY name signal.
5. Commit: one atomic, professional commit (Conventional Commits, `--no-gpg-sign`), e.g.
`feat(sources): per-source name, label and tags focus retrieval and the prompt (phase 191)` — the body names the D1 alias contract, the D2 rule, the D3 knob, the D4 surfaces, the new table/migration 0030, and the battery record location. `.agents/phases/todo/191_source_name_labels_tags/` moves to `.agents/phases/complete/` in the SAME commit (AGENTS rule 8).
## Testing & Quality
- E2E: the suite above, isolated; the regression suites green.
- Battery record: task 04's three-run record is in the phase file (the kill-switch run matches the baseline).
- Coverage: **>90%** on `app/`.
- Lint/types: clean.
## Completion Criteria
- [ ] `tests/e2e/test_source_name_labels.py` green in isolation — (a) the editor for git/local/upload with the saved-state contract, (b) the named question surfaces the named source's document + the prompt attributes, (c) the derived-name locator is untouched by a rename, (d) the 404 note, (e) the reset reverts end to end.
- [ ] `uv run pytest` 100% passing; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `README.md` / `AGENTS.md` consistent (the naming feature + the knob).
- [ ] the phase directory is in `.agents/phases/complete/`; one atomic commit with `--no-gpg-sign`.
@@ -0,0 +1,58 @@
# Phase 192 — Verbatim panels land while the answer is still streaming
**Source:** `TODO.md` L5 — "The verbatim tag appear at the end and only when the llm is done answering which makes the chat read strangely."
**Story:** n/a (owner report)
**Depends on:** none functionally (builds on the completed phase-182 verbatim display, phase-95 tool_result frames, phase-158/164 saved-chat restore).
## Evidence (diagnosed from the code, read this not the chat)
1. **Panels materialize only on the `done` frame.** `app/api/chat.py` resolves the turn's verbatim blocks at done-time (L1319–1347: `resolve_answer_blocks` over `answer_text` → the done frame's additive `answer_text` list); `app.js` hydrates at done (L4506–4511: `hydrateVerbatim(bubble, ev.answer_text, …)` + `appendUnlocatedVerbatim(wrap, ev.answer_text, …)`) and at restore (L3044–3046, the record's `m.answer_text`).
2. **The `show_text` result already exists mid-turn — it just isn't sent.** The agent's show_text branch (app/rag/agent.py L2238–2340) appends the resolved `VerbatimBlock` to `holder.shown_texts` (call order, the `show_text_max_blocks` budget) the instant the call succeeds, and the result text rides the round's `tool` message — but the block itself reaches the client ONLY in the done frame.
3. **So the reading order inverts.** The prose streams first; every verbatim panel (fenced or not) pops in at the END, after the model has already finished talking about the bytes: "…here is the doc: [model finishes] [panel appears]". The unfenced blocks are the worst case — `appendUnlocatedVerbatim` appends them at the answer's tail (the `answer_image` position), detached from the sentence that introduced them.
4. **The fix is additive.** The block is already in hand at the success point (the `shown_texts` append, L2322 area); the round loop already has the snapshot-then-yield pattern for phase-95 truncations (`trunc_before`, L2640 → the `ToolResultPiece` yield at L2690). One new piece + one new SSE event + one client branch; the `done` frame, the record key, the restore and the shared page keep their exact contract.
## Design (shared by all tasks — the executor reads this, not the chat)
- **A1 — the mid-turn frame (task 01).** A new additive SSE event `verbatim_block`, emitted as soon as a `show_text` call succeeds:
- `app/rag/agent.py`: a new `@dataclass VerbatimBlockPiece` (the `ToolResultPiece` pattern, beside it; the `block: VerbatimBlock` field — the class is already imported at L343). In the round loop: `shown_before = len(holder.shown_texts)` BEFORE `_execute_tool` (the `trunc_before` pattern, L2640); after the round's `ToolResultPiece` yield (L2690), `for block in holder.shown_texts[shown_before:]: yield VerbatimBlockPiece(block=block)` — the frame lands AFTER the round's `tool_result` frame and BEFORE the next model round. SUCCESS-ONLY by construction: every refusal path returns before the `shown_texts.append` (L2322), and the already-shown dedupe is a refusal — a refused call yields no piece. The generator union types (L928/L992) gain the piece.
- `app/schemas.py`: `ChatVerbatimBlockEvent` (the `ChatToolResultEvent` pattern, L321) — `type: Literal["verbatim_block"]` + the EXACT `VerbatimBlockRef` field set (L185: `source` ≤ 120 / `path` ≤ 1000 / `title` ≤ 500 / `start_line` / `end_line` / `total_lines` / `text` ≤ 600 000 / `truncated` / `locator` ≤ 2100) — the client's `isVerbatimBlock` check (verbatim.js L70) accepts the event as-is; the framework's `extra` convention of the sibling events is mirrored.
- `app/api/chat.py`: the new dispatch branch after the `ToolResultPiece` branch (L1104): `if isinstance(piece, VerbatimBlockPiece): yield sse_event(ChatVerbatimBlockEvent(**asdict-style fields).model_dump()); continue`. The done-frame `answer_text` resolution (L1319–1347) and the per-turn log line (`verbatim_blocks=N`, done-frame count) are UNTOUCHED — the mid-turn frames are not counted there (the pin stays green).
- KILL-SWITCH: `BOR_ANSWER_TEXT=0` → the tool is not offered (the phase-182 D8 posture) → no success → NO piece, NO frame: the frame sequence is byte-identical to pre-phase (a wire pin, task 01 + the E2E arm).
- **A2 — the client lands the panel mid-stream (task 02).**
- `frontend/assets/verbatim.js`: a NEW `hydrateVerbatimNow(root, block, opts)` — walks the PENDING placeholders (`.verbatim-pending[data-bor-locator]`); for each, `matchVerbatimBlock([block], key)` (the SAME matching semantics as the done path — the exact locator, or the identity fallback when the list holds exactly one block of that identity); REPLACES ONLY the matched nodes with `buildVerbatimPanel(block, opts)`; the UNMATCHED pending nodes STAY PENDING (settling them is the done/failure path's job — a mid-turn pass must never burn a placeholder for a not-yet-shown fence); returns the number matched (0 → the caller appends). Never throws (the house contract — a missing `root` → 0).
- `frontend/assets/app.js`: the new `ev.type === "verbatim_block"` branch beside the `tool_result` branch (L4612): `const n = hydrateVerbatimNow(bubble, ev, VERBATIM_LINK_OPTS); if (n === 0) appendUnlocatedVerbatim(wrap, [ev], VERBATIM_LINK_OPTS);` — a fenced block lands IN PLACE at its fence (the model said "here it is" → the panel is there); an unfenced block lands at the `answer_image` position (the `appendUnlocatedVerbatim` contract — below the current text, above the sources) AT THE MOMENT the call succeeds, not at the end of the answer. `appendUnlocatedVerbatim`'s state-free idempotence (same canonical locator already placed → skip, L463–472) makes the done pass's re-run a no-op for the same block. No client-side kill-switch gate: the server's gate is the only one (a `False` turn sends no frame — the branch never fires).
- **A3 — the done/restore dedupe (task 03).** The one remaining duplicate: a mid-turn APPENDED panel (locator A) + the model later fences the SAME locator A → the fence placeholder hydrates at done into a second panel. `hydrateVerbatim` (the done AND restore path — L423) gains the placed-scan: before building a matched panel for a `key`, if a PLACED resolved panel (``.verbatim[data-bor-locator]`, NOT `.verbatim-unresolved` — the `appendUnlocatedVerbatim` placed-scan convention) carrying the same canonical `locator` exists in the message wrap (`root.closest(".msg")`), that placed panel is REMOVED first — the in-text panel wins (the fence is where the model wanted it; the appended panel has served its mid-turn purpose). TWO DIFFERENT ranges of the same file are LEGITIMATE (the `shown_texts` dedupe is per canonical locator, not per identity) and stay — the scan is per-locator, never per-identity. The restore path (L3044–3046) and the shared page ride the SAME functions — no separate change; the parity pin is the test.
- **A4 — the unchanged contract (pinned across tasks).** The `done` frame's `answer_text` list, the saved-record key, the restore path and the shared page are byte-identical to phase 182 for a turn that never calls `show_text`; a `BOR_ANSWER_TEXT=0` turn is byte-identical end to end (no frame, no placeholder, no panel); the `verbatim_blocks=N` per-turn log line counts the done-frame blocks as before.
## Context
- `app/rag/agent.py` — `VerbatimBlock` (imported L343), `holder.shown_texts` (L1848), the show_text branch (L2238–2340), the round loop (L2640–2697), the piece union types (L928/L992); `app/api/chat.py` — the piece dispatch (L1096–1130), the done-time resolution (L1319–1347); `app/schemas.py` — `VerbatimBlockRef` (L185), `ChatToolResultEvent` (L321), the `extra` convention; `app/rag/verbatim.py` — `VerbatimBlock` (L94), `block_for`, `canonical_locator` (D7).
- `frontend/assets/verbatim.js` — `isVerbatimBlock` (L70), `matchVerbatimBlock` (L404), `hydrateVerbatim` (L423), `appendUnlocatedVerbatim` (L447), `settleVerbatimPlaceholders` (L480); `frontend/assets/app.js` — the `tool_result` branch (L4612), the done branch's hydrate/append (L4506–4511), the restore hydrate/append (L3044–3046).
- `tests/e2e/mock_llm.py` (the mock's show_text flow — the phase-182 E2E precedent; verify the exact mock arm), the phase-182 E2E suites (verify the names in `tests/e2e/`), the JS marker-pin unit module (verify the name — the `test_save_chat_ui.py` convention), `tests/integration/` the chat SSE contract module (verify the name).
## Tasks
1. `01_verbatim_block_sse.md` — the `VerbatimBlockPiece`, the `ChatVerbatimBlockEvent`, the chat dispatch branch; the agent mock pins, the SSE contract pins, the kill-switch wire pin.
2. `02_midturn_hydration_and_append.md` — `hydrateVerbatimNow` + the app.js `verbatim_block` branch (in-place fence hydration at mid-turn; unfenced blocks land at the call moment; unmatched placeholders stay pending).
3. `03_done_and_restore_dedupe.md` — the `hydrateVerbatim` placed-scan dedupe (per-locator; two ranges stay; the restore/shared parity pin; the kill-switch and the `verbatim_blocks=N` pins stay green).
4. `04_e2e_and_gates.md` — the dedicated Playwright suite (the mid-turn arm, the fence arm, the no-double arm, the kill-switch wire arm, the restore arm), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the agent mock pins (a successful `show_text` → exactly one `VerbatimBlockPiece` after the round's `tool_result` and before the next round; the five refusal paths → zero; the `answer_text=False` posture → zero); the `VerbatimBlockPiece`/`ChatVerbatimBlockEvent` shape pins; the JS marker pins (the `hydrateVerbatimNow` matched-only contract, the app.js branch).
- **Integration:** the chat SSE contract — a mock turn with `show_text` → the stream carries the `verbatim_block` frame (the exact `VerbatimBlockRef` fields) before `done`; a `BOR_ANSWER_TEXT=0` app → the captured frame sequence is byte-identical to the pre-phase sequence (the kill-switch wire pin); the done frame's `answer_text` is unchanged (the phase-182 pins stay green).
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_verbatim_inline_midturn.py`, run in isolation — the owner-visible arm: the panel is on screen WHILE the answer is still streaming (before `done`), the fence hydrates in place mid-turn, no double panel, the kill-switch wire arm, the restore parity.
- **Regression:** the phase-182 E2E suites green unedited; `tests/e2e/test_chat_history.py` (the restore contract) green.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); the frontend change is marker-pinned per the house JS convention.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the verbatim section notes the mid-turn landing (a `show_text` panel appears the moment the call succeeds — in place when fenced, at the answer position when not; the done/restore contract is unchanged) (task 04). `AGENTS.md`: the phase-182 clause gains the mid-turn frame clause if it names the done-frame resolution (verify, do not add if absent).
## Completion Criteria
- [ ] a successful `show_text` call yields exactly one `verbatim_block` frame (the unit + integration pins); every refusal yields none.
- [ ] a `BOR_ANSWER_TEXT=0` turn's frame sequence is byte-identical to pre-phase (the wire pin) — no frame, no placeholder, no panel.
- [ ] a pending fence placeholder for a shown block hydrates IN PLACE at mid-turn (before `done`); an unfenced block is on screen at the call moment (the E2E arms).
- [ ] a mid-turn appended panel + a same-locator fence → exactly ONE panel after `done` (the dedupe pin); two different ranges of one file → two panels (the pin).
- [ ] the restore path shows exactly one panel per locator after a reload (the E2E arm); the `done` frame's `answer_text`, the record key and the `verbatim_blocks=N` log line are unchanged (the phase-182 pins green unedited).
- [ ] test suite green, coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed phases (a turn without `show_text` is byte-identical end to end).
- [ ] one atomic Conventional Commits commit with `--no-gpg-sign`.
@@ -0,0 +1,60 @@
# Task 01 — The additive `verbatim_block` SSE frame (A1)
**Phase:** `192_verbatim_inline_midturn` · **Source:** `TODO.md:6 — "The verbatim tag appear at the end and only when the llm is done answering …"`
**Story:** n/a
## Objective
The server half: a `show_text` success yields a new `VerbatimBlockPiece` → a new additive `verbatim_block` SSE frame lands AFTER the round's `tool_result` frame and BEFORE the next model round — the block the agent already holds, sent at the moment it exists. Success-only by construction; `BOR_ANSWER_TEXT=0` → zero frames (byte-identical sequence).
## Work
1. `app/rag/agent.py`:
- a new `@dataclass VerbatimBlockPiece` beside `ToolResultPiece` (the same style — a field docstring naming the phase-95/182 lineage: `block: VerbatimBlock` — the class is already imported at L343);
- the round loop: `shown_before = len(holder.shown_texts)` immediately BEFORE `_execute_tool` (L2640, the `trunc_before` pattern); after the round's `ToolResultPiece` yield (L2690–2697), `for block in holder.shown_texts[shown_before:]: yield VerbatimBlockPiece(block=block)` — one line, the same snapshot-then-yield shape (a round can add at most one block — the already-shown dedupe is a refusal — so the loop is a one-iteration shape, but the snapshot keeps a future multi-add correct);
- the generator union types (L928/L992) gain `VerbatimBlockPiece`; the module docstring's piece paragraph (the L215–225 display-tools region) gains the one-line clause (the mid-turn frame, the done-frame count UNCHANGED — `verbatim_blocks=N` still counts the done frame's blocks, never the mid-turn frames);
- the `run()` docstring: the emission-order contract (the frame is after the round's `tool_result`, before the next round — a refused call emits nothing, the `shown_texts.append` at L2322 is the single success gate).
2. `app/schemas.py` — `ChatVerbatimBlockEvent(BaseModel)` (the `ChatToolResultEvent` L321 pattern, directly after it):
- `type: Literal["verbatim_block"] = "verbatim_block"` + the EXACT `VerbatimBlockRef` field set (L185 — `source` ≤ 120, `path` ≤ 1000, `title` ≤ 500, `start_line`/`end_line`/`total_lines` `ge=0`, `text` ≤ 600 000, `truncated: bool = False`, `locator` ≤ 2100) — the docstring: the mid-turn twin of the done frame's `answer_text` item; the client's `isVerbatimBlock` check (verbatim.js L70) accepts the event as-is (the key set matches — a pin); the sibling events' `model_config`/`extra` convention is mirrored EXACTLY (verify `ChatToolResultEvent`'s config and copy it).
3. `app/api/chat.py` — the dispatch branch after the `ToolResultPiece` branch (L1104–1119):
```python
if isinstance(piece, VerbatimBlockPiece):
# Phase 192 (task 01, A1): the mid-turn verbatim frame — the block
# the show_text call just displayed, sent at the moment it exists.
# Additive: a refused call yields no piece (no frame); a
# answer_text=False turn offers no tool (no frame) — the frame
# sequence is byte-identical to pre-phase in both cases.
yield sse_event(
ChatVerbatimBlockEvent(
source=piece.block.source,
path=piece.block.path,
title=piece.block.title,
start_line=piece.block.start_line,
end_line=piece.block.end_line,
total_lines=piece.block.total_lines,
text=piece.block.text,
truncated=piece.block.truncated,
locator=piece.block.locator,
).model_dump()
)
continue
```
(import `VerbatimBlockPiece` beside `ToolResultPiece` at L290; import `ChatVerbatimBlockEvent` beside the event imports at L310–322).
4. `tests/unit/test_agent.py` (verify the module name — the agent mock suite):
- `test_show_text_success_yields_one_verbatim_block_piece` — a mock turn whose answer calls `show_text` (the phase-182 mock precedent — verify the helper in the module) → exactly ONE `VerbatimBlockPiece`, positioned AFTER the round's `ToolResultPiece` (if a truncation piece exists) and BEFORE the next model round's first `StreamPiece`; the piece's `block` is the `holder.shown_texts` entry (same object);
- `test_show_text_refusals_yield_no_verbatim_block_piece` — the five refusal paths (the missing-arg `MISSING_SHOW_TEXT_ARGS`, the `answer_text=False` `UNKNOWN_TOOL`, the no-document refusal, the `is_image` refusal, the block-cap refusal, the already-shown dedupe — six pins, the module's refusal fixtures) → zero `VerbatimBlockPiece`s in the stream;
- `test_kill_switch_turn_yields_no_verbatim_block_piece` — the `answer_text=False` app posture → the tool is absent from the offered tools (the phase-182 D8 pin stays green) AND zero `VerbatimBlockPiece`s.
5. `tests/integration/test_chat_api.py` (verify the module name — the SSE contract suite):
- `test_verbatim_block_frame_lands_before_done` — a mock-LLM turn with a `show_text` call → the captured SSE stream carries a `type="verbatim_block"` frame with the EXACT `VerbatimBlockRef` field set (the pin asserts every key of `model_dump()`) positioned after the matching `tool_result`/`tool` frames and before `done`; the `done` frame's `answer_text` list is the same list the pre-phase resolution produced (the phase-182 pin, extended — the done-time resolution is untouched);
- `test_kill_switch_stream_is_byte_identical` — the `BOR_ANSWER_TEXT=0` module app: the captured frame sequence (types + payloads) equals the pre-phase expected sequence (the documented baseline in the test — a turn with the mock's `show_text` attempt gets the phase-182 refusal path; NO `verbatim_block` frame anywhere).
6. Verify: `uv run pytest tests/unit/test_agent.py tests/integration/test_chat_api.py -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the success-position pin, the six refusal pins, the kill-switch pin.
- Integration: the frame contract pin, the kill-switch wire pin, the done-frame parity pin.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] a successful `show_text` yields exactly one `verbatim_block` frame (after `tool_result`, before the next round) — unit + integration pins.
- [ ] every refusal path yields zero frames; the `answer_text=False` stream is byte-identical to pre-phase (the wire pin).
- [ ] the done frame's `answer_text` and the `verbatim_blocks=N` log line are unchanged (the phase-182 pins green unedited).
- [ ] full test suite **100% passing**.
- [ ] no behavior change in completed work (a turn without `show_text` is byte-identical — the pre-existing SSE pins green).
@@ -0,0 +1,66 @@
# Task 02 — Mid-turn landing: the fence hydrates in place, the unfenced block lands at the call moment (A2)
**Phase:** `192_verbatim_inline_midturn` · **Source:** `TODO.md:6 — "… which makes the chat read strangely."`
**Story:** n/a
## Objective
The client half: a `verbatim_block` frame settles the pending fence placeholder IN PLACE (the matching semantics of the done path) or, when nothing matches, lands the panel at the answer position AT THE MOMENT the call succeeds — so the reading order is the model's order: "…here it is:" → panel → "…as you can see…". Unmatched placeholders STAY pending (settling is the done/failure path's job).
## Work
1. `frontend/assets/verbatim.js` — a NEW `hydrateVerbatimNow(root, block, opts)` (after `hydrateVerbatim`, L446):
```js
/* hydrateVerbatimNow(root, block, opts) — the MID-TURN pass (phase 192,
* A2): settle the PENDING placeholders THIS ONE block matches, leave
* every other placeholder pending.
*
* The matching is the done path's semantics (matchVerbatimBlock with a
* ONE-block list): the exact canonical locator, or the identity
* fallback when the block is the only one of that identity — a fence
* for range B of a shown file A hydrates with A's server extent (the
* caption shows the SERVER's range, the phase-182 contract), so the
* mid-turn pass and the done pass can never disagree.
*
* ONLY matched nodes are replaced (buildVerbatimPanel); the unmatched
* pending nodes STAY .verbatim-pending — settling them is the
* done/failure path's job (settleVerbatimPlaceholders), and a mid-turn
* pass must never burn a placeholder for a not-yet-shown fence.
*
* Returns the number of placeholders settled (the caller appends the
* block — appendUnlocatedVerbatim — only when it is 0). Never throws:
* a missing root → 0. */
```
- implementation: `const list = [block]` guarded by `isVerbatimBlock(block)` (a malformed frame → 0, the house never-throw contract); the same `querySelectorAll(".verbatim-pending[data-bor-locator]")` walk as `hydrateVerbatim`; `matchVerbatimBlock(list, key)`; replace only when matched; count the replacements.
2. `frontend/assets/app.js` — the new branch beside the `tool_result` branch (L4612):
```js
} else if (ev.type === "verbatim_block") {
// Phase 192 (task 02, A2): the show_text panel lands at the moment
// the call succeeds — the fence it names hydrates IN PLACE; an
// unfenced block lands at the answer position now, not at done.
// The done pass re-runs the same functions: hydrateVerbatim is
// idempotent (only .verbatim-pending is touched) and
// appendUnlocatedVerbatim skips an already-placed locator, so the
// done frame and this frame can never double a panel.
const n = hydrateVerbatimNow(
wrap.querySelector(".bubble"), ev, VERBATIM_LINK_OPTS
);
if (n === 0) appendUnlocatedVerbatim(wrap, [ev], VERBATIM_LINK_OPTS);
}
```
- the import line: `hydrateVerbatimNow` joins `hydrateVerbatim` / `appendUnlocatedVerbatim` in the existing verbatim.js import (verify the import site);
- the `answer_text=False` gate is SERVER-side only (the frame never arrives) — NO client-side settings check (a client gate would be a second, drift-prone gate — the docstring says so).
3. `tests/unit/` — the JS marker pins (the verbatim JS pin module — verify the name; the `test_save_chat_ui.py` convention):
- `hydrateVerbatimNow` exists with the matched-only contract (the marker names the `matchVerbatimBlock` reuse + the "unmatched stay pending" clause + the never-throw guard);
- the app.js handler references `verbatim_block` + `hydrateVerbatimNow` + `appendUnlocatedVerbatim` in the `n === 0` shape (the source marker — the append only on zero matches);
- the `isVerbatimBlock` guard marker (a malformed frame settles nothing).
4. Verify: `uv run pytest tests/unit/<js-pin-module> -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the marker pins above (the matched-only contract is the task's heart — the E2E proves the behavior, the markers prove the shape).
- E2E: task 04 (the owner-visible arm).
- Coverage: **>90%** on `app/` (unchanged — the task is frontend; the app-side gate stays green).
## Completion Criteria
- [ ] `hydrateVerbatimNow` settles ONLY the pending placeholders the single block matches; the rest stay `.verbatim-pending`; returns the match count; never throws (marker pins).
- [ ] the app.js `verbatim_block` branch hydrates in place and appends ONLY on zero matches (marker pins); no client-side kill-switch gate (the marker asserts its absence).
- [ ] the existing `hydrateVerbatim` / `appendUnlocatedVerbatim` / `settleVerbatimPlaceholders` functions are UNTOUCHED (the phase-182 pins green unedited).
- [ ] no behavior change in completed work (a turn without the frame type never enters the branch — the pre-existing E2E suites green).
@@ -0,0 +1,36 @@
# Task 03 — The done/restore dedupe and the unchanged-contract pins (A3/A4)
**Phase:** `192_verbatim_inline_midturn` · **Source:** `TODO.md:6 — "… makes the chat read strangely."`
**Story:** n/a
## Objective
Close the one remaining duplicate: a mid-turn APPENDED panel and a later fence for the SAME canonical locator must end as ONE panel (the in-text one). The dedupe lives in `hydrateVerbatim` (the done AND restore path — one function, so the shared page rides it for free); per-locator, never per-identity — two DIFFERENT ranges of one file are legitimate and stay.
## Work
1. `frontend/assets/verbatim.js` — `hydrateVerbatim` (L423–445) gains the placed-scan, inside the per-node loop:
- before `node.replaceWith(buildVerbatimPanel(block, opts))` for a MATCHED key: `const wrap = root.closest(".msg") || root;` (the message wrap — the same host `appendUnlocatedVerbatim` scans); remove every PLACED resolved panel (`.verbatim[data-bor-locator]`, NOT `.verbatim-unresolved` — the `appendUnlocatedVerbatim` placed-scan convention, L463–467) whose `data-bor-locator` equals the key (the canonical-locator comparison — the exact `===`, the phase-182 D7 key), then build the in-text panel in the node's place;
- UNMATCHED nodes: UNCHANGED (still settle to `buildVerbatimUnresolved`);
- the docstring gains the A3 paragraph: the mid-turn appended panel is superseded by the in-text fence panel (the fence is where the model wanted it); the scan is PER LOCATOR — two different ranges of one file (the `shown_texts` per-locator dedupe admits both) are both legitimate placements and BOTH survive (the scan never compares identities);
- `appendUnlocatedVerbatim` is UNTOUCHED (its state-free idempotence — the `placed` set skip, L468–472 — already makes the done pass a no-op for an appended block).
2. `tests/unit/` — the JS marker pins (the verbatim JS pin module — the task-02 module):
- the `hydrateVerbatim` placed-scan marker: the scan targets `.verbatim[data-bor-locator]` (the class + attribute pair), EXCLUDES `.verbatim-unresolved`, removes on the exact `data-bor-locator` key (the three markers — the per-locator contract), and the "two ranges stay" clause is named in the docstring (the marker asserts the docstring contains the per-locator-not-per-identity sentence — the house docstring-pinning convention);
- the `root.closest(".msg")` wrap marker.
3. The unchanged-contract pins (A4) — the existing suites stay green UNEDITED, and three NEW integration/E2E-preparation pins:
- `tests/integration/test_chat_api.py` — `test_done_frame_and_log_line_unchanged_by_midturn_frames` — a mock turn WITH a `show_text` call: the `done` frame's `answer_text` list equals the phase-182 resolution (the task-01 pin, now extended with the per-turn log line assertion: `verbatim_blocks=N` where `N` = the done-frame block count, the mid-turn frames NOT counted — the captured log line pin, the house log-pin convention).
- `tests/unit/test_schemas.py` (or the phase-182 pin module — verify): `ChatVerbatimBlockEvent`'s `model_dump()` key set == `VerbatimBlockRef`'s key set + `type` (the `isVerbatimBlock` compatibility pin — the client check's required keys are present).
- the restore-parity marker (the app.js restore path L3044–3046 UNTOUCHED — a marker asserting the restore path still calls `hydrateVerbatim` + `appendUnlocatedVerbatim` with `m.answer_text` — the shared page's hydration call site likewise untouched; the BEHAVIOR pin is the E2E arm).
4. Verify: `uv run pytest tests/unit/<js-pin-module> tests/integration/test_chat_api.py -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the placed-scan markers (per-locator, unresolved-excluded, wrap derivation, the two-ranges-stay clause).
- Integration: the done-frame + log-line parity pin; the schema compatibility pin.
- E2E: task 04 (the no-double arm, the two-ranges arm, the restore arm).
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] a mid-turn appended panel + a same-locator fence → exactly ONE panel after `done` (the marker pins now; the E2E proves it) — the in-text panel wins, the placed panel removed.
- [ ] two different ranges of one file → two panels after `done` (the marker clause + the E2E arm).
- [ ] the restore path and the shared page's call sites are untouched (marker pins); the `done` frame's `answer_text` and the `verbatim_blocks=N` log line are unchanged (integration pins).
- [ ] `appendUnlocatedVerbatim` / `settleVerbatimPlaceholders` / the task-02 functions are untouched (the phase-182 pins green unedited).
- [ ] full test suite **100% passing**.
- [ ] no behavior change in completed work.
@@ -0,0 +1,36 @@
# Task 04 — The phase E2E suite, the full gates, the docs pass, the commit
**Phase:** `192_verbatim_inline_midturn` · **Source:** `TODO.md:6`
**Story:** n/a
## Objective
The dedicated Playwright suite proves the owner-visible contract — the panel is on screen WHILE the answer is still streaming, the fence lands in place, nothing doubles, the kill switch is a byte-identical wire — run in isolation per AGENTS rule 4; the full gates pass; the docs are consistent; one atomic commit.
## Work
1. `tests/e2e/test_verbatim_inline_midturn.py` (NEW — the module docstring names the TODO item and the A4 unchanged-contract clause) — the module-app fixture pattern (the phase-182 E2E convention — verify in `tests/e2e/conftest.py` / the sibling verbatim suites; the mock LLM from `tests/e2e/mock_llm.py` — verify the phase-182 mock's `show_text` arm; if the mock drives `show_text` via a fence-only turn, extend the mock with a direct `show_text` tool call arm in `tests/e2e/mock_llm.py` — the mock is a test fixture, the extension is additive and the phase-182 pins must stay green):
- the fixture: a small KB with one document whose content the mock's `show_text` call targets (the phase-182 fixture convention).
- **(a) the mid-turn arm (the TODO regression):** a mock turn: an intro delta ("…here is the exact text:") → the `show_text` call (the mock arm) → the continuation deltas → `done`. Assert: the panel (`.verbatim[data-bor-locator]` with the call's locator — caption + gutter + body) is in the DOM BEFORE the `done` frame is consumed (the Playwright shape: `page.wait_for_selector` on the panel resolves while the stream is still running — a sentinel delta after the call proves the stream is live; the panel's DOM position is BELOW the intro text and ABOVE the final sentinel delta — the inline position assertion, the `answer_image`-position contract of `appendUnlocatedVerbatim`); the `done` pass adds nothing (the final panel count == 1).
- **(b) the fence-in-place arm:** a mock turn: an intro delta → the `show_text` call for `source/path#12-20` → the final answer containing the ` ```bor ` fence for the SAME locator → `done`. Assert: the fence placeholder is hydrated IN PLACE (the panel is at the fence position — between the intro and the final sentinel delta, NOT at the answer tail) before `done` (the mid-turn `hydrateVerbatimNow` pass); after `done`, the total panel count for that locator == 1 (the dedupe — the task-03 arm).
- **(c) the two-ranges arm:** a mock turn with TWO `show_text` calls — range 1–10 and range 11–20 of the same document (the `shown_texts` per-locator dedupe admits both) — no fences. Assert: exactly TWO panels after `done`, both at the answer position in call order, both with their own captions (the per-locator-not-per-identity pin, owner-visible).
- **(d) the kill-switch wire arm:** the `BOR_ANSWER_TEXT=0` module app — the same mock turn (the mock's `show_text` attempt hits the phase-182 refusal) → the captured SSE frame sequence (types + payloads via the house SSE-capture pattern) equals the pre-phase baseline (documented in the test); NO `verbatim_block` frame, NO panel, NO placeholder in the DOM.
- **(e) the restore arm:** run arm (a) to completion (a saved conversation — the phase-158 auto-save), reload the page, restore → exactly ONE panel for the locator (the record's `answer_text` through the same dedupe functions — the A4 parity pin, owner-visible).
2. Regression, run in isolation: the phase-182 E2E suites (verify the exact names in `tests/e2e/` — the verbatim display suite, the fence suite), `tests/e2e/test_chat_history.py` (the restore contract) — green unedited.
3. The full gates, in order:
- `uv run pytest` (unit + integration) — 100% passing;
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on `app/`;
- `uv run pytest tests/e2e/test_verbatim_inline_midturn.py -v --no-cov` — green in isolation;
- `uv run ruff check . && uv run pyright` — clean.
4. Documentation pass (the phase file's Documentation section): `README.md` + `AGENTS.md` reviewed in their entirety against the post-phase app; the verbatim section notes the mid-turn landing (a `show_text` panel appears the moment the call succeeds — in place when fenced, at the answer position when not; the `done`/restore/shared contract is unchanged, the kill switch byte-identical); `AGENTS.md` corrected only where it names the done-frame resolution as the ONLY landing point.
5. Commit: one atomic, professional commit (Conventional Commits, `--no-gpg-sign`), e.g.
`feat(verbatim): show_text panels land mid-turn, at the moment the call succeeds (phase 192)` — the body names the additive `verbatim_block` frame, the `hydrateVerbatimNow` matched-only pass, the per-locator dedupe, and the A4 unchanged contract. `.agents/phases/todo/192_verbatim_inline_midturn/` moves to `.agents/phases/complete/` in the SAME commit (AGENTS rule 8).
## Testing & Quality
- E2E: the suite above, isolated; the regression suites green.
- Coverage: **>90%** on `app/`.
- Lint/types: clean.
## Completion Criteria
- [ ] `tests/e2e/test_verbatim_inline_midturn.py` green in isolation — (a) the panel is on screen before `done`, at the inline position; (b) the same-locator fence hydrates in place mid-turn and never doubles; (c) two ranges of one file → two panels; (d) the `answer_text=False` wire is byte-identical with no panel/placeholder; (e) the restore shows one panel per locator.
- [ ] `uv run pytest` 100% passing; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `README.md` / `AGENTS.md` consistent (the mid-turn landing clause).
- [ ] the phase directory is in `.agents/phases/complete/`; one atomic commit with `--no-gpg-sign`.
@@ -0,0 +1,55 @@
# Phase 193 — The sync label counts DOWN: "N remaining"
**Source:** `TODO.md` L6 — "The sync status should show how many files are left to sync, not how many files out of the total."
**Story:** n/a (owner directive)
**Depends on:** none functionally (builds on the completed phase-64/98/128/129/152 sync-label lineage).
## Evidence (diagnosed from the code, read this not the chat)
1. **The running label counts UP.** `frontend/assets/sources.js` `fmtSyncLabel` (L1053–1058): the import phase and the upload job render `count = total > 0 ? `(${done}/${total})` : ""` — `Syncing… notes/a.md (1234/27000)`. The owner wants the distance to the end, not the position in the whole: `25766 remaining`.
2. **The data already exists.** `SyncStatus.files_done` / `files_total` (the phase-64 payload — the running label's only inputs) carry everything: `N = total − done` is a subtraction, no server change, no new field, no new poll. The `#sync-progress` line (the phase-152 A4: the untruncated full label rides the button title + `#sync-progress` via `setSyncLabel`) gets the new part through the SAME builder — nothing to change there.
3. **The other count parts are different animals — they stay.** `· chunk n/m` (phase 129 — one file's embed batch), `· image n/m` (phase 152 — one PDF's vision span), and the summaries phase's `(n/m)` (phase 98 — the FOLDER position, always emitted, `fmtSyncPhaseLabel` L1089+) are per-file / per-folder position parts, not "files of the total". The terminal result line (`counts_line` — "N added · M updated …") is a different surface and is untouched.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the part (task 01).** `fmtSyncLabel`'s `count` becomes `total > 0 && total - done > 0 ? `${total - done} remaining` : ""` — the owner-confirmed shape (ASSUMPTION A1, confirmed 2026-10-18):
- `N remaining` shows while `N ≥ 1`; at the final file (`N == 0`) the part is HIDDEN — the main part names the file in flight, and a "0 remaining" next to the last file's name is the stale-looking edge the owner's complaint is about;
- plain integer (no thousands separator — the existing count part is plain; the house label style);
- the POSITION is unchanged (main → count → chunk → image — the phase-128 parts rework order);
- the UPLOAD job shares the builder and the format (owner-confirmed A2 — `Importing <file> N remaining`; the upload's `files_total` is the pre-walk count, the phase-98 contract — the same subtraction semantics);
- `total === 0` (the pre-walk null total, the phase-64 contract) → no count part (unchanged);
- the chunk / image parts and the summaries-phase folder position are UNTOUCHED (byte-identical);
- the docstring (L1044–1058 region) is updated to the new part (the phase-64/98 lineage paragraph keeps its history, the `(done/total)` description becomes the `N remaining` one).
- **D2 — the pinned suites move (task 02).** The running-label count pins retarget to the new part; the terminal-line pins stay unedited. The definitive list is found by `grep -rn "Syncing…\|Importing" tests/e2e/*.py | grep -E "\\\\d|/\{N|[0-9]+\)"` at execution time (the known pins: `test_sync_label_mobile.py` `IMPORT_FULL_RE` L226 + the docstring L93/L213; `test_sync_upload_progress.py` L653/L667/L850).
## ASSUMPTIONS (owner-confirmed 2026-10-18)
- **A1:** `N remaining` while `N ≥ 1`, hidden at `N == 0` (the final file — the main part names it).
- **A2:** the upload job shares the format (one builder, one shape).
## Context
- `frontend/assets/sources.js` — `fmtSyncLabel` (L1053–1058), `fmtSyncPhaseLabel` (L1089+, the phase-aware wrapper: overview / summaries / import), `setSyncLabel` (the A4 full-label placement — the button title + `#sync-progress`), the `counts_line` helper (the terminal result line — UNTOUCHED); `app/schemas.py` `SyncStatus` (`files_done` / `files_total` — the payload, UNTOUCHED); `app/api/sync.py` (the status endpoint — UNTOUCHED).
- `tests/e2e/test_sync_label_mobile.py`, `tests/e2e/test_sync_upload_progress.py`, `tests/e2e/test_sync_chunk_progress.py`, `tests/e2e/test_sync_summary_visibility.py` (the phase-64/98/128/129/152 label suites — the count-part pins; the chunk/image/summaries pins stay), `tests/e2e/test_sync_git_freeze.py` + `test_sync_session_reset.py` + `test_fast_path_deferred_content.py` (the `counts_line` terminal pins — unedited).
## Tasks
1. `01_remaining_label.md` — the `fmtSyncLabel` count part becomes `N remaining` (the D1 spec), the docstring update, the JS marker pin.
2. `02_update_pinned_e2e.md` — the running-label count pins retarget to the new part (the D2 list); the full suite green.
3. `03_e2e_and_gates.md` — the phase's own suite (the count-down through the real UI), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit (JS marker pin):** the new count expression (`total - done > 0` guard, the plain-integer shape, the `total > 0` guard kept, the upload kind sharing the builder — the marker names the expression).
- **E2E (regression, task 02):** the retargeted count pins + the chunk / image / summaries pins green; the terminal `counts_line` pins green UNEDITED.
- **E2E (phase, AGENTS rule 4, task 03):** `tests/e2e/test_sync_remaining_count.py`, run in isolation — the count-down arm (the part decreases across polls; hidden at the final file; the chunk part coexists), the upload arm (the shared format).
- **Coverage:** **>90%** on `app/` (unchanged — the task is frontend; the app-side gate stays green).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the sync section's running-label description (the `(done/total)` example, if present) becomes the `N remaining` one; `AGENTS.md` — the phase-64 sync-label clause (if it names the count part) gains the new shape (task 03 — verify, do not add if absent).
## Completion Criteria
- [ ] the running file-count part is `N remaining` (`N = total − done`, plain, shown while `N ≥ 1`, hidden at `N == 0`, `total === 0` → no part) — the marker pin + the E2E count-down arm.
- [ ] the upload label uses the same shape (the E2E upload arm); the chunk / image / summaries parts are byte-identical (the unedited pins green).
- [ ] the terminal result line is byte-identical (the `counts_line` pins green unedited); the `#sync-progress` line carries the new part through the same builder (the A4 pin).
- [ ] test suite green, coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] no behavior change in completed phases (the server, the payload, the phases of the run all untouched — the change is the one label part).
- [ ] one atomic Conventional Commits commit with `--no-gpg-sign`.
@@ -0,0 +1,31 @@
# Task 01 — The `N remaining` count part (D1)
**Phase:** `193_sync_remaining_count` · **Source:** `TODO.md:7 — "The sync status should show how many files are left to sync, not how many files out of the total."`
**Story:** n/a
## Objective
The one-line change the owner asked for: `fmtSyncLabel`'s running file-count part renders the distance to the end (`N remaining`, `N = total − done`) instead of the position in the whole (`(done/total)`) — sync and upload alike (one builder), hidden at the final file, the pre-walk `total === 0` behavior unchanged, the chunk / image / summaries parts untouched.
## Work
1. `frontend/assets/sources.js` — `fmtSyncLabel` (L1053–1058):
- the `count` expression: `const count = total > 0 && total - done > 0 ? `${total - done} remaining` : "";` (the `total > 0` guard kept — the phase-64 pre-walk contract: no count part before the walk knows the total; the plain integer — no thousands separator, the existing part's style; the POSITION in the returned object is unchanged — the phase-128 parts order main → count → chunk → image is preserved by `fmtSyncPhaseLabel` which consumes `count` unchanged);
- the docstring block above the function (L1044–1052) + the `fmtSyncPhaseLabel` docstring (L1060–1088): the `(done/total)` description becomes the `N remaining` one — the phase-64/98 lineage KEPT (the "count part" history), the new clause: `N = total − done`, shown while `N ≥ 1`, hidden at `N == 0` (the final file — the main part names it; a "0 remaining" next to the last file's name is the stale-looking edge), the upload job shares the builder and the shape (the phase-98 D2 "upload keeps the bare fmtSyncLabel parts" clause updated: still bare, now with the `N remaining` part);
- `setSyncLabel` / `#sync-progress` placement: UNTOUCHED (the A4 full label rides the same builder — the new part flows through automatically);
- the terminal `counts_line` helper: UNTOUCHED.
2. `tests/unit/` — the JS marker pin (the sources-JS pin module — verify the name; the `test_save_chat_ui.py` convention):
- the `fmtSyncLabel` count expression marker (the `total > 0 && total - done > 0` guard + the `` `${total - done} remaining` `` shape — the source marker asserts the exact expression);
- the `total === 0` guard marker (no count part before the walk);
- the "hidden at N == 0" clause marker (the docstring names the final-file edge);
- the chunk / image / `fmtSyncPhaseLabel` parts UNTOUCHED marker (the three part builders' source is unchanged — the `· chunk n/m` / `· image n/m` / summaries `(n/m)` strings appear only in their own builders, not in `fmtSyncLabel`).
3. Verify: `uv run pytest tests/unit/<sources-pin-module> -v` green; `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the marker pins above (the expression is the whole change — the markers pin the shape).
- E2E: task 02 (the retargeted pins) + task 03 (the phase arm).
- Coverage: **>90%** on `app/` (unchanged — the task is frontend).
## Completion Criteria
- [ ] `fmtSyncLabel`'s count part is `` `${total - done} remaining` `` with the two guards (the marker pins).
- [ ] the docstrings name the new part + the hidden-at-final-file edge + the upload sharing (the marker pins).
- [ ] the chunk / image / summaries / terminal-line code is byte-identical (the unedited pins green).
- [ ] no behavior change in completed work (the server and the payload are untouched).
@@ -0,0 +1,25 @@
# Task 02 — The pinned running-label count pins move to `N remaining` (D2)
**Phase:** `193_sync_remaining_count` · **Source:** `TODO.md:7`
**Story:** n/a
## Objective
The full suite stays green: every E2E pin that asserts the running file-count part retargets to the new shape; the chunk / image / summaries pins and the terminal `counts_line` pins stay UNEDITED.
## Work
1. The definitive pin list — run at execution time: `grep -rn "Syncing…\|Importing" tests/e2e/*.py | grep -E "\\\\d|/\{N|[0-9]+\)"` — the known pins (verify each against the list):
- `tests/e2e/test_sync_label_mobile.py` — `IMPORT_FULL_RE` (L226): `^Syncing… {FILE_RE} \([1-4]/4\)(?: · chunk \d+/\d+)?$` → `^Syncing… {FILE_RE} [1-4] remaining(?: · chunk \d+/\d+)?$` (a 4-file fixture: while file `i` is in flight `done = i − 1`, so `N = 4 − done ∈ {1, 2, 3, 4}` — `N == 0` never renders, the part is `[1-4] remaining`; the docstring lines L93/L213 updated to the new shape — the `IMPORT_MAIN_RE` (L217, the no-count mobile clip) is UNTOUCHED);
- `tests/e2e/test_sync_upload_progress.py` — L653, L667, L850: `\(\d+/{N_FILES}\)` → `[0-9]+ remaining` (the same module's `N_FILES` fixture; the L667 arm keeps the ` · chunk \d+/\d+` tail; the L653/L850 arms are the bare label).
2. Any ADDITIONAL pin the grep surfaces (a label regex with a count part in another suite — the phase-128/129/152 lineage: `test_sync_chunk_progress.py`, `test_sync_summary_visibility.py`, `test_sync_git_freeze.py`, `test_sync_session_reset.py` are the candidates — verify each: a pin on the RUNNING label's count part → retargeted; a pin on the CHUNK/IMAGE part, the summaries folder position, or the `counts_line` terminal line → UNTOUCHED).
3. The module docstrings of the retargeted suites: the shape description updated (the `(n/m)` running-count clause → the `N remaining` clause, the phase-64/98 lineage KEPT, one sentence: phase 193 moved the part to the distance-to-end per the owner's directive).
4. Verify: `uv run pytest tests/e2e/test_sync_label_mobile.py tests/e2e/test_sync_upload_progress.py -v --no-cov` green (plus every additional retargeted suite); `uv run pytest tests/e2e/test_sync_chunk_progress.py tests/e2e/test_sync_summary_visibility.py tests/e2e/test_sync_git_freeze.py tests/e2e/test_sync_session_reset.py -v --no-cov` green (the unedited chunk/image/terminal pins); `uv run pytest` (unit + integration) green.
## Testing & Quality
- E2E (regression): the retargeted count pins green; the chunk / image / summaries / terminal pins green unedited.
- Coverage: **>90%** on `app/` (unchanged).
## Completion Criteria
- [ ] every running-label count pin in `tests/e2e/` asserts the `N remaining` shape (the grep list is closed — every hit accounted for: retargeted or documented-untouched).
- [ ] the terminal `counts_line` pins and the chunk / image / summaries pins are green UNEDITED.
- [ ] `uv run pytest` 100% passing (the full suite — no pin left asserting the old shape).
- [ ] no behavior change in completed work (test-only + the task-01 label part).
@@ -0,0 +1,34 @@
# Task 03 — The phase E2E suite, the full gates, the docs pass, the commit
**Phase:** `193_sync_remaining_count` · **Source:** `TODO.md:7`
**Story:** n/a
## Objective
The dedicated Playwright suite proves the owner-visible contract — the label counts DOWN as the run progresses and goes quiet at the final file — run in isolation per AGENTS rule 4; the full gates pass; the docs are consistent; one atomic commit.
## Work
1. `tests/e2e/test_sync_remaining_count.py` (NEW — the module docstring names the TODO item, the A1/A2 assumptions and the "server untouched" clause) — the module-app fixture pattern (the phase-64/98 E2E convention — verify in `tests/e2e/conftest.py` / the sibling sync suites):
- the fixture: a small git source with N files (N ≥ 4, the phase-98 fixture convention — the `N_FILES` shape of `test_sync_upload_progress.py`).
- **(a) the count-down arm:** start the sync through the RAG page's sync flow; capture `#sync-progress` (and the button title — the A4 full-label placement) across the run: while files are in flight the label matches `^Syncing… <file> [1-N] remaining` and the displayed `N` is NON-INCREASING across polls (the `total − done` shape — a strict count-down; the pin: the sequence of captured `N`s, when a new value appears, is smaller); the `· chunk n/m` part coexists when a batch is in flight (the phase-129 part, unbroken); when the walk reaches the final file the part is ABSENT (the main part names the file — `^Syncing… <last-file>$`, no count part — the A1 hidden edge); the run's terminal result line is the phase-64 shape (the `counts_line` helper — byte-identical, the terminal-line pin).
- **(b) the upload arm (A2):** the upload job (the same fixture as an archive, the `test_sync_upload_progress.py` convention) → the label shows `Importing <file> [1-N] remaining` (the shared builder, the shared shape — the pin) and the chunk part coexists (the pin).
- **(c) the pre-walk edge:** the first poll before the walk knows the total → NO count part (`^Syncing…$`, the phase-64 contract — the `total === 0` guard, owner-visible).
2. Regression, run in isolation: `tests/e2e/test_sync_label_mobile.py`, `tests/e2e/test_sync_upload_progress.py` (the task-02 retargets), `tests/e2e/test_sync_chunk_progress.py`, `tests/e2e/test_sync_summary_visibility.py` (the chunk/image/summaries pins) — green.
3. The full gates, in order:
- `uv run pytest` (unit + integration) — 100% passing;
- `uv run pytest --cov=app --cov-report=term-missing` — **>90%** on `app/`;
- `uv run pytest tests/e2e/test_sync_remaining_count.py -v --no-cov` — green in isolation;
- `uv run ruff check . && uv run pyright` — clean.
4. Documentation pass (the phase file's Documentation section): `README.md` + `AGENTS.md` reviewed in their entirety against the post-phase app; the sync section's running-label example (the `(done/total)` shape, if present in `README.md`) becomes the `N remaining` one; `AGENTS.md` corrected only where a phase-64/98 clause names the running count part.
5. Commit: one atomic, professional commit (Conventional Commits, `--no-gpg-sign`), e.g.
`feat(sync): the running label counts down — "N remaining" (phase 193)` — the body names the one-line `fmtSyncLabel` change (sync + upload through the shared builder), the hidden-at-final-file edge, and the untouched chunk/image/summaries/terminal parts. `.agents/phases/todo/193_sync_remaining_count/` moves to `.agents/phases/complete/` in the SAME commit (AGENTS rule 8).
## Testing & Quality
- E2E: the suite above, isolated; the regression suites green.
- Coverage: **>90%** on `app/`.
- Lint/types: clean.
## Completion Criteria
- [ ] `tests/e2e/test_sync_remaining_count.py` green in isolation — (a) the count-down (non-increasing `N` across polls, the chunk part coexisting, the part absent at the final file, the terminal line byte-identical); (b) the upload arm (`Importing <file> N remaining`); (c) the pre-walk edge (no count part before the total is known).
- [ ] `uv run pytest` 100% passing; `uv run pytest --cov=app --cov-report=term-missing` >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] `README.md` / `AGENTS.md` consistent (the running-label shape).
- [ ] the phase directory is in `.agents/phases/complete/`; one atomic commit with `--no-gpg-sign`.
@@ -0,0 +1,70 @@
# Phase 194 — "N remaining" counts the files that will actually change
**Source:** owner report 2026-10-19 — "I just synced the prod brain of reese and it said '1216 remaining' when it only had like 10 files to sync" (the prod KB holds ~10,000 importable files; that run changed ~10).
**Story:** n/a (owner directive — phase 193's confirmed assumption A1 proved wrong in production)
**Depends on:** `193_sync_remaining_count` (complete) — builds on the phase-64/98/128/129/152/179/184/185/187 sync lineage.
## Evidence (diagnosed from the code, read this not the chat)
1. **The phase-193 label counts the WALK, not the WORK.** `frontend/assets/sources.js` `fmtSyncLabel` renders `N remaining` with `N = files_total − files_done` (the arithmetic is fine). But `app/rag/importer.py`'s pre-walk (L949–962) sets `total` to the COMBINED COUNT OF EVERY IMPORTABLE FILE across all sources (the 10,000), and the per-file hook (L1034–1039) reports `done = summary.files` (the 1-based position across ALL files walked). The fast path (phase 147/185) skips re-embedding for the unchanged 9,990 but still COUNTS them. Prod: "1216 remaining" = "8,784 files left to walk", while the actual work was ~10 files. Phase 193's E2E suite passed because its fixture KB was a FRESH import where walk == work — the incremental case the owner just hit was never pinned.
2. **The change signal already exists per file.** `_file_hash` (L704–724 — the sha256 of the RAW bytes, streamed in 1 MiB chunks) compared against `documents.file_hash` (nullable `String(64)`, no index — looked up through the unique `(source, path)` constraint, `app/models.py` L119/L138–142) is EXACTLY the fast path's change predicate: stored NULL (no row, or a pre-phase row) or `!= digest` ⇔ the file will be re-indexed (added/updated). A pre-scan reusing that predicate predicts the change set with the SAME semantics the processing loop then applies — pre-phase NULL rows are predicted changed AND actually re-indexed (stamped) by the loop, so the prediction never lies about the stored-hash contract.
3. **The pre-scan must be OFF the loop.** A 10,000-file hash+SELECT span run inline would park the single uvicorn loop for seconds — the exact phase-179 defect class pinned by `tests/integration/test_sync_git_freeze.py` (the 3 s git stub froze the 2 s status poll) and `tests/e2e/test_sync_git_freeze.py`. The git span moved to `asyncio.to_thread` (phase 179); the pre-scan does the same (the worker runs to completion on cancel; the run stops at the next phase-134 boundary — the `CancelledError` clause in `_run_sync`, L178–190 contract).
4. **Only the UI sync pays anything.** `import_sources` has exactly two callers: `app/api/sync.py` (the UI sync — the ONLY one that passes `progress`) and `scripts/import_docs.py` L407 (the CLI — no `progress`). Uploads have no file-level progress at all (phase 90 A2 — `app/api/git_sources.py` L59; their label renders bare, the `test_archive_upload_sources.py` pins). Gating the pre-scan on `progress is not None` (the phase-64 skip rule) keeps every non-UI caller byte-identical.
5. **The count is a prediction, and the drift edges are bounded.** The pre-scan runs seconds before the processing loop reaches a given file. A predicted file that turns out unchanged (reverted mid-run) still advances the count (it was predicted — `total` counted it) so the count reaches 0; a file changed AFTER the scan is indexed for real but does not advance the count, and the phase-193 `total − done > 0` guard hides the excess (never negative, never a stale "0 remaining"). The terminal `counts_line` ("N added · M updated …") stays the exact surface — untouched.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the pre-scan (task 01).** `import_sources` gains ONE optional callback `on_scan_progress: Callable[[int, int], None] | None = None` (the 1-based scan position + the importable total — the "only when provided", "hooks only assign fields", no-try/except contract of its three siblings). While `progress is not None` AND `llm.settings.sync_pending_count` is truthy (read ONCE before the source loop, the phase-184 read-once discipline — `0` = rollback to the phase-193 walk-position semantics, negative value refuses startup loudly, the house kill-switch pattern):
- the EXISTING stats-only pre-walk is REPLACED (not augmented — no third walk) by ONE `asyncio.to_thread` span that, per root with the EXACT `iter_importable_files` rules of the processing loop (same `ignore` / `include_hidden` / `image_extension_set` / images-toggle gating): (a) takes the per-root file list and its length (the scan total — the same number the old pre-walk computed), then (b) per file: `_file_hash(path)` (the existing streamed helper) + ONE scalar `SELECT documents.file_hash WHERE source = :s AND path = :p` (the only column — NO entity load, NO `defer`/`load_only` row, no session retention, one light metadata read per file; phase 184/185 compliant: it is not a whole-entity `select(Document)`, it defers nothing because it loads no row, and the processing loop's `_doc_row_stmt` fast-path row load stays the only ROW load) → `pending = {(source, rel)}` for every file whose stored hash is NULL or `!= digest`;
- `on_scan_progress(position, total)` fires per file (the scan position for the UI's "Scanning… (n/total)" label — called from the worker thread; field assignments are GIL-atomic, the phase-179 off-loop contract), and `files_total` becomes `len(pending)` while the per-file `progress` call's `done` becomes the 1-based POSITION IN THE PENDING SET (it advances for every predicted file, whether or not it turns out changed at processing time — the count and its denominator stay on the SAME set so the arithmetic never runs away);
- `summary.files` (the `import: summary … files=` log count — every file the loop processed, append-only log discipline) is UNTOUCHED; the `sync: done/cancelled/failed … files_done=… files_total=…` lines keep their EXACT shape, now carrying the pending counts (the module docstring names the semantic shift);
- the processing loop's FRESH per-file `_file_hash` is KEPT (the double hash is deliberate: the pre-scan's digest is the prediction, the loop's is the truth — a file edited between the two is still detected and re-indexed; the second read is page-cached — the cost the owner accepted, A3);
- `on_scan_progress` is called only when provided; `progress=None` callers (the CLI) skip the pre-scan entirely — byte-identical behavior (the phase-64 skip rule, extended);
- the degenerate empty KB: the scan total is 0, no callback fires, the run's `phase` stays `"scan"` until the first `progress` call (which there is none) or the next transition/terminal — the UI renders bare "Scanning…" (the `total === 0` guard → no count part) and the terminal clears the phase; documented, pinned in the task-05 arm;
- docstrings: the module `progress` paragraph (the `done`/`total` semantic shift), the new `on_scan_progress` paragraph, the `to_thread`/cancel/cost paragraphs, the `sync_pending_count` gating clause.
- **D2 — the kill switch (task 01).** `app/config.py` gains `sync_pending_count: int = 1` (`BOR_SYNC_PENDING_COUNT`): `1` (default) = the pre-scan pending-count semantics; `0` = the phase-193 WALK-position semantics (the stats pre-walk + `done = summary.files` — the full rollback the owner gets if the cost ever bites); a negative value refuses startup with the field named (the `agent_max_rounds` house pattern). `.env.example` + README settings documentation. The switch is read ONCE per run (the phase-184 discipline — a mid-run settings save cannot change the semantics between two files of the same walk).
- **D3 — the phase machine + status (task 02).** `SyncStatus.phase` gains `"scan"` (additive — the machine: the model-check + clone/pull prelude reports `null`, then `"scan"` (set immediately before `import_sources`, REPLACING today's `= "import"`), `"import"` (flipped by the FIRST `progress` call while `phase == "scan"` — the flip point is the existing per-file hook, so no extra callback), `"overview"`, `"summaries"` — the no-gap summaries skip still keeps the run in `"import"`). The flip + the pre-`import_sources` set are phase transitions, so each gets the phase-187 FORCED `_sample_rss()` (the per-file `_scan_hook` calls the UNFORCED throttled sample, the progress-hook convention — one `/proc` read per 5 s, never one per file). `SyncStatus` gains `scan_done` / `scan_total` (`int`, default 0; reset with the run; KEPT at every terminal — the phase-64 keep-final-values convention, beside the final file counts). The status payload gains the same two keys ADDITIVELY (placed after `files_total`; every pre-existing key keeps its name, type, position, and meaning — the phase-187 additive-only discipline). `current_file` stays `null` during the scan (the scan names no file). Docstrings: the module phase-machine paragraph (L188–200 region), the `SyncStatus` class paragraph (the phase-64/98/187 clauses gain the scan clause), the `GET /api/sync/status` docstring field paragraph.
- **D4 — the UI scan label (task 03).** `fmtSyncPhaseLabel` (the phase-aware wrapper — `frontend/assets/sources.js`) gains ONE explicit `"scan"` branch in its decision tree: main text `Scanning…` (no file — `current_file` is null in the phase, the branch must not read it), count part `(n/total)` rendered in the SAME count slot the import phase's `N remaining` uses (`n = scan_done`, `total = scan_total`; the `total === 0` guard → no count part, the phase-64 contract); the caller passes the scan pair through the existing call sites that already forward the summaries/chunk/image parts (the `phase` value selects which pair the wrapper consumes — the summaries-branch pattern). The `null` prelude + `"import"` fallback clause now covers ONLY those two. `fmtSyncLabel`'s `N remaining` arithmetic is BYTE-IDENTICAL (the phase-193 pins stay unedited — the server-side semantic change flows through the SAME builder unchanged). The A4 untruncated placement (button title + `#sync-progress`) is untouched — `Scanning… (n/total)` rides it automatically; the phase-187 `· peak N MB` trailing clause still lands on the `#sync-progress` counts line while the scan label is visible. The docstring's decision-tree paragraph (L157–167 region) is updated. The walk position surviving as `(n/total)` is HONEST for a scan — it is labeled "Scanning", never "Syncing" (the owner's objection, TODO L6, was the sync count wearing the total's shape).
- **D5 — the pinned suites move (task 04).** See the task for the definitive list (the count-frame pins retarget to the pending-set semantics where the fixture KB is NOT fresh; the fresh-fixture pins (chunk progress, label mobile, upload) stay unedited; the phase-184/185/179 integration pins get the pre-scan's statements distinguished or are verified unaffected).
## ASSUMPTIONS (owner-confirmed 2026-10-19, questionnaire)
- **A1:** `N remaining` counts down the pre-scan's PREDICTED change set (`files_total = len(pending)`, `files_done` = the 1-based position in it); the count advances on the prediction set (a predicted-then-unchanged file still advances it — the count reaches 0; a post-scan change is indexed but uncounted — the phase-193 guard hides the excess). First-time imports (empty DB) count the full walk exactly as phase 193 did; a no-change run shows no count part.
- **A2:** the pre-scan is a NEW VISIBLE `"scan"` phase — `Scanning… (n/total)` before the `import` phase; the scan's walk position is the only surviving `n/m` shape, and it is labeled a scan.
- **A3:** the cost is accepted: one extra full read + sha256 + one scalar metadata SELECT per importable file per run (the second read is page-cached; the span is off-loop via `to_thread`, the phase-179 rule). The rollback knob `BOR_SYNC_PENDING_COUNT=0` restores the phase-193 walk-position semantics; the CLI and the upload job pay nothing.
## Context
- `app/rag/importer.py` — `import_sources` (L756+, the pre-walk L949–962, the per-file `progress` call L1034–1039), `_file_hash` (L704–724), `_doc_row_stmt` (L725+), the module docstring's progress/chunk/image hook paragraphs.
- `app/api/sync.py` — the module docstring (the phase-179/134/98 paragraphs, L100–200 region), `SyncStatus` (L308+), `GET /api/sync/status` (L495+, the payload dict L548+), `_run_sync` (L651+, the reset block, the `_hook`/`_scan_hook`/`_chunk_hook`/`_image_hook` closures L815–860 region, the `_status.phase = "import"` set L880, the terminal log lines L1112/L1133).
- `app/config.py` — the `BOR_SYNC_*` settings block + validators (the `agent_max_rounds` kill-switch pattern); `.env.example`.
- `app/models.py` — `Document` (L117+, the unique `(source, path)`, the `file_hash` column contract L128–142).
- `frontend/assets/sources.js` — `fmtSyncPhaseLabel` (L157–167 docstring, the decision tree), `fmtSyncLabel` (L1043–1071 — UNTOUCHED), the status call sites (L1121–1127, L1449–1458, L1621–1652), `setSyncLabel`/A4 (UNTOUCHED).
- `scripts/import_docs.py` L407 (the CLI caller — `progress` absent — the byte-identity pin), `app/api/git_sources.py` L59 (the upload's no-file-progress contract — UNTOUCHED).
- Pinned suites: `tests/e2e/test_sync_remaining_count.py` (phase 193's own — the count-frame pins + the status key-set pin), `tests/e2e/test_sync_chunk_progress.py`, `test_sync_label_mobile.py`, `test_sync_upload_progress.py`, `test_archive_upload_sources.py` (the upload's "no `N remaining`" pins), `tests/integration/test_import_fast_path_columns.py` (phase 185 — the per-file SELECT pins), `test_import_session_retention.py` (phase 184), `test_sync_git_freeze.py` (phase 179 — the loop-freeze pin), `test_sync_session_reset.py`, `test_fast_path_deferred_content.py`, `test_sync_summary_visibility.py`, `test_sync_image_progress.py`, `test_sync_progress_box.py`, `test_sync_status_freshness.py`, `test_sync_peak_memory_visible.py`, `test_smoke.py`.
## Tasks
1. `01_prescan_importer.md` — the importer's pre-scan (the D1 spec) + the `BOR_SYNC_PENDING_COUNT` kill switch (the D2 spec) + the unit/integration pins.
2. `02_sync_phase_status.md` — the `"scan"` phase + `scan_done`/`scan_total` status fields + the additive payload + the docstrings (the D3 spec) + the integration pins.
3. `03_ui_scan_label.md` — the `fmtSyncPhaseLabel` scan branch (the D4 spec) + the JS marker pins.
4. `04_retarget_pinned_suites.md` — the pinned-suite retarget (the D5 list); the full suite green.
5. `05_e2e_and_gates.md` — the phase's own E2E suite (the pending-count-down through the real UI), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit (marker pins):** the pre-scan gate (the `progress is not None` + `sync_pending_count` read-once), the scalar SELECT's column shape (no entity load), the `pending` predicate (NULL / mismatch / match), the `done`-as-pending-position contract, the kill-switch value handling (1 / 0 / negative).
- **Integration:** the pending set on fresh / unchanged / changed / pre-phase-NULL fixtures; the off-loop pin (the delayed pre-scan span keeps the loop responsive — the phase-179 pattern); the cancel-during-prescan terminal; the phase-184/185/179 pins retargeted or verified; the phase machine sequence (null → scan → import → … → terminal null) + the kept scan counts + the forced/throttled RSS samples.
- **E2E (regression, task 04):** the retargeted pins green; the fresh-fixture + upload pins green UNEDITED.
- **E2E (phase, AGENTS rule 4, task 05):** `tests/e2e/test_sync_pending_count.py`, run in isolation — the first-sync arm, the INCREMENTAL arm (the owner's prod case: K of N changed, the count holds at `K remaining` while the walk moves over unchanged files, hidden at the final predicted file, terminal `K updated`), the no-change arm (no count part), the kill-switch arm (`0` → the phase-193 walk semantics), the empty-KB arm, the peak-clause-on-scan-line pin.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the sync section's running-label clause gains the `scan` phase + the pending-count semantics (the phase-193 `N remaining` example becomes the predicted-change-set one) and the `BOR_SYNC_PENDING_COUNT` setting (the `.env.example` entry is part of task 01); `AGENTS.md` — verify whether any clause names the running count part (phase 193 verified it does not — if still absent, do NOT add; the `BOR_SYNC_PENDING_COUNT` setting row joins the settings documentation wherever the README/AGENTS settings table lives).
## Completion Criteria
- [ ] `N remaining` counts down the pre-scan's predicted change set (`files_total = len(pending)`, `files_done` = the pending position) on a progress caller; a first-time import counts the full walk (A1's degenerate parity); a no-change run shows no count part — the E2E arms a/b/c.
- [ ] the `scan` phase is visible (`Scanning… (n/total)`, the position climbs, the `total === 0` guard) and flips to `import` on the first progress call; `scan_done`/`scan_total` reset with the run and survive the terminals — the E2E arms a/e + the integration phase-machine pin.
- [ ] `BOR_SYNC_PENDING_COUNT=0` restores the phase-193 walk-position semantics byte-for-byte (the E2E kill-switch arm + the marker pins); a negative value refuses startup with the field named.
- [ ] the pre-scan span never parks the loop (the off-loop pin); a cancel during it lands the `cancelled` terminal (the phase-134 contract); the CLI and the upload job are byte-identical (the unedited pins).
- [ ] the chunk / image / summaries parts, the `fmtSyncLabel` arithmetic, the terminal `counts_line`, `summary.files` and every pre-existing log line are byte-identical (the unedited pins green); the status payload is additive-only (the key-set pin).
- [ ] test suite green, coverage >90%; `uv run ruff check . && uv run pyright` clean; README + AGENTS consistent (the final pass).
- [ ] one atomic Conventional Commits commit with `--no-gpg-sign`.
@@ -0,0 +1,48 @@
# Task 01 — The importer's pre-scan + the `BOR_SYNC_PENDING_COUNT` kill switch (D1/D2)
**Phase:** `194_sync_pending_count` · **Source:** owner report 2026-10-19 — "1216 remaining" on a ~10,000-file KB with ~10 changed files.
**Story:** n/a
## Objective
The server-side core: `import_sources` predicts the run's change set BEFORE the walk (the pre-scan, D1) so the `progress` hook's `done`/`total` are the PENDING position (the files that will actually be re-indexed), with the `BOR_SYNC_PENDING_COUNT` kill switch (D2) — `0` restoring the phase-193 walk-position semantics byte-for-byte, negative refusing startup.
## Work
1. `app/config.py` — `sync_pending_count: int = 1` (env `BOR_SYNC_PENDING_COUNT`), the house pattern next to `sync_session_reset_files` (phase 184) / `agent_max_rounds`: `1` = the pre-scan pending-count semantics (default), `0` = the phase-193 walk-position rollback (the stats pre-walk + `done = summary.files`), a NEGATIVE value refuses startup loudly with the field named (the validator). `.env.example` — the entry, documented like the other `BOR_SYNC_*` rows.
2. `app/rag/importer.py` — `import_sources`:
- the signature gains `on_scan_progress: Callable[[int, int], None] | None = None` (the 1-based scan position + the importable total — the "only when provided", "the hooks only assign fields", no-try/except contract of `progress` / `on_chunk_progress` / `on_image_progress`);
- read `llm.settings.sync_pending_count` ONCE before the source loop (the phase-184 read-once discipline — a mid-run settings save cannot flip the semantics between two files of the same walk);
- the GATE: `progress is not None` AND the switch truthy → the PRE-SCAN; `progress is None` (the CLI, `scripts/import_docs.py` L407) → skipped entirely (the phase-64 skip rule, extended — byte-identical); the switch `0` on a progress caller → the EXISTING stats-only pre-walk runs and `done = summary.files` (the phase-193 rollback, byte-for-byte);
- the pre-scan REPLACES the existing stats-only pre-walk (L949–962) — no third walk: ONE `asyncio.to_thread` span (the phase-179 off-loop rule — a 10,000-file hash+SELECT span must not park the single uvicorn loop; the span's worker runs to completion on cancel, the phase-134 contract) that, per root with the EXACT `iter_importable_files` rules of the processing loop (the same `ignore` / `include_hidden` / `image_extension_set` / images-toggle gating — the L930–962 region's rules, byte-identical): (a) the per-root file list + length (the scan total — the same number the old pre-walk computed), then (b) per file: `_file_hash(path)` (the existing streamed 1 MiB-chunk helper, L704–724) + ONE scalar `SELECT documents.file_hash WHERE source = :s AND path = :p` (the ONLY column — NO entity load, NO `defer`/`load_only` row, nothing lands in the session's `identity_map`, no retention — phase 184/185 compliant: it is not a whole-entity `select(Document)` and it defers nothing because it loads no row; the processing loop's `_doc_row_stmt` stays the ONLY row load) → `pending = {(source, rel)}` where stored is NULL (no row, or a pre-phase row) or `!= digest` — the SAME predicate the fast path applies at processing time, so the prediction is consistent with the stored-hash contract (a pre-phase NULL row is predicted changed AND actually re-indexed + stamped by the loop);
- `on_scan_progress(position_1based, total)` fires per file from inside the worker span (the call is a plain field-assignment closure — GIL-atomic; document the thread-safety clause in the docstring);
- `files_total` = `len(pending)` (the `progress` hook's denominator); the per-file `progress(source, rel, done, total)` call's `done` becomes the 1-based POSITION IN THE PENDING SET — it advances for every file in `pending` (whether or not the file turns out changed at processing time — count and denominator stay on the SAME set so the arithmetic never runs away; a file NOT in `pending` does not advance it — the label holds at `K remaining` while the walk moves over the unchanged `N − K`);
- the processing loop's FRESH per-file `_file_hash` stays (the double hash is deliberate: the pre-scan's digest is the prediction, the loop's is the truth — a file edited between the two is still detected; the second read is page-cached — the cost the owner accepted, A3);
- `summary.files` (the `import: summary … files=` count — every file the loop processed) is UNTOUCHED; the prune post-pass, the date refresh, the phase-184 `expunge_all` cadence, the `seen` set: all UNTOUCHED (the pre-scan builds `pending`, not `seen` — the loop keeps building `seen` exactly as today);
- the degenerate empty KB: scan total 0, no callback fires, `pending` empty, `files_total` 0 (the phase-64 `total === 0` no-count-part contract);
- docstrings: the module `progress` paragraph (the `done`/`total` semantic shift + the drift edges — a predicted-then-unchanged file still advances the count so it reaches 0; a post-scan change is indexed but uncounted and the UI's `total − done > 0` guard hides the excess), the new `on_scan_progress` paragraph (the "only when provided" rule, the worker-thread call, the no-try/except contract), the `to_thread`/cancel/cost paragraph (the phase-179/134 clause + the double-hash + the page-cache note + the kill-switch gate), the `sync_pending_count` read-once clause.
3. `tests/unit/` — the marker pins (the importer pin-module convention, the phase-184/185 pin style):
- the gate: `progress is None` → no pre-scan (the callback never fires, no scalar statement — a mocked-callback pin); the switch `0` on a progress caller → the stats pre-walk path (a source marker pin: the old pre-walk code is the `0` branch);
- the scalar SELECT shape: a captured statement names EXACTLY `documents.file_hash` (column-list pin, the phase-185 style — no other column, no entity);
- the `pending` predicate: fresh (no rows → all pending), unchanged (hash match → not pending), changed (mismatch → pending), pre-phase NULL row (→ pending), an ignore-rule file (→ never in the walk, never pending);
- the `done`-as-pending-position: a 3-file fixture (2 predicted, 1 not) → the `progress` calls' `done` sequence is the pending position (the non-predicted file reports the previous position; `total = 2`);
- the switch values: `1`/`0`/negative (the validator names the field on the negative value).
4. `tests/integration/` —
- the pending set through a real `import_sources` call (a `file://`-style temp source dir, the house fixture convention): fresh / unchanged / one-changed fixtures assert the `progress` hook's `done`/`total` pairs AND the terminal `summary.added`/`summary.updated` (the parity pin: the prediction's size equals the terminal's `added + updated` on a stable KB);
- NO entity hydration: `len(session.identity_map) == 0` immediately after the pre-scan span (the `progress` + `on_scan_progress` caller, before any processing-loop row load);
- OFF-LOOP (the phase-179 pattern, `test_sync_git_freeze.py` style): a monkeypatched `_file_hash` that blocks ~3 s over a small multi-file fixture → a loop-side heartbeat (a coroutine timing `asyncio.sleep(0)` / a probe task) stays responsive through the pre-scan span;
- CANCEL: a cancel registered before the (delayed) pre-scan → the worker runs to completion, the run lands the `cancelled` terminal (the phase-134 contract, `test_sync_git_freeze.py`'s cancel-during-clone pattern).
5. Verify: `uv run pytest tests/unit tests/integration -v` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the marker pins above (the gate, the scalar shape, the predicate, the `done` contract, the switch values).
- Integration: the pending-set parity, the no-hydration pin, the off-loop pin, the cancel pin.
- Coverage: **>90%** on this task's new/modified code (the whole gate — `uv run pytest --cov=app --cov-report=term-missing` at the phase's final pass).
## Completion Criteria
- [ ] the pre-scan runs on a progress caller with the switch truthy, is skipped for `progress=None` callers, and the `0` switch runs the phase-193 walk-position path (the marker pins).
- [ ] the scalar SELECT's column list is exactly `[documents.file_hash]` with no entity hydration (the integration pins).
- [ ] `done`/`total` are the pending position on a stable KB (the parity pin: `files_total == summary.added + summary.updated`).
- [ ] the pre-scan span never parks the loop; a cancel during it lands `cancelled` (the integration pins).
- [ ] `summary.files`, `seen`, the prune, the date refresh and every log line are byte-identical (the unedited pins green).
- [ ] `BOR_SYNC_PENDING_COUNT` in `.env.example` + the config validator (negative refuses startup with the field named).
- [ ] full test suite **100% passing**; README/AGENTS: the setting is documented (the phase's final pass re-checks both project-wide).
- [ ] no behavior change in completed work (the CLI, the upload job, the switch-`0` path).
@@ -0,0 +1,43 @@
# Task 02 — The `"scan"` phase + `scan_done` / `scan_total` (D3)
**Phase:** `194_sync_pending_count` · **Source:** owner-confirmed A2 — the pre-scan is a visible phase.
**Story:** n/a
## Objective
The status half of D1: `SyncStatus` carries the scan phase (`"scan"`) and the scan position (`scan_done` / `scan_total`), the `GET /api/sync/status` payload gains the two keys ADDITIVELY, and every docstring names the new machine stage — the phase-187 forced/throttled sampling discipline extended to the scan transition.
## Work
1. `app/api/sync.py` — `SyncStatus`:
- `phase: Literal["scan", "import", "overview", "summaries"] | None = None` (additive value — the machine: prelude `null` → `"scan"` (set immediately before `import_sources`, REPLACING today's `_status.phase = "import"` at L880) → `"import"` (flipped by the FIRST `progress` call while `phase == "scan"` — the flip lives in the existing `_hook` closure: `if _status.phase == "scan": _status.phase = "import"`) → `"overview"` → `"summaries"`; the no-gap summaries skip still keeps the run in `"import"`);
- `scan_done: int = 0` / `scan_total: int = 0` (placed after `files_total` — reset with the run in the `_run_sync` reset block L664–679 region, kept at EVERY terminal like the file counts — the phase-64 keep-final-values convention);
- the class docstring: the phase-64 paragraph (the `files_done`/`files_total` clause now names the pending-set semantics — "the hook's pending position (phase 194: the pre-scan's predicted change set)"), the phase-98 paragraph (the machine gains the `"scan"` stage), a new phase-194 paragraph (the scan fields, the reset/keep convention, the `null` `current_file` during the scan, the empty-KB degenerate clause — the phase stays `"scan"` until the next transition/terminal, the UI renders bare "Scanning…").
2. `app/api/sync.py` — `_run_sync`:
- the reset block: `scan_done = 0` / `scan_total = 0` with the run;
- the pre-`import_sources` point (L880): `_status.phase = "scan"` + the phase-187 FORCED `_sample_rss()` (a phase transition — the forced-sample convention, replacing the current `= "import"` set + forced sample);
- a new `_scan_hook(scan: int, total: int)` closure passed as `on_scan_progress`: sets `scan_done` / `scan_total` + calls the UNFORCED `_sample_rss()` (the per-file throttled convention — one `/proc` read per 5 s, never one per file, LOCKED A4's throttle discipline);
- the `_hook` (progress) closure: the flip clause (above) + a FORCED `_sample_rss()` on the flip (the scan→import transition is a phase transition);
- the `import_sources` call (L881–893): `on_scan_progress=_scan_hook` added;
- the terminal paths (`sync: done/cancelled by user/failed` — L1112/L1133 region): UNTOUCHED in shape (the `files_done=… files_total=…` fields keep their names — now carrying the pending counts; the scan fields keep their final values beside them — the keep-final convention);
- the module docstring: the phase-machine paragraph (L188–200 region — the `"scan"` stage + the flip point), the phase-64 progress paragraph (L100–115 region — the `files_done`/`files_total` semantic shift), the phase-187 sampler paragraph (the scan transition's forced sample + the per-file scan hook's throttled sample), the cancel paragraph (a cancel during the pre-scan span: the `to_thread` worker runs to completion, the run stops at the next boundary — the phase-134/179 contract, the `CancelledError` clause's two paths unchanged).
3. `app/api/sync.py` — `GET /api/sync/status`:
- the payload dict: `"scan_done"` / `"scan_total"` added after `"files_total"` (ADDITIVE ONLY — every pre-existing key keeps its name, type, and position; the phase-187 additive-only discipline);
- the endpoint docstring: the new fields' paragraph (the scan position — 0/0 before the scan starts and in terminal states the final values are kept; `current_file` null during the scan).
4. `tests/integration/` —
- the PHASE MACHINE sequence (a real app, a small git-source fixture with one changed file): a held first status GET (the prelude — `phase: null`, `scan_*: 0/0`) → a `phase: "scan"` frame with `scan_total` set and `scan_done` climbing (≥2 distinct positions on a multi-file fixture) → a `phase: "import"` frame (the flip — `files_total` = the predicted count) → the terminal (`phase: null`, the scan counts KEPT beside the final file counts — the keep-final pin);
- the PAYLOAD key-set pin: the phase-193 key set + `scan_done` / `scan_total` (the additive pin — every pre-existing key byte-identical in name/type);
- the FORCED samples: a forced `_sample_rss` at the pre-`import_sources` set (the `"scan"` transition) and at the flip (the `"import"` transition) — the phase-187 sampling-pin pattern (`test_sync_peak_memory_visible.py` convention);
- the DEGENERATE empty-KB run: the phase reports `"scan"` (no `on_scan_progress` fires — `scan_*` 0/0), never `"import"` (no progress call), the terminal clears the phase (the pin names the D1 degenerate clause);
- the CANCEL-during-scan terminal (a delayed pre-scan, the task-01 cancel pin's app-side twin): `state: "cancelled"`, the phase cleared, the scan counts kept.
5. Verify: `uv run pytest tests/integration -v` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Integration: the phase-machine sequence, the payload key-set, the forced/throttled samples, the degenerate + cancel terminals.
- Coverage: **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] the machine is `null → "scan" → "import" → "overview" → "summaries"` with the flip on the first progress call (the integration sequence pin).
- [ ] `scan_done` / `scan_total` reset with the run, are kept at every terminal, and are additive-only in the payload (the pins).
- [ ] the phase-187 sampling discipline holds (forced at the scan + import transitions, throttled per-file in `_scan_hook`) (the pins).
- [ ] every pre-existing log line, payload key and terminal behavior is byte-identical (the unedited pins green).
- [ ] full test suite **100% passing**; no user-visible doc changes this task (the phase's final pass reviews README/AGENTS).
- [ ] no behavior change in completed work.
@@ -0,0 +1,35 @@
# Task 03 — The `fmtSyncPhaseLabel` scan branch (D4)
**Phase:** `194_sync_pending_count` · **Source:** owner-confirmed A2 — the visible `Scanning… (n/total)` label.
**Story:** n/a
## Objective
The UI half of D1/D3: `fmtSyncPhaseLabel` (the phase-aware wrapper) gains ONE explicit `"scan"` branch — `Scanning…` + the `(n/total)` count part in the SAME count slot the import phase's `N remaining` uses — while `fmtSyncLabel`'s arithmetic stays BYTE-IDENTICAL and the A4 untruncated placement + the phase-187 peak clause flow through untouched.
## Work
1. `frontend/assets/sources.js` — `fmtSyncPhaseLabel` (the decision tree + its docstring, L157–167 region):
- the new `"scan"` branch: main text `Scanning…` (NO file — `current_file` is null in the phase; the branch must not read it), count part `(n/total)` with `n = scan_done`, `total = scan_total` — rendered in the SAME count slot (the `{ main, count }` part shape the import phase's `N remaining` uses, so `setSyncLabel`'s hidden-toggle + the A4 untruncated placement — button title + `#sync-progress` — carry `Scanning… (n/total)` automatically); the `total === 0` guard → NO count part (the phase-64 contract, the empty-KB clause — bare "Scanning…");
- the wrapper consumes the scan pair the way it already consumes the summaries pair (`summaries_done` / `summaries_total` / `current_summary` — the `phase` value selects which pair the branch reads); the status call sites (L1121–1127, L1449–1458, L1621–1652 region) forward `scan_done` / `scan_total` alongside the existing fields (the small caller pass-through — the summaries-forwarding pattern);
- the `null` prelude + `"import"` fallback clause now covers ONLY those two (the docstring's decision-tree paragraph updated: the new branch is named, the fallback narrowed, the phase-193 `N remaining` clause kept);
- the walk position surviving as `(n/total)` is HONEST for a scan — it is labeled "Scanning", never "Syncing" (TODO L6's objection was the sync count wearing the total's shape — the docstring says so);
- `fmtSyncLabel` (L1043–1071): UNTOUCHED (the `N remaining` arithmetic + its guards are byte-identical — the phase-193 pins stay unedited); the chunk / image parts, `setSyncLabel`, the terminal `counts_line`: UNTOUCHED;
- the phase-187 peak clause (`· peak N MB` on the `#sync-progress` counts line — `fmtSyncPeakClause`): UNTOUCHED (it lands on the scan label's line while the scan is visible — no new element, no new live region).
2. `tests/unit/` — the JS marker pins (the sources-JS pin-module convention, the phase-193 marker style):
- the `"scan"` branch marker (the `Scanning…` main + the `phase === "scan"` selection in the decision tree);
- the `(n/total)` shape marker (the exact template over `scan_done` / `scan_total` — the source marker asserts the expression);
- the `total === 0` guard marker (no count part before the scan knows the total);
- the caller pass-through marker (the call sites forward the scan pair);
- the UNTOUCHED markers: `fmtSyncLabel`'s `N remaining` expression (the phase-193 pin — byte-identical), the chunk / image / summaries part builders, `setSyncLabel`, the `counts_line` helper.
3. Verify: `uv run pytest tests/unit -v` green; `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- Unit: the marker pins above (the branch, the shape, the guard, the pass-through, the untouched set).
- E2E: task 04 (the retargeted pins) + task 05 (the phase arms — the scan label through the real UI).
- Coverage: **>90%** on `app/` (the task is frontend — the app-side gate stays green at the phase's final pass).
## Completion Criteria
- [ ] the `"scan"` branch renders `Scanning…` + `(n/total)` in the count slot with the `total === 0` guard (the marker pins).
- [ ] `fmtSyncLabel`'s `N remaining` arithmetic is byte-identical (the unedited phase-193 pins green).
- [ ] the A4 untruncated placement + the phase-187 peak clause are untouched (the unedited pins green).
- [ ] full test suite **100% passing**; no user-visible doc changes this task (the phase's final pass reviews README/AGENTS).
- [ ] no behavior change in completed work (the prelude / import / overview / summaries labels byte-identical).
@@ -0,0 +1,38 @@
# Task 04 — The pinned suites move (D5)
**Phase:** `194_sync_pending_count` · **Source:** the D5 retarget list.
**Story:** n/a
## Objective
Every existing pin that asserts the phase-193 WALK-position semantics (or the pre-phase statement counts the pre-scan now disturbs) is retargeted to the pending-set semantics — and every pin that is unaffected by the change (fresh-fixture count pins, the upload's bare-label pins, the terminal-line pins) is verified green UNEDITED. The full suite ends green.
## Work
The definitive list (per suite: decide from the fixture's KB state — a FRESH-DB fixture has pending == walk, so its count pins survive unedited; a RE-SYNC fixture's walk-position pins retarget):
1. `tests/e2e/test_sync_remaining_count.py` (phase 193's own suite — the primary retarget):
- the docstring's `files_done` / `files_total` clause (the "the 1-based …" wording, L15–16 region) retargets to the PENDING-set semantics (the pre-scan's predicted change set; the phase-193 walk wording moves to the kill-switch history note);
- the status key-set pin (the "the phase-64 key set" clause) moves to the phase-64 set + `scan_done` / `scan_total` (the additive pin);
- the arm's phase-sequence pins: the prelude → `import` sequence gains the `"scan"` stage (a `phase: "scan"` frame with `scan_total` set before the first `files_total`-bearing frame — the D3 machine); the pre-scan's label frames (`Scanning… (n/total)`) are pinned where the arm's held-first-GET / pre-first-file region stands;
- the count frames: on the suite's FRESH-DB fixture, pending == walk — the `1 … N−1` frames and the "hidden at the final file" pin survive (the phase-193 "Notable" deviation note retargets to the pending-position wording); ADD the re-sync arm if the suite's fixture allows it (a second sync of the same repo: no count part — `files_total = 0`, the phase-193 A1 exception now driven by the empty pending set) — otherwise the re-sync pin lives in the task-05 suite;
- the upload arm (arm c): UNEDITED (the upload status endpoint gains no scan fields — phase 90 A2 stands).
2. `tests/integration/test_import_fast_path_columns.py` (phase 185 — the per-file SELECT pins): inspect the invocation. If it calls `import_sources` WITHOUT `progress` → the pre-scan never runs → verify green UNEDITED. If WITH `progress` → retarget the "exactly one document-wide `SELECT` per unchanged file" pin to distinguish the pre-scan's scalar statement (column list exactly `[documents.file_hash]` — the task-01 shape pin) from the fast-path ROW statement (content deferred — the phase-185 pin intact); the "one PK follow-up for the backfill" pin is UNTOUCHED either way (the pre-scan's scalar is a `(source, path)`-keyed statement, not a PK follow-up).
3. `tests/integration/test_import_session_retention.py` (phase 184 — the bounded/unbounded retention pins): the pre-scan hydrates NO entities (the task-01 no-hydration pin) → the retention bound is UNCHANGED; verify green (if the test's fixture invokes with `progress`, confirm the worker-thread scalar statements disturb no retention assertion — the scalar loads no row).
4. `tests/integration/test_sync_git_freeze.py` (phase 179 — the loop-freeze pins): the pre-scan span is itself off-loop (the task-01 off-loop pin lives in its own module, the phase-179 pattern) — verify this suite's git-span pins stay green; if its measurement window now overlaps the pre-scan, extend it with the pre-scan heartbeat (the loop must stay responsive through BOTH off-loop spans).
5. `tests/e2e/test_sync_chunk_progress.py` (the `Syncing… chunkprog/big.md 1 remaining · chunk n/m` pins): the fixture is a fresh-DB import → pending == walk → verify green UNEDITED (the `1 remaining` part reads identically — the pending denominator equals the walk denominator on a fresh KB).
6. `tests/e2e/test_sync_label_mobile.py` (the `IMPORT_FULL_RE` pin + the label docstring): inspect the fixture's KB state — fresh-DB → verify green UNEDITED; any re-sync walk-position pin retargets (the count part absent on a no-change re-sync).
7. `tests/e2e/test_sync_upload_progress.py` + `tests/e2e/test_archive_upload_sources.py` (the upload's "no `N remaining` count" pins — phase 90 A2): the upload job has no pre-scan and no scan fields → verify green UNEDITED.
8. `tests/e2e/test_sync_session_reset.py`, `tests/e2e/test_fast_path_deferred_content.py`, `tests/e2e/test_sync_summary_visibility.py`, `tests/e2e/test_sync_image_progress.py`, `tests/e2e/test_sync_progress_box.py`, `tests/e2e/test_sync_status_freshness.py`, `tests/e2e/test_sync_peak_memory_visible.py`, `tests/e2e/test_smoke.py` — the sync-lineage sweep: run each in isolation (`--no-cov`), verify green; a pin that breaks on the new `"scan"` stage of the phase sequence or on the pre-scan's seconds (e.g. a freshness window that now spans the scan span) retargets minimally (the window/sequence pin names the scan stage — never the arithmetic).
9. `AGENTS.md` / `README.md` are NOT edited this task (the phase's final pass owns the docs — task 05).
10. Verify: `uv run pytest -v` (unit + integration) green; the ISOLATED `--no-cov` sweep of items 1–8 green (the phase-193 14-suite convention, extended by this phase's suite in task 05); `uv run ruff check . && uv run pyright` clean.
## Testing & Quality
- The retargeted pins (the pending-set wording, the key-set pin, the phase-sequence pins).
- The verified-unedited pins (fresh-fixture count pins, the upload's bare-label pins, the terminal `counts_line` pins, the phase-184/185/179 pins).
- Coverage: **>90%** on `app/` (the gate stays green — this task edits tests + docstrings).
## Completion Criteria
- [ ] the retargeted list above is settled: every broken pin retargeted with the reason in its docstring, every unaffected pin green UNEDITED (the sweep).
- [ ] `tests/e2e/test_sync_remaining_count.py` pins the new machine (the `"scan"` stage + the key-set + the pending-position wording) and its fresh-fixture count frames survive.
- [ ] the terminal `counts_line` pins, the chunk / image / summaries pins and the upload pins are green UNEDITED.
- [ ] full test suite **100% passing** (unit + integration + the isolated E2E sweep).
- [ ] no behavior change in completed work (only test files + their docstrings move).
@@ -0,0 +1,42 @@
# Task 05 — The phase's E2E suite + gates + docs + commit
**Phase:** `194_sync_pending_count` · **Source:** AGENTS rule 4 — "One Story, One Phase" (the owner-directive phase's own suite).
**Story:** n/a
## Objective
The owner-visible proof through the real UI: the count counts down the PREDICTED change set (the owner's prod case — K of N changed), the scan phase is visible and honest, the kill switch rolls back to phase 193, and the no-change / first-sync / empty-KB edges behave per D1 — then the full gates, the docs pass, and the atomic commit.
## Work
1. `tests/e2e/test_sync_pending_count.py` (NEW — run in isolation: `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov`):
- the house fixture convention (the phase-193/107 suite shape): a real git repo over the `file://` transport, the mock-LLM proxy with the house per-request delay, a unique E2E port pair (check the existing allocations in the sibling suites — the one-suite-at-a-time rule makes collisions harmless, but pick a free pair anyway), a `file://`-repo N = 8 importable files (keep the suite under ~2 min; ≥2 multi-chunk files so the chunk part coexists, the phase-129 convention);
- the module app: the default `BOR_SYNC_PENDING_COUNT` (1); a second module app for the kill-switch arm (`BOR_SYNC_PENDING_COUNT=0`) — the phase-184 two-module-app convention;
- **arm a — the first sync (fresh KB):** the phase sequence `null` → `"scan"` (the label `Scanning… (n/8)` — the position climbs: pin ≥2 distinct `scan_done` values at `scan_total = 8`; `current_file` null in the phase) → `"import"` (the count part `N remaining` with `files_total = 8` — pending == walk on a fresh KB, the phase-193 parity: the frames count down, hidden at the final file) → the terminal (`phase: null`, `scan_done = scan_total = 8` KEPT beside `files_done = 8`, the `sync: done` counts line `8 added`);
- **arm b — the incremental sync (the owner's prod case — the phase's heart):** modify K = 2 of the 8 (the fixture's changed-set), re-sync: the scan phase climbs `(n/8)` (the full walk is scanned), then the import: the count part reads `2 remaining` and HOLDS while `current_file` moves over UNCHANGED files (pin: ≥2 consecutive unchanged-file frames at `2 remaining` — the count does not advance on them), advances to `1 remaining` on the first predicted file, HIDES on the final predicted file (the phase-193 A1 edge — the main part names it), terminal `2 updated`;
- **arm c — the no-change re-sync:** the scan completes, the import renders BARE `Syncing… <file>` (NO count part — `files_total = 0`, the phase-64 guard), the terminal `0 added · 0 updated` (the `counts_line` byte-identical shape, the phase-64 pin);
- **arm d — the kill switch:** the second module app (`BOR_SYNC_PENDING_COUNT=0`), arm b's fixture: NO `"scan"` phase (the machine is `null` → `import` — the phase-193 machine), the count part counts down the WALK (`8 remaining` → hidden at the final file — the phase-193 semantics, the rollback pin), the terminal `2 updated`;
- **arm e — the empty KB:** a repo with zero importable files: `phase: "scan"` with `scan_*` 0/0 (bare `Scanning…` — the `total === 0` guard, NO count part), never `phase: "import"`, the terminal clears the phase (the D1 degenerate pin);
- **the phase-187 peak clause** (the `· peak N MB` / `fmtSyncPeakClause` trailing text on the `#sync-progress` counts line) is visible while the scan label is showing (the clause lands on the scan line — no new element, the phase-187 pin);
- **the upload arm:** a background upload run renders its bare label with NO file and NO count part (the phase-90 A2 pin — the `test_archive_upload_sources.py` convention; the upload status endpoint carries no scan fields);
- the `#sync-progress` A4 untruncated placement (the button `title` + `#sync-progress` text equal the full `Scanning… (n/total)` label, the phase-64 A4 pin).
2. The full gates (the phase's final pass — run in order, all green before the commit):
- `uv run pytest -q --cov=app --cov-report=term-missing` → exit 0, coverage **>90%** on `app/`;
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → all arms pass in ISOLATION;
- the isolated `--no-cov` sweep of the sync-lineage suites (task 04's list + this suite): `test_sync_remaining_count`, `test_sync_label_mobile`, `test_sync_chunk_progress`, `test_sync_image_progress`, `test_sync_progress_box`, `test_sync_summary_visibility`, `test_archive_upload_sources`, `test_sync_upload_progress`, `test_sync_status_freshness`, `test_sync_peak_memory_visible`, `test_sync_session_reset`, `test_fast_path_deferred_content`, `test_sync_git_freeze`, `test_smoke`;
- `uv run ruff check . && uv run pyright` clean.
3. The documentation & consistency pass (AGENTS rule + the phase's D-docs clause):
- `README.md` in full: the sync section's running-label clause — the phase-193 `N remaining` example becomes the PREDICTED-change-set one (the `scan` phase + `Scanning… (n/total)` + `N remaining` over the pending set + the no-change/first-sync/empty-KB edges + the `BOR_SYNC_PENDING_COUNT` setting: `1` default, `0` = the phase-193 walk-position rollback, negative = startup refusal); the `.env.example` entry was added in task 01 — verify it is documented here too; every pre-existing inconsistency found in the full review is fixed (the phase-193 docs-pass convention);
- `AGENTS.md` in full: verify whether any clause names the running count part (phase 193 verified one does NOT — if still absent, do NOT add one; if one exists, it gains the pending-set wording + the kill switch); the `BOR_SYNC_PENDING_COUNT` row joins wherever the settings documentation lives.
4. The atomic commit (AGENTS rule 8): ONE Conventional Commit, `--no-gpg-sign`, the phase directory moved `todo/` → `complete/` in the SAME commit (the house convention — the phase-193 commit shape: the `phase: 194_sync_pending_count` message with the verified-evidence block). Suggested subject: `feat(sync): count down the sync's predicted change set (pre-scan)`.
## Testing & Quality
- E2E: this suite, isolated (the six arms + the peak-clause + the A4 + the upload pins).
- Gates: the full suite + coverage >90% + the 15-suite sweep + lint/types (above).
- Docs: README + AGENTS reviewed in full, every inconsistency fixed.
## Completion Criteria
- [ ] arm b proves the owner's prod case through the real UI (the count holds at `2 remaining` over the unchanged files, hides at the final predicted file, terminal `2 updated`).
- [ ] arms a/c/d/e pin the first-sync parity, the no-change no-count edge, the kill-switch rollback and the empty-KB degenerate.
- [ ] the peak clause + the A4 placement pins hold on the scan label.
- [ ] all gates green (above) — full suite 100%, coverage >90%, the sweep, lint/types.
- [ ] README + AGENTS accurate and consistent with the whole project (the final pass).
- [ ] ONE atomic Conventional Commit with `--no-gpg-sign`, the phase directory in `complete/`.
@@ -0,0 +1,15 @@
All gates green; working tree holds exactly the phase's changes for the harness to commit.
## Phase 179 — Final verification report
- **Verified (all 3 tasks already complete in `complete/`)**: repro + CONFIRMED verdict, `asyncio.to_thread` fix, dedicated E2E suite.
- `app/api/sync.py::_run_sync`: both git ops (`clone_or_pull`, `file_commit_dates`) now `await asyncio.to_thread(...)`; phase-134 boundary note + module docstring document worker-thread-runs-to-completion and the A3 DB-calls-stay-on-loop note.
- **Tests / lint / coverage** (full suite 100% green):
- `uv run pytest tests/integration/test_sync_git_freeze.py -v` → **4 passed** (post-fix probe <1.5 s during 3 s stub)
- `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` → **2 passed in 25.65 s in isolation** (wall guard 120 s holds)
- `uv run pytest --cov=app --cov-report=term-missing` → **4173 passed**, `app/` coverage **99%** (`app/api/sync.py` 100%)
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
- **Completion criteria**: ① integration probe green + verdict (CONFIRMED, mechanism, measured 3.04 s→0.003 s) in docstring — **met**; ② off-loop git ops + boundary/A3 notes — **met**; ③ full suite green, coverage 99% >90%, ruff+pyright clean — **met**; ④ E2E green in isolation (responsive-poll + cancel-during-clone contracts) — **met**; ⑤ atomic commit + move to `complete/` — **harness-owned** (pipeline rule overrides per-phase self-commit; all changes staged-ready in the working tree).
- **Documentation**: full README (1526 l) + AGENTS.md review — phase-179 bullets present and accurate (verified against fixture constants, `.env.example` fully covered by the config reference, pages table matches `app/main.py` shell routes, quick-reference commands/artifacts exist); **consistent — no further changes needed**.
- **Notable**: task-03 pipeline sessions died on context overflow but their harness validation passed; this pass independently re-ran every gate to confirm. No live infra touched; no git index commands run.
- **Next pending phase**: `180_markdown_inline_links_images`.
@@ -0,0 +1,136 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
..................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6468 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,8 @@
**Task 01 complete — verdict: CONFIRMED.**
- **`tests/integration/test_sync_git_freeze.py`** (new, test-only; no `app/` change): deterministic in-process repro of the owner's TODO L3 suspicion — 3.0 s blocking stubs on `app.api.sync.clone_or_pull`/`file_commit_dates` (the L233 module names), env-fallback `BOR_GIT_SOURCES` source, real one-commit fixture repo, house `sync_client`/hygiene-fixture patterns, admin/anonymous 403 pins.
- **Verdict recorded in the module docstring**: the sync's synchronous `clone_or_pull → run_git → subprocess.run` runs ON the single uvicorn event loop (`create_task`, no `to_thread`/`create_subprocess_exec` anywhere in `app/`/`scripts/` — grep-verified), stalling the whole HTTP surface. Measured: status probe mid-clone **3.04 s** (≥2.5 floor; slept inside the stub's recorded window), `/api/health` **3.02 s**, idle baseline **0.001 s**, run lands `success` (freeze ≠ failure).
- **Deviation (documented in docstring + S2)**: under `TestClient` the trigger's own `202` is also held ≈3.2 s (the `create_task` body outruns the response dispatch on the shared portal loop), so the trigger POST rides a side thread and the main thread probes — one loop, two concurrent clients, the real uvicorn topology. **Task 02 note**: post-fix both the probe assertion flips to <1.5 s and S2's held-`202` assertion must be flipped/removed (the `to_thread` await yields the loop, freeing the trigger response).
- **Gates (all green)**: `uv run pytest tests/integration/test_sync_git_freeze.py -v` → 3 passed; `uv run pytest --cov=app --cov-report=term-missing` → **4172 passed**, `app/` coverage **99%** (>90%); `uv run ruff check .` → clean; `uv run pyright` → 0 errors.
- **Docs**: no doc impact (test-only task; nothing user-visible changed).
- **Next pending task**: `179_git_clone_server_freeze/02_offloop_git_ops.md` (the `asyncio.to_thread` fix + probe flip).
@@ -0,0 +1,136 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
.................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6468 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1,9 @@
**Task 02 complete — fix landed, all gates green.**
- `app/api/sync.py::_run_sync`: both per-source git ops now `await asyncio.to_thread(...)` (`clone_or_pull`, `file_commit_dates`); module docstring gained the phase-179 freeze paragraph (TODO L3 CONFIRMED + mechanism + A3 note that the short-lived `SessionLocal`/`effective_sources` calls stay on the loop, deliberately out of scope); the phase-134 boundary comment, `_check_cancelled` and `_run_sync` docstrings now state the worker-thread-runs-to-completion semantics (cancel unblocks the run at the `await`, loop free, `cancelled` terminal unchanged). `check_models`, the local-kind branch and `scripts/git_sync.py` untouched.
- `tests/integration/test_sync_git_freeze.py`: probe flipped to post-fix (`< 1.5 s`), plus pins that the probe lands *inside* the clone window (`finished_at is None`), that the git stub's thread ident differs from the loop's (`to_thread` worker, not the loop), the UI's second poll ~1 s in, the prompt `202`, and a NEW `test_cancel_during_a_clone_lands_cancelled_with_a_free_loop` (202 prompt, `cancelled` terminal inside the still-running clone, `error` null / `detail` `{}`, import never ran). Docstring records CONFIRMED → FIXED with measured numbers: probe 0.0031 s, 2nd poll 0.0027 s, health 0.0033 s, trigger 202 0.026 s, cancel 0.0037 s / terminal 0.0056 s (pre-fix: 3.04 / 3.02 / ~3.2 s).
- Fixed two suites that silently depended on the old synchronous scheduling: `test_sync_api.py` (`test_double_trigger_while_running_returns_409` now `_wait_phase(...)`-polls for `phase=="import"` instead of assuming it; import delay 0.5→1.5 s) and `tests/unit/test_sync_button.py` (new `_pump_until` helper — the loop-thread actions' `Event.wait()` deadlocked once the prelude yielded the loop; 5 action sites + driver docs updated).
- Gates: `uv run pytest tests/integration/test_sync_git_freeze.py -v --no-cov` → **4 passed** (×3 runs, stable); `uv run pytest --cov=app --cov-report=term-missing` → **4173 passed**, `app/` coverage **99%** (>90%; `app/api/sync.py` **100%**); `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings. E2E spot-checks in isolation: `test_sync_cancel` 3 passed, `test_sync_button` 3 passed, `test_sync_status_freshness` 4 passed, `test_force_push_resync` 2 passed.
- Docs: README "Sync from the UI" gained the "a long clone never freezes the server" bullet + the cancel bullet's worker-thread wording; AGENTS.md gained rule 11 ("Never block the event loop", `asyncio.to_thread`, pointing at the pinned integration test).
- Note: no live-infra touched; no git index/history commands run — all changes left in the working tree.
- Next pending task: `179_git_clone_server_freeze/03_e2e_freeze_contract.md` (the dedicated `tests/e2e/test_sync_git_freeze.py` Playwright suite with the real generated `file://` repo).
@@ -0,0 +1,136 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 12%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 31%]
........................................................................ [ 32%]
........................................................................ [ 34%]
........................................................................ [ 36%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 43%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 56%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 63%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
..................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6468 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
@@ -0,0 +1 @@
Phase-134 regression spot-check passes. Now the full suite with coverage:

Some files were not shown because too many files have changed in this diff Show More