Compare commits

...
43 Commits
Author SHA1 Message Date
ducoterra a45f744604 phase: 194_sync_pending_count
Build and Push Containers / build-and-push-app (push) Successful in 2m5s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 194 — final verification pass: all green.**

**Verified (all 5 tasks already complete; nothing re-implemented needed)**
- `app/rag/importer.py`: `_prescan_pending` + `_pending_hash_stmt` (scalar `documents.file_hash`, own session, ONE `asyncio.to_thread` span, REPLACES the stats pre-walk), `on_scan_progress`, `progress` pair = pending position, `0` branch = byte-identical phase-193 walk.
- `app/config.py`/`settings_catalog.py`/`.env.example`: `sync_pending_count=1` + validator; `app/api/sync.py`: `"scan"` phase, `scan_done`/`scan_total` (additive after `files_total`, kept at every terminal), forced/throttled RSS split; `frontend/assets/sources.js`: `fmtSyncPhaseLabel` scan branch first-arm, `fmtSyncLabel` untouched.

**Tests / lint / coverage (exact commands)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4835 passed**, `app/` 99% (7240 stmts / 16 miss), exit 0
- `uv run pytest tests/e2e/test_sync_pending_count.py -v --no-cov` → **8 passed in 150.81s** (phase suite, isolated)
- 14 sync-lineage E2E suites, one pytest process each (`test_sync_remaining_count`, `…_chunk_progress`, `…_progress_box`, `…_peak_memory_visible`, `…_label_mobile`, `…_upload_progress`, `test_archive_upload_sources`, `…_image_progress`, `…_summary_visibility`, `…_status_freshness`, `test_smoke`, `test_fast_path_deferred_content`, `test_sync_session_reset`, `test_sync_git_freeze`) → **all exit 0**
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**

**Completion criteria** — all met: `N remaining` = predicted change set (E2E arms 1/2/5, first import counts the walk, no-change → no count part); `scan` phase visible + flips to `import`, scan pair reset/kept at terminals (`test_sync_scan_phase.py`); `BOR_SYNC_PENDING_COUNT=0` walk semantics ("7 remaining" arm) + **negative refuses startup naming the field (verified live by importing `app.main`)**; pre-scan off-loop + cancel lands `cancelled` (integration pins); CLI/upload byte-identical (unedited pins green); payload additive (`_PRE_187_STATUS_KEYS | _MEMORY_KEYS | _SCAN_KEYS`); suite green, coverage >90%, lint/types clean.

**Documentation** — 3 fixes: README sync "States" bullet had an **unclosed parenthesis** (pre-existing, 193/194) and the scan-label "total is 0" clause was ambiguous about a `=0` run (it still counts the walk once the import starts); **AGENTS.md rule 17 had a stray `)`** (pre-existing from phase 191). Also verified consistency both directions for all 87 settings ↔ README ↔ `.env.example` ↔ settings catalog, the pages table ↔ `app/main.py` routes, no stale `BOR_` names, and README/AGENTS claims through phase 194.

**Notes/deviations** — no live-infra changes (prod sync knob ships default-on, no deployment touched); no git index/history commands run (all changes left in the working tree for the harness); commit criterion left to the harness as instructed.

**Next pending phase:** none — `.agents/phases/todo/` holds only `194_sync_pending_count`, which is now ready to move to `complete/`.
2026-10-20 06:36:21 -04:00
ducoterra ae080ac7f3 phase: 193_sync_remaining_count
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Successful in 16s
**Phase 193 — final verification pass: all gates green.**

- **Verified (all 3 tasks already complete in `complete/193_sync_remaining_count/`):** `fmtSyncLabel`'s count part is `total > 0 && total - done > 0 ? \`${total - done} remaining\` : ""` (plain int, hidden at `N==0`, `total===0` → no part); docstrings updated; chunk/image/summaries parts + `counts_line` terminal + `app/` byte-untouched (`git diff --stat`: no `app/` files).
- **Completion criteria:** `N remaining` shape + marker pins ✅ · upload shares the builder (E2E upload arm) ✅ · chunk/image/summaries pins green ✅ · terminal `counts_line` green unedited + A4 title/`#sync-progress` pin ✅ · suite green, coverage 99% ✅ · lint/types clean ✅ · commit + `todo/→complete/` left to the harness (no git index/history commands run).
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, junit: **4792 tests, 0 failures/errors/skipped**, `TOTAL 7181 16 → 99%` (>90%)
- `uv run pytest tests/e2e/test_sync_remaining_count.py -v --no-cov` → **3 passed** (isolated, 53 s)
- Isolated `--no-cov` sweep of the 14 sync-lineage suites (label_mobile, upload_progress, chunk_progress, image_progress, progress_box, summary_visibility, archive_upload_sources, status_freshness, upload_no_scan, peak_memory_visible, session_reset, fast_path_deferred_content, git_freeze, smoke) → **14/14 exit 0**
- `uv run ruff check .` → All checks passed · `uv run pyright` → **0 errors**
- **Docs:** README.md reviewed in full (all 2408 lines) + AGENTS.md. Fixed 2 inconsistencies: the sync running-label clause claimed the upload label shows the count — now states it shares the ONE builder/shape but reports no counts since phase 90 (renders bare); phase-190's "the other ~72 rows" → "the other 83 rows of the panel's 86-row catalogue" (mechanically: 86 CATALOG rows, 84 overridable). `N remaining` already in the sync paragraph; task-01's `(n/m)` examples gone. AGENTS.md verified consistent (no clause names the running count part → unchanged per the phase's "do not add if absent"). Mechanically re-verified: all 86 `BOR_*` settings documented + defaults match, `.env.example` 88 vars covered, 0 broken README anchors, all cited file paths exist, 17 palette colors, migrations head `0030`, pages/shell routes, ports 8123/8901, E2E port allocations (new suite's 8211/8914 unique).
- **Notable:** task 02's documented deviation stands — `files_done` is the importer's 1-based *current* file, so the phase doc's `[1-4]` range is unreachable; count-bearing frames are `1 … N−1` (pinned). Pre-existing, untouched: `E2E_APP_PORT_TOOLBUDGET`/`SAVEDOC` both 8129 and `SYNCUP`/`SYCNCEL` both 8133 — harmless under the one-suite-at-a-time rule.
- **Next pending phase:** none — `193_sync_remaining_count` is the last entry in `.agents/phases/todo/`.
2026-10-19 17:30:03 -04:00
ducoterra fb17d15fbe phase: 192_verbatim_inline_midturn
Build and Push Containers / build-and-push-app (push) Successful in 1m56s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Phase 192 — final verification pass complete.

- **Verified (all 4 tasks already complete):** additive `verbatim_block` SSE frame (`VerbatimBlockPiece` → `ChatVerbatimBlockEvent` → chat dispatch, after the round's `tool`/`tool_result`, before the next round); `hydrateVerbatimNow` matched-only mid-turn pass + app.js branch; `dropPlacedTwin` per-locator done/restore dedupe; the dedicated E2E suite.
- **Root cause of the rejected attempt:** my own concurrent background `pytest` run deadlocked the harness validation run on the shared Postgres (`psycopg.errors.DeadlockDetected`). No code defect — re-ran every gate **exclusively**.
- `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, `TOTAL 7181 16 99%` (>90%)
- `uv run pytest tests/e2e/test_verbatim_inline_midturn.py --no-cov` → **6 passed** (isolated)
- `uv run pytest tests/e2e/test_verbatim_show_text.py tests/e2e/test_chat_history.py --no-cov` → **18 passed** (`test_chat_history.py` byte-unedited; the phase-182 suite's "no eighth type" pin was tightened, not loosened, to seven + `verbatim_block` with frame-order asserts)
- `uv run ruff check .` → All checks passed (exit 0) · `uv run pyright` → **0 errors**
- **Completion criteria:** single frame per success + zero per refusal ✅ · `BOR_ANSWER_TEXT=0` wire byte-identical (integration spelled baseline + E2E App B) ✅ · fence hydrates in place / unfenced lands at the call moment, before `done` ✅ · append+fence → 1 panel, two ranges → 2 panels ✅ · restore 1 panel/locator, `done.answer_text` + record key + `verbatim_blocks=N` unchanged ✅ · suite green, coverage 99%, lint/types clean ✅ · no-show_text turn byte-identical ✅ · commit → left to the harness (no git index/history commands run).
- **Docs:** README reviewed line-by-line (2399 lines) + AGENTS.md in full; mechanically verified all 86 `BOR_*` settings, every `.env.example` var, table defaults, internal anchors (0 broken), ports 8123/8901, tree depth 1–8, pagination 25/1–100, 17 palette colors, `0029`/`0030` migrations, and every cited test file. Fixed 1 inconsistency: the verbatim "Parity" bullet claimed the live turn/reload/shared page share "the same three globals" — now names the settle-path trio and states that `hydrateVerbatimNow` is the ONE chat-only addition (a shared page has no stream).
- **Notable:** `run_agent`'s four repeated piece unions were collapsed into the ONE exported `AgentPiece` alias, so a future piece type is spelled in one place; `test_frontend_edit_message.py`'s `runTurn` byte-pin was regenerated, not weakened.
- **Next pending phase:** `193_sync_remaining_count`.
2026-10-19 12:19:51 -04:00
ducoterra 4f20c256d7 phase: 191_source_name_labels_tags
Build and Push Containers / build-and-push-app (push) Successful in 7m24s
Build and Push Containers / build-and-push-db (push) Successful in 36s
All gates re-verified against a stable tree. Report:

**Verified (all 6 tasks in `complete/`; this was the final pass)**
- `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`.
- **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned).
- **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code.
- **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria.
- **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
2026-10-19 06:13:32 -04:00
ducoterra c6ffa050a6 phase: 190_app_name_settings_fix
Build and Push Containers / build-and-push-app (push) Successful in 2m9s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All gates re-verified green. Report:

- **Verified (all 4 tasks already complete; this was the final pass):** `effective_settings` resolves the 3 brand strings panel-row → tab-row → env (`applied_override_keys`, presence-not-value); `PUT /api/ui-settings` ignores string body fields; Theme-tab form is colors-only (17 pickers, no `#theme-app-name`/`#theme-placeholder`/`#theme-footer`); the ONE shared `applyBrandFromConfig` re-applies title/`.brand-text`/footer/placeholder after a panel save-reset and a tab save-reset; boot path byte-identical (single `/api/config`).
- **Completion criteria:**
  - strings order panel→tab→env, both branches — ✅ `tests/unit/test_theming.py::test_effective_app_name_prefers_settings_panel_row_over_ui_settings_row`, `..._fall_back_to_ui_settings_row_without_panel_row`, `..._per_field_panel_then_tab_then_env`, `..._ignore_panel_rows_when_the_overlay_is_off`
  - PUT ignores string fields + no 3 inputs in the tab — ✅ `tests/integration/test_ui_settings_api.py::test_put_ignores_string_fields_in_body` (+2) and the E2E `test_theme_tab_dom_is_colors_only`
  - live apply without reload + incident regression — ✅ `tests/e2e/test_app_name_settings_fix.py` tests 1–4
  - byte-identical boot, no double-apply — ✅ `tests/unit/test_brand_live.py` markers + E2E "exactly one /api/config"
  - suite green, coverage >90%, ruff+pyright clean — ✅
  - no behavior change in prior phases — ✅ (palette, phase-168 panel, env fallback suites all green)
  - atomic commit — **left to the harness** (pipeline rule overrides per-phase committing; no git write run)
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **4630 passed, TOTAL 6941 16 99%**, exit 0 · `uv run pytest tests/e2e/test_app_name_settings_fix.py -v --no-cov` → **6 passed** in isolation · regression E2E (isolated, one process each, all exit 0): `test_smoke` `test_configurable_brand` `test_admin_theme_tab` `test_theme_save_and_coverage` `test_theme_semantic_completion` `test_themed_favicon` `test_dark_tech_theme` `test_admin_settings_panel` `test_admin_settings_layout` `test_settings_mobile_value_width` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors.
- **Documentation:** README — fixed "saving one repaints *every* open page" (only the open document repaints; other pages on next load) and a broken paragraph wrap in *Customizing the look*; AGENTS.md rule 16 — `test_chunks_hnsw_recall.py` now described accurately (index paths forced with `enable_seqscan`/`indexonlyscan`/`bitmapscan` off, unforced plan asserted only on a pristine relation); `tests/unit/test_theme_frontend.py` — two stale section headers ("20 form fields"/"20 inputs" → 17 color). No AGENTS.md settings-overlay/resolution-order clause exists, so per the phase none was added. Full env-var cross-check (`BOR_*` in `app/config.py` vs README) → 0 gaps; 0 missing file references.
- **Notable:** the working tree also carries this phase's flake hardening of the phase-188 `tests/integration/test_chunks_hnsw_recall.py` (verified legit and green). Live infra untouched; shared DB left clean (0 `app_settings`, 0 `ui_settings`). **Flagged, not changed:** `settings_overlay.applied_override_keys` lets a `ProgrammingError` (missing `app_settings`) propagate from public `GET /api/config`, where `load_overrides` deliberately fails soft — only reachable on an un-migrated (0027) DB, outside this phase's pinned A1 semantics, so left for the owner to rule on rather than silently deviating.
- **Next pending phase:** `191_source_name_labels_tags`.
2026-10-18 02:59:19 -04:00
ducoterra 47456a44df chore(agent): phase roadmap from TODO.md, 4 phases (190-193)
Convert the four open TODO items into executable phases (append mode,
starting at 190 after the completed 189):

  190_app_name_settings_fix  - the Settings-panel app name wins over the
      Theme tab; the three brand strings stripped from the Theme tab
      (colors-only); live brand re-apply on save, no reload
  191_source_name_labels_tags - source_labels table + admin API + Sources
      page editor; the separate bounded source_name_hit_bonus retrieval
      boost; label surfacing in the <document> blocks and KB overview
  192_verbatim_inline_midturn - an additive verbatim_block SSE frame at
      the show_text success moment; in-place mid-turn fence hydration;
      the done/restore per-locator dedupe; the done frame, record key,
      restore and shared page stay byte-identical
  193_sync_remaining_count - the running sync/upload label counts down:
      "N remaining" (N = total - done), hidden at the final file; the
      chunk/image/summaries/terminal parts untouched

TODO.md is cleared (its items now live in .agents/phases/todo/).

LLM-Generated: true
2026-10-17 19:35:55 -04:00
ducoterra 140a343b92 phase: 189_chat_save_long_turns
Build and Push Containers / build-and-push-app (push) Successful in 3m4s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 189 final verification — all green**

- **Verified (all 4 tasks already implemented):** A1 caps in `app/schemas.py` — `sources`/`related`/`tools` 200, `suggestions` 100, `segments` 450; `text`/`thinking` 600 000, `error`/`image` 500, `answer_text` 50, `messages` 500, 8 M budget all UNCHANGED; rationale docstrings rewritten to `merge_cited_docs`/`agent_max_rounds`. A2 `shareCurrentChat` linked branch upserts (PUT, 404→create+re-link) before `POST /{id}/share`, aborts on failure (banner, no token); unlinked branch byte-identical. A3 `savePending` + `finally` re-run. A4 `saveFailedSticky` + one 5 s `setTimeout` retry on network/5xx only, 4xx never retries.
- `uv run pytest --cov=app --cov-report=term-missing` → **4609 passed, 1 warning, EXIT=0**, `TOTAL 99%`, `app/schemas.py 100%` (>90%).
- `uv run pytest tests/e2e/test_chat_save_long_turns.py -v --no-cov` → **4 passed, EXIT=0** (isolation).
- Regression green & **unedited** (`git status` clean): `test_chat_save_payload_limits.py` 2, `test_chat_history.py` 5, `test_share_chat.py` 4. Extra blast-radius sweep (share/save/persist/segments surface, all EXIT=0): `test_save_share_ux`, `test_stale_saved_chats`, `test_history_pagination`, `test_history_copy`, `test_bottom_chat_actions`, `test_reasoning_bubble`, `test_tool_budget_telemetry`, `test_verbatim_show_text`, `test_smoke`. No leaked `saved_chats` rows (`save-long-%` = 0).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- **Completion criteria:** caps ✓ · incident payload 22/33/51 → 201/200 anonymously + 201 tools/451 segments/101 suggestions → 422 naming the offender, row unchanged ✓ · share-saves-first + abort + unlinked byte-identical ✓ · overlap re-run + sticky note + one-shot retry ✓ · phase E2E in isolation + 3 regression suites unedited ✓ · full suite / coverage / lint / types ✓ · `git diff --stat` = `app/schemas.py`, `frontend/assets/app.js`, 3 test files, `README.md` + new E2E & phase files (no `app/api/chats.py`, no migration, no new dependency) ✓ · commit + `todo/`→`complete/` move left to the harness (rules forbid git index writes).
- **Docs:** full README review (all 2 258 lines) + machine cross-check — all 85 settings documented in the Configuration reference, all 34 `.env.example` vars documented, documented defaults match `app/config.py`. Fixed 2 inconsistencies: (1) the 117-char prose line task 04 left when reflowing the Saved-chats pager paragraph; (2) the phase-189 sentence implying the 8 M total bounds the widened arrays — now states precisely that the total counts `text`/`thinking`/verbatim-block text only, and that the arrays are bounded by fixed item COUNT (matches `_require_within_total_budget`). AGENTS.md verified consistent — it enumerates no save contract or list cap (rule 5's phase-166 in-flight-turn clause untouched), so **no AGENTS.md change** per the phase mandate.
- **Notable/deviation:** the wider caps raise the *theoretical* anonymous-POST worst case (arrays are outside the 8 M char budget); left as designed (A1 fixes every other bound), documented honestly in README — realistic payloads stay ~10 MB, so the existing `client_max_body_size ≥ 16 MB` guidance still holds. No live-infra changes were made (data-repair of the two stale shared rows stays owner action).
- **Next pending phase:** none — `189_chat_save_long_turns` is the last entry in `.agents/phases/todo/`.
2026-10-17 07:36:28 -04:00
ducoterra f20d40e4fe phase: 188_chunks_hnsw_index
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
2026-10-17 03:18:23 -04:00
ducoterra c054b47df5 phase: 187_sync_peak_memory_line
Build and Push Containers / build-and-push-app (push) Successful in 1m47s
Build and Push Containers / build-and-push-db (push) Successful in 13s
**Phase 187 — final verification pass: all criteria met, repo 100% green.**

Verified (all four tasks already in `complete/`)
- `app/core/memory.py`: `rss_mb` / `rss_peak_mb` / `cgroup_peak_mb` / `fmt_mb`, pure stdlib, never-raise, **100%** coverage, 24 branch tests.
- Lifecycle lines append-only: `sync: started … rss_mb=`, all three terminals `… rss_mb= rss_peak_mb=` (failed = sanitized one-liner beside the surviving `logger.exception`), `import: summary … rss_mb=`; prefixes byte-identical.
- `GET /api/sync/status`: additive `rss_mb`/`rss_peak_mb` (`int|null`), keyed off the ONE `_sample_rss` (run start, phase transitions, forced terminals, 5 s-throttled hook), `peak >= current` invariant, anonymous 403.
- `sources.js` `fmtSyncPeakClause`: ` · peak <N> MB` trailing the counts line / "Sync cancelled", nothing when null, no new control/live region.

Gates (exact commands)
- `uv run pytest --cov=app --cov-report=term-missing` → **4570 passed**, TOTAL **99%**, `app/api/sync.py` **100%**, `app/core/memory.py` **100%**
- `uv run ruff check .` → clean · `uv run pyright` → **0 errors**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` → **5 passed in isolation** (ran twice)
- All 27 touched E2E families re-run **in isolation**, all green (progress_box 8, cancel 3, status_freshness 4, summary_visibility 3, sync_button 3, session_reset 4, git_freeze 2, git_log_stream_dates 5, PDF/image suites, …)

Completion criteria: every checkbox ✅ except the commit + phase-dir move, which belong to the harness.

Documentation
- README reviewed in full (pages, commands, config tables cross-checked against `app/config.py` — 85 env rows, all defaults match; phase-187 ops section, greps, process-vs-cgroup caveat): consistent, no changes needed.
- `AGENTS.md`: **fixed one pre-existing inconsistency** — rule 6 claimed the no-CDN test covers "the index page"; it actually parametrizes 10 pages (all but `/doc-edit.html`).

Notable
- Two failures early in the pass were self-inflicted by running the E2E module concurrently with the full suite on the shared Postgres (TRUNCATE deadlock + one order-dependent `test_overview` flake); both vanish under the documented serial isolation — no code defect, no change made.
- **PLAN §9 still needs the owner's reconciliation** (phase 187 adds the sync memory fields + two status keys; PLAN is never agent-edited — AGENTS rule 10 carries the spec). `TODO.md` holds an unrelated owner note, left untouched.

Next pending phase: `188_chunks_hnsw_index`.
2026-10-16 15:22:51 -04:00
ducoterra 644f7de696 phase: 186_git_log_streaming_dates
Build and Push Containers / build-and-push-app (push) Successful in 1m42s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Final report — phase 186 `git_log_streaming_dates`

**State found:** all three task files were already implemented and in `.agents/phases/complete/186_git_log_streaming_dates/`, so this run was the final verification + documentation pass.

### Verification (all green)
| Gate | Result |
|---|---|
| `pytest --cov=app --cov-report=term-missing` | **4531 passed, exit 0** · coverage `TOTAL 6872 stmts → 99%` (gate >90%) |
| `tests/e2e/test_git_log_stream_dates.py -v --no-cov` (isolation) | **5 passed** |
| `ruff check .` | All checks passed |
| `pyright` | 0 errors |

Implementation shape confirmed against the phase contracts: `_parse_commit_dates(str | Iterable[str])` (existing string pins untouched — only 3 non-test lines removed from `test_git_sync.py`), `stream_git_lines` (`Popen` + daemon stderr drain + per-line stdout + terminate-on-abandon, `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream with byte-identical dates and the fail-soft funnel unchanged, `async_sync` propagating the `to_thread` errors to the existing `except`. A11 holds: two `subprocess.` sites, both in `scripts/git_sync.py`, both off-loop.

### Documentation-consistency pass
Verified (clean): all ~100 `BOR_*` README defaults against a bare `Settings()` (the local `.env` produced false positives — none real), `.env.example` completeness, every module/route/page/test path referenced, `compose.yaml`/`Containerfile`/`entrypoint.sh` claims, E2E port conventions (8901 / 8123), palette count (17), chip cap (90), token format, payload limits, retrieval battery + audit commands.

Fixed (both files, prose only):
- **Page-naming drift** — README/AGENTS called the sync/catalog page "the Sources page" in 5 places, contradicting the README's own convention (and `frontend/index.html`): the **Sync sources** button and the document catalogue are the **RAG** page (`/sources.html`); "Sources" is `/git-sources.html`. Now `RAG page (/sources.html)` in README:186, 376, 418, 682, 713, 745, 1006, 1057 and AGENTS:19, 38, 183.
- **Phase-186 AST-pin wording** — "fails if a third spawn appears anywhere in `app/` or `scripts/`" overstated it (the pin *enumerates* the spawns, including the non-git Valkey probe in `app/core/caching.py`); rewritten to describe the enumeration.
- Moved `00_phase.md` into `complete/186_git_log_streaming_dates/` (AGENTS rule 2 moves the whole directory); `todo/` now starts at 187.

⚠️ **One mistake to disclose:** while tidying line wraps I ran `git checkout README.md`, which discarded task 03's uncommitted phase-186 README bullet. I restored it verbatim from the full `git diff` captured earlier in this session (plus the two corrections above) and re-verified the section renders as intended — no other README content was uncommitted, and AGENTS.md was untouched by the checkout. A blanket line-length rewrap also reflowed 4 unrelated lines; all four were restored to their exact original text (`git diff` now shows only intended changes).

Nothing was committed or staged — the working tree is left for the harness: modified `scripts/git_sync.py`, `app/api/sync.py`, `app/rag/git_sources.py`, `tests/unit/test_git_sync.py`, `tests/integration/test_git_file_dates.py`, `README.md`, `AGENTS.md`; new `tests/e2e/test_git_log_stream_dates.py`, `.agents/phases/complete/186_git_log_streaming_dates/`, `.agents/reports/186_git_log_streaming_dates/`.
2026-10-16 07:50:16 -04:00
ducoterra 632fb88e43 phase: 185_sync_defer_content_fast_path
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All gates green. Report:

**Implemented / verified (phase 185)**
- `_doc_row_stmt` is the ONE deferral helper for all four fast-path row loads; only `Document.content` is deferred, no `raiseload`/`noload` (LOCKED A1/A2) — pinned by `tests/unit/test_importer_defer_content.py`.
- `tests/integration/test_import_fast_path_columns.py` (5), `tests/e2e/test_fast_path_deferred_content.py` (3, isolation) green; added 2 constraint pins to `test_import_session_retention.py` (summary-NULL backfill survives the cadence; both prune guards cost the same statement budget).
- **Fixed 6 pre-existing E2E failures** (all reproduce at HEAD with this phase's importer reverted): stale pins from phase 168 `Settings` nav link (`test_nav_rename_sources`), phase 182 `verbatim.js` script count (`test_steering`), phase 175 `/api/chats?limit=…` pagination breaking `endswith` counting (`test_navbar_refresh`), phase-167 lazy-drill race in the breadcrumb read (`test_modal_back_button`), nav-length tab budget (`test_suggestion_chips`).
- **Fixed a genuine flake** in `test_source_chip_quality` (passed/failed across identical runs): `ORDER BY embedding <=> q` is not a total order, so equal-cosine fixtures get a different RRF rank each run. Pinned it as an explicit tie-group (`_assert_ranked` / `_assert_related_pair`) instead of an exact order.

**Tests / lint (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4519 passed**, `app/rag/importer.py` **99%**, `TOTAL` **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors.
- Full E2E sweep, one pytest process per file (the README's sanctioned loop) → **167/167 files OK, 0 FAIL** (re-run after the fixes).

**Completion criteria**: helper shared by 4 loads ✅ · column-list + 1-SELECT + backfill pins ✅ · `test_importer_unchanged_fast_path.py` 24 passed, zero expectation edits ✅ · full suite + coverage + lint/types ✅ · phase E2E in isolation ✅ · docs accurate ✅ · commit left to the harness (no git commands run).

**Documentation**: README + AGENTS.md already carry the phase-185 rule; full pass found and fixed no doc inconsistencies (env vars: all 85 `Settings` fields documented in `.env.example` + README, no ghosts; every `app/… tests/… scripts/…` path referenced in both files exists).

**Deviations**: (1) the ranking non-determinism behind the chip-quality flake is real in `app/rag/retriever.py` — a SQL tie-break would cost pgvector index usage, so I kept the fix test-side; a follow-up phase should make vector-candidate ranking a total order. (2) Two tracked E2E screenshots were regenerated by the sweep (artifacts, left modified).

**Next pending phase**: `186_git_log_streaming_dates`.
2026-10-16 04:54:01 -04:00
ducoterra 065304b52f phase: 184_sync_bounded_session
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green; no code changes were needed — the prior failure was a missing final report, not a test failure.

**Verified / implemented (phase 184 final pass)**
- All 4 tasks present in `.agents/phases/complete/184_sync_bounded_session/`; confirmed their output in the worktree: `sync_session_reset_files=200` / `sync_prune_batch=500` on `Settings` (+ non-negative validators), both `SettingMeta` rows under `"Import & sources"`, `.env.example` + `README.md` entries.
- `import_sources` expunges at **exactly two** sites (per-file loop top L972, source boundary L1012) — grep proves no third `expunge`/`expire_all` anywhere in `app/`; `app/db.py:30` untouched.
- `_prune` uses `load_only(id, source, path, is_image, image_path, page_count).order_by(Document.id)` + per-batch commits, with the phase-122/126 guards and `changed_paths` intact.

**Tests / lint / coverage (all 100% green)**
- `uv run pytest --cov=app --cov-report=term-missing` → **4506 passed**, `app/` TOTAL **99%** (>90%).
- `uv run pytest tests/integration/test_import_session_retention.py -v --no-cov` → **12 passed**.
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` → **4 passed in isolation**.
- Regression sweep of neighbouring sync suites (each isolated): `test_sync_changed_only` 3, `test_sync_summary_visibility` 3, `test_prune_cascade_cleanup` 3, `test_sync_git_freeze` 2 — all passed.
- `uv run ruff check .` → All checks passed. `uv run pyright` → 0 errors, 0 warnings.

**Completion criteria**
1. Settings + CATALOG + `.env.example` + README; `test_settings_catalog.py` → ✅ green
2. Retention test green with the post-fix bound and both measured numbers in the docstring (pre-fix 12-row monotone staircase / ≈265 MB extrapolated; post-fix ends at 0, add-path peak 14 vs 21→49) → ✅
3. Expunge only at the two documented-safe points; light-row batched prune with guards intact → ✅
4. Every existing importer/sync/removal test green with unchanged expectations → ✅
5. Full suite green, coverage >90%, ruff + pyright clean → ✅
6. Phase E2E green in isolation → ✅
7. README + AGENTS accurate/consistent → ✅
8. Atomic commit — left to the harness (I ran no git write commands).

**Documentation** — Full README (1886 lines) + AGENTS.md (189 lines) review: **verified consistent — no changes needed.** Programmatic cross-checks: every `Settings` field appears in `.env.example` and in the README config reference; every `.env.example` var is documented in that reference; every `tests/…` and `app/…` path cited in README/AGENTS exists; all 10 README page URLs are registered in `app/main.py`; documented E2E port vars (`E2E_APP_PORT_SYNCRESET[_ZERO]`) exist in the test. The only reference-var without a `Settings` field is `BOR_THEME`, which the README explicitly documents as retired (correct as written).

**Notable decisions/deviations** — None. No code, test, or doc edits were made this pass; changes remain entirely in the working tree for the harness to stage. `00_phase.md` still sits in `todo/` while tasks 01–04 are in `complete/` — harness-owned, so I left it untouched.

**Next pending phase:** `185_sync_defer_content_fast_path`.
2026-10-15 23:19:27 -04:00
ducoterra 7b5ea7c2ee phase: 183_sync_passive_delete_cascade
Build and Push Containers / build-and-push-app (push) Successful in 2m7s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green. Final state verified.

**Implemented / verified**
- Task 03's work was **never actually done** (its report was a stub, `tests/e2e/test_prune_cascade_cleanup.py` did not exist) → wrote it: 3 tests — 9-file local source synced from the Sources page (`9 added`), 6 files deleted upstream, re-sync (`0 added · 3 unchanged · 6 pruned`), documents vanished from the tree + `GET /api/documents/content` 404 + total 3, **0 orphan chunk rows** via the test's own SQL, survivor chunk total unchanged; partial-prune chat turn (grounded, non-deflected, `query_log` names the survivor and no pruned doc); total source removal (`DELETE /api/git-sources/{id}`) wipes every doc+chunk of the source with 0 orphans, sibling source untouched. Mock-only (`E2E_REAL_LLM=1` skipped), own port 8199, truncate before/after.
- Tasks 01/02 re-verified: `passive_deletes=True` + mapper pin, probe flipped to `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, `_prune`/`delete_git_source` docstrings.
- **Drift guard proven non-tautological by my own one-off run**: scratch DB (`alembic upgrade head`) → FK `confdeltype='c'`, guard PASSES, CASCADE delete leaves 0 chunks; FK recreated `ON DELETE NO ACTION` → guard FAILS verbatim and a `passive_deletes` delete raises `ForeignKeyViolation` (never silent orphans). Scratch DB dropped; shared test DB untouched (`'c'`).

**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4488 passed**, `app/` TOTAL **99%** (>90%)
- `uv run pytest tests/integration/test_prune_cascade_statements.py -v` → 4 passed
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` → **3 passed in isolation**; `test_source_removal_cleanup.py` 6 passed, `test_force_push_resync.py` 2 passed (both isolated)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors

**Completion criteria**: passive_deletes+pin ✅ · probe post-fix ✅ · drift guard green + proven-failing variant ✅ · full suite/coverage/lint ✅ · E2E in isolation ✅ · docs consistent ✅ · commit = harness-only (per pipeline rule I ran no git write commands).

**Docs**: fixed 4 pre-existing README inconsistencies — per-module E2E ports (claim of "one pair of fixed ports" predates phase 79), debugpy `launch.json` snippet vs the shipped file (`name`/`remoteRoot`), and the page-naming wobble (navbar labels `/sources.html` **RAG**, `/git-sources.html` **Sources** — the Sync button's home is now named correctly in 4 places); added the phase-183 E2E pointer. AGENTS.md verified accurate (rule 12 matches the delete-site audit).

**Deviation**: none from the locked decisions; the E2E is a behaviour pin (byte-identical owner-visible behaviour), the statement-count probe remains the memory teeth, as phase 183 D1/A4 specify.

**Next pending phase**: `184_sync_bounded_session`.
2026-10-15 16:00:28 -04:00
ducoterra b34b8ca672 phase: 182_verbatim_show_text
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**

- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
2026-10-15 08:33:12 -04:00
ducoterra 814a400e6d phase: 181_svg_document_preview
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Build and Push Containers / build-and-push-db (push) Successful in 15s
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`![alt](/api/documents/<id>/image)` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
2026-10-14 22:54:43 -04:00
ducoterra 6880bd0866 phase: 180_markdown_inline_links_images
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 15s
All verification complete. Final state summary:

**Report — phase 180 final verification pass**
- Verified (all already implemented in working tree): `inline()` chain = code→strong→em→linked-image→image→link→headings→lists in `frontend/assets/markdown.js`; `allowLinkUrl`/`allowImageUrl` allowlists (A5), shared `mdAnchorOpen` (target/rel only for absolute http(s)); `####`→`<h5>`, `#####`/`######`→`<h6>`, `#`/`##`→`<h3>`/`###`→`<h4>` unchanged; `.md-link`/`.md-img`/h5-h6 scale in `styles.css` (palette vars only). All surfaces (chat, thinking, restore, viewer, modal) route through the one renderer (pinned in `app.js`/`document.js`).
- Completion criteria:
  - `[text](url)`→`a.md-link`, `javascript:`/`data:` inert — ✅ (node probe + unit + E2E)
  - `![alt](src)`→`img.md-img loading="lazy"` with `alt`; bad src inert — ✅
  - `####`/`#####`/`######` ladder; shallow rungs unchanged — ✅
  - Fenced link/image/`####` stay code; cell-level link renders in `.md-table` — ✅ (node probe)
  - Byte-identity fixture + phase-44/138 suites pass unedited (git-verified untouched; 6 + 3 E2E pass in isolation) — ✅
  - `uv run pytest --cov=app --cov-report=term-missing`: **4198 passed, TOTAL 99%**; `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov`: **4 passed**; `uv run ruff check .`: clean; `uv run pyright`: **0 errors** — ✅
- Docs: full README/AGENTS.md review vs the whole project — env-example↔README table complete, page routes match `app/main.py`, AGENTS quick-ref artifacts exist. Fixed 1 inconsistency: README understated the image allowlist (impl per task 02 accepts first segments ≤2 chars, `^.{0,2}/`, not only `./`/`../`); phase-180 paragraphs otherwise accurate.
- Notable: image allowlist `.{0,2}/` is broader than the overview's `^\.{1,2}/` but is what task 02 explicitly specifies and the docstring documents — not a deviation from the tasks; SVG sizing deliberately deferred to phase 181.
- Next pending phase: `181_svg_document_preview`.
2026-10-14 16:03:36 -04:00
ducoterra 3d82a72b6f phase: 179_git_clone_server_freeze
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All gates green; working tree holds exactly the phase's changes for the harness to commit.

## Phase 179 — Final verification report

- **Verified (all 3 tasks already complete in `complete/`)**: repro + CONFIRMED verdict, `asyncio.to_thread` fix, dedicated E2E suite.
- `app/api/sync.py::_run_sync`: both git ops (`clone_or_pull`, `file_commit_dates`) now `await asyncio.to_thread(...)`; phase-134 boundary note + module docstring document worker-thread-runs-to-completion and the A3 DB-calls-stay-on-loop note.
- **Tests / lint / coverage** (full suite 100% green):
  - `uv run pytest tests/integration/test_sync_git_freeze.py -v` → **4 passed** (post-fix probe <1.5 s during 3 s stub)
  - `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` → **2 passed in 25.65 s in isolation** (wall guard 120 s holds)
  - `uv run pytest --cov=app --cov-report=term-missing` → **4173 passed**, `app/` coverage **99%** (`app/api/sync.py` 100%)
  - `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**
- **Completion criteria**: ① integration probe green + verdict (CONFIRMED, mechanism, measured 3.04 s→0.003 s) in docstring — **met**; ② off-loop git ops + boundary/A3 notes — **met**; ③ full suite green, coverage 99% >90%, ruff+pyright clean — **met**; ④ E2E green in isolation (responsive-poll + cancel-during-clone contracts) — **met**; ⑤ atomic commit + move to `complete/` — **harness-owned** (pipeline rule overrides per-phase self-commit; all changes staged-ready in the working tree).
- **Documentation**: full README (1526 l) + AGENTS.md review — phase-179 bullets present and accurate (verified against fixture constants, `.env.example` fully covered by the config reference, pages table matches `app/main.py` shell routes, quick-reference commands/artifacts exist); **consistent — no further changes needed**.
- **Notable**: task-03 pipeline sessions died on context overflow but their harness validation passed; this pass independently re-ran every gate to confirm. No live infra touched; no git index commands run.
- **Next pending phase**: `180_markdown_inline_links_images`.
2026-10-14 13:06:39 -04:00
ducoterra 2b57180998 chore(repo): remove stray image.png screenshot artifact
Build and Push Containers / build-and-push-app (push) Successful in 17s
Build and Push Containers / build-and-push-db (push) Successful in 12s
The root-level image.png was committed accidentally during phase
160 (tool budget telemetry) as a bug-report screenshot. It is not
served or referenced as an asset by the app; the remaining
"image.png" mentions in tests and comments only describe the
owner's screenshot in prose, so removing the file is safe.

LLM-Generated: true
2026-10-14 10:41:04 -04:00
ducoterra 76887639f3 docs(phases): add phase 182 (verbatim file display)
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Answers to "show me my pi.dev config" re-type file contents from context,
so the code block in the answer can differ from the indexed file by a
character and cannot be trusted for copy-paste — the owner opens the
source bubble and ignores the answer.

Phase 182 makes the server the only party that supplies bytes:

- show_text(path, offset?, limit?), the display sibling of show_image,
  slices documents.content and hands the same bytes back to the model
  with a "do not retype" instruction
- inline ```bor source/path 12-40 locator fences place a resolved block
  anywhere in the prose; the model's fence body is never painted, and an
  unresolvable locator says so instead of showing guessed text
- the bytes ride inline in the done frame's additive answer_text array
  and persist in the saved chat, so restore and the shared page match
- line numbers are always the server's resolved extent (overshoot
  clamps, past-the-end refuses to the model), which removes the
  "did the model invent those numbers?" problem entirely
- a Copy button on every answer code block, a VERBATIM badge only on
  server-resolved ones
- gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the
  flag off every frame, record and byte budget is exactly pre-phase

Overview plus eight tasks, each with its own test gate; the phase E2E
runs in isolation and asserts the clipboard content equals the stored
bytes. Run 180 first — both touch markdown.js.

LLM-Generated: true
2026-10-14 10:33:06 -04:00
ducoterra 50658580e9 chore(agent): phase roadmap from TODO.md, 2 phases
Convert the 3 open TODO.md items into sequential phases appended to the
existing pipeline (Protocol B, numbered after 179):

- 180_markdown_inline_links_images: the shared escape-first renderer
  gains the [text](url) link transform, the ![alt](src) image
  transform, and the ####-###### heading levels, with CSS and the
  dedicated Playwright suite.
- 181_svg_document_preview: SVGs index as vector image documents
  (text extracted from title/desc/text, no vision call), are served as
  sandboxed image/svg+xml, and preview as graphics instead of raw XML.

TODO.md is cleared; its items now live in .agents/phases/.

LLM-Generated: true
2026-10-14 09:54:01 -04:00
ducoterra dad54a2854 chore(agent): phase roadmap from TODO.md (179 git clone server freeze)
TODO L3 (a long git clone seems to freeze the server) converted to phase 179:
deterministic in-process repro + verdict, the asyncio.to_thread fix for the
sync's git ops, and the dedicated Playwright E2E suite. TODO.md cleared
(items now live in .agents/phases/).

LLM-Generated: true
2026-10-12 09:27:58 -04:00
ducoterra 149e753952 phase: 178_split_thinking_levels
Build and Push Containers / build-and-push-app (push) Successful in 1m53s
Build and Push Containers / build-and-push-db (push) Successful in 14s
## Phase 178 — final verification pass: 100% green

**Verified (all 5 tasks complete in `complete/178_split_thinking_levels/`):**
- `config.py` rename (`llm_chat_thinking_level` default `low`) + new `llm_summary_thinking_level` (default `""`); two `CATALOG` rows; `.env.example` replaced; no stale old-key references (only intentional negative pins in tests)
- Wire split: `chat_stream()` → chat level, one-shot `chat()` → summary level (absent `extra_body` when empty); per-turn line reads chat level only

**Gates:**
- `uv run pytest -k "config and thinking" -v` → 7 passed (defaults + env overrides both knobs)
- `uv run pytest` (unit + integration) → 4170 tests, exit 0; `--cov=app` → **99%** (>90% ✓)
- `uv run ruff check .` clean; `uv run pyright` clean
- `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` → 3 passed in isolation
- `test_admin_settings_panel.py` + `test_admin_settings_layout.py` → 17 passed in isolation

**Completion criteria:** all 6 test/doc criteria ✅ (admin GET lists both keys, old key gone — integration + E2E pins); commit + phase-dir move = harness's job.

**Documentation:** README.md — both new env vars correctly documented in the LLM (aipi) table with defaults/wire-mapping/off-rule, defaults match `config.py`, all `.env.example` vars covered: verified consistent, no changes needed. AGENTS.md — all referenced files/commands verified to exist, no phase-178 impact (no UI code change, A4): verified consistent, no changes needed.

**Decisions:** none required — no live changes made (E2E booted its own app subprocess against the existing DB).

**Next pending phase:** none — 178 is the last phase in `todo/`.
2026-10-11 23:38:18 -04:00
ducoterra c6223c93dc chore(agent): phase roadmap from TODO.md (178 split thinking levels)
LLM-Generated: true
2026-10-11 20:56:16 -04:00
ducoterra 36eddce672 phase: 177_themed_favicon
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 12s
Everything is verified. Final report:

## Phase 177 — Themed favicon: completion pass (all tasks pre-done, verified end-to-end)

**Verified (no code fixes needed — implementation matched the design D1–D4 exactly):**
- `theming.favicon_svg()` pure renderer + golden byte-identical pin; `GET/HEAD /favicon.svg` route (no-cache, `image/svg+xml`, built-in 200 fallback on DB blip, routes-first before static mount); 5 templates → bare `href="/favicon.svg"`, static file deleted (Containerfile cp removed); Theme-tab `data:` live preview with 4 restore points; JS↔server mirror pins.

**Tests / lint / coverage (all green):**
- `uv run pytest --cov=app --cov-report=term-missing` → 4161 passed, `app/` 99% (>90% gate)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
- `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov` (isolated) → 6 passed
- `test_theme_save_and_coverage.py` (92) + `test_theme_semantic_completion.py` (93), isolated → 3 + 8 passed

**Completion criteria:** all met — default golden bytes/headers; themed roles + `#bor-theme` non-regression; DB-failure 200 fallback; 5 bare links + `/assets/favicon.svg` 404 + browser rendering; live-preview data: repaint + Save/Reset/re-show restore; gates green; E2E suites green in isolation. Commit: harness commits atomically (overrides AGENTS rule 8 — no git commands run).

**Documentation:** README.md verified consistent in full (the phase's favicon paragraph is accurate; all commands, env vars, routes, ports, scripts cross-checked) — no changes needed. AGENTS.md verified consistent — no changes needed.

**Notable:** `image-1.png` (owner's TODO screenshot, referenced by phase 176's overview) sits untracked at the repo root — pre-existing, explicitly out of scope per the phase-176 report; left untouched.

**Next pending phase:** none — 177 was the last; TODO.md is cleared.
2026-10-11 15:33:39 -04:00
ducoterra dd119d534a phase: 176_mobile_settings_value_width
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 14s
All documentation claims verify against the code. Everything is green — final verification complete.

## Phase 176 — Final Verification Report

- **Verified (all 3 tasks in `complete/`):** the mobile Value-column fix is in the working tree — `frontend/index.html` L1602 carries `class="settings-value-th"`; `frontend/assets/styles.css` has exactly two new rules inside the `@media (max-width: 640px)` block: `.settings-table { min-width: 720px }` and `.settings-value-cell, .settings-value-th { min-width: calc(13rem + 2rem) }`; `settings.js` + `app/` byte-unchanged (diff = index.html + styles.css + unit test only).
- **Documented deviation (flagged in task-03 report, not silent):** spec's `min-width: 13rem` on the border-box cell left the control at 176px (< the 192px A3 pin); executor corrected the cell to `calc(13rem + 2rem)` so the control (100% of content) lands at exactly 13rem ≈ 208px — the A2 lock intent preserved, unit pin + CSS comment updated to match.
- **Tests/lint/coverage (exact commands, all green):** `uv run pytest --cov=app` → **4138 passed**, `app/` coverage **99%** (>90% ✓); `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings.
- **Completion criteria:** 375px controls ≥ 192px — ✅ (E2E `test_mobile_value_inputs_are_usable_width`); card scrolls / page doesn't overflow at 375px — ✅ (`test_mobile_table_still_scrolls_and_page_does_not_overflow`); 1440px media query inert — ✅ (`test_desktop_value_width_is_unchanged`); tab order + zero external requests — ✅; phase E2E isolated — ✅ 4/4; phase 171 isolated — ✅ 5/5; phase 168 isolated — ✅ 12/12; unit pins — ✅ 17/17; commit/move — the harness commits; phase dir already in `complete/`.
- **Documentation:** README.md and AGENTS.md reviewed in full — every claim spot-checked against code (E2E ports 8123/8901, defaults `BOR_COMPRESS_MIN_BYTES=860` / `BOR_LOG_LARGE_RESPONSE_BYTES=65536`, audit budget 262144, `.cell-clip`, `.bubble:empty`, `min(70vh, 64rem)`, aria-pressed toggle, CDN integration test, `.gitignore` entries, all referenced scripts/routes) — verified consistent, no changes needed.
- **Notable:** `image-1.png` (owner's TODO screenshot, referenced by the phase overview) sits untracked in the tree — left alone; not part of the phase's commit scope.
- **Next pending phase:** `177_themed_favicon`.
2026-10-11 13:23:04 -04:00
ducoterra 2a5df7cb31 chore(agent): phase roadmap from TODO.md (177 themed favicon)
LLM-Generated: true
2026-10-11 11:44:07 -04:00
ducoterra bd271a6f52 chore(agent): phase roadmap from TODO.md (176 mobile settings value width)
The single TODO item (mobile settings Value inputs impossibly small) becomes phase 176_mobile_settings_value_width: 3 tasks (CSS fix in the <=640px block, unit pins, dedicated E2E suite) with owner-locked A1-A3. Protocol B append - PLAN.md and AGENTS.md untouched.

LLM-Generated: true
2026-10-11 10:28:54 -04:00
ducoterra 375169bbfc phase: 175_history_pagination
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All criteria verified. This pass found no defects — the phase was fully implemented by prior task runs, and every gate is green.

**Phase 175 final verification pass — report**

- Verified tasks 01–03 implementation in-tree: `list_chats` `?limit=` (25, 1–100) / `?offset=` + `func.count` `total`; `SavedChatList.total` required; `#history-pager` below `#history-table-wrap` (hidden default, aria-labels, aria-hidden SVGs); `history.js` `PAGE_SIZE=25`/offset state, parameterized fetch, `updatePager`, refresh-keeps-current-page; CSS pill union + ≤640px block. No code changes were needed.
- Full suite: `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99%** (>90% gate).
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors, 0 warnings.
- Phase E2E in isolation: `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` → **9 passed** (page 1-of-2 / 2-of-2 walk, replace-not-append, refresh-on-page-2, delete-to-empty + single-page hide, anonymous no-fetch, 375px mobile, a11y/no-CDN).
- Regression set, each in isolation: test_chat_history 5✓, test_history_page_width 5✓, test_share_chat 4✓, test_stale_saved_chats 3✓, test_history_copy 1✓; integration `test_chats_api.py` green (incl. updated `test_list_empty` two-key pin and all five new pagination pins).
- Completion criteria: all nine boxes satisfied (API slice/422/honest-total pins ✓; E2E walk + status lines ✓; refresh/delete/hide ✓; anonymous gate ✓; mobile ✓; gates ✓; E2E + regression ✓; docs ✓; commit/phase-move is the harness's).
- Documentation: README.md reviewed in full — the saved-chats section already carries the `?limit=`/`?offset=`/`total` contract from this phase; all 31 `.env.example` vars documented; AGENTS.md verified accurate. **Consistent — no changes needed.**
- Deviations: none. No defects found in previously completed work.
- Next pending phase: none — 175 is the only entry in `todo/`.
2026-10-11 04:12:53 -04:00
ducoterra 938d6cf09d phase: 174_git_sources_card_rhythm
Build and Push Containers / build-and-push-app (push) Successful in 1m51s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All checks pass. Both docs verified consistent. Final state — no live-infrastructure changes were needed (pure CSS fix; verification ran against the already-running local dev DB).

**Phase 174 final verification pass — report**

- **Verified (no new work needed; both tasks already complete in `complete/`):** the single `#git-sources-content { display:flex; flex-direction:column; gap:1.25rem }` rule in `frontend/assets/styles.css` (with the phase-174 root-cause comment), `.git-sources-shell` untouched, the 4-test unit pin module, and the 4-test Playwright suite.
- **Full suite:** `uv run pytest --cov=app --cov-report=term-missing` → **4116 passed, 99% `app/` coverage** (gate >90% ✓).
- **Lint/types:** `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings.
- **Dedicated E2E in isolation:** `uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov` → **4 passed** (1440px + 375px bounding-box rhythm pins 20±2px, hidden-children + revealed-joins-rhythm, anonymous gate, a11y/no-CDN).
- **Regression set, each in isolation:** `test_git_sources_admin.py` **6 passed**, `test_archive_upload_sources.py` **5 passed**, `test_git_source_dates.py` **4 passed**.
- **Completion criteria:** cards stack with uniform ≈20px gap at both viewports ✓; hidden children stay hidden / gate+content flip intact ✓; ≤640px forms column-stack + table scrolls inside its card (no page-level h-scroll) ✓; full suite + coverage + ruff + pyright ✓; dedicated + regression E2E suites ✓.
- **Documentation:** README.md reviewed in full (every `.env.example` var documented; all documented routes/pages/endpoints exist; every config default cross-checked against `app/config.py`; payload-limit table, ports 8123/8901, Containerfile/compose/entrypoint claims all match) — **verified consistent, no changes needed**. AGENTS.md verified (all referenced files/sections exist: PLAN §2/§7.1/§7.4/§9, `response_budget.py`, `debugging.py` conditional import, `.agents/` tracked + gitignore rules, `commit.gpgsign=false`, phase-pinned CSS classes) — **verified consistent, no changes needed**.
- **Notable:** commit + phase-dir move are handled by the harness per pipeline rules (no git history/index touched by this pass); no live-infrastructure changes were required.
- **Next pending phase:** `175_history_pagination` (in `todo/`).
2026-10-11 01:49:49 -04:00
ducoterra da441ea799 chore(agent): phase roadmap from TODO.md, phase 175 (history pagination)
LLM-Generated: true
2026-10-11 00:19:04 -04:00
ducoterra a86caec821 phase: 173_docs_push_creds_split
Build and Push Containers / build-and-push-app (push) Successful in 1m41s
Build and Push Containers / build-and-push-db (push) Successful in 11s
All gates green. Final report:

**Phase 173 — final verification pass (all tasks already complete in `complete/`)**

- Verified the full implementation is in the working tree: `docs_user`/`docs_token` settings + catalogue rows (task 01), `docs_repo` PUT 422 guard + `docs_token` write-only pin (task 02), `run_git(env)` + `push_document(user, token)` credential env (task 03), `settings_migrate.py` wired into `create_app()` (task 04), E2E suite (task 05).
- **Completion criteria:**
  1. GET `/api/admin/settings` lists `docs_user` (plain) + `docs_token` (secret, `value=null`), token never in any body — ✅ `test_docs_token_is_write_only_masked_end_to_end` + E2E sentinel-absent-from-DOM
  2. PUT `docs_repo` with `://user:pass@` → 422 naming the fields; clean URL passes — ✅ integration + E2E UI-side
  3. Token set → `GIT_CURL_USERNAME`/`GIT_CURL_PASSWORD`/`GIT_TERMINAL_PROMPT=0` on every git step; empty → `env=None` — ✅ mocked-git unit tests
  4. Legacy env split once at startup, idempotent, one INFO line, `.env` untouched — ✅ `test_settings_migrate.py` (4 shapes) + `test_docs_push_migrate.py`
  5. Full suite green, coverage >90%, lint clean — ✅
  6. E2E isolated: `test_docs_push_creds.py` 4/4; regression set (panel + layout + response_to_docs) 21/21 — ✅
  7. Commit + phase move — harness's job (per instructions, not mine).
  8. README/AGENTS consistent — ✅ (below)
- Tests/lint: `uv run pytest` exit 0 (4112 passed) · `uv run pytest --cov=app --cov-report=term-missing` TOTAL 99% · `uv run ruff check .` clean · `uv run pyright` 0 errors
- Docs: README.md (incl. Docs-push config section, guard/migration paragraph, `.env.example` block) and AGENTS.md verified consistent in full; fixed two stale docstrings in `app/config.py` + `app/core/settings_catalog.py` (`is_set` → the API's actual `has_value` field).
- No live-infrastructure changes (no live step in this phase). No next pending phase — `todo/` held only 173.
2026-10-10 16:47:39 -04:00
ducoterra f27106a638 phase: 172_thinking_level
Build and Push Containers / build-and-push-app (push) Successful in 1m43s
Build and Push Containers / build-and-push-db (push) Successful in 12s
All verification complete. Final report:

**Phase 172 final verification pass — all tasks (01–05) were already complete; verified the whole phase, no code fixes needed.**

- **Verified implementation:** `Settings.llm_thinking_level: str = "low"` + `CATALOG` row (`LLM (aipi)` / "Thinking level", plain: not secret/restart/overridable-exempt) + `.env.example` line under `BOR_LLM_CHAT_MODEL` + README table row; `_thinking_extra_body()` → `reasoning_effort` via `extra_body` on `chat()` + `chat_stream()` when trimmed-non-empty, key absent (byte-identical) when empty, embeddings untouched; per-turn line gains `thinking=<value>` (`-` when empty) before `thinking_chars`.
- **Tests/lint/coverage (all green):** `uv run pytest` → 4084 passed · `uv run pytest --cov=app --cov-report=term-missing` → 4084 passed, `app/` **99%** (>90%) · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors, 0 warnings.
- **Completion criteria:** (1) `pytest -k "config and thinking"` 4 passed + `.env.example` grep ✓ (2) live `GET /api/admin/settings` curl shows the row (str, `low`, `default`) + integration PUT/DELETE round-trip ✓ (3) 9 mocked-client unit tests assert `extra_body` present/absent ✓ (4) log-line tests assert `low`/`high`/`-`/whitespace ✓ (5) suite + coverage + lint ✓ (6) `test_thinking_level.py` 3/3 in isolation; `test_admin_settings_panel.py` 12/12; `test_admin_settings_layout.py` 5/5 ✓ (7) atomic commit + phase move = the harness's job (task files already in `complete/`; I ran no git index/history commands).
- **Documentation:** README.md reviewed in full — verified consistent (env table complete vs `.env.example`, catalog restart/non-overridable claims re-verified at runtime); AGENTS.md reviewed in full — verified consistent (no rule affected, A5 = no UI change). No changes needed. **Flag (not fixable here):** PLAN.md §9's per-turn-line spec predates the phase-172 `thinking=` field; PLAN.md is off-limits per protocol.
- **Next pending phase:** none — `todo/` contains only phase 172 (the harness moves it on this pass's success).
2026-10-10 13:19:10 -04:00
ducoterra b21b05eaee chore(agent): phase roadmap from TODO.md, phase 172 (thinking level)
LLM-Generated: true
2026-10-10 10:28:46 -04:00
ducoterra e9ae6e2ba3 phase: 171_admin_settings_layout
Build and Push Containers / build-and-push-app (push) Successful in 1m34s
Build and Push Containers / build-and-push-db (push) Successful in 14s
Expected state: only `00_phase.md` remains in `todo/` (the harness moves it on success; I must not touch it). All verification is complete.

---

**Phase 171 final verification pass — report**

- **Verified (all 4 tasks complete, CSS-only diff confirmed byte-exact to contract):** `#settings-content` flex column + `gap: 1.25rem` (tasks 01); `.settings-group-row th` band `--brand-soft` + 700 + `--line` top hairline (02); `#settings-save:disabled` `cursor: default` (03); new phase-dedicated Playwright suite (04). Mobile media queries and `settings.js`/`app/` byte-untouched, as pinned.
- **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **4069 passed, app/ 99%** (>90%); `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**.
- **E2E (in isolation):** `test_admin_settings_layout.py` → **5 passed**; `test_admin_settings_panel.py` (phase 168) → **12 passed**; `test_wide_desktop_column.py` → **3 passed**.
- **Completion criteria:** search/table no-intersection + gapped action row at 1440/375 px ✅ · distinct group bands (bg ≠ card, weight 700, ≥1px top hairline) ✅ · disabled Save `cursor: default` with §7.4 dim + aria-busy lifecycle intact ✅ · full suite + coverage + lint ✅ · all three E2E suites isolated ✅ · atomic commit + dir move = **harness's job** (task files already in `complete/`; per protocol I left every change uncommitted in the working tree).
- **Documentation:** README.md (all 1487 lines) and AGENTS.md (all 80 lines) fully reviewed — every config default cross-checked vs `app/config.py`, all quick-reference commands verified against existing files, AGENTS.md rules/contracts verified vs code — **verified consistent, no changes needed** (phase 171 is CSS-only: no setup/command/env/API impact).
- **Notable:** one phase-168 test failed on the first isolated run solely because an unrelated `undercooked` dev server held port 8124 (the suite's `DISABLED_PORT`); re-ran with the suite's built-in `E2E_APP_PORT=8300` override — no code change, foreign process untouched. Post-run DB state clean (`app_settings` 0, `ui_settings` 0, e2e tokens 0).
- **Next pending phase:** none — phase 171 was the last `todo/` phase.
2026-10-10 04:45:34 -04:00
ducoterra c4f098d0c0 phase: 170_edit_bubble_sizing
Build and Push Containers / build-and-push-app (push) Successful in 1m42s
Build and Push Containers / build-and-push-db (push) Successful in 13s
All verification complete. Final report:

**Phase 170 — final verification pass (all 3 tasks were in `complete/`):**

**Verified implemented:**
- Editor renders inside the bubble: `bubble.replaceChildren(form)`, `display:none` rule deleted, `form.style.minWidth` seeded with pre-edit width, `restoreBubble()` (markdown + image re-prepend) before `closeEditor()` on Cancel/Escape, `closeEditor()` reordered before submit re-render
- Label: `editSaveBtn.textContent = "Send"` (app.js L1241)

**Gates (all 100% green):**
- `uv run pytest` → 4066 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 99% on `app/` (gate >90%)
- `uv run ruff check . && uv run pyright` → clean (0 errors, 0 warnings)
- `uv run pytest tests/e2e/test_edit_bubble_sizing.py -v --no-cov` (in isolation) → 6/6 passed
- Regression set in isolation: `test_edit_and_resend.py` 7✓, `test_retry_answer.py` 4✓, `test_failed_turn_retry.py` 4✓, `test_chat_persistence.py` 4✓

**Completion criteria:** in-bubble editor + width pin ✅ · "Send" label + cancel restore + trim re-ask + "Question too long" driver case ✅ · suite/coverage/lint ✅ · E2E + regression isolation ✅ · phase dir moved to `complete/` ✅ (atomic commit is the harness's to make — changes left in the working tree per protocol)

**Documentation:** README.md reviewed in full — edit section already aligned to phase 170 (in-bubble editor + "Send"), pages table matches actual routes, all 29 `.env.example` vars documented, setup commands accurate; AGENTS.md reviewed — all referenced files/commands exist. Both verified consistent — no changes needed.

**Notable decisions:** none — verification pass only, no code changes required.

**Next pending phase:** `171_admin_settings_layout`
2026-10-10 03:25:40 -04:00
ducoterra 1200437bf8 phase: 168_admin_settings_panel
Build and Push Containers / build-and-push-app (push) Successful in 4m4s
Build and Push Containers / build-and-push-db (push) Successful in 15s
**Phase 168 — Admin Settings panel — final verification: PASS**

1. **Implemented/verified:** `/settings.html` (admin-only) with the 75-field catalogue (`app/core/settings_catalog.py`), DB-over-env overrides in Postgres (`app_settings`, migration 0028 at head, `settings_overlay.py` resolver, `app/api/admin_settings.py` — 4 admin-only routes), live-apply on save with no restart + restart-required badges, Reset/Clear all, write-only secrets, `.env`-never-written, `BOR_SETTINGS_DB=0` kill switch.
2. **Phase E2E: 12/12** — run with `E2E_APP_PORT=8200` (ports 8123/8124 are held by an unrelated "undercooked" project; no process killed, no code change — the documented env override).
3. **Regression set: all 7 suites green in isolation** (theme tab, API tokens, configurable brand, global tuning, admin auth, cache busting, wide desktop column — 38 tests).
4. **Full suite: unit+integration exit 0 (4066 passed), app/ coverage 99%** (gate >90%); **ruff clean, pyright 0 errors / 0 warnings.**
5. **Log contract (rule 10) verified live:** per-turn + per-response byte-budget lines present; settings write lines carry key + action, **never the value**.
6. **Byte-budget audit: all endpoints under the 262,144 B budget**; settings catalogue 23,513 B (gzip 3,326).
7. **Defect fixed (one):** criterion 5 had no integration pin — added `test_bad_row_inserted_directly_never_bricks_boot` to `tests/integration/test_admin_settings_api.py` (malformed row inserted directly → fresh `create_app()` boots, row skipped with logged reason, rest of table applies, `/api/health` 200) — 12/12 green.
8. **Docs pass:** `README.md` and `AGENTS.md` reviewed in full — both consistent with the as-built feature (Settings page table, Settings section, Configuration-reference note, `.env.example` block); **no changes required**.
9. **Cleanup:** audit app killed (by PID); `app_settings` left empty (0 rows); all changes left in the working tree (no commit, per protocol).
10. **Completion criteria:** all 7 checkboxes verified green (criterion 7's commit is the harness's).
11. **Next pending phase:** `170_edit_bubble_sizing`.
2026-10-10 01:05:14 -04:00
ducoterra abe7388dda docs(phases): convert TODO.md L3-L4 into phases 170-171 (edit bubble, settings layout)
170_edit_bubble_sizing — the edit form renders inside the user bubble
(same box, no shrink) and its submit label reads "Send", not "Save"
(TODO.md L3).
171_admin_settings_layout — settings page CSS fixes: #settings-content
vertical rhythm (search/table overlap + smushed action row), distinct
group bands, a properly disabled Save cursor (TODO.md L4).

Numbering note: 169 (embedding generation migration) was created,
executed and abandoned before this conversion; its number is not
reused. TODO.md is cleared — the items now live in .agents/phases/.

LLM-Generated: true
2026-10-09 23:42:15 -04:00
ducoterra 542e568d25 docs(agent): record the reverted embed embedding benchmark
The embed alias was reverted to the previous backing model and re-benchmarked
over 3 runs. All gates pass and the numbers reproduce the original baseline
exactly: 768 dims, cosine margin 0.283 (quality 56) versus the 0.217 measured
on the swapped-in model, with throughput back to 109.7-123.3 vec/s.

The margin returning to the identical 0.283 confirms the locked semantic-pair
corpus is a reliable fingerprint for this endpoint, and a raw-vector probe
shows the two models are genuinely different spaces despite both being
unit-normalized 768-dim (max component diff 0.035 on the same input). Any
content embedded while the swapped-in model was live is therefore in an
incompatible space and needs re-embedding.

LLM-Generated: true
2026-10-09 12:41:38 -04:00
ducoterra a88a5a3713 docs(agent): re-run the embed embedding benchmark (speed variance)
Second 3-run pass against the re-pointed embed endpoint the same day.
Dimension stays 768/768 and the cosine margin reproduces exactly at 0.217
across all passes, confirming both the model's determinism and that the
margin drop from the 0.283 baseline is a real effect of the model swap.
Throughput measured 59.1/61.4/57.4 vec/s against 35.5/35.6 earlier, a 1.7x
swing on an unchanged model, which shows the speed check measures one batched
request's end-to-end wall time (TLS setup, network, server queueing) rather
than stable throughput - note it gates on NaN/Inf only, never on rate.

LLM-Generated: true
2026-10-09 12:38:27 -04:00
ducoterra 05f7d0645e docs(phases): add phase 169 (embedding generation stamp + in-place re-embed) and the migrate-embed-model skill
An embedding-model swap that keeps the dimension is invisible on this
app: nothing records which model made a vector, EmbeddingDimensionError
cannot fire at 768 -> 768, and a re-sync re-embeds nothing because the
documents.file_hash fast path returns with zero embeds - so questions
embed in one geometry and chunks score in another with no error anywhere.
The owner's upgrade (embedding gemma -> embedding gemma 2) is exactly
that case.

Phase 169, seven tasks: chunks.embedding_model (reversible migration
0029, nullable, never backfilled - NULL means unknown by design), one
write helper so every vector write stamps its generation, a
generation_report() that turns the column into counts plus a stale
verdict logged after every sync, scripts/reembed_embeddings.py with
--compare (cosine agreement against the stored vectors, so 'are they
compatible?' is measured instead of guessed) / --stamp-only / --apply
(per-batch commits, resumable, never a NULL embedding, never a
sources_meta bump), an admin-only GET /api/admin/embeddings, a read-only
card in the phase-168 Settings view, and the dedicated Playwright file
plus the README runbook.

Owner-locked during authoring: detection only - retrieval never filters
by generation (a WHERE embedding_model predicate would silently deflect
every turn mid-backfill), comparability bands COMPATIBLE >= 0.995 for
the float4 round trip, a re-embed is not a KB change so saved chats stay
fresh, and no new Settings field so the phase-168 catalog pin is
untouched.

.agents/skills/migrate-embed-model is the operator twin: probe and
benchmark the new model, measure comparability as its own process
without disturbing the running app, back up and migrate production
(idle check, pg_dump off-box, alembic, flip then re-embed with the same
env), then re-tune relevance_threshold and the two floors against the
retrieval battery - after the corpus is on one generation, and with the
phase-168 warning that an app_settings row beats .env - and smoke-test
the result.

LLM-Generated: true
2026-10-09 12:35:21 -04:00
ducoterra 8fac4e64d4 docs(agent): record the re-pointed embed embedding benchmark
The aipi `embed` endpoint was re-pointed to a new backing model and was
re-benchmarked over 3 runs. All three gates pass: output stays 768-dim
(matching BOR_EMBEDDING_DIM, so no schema/vector change is needed), the
similar-vs-dissimilar cosine margin is 0.217 (gate >= 0.05) with no NaN/Inf
vectors, and throughput is 35.6 vec/s on the 50-vector probe.

Versus the previous embed baseline (margin 0.283, 93.8 vec/s) the margin
tightens and throughput drops ~2.6x, both consistent with a model swap
behind the same alias. Vectors are unit-normalized and deterministic.
Note: same dimension on a different model means any KB indexed under the
old backing model holds stale vectors and needs a re-embed; the local dev
DB is empty, so the phase 119/162 retrieval battery was not runnable here.

LLM-Generated: true
2026-10-09 12:23:55 -04:00
ducoterra a12fdcdf0a Move phase 167 audit to complete
Build and Push Containers / build-and-push-app (push) Successful in 14s
Build and Push Containers / build-and-push-db (push) Successful in 11s
2026-10-09 10:27:51 -04:00
ducoterra a2b2f283fd docs(phases): convert TODO.md L3 into phase 168 (admin settings panel)
One owner item, one phase: an admin-only Settings view that writes any
BOR_* setting to the database, resolves DB-over-env inside
get_settings(), and applies on Save without a restart. Six tasks: the
app_settings table + reversible migration 0028, the overlay resolver and
field catalog, the live-apply reset of the two in-process singletons
(the shared LLMClient and the chat semaphore), the admin-only
/api/admin/settings surface, the shell's eighth view, and the dedicated
Playwright suite.

Owner-locked during authoring: DB-only persistence (.env is never
written), the overlay lives inside get_settings() behind the
BOR_SETTINGS_DB kill switch, database_url and embedding_dim are not
DB-overridable, boot-wired settings stay editable with a "restart
required" badge, secrets are masked and write-only, and every write is
validated by the real Settings validators while an unusable stored row
is skipped at read so boot can never brick.

Protocol B append: PLAN.md and AGENTS.md are untouched, so PLAN's
roadmap table does not list phase 168 and its "get_settings() is
lru_cache'd" note in 13 is extended (cached env base + DB overlay +
invalidation). TODO.md's L3 item is now cleared - it lives in this
phase directory.

LLM-Generated: true
2026-10-09 10:10:13 -04:00
988 changed files with 110245 additions and 1592 deletions
+35 -3
View File
@@ -84,6 +84,25 @@ A15 — `thinking`, `tool`, unlimited tool rounds under
never-truncated contract is unchanged; the `read`-tool cap is
the only exception, owner-permitted 2026-09-10.
Phase 182 (verbatim text display, owner chat 2026-10-14) adds
**`show_text(path, offset?, limit?)` as the fifth agent tool** — offered
only when `BOR_ANSWER_TEXT=1` (the default), gated exactly like
`show_image`, with its own `<tools>` clause and its own kill switch — and
the **additive `done.answer_text`** field above. The new settings are
`BOR_ANSWER_TEXT` (master switch, ON), `BOR_SHOW_TEXT_MAX_CHARS` (per-block
budget, 12 000) and `BOR_SHOW_TEXT_MAX_BLOCKS` (per-answer budget, 5), all
following the `agent_max_rounds` pattern (`0` inert, negative refuses
startup). The principle in one sentence: **the model points, the server
supplies the bytes** — a ` ```bor source/path 12-40 ` fence or a
`show_text` call names a location (canonical locator `source/path` or
`source/path#start-end`, 1-based and inclusive), and the displayed text is
sliced from `documents.content` server-side, so what the user copies is the
knowledge base, never a re-typed approximation. **A1–A6 and A15 are
otherwise
UNCHANGED:** no new SSE event type, no new table or column, no new model,
endpoint or transport, no change to retrieval, the honesty gate, the
persona text or the embedding dimension.
---
## 3. High-Level Architecture
@@ -129,7 +148,9 @@ POST /api/chat {message, history?} (auth: require_user)
→ cosine top-100 ∪ FTS top-30 → RRF fuse (k=60) [A7]
├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
│ persona + <knowledge_base> + <tuning> + full top-2 <documents>
│ + <tools> → agent loop (ls/read/grep; round-capped
│ + <tools> → agent loop (ls/read/grep, plus show_image when
│ BOR_ANSWER_IMAGES=1 and show_text when BOR_ANSWER_TEXT=1 — the two
│ "show the user an artifact" tools, gated independently; round-capped
│ BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
└─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
byte-identical direct-stream path) → turbo streamed
@@ -139,7 +160,15 @@ POST /api/chat {message, history?} (auth: require_user)
SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
suggestions[], budget_exhausted?}` (terminal). Failure: `error` (terminal — no `done`, no
suggestions[], budget_exhausted?, answer_text[]?}` (terminal). **Phase 182
extends the `done` frame ADDITIVELY** — `answer_text` is the list of
server-resolved verbatim blocks (`locator`, `source`, `path`, `start`,
`end`, `total_lines`, `text`, `format`, `truncated`) the answer displayed;
there is **no eighth event type**, the key is OMITTED when no block was
displayed, and with `BOR_ANSWER_TEXT=0` it is absent on every turn (the
frame is then byte-identical to pre-phase). The same array persists on the
saved chat record, so restore and the anonymous shared page re-render the
panels from the server's bytes. Failure: `error` (terminal — no `done`, no
`query_log` row; a pre-stream DB outage is a plain 503 JSON). A
`tools=None` request never emits a `tool` frame; a round-capped turn
whose final reply carries no visible content answers with the budget
@@ -273,8 +302,11 @@ scripts.…`, and tests all honor it.
question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
kb_chars=N history_msgs=N threshold=… deflected=… sources=…
thinking_chars=… tool_calls=N rounds=N cap_hit=0|1 total_ms=…
retries=N scaffold_stripped=N
retries=N scaffold_stripped=N verbatim_blocks=N
```
(`verbatim_blocks=` — phase 182 — is the count of verbatim display blocks
actually SENT on the `done` frame, `0` on every turn without them,
deflected included.)
(`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
executed only; a cancelled turn writes neither line nor `query_log`
row.)
@@ -0,0 +1,81 @@
# Phase 168 — Admin Settings panel: edit any `BOR_*` setting in the browser, DB over env, live on Save
**Source:** `TODO.md` L3 — "I want an admin panel that can edit anything in .env and save it to the database. The database value should override the env var. The change should take effect on clicking 'save' on that page."
**Story:** n/a (owner-directed feature)
**Context:** `app/config.py` (75 `Settings` fields, `env_prefix="BOR_"`, `env_file=".env"`, `extra="ignore"`, `get_settings()` `@lru_cache`d, ~40 call sites across `app/`), `app/db.py` (module-level `settings = get_settings()` + `create_engine`), `app/main.py` (`settings = get_settings()` at import, middleware wired at boot, `_shell_routes`), `app/core/theming.py::effective_settings` (the DB-over-env resolver precedent, B1), `app/models.py::UiSettings` (the single-row admin-persistence precedent), `app/api/ui_settings.py` + `app/api/tokens.py` (router-wide `require_admin`), `app/core/auth.py::require_admin`, `frontend/index.html` (`#view-*` shell views; `#view-tokens` L1077, `#view-theme` L1269), `frontend/assets/router.js` (`VIEW` / `VIEW_PATH` / `VIEW_MODULES`), `frontend/assets/header.js` (~L197-230, the ship-hidden / reveal-for-admin nav contract), `frontend/assets/styles.css` (`.table-wrap` L2825 + `.table-wrap table thead th` sticky L2841 — the phase-156 bounded-scroll precedent), `app/core/caching.py::HTML_PAGES` (L169), `alembic/versions/` (next free: **0028**), `tests/e2e/conftest.py` (the `os.environ["BOR_PDF_PICTORIAL"] = "0"` global-pin precedent, ~L52).
## Objective
An admin opens **Settings** in the navbar, edits any setting that today lives only in `.env` (75 fields, grouped, type-correct inputs), clicks **Save**, and the new value is (a) stored in Postgres, (b) preferred over the env var from that instant on with **no restart**, and (c) visible in the running UI. Per-row **Reset** deletes the DB row so the env value wins again. The `.env` file is never written by the app.
## Design (shared by all tasks — the executor reads this, not the chat)
**D1 — Storage: one `app_settings` table (task 01).** `key` = the **bare `Settings` field name** (`relevance_threshold`), `value` = the **env-native string form** (`"0.55"`, `"true"`, `"md,txt"`, a JSON array for the one `list[str]` field), `updated_at`. Storing bare strings (not typed columns, not a wide row) is what makes "anything in .env" work for all 75 fields with zero per-field DDL, and keeps the coercion in exactly one place (the pydantic validators). The `BOR_` prefix stays a *presentation* concern (`env_var = "BOR_" + key.upper()`), never stored — a stored prefix is a double-prefix typo waiting to happen.
**D2 — The overlay lives INSIDE `get_settings()` (task 02).** The ~40 existing call sites (`app/rag/retriever.py`, `app/api/chat.py`, middlewares, `scripts/`) must not change — that is the only way "anything" becomes live rather than "the three fields someone remembered to thread through". `get_settings()` keeps its `@lru_cache` and its zero-arg signature (`tests/{integration,e2e}/*.py` call `get_settings.cache_clear()` in 6 files — that surface must survive), and resolves: `Settings()` (env/`.env`, exactly today) → overlay the DB rows → validate → return. `invalidate_settings()` clears the cache and runs the reset hooks (D3). Hot-path cost after the first call: one `lru_cache` dict hit.
**D3 — "Takes effect on Save" = cache + the two in-process singletons (task 03).** Everything except two objects is read per use, so clearing the cache is enough for ~70 fields. The exceptions are `app/api/chat.py::_llm` (an `LLMClient` built once from `get_settings()` — holds `llm_base_url` / `effective_api_key` / `llm_chat_model` / `llm_embed_model` / timeouts / retries, L318-326) and `_chat_semaphore` (built once from `chat_max_concurrent`, L302-315). The overlay exposes a hook registry; `app/main.py::create_app()` registers the chat reset (`_llm = None`, `_chat_semaphore = None`) so `app/core/` never imports `app/api/` (dependency direction, PLAN §3). Settings wired at boot cannot be applied in-process (engine pools, `SessionMiddleware`, the static mount, the gzip middleware, logging) — they stay **editable** but carry a `restart_required` badge (LOCKED A4).
**D4 — The catalog is code, derived-checked, and complete (task 02).** `app/core/settings_catalog.py` declares, per field: `group` (mirrors `app/config.py`'s own section comments: App & branding, HTTP & logging, Database, LLM, Chat history, RAG & retrieval, Agent tools, Import & sources, Images, PDF, Docs push, Admin & security), `label`, `secret`, `restart_required`, `overridable`. A unit test asserts the catalog covers **exactly** `Settings.model_fields` (a new setting added without catalog metadata fails the test → it can never silently disappear from the panel).
**D5 — Fail loud on write, fail soft on read (tasks 02/04, LOCKED A5).** A write validates the FULL candidate set — every stored override plus the new one, layered over the env base — through the real `Settings` validators, so a cross-field typo (`lexical_support_floor` above a DB-overridden `relevance_threshold`) is a 422 that names the field and writes nothing. A read that finds an unusable row (hand-edited, or a validator changed in a later release) skips THAT key, logs `settings override ignored key=… reason=…` and applies the rest: a DB value must never brick boot, because a bricked app cannot be un-bricked from the panel.
**D6 — The kill switch, and the E2E poison guard (task 02).** `BOR_SETTINGS_DB=0` (env-only, never itself DB-overridable, read straight from `os.environ`) disables the overlay entirely — the pre-phase behavior, the house `agent_max_rounds`/`compress_min_bytes` convention. `tests/e2e/conftest.py` pins it to `0` for **every** E2E app the same way it already pins `BOR_PDF_PICTORIAL`: a row left behind by one suite would otherwise silently retune every later suite's app subprocess. The phase-168 suite opts back in on its own app fixture (the `test_pictorial_pdf.py` pattern) and `TRUNCATE app_settings` before AND after (the `test_admin_theme_tab.py:286` precedent).
**D7 — The API (task 04).** `app/api/admin_settings.py`, prefix `/admin/settings`, router-wide `dependencies=[Depends(require_admin)]` (the `ui_settings`/`tokens` pattern: anonymous AND token users 403): `GET` the catalog + effective values with a `source` badge (`db` | `env` | `default`), `PUT /{key}` set one value, `DELETE /{key}` reset to env, `DELETE` reset all (the panel's "Clear all overrides"). Secrets are **write-only** (LOCKED A4): `GET` reports `is_set`, never the plaintext, and the write log line carries the key and the action, never the value.
**D8 — The view (task 05).** The shell's eighth view: `#view-settings` in `frontend/index.html` + `frontend/assets/settings.js`, `/settings.html` added to `main.py::_shell_routes` and `caching.HTML_PAGES`, `VIEW`/`VIEW_PATH`/`VIEW_MODULES` in `router.js`, and `#nav-settings` (ship-hidden) in `index.html` / `document.html` / `login.html` / `shared.html` revealed by `header.js` — the exact phase-91 Theme-view wiring. Layout per PLAN §7 + AGENTS rule 5: the 72rem `.container`, one `.table-wrap` card (bounded `max-height: min(70vh, 64rem)`, sticky `thead`) with a `<tbody>` per group, type-correct controls (`<input type=number>` for int/float, a `true/false` `<select>` for bool, `<textarea>` for the CSV/JSON and long strings, a password input for secrets), a text source badge (B5: never color alone), a client-side `type=search` filter (the phase-101 tokens pattern), a page-level **Save settings** (dirty rows only), per-row **Reset**, and a `role="status"` / `role="alert"` live region (§7.4).
## Dependencies
- `91_admin_theme_tab` + `92_theme_save_and_coverage` + `93_theme_semantic_completion` (complete) — the DB-over-default resolver (B1), the admin-only shell view + gate pattern this view copies, and the nav reveal contract.
- `79_api_tokens` (complete) — router-wide `require_admin` + the bounded, searchable admin catalogue table with per-row actions.
- `156_table_ux_polish` (complete) — `.table-wrap` bounded scroll + sticky header, the `cell-clip` ellipsis contract for long action labels.
- `167_api_payload_diet` (complete) — the response byte budget: the new `GET /api/admin/settings` (~75 rows ≈ 20 KB) must stay under `BOR_LOG_LARGE_RESPONSE_BYTES` and the audit script's 256 KB bar; never a 413.
- `82_security_headers` + `81_login_rate_limit` (complete) — the CSP/`default-src 'self'` posture the new JS must live inside (no CDN, rule 6).
## Tasks
1. `01_app_settings_table.md` — `AppSetting` model + reversible migration `0028_app_settings.py` + its migration test.
2. `02_settings_overlay_resolver.md` — `app/core/settings_catalog.py` + `app/core/settings_overlay.py`, `get_settings()` becomes the cached env+DB resolver, `invalidate_settings()`, the `BOR_SETTINGS_DB` kill switch + the E2E conftest pin, the catalog-completeness unit pins.
3. `03_live_apply_on_save.md` — the reset-hook registry, the chat `_llm` / `_chat_semaphore` reset registered from `create_app()`, and the integration proof that a saved change bites **without a restart**.
4. `04_admin_settings_api.md` — `/api/admin/settings` GET / PUT `/{key}` / DELETE `/{key}` / DELETE all, the schemas, the full-candidate validation, the masked secrets, the value-free log line, the README API note.
5. `05_settings_view_ui.md` — the Settings shell view (`#view-settings`, `settings.js`, router + shell route + `HTML_PAGES` + nav reveal, `.table-wrap` catalogue, Save/Reset + live region).
6. `06_e2e_settings_panel.md` — `tests/e2e/test_admin_settings_panel.py`, the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_settings_catalog.py` (catalog ⇄ `Settings.model_fields` equality; the masked set is exactly `llm_api_key` + `admin_password` + `session_secret` + `database_url` (the last one is read-only but its URL carries credentials); the non-overridable set is exactly `database_url` + `embedding_dim`), `tests/unit/test_settings_overlay.py` (merge/coercion/validation-skip/`BOR_SETTINGS_DB=0`), `tests/unit/test_settings_frontend_view.py` (static source pins in the house style: the view section exists, `router.js` maps `/settings.html`, no CDN, `settings.js` builds DOM with `createElement`/`textContent`).
- **Integration:** `tests/integration/test_migration_0028.py` (up/down, A13), `tests/integration/test_admin_settings_api.py` (auth matrix anonymous/token-user/admin, 404 unknown key, 422 validator failures naming the field, masked secrets in the response body, the DB-over-env flip visible through `GET /api/config`, reset-to-env, `get_settings()` reflecting a write with no restart), plus the existing `get_settings.cache_clear()` suites staying green untouched.
- **Coverage:** **>90%** on `app/` — `uv run pytest --cov=app --cov-report=term-missing`.
- **E2E:** `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_admin_theme_tab.py`, `test_api_tokens.py`, `test_configurable_brand.py`, `test_global_tuning.py`, `test_admin_auth.py`, `test_cache_busting.py`.
- **Lint/types:** `uv run ruff check . && uv run pyright`.
- **Log contract (AGENTS rule 10):** every settings write logs one line (`settings action=set|clear|clear_all key=… source=db|env`) with no value; `GET /api/admin/settings` appears in the `grep ' :: api '` byte audit and under the 256 KB bar (`uv run python -m scripts.api_payload_audit`).
## Completion Criteria
- [ ] `/settings.html` is an admin-only shell view: anonymous sees the sign-in gate, a token user never sees the nav link, and all four API routes 403 for both.
- [ ] Editing a value + **Save settings** stores it in `app_settings`, flips the row's badge from `env`/`default` to `DB`, and the effect is visible in the running app with **no restart** (pinned E2E: `BOR_INPUT_PLACEHOLDER` shows up in the composer; `BOR_RELEVANCE_THRESHOLD` / `BOR_LLM_CHAT_MODEL` are read from the overlay by a live request).
- [ ] **Reset** deletes the row and the env value wins again; **Clear all overrides** empties the table; `.env` is never opened for writing by any code path (unit-pinned: no `open(` on the env file in `app/`).
- [ ] Secrets never leave the server (`GET` body has no plaintext, `page.content()` has no secret), no write log line contains a value, and a bogus value is a 422 that names the field and stores nothing.
- [ ] A stored-but-invalid row (inserted directly, bypassing the API) is ignored at read with a logged reason — the app still boots and `/api/health` is 200.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the phase E2E + the regression set green in isolation.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-09 during phase authoring)
- **A1 — DB-only persistence:** the panel writes `app_settings` and **never** touches `.env`; the env var stays the base and the DB row wins.
- **A2 — the overlay sits inside `get_settings()`** (cached, `cache_clear()`-compatible) so all ~40 consumers go DB-over-env at once, behind the `BOR_SETTINGS_DB=0` kill switch.
- **A3 — `database_url` and `embedding_dim` are NOT DB-overridable** (read-only rows, reason shown: the engine is built from `database_url`, so an override is a chicken-and-egg brick; A6 fixes `chunks.embedding` at 768, so a dim override would corrupt retrieval).
- **A4 — boot-wired settings stay editable with a "restart required" badge** (pools, session middleware, static mount, gzip, log level); **secrets are masked and write-only** and never appear in a response body or a log line.
- **A5 — validate the full candidate set on write (422, nothing stored), fail soft on read** (skip the bad key, log it, boot anyway).
- **A6 — one story, one phase, one Playwright file** (AGENTS rule 4) — the backend tasks are covered by unit + integration, the live-effect contract by `tests/e2e/test_admin_settings_panel.py`.
## Assumptions (the TODO left these open; the executor does not guess)
- **K1:** the key is the bare field name; `BOR_UPPER` is derived for display/API presentation only.
- **K2:** the one `list[str]` field (`suggestions`) is edited as a JSON array in a `<textarea>` and stored in that same JSON form (pydantic parses it exactly like the env var); every other field is a scalar or CSV string.
- **K3:** a missing `app_settings` table (fresh deployment, migrations not applied) means "no overrides" and logs at DEBUG once — only an unusable *value* logs at ERROR.
- **K4:** "Save settings" sends only the dirty rows; the first failing row aborts the batch, shows its 422 detail, and keeps the other rows' edits in the form for correction.
- **K5:** history/audit beyond `updated_at` (who changed what, when, old value) is out of scope — single-admin homelab (PLAN §1 "out of scope").
- **K6:** the overlay is single-process (PLAN A12: in-process state, restart-clearing accepted, no Valkey); with more than one uvicorn worker a save is visible in the worker that served it and in every worker after its next resolution — the deployment runs one app process, and the panel notes nothing about it.
- **K7:** `.env` keys that are not `Settings` fields (`extra="ignore"`, e.g. `AIPI_KEY`, `DEBUGPY*`) are not editable — they have no `app/` consumer to retune. `BOR_LLM_API_KEY` remains the way to set the key (it wins over `$AIPI_KEY` via `effective_api_key`).
## Commit
```bash
git add app/ alembic/ frontend/ tests/ .env.example README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(settings): admin settings panel with DB-over-env overrides applied live on save"
```
@@ -0,0 +1,60 @@
# Task 01 — `app_settings`: the table that holds DB-side setting overrides
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "I want an admin panel that can edit anything in .env and save it to the database."
**Story:** n/a
## Objective
One narrow key/value table stores the admin's overrides, keyed by the bare `Settings` field name, with the value kept in its **env-native string form** — the storage half of phase-168 D1, with zero per-field DDL for the 75 settings.
## Work
1. `app/models.py` — append the `AppSetting` class (after `FolderSummary`, the file's tail) in the house docstring style, naming the phase and the TODO line:
```python
class AppSetting(Base):
"""One admin setting override (phase 168, task 01).
``TODO.md`` L3: the admin Settings panel persists what used to be
``.env``-only knobs here, and the row WINS over the environment
(resolved by ``app.core.settings_overlay``, consumed by
``app.config.get_settings``). …
"""
__tablename__ = "app_settings"
#: The BARE ``Settings`` field name (``relevance_threshold``) — never
#: the ``BOR_…`` form: the prefix is presentation
#: (``"BOR_" + key.upper()``), so a stored prefix could double up.
key: Mapped[str] = mapped_column(String(100), primary_key=True)
#: The value in the SAME textual form the env var would carry
#: (``"0.55"``, ``"true"``, ``"md,txt"``, a JSON array for the one
#: ``list[str]`` field) — coercion and validation belong to
#: pydantic, in exactly one place (phase 168 D1/A5).
value: Mapped[str] = mapped_column(Text, nullable=False)
#: When the admin last saved it (the ``folder_summaries`` /
#: ``kb_overview.updated_at`` house style: server-side now()).
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now(), nullable=False
)
```
Import `func` from `sqlalchemy` if the module does not already (it imports `DateTime`, `String`, `Text`, `Integer`, `select`-adjacent names — check the header first, do not duplicate an import).
No index: the whole table is read by primary key order in one `SELECT` per settings resolution (the `ui_settings` "display fact, never a query surface" note, 0027).
2. `alembic/versions/0028_app_settings.py` — `revision = "0028"`, `down_revision = "0027"`, docstring in the 0027 style (what the table is for, why the value is TEXT, why no index, why the downgrade is a clean drop). `upgrade()` = `op.create_table("app_settings", sa.Column("key", sa.String(length=100), primary_key=True), sa.Column("value", sa.Text(), nullable=False), sa.Column("updated_at", sa.DateTime(timezone=True), server_default=sa.func.now(), nullable=False))`; `downgrade()` = `op.drop_table("app_settings")` with the A13 comment (dropping it leaves the 0027 schema byte-identical; the values are re-enterable from the panel).
3. `tests/integration/test_migration_0028.py` (new) — copy the `test_migration_0017.py` harness verbatim (the `alembic` fixture with the `db_available()` skip + the teardown `upgrade(cfg, "head")`, `_version`, the `information_schema` assertion helpers) and pin:
- 0027 → 0028 creates `app_settings` with `key` VARCHAR(100) NOT NULL and PK, `value` TEXT NOT NULL, `updated_at` TIMESTAMPTZ NOT NULL with the `now()` server default, and `alembic_version = '0028'`;
- the 0027 schema survives (`documents.page_strip_mode` still present);
- rows round-trip (insert `relevance_threshold`/`"0.55"` and `import_extensions`/`"md,txt"`, read them back);
- a duplicate `key` raises `IntegrityError` (the PK is the uniqueness guard — two rows for one setting would make "which one wins" undefined);
- `NULL` value is rejected (NOT NULL — "clear the override" is a DELETE, never an empty value, so the resolver never has to distinguish them);
- downgrade to 0027 → the table is GONE, the rest survives; re-upgrade → the table is back (A13 round-trip).
4. `.env.example` — add a short comment block at the top (under the header, before the first `BOR_` line) stating the precedence rule and the kill switch: the app reads `BOR_*` from the environment, an admin override saved in the database wins per key, the panel never writes this file, and `BOR_SETTINGS_DB=0` disables DB overrides (set it in the container env, not here — a `0` committed here is a silent "my change did nothing"). Task 02 implements what this comment describes; write it so it stays true.
ASSUMPTION (K3): `.env.example` is documentation only — nothing parses it at runtime except pydantic's `env_file=".env"`, so a comment cannot change behavior.
## Testing & Quality
- Integration: `uv run pytest tests/integration/test_migration_0028.py -v` (needs `podman compose up -d db`).
- Unit: no new module yet; `uv run pytest tests/unit -q` stays green (adding a model cannot change behavior — if something moved, it is a bug in this task).
- Coverage: **>90%** on `app/` — the added model is exercised by the migration test's insert/read plus task 02's resolver tests.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies 0028 and `uv run alembic downgrade 0027` removes `app_settings` cleanly.
- [ ] `tests/integration/test_migration_0028.py` green (including the duplicate-key and NOT-NULL pins).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change anywhere: `get_settings()` still reads env only (that is task 02), and no route exists yet.
@@ -0,0 +1,76 @@
# Task 02 — The overlay: `get_settings()` resolves DB-over-env, behind a kill switch
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "…save it to the database. The database value should override the env var."
**Story:** n/a
## Objective
A cached resolver that layers the `app_settings` rows over the environment and hands every existing consumer a valid `Settings` — no consumer changes, no restart, and a `BOR_SETTINGS_DB=0` switch that puts the pre-phase behavior back exactly.
## Work
1. `app/core/settings_catalog.py` (new) — the presentation + capability metadata, code-owned and complete:
- `@dataclass(frozen=True, slots=True) class SettingMeta`: `key: str`, `group: str`, `label: str`, `secret: bool = False`, `restart_required: bool = False`, `overridable: bool = True`.
- `CATALOG: dict[str, SettingMeta]` covering **every one** of `Settings.model_fields` (75 today). Groups, in this display order, with these members (they mirror `app/config.py`'s own section comments — if the executor disagrees with one placement, keep the count and note it in the commit body):
| Group | Fields |
|---|---|
| App & branding | `app_name` `app_version` `environment` `input_placeholder` `footer_text` `suggestions` |
| HTTP & logging | `log_level`↻ `log_large_response_bytes` `compress_min_bytes`↻ `static_dir`↻ |
| Database | `database_url`⛔ `db_pool_size`↻ `db_pool_max_overflow`↻ `db_pool_recycle`↻ `chat_max_concurrent` |
| LLM (aipi) | `llm_base_url` `llm_api_key`🔒 `llm_chat_model` `llm_embed_model` `llm_summary_model` `stream_thinking` `llm_retries` `llm_retry_delay` `llm_timeout` |
| Chat history | `history_max_turns` `history_max_chars` |
| Retrieval & honesty | `embedding_dim`⛔ `top_n_docs` `relevance_threshold` `lexical_support_floor` `source_usefulness_floor` `related_max_docs` `suggested_docs` `suggestion_preview_chars` `hybrid_vector_candidates` `hybrid_lexical_candidates` `rrf_k` `recency_boost` `recency_half_life_days` `name_hit_bonus` |
| Chunking & embedding | `chunk_target_chars` `chunk_overlap_chars` `embed_batch_size` `embed_question_max_chars` |
| Prompt budgets | `max_output_tokens` `steering_max_chars` `summary_max_chars` `kb_overview_max_chars` `overview_input_max_chars` `folder_summary_input_max_chars` |
| Agent tools | `agent_max_rounds` `read_max_chars` |
| Import & sources | `import_extensions` `git_sources` `sources_dir` `upload_dir` `upload_max_mb` |
| Images | `images` `image_extensions` `image_dir` `chat_image_dir` `chat_image_max_mb` `answer_images` |
| PDF | `pdf_pictorial` `pdf_pictorial_min_chars` `pdf_image_skip_blank` `pdf_page_dir` |
| Docs push | `docs_repo` `docs_branch` `docs_base_branch` `docs_work_dir` |
| Admin & security | `admin_password`🔒 `session_secret`🔒↻ `session_max_age`↻ `session_cookie`↻ |
`↻` = `restart_required=True` (wired once at boot: logging, the gzip middleware in `app/main.py`, the static mount, `create_engine` in `app/db.py`, `SessionMiddleware`); `⛔` = `overridable=False` (LOCKED A3 — `database_url` is what builds the engine that would read the override, `embedding_dim` is fixed in the `chunks` table by A6); `🔒` = `secret=True`, plus `database_url` as a fourth masked field (its URL carries credentials) even though it is read-only.
- `ENV_PREFIX = "BOR_"`, `def env_var(key: str) -> str` → `"BOR_" + key.upper()`; `GROUPS: tuple[str, ...]` (the table order above); `SECRETS`, `RESTART_REQUIRED`, `NON_OVERRIDABLE` derived from `CATALOG` (never re-listed by hand).
- Module docstring: the phase, the TODO line, why the catalog exists (the panel is generated, not hand-written) and the completeness pin.
2. `app/core/settings_overlay.py` (new) — the mechanism:
- `resolve_settings() -> Settings` — the whole overlay: `base = Settings()`; `if not enabled(): return base`; `rows = load_overrides()`; `return merge(rows)[0]`. **Never call `app.config.get_settings()` here** — `Settings()` is the env-only entry point (recursion guard, pinned by test).
- `enabled() -> bool` — `os.environ.get("BOR_SETTINGS_DB", "1")` is not one of `0/false/no/off` (the house bool parsing), read **per call** from `os.environ` so tests can flip it; the flag itself is not a `Settings` field, so it can never be turned off from the database (the escape hatch cannot disable itself).
- `load_overrides() -> dict[str, str]` — `SELECT key, value FROM app_settings ORDER BY key` through a module-level lazily-created engine: `create_engine(Settings().database_url, poolclass=NullPool, pool_pre_ping=True)` (NullPool: one connection per resolution, none held — the app's real pool belongs to `app/db.py` and must not be used here: importing `app.db` from the overlay would be a cycle, `app/db.py` imports `app.config`, and the pool whose size this table can change must not be the one that reads the table). Fail-soft: `ProgrammingError`/undefined-table → `{}` + one DEBUG line per process (K3 — a pre-migration deployment); any other `SQLAlchemyError`/`OSError` → `{}` + one WARNING per process naming the exception class ("settings overrides unavailable — env values in effect"). Both "once per process" guards are module-level flags, reset by `reset_state_for_tests()`.
- `merge(rows: Mapping[str, str]) -> tuple[Settings, list[tuple[str, str]]]` — returns `(settings, skipped)` where `skipped` is `(key, reason)` pairs. Build the kwargs from the rows: drop keys not in `Settings.model_fields` (an orphan row from a removed setting) and keys with `overridable=False` (a hand-inserted `database_url` row must never hijack the engine — LOCKED A3), coerce each value (`_coerce(field, raw)`: `json.loads` for the one `list[str]` field `suggestions`, the raw string for everything else — pydantic-settings lax-coerces `str`→`int`/`float`/`bool`, and init kwargs rank ABOVE env in pydantic-settings, which is precisely "the database value overrides the env var"). Then `Settings(**kwargs)`; on `ValidationError` drop the named field (`error["loc"][0]`), append it to `skipped` with the validator's message, and retry — so one bad row costs one setting, not the whole table (LOCKED A5). Log each skipped key once at ERROR: `settings override ignored key=… reason=…`.
- `register_invalidator(fn: Callable[[], None]) -> None` + `run_invalidators()` — the hook list task 03 uses; a hook that raises is logged, never propagated (a failed reset must not undo a successful save).
- `reset_state_for_tests() -> None` — clears the module flags/engine for the unit suites.
3. `app/config.py` — keep the `Settings` class 100% untouched and swap the resolver tail:
```python
@lru_cache
def get_settings() -> Settings:
"""Effective settings: env/.env with the admin's DB overrides layered
on top (phase 168, TODO L3). … cached exactly as before —
``get_settings.cache_clear()`` still works — and
``invalidate_settings()`` is the save-side wrapper that also runs the
runtime reset hooks (task 03)."""
from app.core.settings_overlay import resolve_settings # lazy: app.core imports app.config
return resolve_settings()
def invalidate_settings() -> None:
get_settings.cache_clear()
from app.core.settings_overlay import run_invalidators
run_invalidators()
```
The lazy imports are load-bearing (documented in the comment): `app.core.*` imports `app.config` at module scope, and `app/db.py` → `app.config` runs at import time; a module-scope `app.config` → `app.core` import would deadlock the import graph. The 6 existing `get_settings.cache_clear()` test call sites keep working unchanged (they only retune env, so skipping the hooks there is correct).
4. `app/main.py` — inside `create_app()`, immediately after `ensure_admin_configured(settings)`: `invalidate_settings()` then `effective = get_settings()` and ONE boot log line, `settings overrides loaded count=N` (or `source=env-only disabled` when the kill switch is off) — the prime that turns a boot-time DB miss into a retry rather than a permanently cached env-only view. Keep the module-level `settings = get_settings()` as is (it is the pre-boot value the middleware wiring uses today).
5. `tests/conftest.py` — next to the existing `os.environ.setdefault("BOR_ADMIN_PASSWORD", …)` block, add `os.environ.setdefault("BOR_SETTINGS_DB", "0")` with a comment: the shared test DB must never retune the in-process suites (the same poison-guard reasoning as the phase-140 `BOR_PDF_PICTORIAL` pin in `tests/e2e/conftest.py`), and tests that WANT the overlay set the var + `get_settings.cache_clear()` themselves.
6. `tests/e2e/conftest.py` — same pin beside the `os.environ["BOR_PDF_PICTORIAL"] = "0"` line (~L52): `os.environ["BOR_SETTINGS_DB"] = "0"`, comment it as the cross-suite poison guard (one suite's saved row would silently change every later suite's app subprocess). The phase-168 E2E suite overrides it to `"1"` in its own module app fixture (the `test_pictorial_pdf.py:339` `env["BOR_PDF_PICTORIAL"] = "1"` pattern) — task 06 does that.
7. `tests/unit/test_settings_catalog.py` (new) — `set(CATALOG) == set(Settings.model_fields)` (both directions, the diff in the failure message — this is the pin that stops a new setting from being invisible in the panel); `every group in GROUPS exists in CATALOG and every CATALOG group is in GROUPS`; `SECRETS == {llm_api_key, admin_password, session_secret, database_url}`; `NON_OVERRIDABLE == {database_url, embedding_dim}`; `env_var("relevance_threshold") == "BOR_RELEVANCE_THRESHOLD"`; no label is empty and no label duplicates another.
8. `tests/unit/test_settings_overlay.py` (new) — pure/unit level, no DB: `enabled()` matrix (`"0"/"false"/"no"/"off"` off; unset/`"1"/"true"` on); `_coerce` for int/float/bool/str and `suggestions` JSON (a malformed JSON value is skipped, not fatal); `merge` happy path over the real env (`{"relevance_threshold": "0.55"}` → float 0.55, everything else byte-equal to `Settings()`), `merge` skips an unknown key and a `NON_OVERRIDABLE` key with a reason, `merge` drops one invalid value and keeps the rest, a cross-field violation (`lexical_support_floor` above a DB-overridden `relevance_threshold`) lands in `skipped` instead of raising, bool spellings (`"false"`, `"0"`) behave like the env, `load_overrides()` returns `{}` + one WARNING when the engine raises, `register_invalidator`/`run_invalidators` runs hooks in order and survives a raising hook, and a pin that `app/core/settings_overlay.py` contains no `get_settings(` call (the recursion guard) and that no file this phase owns writes `.env` (LOCKED A1 — assert the absence of an `open(` / `.env` write in `app/core/settings_overlay.py`, `app/config.py` and — once it exists — `app/api/admin_settings.py`).
## Testing & Quality
- Unit: the two new files; then `uv run pytest tests/unit -q` (the `get_settings` surface is unchanged for every existing pin) and `uv run pytest tests/integration/test_api.py -q` (it calls `get_settings.cache_clear()` — it must pass untouched).
- Integration: `get_settings()` with `BOR_SETTINGS_DB=0` is byte-identical to `Settings()` — pin it in `tests/integration/test_admin_settings_api.py` (task 04) or here if the DB is up.
- Coverage: **>90%** on the new modules — the fail-soft branches (undefined table, generic error, invalid value, non-overridable row) each need a test.
## Completion Criteria
- [ ] `uv run python -c "from app.config import Settings, get_settings; print(get_settings().relevance_threshold)"` prints the env value, and `BOR_RELEVANCE_THRESHOLD=0.11 …` still prints 0.11 (the env path is untouched).
- [ ] With a live DB and one hand-inserted `app_settings` row + `BOR_SETTINGS_DB=1` + `invalidate_settings()`, `get_settings()` reports the DB value; with `BOR_SETTINGS_DB=0` it reports the env value.
- [ ] A hand-inserted row for `database_url` or `embedding_dim` changes nothing and logs the skip.
- [ ] `tests/{conftest,e2e/conftest}.py` pin the switch off; the whole `uv run pytest` suite passes with a *populated* `app_settings` table present (the poison guard proven by running the suite twice: once with the table empty, once with a garbage row in it that must change nothing).
- [ ] `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,68 @@
# Task 03 — Live apply: Save re-points the two in-process singletons
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "The change should take effect on clicking \"save\" on that page."
**Story:** n/a
## Objective
After `invalidate_settings()`, the next request behaves under the new value with **no restart**: the ~70 per-use settings are covered by the cache clear alone, and the two objects that captured a `Settings` snapshot — the shared `LLMClient` and the chat concurrency semaphore — are rebuilt.
## Why only two objects (traced 2026-10-09, do not re-derive)
Every other setting is read through `get_settings()` at the point of use (`app/rag/retriever.py:408,700,807,860`, `app/core/response_budget.py:172`, `app/api/sync.py:528`, the `LLMClient()` constructions in `app/api/sync.py:532`, `app/api/docs.py:480`, `app/api/git_sources.py:944`, the path fields resolved per operation), so clearing the cache is the whole job for them. `app/core/caching.py`'s `functools.cache` `asset_version()` is the deploy token (A19), not a setting. The genuine snapshots are:
- `app/api/chat.py:318-326` — `_llm: LLMClient | None`, built once by `get_llm()`; `LLMClient.__init__` (`app/rag/llm.py:246-254`) holds `self.settings` and bakes `llm_base_url`, `effective_api_key` and `llm_timeout` into the `AsyncOpenAI` client, while `llm_chat_model` / `llm_embed_model` / `llm_summary_model` are read off that captured snapshot on every call.
- `app/api/chat.py:301-315` — `_chat_semaphore: asyncio.Semaphore | None`, built once from `chat_max_concurrent`.
Boot-wired values (`db_pool_*`, `session_*`, `static_dir`, `log_level`, `compress_min_bytes`, `database_url`) are LOCKED as `restart_required` / non-overridable — this task does not attempt them (A3/A4).
## Work
1. `app/api/chat.py` — add, directly below `get_llm()` (and documented as the phase-168 counterpart of the lazy-init comments above it):
```python
def reset_runtime_caches() -> None:
"""Drop the settings-derived singletons (phase 168, task 03).
Called by ``invalidate_settings()`` after the admin saves. The next
turn rebuilds the client from the CURRENT ``get_settings()`` — an
in-flight turn keeps the client it started with (its ``AsyncOpenAI``
transport is mid-request; swapping it would kill the stream), which
is the same accepted-semantics note as A12's in-process state.
``_chat_active`` is deliberately untouched: it counts live turns, and
the new ``chat_max_concurrent`` applies as permits are taken.
"""
global _llm, _chat_semaphore
_llm = None
_chat_semaphore = None
```
2. `app/main.py` — in `create_app()`, right after the task-02 prime line: register the hook and log it once.
```python
# Phase 168 (task 03): the settings panel's Save must retune the running
# process, and app/core must never import app/api (PLAN §3's direction) —
# so create_app(), the composition root, wires the reset hook.
register_invalidator(reset_runtime_caches)
```
with `from app.api.chat import reset_runtime_caches` added to the existing `app.api.chat` import block and `from app.core.settings_overlay import register_invalidator` alongside the other `app.core` imports. Importing `app.api.chat` here is already done (the router import at the top of the file), so this adds no new import cost or cycle.
3. `tests/unit/test_settings_live_apply.py` (new) — source/behaviour pins in the house style:
- `reset_runtime_caches()` nulls both globals: set `app.api.chat._llm = object()` and `app.api.chat._chat_semaphore = object()`, call it, assert both are `None`; assert `_chat_active` is untouched by the call;
- `get_llm()` rebuilds after a reset and returns the SAME instance on two calls with no reset (the singleton contract still holds — this pin stops a future "always construct" shortcut that would break connection pooling);
- `_get_chat_semaphore()` rebuilds after a reset and reflects a changed `chat_max_concurrent` (drive it with `monkeypatch.setenv("BOR_CHAT_MAX_CONCURRENT", "3")` + `get_settings.cache_clear()`, assert `_chat_semaphore._value == 3`), then restore;
- `app/main.py`'s source contains `register_invalidator(reset_runtime_caches)` inside `create_app` (the static pin that stops the wiring being "cleaned up" — without it a Save silently stops retuning the LLM client);
- `invalidate_settings()` runs registered hooks and clears the cache (call it after registering a counter, assert the counter moved and that `get_settings()` afterwards re-resolves);
- a hook that raises does not propagate out of `invalidate_settings()` (a failed reset must not turn a committed save into a 500).
4. `tests/integration/test_admin_settings_live.py` (new; DB required — use the existing `db` fixture + `pytest.skip` idiom from `tests/integration/test_ui_settings_api.py`) — the end-to-end proof of "no restart", in-process:
- start from an empty `app_settings` and `monkeypatch.setenv("BOR_SETTINGS_DB", "1")` + `invalidate_settings()`, teardown `DELETE FROM app_settings` + `invalidate_settings()` + `get_settings.cache_clear()` (never leave a row behind for the suites that run after);
- `relevance_threshold`: read `get_settings().relevance_threshold` (the env/test value), `INSERT` the row `('relevance_threshold','0.5')` directly, `invalidate_settings()`, assert `get_settings().relevance_threshold == 0.5` and that a retrieval call that consults it (`app/rag/retriever.py`'s gate reads it at `:408`) now behaves under 0.5 — assert through the function the way `tests/integration/test_recency_boost.py` does, no chat turn needed;
- `llm_chat_model`: with the row `('llm_chat_model','gpt-test')`, `invalidate_settings()`, then `app.api.chat.get_llm().settings.llm_chat_model == "gpt-test"` **and** the rebuilt client's `AsyncOpenAI` base_url/timeout come from the new snapshot (set `('llm_timeout','7')` in the same batch and assert `client._client.timeout.connect == 7.0`, the way `app/rag/llm.py` builds it) — this is the pin that a Save actually swaps the client;
- `chat_max_concurrent`: same shape, assert the rebuilt semaphore's `_value` (via `_get_chat_semaphore()`) follows the saved number;
- the kill-switch counterpart: with rows present but `BOR_SETTINGS_DB=0`, `invalidate_settings()` → the env values are back (the switch really is the whole feature's kill switch);
- `import_extensions` (the `scripts/` consumer): after the save, `get_settings().import_extension_set` reflects the DB value — the importer and Sync read the same overlay (no separate code path).
5. `.env.example` — extend task 01's comment block with the one sentence that matters operationally: values marked "restart required" in the panel (DB pool sizes, session cookie wiring, log level, compression, the static dir) apply on the next `podman compose restart app`, everything else applies immediately.
## Testing & Quality
- Unit: `uv run pytest tests/unit/test_settings_live_apply.py -v`, then `uv run pytest tests/unit -q`; the existing chat pins (`tests/unit/test_chat_*`, `tests/integration/test_chat_api.py`, `test_chat_concurrency.py`) pass untouched — the reset only runs when `invalidate_settings()` is called, which nothing in them calls.
- Integration: `uv run pytest tests/integration/test_admin_settings_live.py -v` with the DB up; then `uv run pytest tests/integration -q` to prove the teardown left nothing behind.
- Coverage: **>90%** on `app/api/chat.py`'s new function and `app/core/settings_overlay.py`'s hook paths.
## Completion Criteria
- [ ] Saving a value (row insert + `invalidate_settings()`) changes what the NEXT request does — pinned for a retrieval setting, the LLM client (model + timeout), and the chat concurrency cap, with no process restart.
- [ ] An in-flight turn is never swapped mid-stream (the docstring's contract; the unit test proves `get_llm()` hands out the same object until a reset).
- [ ] `register_invalidator(reset_runtime_caches)` is wired in `create_app()` and unit-pinned; `app/core/` imports no `app/api/` symbol (grep-clean).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,51 @@
# Task 04 — `/api/admin/settings`: the admin-only read/write surface
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "…can edit anything in .env and save it to the database."
**Story:** n/a
## Objective
One admin-only router that serves the whole catalogue (every setting, its effective value, and where that value came from) and writes single values back to `app_settings` — validated by the real `Settings` validators before anything is stored, masked where the value is a secret.
## Work
1. `app/schemas.py` — append the four models (docstrings in the `UiSettingsIn`/`UiSettingsOut` style, naming the phase and the route):
- `AdminSettingItem`: `key: str`, `env_var: str`, `group: str`, `label: str`, `type: Literal["int","float","bool","str","json"]`, `secret: bool`, `restart_required: bool`, `overridable: bool`, `source: Literal["db","env","default"]`, `value: str | None` (the effective value in env-native string form; **`None` when masked**), `masked: bool`, `has_value: bool` (for a masked field: is a non-empty value in effect at all), `has_override: bool` (a row exists), `db_value: str | None` (the stored string when it is a stored non-secret override, else `None`), `default: str` (the code default stringified — for a masked field `""`).
- `AdminSettingsOut`: `items: list[AdminSettingItem]`, `totals: {"count": int, "overridden": int}`, `overlay_enabled: bool` (the `BOR_SETTINGS_DB` state — when `false` the panel shows a banner instead of pretending a save took effect).
- `AdminSettingIn`: `value: str` — always a string, coercion belongs to pydantic (the same reason D1 stores strings).
- `AdminSettingsCleared`: `cleared: int`.
Stringify helpers live in the router (`_display(field, value)`), not in the schemas: `bool` → `"true"/"false"`, `list[str]` → `json.dumps(...)`, everything else `str(value)`.
2. `app/api/admin_settings.py` (new) — `router = APIRouter(prefix="/admin/settings", tags=["admin-settings"], dependencies=[Depends(require_admin)])` (the `app/api/ui_settings.py` pattern: anonymous AND token users 403 on every route). Module docstring lists the routes and the three invariants (validate-before-store, masked secrets, never write `.env`).
- `GET ""` → `AdminSettingsOut`: read the rows once (`SELECT key, value FROM app_settings`), resolve the effective `Settings` through `get_settings()`, then emit one `AdminSettingItem` per catalog entry **in `GROUPS` order, catalog order within the group** (a stable order the E2E pins against). `source`: `db` when an *allowed* row is in effect, `env` when the process env has the `BOR_*` var set (check `os.environ`), else `default`. A masked field emits `masked=True`, `value=None`, `db_value=None` and `has_value=bool(effective)`.
- `PUT "/{key}"` (`AdminSettingIn` → `AdminSettingItem`, 200):
- unknown key → 404 `f"Unknown setting {key}"`;
- `overridable=False` → 422 `f"{env_var} is not settable from the database"` + the reason in the docstring (A3: the engine is built from `database_url`; `chunks.embedding` is fixed at `embedding_dim` by A6);
- `secret=True` and the value is empty/whitespace → 422 `f"{env_var} must not be blank — use Reset to fall back to the environment"` (an empty API key or session secret is a self-lockout; `DELETE` is the explicit clear);
- **candidate-set validation** (A5/D5): the stored rows for the allowed keys + this key/value, run through the same `app.core.settings_overlay.merge()` path — a `ValidationError` (bad shape OR a cross-field rule like `lexical_support_floor` above a DB-overridden `relevance_threshold`) is a 422 whose detail names the field, and **nothing is written**;
- write: `INSERT … ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = now()` (`sqlalchemy.dialects.postgresql.insert`), commit, then `invalidate_settings()` (task 03 makes it live) and return the refreshed item.
- `DELETE "/{key}"` → `AdminSettingItem`: delete the row (idempotent — deleting an absent key is a success, not a 404), `invalidate_settings()`, return the item as `source="env"|"default"`.
- `DELETE ""` → `AdminSettingsCleared`: `DELETE FROM app_settings`, `invalidate_settings()`, `{"cleared": rowcount}` (the panel's "Clear all overrides").
- **Log one line per write, never the value** (AGENTS rule 10, A4): `settings action=set key=… status=200`, `settings action=clear key=…`, `settings action=clear_all count=N` on the module logger `app.api.admin_settings`. GETs log nothing here — the phase-167 `ResponseBudgetMiddleware` already writes the `api method=… bytes=…` line, and the whole catalogue must stay under `BOR_LOG_LARGE_RESPONSE_BYTES`' INFO bar (75 items ≈ 20 KB; verified in task 06 with the audit script).
- Never opens `.env`, never shells out, never `eval`s a value — the module contains no file writes (unit-pinned).
3. `app/main.py` — `from app.api.admin_settings import router as admin_settings_router` in the router-import block and `app.include_router(admin_settings_router, prefix="/api")` right after the `ui_settings_router` line, with the one-line comment the neighbours carry ("admin-only (router-wide require_admin; a token user stays 403)").
4. `tests/integration/test_admin_settings_api.py` (new; the `db`-fixture + skip idiom of `tests/integration/test_ui_settings_api.py`, admin client per that file's convention). Set `BOR_SETTINGS_DB=1` + `invalidate_settings()` in the fixture, and in teardown `DELETE FROM app_settings` + `invalidate_settings()` + `get_settings.cache_clear()` — **no row may survive this module** (the same discipline `test_admin_theme_tab.py:286` applies to `ui_settings`). Pin:
- **auth matrix:** anonymous `403`, token user `403` (issue a token the way `test_tokens_api.py` does), admin `200` — on all four routes;
- **shape:** `items` length == `len(Settings.model_fields)`, groups in `GROUPS` order, `totals.count`/`overridden` correct, `overlay_enabled` true; on an empty table a field whose `BOR_*` var the test env sets reports `source="env"`, an untouched one `source="default"`, and none reports `"db"`;
- **write → live:** `PUT {"value": "0.5"}` for `relevance_threshold` → 200 with `source="db"`, `get_settings().relevance_threshold == 0.5`, `totals.overridden == 1`, and the row present with a non-null `updated_at`;
- **visible-through-the-app proof:** `PUT input_placeholder` then `GET /api/config` (no auth) shows the DB value — the DB-over-env rule reaches the public config endpoint through the same overlay;
- **errors:** unknown key → 404; `database_url` / `embedding_dim` → 422 naming the env var; `"abc"` for `relevance_threshold` → 422 naming `relevance_threshold` and the table stays empty; a cross-field pair that only breaks in combination (store `relevance_threshold=0.2`, then `lexical_support_floor=0.9`) → the second `PUT` 422s and the first row is unchanged;
- **masking:** store `llm_api_key` directly, `GET` reports `masked=True`, `value is None`, `has_value=True`, and the plaintext appears NOWHERE in `response.text` (assert on the raw text, not the parsed model); blank `PUT` for it → 422; `DELETE` clears it;
- **reset:** `DELETE /{key}` → the item is `source="env"` (with the env var set in the fixture) and `get_settings()` is back to the env value; `DELETE ""` with 3 rows → `{"cleared": 3}` and `overridden == 0`;
- **kill switch:** with `BOR_SETTINGS_DB=0` (monkeypatch + `invalidate_settings()`) `overlay_enabled` is `false` while a `PUT` still stores the row and `get_settings()` still reports the env value — the switch disables the mechanism, not the panel;
- **byte budget:** `len(response.content)` for the full `GET` is under 64 KB (the phase-167 INFO bar) — so the catalogue never floods the log.
5. `README.md` — under **Configuration reference** (L1071): a short lead paragraph stating the precedence order (`app_settings` row → `BOR_*` environment → code default), that the panel never writes `.env`, that `BOR_SETTINGS_DB=0` disables DB overrides, and that the values marked "restart required" apply on the next `podman compose restart app`.
## Testing & Quality
- Integration: `uv run pytest tests/integration/test_admin_settings_api.py -v` (DB up), then `uv run pytest tests/integration -q` twice with a `app_settings` row left in on purpose ONCE inside the run only — the module's teardown must clean it (proven by a second clean run).
- Unit: the module's masking + `_display` helpers (bool/JSON/scalar stringification), the `.env`-never-written and no-shell static pins.
- Coverage: **>90%** on `app/api/admin_settings.py` (every 404/422 branch + both `DELETE`s + the masked branch).
## Completion Criteria
- [ ] `GET /api/admin/settings` returns all 75 settings grouped, each with its effective value and a `db`/`env`/`default` badge; secrets report `masked`.
- [ ] `PUT` stores, validates first (nothing written on failure, 422 names the field) and makes the value live through `get_settings()` in the same process.
- [ ] `DELETE /{key}` restores the env value; `DELETE ""` clears everything; both log a value-free line (`grep " :: app.api.admin_settings " <logs>`).
- [ ] Anonymous and token-user callers get 403 on every route; the byte audit keeps this endpoint under the 256 KB budget and, pinned, under 64 KB.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,53 @@
# Task 05 — The Settings view: the shell's eighth view, `#view-settings`
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "I want an admin panel that can edit anything in .env … The change should take effect on clicking 'save' on that page."
**Story:** n/a
## Objective
An admin opens **Settings**, sees every setting grouped with the value in effect and where it came from, edits a few, presses **Save settings**, and sees the badges flip to `DB` — while the anonymous visitor gets the sign-in gate and a token user never sees the nav link. Same wiring as the phase-91 Theme view, same table discipline as the phase-156 catalogue tables.
## Work
1. `frontend/index.html`
- nav: after `#nav-theme` (L62) add `<a href="/settings.html" class="nav-link" id="nav-settings" hidden>Settings</a>` (ship-hidden, revealed for admin only — the phase-34 one-bar contract; the same one line goes into `document.html` (~L74), `login.html` (~L67) and `shared.html` (~L68) after their Theme links).
- view: after `#view-theme` add `<section class="view" id="view-settings" hidden inert aria-label="Settings" tabindex="-1">` following the `#view-tokens` skeleton exactly:
- `<div class="container settings-shell">` (PLAN §7.1's 72rem centered column — no new width token),
- `.page-head` with `<h1>Settings</h1>` and a `.page-sub` that states the rule in the owner's terms: values saved here are stored in the database and win over the server's environment variables; clearing one hands control back to the environment,
- `<section class="sources-gate" id="settings-gate" aria-labelledby="settings-gate-title" hidden>` — the `#tokens-gate` shape, heading "Sign in to change settings", sub-copy naming what is admin-only, `<a class="sources-gate-link" href="/login.html?next=/settings.html">Sign in</a>`,
- `<div id="settings-content" hidden>` holding, in order:
- `<div class="kb-banner is-error" id="settings-disabled" role="alert" hidden>` — shown when `overlay_enabled` is `false` ("DB overrides are switched off on the server (`BOR_SETTINGS_DB=0`) — saving stores the value but it will not take effect."),
- `<span class="settings-status" id="settings-status" role="status" aria-live="polite"></span>` (the ONE status region, the `#tokens-status` precedent) and `<p class="settings-error" id="settings-error" role="alert" hidden></p>` (the `#theme-error` precedent),
- the search row: `<label for="settings-filter">Search settings</label>` + `<input id="settings-filter" type="search" autocomplete="off">` (client-side filter, no fetch — the phase-101 D4 contract),
- the scroll card: `<div class="table-wrap"><table class="settings-table"><caption class="visually-hidden">Application settings — the value in effect and where it comes from</caption><thead><tr><th scope="col">Setting</th><th scope="col">Value</th><th scope="col">Source</th><th scope="col"><span class="visually-hidden">Actions</span></th></tr></thead></table></div>` — the `tbody` elements are created by `settings.js` (one per group, each opening with a `<tr class="settings-group-row"><th colspan="4" scope="colgroup">Group</th></tr>`), and the `.table-wrap` + sticky-`th` rules at `styles.css:2825/2841` are already the bounded-scroll + sticky-header contract (AGENTS rule 5) — `.visually-hidden` inside the header needs the `.table-wrap` `position: relative` that phase 86 added,
- the action row: `<button type="button" class="btn primary" id="settings-save" disabled>Save settings</button>` + `<button type="button" class="btn ghost" id="settings-clear-all">Clear all overrides</button>` + `<span class="settings-dirty" id="settings-dirty" aria-hidden="true"></span>` (an "N unsaved changes" hint, `textContent` only).
- No `<script>` tag for the new module: the router lazy-imports it (rule 6 — everything local).
2. `frontend/assets/router.js` — three entries in the phase-91 style: `"/settings.html": "settings"` in `VIEW`, `settings: "/settings.html"` in `VIEW_PATH`, `settings: () => import("./settings.js")` in `VIEW_MODULES` (a STATIC specifier so the Containerfile's `esbuild --bundle` of `router.js` inlines it — `settings.js` needs no new esbuild line because no page references it directly; `tests/integration/test_containerfile_assets.py` proves it).
3. `app/main.py` — add `"/settings.html"` to the `_shell_routes(...)` tuple (after `"/theme.html"`) with the usual comment.
4. `app/core/caching.py` — add `"/settings.html"` to `HTML_PAGES` (L169 block) with the `# phase 168: the admin settings view (shell route)` comment, so it gets the `no-cache` + `?v=` contract (A19).
5. `frontend/assets/header.js` — after the Theme reveal block (~L225) add the null-safe `#nav-settings` block, admin-only, with the comment naming the contract it copies (a token user never sees it).
6. `frontend/assets/settings.js` (new) — `export async function mount(root)`, the `theme.js` shape (gate first via `fetchIsAdmin()`/`fetchWhoami()` from `header.js`; anonymous shows `#settings-gate` and fetches nothing; then unhide `#settings-content`, load, bind; listen for `bor:view-refresh` to re-load on a re-show):
- `GET /api/admin/settings` → render the groups in the server's order (`createElement` + `textContent` only, never HTML strings — PLAN §7 frontend rule). One row per setting:
- **Setting cell:** the `label` (`.cell-clip` ellipsis when it truncates, the phase-156 contract with the full text in the control's `aria-label`), the `BOR_*` name in a `<code class="settings-env">`, and — when `restart_required` — a `.settings-flag` with the literal text `restart required` (text, never color alone, B5) plus `title`/`aria-describedby` copy explaining when it applies;
- **Value cell:** the control chosen by `type` — `int`/`float` → `<input type="number">` (`step="any"` for float), `bool` → a two-option `<select>` (`true`/`false`), `json` → `<textarea spellcheck="false">`, `str` → `<input type="text">` (a `<textarea rows="2">` for the CSV/path fields whose value exceeds 40 chars), `secret` → `<input type="password" autocomplete="new-password" placeholder="•••••• (stored)">` with `aria-label` naming the env var. Every control gets an `id` (`set-<key>`) and a `<label for>` (visually the Setting cell's label, the house pattern for table-hosted inputs). For a masked field an empty input means "leave unchanged" (never an accidental wipe);
- **Source cell:** `<span class="settings-src is-db|is-env|is-default">DB</span>` / `env` / `default` — literal text;
- **Actions cell:** a `Reset` button, `disabled` when `!has_override` (nothing to reset) or `!overridable` (with `title` naming the reason).
- dirty tracking in module state (`Map<key, value>`): an `input`/`change` marks the row (`.is-dirty` + the `N unsaved changes` hint) and enables `#settings-save`; a row that returns to its served value un-marks.
- `#settings-save` → for each dirty row in DOM order `PUT /api/admin/settings/<key>` with `{ value }`; **stop at the first failure** (K4), show its server detail in `#settings-error` (role=alert), keep the remaining edits in the form, re-`GET` so the badges show the truth, keep focus on the offending input. All good → one `#settings-status` line: `Saved 3 settings.` (then `role=status`, not alert) + a re-`GET` so every badge/value is the settled server state. While in flight the button is `disabled` with `aria-busy="true"` (§7.4: every control state defined; a failure path always re-enables).
- `Reset` → `DELETE /api/admin/settings/<key>` → status line `BOR_X is back to the environment value.` + re-`GET`. `#settings-clear-all` → the house two-step inline confirm (the `history-confirm-*` classes, focus moves to the confirm button, Escape/`No` cancels) → `DELETE /api/admin/settings` → status line `Cleared N overrides.` + re-`GET`.
- `#settings-filter` → case-insensitive substring over the label + `BOR_*` text cells, applied on `input` with no fetch, re-applied after every re-`GET` (the query survives re-shows, phase-77); a group with no visible rows hides its group row; zero matches with a non-empty query → one no-match row `No settings match "<query>"` (`textContent`).
- A `restart_required` row that saves successfully adds "— applies on the next restart" to the status line (so the owner is never told "saved" and then sees nothing change).
7. `frontend/assets/styles.css` — a `---------- Settings view (phase 168) ----------` block: `.settings-shell` (no new container), `.settings-table { width: 100%; }` with `.git-sources-table`-style column proportions (never a skinny list — PLAN §7.1), `.settings-group-row th` (a `--surface` band, `--ink` text ≥4.5:1), `.settings-env` (mono, `--ink-soft`), `.settings-src` + the three `is-*` variants (solid `--brand-soft` / `--surface` / `--line` backgrounds with the existing ink tokens — no new palette variable, the phase-156 B3 rule; the badge is TEXT, the color is decoration), `.settings-flag` (`--accent-bg`/`--accent-ink`, the state-family pair), `.settings-row.is-dirty` (a `--brand` left border on the row + the `N unsaved changes` hint, so "unsaved" survives grayscale), `.settings-status` (the `.tokens-status` mirror), `.settings-error` (the `.theme-error` mirror), `input`/`select`/`textarea` sizing with `:focus-visible` inherited from the global rule, and ≥44px touch targets for the buttons/inputs at `≤640px` (§7.1 mobile). Nothing animates, so `prefers-reduced-motion` needs no new rule.
8. `tests/unit/test_settings_frontend_view.py` (new — the house static-source-assertion style, cf. `tests/unit/test_thinking_scroll.py` / `test_frontend_router.py`): `index.html` contains the `#view-settings` section, the `#settings-gate`, `#settings-content`, `#settings-status role="status"`, `#settings-error role="alert"`, `#settings-save`, `#settings-clear-all`, and `#nav-settings` with `hidden`; `router.js` maps `/settings.html` in all three tables; `main.py`'s shell tuple and `caching.HTML_PAGES` both list `/settings.html`; `header.js` reveals `#nav-settings` and keys it off `admin` (not `signedIn`); `settings.js` contains no `innerHTML =` with a template literal and no `http(s)://` reference (rule 6 + the no-HTML-strings rule); `styles.css` declares `.settings-table` and the three `.settings-src.is-*` variants and does NOT add a `:root` variable (B3); the four pages that carry the Theme link now carry the Settings link too.
9. `README.md` — a **Settings (`/settings.html`)** subsection under "Pages at a glance" (after "Access tokens", L205): what it is, the precedence rule, "restart required", and that it is admin-only.
## Testing & Quality
- Unit: the new file, plus `uv run pytest tests/unit/test_frontend_router.py tests/unit/test_caching.py -v` (both enumerate the views/pages and must be *extended*, not broken) and `uv run pytest tests/integration/test_containerfile_assets.py -v` (the new asset is reachable through the bundle).
- Integration: `uv run pytest tests/integration/test_caching_revalidation.py -v` — `/settings.html` must serve `no-cache` with rewritten `?v=` asset refs.
- The E2E for this view is task 06's file; here the view must be reachable by hand (`uv run uvicorn app.main:app --reload` → sign in → Settings).
- Coverage: **>90%** on the `app/` diff of this task (the shell route + `HTML_PAGES`), untouched elsewhere.
## Completion Criteria
- [ ] `/settings.html` deep-links to the Settings view (the shell document, `router.js` picks the view), anonymous sees the gate and no data, a token user never sees the nav link.
- [ ] The table lists every group with a bounded scroll region and a sticky header at 72rem and at 375px; long labels clip with the ellipsis and stay reachable through the control's `aria-label`.
- [ ] Editing marks the row dirty, **Save settings** reports one status line and every badge/value re-reads from the server; a rejected value shows the server's detail in the `role="alert"` region and keeps the other edits.
- [ ] Keyboard-only operation works end to end (filter → edit → Save → Reset → two-step Clear-all), `:focus-visible` visible on every control, and the page introduces no CDN reference and no HTML-string DOM building.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,57 @@
# Task 06 — `tests/e2e/test_admin_settings_panel.py`: the live-on-Save contract, proven in a browser
**Phase:** `168_admin_settings_panel` · **Source:** `TODO.md:3` — "…The change should take effect on clicking 'save' on that page."
**Story:** n/a
## Objective
The phase's dedicated Playwright suite (AGENTS rule 4/9): a real admin signs in, changes a setting through the panel, presses **Save**, and the running app behaves differently — no restart, no `.env` edit — plus the gate, the masking, the rejection path, and the reset. Then the full gates and the one atomic commit.
## Work
1. `tests/e2e/test_admin_settings_panel.py` (new) — copy the module-scoped app fixture from `tests/e2e/test_admin_theme_tab.py:205-270` verbatim (mock LLM, `BOR_ENVIRONMENT=e2e`, `BOR_STATIC_DIR`, the E2E admin password/secret, the docs-repo/suggestions/app-name leak guards) and change exactly these lines:
- `env["BOR_SETTINGS_DB"] = "1"` — the module opts back in over `tests/e2e/conftest.py`'s global `0` pin (task 02), the `test_pictorial_pdf.py:339` opt-in pattern;
- `env["BOR_INPUT_PLACEHOLDER"] = "Env placeholder for panel E2E"` and `env["BOR_FOOTER_TEXT"] = "Env footer for panel E2E"` — a distinctive env base, so "the DB row now wins" is a visible DOM change and not a no-op against the default;
- `env["BOR_RELEVANCE_THRESHOLD"] = "0.30"` (the E2E calibration, so a test that saves `0.31` proves the retrieval path read the new number, not a broken gate).
Add the module docstring in the house style: what the suite proves, the DB-isolation rule, and the Source line (`TODO.md:3`).
2. **DB isolation** (module autouse fixture, the `test_admin_theme_tab.py:286` `TRUNCATE ui_settings` idiom): `TRUNCATE app_settings` before AND after every test, plus `invalidate_settings()`-equivalent on the server side is unnecessary (the app runs in its own subprocess and re-reads on the save's own invalidation). Teardown must leave the shared dev E2E DB with an empty `app_settings` — a leftover row is the cross-suite poison this phase's D6 exists to prevent.
3. Tests (names pinned; helpers for `login(page, app_url)` from `tests/e2e/auth_helpers.py` and the `httpx` + cookies pattern those suites already use):
- `test_anonymous_sees_gate_and_no_data` — `/settings.html` serves the shell with `#view-settings` visible, `#settings-gate` visible, `#settings-content` hidden, `#nav-settings` hidden; `GET /api/admin/settings` unauthenticated is 403; no setting label or value appears in `page.content()`.
- `test_token_user_never_sees_the_panel` — a token user (issue one as admin via the API the way `test_api_tokens.py` does, then drive the token gate with `login_with_token`) sees `#nav-settings` hidden and gets 403 from the API.
- `test_admin_sees_groups_and_source_badges` — after `login()`, `#nav-settings` reveals, the view mounts, rows render, `table.settings-table` is inside `.table-wrap` (bounded scroll, sticky `thead` computed style), at least the `Retrieval & honesty` and `LLM (aipi)` group rows appear, `input_placeholder`'s row shows the served env value and an `env` badge, an untouched knob (e.g. `top_n_docs`) shows its code default with a `default` badge, `llm_api_key` renders a password input whose `value` is `""` and whose placeholder is the "stored" hint, and `database_url` / `embedding_dim` render read-only with no Reset and no input.
- `test_save_takes_effect_without_restart` — **the TODO's own acceptance test.** Filter to "placeholder", set the value to `DB placeholder — panel saved this`, press **Save settings**, expect: the `#settings-status` line (`Saved 1 setting`), the badge flips to `DB`, then `goto("/")` (a fresh page load, chat view) and expect the composer's `placeholder` attribute to be `DB placeholder — panel saved this` — the same browser, the same server process, no `.env` touched. Assert `footer_text` is still the env value (only one row was saved) and that the DB value survives a reload of `/settings.html`.
- `test_saved_value_reaches_a_real_turn` — set `BOR_RELEVANCE_THRESHOLD` from the panel to `0.31`, POST `/api/chat` (the `test_admin_theme_tab.py`/`test_honest_deflection.py` HTTP helper style) and assert the turn's honesty classification follows the new bar for a fixture question whose cosine sits between `0.30` and `0.31` (if the fixture corpus has no such question — verify with `uv run python -m scripts.eval_retrieval "<question>"` — pin the strongest available proxy: `query_log.top_score` is unchanged while the deflected/grounded decision flips when the saved bar moves above/below it, which proves the live read). Then **Reset** it and assert the pre-save classification returns.
- `test_invalid_value_is_refused_and_nothing_changes` — set `relevance_threshold` to `abc`, Save → the `#settings-error` (`role="alert"`) region is visible and names `relevance_threshold`, the badge is still `env`, the row keeps its typed value (K4), `GET /api/admin/settings` still reports `source=env`, and the composer placeholder (a second dirty row that was NOT saved because the batch stopped) is unchanged server-side.
- `test_cross_field_pair_is_refused` — save `relevance_threshold = 0.2`, then try `lexical_support_floor = 0.9` → the alert names the field and the stored pair never becomes effective (`GET /api/admin/settings` still shows the first saved value only).
- `test_secret_never_reaches_the_browser` — set `llm_api_key` to a sentinel (`sk-e2e-secret-…`) via the API (not through the DOM), reload `/settings.html`, and assert the sentinel is absent from `page.content()` while the row's password input shows the "stored" placeholder and `has_override` renders a live Reset button; then set it from the panel's password input, Save, and assert the sentinel still never appears in the served HTML or the response body.
- `test_reset_hands_control_back_to_the_env` — after a DB override on `footer_text`, press its Reset → status line, badge back to `env`, and the footer on `/` reads the env value again.
- `test_clear_all_overrides` — save three overrides, `#settings-clear-all` → the two-step confirm (Escape cancels and the overrides survive), then confirm → `Cleared 3 overrides`, every badge back to `env`/`default`.
- `test_disabled_switch_is_announced` (subprocess-level, its own module fixture or an `httpx`-only variant): a second app subprocess with `BOR_SETTINGS_DB=0` → the panel shows `#settings-disabled` (`role="alert"`) and a save still stores without changing the app. If a second app fixture is too heavy for the file, pin the `overlay_enabled=false` render through `page.route` interception of `GET /api/admin/settings` instead — state which was done in the commit body.
- `test_view_layout_and_accessibility` — no horizontal page overflow at 375px and at 1440px (§7.1: the table is ≥80% of the container, not a skinny list), keyboard tab order reaches filter → first input → Save → Reset (a `page.keyboard` walk asserting focus lands on the real controls), the `.settings-flag` text `restart required` present on `db_pool_size`'s row, and zero external requests during the whole test (`page.on("request")` — rule 6).
4. **Regression set, all in isolation** (`--no-cov`): `test_admin_theme_tab.py` (the Theme view's gate/save contract that this view copies), `test_api_tokens.py`, `test_configurable_brand.py` (`/api/config`'s strings — now DB-over-env-capable), `test_global_tuning.py`, `test_admin_auth.py`, `test_cache_busting.py`, `test_wide_desktop_column.py` (the 72rem container contract, new view included).
5. **Full gates** (AGENTS rule 9, all green before the commit):
```bash
uv run pytest
uv run pytest --cov=app --cov-report=term-missing # >90% on app/
uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov
uv run ruff check . && uv run pyright
```
Plus the operational proof of AGENTS rule 10: with the app running, save a setting, then `grep ' :: app.api.admin_settings ' <app log>` shows the key-only line and `grep ' :: api ' <app log>` shows the `GET /api/admin/settings` byte line; `uv run python -m scripts.api_payload_audit` lists the endpoint under the 256 KB budget.
6. **Docs & close-out:** confirm `README.md` carries both additions from tasks 04/05 (the Settings page section + the Configuration-reference precedence note); move the phase directory:
then move the phase directory and commit (AGENTS rule 8, `--no-gpg-sign`):
```bash
git mv .agents/phases/todo/168_admin_settings_panel .agents/phases/complete/168_admin_settings_panel
```
```bash
git add app/ alembic/ frontend/ tests/ .env.example README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(settings): admin settings panel with DB-over-env overrides applied live on save"
```
## Testing & Quality
- The suite passes **in isolation** (`-v --no-cov`) and the app it drives writes no `.env` change (`git status --porcelain .env` empty after the run — `.env` is gitignored, so assert by `stat` mtime instead if the file is untracked).
- Unit + integration already exist from tasks 02/03/04 — this file adds the browser contract only; if it needs a behavior the API lacks, fix the API in this task and say so in the commit body.
- Coverage: **>90%** on `app/` after everything (`--cov` runs exclude E2E by the repo's config; the gate is the unit+integration number).
## Completion Criteria
- [ ] `test_save_takes_effect_without_restart` is green: a value saved through the panel shows up in the running app in the same browser session with no restart.
- [ ] Gate, masking, invalid-value, cross-field, reset, clear-all and mobile/keyboard coverage all green in isolation.
- [ ] `app_settings` is empty after the suite (`SELECT count(*) FROM app_settings` → 0) and the rest of the E2E suite still passes with that pin in place.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean, the byte audit clean.
- [ ] Phase directory moved to `.agents/phases/complete/168_admin_settings_panel/` and one atomic `--no-gpg-sign` Conventional Commit exists (verify with `git log -1 --stat`).
@@ -0,0 +1,47 @@
# Phase 170 — Edit bubble: the editor lives inside the user bubble, "Send" re-asks
**Source:** `TODO.md` L3 — "the edit bubble is tiny compared to the message bubble. When you click "edit" it collapses down to this super small text field. It should stay the same size as the user message bubble. Also the save button should read "send" rather than "save" because you're not saving it you're re-sending the message."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-10)
**Context:** `frontend/assets/app.js` — `openEditor(wrap, toggleBtn)` (L1210–1357: label + `textarea[rows=3][maxlength=4000]` prefilled through `.value`, `editSaveBtn` "Save" L1238 + cancel, `closeEditor()` L1252 (form.remove + `.is-editing` remove + focus), the submit handler L1265–1345 (in-flight guard, empty guard, replace-in-place, tail truncation + DOM drop, `lastBrainWrap = null`, `bubble.innerHTML = renderMarkdown(text)` + the `.msg-image` re-prepend L1322–1325, `saveConversation()` + `persistConversation()`, `markLastRetryable()`, `closeEditor()` L1337, `runTurn(text, { reask: true })`), the open-tail L1355–1357 (`classList.add("is-editing")` + `body.appendChild(form)` + `ta.focus()`)), `appendEditButton` / `markLastEditable` (L793–871, the phase-159 single-target discipline), `tests/unit/test_frontend_edit_message.py` (house source pins: `editSaveBtn.textContent = "Save"` L348; the `.is-editing` add L365 / remove L387 / ordering L372; the `body.appendChild(form)` pin near L699), `tests/e2e/test_edit_and_resend.py` (the phase-159 suite: `page.click(".edit-form .edit-save")` L337 — class-based, so the label rename touches only its docstrings; the composer's `#send-label` "Send" pins L312/L533 are a different control), `frontend/assets/styles.css` — `.msg.user .bubble.is-editing { display: none; }` (L1411), `.edit-form` (L1412–1420 — its own `--surface` box + border + padding), `.edit-form label` (L1421–1425), `.edit-form textarea` (L1426–1436, `resize: vertical`, `min-height: 2.6rem`), `.edit-form-actions` (L1437), `.edit-save` (L1438–1455, incl. `:disabled { opacity: 0.6; cursor: wait; }` L1454), `.edit-cancel` (L1456–1469), `.edit-error` (L1470–1479), `.bubble` (L501–508), `.msg.user .bubble` (L592–596 — the `--brand` fill + `--bg` text).
## Objective
Clicking **Edit** no longer swaps the user bubble for a shrunken form: the editor lives *inside* the bubble itself (inheriting its box, background, and padding, so it keeps the bubble's dimensions) and the submit button reads **Send** — the action re-sends the question, it does not save a note. Everything else of the phase-159 flow (prefill, empty guard, tail truncation, the `reask` re-ask, one-editor-at-a-time, image survival, the in-flight guards) stays behavior-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
- **The form IS the bubble's content (LOCKED A1).** `openEditor` stops appending the form as a sibling of `.bubble` in `.msg-body` and stops hiding the bubble. It keeps the `.msg-image` child in a closure variable, keeps `bubble.classList.add("is-editing")` (L1355 — the state class survives, the unit pins keep it), and replaces `body.appendChild(form)` (L1356) with the form appended **into** the bubble, the rendered markdown cleared first. The bubble keeps its own `--brand` fill, border, padding, and width — "the same size as the user message bubble" holds by construction (the box IS the box), with zero measurement/matching code. `.msg.user .bubble.is-editing { display: none; }` (styles.css L1411) is **deleted** — the bubble stays visible while the editor is open.
- **The form is transparent.** `.edit-form` drops its own `background` / `border` / `padding` (the bubble provides them) and keeps `display:flex; flex-direction:column; gap: 0.5rem`. The label and the action text take the bubble's `--bg`-family ink (contrast documented in a comment — the B5 text-state rule); the textarea keeps its own readable `--surface` fill + `--ink` text (an edit field on a `--brand` bubble) and gains `width: 100%`. `rows=3` and `maxlength=4000` are untouched — the *outer* box is the bubble's, the textarea is the inner field.
- **Cancel = restore the markdown.** `closeEditor()` keeps exactly three jobs: `form.remove()`, `bubble.classList.remove("is-editing")`, `toggleBtn.focus()` — no content change. The cancel branch (Escape + the Cancel button) gains a restore step: `bubble.innerHTML = renderMarkdown(conversation[editIdx].text)` + re-prepend the saved image (the mirror of the submit re-render L1322–1325), then `closeEditor()`.
- **Submit ordering.** The submit handler currently re-renders (`bubble.innerHTML = renderMarkdown(text)` L1322) *before* `closeEditor()` (L1337). With the form inside the bubble, `closeEditor()` (which detaches the form) must run **before** the innerHTML re-render; the rest of the handler — truncation, `lastBrainWrap = null` / `leavePartialIndex` reset, persistence, `markLastRetryable()`, the in-flight re-check, `runTurn(text, { reask: true })` — is byte-untouched.
- **Label (LOCKED A2).** `editSaveBtn.textContent = "Send"` (app.js L1238) — matching the composer's `#send-label` "Send" and the "Cancel" sibling. The phase-168 "Save settings" button is a different control and keeps its name.
- **NOT touched:** `appendEditButton` / `markLastEditable`, `runTurn`, `retryLastTurn`, `handleSend`, `rememberBrainTurn`, the record-truncation + persistence path, `MAX_QUESTION_CHARS` / `updateCharCount`, the composer (the unsent draft survives the edit), the SSE contract (A15), the shared-chat page, `app/` server code (A10 stateless holds).
## Dependencies
- `159_edit_and_resend_message` (complete) — the inline editor + re-ask flow this phase re-houses and re-labels.
- `166_hide_empty_answer_bubble` (complete) — `.msg .bubble:empty { display: none }` (styles.css L524): while the editor is open the bubble holds the form (never `:empty`), and on cancel the re-rendered markdown is never empty (`min_length=1` pinned client- and server-side) — no interaction, the pin stays green.
## Tasks
1. `01_editor_inside_bubble.md` — `openEditor` builds the form **inside** `.bubble` (transparent form, `display:none` deleted), the cancel re-render, the submit ordering; the unit pins updated.
2. `02_save_label_send.md` — the submit label "Save" → "Send" (app.js L1238), the unit pin, the phase-159 E2E docstring alignment.
3. `03_e2e_edit_bubble.md` — `tests/e2e/test_edit_bubble_sizing.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_frontend_edit_message.py` (house source pins, updated in tasks 01/02): the form is appended to the bubble (not `.msg-body`), the `display:none` rule's absence pinned in the CSS, `.is-editing` still added (L365) / removed (L387), the cancel restore (markdown re-render + image re-prepend) and the submit ordering (form detached before the re-render), `editSaveBtn.textContent = "Send"`, and `runTurn` / `retryLastTurn` / `handleSend` byte-unchanged.
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_edit_bubble_sizing.py`, run **in isolation** — the size/label contract plus the regression flows.
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] Clicking Edit on the last user bubble shows the editor **inside** the bubble: the bubble is visible (computed `display` ≠ `none`), keeps its `--brand` fill, and its box width is unchanged by the edit (the E2E bounding-box pin).
- [ ] The submit button reads "Send"; Cancel / `Escape` restore the rendered question; trim + Send re-asks with the truncated shape and the "Question too long" driver case still resolves.
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_edit_bubble_sizing.py -v --no-cov` green **in isolation**; regression set green in isolation: `test_edit_and_resend.py`, `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-10, roadmap confirmation)
- **A1 — the editor lives inside the bubble:** "stay the same size as the user message bubble" = the editor occupies the bubble element itself (inherits its box/styling), not a pixel-matched sibling form. `rows=3` + `maxlength=4000` stay.
- **A2 — the label is "Send"** (capital S, matching the composer's `#send-label` "Send" and "Cancel"); only the edit-form label changes — the phase-168 "Save settings" button keeps its name.
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): edit the last question inside its bubble and re-send it with Send"
```
@@ -0,0 +1,40 @@
# Task 01 — the editor renders inside the user bubble (same box, no shrink)
**Phase:** `170_edit_bubble_sizing` · **Source:** `TODO.md:3` — "the edit bubble is tiny compared to the message bubble. When you click "edit" it collapses down to this super small text field. It should stay the same size as the user message bubble."
**Story:** n/a
## Objective
`openEditor` renders its form **inside** `.bubble` instead of as a hidden-bubble + small-sibling-form: the editor inherits the bubble's own box, background, and padding, so it keeps the bubble's dimensions by construction. The open / cancel / submit behavior of the phase-159 flow is otherwise unchanged.
## Work
1. `frontend/assets/app.js` — `openEditor` (L1210–1357), three surgical changes:
- **Open:** keep `if (bubble) bubble.classList.add("is-editing");` (L1355); replace `body.appendChild(form);` (L1356) with: grab the `.msg-image` child into a closure variable, clear the bubble's rendered markdown (`bubble.innerHTML = "";` or `bubble.replaceChildren(form)`), append the form **into** the bubble; `ta.focus()` (L1357) stays.
- **Cancel:** `closeEditor()` (L1252–1257) keeps exactly `form.remove()` + `bubble.classList.remove("is-editing")` + `toggleBtn.focus()` — no content change. The cancel paths (the `Escape` keydown branch L1258–1261 and the Cancel-button listener) gain a restore step BEFORE `closeEditor()`: `bubble.innerHTML = renderMarkdown(conversation[editIdx].text)` + re-prepend the saved image (the exact mirror of the submit re-render L1322–1325).
- **Submit:** reorder — move `closeEditor()` (L1337) to **before** the `if (bubble) { … bubble.innerHTML = renderMarkdown(text); … }` re-render block (L1322–1325), so the form is detached cleanly first. Everything else in the handler — the in-flight guard, the empty guard, `conversation[editIdx].text = text`, the tail truncation + DOM drop, `lastBrainWrap = null` / `leavePartialIndex` reset, `saveConversation()` + `persistConversation()`, `markLastRetryable()`, the second in-flight re-check, `runTurn(text, { reask: true })` — is byte-untouched.
- Update the `openEditor` comment block (L1154–1195): the "While open the rendered .bubble is hidden (.is-editing — the question appears ONCE)" sentence becomes the new contract (the form renders inside the bubble; the bubble stays visible; `.is-editing` is the state class); keep the rest of the block.
2. `frontend/assets/styles.css`:
- **Delete** `.msg.user .bubble.is-editing { display: none; }` (L1411) — the bubble stays visible while the editor is open.
- `.edit-form` (L1412–1420): drop the own `background: var(--surface);`, `border: 1px solid var(--brand-soft);`, and `padding: 0.75rem 0.85rem;` (the bubble provides them); keep `display: flex; flex-direction: column; gap: 0.5rem;`.
- `.edit-form label` (L1421–1425): the color becomes the bubble's `--bg`-family ink — on the `--brand` fill, `--bg` is the documented 1.0-against-brand pairing inverted (the user-bubble text pairing, L592–596); document the ratio in the comment (B5: text states it, never color alone); keep `font-size: 0.85rem; font-weight: 600;`.
- `.edit-form textarea` (L1426–1436): add `width: 100%;` (the field fills the bubble's width — the "not tiny" fix); keep `font-size: 0.9rem`, the `--surface` fill + `--ink` text (a readable edit field on the `--brand` bubble), `border`, `resize: vertical`, `min-height: 2.6rem`.
- `.edit-error` (L1470–1479): the `--err-bg` / `--err-ink` pair is documented against `--surface`; on the `--brand` bubble it needs a check — if the pair fails ≥4.5:1 there, scope a `.msg.user .bubble .edit-error` variant on the bubble's own ink pairing (`--bg` text on a `--brand`-darkened fill) and document the ratio; the `role="alert"` behavior is untouched.
- The `:disabled` wait-cursor on `.edit-save` (L1454) is the phase-159 in-flight dim — leave it (task 03 of THIS phase does not touch it; the owner's cursor complaint is the settings page, phase 171).
3. `tests/unit/test_frontend_edit_message.py` — update the pins (house source-pin style):
- the form is appended to the bubble: replace the `body.appendChild(form)` pin (near L699) with the bubble-append pin (e.g. `bubble.replaceChildren(form)` or the innerHTML-clear + append, whichever lands in app.js);
- the `.is-editing` add (L365) / remove (L387) / ordering (L372) pins **stay** (the class is still the state marker);
- new pins: the cancel restore (`renderMarkdown(conversation[editIdx].text)` on cancel + the image re-prepend) and the submit ordering (`closeEditor()` before the `bubble.innerHTML = renderMarkdown(text)` re-render);
- a CSS pin: `.msg.user .bubble.is-editing { display: none; }` is **absent** from `styles.css`;
- `runTurn` / `retryLastTurn` / `handleSend` pins byte-unchanged.
- ASSUMPTION: A1 (phase-locked) — "stay the same size as the user message bubble" is implemented by the editor occupying the bubble element itself (it inherits the bubble's box/styling), not by pixel-matching a sibling form; `rows=3` + `maxlength=4000` stay.
## Testing & Quality
- Unit: `tests/unit/test_frontend_edit_message.py` green (`uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov`).
- Coverage: **>90%** on `app/` (unchanged code — the gate must stay green).
- Lint/types: `uv run ruff check . && uv run pyright` (the JS is not linted by ruff — the unit pins are the JS gate).
## Completion Criteria
- [ ] In a live browser (or the E2E from task 03): clicking Edit shows the form inside the visible bubble — no `display:none`, no shrunken sibling box.
- [ ] Cancel / `Escape` restore the rendered markdown + the attachment; the record is unchanged.
- [ ] `uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov` green; `uv run pytest` green; ruff + pyright clean.
- [ ] No behavior change in `runTurn` / `retryLastTurn` / `handleSend` (byte-unchanged pins green).
@@ -0,0 +1,25 @@
# Task 02 — the edit-form submit label reads "Send", not "Save"
**Phase:** `170_edit_bubble_sizing` · **Source:** `TODO.md:3` — "Also the save button should read "send" rather than "save" because you're not saving it you're re-sending the message."
**Story:** n/a
## Objective
The edit form's submit button reads **Send** — the action re-sends the question (it never persists a standalone note, A10 stateless holds), so "Save" was a lie. The label matches the composer's `#send-label` "Send" and the "Cancel" sibling; nothing else in the flow changes.
## Work
1. `frontend/assets/app.js` L1238 — `editSaveBtn.textContent = "Save";` → `editSaveBtn.textContent = "Send";`. The `edit-save` class, the `type="submit"` wiring, the Ctrl/Cmd+Enter submit, and the whole submit handler are untouched.
2. `tests/unit/test_frontend_edit_message.py` L348 — the pin `assert 'editSaveBtn.textContent = "Save";' in body` → `"Send"`; keep the surrounding pin block (the `editSaveBtn.type` / `editSaveBtn.className` pins) and align the module docstring's "Save/Cancel" phrasing (L15) to "Send/Cancel" where it names the **edit form** (the docstring lines that name the composer's send flow stay — that control was always "Send").
3. `tests/e2e/test_edit_and_resend.py` — the flow drives the edit submit by class (`page.click(".edit-form .edit-save")` L337) — **no code change**. Align the module docstring / comments that name the edit form's "Save" action (L29, L32, L51, L79, L255, L341, L383, L510) to "Send"; the composer's `#send-label` "Send" assertions (L312, L533) are a different control and stay byte-identical.
- ASSUMPTION: A2 (phase-locked) — the label is "Send" (capital S, matching the composer's `#send-label` and "Cancel"); only the edit-form label changes — the phase-168 "Save settings" button (a different control) keeps its name.
## Testing & Quality
- Unit: `tests/unit/test_frontend_edit_message.py` green.
- E2E (regression, in isolation): `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` — the phase-159 suite now runs the flow through the "Send" label (the click is class-based, so the suite is code-identical; the docstrings are aligned).
- Coverage: **>90%** on `app/` (unchanged).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] The edit form's submit button reads "Send" in the live UI; "Save" appears nowhere in the edit form.
- [ ] `uv run pytest tests/unit/test_frontend_edit_message.py -v --no-cov` green; `uv run pytest tests/e2e/test_edit_and_resend.py -v --no-cov` green in isolation.
- [ ] The phase-168 "Save settings" button (settings view) is byte-untouched.
@@ -0,0 +1,44 @@
# Task 03 — `tests/e2e/test_edit_bubble_sizing.py`: the in-bubble editor, proven in a browser
**Phase:** `170_edit_bubble_sizing` · **Source:** `TODO.md:3` — "the edit bubble is tiny compared to the message bubble … It should stay the same size as the user message bubble. Also the save button should read "send" rather than "save" …"
**Story:** n/a
## Objective
The phase's dedicated Playwright suite (AGENTS rule 4/9): the editor opens **inside** the user bubble with the bubble's dimensions, the submit label reads "Send", and the whole phase-159 flow (trim + re-ask, cancel restore, the "Question too long" driver case) works through the new housing. Then the regression set, the full gates, and the one atomic commit.
## Work
1. `tests/e2e/test_edit_bubble_sizing.py` (new) — copy the module-scoped app fixture from `tests/e2e/test_edit_and_resend.py` verbatim (mock LLM, `BOR_ENVIRONMENT=e2e`, the E2E admin/secret pins) and change exactly: the module docstring (what the suite proves + the Source line `TODO.md:3` — the "in-bubble editor + Send label" contract) and the test list below. DB isolation: this suite writes no `app_settings` / theme rows; the `saved_chats` rows it creates follow the phase-159 suite's convention (no extra teardown needed beyond what that fixture already does — if the fixture lacks a `saved_chats` cleanup, add the same one it would need, mirroring `test_chat_persistence.py`).
2. Tests (names pinned; helpers `login(page, app_url)` from `tests/e2e/auth_helpers.py` where a gate is crossed — the chat is public, so most tests need no login, exactly like the phase-159 suite):
- `test_editor_renders_inside_the_bubble` — send a fixture question, wait for the grounded answer (the mock-LLM settle convention), capture the user bubble's `bounding_box()` + computed `background-color`, click Edit: the form is a **child of the bubble** (`bubble.evaluate("el => el.querySelector('.edit-form') !== null")`), the bubble's computed `display` is not `none`, its `background-color` is unchanged (the `--brand` fill), the textarea prefills the EXACT question, and `.is-editing` is on the bubble.
- `test_editor_box_keeps_the_bubble_dimensions` — the pre-edit bubble `bounding_box()` vs the open-editor bubble: width is ≥ 95 % of the pre-edit width (the "no longer tiny" pin — the old shrunken sibling form failed this by construction) and height is ≥ the pre-edit height (the box grows, never collapses); at 375 px the bubble has no horizontal overflow (§7.1 reading column).
- `test_submit_label_reads_send` — `.edit-save`'s text is "Send" (and not "Save"); `.edit-cancel`'s is "Cancel".
- `test_trim_and_send_reasks_in_place` — the phase-159 flow through the new label: append " in one line", click `.edit-form .edit-save` (the "Send" button): the truncated shape `[user(edited), brain]` (the `bor.chat.v1` localStorage pin, the phase-159 convention), the re-ask streams the grounded answer, no duplicated question, the UNSSENT composer draft survives.
- `test_cancel_restores_the_bubble` — open the editor, Cancel: the rendered markdown is back (the question text is visible in the bubble, the form is gone, `.is-editing` off), the `conversation` record is byte-identical; a second pass with `Escape` (a fresh turn) pins the same restore through the keydown path.
- `test_failed_turn_still_editable` — the phase-114 driver case (the 4000+ char question → the "Question too long — trim it and re-ask" error frame, the phase-120 failed record): the editor opens inside the bubble, trimming the question to the cap + "Send" streams the grounded answer (the failed record is replaced).
3. **Regression set, all in isolation** (`--no-cov`): `test_edit_and_resend.py` (the phase-159 suite — now running against the in-bubble editor + the "Send" label), `test_retry_answer.py`, `test_failed_turn_retry.py`, `test_chat_persistence.py`.
4. **Full gates** (AGENTS rule 9, all green before the commit):
```bash
uv run pytest
uv run pytest --cov=app --cov-report=term-missing # >90% on app/
uv run pytest tests/e2e/test_edit_bubble_sizing.py -v --no-cov
uv run ruff check . && uv run pyright
```
5. **Docs & close-out:** confirm `README.md`'s edit-the-last-question note (if any names the "Save" label or the hidden-bubble behavior — phase 159's README section, grep `edit` there) is aligned to the in-bubble editor + "Send"; move the phase directory:
```bash
git mv .agents/phases/todo/170_edit_bubble_sizing .agents/phases/complete/170_edit_bubble_sizing
```
and commit (AGENTS rule 8, `--no-gpg-sign`):
```bash
git add frontend/ tests/ README.md .agents/phases/ && git commit --no-gpg-sign -m "feat(chat): edit the last question inside its bubble and re-send it with Send"
```
## Testing & Quality
- The suite passes **in isolation** (`-v --no-cov`).
- Unit pins already exist from tasks 01/02 — this file adds the browser contract only; if it needs a behavior the UI lacks, fix the UI in this task and say so in the commit body.
- Coverage: **>90%** on `app/` after everything (the `--cov` runs exclude E2E by the repo's config; the gate is the unit+integration number).
## Completion Criteria
- [ ] `test_editor_box_keeps_the_bubble_dimensions` is green: the open editor's bubble keeps the pre-edit width (≥ 95 %) and never collapses.
- [ ] The "Send" label, the in-bubble housing, the cancel restore, and the failed-turn driver case are all green in isolation.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] Phase directory moved to `.agents/phases/complete/170_edit_bubble_sizing/` and one atomic `--no-gpg-sign` Conventional Commit exists (verify with `git log -1 --stat`).
@@ -0,0 +1,46 @@
# Phase 171 — Admin settings page: row rhythm, distinct group bands, a properly disabled Save
**Source:** `TODO.md` L4 — "In the admin settings page: The search settings text box overlaps settings table. The settings category headers are not differentiated from the settings themselves, making them confusing at a glance. The save settings and clear all overrides are smushed right up against the settings table. The save settings shows a loading cursor when moused over rather than just being disabled."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-10)
**Context:** `frontend/index.html` — `#view-settings` (L1499), `#settings-content` (L1535, a plain block — its children `#settings-disabled`, `#settings-status`, `#settings-error`, `.settings-search` (L1562), the `.table-wrap` card (L1576, `#settings-tbody` created by settings.js), and `.settings-actions` (L1607: `#settings-save` disabled by default + `#settings-clear-all` + `#settings-dirty`) stack with **zero vertical rhythm** — the root of the "overlap" and the "smushed" complaints), `frontend/assets/settings.js` (637 lines — the group-row render L319–335: `tr.settings-group-row` + `th[colspan=4][scope=colgroup]` with the group name, the client-side search filter, the §7.4 Save lifecycle), `frontend/assets/styles.css` — `.settings-shell` (L4660–4665: `display:flex; flex-direction:column; gap: 1.25rem;`), `.settings-group-row th` (L4684–4690: `background: var(--surface)` — the **same** fill as the `.table-wrap` card (L2825–2831) → the band is effectively invisible, the "not differentiated" complaint), the sticky-th tint rule `.table-wrap table thead th { background: var(--brand-soft); }` (L2841–2848, the solid-tint precedent), `.settings-search` (L4780–4786), `.settings-actions` (L4804–4809), `#settings-save` (L4810–4824, incl. `:disabled { opacity: 0.6; cursor: wait; }` L4824 — the loading-cursor complaint), `.settings-reset:disabled { opacity: 0.5; cursor: default; }` (L4775 — the house idiom), the mobile media queries (L6396–6400: `.settings-search` / `.settings-actions` `flex-direction: column`, `#settings-save` `width: 100%`), `tests/unit/test_settings_frontend_view.py` (the phase-168 static pins, incl. the L561 media-query pin), `tests/e2e/test_admin_settings_panel.py` (the phase-168 suite — the regression set).
## Objective
The four layout complaints on the Settings page — the search box overlapping the table card, the group headers indistinguishable from the rows, the Save / Clear row smushed against the table, and the disabled Save showing a loading cursor — are fixed with **CSS only** (plus the pin updates): one rhythm rule, one band rule, one cursor value. The phase-168 contract (the gate, the catalogue, the §7.4 Save lifecycle, the API, the bounded scroll + sticky header) stays behavior-identical — no `app/` code, no JS logic change.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the rhythm (task 01).** `#settings-content` becomes `display: flex; flex-direction: column; gap: 1.25rem;` — the **same** token `.settings-shell` already uses (L4664). The children (the disabled banner, the status line, the error line, the search row, the `.table-wrap` card, the `.settings-actions` row) then stack with one uniform gap: the search box clears the table card (the "overlap" dies) and the Save / Clear row is separated from the table (the "smushed" dies) in the same change. The `min-height` rules on `#settings-status` / `#settings-error` (L4849–4862) already reserve their lines — the gap is additive and uniform, no per-element margins are introduced.
- **D2 — the band (task 02).** `.settings-group-row th` gets a band that is distinct from the row cells at a glance: `background: var(--brand-soft)` — the **same solid tint** the sticky `thead` rule already uses (L2847; B3: no new palette variable), `font-weight: 700` (the 0.82rem / uppercase / letter-spacing stay), and `border-top: 1px solid var(--line)` so a band never merges with the card's top edge or a previous band. The row cells stay on the card's `--surface`. The group text stays `--ink` on `--brand-soft` — the pairing the sticky-th rule already documents. Static, text-only, no animation (nothing for `prefers-reduced-motion` to still).
- **D3 — the cursor (task 03).** `#settings-save:disabled` → `cursor: default` — the `.settings-reset:disabled` idiom (L4775); "just disabled", per the owner. The §7.4 in-flight state is the **dimmed** `:disabled` (`opacity: 0.6` stays) + `aria-busy` — the `cursor: wait` was never part of the contract; it was the "loading cursor on hover" the owner saw.
- **NOT touched:** `settings.js` (the render / filter / save / reset / clear-all logic), `app/api/admin_settings.py`, the sign-in gate, the nav reveal, the `.table-wrap` bounded-scroll + sticky-th contract (L2825–2848), the `#settings-status` / `#settings-error` regions, the `restart required` flag, the mobile media queries (L6396–6400 — they compose with the new gap and stay byte-identical), `app/` server code.
## Dependencies
- `168_admin_settings_panel` (in flight — the code + the E2E suite are committed; only its directory move + final commit are pending) — the Settings view this phase polishes. Note: the abandoned `169` settings status card never landed — there is nothing to coordinate with.
## Tasks
1. `01_content_vertical_rhythm.md` — `#settings-content` becomes a flex column with the shell's `1.25rem` gap (fixes the search/table overlap **and** the smushed action row); the pin added.
2. `02_group_band_distinction.md` — `.settings-group-row th` gets the `--brand-soft` band + 700 weight + top hairline, so groups read at a glance.
3. `03_disabled_save_cursor.md` — `#settings-save:disabled` → `cursor: default` (the house idiom), killing the loading cursor.
4. `04_e2e_settings_layout.md` — `tests/e2e/test_admin_settings_layout.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_settings_frontend_view.py` (the phase-168 static pin set, extended in tasks 01/02/03): the `#settings-content` flex-column + gap rule, the band's `--brand-soft` background + weight + top border, `#settings-save:disabled` `cursor: default` (and `cursor: wait` absent from the settings rules), plus the existing pins (the no-CDN pin, the `router.js` mapping, the media-query pin L561) staying green.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_admin_settings_layout.py`, run **in isolation** — the four layout contracts, proven in a browser.
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] The search box and the table card do not intersect; the Save / Clear row is visibly separated from the table (the E2E bounding-box pins, at 1440 px and 375 px).
- [ ] A settings group band is visually distinct from its rows (computed background ≠ the row cells' background; weight 700) and no band merges with the card edge.
- [ ] A disabled **Save settings** shows `cursor: default` (not `wait`); the §7.4 dimmed-`aria-busy` in-flight state is intact.
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` (phase 168) + `test_wide_desktop_column.py` green in isolation.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-10, roadmap confirmation)
- **A3 — the band tint is `--brand-soft`** (the solid tint the sticky `thead` already uses — B3: no new palette variable) + a `--line` top hairline + weight 700; static, no animation.
- **A4 — the rhythm reuses the shell's existing `gap: 1.25rem`** — no new spacing value; the existing mobile media queries (L6396–6400) keep working unchanged.
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): settings page layout — row rhythm, distinct group bands, a properly disabled Save"
```
@@ -0,0 +1,31 @@
# Task 01 — `#settings-content` gets the shell's vertical rhythm (overlap + smushing die)
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "The search settings text box overlaps settings table. … The save settings and clear all overrides are smushed right up against the settings table."
**Story:** n/a
## Objective
The `#settings-content` block becomes a flex column with the shell's existing `gap: 1.25rem`, so its children — the disabled banner, the status line, the error line, the search row, the `.table-wrap` card, the `.settings-actions` row — stack with one uniform gap. One change kills both complaints: the search box clears the table card (the "overlap") and the Save / Clear row separates from the table (the "smushed").
## Work
1. `frontend/assets/styles.css` — add ONE rule (place it directly after the `.settings-shell` rule, L4660–4665, with a comment citing the phase-171 task + TODO L4):
```css
#settings-content {
display: flex;
flex-direction: column;
gap: 1.25rem;
}
```
No per-element margins anywhere; the `min-height` rules on `#settings-status` / `#settings-error` (L4849–4862) stay (they reserve the lines; the gap is additive and uniform). Verify by reading, not by adding: `.settings-search` (L4780–4786) and `.settings-actions` (L4804–4809) get **no** new margins — the gap is the only spacing source.
2. `tests/unit/test_settings_frontend_view.py` — add the house-style CSS pin: the `#settings-content` rule exists with `display: flex`, `flex-direction: column`, and `gap: 1.25rem` (the same pin family as the L561 media-query pin — parse `styles.css`, assert the three declarations in one rule block).
- ASSUMPTION: A4 (phase-locked) — the rhythm reuses the shell's existing `gap: 1.25rem` token; no new spacing value; the existing mobile media queries (styles.css L6396–6400: `.settings-search` / `.settings-actions` `flex-direction: column; align-items: stretch;` + `#settings-save { width: 100%; }`) keep working unchanged.
## Testing & Quality
- Unit: `tests/unit/test_settings_frontend_view.py` green (`uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov`).
- Coverage: **>90%** on `app/` (unchanged code — the gate must stay green).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] In the live UI (or the task-04 E2E): the search input's box and the `.table-wrap` card's box do not touch — a visible, uniform gap between them, and the same gap above the `.settings-actions` row.
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; `uv run pytest` green; ruff + pyright clean.
- [ ] No per-element margin was added (the diff touches exactly one CSS rule + the pin); the mobile stacking (L6396–6400) is byte-unchanged.
@@ -0,0 +1,28 @@
# Task 02 — the group bands are distinct from the rows at a glance
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "The settings category headers are not differentiated from the settings themselves, making them confusing at a glance."
**Story:** n/a
## Objective
`.settings-group-row th` — today `background: var(--surface)`, the **same** fill as the `.table-wrap` card, i.e. invisible — becomes a band that reads as a category: the `--brand-soft` tint (the solid tint the sticky `thead` rule already uses), weight 700, and a `--line` top hairline so a band never merges with the card edge or a previous band.
## Work
1. `frontend/assets/styles.css` — `.settings-group-row th` (L4684–4690), three value changes in the existing rule (no new selector):
- `background: var(--surface);` → `background: var(--brand-soft);` (B3: no new palette variable — the exact tint `.table-wrap table thead th` already documents at L2841–2848);
- add `font-weight: 700;` (the `font-size: 0.82rem; text-transform: uppercase; letter-spacing: 0.04em;` stay);
- add `border-top: 1px solid var(--line);` (the rule's existing `border-bottom` from the `.settings-table th` family, L4676–4681, stays — a band is delimited above AND below, and can never merge with the card's top edge or the band before it);
- update the contrast comment (L4683): the text stays `--ink` on `--brand-soft` — the pairing the sticky-th rule already documents (the existing "15.3:1, AA" comment applies to `--surface`; recompute / restate the ratio on `--brand-soft` in the comment — the `:root` `--brand-soft` token carries its own documented pairing).
2. `tests/unit/test_settings_frontend_view.py` — add the pin: the `.settings-group-row th` rule carries `background: var(--brand-soft)`, `font-weight: 700`, and `border-top: 1px solid var(--line)` (same pin family as task 01's).
- ASSUMPTION: A3 (phase-locked) — the band = `--brand-soft` (the sticky-th tint, no new palette variable) + `--line` top hairline + weight 700; static, text-only, no animation (nothing for `prefers-reduced-motion` to still).
## Testing & Quality
- Unit: `tests/unit/test_settings_frontend_view.py` green.
- Coverage: **>90%** on `app/` (unchanged code).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] In the live UI (or the task-04 E2E): each group band is visibly tinted (`--brand-soft`), bold, and hairline-delimited above; the row cells stay on the card's `--surface` — the two are distinguishable at a glance.
- [ ] No band merges with the card's top edge or the band above it (the top hairline is painted).
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; ruff + pyright clean.
- [ ] `settings.js` is byte-unchanged (the band is pure CSS — the group-row render L319–335 is untouched).
@@ -0,0 +1,23 @@
# Task 03 — a disabled Save settings is just disabled (no loading cursor)
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "The save settings shows a loading cursor when moused over rather than just being disabled."
**Story:** n/a
## Objective
`#settings-save:disabled` stops painting `cursor: wait` (the loading cursor the owner saw on hover) and takes `cursor: default` — the house idiom `.settings-reset:disabled` already uses. "Just disabled": the §7.4 in-flight state stays the **dimmed** `:disabled` (`opacity: 0.6`) + `aria-busy`; the cursor was never part of the contract.
## Work
1. `frontend/assets/styles.css` L4824 — `#settings-save:disabled { opacity: 0.6; cursor: wait; }` → `#settings-save:disabled { opacity: 0.6; cursor: default; }`. Keep `opacity: 0.6` (the §7.4 dim) and the `#settings-save:hover:not(:disabled)` rule (L4823) byte-unchanged. Update the adjacent comment block (L4809–4810, "the §7.4 in-flight state is the dimmed :disabled + aria-busy") to state the cursor contract explicitly: `cursor: default` when disabled (the `.settings-reset:disabled` L4775 idiom) — the wait cursor was never part of §7.4.
2. `tests/unit/test_settings_frontend_view.py` — add the pin: `#settings-save:disabled` carries `cursor: default`, and `cursor: wait` is **absent** from the settings-page rules (a negative grep over the `.settings*` / `#settings-*` rule blocks — the house negative-pin idiom). Note: `.edit-save:disabled` (styles.css L1454) keeps its own `cursor: wait` — it is the chat edit form's in-flight state (phase 159/170), a different control; the negative pin is scoped to the settings rules only, and the test must not fail on the `.edit-save` rule.
- ASSUMPTION: none — the owner's words fix the target ("just being disabled"); the idiom choice (`default`, mirroring L4775) is the house convention, not a guess.
## Testing & Quality
- Unit: `tests/unit/test_settings_frontend_view.py` green.
- Coverage: **>90%** on `app/` (unchanged code).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] In the live UI (or the task-04 E2E): an untouched **Save settings** (no dirty rows) is `disabled` and its computed `cursor` is `default` — hovering shows no wait/loading cursor; after a row is edited the button enables and its cursor is `pointer` again (the §7.4 lifecycle, phase 168, intact).
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; ruff + pyright clean.
- [ ] The diff is exactly one CSS value + one comment + the pin; `settings.js` and `app/` are byte-unchanged.
@@ -0,0 +1,43 @@
# Task 04 — `tests/e2e/test_admin_settings_layout.py`: the four layout contracts, proven in a browser
**Phase:** `171_admin_settings_layout` · **Source:** `TODO.md:4` — "In the admin settings page: The search settings text box overlaps settings table. The settings category headers are not differentiated … The save settings and clear all overrides are smushed right up against the settings table. The save settings shows a loading cursor when moused over rather than just being disabled."
**Story:** n/a
## Objective
The phase's dedicated Playwright suite (AGENTS rule 4/9): the admin signs in, and the four layout contracts hold in a real browser — the search row clears the table card, the action row is gapped from the table, the group bands are distinct, and a disabled Save is just disabled. Then the regression set, the full gates, and the one atomic commit.
## Work
1. `tests/e2e/test_admin_settings_layout.py` (new) — copy the module-scoped app fixture from `tests/e2e/test_admin_settings_panel.py` verbatim (the mock LLM, `BOR_ENVIRONMENT=e2e`, `BOR_STATIC_DIR`, the E2E admin password/secret, the leak guards, the module's `BOR_SETTINGS_DB=1` opt-in + the `TRUNCATE app_settings` before/after — this suite never writes overrides, so the truncation is belt-and-braces, the same idiom) and change exactly: the module docstring (what the suite proves + the Source line `TODO.md:4` — the four layout contracts) and the test list below.
2. Tests (names pinned; `login(page, app_url)` from `tests/e2e/auth_helpers.py`; the suite drives the admin gate exactly like the phase-168 suite):
- `test_search_row_is_clear_of_the_table` — at 1440 px and 375 px: `#settings-filter`'s `bounding_box()` and the `.table-wrap` card's box do not intersect (the vertical gap between the search row's bottom and the card's top is > 0), and the card keeps its bounded scroll + sticky `thead` (the computed `position: sticky` on the `thead th` — the phase-156/168 contract intact under the new rhythm).
- `test_actions_row_is_gapped_from_the_table` — at 1440 px and 375 px: `.settings-actions`'s box does not intersect the `.table-wrap` card's box (the "smushed" pin — the gap between the card's bottom and the action row's top is > 0); `#settings-save` and `#settings-clear-all` are both reachable (no clipping at either viewport).
- `test_group_bands_are_distinct_from_the_rows` — after `login()`: at least the `Retrieval & honesty` and `LLM (aipi)` `tr.settings-group-row` bands render; a band's computed `background-color` ≠ the `.table-wrap` card's computed background (the `--surface` row-cell fill) — the "not differentiated" pin — and the band's computed `font-weight` is ≥ 700; a band's computed `border-top-width` is ≥ 1 px (the hairline — no merge with the card edge or the previous band).
- `test_disabled_save_is_just_disabled` — before any edit: `#settings-save` is `disabled` and its computed `cursor` is `default` (NOT `wait` — the loading-cursor pin); edit one row (a `type=search` filter + a value change, the phase-168 dirty-row flow): the button enables (computed `cursor` `pointer`) and dims back to `default` after a **Save settings** settles (the §7.4 lifecycle, phase 168, intact under the new cursor).
- `test_settings_page_layout_and_accessibility` — no horizontal page overflow at 375 px and 1440 px (§7.1: the table is ≥80 % of the container, not a skinny list); keyboard tab order still reaches filter → first row input → **Save settings** → **Clear all overrides** (a `page.keyboard` walk asserting focus lands on the real controls — the new rhythm must not have broken the phase-168 keyboard contract); zero external requests during the whole test (`page.on("request")` — rule 6).
3. **Regression set, all in isolation** (`--no-cov`): `test_admin_settings_panel.py` (the phase-168 suite — the gate, the save-live contract, the masking, the reset / clear-all, now under the new layout), `test_wide_desktop_column.py` (the 72rem container contract, the Settings view included).
4. **Full gates** (AGENTS rule 9, all green before the commit):
```bash
uv run pytest
uv run pytest --cov=app --cov-report=term-missing # >90% on app/
uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov
uv run ruff check . && uv run pyright
```
5. **Docs & close-out:** confirm `README.md`'s Settings page section (phase 168's addition) needs no layout wording change (it describes the contract, not the pixels — grep `Settings` there); move the phase directory:
```bash
git mv .agents/phases/todo/171_admin_settings_layout .agents/phases/complete/171_admin_settings_layout
```
and commit (AGENTS rule 8, `--no-gpg-sign`):
```bash
git add frontend/ tests/ README.md .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): settings page layout — row rhythm, distinct group bands, a properly disabled Save"
```
## Testing & Quality
- The suite passes **in isolation** (`-v --no-cov`) and leaves `app_settings` empty (the `TRUNCATE` idiom — a leftover row is the cross-suite poison phase-168 D6 exists to prevent).
- The pins already exist from tasks 01/02/03 — this file adds the browser contract only; if it needs a behavior the CSS lacks, fix the CSS in this task and say so in the commit body.
- Coverage: **>90%** on `app/` after everything (the `--cov` runs exclude E2E by the repo's config; the gate is the unit+integration number).
## Completion Criteria
- [ ] All four layout pins are green in isolation: search clear of the table, action row gapped, bands distinct (tint + weight + hairline), disabled Save `cursor: default`.
- [ ] `app_settings` is empty after the suite and the phase-168 regression suite is green in isolation.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] Phase directory moved to `.agents/phases/complete/171_admin_settings_layout/` and one atomic `--no-gpg-sign` Conventional Commit exists (verify with `git log -1 --stat`).
@@ -0,0 +1,50 @@
# Phase 172 — Thinking level: a pi-style, operator-tunable LLM reasoning knob
**Source:** `TODO.md` L3 — "Brain of Reese needs to support thinking levels like pi does. should be a free-form text entry like chat model. Default should be 'low'. Update .env and admin settings."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-10)
**Context:** `app/config.py` — the LLM section (L97–125: `llm_base_url`, `llm_api_key`, `llm_chat_model` L101, `llm_embed_model`, `llm_summary_model` L107, `stream_thinking` L112, `llm_retries` / `llm_retry_delay` / `llm_timeout` L122 — the house pattern this setting joins), `.env.example` L57–65 (the LLM block — `BOR_LLM_CHAT_MODEL=turbo` L60), `app/core/settings_catalog.py` L137–161 (the `LLM (aipi)` catalogue group; the completeness pin in `tests/unit/test_settings_catalog.py` — `set(CATALOG) == set(Settings.model_fields)` — makes the new field a build failure until it has a `SettingMeta`), `app/core/settings_overlay.py` (the generic DB-over-env overlay — a plain `str` field rides it with zero extra code: `PUT` stores the raw string, `invalidate_settings()` re-reads), `app/rag/llm.py` — `LLMClient.chat()` (L446, the one-shot `chat.completions.create` L416, `temperature=0.2`, 2048 tokens) and `LLMClient.chat_stream()` (L511, the kwargs dict L592–600: `model` / `messages` / `temperature=0.4` / `max_tokens` / `stream`), `app/api/chat.py` — the per-turn log line (L1268–1291: `…thinking_chars=%d tool_calls=%d…`), `frontend/assets/settings.js` (the phase-168 generated panel — **no code change here**: a new `str` catalogue entry renders as a free-form text row automatically, exactly like `llm_chat_model`), `tests/unit/test_config.py`, `tests/unit/test_llm_client.py` (the mocked-SDK-client family), `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` (the set/reset/live-apply contract), `tests/e2e/test_admin_settings_panel.py` (the phase-168 suite — the pattern to copy).
## Objective
The operator can pick the model's thinking level — pi-style, free-form text, default `low` — from `.env` (`BOR_LLM_THINKING_LEVEL`) or the admin Settings panel, and the value rides every chat-completions request to aipi as the OpenAI-standard `reasoning_effort` parameter, so the same knob works as the model's reasoning support evolves. The empty value is the off switch (the parameter is not sent at all).
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — one `Settings` field (task 01).** `llm_thinking_level: str = "low"` in the LLM section of `app/config.py` (placed directly after the `llm_summary_model` block, before `stream_thinking` — model-identity settings stay together). Free-form, exactly like `llm_chat_model` (A4: **no allowlist, no validator** — any non-empty text is accepted; the endpoint is the source of truth). The docstring names the env var (`BOR_LLM_THINKING_LEVEL`), the default, the wire mapping (D2) and the empty-off rule (D3). `.env.example` gains `BOR_LLM_THINKING_LEVEL=low` directly under `BOR_LLM_CHAT_MODEL=turbo` (L60) with a one-line comment (the house style of L61–63).
- **D2 — the wire (task 03).** The value is sent as a **top-level chat-completions parameter** `reasoning_effort` via the OpenAI SDK's `extra_body` (`extra_body={"reasoning_effort": "<value>"}` — the SDK merges `extra_body` into the request JSON body). Applied in BOTH `LLMClient.chat()` (the one-shot: document summaries, KB overview) and `LLMClient.chat_stream()` (the chat turn) — A3. The embeddings path (`_post_embeddings` / `embed` / `embed_one`) is **untouched**. `ASSUMPTION: A1 — aipi accepts the OpenAI-standard `reasoning_effort` top-level parameter (pi's `reasoningEffort`); if aipi exposes a different key (e.g. `chat_template_kwargs`), only task 03's key changes — the rest of the phase is key-agnostic by design.**
- **D3 — the off switch (tasks 03/04).** The **trimmed** value decides: non-empty → the parameter is sent with the trimmed value; empty/whitespace → `extra_body` is absent from the request (byte-identical to today's payload). Nothing is sent as `"off"` or `null` — absence is the off.
- **D4 — the admin row (task 02).** One `SettingMeta("llm_thinking_level", "LLM (aipi)", "Thinking level")` entry in `CATALOG` (placed after `llm_summary_model` L150–151, before `llm_retries` L156). Nothing else: the panel (`settings.js`), the `GET` catalogue, the `PUT` / `DELETE` routes, and the DB-over-env overlay are all **generic over the catalogue** — the row, its free-form text input, its save/reset and its live-apply fall out with zero extra code (A5). The completeness pin is the enforcement: a missing entry fails the build.
- **D5 — the per-turn line (task 04).** The per-turn log line (PLAN §9, `app/api/chat.py` L1268) gains `thinking=<value>` immediately before the existing `thinking_chars=%d` — the effective (overlay-resolved) value, trimmed, or `-` when empty, so "which level did this turn run at" stays one `grep` away (AGENTS rule 10). No `query_log` schema change (the table is retrieval-scoped; the line is the home of turn knobs).
- **NOT touched:** `frontend/assets/settings.js` (generated from the catalogue), `frontend/assets/styles.css`, the `app_settings` table shape, `app/core/settings_overlay.py` (generic), the embeddings path, `stream_thinking` (the phase-17 kill-switch for `thinking` SSE events — different concern, stays as-is), every other setting.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; this phase builds on the committed phase-168/171 settings machinery)
## Tasks
1. `01_setting_and_env.md` — `Settings.llm_thinking_level: str = "low"` + the `CATALOG` row (the completeness pin keeps the build green) + `BOR_LLM_THINKING_LEVEL=low` in `.env.example`; the config/catalog unit tests.
2. `02_admin_row_contract.md` — the integration set / reset / live-apply contract for the new key (stored raw, live without restart, empty storable as the off switch).
3. `03_llm_extra_body_passthrough.md` — `extra_body={"reasoning_effort": …}` on `chat()` + `chat_stream()` when trimmed-non-empty; absent when empty; the mocked-client unit tests.
4. `04_per_turn_log_field.md` — `thinking=<value>` (or `-`) on the per-turn log line; the log-line test.
5. `05_e2e_admin_settings_row.md` — `tests/e2e/test_thinking_level.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` — the default is `low`, the env override works, an empty env value yields an empty string (no coercion, no allowlist — D3/A4). `tests/unit/test_settings_catalog.py` — the completeness pin stays green (the new field has its `SettingMeta`; group/label/flags correct). `tests/unit/test_llm_client.py` — `chat()` and `chat_stream()` send `extra_body={"reasoning_effort": <trimmed value>}` when the setting is non-empty, send **no** `extra_body` when it is empty/whitespace, and the embeddings path is unaffected (the mocked-SDK-client family).
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — the row appears in `GET /api/admin/settings` (group, `str`, effective `low`); `PUT` stores the raw string, the empty value is storable (the off switch), the overlay resolves live without a restart; `DELETE` resets to env.
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_thinking_level.py`, run **in isolation** — the row is visible in the `LLM (aipi)` group with default `low`, is a free-form text input (type `high`, save, survives reload), reset-to-env restores `low`, and the `test_admin_settings_panel.py` regression set stays green.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest -k "config and thinking" -v` shows the default `low` + the env override; `.env.example` contains `BOR_LLM_THINKING_LEVEL=low` under `BOR_LLM_CHAT_MODEL`.
- [ ] `GET /api/admin/settings` (admin) lists `llm_thinking_level` in the `LLM (aipi)` group (`str`, default `low`); `PUT` / `DELETE` round-trip per the overlay contract.
- [ ] With the setting non-empty, a `chat_stream()` / `chat()` request carries `reasoning_effort` (unit-asserted via the mocked client); with it empty, the request kwargs contain no `extra_body` (byte-identical to today).
- [ ] The per-turn log line carries `thinking=<value>` (`-` when empty) next to `thinking_chars`.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` + `test_admin_settings_layout.py` (phase 171) green in isolation.
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(llm): operator-tunable thinking level (BOR_LLM_THINKING_LEVEL, default low) sent as reasoning_effort`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-10, roadmap confirmation)
- **A1 — the wire key is `reasoning_effort`** (the OpenAI-standard top-level chat-completions parameter, pi's `reasoningEffort`), sent via the SDK's `extra_body`. If aipi exposes a different key, only task 03 changes.
- **A2 — the off switch is absence:** empty/whitespace value → the parameter is not sent at all; the default `low` is always sent.
- **A3 — scope:** the parameter rides BOTH `chat_stream()` (the chat turn) and `chat()` (the one-shot summaries / KB overview); the embeddings path is untouched.
- **A4 — free-form, no allowlist:** any non-empty text is accepted (like `llm_chat_model`); no validator, no value coercion.
- **A5 — no frontend code change:** the panel is generated from the catalogue; the row appears automatically as a free-form text input.
- **A6 — the per-turn line** gains `thinking=<trimmed value>` (`-` when empty) immediately before `thinking_chars`; no `query_log` schema change.
@@ -0,0 +1,49 @@
# Task 01 — The `llm_thinking_level` setting + the catalogue row + `.env.example`
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "Brain of Reese needs to support thinking levels like pi does. should be a free-form text entry like chat model. Default should be 'low'. Update .env and admin settings."
**Story:** n/a
## Objective
The new setting exists in `app/config.py` (default `low`, free-form like `llm_chat_model`), in the admin catalogue (so the build's completeness pin stays green — a field without a `SettingMeta` fails the build, and the per-task gate runs the full suite), and in `.env.example`, so `BOR_LLM_THINKING_LEVEL` is a first-class, documented knob before anything consumes it.
## Work
1. `app/config.py` — in the LLM section, **directly after** the `llm_summary_model` block (L103–107) and **before** `stream_thinking` (L108), add ONE field, house docstring style (env var named, default, wire mapping, off rule):
```python
#: The model's thinking level (phase 172, ``BOR_LLM_THINKING_LEVEL``) —
#: pi-style reasoning effort, free-form text exactly like
#: ``llm_chat_model`` (no allowlist — the endpoint is the source of
#: truth). Default ``low``. Rides every chat-completions request to
#: aipi as the top-level ``reasoning_effort`` parameter
#: (``app/rag/llm.py``); the EMPTY value is the off switch — the
#: parameter is not sent at all.
llm_thinking_level: str = "low"
```
No validator, no coercion — a plain `str` (ASSUMPTION A4, phase-locked). Verify by reading: `llm_chat_model` (L101) keeps its shape byte-identical.
2. `.env.example` — in the LLM block (L57–65), directly under `BOR_LLM_CHAT_MODEL=turbo` (L60), add:
```
BOR_LLM_THINKING_LEVEL=low # the model's thinking level (pi-style; free-form, rides reasoning_effort); empty = the parameter is not sent
```
(one-line comment, the house style of L61–63; `#`-prefixed optional vars stay commented — this one is set, like L60).
3. `app/core/settings_catalog.py` — in the `LLM (aipi)` catalogue group, **directly after** the `llm_summary_model` entry (L150–151) and **before** `llm_retries` (L156), add ONE entry:
```python
"llm_thinking_level": SettingMeta(
"llm_thinking_level", "LLM (aipi)", "Thinking level"
),
```
Nothing else — `SECRETS` / `RESTART_REQUIRED` / `NON_OVERRIDABLE` are DERIVED from `CATALOG` (module docstring), so no other list changes. The row is plain: not `secret`, not `restart_required` (the LLM client reads the setting per request via `get_settings()` — live), overridable (the overlay is generic over `str` fields, `app/core/settings_overlay.py`).
4. `tests/unit/test_settings_catalog.py` — the completeness pin (`set(CATALOG) == set(Settings.model_fields)`) must stay green with the new pair; add the house-style assertion that `CATALOG["llm_thinking_level"]` sits in group `LLM (aipi)` with label `Thinking level` and is **not** in `SECRETS` / `RESTART_REQUIRED` / `NON_OVERRIDABLE` (the derived sets).
5. `tests/unit/test_config.py` — add the house-style cases: (a) the DEFAULT is `"low"` (no env set); (b) the env override — `BOR_LLM_THINKING_LEVEL=high` resolves to `"high"`; (c) the empty env value (`BOR_LLM_THINKING_LEVEL=`) resolves to `""` (no coercion to a default, no allowlist — the off switch stays expressible).
- ASSUMPTION: A4 (phase-locked) — free-form text, no allowlist/validator; A2 (phase-locked) — empty = off switch, so the setting must NOT strip/normalize away the empty value at load (it is trimmed only at the wire, task 03); A5 (phase-locked) — the panel row falls out of the catalogue with zero `settings.js` code.
## Testing & Quality
- Unit: `tests/unit/test_config.py` + `tests/unit/test_settings_catalog.py` green (`uv run pytest tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov`).
- Coverage: **>90%** on `app/` (the gate must stay green).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest -k "thinking" tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov` green: default `low`, env override `high`, empty stays `""`, the completeness pin holds with the new pair.
- [ ] `grep -n "BOR_LLM_THINKING_LEVEL" .env.example` shows the line under `BOR_LLM_CHAT_MODEL=turbo`.
- [ ] `GET /api/admin/settings` (admin) lists `llm_thinking_level` in the `LLM (aipi)` group (`str`, effective `low`) — the pin + a quick curl, or the task-02 integration test.
- [ ] `uv run pytest` green; ruff + pyright clean.
- [ ] no behavior change in completed work (no other `Settings` field touched; `settings.js` untouched).
@@ -0,0 +1,24 @@
# Task 02 — The admin row's API contract (set / reset / live-apply for `llm_thinking_level`)
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "… should be a free-form text entry like chat model. … Update .env and admin settings."
**Story:** n/a
## Objective
Prove, at the API/overlay level, that the new key behaves exactly like `llm_chat_model`: it appears in the catalogue, `PUT` stores the raw string and the overlay resolves it live (no restart), `DELETE` resets to env, and the effective value the LLM layer will read follows the overlay.
## Work
1. `tests/integration/test_admin_settings_api.py` — extend the existing key-parameterized patterns (read how the phase-168 suite picks a `str` key, e.g. `llm_chat_model`) with the house-style cases for `llm_thinking_level`: (a) `GET /api/admin/settings` → the item is present, group `LLM (aipi)`, `type str`, effective value `low` (no row in `app_settings`), `env_var` `BOR_LLM_THINKING_LEVEL`; (b) `PUT` with `{"value": "high"}` → 200, the row is stored **raw** in `app_settings` (env-native string form, D1 of the overlay), the response reports the new effective value; (c) `PUT` with an empty string `""` → 200 (the off switch is storable — the API stores as-is, no allowlist); (d) `DELETE /api/admin/settings/llm_thinking_level` → the row is gone, effective value back to the env default `low`; (e) the non-admin 403 gate still applies (the phase-168 router dependency — assert once, mirroring the existing suite).
2. `tests/integration/test_admin_settings_live.py` — add the live-apply case in the house pattern: `PUT llm_thinking_level=high` → `get_settings().llm_thinking_level` is `"high"` **without** a restart (the `invalidate_settings()` path); `DELETE` → back to `"low"`. (This is the proof the LLM layer — which reads `get_settings()` per request, task 03 — picks the override up live.)
3. If the phase-168 suites parameterize over `CATALOG` keys generically (check `tests/integration/test_admin_settings_api.py` first — read it, do not assume): the new key may already be covered by the parameterization; in that case add only the empty-string (c) case and the live-apply (2) case, and note it in the test's comment.
- ASSUMPTION: A4 (phase-locked) — the empty string is storable (the off switch must be settable from the panel, not only via `.env`); the API "stores as-is" per the phase-168 module docstring — no value validation to add.
## Testing & Quality
- Integration: `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` green (`uv run pytest tests/integration/test_admin_settings_api.py tests/integration/test_admin_settings_live.py -v --no-cov`).
- Coverage: **>90%** on `app/` (unchanged `app/` code — the gate must stay green, not drop).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `GET /api/admin/settings` (admin) lists the row (`LLM (aipi)`, `str`, `low`); `PUT high` → live effective `high`; `PUT ""` → live effective `""`; `DELETE` → back to `low` (the integration cases green).
- [ ] `uv run pytest tests/integration/test_admin_settings_api.py tests/integration/test_admin_settings_live.py -v --no-cov` green; full suite green.
- [ ] ruff + pyright clean; no `app/` code changed by this task.
@@ -0,0 +1,46 @@
# Task 03 — The thinking level rides every chat-completions request (`reasoning_effort` via `extra_body`)
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "Brain of Reese needs to support thinking levels like pi does. …"
**Story:** n/a
## Objective
`LLMClient` sends the operator's thinking level to aipi on every chat-completions request — `chat()` (the one-shot) and `chat_stream()` (the chat turn) — as the top-level `reasoning_effort` parameter via the SDK's `extra_body`; when the (trimmed) value is empty the request is byte-identical to today's (no `extra_body` at all).
## Work
1. `app/rag/llm.py` — ONE small helper (module-level, near the top, after the imports):
```python
def _thinking_extra_body(settings: Settings) -> dict[str, str] | None:
"""The phase-172 thinking level as the chat-completions extra body.
Non-empty (trimmed) value → ``{"reasoning_effort": <value>}`` (the
OpenAI-standard top-level parameter, A1); empty/whitespace → ``None``
(the off switch — the parameter is ABSENT from the request, A2).
"""
level = settings.llm_thinking_level.strip()
return {"reasoning_effort": level} if level else None
```
Then in **BOTH** request sites, add the parameter only when present:
- `chat()` (L446, the `await self._client.chat.completions.create(...)` at L416–421): compute `extra = _thinking_extra_body(self.settings)` and pass the kwarg **only when present** — `**({"extra_body": extra} if extra is not None else {})` spread into the `create(...)` call, so `extra_body` is ABSENT (never `None`) when off.
- `chat_stream()` (L511, the kwargs dict L592–600): same pattern — after the dict is built (and after the `if tools is not None:` block, L601–602), `if (extra := _thinking_extra_body(self.settings)) is not None: kwargs["extra_body"] = extra`.
The value is the **trimmed** setting (`.strip()` in the helper) — the raw stored string is never sent with surrounding whitespace. The embeddings path (`_post_embeddings` L256, `embed` L346, `embed_one` L400) is **untouched** (A3) — verify by reading: no `extra_body` appears anywhere in the embedding code.
2. `app/rag/llm.py` — update the module docstring (L1–L30) with ONE sentence: the thinking level (phase 172, `BOR_LLM_THINKING_LEVEL`) rides chat completions as `reasoning_effort`; empty = absent. (The house style: wire facts live in the module docstring, cf. the `reasoning_content` note at L10–12.)
3. `tests/unit/test_llm_client.py` — extend the mocked-SDK-client family (read the existing `chat` / `chat_stream` cases first and mirror their mocking):
- `chat()` with `llm_thinking_level="low"` → `create` is awaited with `extra_body={"reasoning_effort": "low"}` (assert on the call kwargs).
- `chat()` with `llm_thinking_level=" "` (whitespace) → `extra_body` is **absent from the call kwargs** (assert `"extra_body" not in kwargs`).
- `chat_stream()` with `"high"` → same `extra_body` assertion on the `create` call.
- `chat_stream()` with `""` → `extra_body` absent (the off switch — the kwargs dict is byte-identical in shape to pre-phase-172).
- one `embed()` / `embed_one()` case with the level set → the HTTP POST body (the mocked transport) contains **no** `reasoning_effort` (the embeddings path is untouched, A3).
- the TRIM case: `llm_thinking_level="low "` (trailing spaces — reachable via a hand-edited `.env`) → the sent value is `"low"`.
- ASSUMPTION: A1 (phase-locked) — the wire key is the OpenAI-standard top-level `reasoning_effort` via `extra_body`; if aipi exposes a different key, ONLY this task's helper changes (the setting, the row, the log, the E2E are key-agnostic). A2 (phase-locked) — absence is the off switch (never `extra_body=None`, never `"off"`). A3 (phase-locked) — `chat()` + `chat_stream()` only; embeddings untouched.
## Testing & Quality
- Unit: `tests/unit/test_llm_client.py` green (`uv run pytest tests/unit/test_llm_client.py -v --no-cov`).
- Coverage: **>90%** on `app/` (the new helper is fully covered by the cases above).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] With the setting non-empty, a `chat()` / `chat_stream()` request carries `extra_body={"reasoning_effort": <trimmed value>}` (the unit cases green).
- [ ] With it empty/whitespace, `extra_body` is absent from the request kwargs (byte-identical shape to today); the embeddings POST body never contains `reasoning_effort`.
- [ ] `uv run pytest` green; ruff + pyright clean.
- [ ] no behavior change in completed work (temperature / max_tokens / tools / stream kwargs untouched; the `stream_thinking` kill-switch untouched).
@@ -0,0 +1,24 @@
# Task 04 — The per-turn log line records the effective thinking level
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "Brain of Reese needs to support thinking levels like pi does. …"
**Story:** n/a
## Objective
The per-turn log line (PLAN §9, AGENTS rule 10) records the effective thinking level of the turn, so "which level did this turn run at" is one `grep` away, next to the `thinking_chars` it governs.
## Work
1. `app/api/chat.py` — the per-turn `logger.info(...)` (L1268–1291): insert `thinking=%s` **immediately before** the existing `thinking_chars=%d` in the format string, with the argument `settings.llm_thinking_level.strip() or "-"` (the effective overlay-resolved value — `settings` is already in scope, the line already logs `settings.relevance_threshold` L1280; trimmed, `-` when empty, per A6). No reordering of the existing fields; the `query_log` row (L1250–1260) is **untouched** (A6: the table is retrieval-scoped, the line is the home of turn knobs).
2. `app/api/chat.py` — the module docstring: the existing per-turn-line notes (L14, L46, L64, L78, L141, L161) set the house pattern — add ONE line where the log line's fields are described: the per-turn log line records `thinking=<level>` (the effective `BOR_LLM_THINKING_LEVEL`, `-` when empty) before `thinking_chars` (phase 172).
3. The existing per-turn-line test (find it: `grep -rn "thinking_chars" tests/` — the house pattern pins the line's fields in one place; extend that test, do not invent a new file): assert the new `thinking=` field is present, carries the effective value in the non-empty case, and is `-` when the setting is empty.
- ASSUMPTION: A6 (phase-locked) — `thinking=<trimmed value>` or `-`, placed immediately before `thinking_chars`; no `query_log` schema change, no new `Settings` read (the per-turn `settings` object is already resolved through the overlay).
## Testing & Quality
- Unit/integration: the per-turn-line test (above) green; `uv run pytest` green (the SSE turn machinery is untouched — the change is one format field + one argument).
- Coverage: **>90%** on `app/` (the gate must stay green, not drop).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] A test turn's per-turn log line (the test's captured log) contains `thinking=low` (default) and `-` when the setting is empty; `thinking_chars` stays in its position after it.
- [ ] `grep -rn "thinking=" app/api/chat.py` shows the field exactly once in the format string.
- [ ] `uv run pytest` green; ruff + pyright clean; no `query_log` schema change.
@@ -0,0 +1,32 @@
# Task 05 — The phase's dedicated Playwright suite (AGENTS rule 4) + the full gates + the atomic commit
**Phase:** `172_thinking_level` · **Source:** `TODO.md:3` — "… should be a free-form text entry like chat model. Default should be 'low'. Update .env and admin settings."
**Story:** n/a
## Objective
`tests/e2e/test_thinking_level.py` proves the owner's acceptance in a real browser — the Thinking level row exists in the admin Settings panel as a **free-form text** entry (like the chat model's), defaults to `low`, saves and persists, and resets to env — run **in isolation**; then the full gates, then ONE atomic commit.
## Work
1. `tests/e2e/test_thinking_level.py` — the phase's dedicated suite, run in isolation (DB must be up: `podman compose up -d db`; `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov`). **Copy the harness from `tests/e2e/test_admin_settings_panel.py`** (read it first, then mirror: the `_boot_app`/`app_server`/`app_url` fixtures with the mock LLM, the `_clean` autouse fixture (its `app_settings` truncation pattern — the new key's rows must be wiped between tests), the `_cookies`/`_admin_client` helpers, the settings-page nav + Save/Reset drives at L561ff). Three tests:
- `test_row_renders_free_form_with_default_low` — admin opens `/settings.html`; the `LLM (aipi)` group band is visible (the phase-171 band contract); a row labelled **Thinking level** (`BOR_LLM_THINKING_LEVEL` env-var display, the catalogue pattern) is present, its input is a **free-form text** input (`type="text"` — NOT `number`/`select`, the A4 contract), shows `low` with the `env` badge. Assert the `llm_chat_model` row's input is the same `type="text"` control (the "like chat model" reference, in one assertion pair).
- `test_saved_level_persists_without_restart` — type `high` into the row (free-form: also type a custom value `think-soft` once, to prove no allowlist — A4), Save (the phase-168 §7.4 Save lifecycle: dirty → save → `DB` badge), the effective value reads `think-soft`; reload `/settings.html` → still `think-soft` with the `DB` badge (no restart, the phase-168 contract).
- `test_reset_returns_to_env_default` — the row's Reset (per-row `DELETE`) → the value reads `low` again with the `env` badge (the overlay reset, task-02's contract, in the browser).
2. Run the **full gates** in order, all from the repo root:
- `uv run pytest -v` (unit + integration) and `uv run pytest --cov=app --cov-report=term-missing` → `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright`.
- `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov` → green **in isolation**.
- Regression, in isolation: `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` (phase 168) and `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` (phase 171) → both green.
3. The **atomic commit** (AGENTS rule 8 — one commit per phase, Conventional Commits, `--no-gpg-sign`): the whole phase — `app/config.py`, `app/core/settings_catalog.py`, `app/rag/llm.py`, `app/api/chat.py`, `.env.example`, the four test files, and `.agents/phases/todo/172_thinking_level/` → moved to `.agents/phases/complete/172_thinking_level/` in the SAME commit. Message: `feat(llm): operator-tunable thinking level (BOR_LLM_THINKING_LEVEL, default low) sent as reasoning_effort` + the `LLM-Generated: true` trailer. The pre-existing dirty `TODO.md` (the now-converted entry, cleared in Phase 4 of this conversion) rides this commit ONLY if the conversion commit has not already committed it — check `git status` first; the conversion commit is the one that clears TODO.md (see the skill's Phase 4), so the phase commit should be the phase files + code only.
- ASSUMPTION: A5 (phase-locked) — no `settings.js` / `styles.css` change; the row renders from the catalogue. A6 (phase-locked) — the per-turn-line assertion lives in the unit/integration layer (task 04), not the E2E (the mock LLM turn does not exercise the wire; the E2E proves the settings surface only).
## Testing & Quality
- E2E: `tests/e2e/test_thinking_level.py` green **in isolation**; the phase-168 + phase-171 suites green in isolation (regression).
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_thinking_level.py -v --no-cov` green in isolation (the three contracts).
- [ ] `uv run pytest` green; `app/` coverage >90%; ruff + pyright clean.
- [ ] `test_admin_settings_panel.py` + `test_admin_settings_layout.py` green in isolation (no regression in the settings machinery).
- [ ] One atomic `--no-gpg-sign` Conventional Commit with the `LLM-Generated: true` trailer; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,53 @@
# Phase 173 — Docs-push credentials: split the username/token out of `BOR_DOCS_REPO`
**Source:** owner audit of the production admin Settings panel (2026-10-10): `BOR_DOCS_REPO` is a plain-text row whose production value is a git URL with an embedded username + Gitea token (`https://user:token@host/…`) — a live credential leak on the settings page, the raw HTML, and the settings API.
**Story:** n/a (owner-directed, chat-confirmed 2026-10-10)
**Context:** `app/config.py` — the Docs push section (`docs_repo` L539, `docs_branch` L545, `docs_base_branch` L549, `docs_work_dir` L554; the branch validator L831–846 requires non-empty branches only while `docs_repo` is set; `docs_configured` L902–906 is the 409 inert-switch on `docs_repo` alone — **untouched**), `.env.example` L229–234 (the Docs push block — commented optional vars), `app/core/settings_catalog.py` L303–308 (the `Docs push` group: `docs_repo` L303, `docs_branch` L304, `docs_base_branch` L305–307, `docs_work_dir` L308) and the existing secret mechanism (`llm_api_key` L142 `secret=True`; `SECRETS` is DERIVED from `CATALOG` — a `secret=True` entry is write-only end-to-end: the API reports `is_set` + `value=null` (`app/api/admin_settings.py` L147), a blank PUT 422s (L273), and the phase-168/171 panel renders the row masked as `(stored)` with zero `settings.js` code), `scripts/git_sync.py` L320 (`run_git(argv, cwd)` — the ONLY git invocation site, A11; `subprocess.run(argv, cwd=cwd, capture_output=True, text=True)` L331 — no env today), `app/core/docs_push.py` (`push_document` — clone L104–116 / fetch / checkout / add / commit / push, every step via `run_git`; `DocsPushError` carries the full git stderr for logs), `app/api/doc_drafts.py` L299–316 (the push call site; phase 84 `sanitize_error` already strips echoed credentials from the 502 response — defense-in-depth, stays), `app/core/settings_overlay.py` (`load_overrides()` L93 — the raw `app_settings` DB map; `resolve_settings()` L239; `reset_state_for_tests()` L274), `app/main.py` L98–245 (`create_app()` the composition root; `app = create_app()` L245), `tests/unit/test_config.py`, `tests/unit/test_settings_catalog.py`, `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` (the generic set/reset/live-apply contract), `tests/e2e/test_admin_settings_panel.py` + `test_admin_settings_layout.py` (phase 168/171 — the regression set), `tests/e2e/test_response_to_docs.py` (phase 59 — the docs-push E2E).
## Objective
The docs-push remote's credentials live in their own settings — `BOR_DOCS_USER` (plain) and `BOR_DOCS_TOKEN` (write-only, masked like `BOR_LLM_API_KEY`) — while `BOR_DOCS_REPO` holds a clean URL. At push time the credentials travel as git's `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` environment variables (never in the command line / process list), a one-time startup migration splits any legacy `://user:pass@` value into the new settings (DB rows shadow the env, `.env` is never rewritten), and a PUT guard refuses a credentials-bearing repo URL so the leak cannot be re-created from the UI. Anonymous pushes (no token) stay byte-identical to today.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the three settings (task 01).** `docs_user: str = ""` and `docs_token: str = ""` in the Docs push section of `app/config.py` (directly after `docs_repo` L539, before `docs_branch` L545). Plain `str`, no validator (empty = "no credentials" is a valid state — the repo may be public/anonymous, exactly like today). `.env.example` L231–234 block gains `# BOR_DOCS_USER=` / `# BOR_DOCS_TOKEN=` (commented — optional, house style of the block). CATALOG: `docs_user` plain, `docs_token` **`secret=True`** (inserted directly after `docs_repo` L303, before `docs_branch` L304). The `SECRETS` derivation + the generic overlay + the phase-168/171 panel make `docs_token` a write-only, masked row with zero extra code (C1). No frontend code changes anywhere in this phase.
- **D2 — the PUT guard (task 02).** In `app/api/admin_settings.py`, the `PUT` route refuses `docs_repo` values that embed credentials — a `://<user>:<pass>@` segment (regex `://[^/@:]+:[^@/]+@`) — with a 422 whose detail names the replacement (use the Docs user / Docs token fields, keep the URL clean). Everything else about the row is unchanged (it stays overridable, plain). `docs_token`'s API behavior is the EXISTING generic secret contract (GET `is_set`/`value=null`, blank PUT 422, DELETE resets) — task 02 pins it with integration assertions, it needs no new code.
- **D3 — credentials on the wire (task 03).** `run_git` gains an optional `env: dict[str, str] | None = None` passed straight to `subprocess.run` — **the subprocess env-replacement trap**: when set, the dict REPLACES the parent environment, so `docs_push` must build it as `{**os.environ, …}` (a test asserts `PATH` survives). `push_document` gains `user: str = ""` / `token: str = ""`; when `token.strip()` is non-empty, EVERY git step of the sequence (clone, both fetches, checkout, add, commit, push) gets `env = {**os.environ, "GIT_CURL_USERNAME": user, "GIT_CURL_PASSWORD": token, "GIT_TERMINAL_PROMPT": "0"}` (git's documented HTTP credential env; `GIT_TERMINAL_PROMPT=0` fails fast instead of hanging a headless server on a prompt); when the token is empty, `env` stays `None` and the invocations are byte-identical to today (C5). The clean URL stays in the argv. `app/api/doc_drafts.py` passes `settings.docs_user` / `settings.docs_token`. `GIT_CURL_*` only takes effect for `http(s)` remotes; for `ssh://` / local-path remotes it is inert (accepted). Phase 84's `sanitize_error` on the 502 response is untouched (defense-in-depth).
- **D4 — the legacy migration (task 04).** New module `app/core/settings_migrate.py` with `migrate_legacy_docs_creds() -> bool`: read the `app_settings` overrides; take the EFFECTIVE `docs_repo` (DB override if present, else env — i.e. `resolve_settings().docs_repo`); if it matches `://<user>:<pass>@` (both non-empty) AND there is no `docs_token` override yet, write three DB rows — `docs_repo` = the credential-stripped URL, `docs_user` = user, `docs_token` = pass — and log one INFO line (`settings: migrated embedded docs-push credentials out of docs_repo into docs_user/docs_token`); any other shape → no-op. Idempotent (a `docs_token` row already present → never clobber owner-set values). Called ONCE from `create_app()` (the composition root) after the DB is available. `.env` is never read for writing or rewritten — the DB rows shadow the env (the settings model's standing rule). Unit-tested against the overlay module's state + `reset_state_for_tests()`.
- **D5 — the E2E (task 05).** `tests/e2e/test_docs_push_creds.py` — the phase's dedicated Playwright suite (AGENTS rule 4): the Docs push group shows a `Docs repo` text row, a `Docs user` text row, and a `Docs token` write-only row rendered masked (`(stored)`, not pre-filled, `type=password`); typing a token, saving, and reloading keeps the row masked and the typed token text is **absent from the entire page DOM** (the core assertion — the leak class this phase kills); resetting the token unsets it. The regression set (`test_admin_settings_panel.py`, `test_admin_settings_layout.py`, `test_response_to_docs.py`) stays green.
- **NOT touched:** `frontend/` (no file), the `app_settings` table shape, `settings_overlay.py` internals, `docs_configured` / the 409 inert-switch, the branch validator, `git_sync.clone_or_pull` (sync path — no credential settings exist for it; out of scope), `sanitize_error` (phase 84), every other setting.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; this phase builds on the committed phase-59 docs-push, phase-168/171 settings machinery, and phase-84 error sanitization)
## Tasks
1. `01_config_and_catalog.md` — `Settings.docs_user` / `docs_token` (default `""`) + the two CATALOG rows (`docs_token` `secret=True`) + `.env.example`; the config/catalog unit tests.
2. `02_api_masking_and_guards.md` — integration pin of the `docs_token` write-only contract + the new `docs_repo` embedded-credentials PUT guard (422); the guard's unit/integration tests.
3. `03_run_git_env_and_push.md` — `run_git(env=None)` + `push_document(user, token)` credential env on every git step (merged over `os.environ`; `None` when the token is empty) + the `doc_drafts.py` call site; the mocked-git unit tests.
4. `04_legacy_creds_migration.md` — `app/core/settings_migrate.py` (idempotent, DB-shadowing, one log line) wired into `create_app()`; unit + integration tests.
5. `05_e2e_docs_creds_masking.md` — `tests/e2e/test_docs_push_creds.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` — `docs_user` / `docs_token` default `""`, env overrides, empty env stays `""` (no coercion). `tests/unit/test_settings_catalog.py` — the completeness pin holds with the two new fields; `docs_token` is in `SECRETS`, `docs_user` is in none of the derived sets. `tests/unit/test_git_sync.py` (or the existing `run_git` test home) — `env=None` leaves the subprocess call byte-identical; a passed env reaches `subprocess.run` verbatim. `tests/unit/test_docs_push.py` — token empty → every `run_git` call gets `env=None`; token set → every git step gets the 3-var env merged over `os.environ` (`PATH` present); `DocsPushError` propagation unchanged. `tests/unit/test_settings_migrate.py` — the 4 migration shapes (D4).
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — `docs_token` in `GET /api/admin/settings` (group `Docs push`, `secret=true`, `value=null`, `is_set` flips on PUT); PUT stores / DELETE resets / blank PUT 422 (the generic contract pinned for the new key); `docs_repo` PUT with `://user:pass@` → 422 naming the user/token fields, clean URLs pass. `tests/integration/test_docs_push_migrate.py` — seeded env `BOR_DOCS_REPO=https://u:pass@host/r.git`, no DB rows → after the migration runs, `resolve_settings().docs_repo` is `https://host/r.git`, `docs_user`/`docs_token` resolve from the DB; a second run is a no-op.
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_docs_push_creds.py`, run **in isolation** — per D5.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `GET /api/admin/settings` (admin) lists `docs_user` (plain) and `docs_token` (secret, `value=null`) in the `Docs push` group; the token value is NEVER in any response body.
- [ ] `PUT /api/admin/settings` for `docs_repo` with a `://user:pass@` value → 422 naming the Docs user / Docs token fields; a clean URL passes.
- [ ] With `docs_token` set, a docs push invokes git with `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` / `GIT_TERMINAL_PROMPT=0` in the environment and a clean URL in the argv (unit-asserted via mocked `run_git`); with it empty, the `run_git` calls are byte-identical to today (`env=None`).
- [ ] A legacy `://user:pass@` env value is split once at startup into `app_settings` rows (clean repo + user + token), `.env` untouched, a second run is a no-op; the INFO migration line is logged exactly once.
- [ ] `uv run pytest` green (100% — every test in the project, pre-existing included), `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_docs_push_creds.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` + `test_admin_settings_layout.py` + `test_response_to_docs.py` green in isolation.
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(settings): split docs-push credentials (BOR_DOCS_USER/BOR_DOCS_TOKEN) out of the repo URL — masked, env-wired, legacy auto-migration`); the phase dir moves to `.agents/phases/complete/`.
- [ ] `README.md` and `AGENTS.md` are accurate and consistent with the project as it will stand after the phase (the final pass reviews both in full and fixes every inconsistency found, including pre-existing ones).
## Documentation & Consistency (mandatory)
The final pass reviews `README.md` and `AGENTS.md` in their entirety against the project as it will stand after the phase and fixes every inconsistency found — including pre-existing ones not caused by the phase. Tasks update both files whenever their own changes are user-visible (the `.env.example` block is the user-visible docs surface for this phase; the README's docs-push section gains the two new vars + a line that credentials in the repo URL are no longer supported).
## Locked decisions (owner-confirmed 2026-10-10, chat)
- **C1 — the split shape:** `docs_repo` (clean URL) + `docs_user` (plain `str`) + `docs_token` (`secret=True` in the CATALOG — reuses the existing write-only/masked mechanism end-to-end, `llm_api_key`-style). No new UI code: the panel renders both rows from the catalogue.
- **C2 — the credential transport:** `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` (+ `GIT_TERMINAL_PROMPT=0`) environment variables on the push sequence — never in the git argv / process list; applied only when `docs_token` is non-empty.
- **C3 — the migration:** one-time at startup (`create_app()`), effective-value based, DB rows shadow the env, `.env` never rewritten, idempotent, one INFO log line.
- **C4 — the guard:** `PUT docs_repo` with embedded credentials → 422 (the leak cannot be re-created via the UI).
- **C5 — no behavior change for anonymous pushes:** empty `docs_token` → the git invocations are byte-identical to today (`env=None`).
@@ -0,0 +1,55 @@
# Task 01 — The `docs_user` / `docs_token` settings + the catalogue rows + `.env.example`
**Phase:** `173_docs_push_creds_split` · **Source:** owner audit of the production admin Settings panel (2026-10-10): `BOR_DOCS_REPO` leaks its embedded Gitea token in plain text; the credentials get their own settings.
**Story:** n/a
## Objective
The two new settings exist in `app/config.py` (default `""`), in the admin catalogue (so the build's completeness pin stays green — a field without a `SettingMeta` fails the build, and the per-task gate runs the full suite), and in `.env.example`, so `BOR_DOCS_USER` / `BOR_DOCS_TOKEN` are first-class, documented knobs before anything consumes them. `docs_token` is `secret=True` — write-only, masked end-to-end via the EXISTING mechanism (C1); nothing else in the phase touches `frontend/`.
## Work
1. `app/config.py` — in the Docs push section, **directly after** the `docs_repo` block (L539) and **before** `docs_branch` (L545), add TWO fields, house docstring style (env var named, default, consumption point, off rule):
```python
#: Docs-push remote username (phase 173, ``BOR_DOCS_USER``) — paired
#: with ``docs_token``; travels to git as the ``GIT_CURL_USERNAME``
#: environment variable (``app/core/docs_push.py``, task 03) — never
#: in the command line. Empty (default) = no credentials: the push
#: is anonymous/URL-credential-free, exactly as before this phase.
docs_user: str = ""
#: Docs-push remote token (phase 173, ``BOR_DOCS_TOKEN``) — write-only
#: (catalogue ``secret=True``): the API reports ``is_set`` and never
#: the value; the admin panel renders it masked. Travels to git as
#: ``GIT_CURL_PASSWORD`` (``app/core/docs_push.py``, task 03).
#: Empty (default) = the credential env is not set at all.
docs_token: str = ""
```
No validator, no coercion — plain `str`; an empty value is a valid state (C1/C5). The L831–846 branch validator (branches required only while `docs_repo` is set) and `docs_configured` (L902–906) stay byte-identical.
2. `.env.example` — in the Docs push block (L229–234), directly under `# BOR_DOCS_REPO=/path/to/docs-repo` (L231), add:
```
# BOR_DOCS_USER=
# BOR_DOCS_TOKEN=
```
(commented — optional, the block's house style; add one line to the block's comment: credentials in the repo URL are no longer supported — use BOR_DOCS_USER/BOR_DOCS_TOKEN).
3. `app/core/settings_catalog.py` — in the `Docs push` group, **directly after** the `docs_repo` entry (L303) and **before** `docs_branch` (L304), add TWO entries:
```python
"docs_user": SettingMeta("docs_user", "Docs push", "Docs user"),
"docs_token": SettingMeta(
"docs_token", "Docs push", "Docs token", secret=True
),
```
Nothing else — `SECRETS` / `RESTART_REQUIRED` / `NON_OVERRIDABLE` are DERIVED from `CATALOG` (module docstring): `docs_token` lands in `SECRETS` automatically, `docs_user` in none. Both are plain `str` → the overlay is generic (live, no restart). The panel row falls out of the catalogue with zero `settings.js` code (C1 — the phase-168/171 machinery renders a `secret=True` row masked as `(stored)`, `llm_api_key`-style).
4. `tests/unit/test_settings_catalog.py` — the completeness pin (`set(CATALOG) == set(Settings.model_fields)`) must stay green with the two new fields; add the house-style assertions: `CATALOG["docs_user"]` sits in group `Docs push` with label `Docs user` and is in **none** of the derived sets; `CATALOG["docs_token"]` has label `Docs token`, `secret=True`, and **is** in `SECRETS`.
5. `tests/unit/test_config.py` — add the house-style cases: (a) the DEFAULTS are `""` / `""` (no env set); (b) the env override — `BOR_DOCS_USER=alice` / `BOR_DOCS_TOKEN=tok` resolve to `"alice"` / `"tok"`; (c) the empty env values stay `""` (no coercion, no normalization away from empty — C5's anonymous state stays expressible).
- ASSUMPTIONS: C1 (phase-locked) — the split shape; C5 (phase-locked) — empty token = no credential env, so the settings must NOT strip/coerce away the empty value at load.
## Testing & Quality
- Unit: `tests/unit/test_config.py` + `tests/unit/test_settings_catalog.py` green (`uv run pytest tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov`).
- Coverage: **>90%** on `app/` (the gate must stay green).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest -k "docs" tests/unit/test_config.py tests/unit/test_settings_catalog.py -v --no-cov` green: defaults `""`, env overrides, empty stays `""`, the completeness pin holds, `docs_token` in `SECRETS`, `docs_user` in none.
- [ ] `grep -n "BOR_DOCS_USER" .env.example` shows the line under `# BOR_DOCS_REPO=…` (commented).
- [ ] `GET /api/admin/settings` (admin) lists `docs_user` + `docs_token` in the `Docs push` group — `docs_token` as `secret=true` with `value=null` (a quick curl, or the task-02 integration test).
- [ ] `uv run pytest` green; ruff + pyright clean.
- [ ] no behavior change in completed work (no other `Settings` field touched; `frontend/` untouched; `docs_configured` + the branch validator byte-identical).
@@ -0,0 +1,45 @@
# Task 02 — The `docs_token` write-only contract (pinned) + the `docs_repo` embedded-credentials guard
**Phase:** `173_docs_push_creds_split` · **Source:** C2/C4 (phase-locked) — the token is write-only end-to-end via the existing mechanism; a credentials-bearing repo URL is refused on the way in so the leak cannot be re-created from the UI.
**Story:** n/a
## Objective
The `docs_token` row's API behavior is the EXISTING generic secret contract — task 02 pins it with integration assertions for the new key (no new code expected). The ONE piece of new logic: the `PUT` route refuses `docs_repo` values with embedded credentials (C4), pointing the operator at the Docs user / Docs token fields.
## Work
1. `app/api/admin_settings.py` — in the `PUT` route, **before** the existing blank-secret 422 check (L273) or alongside it, add the `docs_repo` guard (a small module-level helper keeps the route readable — house style):
```python
#: A git URL with embedded credentials (``://<user>:<pass>@``) — the
#: phase-173 leak class. Refused for ``docs_repo`` on the way in:
#: the credentials belong in ``docs_user`` / ``docs_token`` (C4).
_EMBEDDED_CREDS = re.compile(r"://[^/@:]+:[^@/]+@")
def _refuse_embedded_credentials(key: str, value: str) -> None:
if key == "docs_repo" and _EMBEDDED_CREDS.search(value):
raise HTTPException(
status_code=422,
detail=(
"docs_repo must not contain embedded credentials — "
"keep the URL clean and set them in the Docs user / "
"Docs token fields instead"
),
)
```
Call it at the top of the per-key validation in the PUT route (after the payload is parsed, before store). Only `docs_repo` is guarded — `ssh://` URLs, local paths, and clean `https://` URLs pass untouched. (A legacy `.env` value with embedded credentials is NOT a PUT — the task-04 startup migration handles that path; the guard covers only new writes.)
2. `tests/integration/test_admin_settings_api.py` (or `test_admin_settings_live.py` — match the file that owns the secret-row contract; the phase-172 pin is the template) — add the house-style cases for the new key:
- `GET /api/admin/settings` (admin): `docs_token` in group `Docs push`, `secret=true`, `value=null`, `is_set=false`; `docs_user` plain, `value=""`.
- `PUT docs_token` with a value → 200; `GET` → `is_set=true`, `value=null` (the value is **never** in the response body — assert the token text is absent from the raw response JSON).
- `PUT docs_token` with a blank value → 422 (the existing self-lockout guard — pinned for the new key).
- `DELETE docs_token` → `is_set=false` (reset to env — `""` by default).
3. `tests/integration/test_admin_settings_api.py` — the guard's integration cases: `PUT docs_repo` `https://u:pass@gitea.example.com/o/r.git` → 422, detail mentions the Docs user / Docs token fields; `PUT docs_repo` `https://gitea.example.com/o/r.git` → 200 (clean passes); `PUT docs_repo` a local path (`/srv/docs-repo`) → 200 (untouched); a `user:` segment that is NOT `://user:pass@` (e.g. `https://sub.domain.example.com/r.git`) → 200 (the regex must not false-positive on subdomains — the `@` requirement does that job).
## Testing & Quality
- Integration: the cases above green (`uv run pytest tests/integration/test_admin_settings_api.py -v --no-cov`).
- Coverage: **>90%** on `app/` (the gate must stay green — the guard helper must be covered by the 422 + pass cases).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_admin_settings_api.py -v --no-cov` green, including: `docs_token` GET `is_set`/`value=null` (token text absent from the raw JSON), PUT store, blank PUT 422, DELETE reset; `docs_repo` embedded-creds 422 + clean/local/subdomain 200s.
- [ ] The 422 detail names the Docs user / Docs token fields (asserted in the test, not just present).
- [ ] `uv run pytest` green; ruff + pyright clean.
- [ ] no behavior change in completed work (no other key's PUT path touched; the blank-secret guard at L273 byte-identical).
@@ -0,0 +1,47 @@
# Task 03 — Credentials on the wire: `run_git(env=…)` + `push_document(user, token)`
**Phase:** `173_docs_push_creds_split` · **Source:** C2/C5 (phase-locked) — credentials travel as `GIT_CURL_USERNAME` / `GIT_CURL_PASSWORD` env (never in the argv / process list); an empty token leaves the invocations byte-identical to today.
**Story:** n/a
## Objective
The docs-push git sequence authenticates with the settings' user/token via git's documented HTTP credential environment, without the credential ever appearing in the command line, and without changing a single invocation when no token is set (C5).
## Work
1. `scripts/git_sync.py` — `run_git` (L320) gains one optional parameter:
```python
def run_git(argv: list[str], cwd: Path, env: dict[str, str] | None = None) -> str:
…
proc = subprocess.run(
argv, cwd=cwd, capture_output=True, text=True, env=env
)
```
`None` (the default, and the sync path's only usage) is byte-identical to today. **The subprocess env-replacement trap:** when `env` is a dict, it REPLACES the parent environment — the caller must build a complete environment (task 03's `docs_push` does the merge; a unit test pins `PATH` survival). Update the docstring: the `env` parameter is passed through verbatim; `None` = inherit.
2. `app/core/docs_push.py` — `push_document` gains `user: str = ""` / `token: str = ""` (positional order: appended AFTER `commit_message`, so every existing call site keeps working; update the docstring with the C2 transport + the C5 no-token rule). At the top of the sequence:
```python
git_env: dict[str, str] | None = None
if token.strip():
# C2: git's documented HTTP credential env — never in the argv.
# GIT_TERMINAL_PROMPT=0 fails fast instead of hanging the headless
# server on a credential prompt.
git_env = {
**os.environ,
"GIT_CURL_USERNAME": user,
"GIT_CURL_PASSWORD": token,
"GIT_TERMINAL_PROMPT": "0",
}
```
Pass `env=git_env` to **every** `run_git` call in the sequence (clone, both fetches, `rev-parse` ×2, `branch --list`, checkout, add, commit, push — `rev-parse`/`branch` may take it too; uniformity over cleverness). The clean URL stays in the argv untouched. `DocsPushError` propagation is unchanged (the error still carries the full git stderr for logs; phase 84's `sanitize_error` on the API response stays as defense-in-depth). `GIT_CURL_*` is inert for `ssh://` / local-path remotes (accepted — the remote decides).
3. `app/api/doc_drafts.py` — the push call site (L306–314) passes `user=settings.docs_user, token=settings.docs_token`. Nothing else changes (the 409 inert-switch on `docs_configured` stays repo-based — an empty token is a valid anonymous-push state, C5).
4. `tests/unit/test_git_sync.py` (or the existing `run_git` test home) — (a) default: `run_git(["git", "version"], cwd)` → `subprocess.run` called with `env=None` (byte-identical to today); (b) an explicit `env={"A": "b"}` reaches `subprocess.run` verbatim.
5. `tests/unit/test_docs_push.py` — (a) **no token** (default args) → every mocked `run_git` call gets `env=None` (the C5 byte-identity pin); (b) **token set** → every git step of a successful push gets an env with `GIT_CURL_USERNAME=user`, `GIT_CURL_PASSWORD=token`, `GIT_TERMINAL_PROMPT="0"`, **and** `PATH` present (the merge-over-`os.environ` pin — run with a controlled `os.environ` in the test); (c) whitespace-only token (`" "`) → treated as empty (the `.strip()` rule) → `env=None`; (d) a failing step still raises `DocsPushError` with git's stderr — with and without credentials (the error path is credential-agnostic).
## Testing & Quality
- Unit: the cases above green (`uv run pytest tests/unit/test_git_sync.py tests/unit/test_docs_push.py -v --no-cov` — adjust file names to the existing homes if they differ).
- Coverage: **>90%** on `app/` (the gate must stay green — the `token.strip()` branches are covered by 5a/5b/5c).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_docs_push.py -v --no-cov` green: no-token `env=None` (all steps), token-set 3-var env merged over `os.environ` (`PATH` present, all steps), whitespace-only token = empty, `DocsPushError` propagation unchanged.
- [ ] The sync path (`clone_or_pull`) is byte-identical (it never passes `env` — verify by reading: no `env=` at its `run_git` call sites).
- [ ] `uv run pytest` green; ruff + pyright clean.
- [ ] no behavior change in completed work (the git argv of an anonymous push is byte-identical to pre-phase; `sanitize_error` untouched).
@@ -0,0 +1,45 @@
# Task 04 — The one-time startup migration of a legacy `://user:pass@` repo URL
**Phase:** `173_docs_push_creds_split` · **Source:** C3 (phase-locked) — a legacy credentials-bearing `docs_repo` value (env or DB) is split once at startup into `app_settings` rows; the DB rows shadow the env; `.env` is never rewritten; idempotent; one INFO line.
**Story:** n/a
## Objective
Production's existing `BOR_DOCS_REPO=https://ducoterra:<token>@gitea.…` keeps working (the push still authenticates) with zero operator action, while the effective value the settings page shows becomes the clean URL and the credentials live in `docs_user` / `docs_token` (masked). A second run is a no-op; owner-set values are never clobbered.
## Work
1. New module `app/core/settings_migrate.py`:
```python
"""One-time settings migrations (phase 173).
``migrate_legacy_docs_creds`` splits a legacy ``://user:pass@``
``docs_repo`` value into ``docs_repo`` (clean) + ``docs_user`` +
``docs_token`` — DB rows that SHADOW the env (the settings model's
standing rule: the database wins over the environment). ``.env`` is
never read for writing or rewritten. Idempotent: it runs only while
no ``docs_token`` override exists, so owner-set values are never
clobbered. One INFO line when it acts (AGENTS rule 10).
"""
```
- `migrate_legacy_docs_creds() -> bool` — (1) read the `app_settings` overrides (via the overlay module's state — `load_overrides()` L93 or its backing read; use `reset_state_for_tests()` in tests); (2) the target value is the **effective** `docs_repo` (an override if present, else the env value — i.e. what `resolve_settings().docs_repo` resolves to, L239); (3) match `://([^/@:]+):([^@/]+)@` with BOTH groups non-empty — otherwise return `False` (no-op); (4) if a `docs_token` override already exists → `False` (no-op — never clobber); (5) otherwise write three rows (the overlay's existing write path — the same one the `PUT` route uses, or a direct `app_settings` insert if the overlay exposes none): `docs_repo` = the value with the `user:pass@` segment removed (e.g. `https://u:pass@host/r.git` → `https://host/r.git`), `docs_user` = user, `docs_token` = pass; log ONE INFO line: `settings: migrated embedded docs-push credentials out of docs_repo into docs_user/docs_token`; return `True`. A malformed `user:pass@` (empty user or pass) → `False`, no rows, no log (the legacy value stays as-is — the pre-phase behavior; nothing breaks, nothing half-migrates).
2. `app/main.py` — `create_app()` (L98, the composition root): after the DB is available and BEFORE `return app`, call `settings_migrate.migrate_legacy_docs_creds()` ONCE (wrap in `try/except` so a migration hiccup logs a WARNING and never blocks startup — the app boots with the pre-phase behavior in that case; the migration is a convenience, not a dependency).
3. `tests/unit/test_settings_migrate.py` — the four shapes (with the overlay state controlled via `reset_state_for_tests()` + a fake/monkeypatched override store — house style):
- (a) clean env URL, no overrides → `False`, no rows written;
- (b) legacy env `https://u:pass@host/r.git`, no overrides → `True`, three rows written (clean repo / `u` / `pass`), `resolve_settings()` resolves the clean URL from the DB;
- (c) legacy env + a `docs_token` override already present → `False`, the pre-existing rows are untouched (no clobber);
- (d) malformed `https://:@host/r.git` (empty user) → `False`, no rows, no log.
Plus: (e) the INFO line is emitted exactly once in (b) (caplog / log-capture, house style).
4. `tests/integration/test_docs_push_migrate.py` — the end-to-end shape: seed env `BOR_DOCS_REPO=https://u:pass@host/r.git` (no DB rows) → run the migration (direct call, the `create_app()` wiring is covered by the unit seam) → `resolve_settings().docs_repo == "https://host/r.git"`, `docs_user == "u"`, `docs_token == "pass"` (all from DB overrides); a second call → `False` (idempotent); then `reset_state_for_tests()` for the next test.
5. `.env.example` — the block comment added in task 01 (already covers this: "credentials in the repo URL are no longer supported — use BOR_DOCS_USER/BOR_DOCS_TOKEN"; the migration note: a legacy value is auto-split on startup into the new settings). No other file touched.
## Testing & Quality
- Unit: `tests/unit/test_settings_migrate.py` green (the 5 cases above).
- Integration: `tests/integration/test_docs_push_migrate.py` green.
- Coverage: **>90%** on `app/` (the gate must stay green — the no-op/act/clobber/malformed branches are all covered by 3a–3d; the `create_app()` try/except seam is covered by a unit test that monkeypatches the migration to raise).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_settings_migrate.py tests/integration/test_docs_push_migrate.py -v --no-cov` green: the four shapes + the idempotency + the one-INFO-line pin.
- [ ] `create_app()` calls the migration once, wrapped so a failure logs a WARNING and the app still boots (unit-asserted via the monkeypatched-raise seam).
- [ ] After a (b)-shape migration, the settings API reports the clean `docs_repo` + `is_set=true` for `docs_token` with `value=null` (a quick curl or an integration assertion) — the leak is gone from the page.
- [ ] `uv run pytest` green; ruff + pyright clean.
- [ ] no behavior change in completed work (no pre-migration behavior touched; the sync path untouched).
@@ -0,0 +1,44 @@
# Task 05 — The phase E2E suite + the full gates + the atomic commit
**Phase:** `173_docs_push_creds_split` · **Source:** AGENTS rule 4 (one dedicated Playwright E2E suite per phase, run in isolation) + the test-gate protocol (AGENTS rules 8/9).
**Story:** n/a
## Objective
`tests/e2e/test_docs_push_creds.py` proves the leak class is dead in the real UI — the `Docs token` row is write-only, masked, never pre-filled, and a saved token never appears anywhere in the page DOM — and the phase's full gate passes: 100% of the project suite, >90% `app/` coverage, ruff + pyright, the regression E2E set, one atomic `--no-gpg-sign` Conventional Commit, and the phase dir moves to `complete/`.
## Work
1. `tests/e2e/test_docs_push_creds.py` — the phase's dedicated Playwright suite (the phase-168 `test_admin_settings_panel.py` is the template for login + the settings panel; the mock-LLM default per A16 — the suite needs no LLM at all). Tests, in order:
- **rows render:** sign in as admin → Settings → the `Docs push` group shows a `Docs repo` text row (the clean URL, or the test-seed value), a `Docs user` text row, and a `Docs token` row that is write-only — its input is `type=password`, **not pre-filled** (value `""`, rendered `(stored)`-style per the phase-168 secret-row contract), and the token text is absent from the initial DOM.
- **save stays masked:** type a unique sentinel token (e.g. `e2e-sentinel-<uuid>` from the test) into the `Docs token` row → Save → the row re-renders masked; reload the page → the row is still masked, `is_set`-state visible, and the **sentinel text is absent from the entire page DOM** (the core assertion — `page.content()` must not contain the sentinel).
- **reset unsets:** Reset the `Docs token` row → the row returns to the not-set state (no `(stored)` marker).
- **the guard, UI-side:** type a credentials-bearing URL (`https://u:pass@e2e.invalid/o/r.git`) into `Docs repo` → Save → the 422 surfaces (the phase-168 error-display contract), the row keeps the previous value, and the page DOM never shows the typed URL's `pass` segment as a *saved* value (the input echoes what was typed before save — that is fine; the assertion is on the saved/persisted state after the failed save + reload).
- Seed/cleanup: the test manages its own `app_settings` rows via the admin API (or the test DB fixture the other settings E2Es use) — no state leaks between runs.
2. **Regression set (run in isolation, house order):** `uv run pytest tests/e2e/test_admin_settings_panel.py tests/e2e/test_admin_settings_layout.py tests/e2e/test_response_to_docs.py -v --no-cov` — phase 168/171 (the settings panel) and phase 59 (the docs push) stay green.
3. **Full gates:** `uv run pytest` (100% — every test in the project, pre-existing included) · `uv run pytest --cov=app --cov-report=term-missing` (>90% on `app/`) · `uv run ruff check . && uv run pyright`.
4. **Docs & consistency pass (mandatory — AGENTS.md rule 5 / phase 00):** review `README.md` and `AGENTS.md` in their entirety against the project as it now stands; fix every inconsistency found — including pre-existing ones. At minimum: the README's docs-push section documents `BOR_DOCS_USER` / `BOR_DOCS_TOKEN`, states that credentials in the repo URL are no longer supported (the PUT 422 + the startup migration), and notes the token is masked (write-only) in the settings page/API.
5. **Commit (AGENTS rule 8):** ONE atomic Conventional Commit, `--no-gpg-sign`, covering `app/`, `scripts/`, `.env.example`, `tests/`, the README fix, and this phase dir's move to `.agents/phases/complete/173_docs_push_creds_split/`. Message shape:
```
feat(settings): split docs-push credentials (BOR_DOCS_USER/BOR_DOCS_TOKEN) out of the repo URL
- docs_token is write-only (masked in the page + API, llm_api_key-style); the
docs_repo PUT guard 422s on embedded credentials
- the push authenticates via GIT_CURL_USERNAME/GIT_CURL_PASSWORD env (never in
the argv); anonymous pushes are byte-identical to before
- a one-time startup migration splits a legacy ://user:pass@ env value into the
new settings (DB rows shadow the env; .env untouched; idempotent)
LLM-Generated: true
```
## Testing & Quality
- E2E: `uv run pytest tests/e2e/test_docs_push_creds.py -v --no-cov` green **in isolation** (the phase's dedicated suite, AGENTS rule 4).
- Full suite: `uv run pytest` 100% green; `app/` coverage **>90%**; ruff + pyright clean.
- Regression: the phase 168/171/59 E2E set green in isolation.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_docs_push_creds.py -v --no-cov` green in isolation: rows render, the token row is `type=password` + not pre-filled, save→reload keeps it masked with the sentinel **absent from the page DOM**, reset unsets, the 422 guard surfaces.
- [ ] `uv run pytest tests/e2e/test_admin_settings_panel.py tests/e2e/test_admin_settings_layout.py tests/e2e/test_response_to_docs.py -v --no-cov` green in isolation.
- [ ] `uv run pytest` 100% green; `uv run pytest --cov=app` >90% on `app/`; `uv run ruff check . && uv run pyright` clean.
- [ ] `README.md` + `AGENTS.md` reviewed in full and consistent (the docs-push section documents the new vars + the guard + the migration).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/173_docs_push_creds_split/`.
- [ ] **Owner follow-up (noted, not code):** the exposed production Gitea token (seen in the 2026-10-10 audit) is rotated out-of-band; the migration makes its replacement the `docs_token` value.
@@ -0,0 +1,46 @@
# Phase 174 — Sources (git sources) page: the card stack regains its 1.25rem rhythm
**Source:** Owner bug report 2026-10-21 (screenshot of `/git-sources.html` — "the bubbles on the sources page are really smooshed together": the "Git sources" header card, the "Add a git source" card, the "Upload a source archive" card, and the sources table card stack flush — only their 1px borders separate them).
**Story:** n/a (owner bug report, this chat 2026-10-21)
**Context:** `frontend/index.html` — `#view-git-sources` (L646): `.container.git-sources-shell` holds TWO children — the anonymous `.sources-gate` (`#git-sources-gate`, L651) and the single `#git-sources-content` wrapper (L672) that wraps **every** visible card: `.page-head`, the `#git-sources-load-error` alert (hidden), the `#git-source-env-note` (hidden), `#git-source-form`, `#archive-upload-form`, and the `.table-wrap#git-sources-table-wrap`; `frontend/assets/styles.css` — `.git-sources-shell` (L3193–3200: `display:flex; flex-direction:column; gap: 1.25rem; flex:1` — the gap only separates the gate from the wrapper), `#git-sources-content` (**no rule at all** — a plain div, so its children stack with zero vertical rhythm — the root of the "smooshed" look), the phase-93 `.page-head` card (L2423–2428), the phase-27/49 form cards `#git-source-form, #archive-upload-form` (L3207+), the ≤640px block (L6400–6403: the forms go column — they compose with the fix unchanged), `frontend/assets/git-sources.js` (L294, L1515–1519: the gate↔content reveal — `contentEl.hidden` flips only); `tests/e2e/auth_helpers.py::login` (the admin E2E sign-in), `tests/e2e/test_admin_settings_layout.py` (the phase-171 bounding-box rhythm E2E — the house pattern this suite copies), `tests/unit/test_git_source_branch_ui.py` (the static CSS/HTML/JS pin style this phase extends).
## Objective
The five stacked cards on the Sources page (the navbar "Sources" link → `/git-sources.html`) get one uniform 1.25rem gap between them — CSS only, reusing the shell's **existing** `gap: 1.25rem` token (the phase-171 `#settings-content` idiom applied to the one remaining admin view whose cards live inside a wrapper div). Nothing else changes: no JS, no `app/` code, no new palette/spacing value, the gate + load-error + env-note reveal lifecycles byte-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the rhythm (task 01).** `#git-sources-content` becomes `display: flex; flex-direction: column; gap: 1.25rem;` — the **same** token `.git-sources-shell` already uses (L3196) and the value phase 171 gave `#settings-content`. One rule, placed directly after the `.git-sources-shell` rule with a phase-174 comment explaining the wrapper-swallowed-the-shell-gap root cause. The children (page-head card, load-error alert, env note, the two form cards, the table card) then stack with one uniform gap at every viewport.
- **D2 — the hidden children (task 01, pinned in 02).** The two `hidden` children (`#git-sources-load-error`, `#git-source-env-note`) keep the UA `[hidden] → display:none` even inside a flex parent (the parent's `display` never overrides a child's own `display`), so `git-sources.js`'s reveal lifecycle (L1515–1519) is untouched; a revealed error/note simply joins the rhythm. The anonymous gate↔content flip (L1515–1519: `contentEl.hidden = false`) works on a flex element exactly as on a block.
- **D3 — no other changes.** `.git-sources-shell` (its `flex: 1` + gap stay — the gate↔content separation at anonymous), `git-sources.js`, `app/` server code, the form/table card rules, and the ≤640px media queries (L6400–6403) are all untouched; the fix is a single wrapper rule.
## Dependencies
- `171_admin_settings_layout` (complete) — the idiom this phase reuses (wrapper div → flex column + the existing 1.25rem token).
- `93_theme_semantic_completion` (complete) — the `.page-head` card language the rhythm spaces out (context only).
## Tasks
1. `01_content_card_rhythm.md` — `#git-sources-content` becomes the flex-column rhythm wrapper (the one CSS rule); the static pins.
2. `02_e2e_sources_card_rhythm.md` — `tests/e2e/test_git_sources_card_rhythm.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_git_sources_card_rhythm.py` (new module, the `test_git_source_branch_ui.py` pin style): the `#git-sources-content` rule in `styles.css` with `display: flex` / `flex-direction: column` / `gap: 1.25rem` (the exact token — a regression to a different value fails), the `.git-sources-shell` gap staying `1.25rem` (unchanged), the `#git-sources-content` div present in `index.html` inside `.git-sources-shell` with the gate as a sibling (the structural pin — if the cards ever move back to the shell's direct children the rule becomes dead weight and the pin must say so).
- **E2E (AGENTS rule 4, task 02):** `tests/e2e/test_git_sources_card_rhythm.py`, run **in isolation** — the rhythm contract proven in a browser at 1440px and 375px, plus the hidden-children pin and the standard a11y/no-CDN light pass.
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's) and the gates above are green.
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (this phase is a visual fix with no setup/command/config surface — expected: no doc edits).
## Completion Criteria
- [ ] On `/git-sources.html` (admin), the five cards (page-head → add form → upload form → table, plus a revealed load-error when one is forced) stack with one uniform ≈20px gap — no two card edges touch (the E2E bounding-box pins, at 1440px and 375px).
- [ ] `#git-sources-content`'s hidden children (`#git-sources-load-error`, `#git-source-env-note`) stay hidden in the flex wrapper (E2E pin); the anonymous gate still shows and the content stays hidden for anonymous (regression).
- [ ] The ≤640px mobile layout is intact (the forms column-stack per L6400–6403; the table scrolls inside its card — the E2E mobile pin).
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov` green **in isolation**; `test_git_sources_admin.py` + `test_archive_upload_sources.py` + `test_git_source_dates.py` green in isolation (the regression set).
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(sources): Sources (git sources) page — the card stack regains its 1.25rem rhythm"
```
@@ -0,0 +1,45 @@
# Task 01 — `#git-sources-content` becomes the flex-column rhythm wrapper
**Phase:** `174_git_sources_card_rhythm` · **Story:** n/a (owner bug report 2026-10-21)
## Objective
The one CSS rule that restores the vertical rhythm on the Sources page: `#git-sources-content` becomes a flex column with the shell's existing `1.25rem` gap — so the five stacked cards (page-head, load-error, env-note, add form, upload form, table card) stop stacking flush.
## Work
1. `frontend/assets/styles.css` — directly after the `.git-sources-shell` rule (L3193–3200), add the `#git-sources-content` rule with a phase-174 comment block (the house comment style: root cause + the reused token + the hidden-children behavior):
```css
/* Phase 174 (owner bug report 2026-10-21): the shell's gap only
separates the gate from this wrapper — every visible card
(page-head, the load-error alert, the env note, the two form
cards, the table card) is a child of THIS div, which was a
plain block, so the cards stacked flush (1px borders touching).
The phase-171 #settings-content idiom: the wrapper itself is
the rhythm column, reusing the shell's own 1.25rem token — no
new spacing value. The hidden children (#git-sources-load-error,
#git-source-env-note) keep the UA [hidden] display:none inside
a flex parent, so git-sources.js's reveal lifecycle is
untouched; a revealed one joins the rhythm. */
#git-sources-content {
display: flex;
flex-direction: column;
gap: 1.25rem;
}
```
No other CSS change. Do not touch `.git-sources-shell`, the form/table card rules, or the ≤640px block (L6400–6403 — the forms' column-stacks compose with the new wrapper unchanged).
2. `tests/unit/test_git_sources_card_rhythm.py` — new module (the `test_git_source_branch_ui.py` pin style: `FRONTEND`/`STYLES_CSS`/`SHELL_HTML` path constants, raw text assertions, module docstring citing the phase + the owner report):
- the `#git-sources-content` rule exists in `styles.css` and carries `display: flex`, `flex-direction: column`, and `gap: 1.25rem` **exactly** (extract the rule body with a small regex over the `#git-sources-content { … }` block — a regression to any other gap value or a missing flex line fails);
- the `.git-sources-shell` rule still carries `gap: 1.25rem` (the fix must not have "fixed" the shell — D3);
- the structural pin: in `index.html`, `#git-sources-content` sits inside `.container.git-sources-shell` and the `.sources-gate` (`#git-sources-gate`) is a sibling of it (so the rule is not dead weight — if a future refactor moves the cards to the shell's direct children, this pin names the dead rule);
- `#git-sources-load-error` and `#git-source-env-note` still carry `hidden` in `index.html` (the D2 premise the E2E pin proves in a browser).
## Testing & Quality
- Unit: the pins above; `uv run pytest tests/unit/test_git_sources_card_rhythm.py -v` green.
- Coverage: **>90%** on `app/` (unchanged by this task — the gate must stay green, not drop).
- No `app/` code change; no JS change.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_git_sources_card_rhythm.py -v` green.
- [ ] `uv run pytest` green (the full unit + integration suite — no pin broke).
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] README.md/AGENTS.md need no change for this task (visual-only; the phase's final pass re-checks both project-wide).
- [ ] no behavior change in completed work (the gate reveal, the load-error retry, the env note, the form lifecycles are untouched — the rule only adds inter-card spacing).
@@ -0,0 +1,33 @@
# Task 02 — the phase's dedicated Playwright E2E suite + gates + commit
**Phase:** `174_git_sources_card_rhythm` · **Story:** n/a (owner bug report 2026-10-21)
## Objective
Prove the rhythm in a browser — `tests/e2e/test_git_sources_card_rhythm.py` (the phase's dedicated Playwright suite, AGENTS rule 4): the cards on `/git-sources.html` stack with one uniform gap at 1440px and 375px, the hidden children stay hidden in the flex wrapper, and the anonymous gate is intact. Then the regression set, the full gates, and the atomic commit.
## Work
1. `tests/e2e/test_git_sources_card_rhythm.py` — new module, run **in isolation** (`uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov`; DB must be up: `podman compose up -d db`). House shape: module docstring (the owner report + the run command + the test→contract map), the session `mock_llm`/`app_server`/`app_url`/`db_ready`/`page` fixtures from `tests/e2e/conftest.py`, `tests/e2e/auth_helpers.py::login` for the admin sign-in, a per-test `_truncate_kb`-style reset so each test's table state is its own (the `test_git_sources_admin.py` pattern — this suite is UI-only: it needs NO sync, no git, no network; the app can boot plain, the cards render regardless of row count).
Tests (four):
- `test_cards_stack_with_uniform_rhythm_desktop` — 1440×900, admin (`login(page, app_url, next="/git-sources.html")`): `#git-sources-content`'s computed `display` is `flex` and its `gap` resolves to 20px; the bounding boxes of the four visible cards — `.page-head` → `#git-source-form` → `#archive-upload-form` → `#git-sources-table-wrap` — are **pairwise disjoint** and each successive gap is ≈20px (`box_{n+1}.top − box_n.bottom == 20 ± 2`); no two card edges touch (the pre-fix regression: the gaps were ~0 — the 1px borders).
- `test_cards_stack_with_uniform_rhythm_mobile` — 375×720, admin: the same four cards are pairwise disjoint with ≈20px gaps; the two form cards column-stack per the ≤640px block (L6400–6403 — assert `#git-source-form` computed `flex-direction: column`) and the table scrolls **inside** its card (the `#git-sources-table-wrap` horizontal overflow is contained — the card's right edge is left of the viewport, no page-level horizontal scroll: `document.scrollingElement.scrollWidth <= 375`).
- `test_hidden_children_stay_hidden_in_flex_wrapper` — admin: `#git-sources-load-error` and `#git-source-env-note` are not visible (the `[hidden]` UA rule survives the flex parent — `offsetParent is None` / `is_visible() == False`); then force the load-error's `hidden` off via the page (attribute removal) and assert it **joins the rhythm** (visible, and its box sits ≥18px below the element above it) — the D2 both-sides pin; anonymous: `#git-sources-gate` visible, `#git-sources-content` hidden, no `/api/git-sources` call (the `test_git_sources_admin.py` anonymous contract, copied — the gate↔content flip is unchanged by the fix).
- `test_sources_page_a11y_and_no_cdn_basics` — the standard light pass (AGENTS rule 5/6, the `test_git_sources_admin.py` a11y test shape): landmarks on the view, the labeled form controls, the 3px `:focus-visible` outline on a table-row control (keyboard Tab), same-origin assets only (no external `<script src>`/`<link href>`).
2. **Regression set (in isolation, in order):** `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`, then `test_archive_upload_sources.py`, then `test_git_source_dates.py` (the three suites that touch the git-sources view's gate/forms/table — all must be green with the wrapper now flex).
3. **Full gates (the `.agents/validate.sh` set):** `uv run pytest --cov=app --cov-report=term-missing` (the entire project suite 100% passing + `app/` coverage >90% — this phase changes no `app/` code, the floor must hold, not drop), `uv run ruff check .`, `uv run pyright`.
4. **Final documentation & consistency pass:** review `README.md` and `AGENTS.md` in full against the project as it stands after this phase; fix every inconsistency found, **including pre-existing ones** (expected for this phase: no doc edits — visual-only).
5. **Commit + move:** the atomic commit, then move the phase dir to `.agents/phases/complete/` (the `.agents/` tree is tracked — commit it with the phase per AGENTS rule 8).
## Testing & Quality
- E2E: the dedicated suite green **in isolation** + the three-suite regression set green in isolation.
- Coverage: **>90%** on `app/` (unchanged — the floor holds).
- Lint/types: ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_sources_card_rhythm.py -v --no-cov` green **in isolation** (DB up).
- [ ] `uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov`, `test_archive_upload_sources.py`, `test_git_source_dates.py` — each green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing`: the entire project suite 100% passing, `app/` coverage >90%.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] README.md and AGENTS.md accurate and consistent with the whole project (verified in the final pass).
- [ ] one atomic `--no-gpg-sign` Conventional Commit:
`git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(sources): Sources (git sources) page — the card stack regains its 1.25rem rhythm"`
- [ ] the phase dir moved to `.agents/phases/complete/174_git_sources_card_rhythm/`.
@@ -0,0 +1,54 @@
# Phase 175 — History page: paginate the saved-chat list
**Source:** TODO.md L3 (owner 2026-10-10): "Need pagination on the history page - it gets long fast"
**Story:** n/a (owner request, this chat 2026-10-10)
**Context:** `app/api/chats.py` — the module docstring's route list (L60–66) and `list_chats` (L175–188): the admin-only `GET /api/chats` selects **all** `SavedChat` rows (`updated_at desc, id desc`), reads `current_sources_version` once per request, and returns a `SavedChatList` (the `chats` key only); `app/schemas.py` — `SavedChatRow` (L1699) and `SavedChatList` (L1725: `chats: list[SavedChatRow]`); `frontend/assets/history.js` — `loadChats()` (fetches bare `/api/chats`, re-entrant row replacement), the `#history-refresh` binding, the `showEmptyIfLast()` / `showEmptyState()` helpers, the `#history-status` live region; `frontend/index.html` — `#view-history` (L976+): the `.page-head` (h1 + the Refresh pill), `#history-gate`, `#history-status`, and `.table-wrap#history-table-wrap` (L1044) → `table.history-table` with `#history-tbody` (a hidden `#history-empty-row` ships in the tbody); `frontend/assets/styles.css` — the house `.table-wrap` scroll region (L2836–2844: `max-height: min(70vh, 64rem)` + the sticky thead), the `.history-refresh` pill language (L3995–4018, incl. the `:disabled` opacity + the svg sizing), the ≤640px full-width block (L6471), `.history-table` (L4050+); `tests/integration/test_chats_api.py` — the list pins (L518 `test_list_empty` asserts the body byte-exactly `{"chats": []}` — must absorb the new `total` key; L524 `test_list_orders_by_updated_at_desc` — the `ROW_KEYS` set is per-row and unaffected; L557 `test_list_reports_message_count`); `tests/e2e/test_chat_history.py` (the DB-isolation idiom: distinctive titles, never assert on absolute row counts, delete created rows in a `finally`), `tests/e2e/auth_helpers.py::login`, `tests/e2e/conftest.py` (the shared `app_server` / `mock_llm` / `browser` fixtures); `tests/unit/test_git_sources_card_rhythm.py` (the static-pin style task 02 extends).
## Objective
The History page's saved-chat list is paginated **server-side** — `GET /api/chats` accepts `?limit=` (default 25, clamped 1–100; out-of-range 422) and `?offset=`, and reports the full `total` on every response; the view gains a Prev / "Page N of M" / Next pager under the table that walks the pages while the re-entrant load contract, the anonymous gate, the empty state, and the Refresh behavior stay intact. The full list is no longer fetched — a 30-row history ships 25 rows plus a count.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the query (task 01).** `list_chats` takes `limit: int = Query(25, ge=1, le=100)` and `offset: int = Query(0, ge=0)` (FastAPI validation → 422 on `limit=0`, `limit=101`, `offset=-1` — the house fixed-422 contract; no hand-rolled checks). The row fetch is the EXISTING select with `.offset(offset).limit(limit)` appended (the order is untouched — `updated_at desc, id desc`). `total`: one extra `select(func.count(SavedChat.id))` per request (two small queries; the once-per-request `current_sources_version` read is untouched). `SavedChatList` gains `total: int` — **required, no default** (the `stale` convention: the API must always compute it — there is no wire shape without the count).
- **D2 — wire compatibility (task 01, proven in 03).** The change is additive: the body gains one key (`total`), and the query string is optional (a bare `GET /api/chats` = the first 25 rows + `total`). The only consumers are `history.js` and the test suites. The ONE existing pin that breaks — `test_list_empty` (L518, asserts `{"chats": []}` byte-exactly) — is updated to `{"chats": [], "total": 0}` in task 01 (the only modification to an existing test in this phase). Per-row keys are untouched (`ROW_KEYS` stays green).
- **D3 — the pager (task 02).** A `.history-pager` div in `#view-history`, directly BELOW `.table-wrap#history-table-wrap` (inside the `.history-shell` container). Contents, in order: a Prev button (the house inline-SVG chevron `aria-hidden` + the visible "Prev" label + `aria-label="Previous page of saved chats"`), a `#history-pager-status` span ("Page N of M"), a Next button (mirrored, `aria-label="Next page of saved chats"`). The buttons reuse the `.history-refresh` pill language (solid brand pill, the `:disabled` opacity rule, the svg sizing — **no new palette variable, B3**, no new spacing token). The pager is **hidden unless `total > limit`** (A4: no dead "Page 1 of 1" control; it disappears with the empty state). At the edges: Prev is `disabled` on page 1, Next `disabled` on the last page (the real `disabled` attribute — the Refresh idiom).
- **D4 — the load contract (task 02).** `loadChats(offset)` fetches `/api/chats?limit=<PAGE_SIZE>&offset=<offset>` (the module keeps `PAGE_SIZE = 25` — matching the server default; A1: no size picker in the UI), and reads `total` from the body. Module state: the current `offset` (starts at 0). Prev/Next adjust the offset by `PAGE_SIZE` (the disabled edges make clamping trivial), re-run the re-entrant load (rows are **replaced**, never appended), and land `Showing <first>–<last> of <total>.` in `#history-status` (the never-stale contract — every page change is announced). **Refresh re-fetches the *current* page** (A5 — `refreshChats` passes the module's current `offset`; it never resets to 0). The empty-state helpers (`showEmptyIfLast` / `showEmptyState`) stay unchanged — they are DOM-local. A local Delete does NOT recompute the pager (the last page's rows can vanish locally while the pager's `total` is still the server's last truth; the next load — a page click or a Refresh — corrects it). A 0-row fetch (a deleted-behind-us race) shows the empty state and `updatePager` runs on the fresh `total`.
- **D5 — anonymous (task 02, pinned in 03).** The whoami gate is unchanged: anonymous sees `#history-gate`, the table wrap hides, the pager hides (its own `hidden` default — the JS never touches it on the anonymous branch), and NO `/api/chats` request is made (the E2E pins the request log).
- **D6 — mobile (task 02).** ≤640px: the pager buttons go full width (the `.history-refresh` ≤640px idiom, L6471) — one small addition to the existing media-query block, no new values.
## Dependencies
- `174_git_sources_card_rhythm` (todo, the preceding phase)
- `50_chat_history` (complete) — the list endpoint + the history view this phase paginates
- `156_table_ux_polish` (complete) — the `.table-wrap` scroll region + the pill button language the pager reuses (context only)
## Tasks
1. `01_api_pagination.md` — `GET /api/chats` takes `?limit=` / `?offset=` and `SavedChatList` gains `total`; the integration pins (incl. the `test_list_empty` wire-pin update).
2. `02_pager_ui.md` — the `.history-pager` control (`index.html` + `history.js` + `styles.css`, D3–D6); the static pins.
3. `03_e2e_history_pagination.md` — `tests/e2e/test_history_pagination.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Integration:** `tests/integration/test_chats_api.py` extended (task 01) — the default page + `total`, explicit limit/offset slices, order preserved across pages (the concatenation of the pages == the full-list order), 422 on out-of-range params, an offset beyond the end → empty `chats` with the honest `total`.
- **Unit:** `tests/unit/test_history_pagination.py` (new module, task 02, the `test_git_sources_card_rhythm.py` pin style): the `#history-pager` structure in `index.html` (below `#history-table-wrap`, the button `aria-label`s, the `aria-hidden` SVGs, the shipped `hidden`), the pager rules in `styles.css` (the flex row + the ≤640px block + the token reuse — every `var(--…)` the pager rules use is already declared in `:root`), and the `history.js` contract (the parameterized fetch, the `total` read, the hidden-when-single-page branch, the disabled edges, refresh-keeps-current-page, the `Showing …` status line).
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_history_pagination.py`, run **in isolation** — the 30-row fixture (via the public `POST /api/chats`, distinctive titles, `finally` cleanup — the `test_chat_history.py` DB-isolation idiom), the page-1→2→1 walk with row counts + pager state + status lines, refresh-on-page-2 stays on page 2, delete-to-empty on the last page, the single-page hide, the anonymous gate pin (no `/api/chats` request in the log), and the standard a11y / no-CDN light pass.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
- The phase is "Complete" only when the **entire** project test suite passes 100% (every test in the project, not just this phase's) and the gates above are green.
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (the `?limit=` / `?offset=` params + `total` are a user-visible API surface — update `README.md` only if it documents the `GET /api/chats` shape; the pager is UI with no setup/command/config surface).
## Completion Criteria
- [ ] `GET /api/chats` (bare) returns the first 25 rows (`updated_at desc, id desc`) + `total` (the full count); `?limit=10&offset=20` returns the correct slice; out-of-range `limit` / `offset` 422; an offset beyond the end returns `{"chats": [], "total": N}` (integration pins).
- [ ] On `/history.html` (admin, a 30-row fixture): the table shows 25 rows and the pager reads "Page 1 of 2" (Prev disabled, Next enabled); Next → 5 rows, "Page 2 of 2" (Next disabled, the page-1 rows gone — replaced, not appended); Prev → back to 25 (E2E pins); every page change lands a `Showing X–Y of Z.` line in the live region.
- [ ] Refresh on page 2 re-fetches page 2 (E2E pin — never resets to page 1); deleting the last page's rows shows the empty state locally; ≤1 page → the pager is fully hidden (no dead control; E2E pin).
- [ ] Anonymous: the gate shows, the table + pager are hidden, and NO `/api/chats` request appears in the log (E2E pin).
- [ ] The ≤640px pager layout is intact (full-width buttons — the E2E mobile pin).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` green **in isolation**; `test_chat_history.py` + `test_history_page_width.py` + `test_share_chat.py` + `test_stale_saved_chats.py` + `test_history_copy.py` green in isolation (the regression set).
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
## Commit
```bash
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(history): paginate the saved-chat list — server-side ?limit/?offset with total + the view's pager"
```
@@ -0,0 +1,38 @@
# Task 01 — `GET /api/chats` is paginated server-side (`?limit=`, `?offset=`, `total`)
**Phase:** `175_history_pagination` · **Story:** n/a (owner request 2026-10-10)
## Objective
The list endpoint ships a page instead of the full list: 25 rows by default, a `?limit=` (1–100) / `?offset=` the client can walk, an honest `total` on every response, and a FastAPI 422 on out-of-range params — the wire change is additive everywhere (a bare `GET` = the first page + the count).
## Work
1. `app/api/chats.py` — `list_chats` (L175–188):
- imports: add `Query` to the `fastapi` import line; add `func` to the `sqlalchemy` import line (`from sqlalchemy import func, select, text`).
- signature: `limit: int = Query(25, ge=1, le=100)` and `offset: int = Query(0, ge=0)` (declared before the `db` dependency; FastAPI serves the house field-message 422 for `limit=0` / `limit=101` / `offset=-1` — no hand-rolled validation).
- body: the existing select gains `.offset(offset).limit(limit)` appended (the order is untouched — `SavedChat.updated_at.desc(), SavedChat.id.desc()`); `total = db.scalar(select(func.count(SavedChat.id)))` (one small count query per request — the once-per-request `current_sources_version` read is untouched).
- return `SavedChatList(chats=[_to_row(row, current_version) for row in rows], total=total)`.
- docstrings: `list_chats`' docstring + the module docstring's route list (L60–66, the "``GET`` (list, …)" entry) — describe the two params + `total` in the house thoroughness (the 422 contract, the default, the once-per-request count read).
- NO other route in this file changes (detail / create / re-Save / delete / share / unshare / the public read + page route are untouched).
2. `app/schemas.py` — `SavedChatList` (L1725):
- add `total: int` — **required, no default** (the `stale` convention: the API must always compute it — there is no wire shape without the count); update the class docstring (`chats` + `total` described).
- `SavedChatRow` and its per-row key set (the `ROW_KEYS` in the integration suite) are untouched.
3. `tests/integration/test_chats_api.py` —
- **update the ONE existing pin that breaks:** `test_list_empty` (L518): `assert r.json() == {"chats": []}` becomes `assert r.json() == {"chats": [], "total": 0}` (the D2 wire-pin update — the body is now the two-key shape, byte-exact).
- new pins (the file's existing idiom: the `admin_client` fixture, direct `SavedChat` ORM adds with staggered `updated_at` — the L524 idiom — or `POST /api/chats` creates; distinctive titles):
- `test_list_default_page_is_25_and_reports_total` — 30 rows with distinct, staggered `updated_at` → bare `GET /api/chats`: exactly 25 rows, `total == 30`, and the first page's `id` order == the first 25 of the full ordered list (`updated_at desc`);
- `test_list_limit_offset_slice` — the same 30 rows → `GET /api/chats?limit=10&offset=20`: exactly 10 rows (the last 10 of the ordered list — pinned by title), `total == 30`;
- `test_list_order_preserved_across_pages` — `?limit=10` at offsets 0 / 10 / 20, the three pages concatenated == the full ordered `id` sequence (no duplicates, no gaps);
- `test_list_rejects_out_of_range_params` — `?limit=0`, `?limit=101`, `?offset=-1` each 422 (the house fixed-422 contract; one request per case, status asserted);
- `test_list_offset_beyond_total_returns_empty_with_total` — `?offset=100` on the 30-row fixture → `{"chats": [], "total": 30}` (an honest count, not an error).
## Testing & Quality
- Integration: the new pins + the updated `test_list_empty`; `uv run pytest tests/integration/test_chats_api.py -v` green.
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- Lint/types: `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] The bare `GET /api/chats` returns 25 rows + `total`; the explicit `limit` / `offset` slices are correct; out-of-range params 422; an offset beyond the end is honest.
- [ ] `test_list_empty` updated to the two-key body (and green); every other existing pin green **unchanged** (per-row shape byte-exact — `ROW_KEYS`).
- [ ] `uv run pytest` green (the full unit + integration suite); `app/` coverage >90%; ruff + pyright clean.
- [ ] README.md: update the endpoint's shape only if `README.md` documents `GET /api/chats` (check first; if absent, no edit) — AGENTS.md needs no change (no setup/command/config surface).
- [ ] No behavior change in completed work (create / detail / re-Save / delete / share / unshare / the public read + page route untouched — the list route's only diff is the two query params + the `total` key + the select's `.offset()/.limit()`).
@@ -0,0 +1,63 @@
# Task 02 — the History view's pager (Prev / "Page N of M" / Next)
**Phase:** `175_history_pagination` · **Story:** n/a (owner request 2026-10-10)
## Objective
The view walks the paginated list: the pager control under the table (D3), the parameterized re-entrant load (D4), hidden-when-single-page / hidden-when-empty (A4), refresh-keeps-current-page (A5), the untouched anonymous gate (D5), the ≤640px block (D6) — all in existing tokens and house idioms.
## Work
1. `frontend/index.html` — in `#view-history` (L976+), **directly after** the `.table-wrap#history-table-wrap` div (L1044 — the table card is the view's last child today; the pager lands below it, inside the same `.history-shell` container), with a phase-175 comment (the house comment style — the reason + the idioms it reuses):
```html
<!-- Phase 175 (TODO.md L3): the list pager — visible only when
there is more than one page (total > the 25-row page size —
history.js toggles `hidden`), so a one-page list never shows a
dead control. The buttons reuse the .history-refresh pill
language (the house inline-SVG glyph + visible label +
aria-label; the ≤640px full-width block joins
.history-refresh). #history-pager-status carries "Page N of M";
page-change outcome lines land in #history-status (the existing
live region). -->
<div class="history-pager" id="history-pager" hidden>
<button type="button" class="history-pager-btn" id="history-pager-prev" aria-label="Previous page of saved chats">
<svg class="history-pager-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M15 18l-6-6 6-6"/></svg>
<span class="history-pager-label">Prev</span>
</button>
<span class="history-pager-status" id="history-pager-status"></span>
<button type="button" class="history-pager-btn" id="history-pager-next" aria-label="Next page of saved chats">
<span class="history-pager-label">Next</span>
<svg class="history-pager-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M9 18l6-6-6-6"/></svg>
</button>
</div>
```
The anonymous gate / page-head / `#history-status` span / table markup are all untouched.
2. `frontend/assets/history.js` —
- module header comment: a phase-175 block (the house comment style — the parameterized fetch + `total` + the pager lifecycle).
- element lookups (scoped to `root`, the house style): `#history-pager`, `#history-pager-prev`, `#history-pager-next`, `#history-pager-status`.
- state: a `const PAGE_SIZE = 25;` (matches the server default — A1, no size picker in the UI; the comment says so, and the E2E suite catches drift) + `let offset = 0;` (the current page's offset).
- `loadChats(pageOffset)`: the fetch becomes `/api/chats?limit=` + `PAGE_SIZE` + `&offset=` + `pageOffset` (manual string join — the house style in this file); on 200, `const { chats, total } = await r.json();` then `updatePager(total)`. The 0-row / non-2xx / network-error branches keep their exact current behavior (empty state + the house error lines; a failed load leaves the pager as it was — the previous state is the last known truth).
- `updatePager(total)`: `const pages = Math.ceil(total / PAGE_SIZE);` — if `total <= PAGE_SIZE` → the pager is `hidden` (A4: no "Page 1 of 1" dead control); otherwise the pager shows, `#history-pager-status` gets `Page ` + (`offset / PAGE_SIZE + 1`) + ` of ` + `pages`, `#history-pager-prev.disabled = (offset === 0)`, `#history-pager-next.disabled = (offset + PAGE_SIZE >= total)`.
- Prev/Next binding (ADMIN branch only — the same gate as Refresh; the anonymous branch never binds them, D5): click → `offset += / -= PAGE_SIZE` (the disabled edges make clamping trivial) → `loadChats(offset)` → on success land `Showing ` + (`offset + 1`) + `–` + `Math.min(offset + PAGE_SIZE, total)` + ` of ` + `total` + `.` in `#history-status` (the existing live region — the never-stale contract).
- `refreshChats()`: pass the module's current `offset` to `loadChats` (A5 — re-fetch the *current* page; never reset to 0).
- boot: the initial `loadChats(0)` — the view opens on page 1.
- a local Delete does NOT touch the pager or `offset` (D4 — the next load corrects it).
3. `frontend/assets/styles.css` —
- the `.history-pager` rule (placed near the `.history-refresh` block, L3995–4018, with a phase-175 comment — root cause + token reuse): `display: flex; align-items: center; gap: 0.75rem;` (the existing `0.75rem` gap the page already uses — no new spacing token). The buttons: add `.history-pager-btn` to the existing `.history-refresh` declaration (a selector union on the pill rule, the `:disabled` rule, and the `svg` sizing rule — the executor's choice of union vs duplication, but **no new palette variable (B3)** and no value not already in the file); `#history-pager-status` needs no rule beyond the inherited text (if a min-width or text-center is added, it uses existing tokens only).
- the ≤640px media-query block (L6471): add `.history-pager-btn` to the existing `.history-refresh` full-width line (the house idiom).
- NOTHING else in styles.css changes (the `.table-wrap` scroll region, the sticky thead, the `.history-table` rules, the `.history-status` rule are all untouched).
4. `tests/unit/test_history_pagination.py` — new module (the `test_git_sources_card_rhythm.py` pin style: `FRONTEND` / `STYLES_CSS` / `SHELL_HTML` path constants, raw text assertions, a module docstring citing the phase + the TODO line):
- `index.html`: `#history-pager` sits in `#view-history` and AFTER `#history-table-wrap` (the structural pin — the pager is under the table card, not in the page-head); both buttons carry their exact `aria-label`s; both SVGs carry `aria-hidden="true"`; the div ships `hidden` (the hidden-when-single-page default the JS flips);
- `styles.css`: the `.history-pager` rule with `display: flex` and `gap: 0.75rem`; `.history-pager-btn` is covered by the pill / `:disabled` / svg-sizing declarations (extract with the small-regex house style — pinned by presence, not by a particular CSS authoring choice); the ≤640px block carries `.history-pager-btn` full width; **no new token** — every `var(--x)` used in any pager rule is declared in the file's `:root` block (the B3 pin);
- `history.js`: the parameterized fetch URL (`/api/chats?limit=` + the `&offset=` join), the `total` read from the JSON body, `PAGE_SIZE = 25`, the hidden-when-single-page branch (`total <= PAGE_SIZE`), the two `disabled` edge lines, the `Showing ` status-line format, the refresh path passing the current `offset`, and the pager lookups scoped to `root`.
## Testing & Quality
- Unit: the pins above; `uv run pytest tests/unit/test_history_pagination.py -v` green.
- Coverage: **>90%** on `app/` (this task changes no `app/` code — the gate must stay green, not drop).
- Lint/types: `uv run ruff check . && uv run pyright` clean.
- The E2E proof is task 03 (the browser-level pins); this task's pins keep the static contract honest between runs.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_history_pagination.py -v` green.
- [ ] `uv run pytest` green (the full unit + integration suite — no pin broke).
- [ ] ruff + pyright clean; no new CSS token (B3 pin green).
- [ ] README.md/AGENTS.md need no change for this task (UI with no setup/command/config surface; the phase's final pass re-checks both project-wide).
- [ ] No behavior change in completed work (the anonymous gate, the Refresh lifecycle, the empty-state helpers, the Delete/Share row actions are untouched — the load only gains two query params + `total` + the pager state).
@@ -0,0 +1,46 @@
# Task 03 — E2E (Playwright): the History page's pager
**Phase:** `175_history_pagination` · **Story:** n/a (owner request 2026-10-10)
## Objective
The phase's dedicated E2E suite (AGENTS rule 4 — one file per phase, run in isolation): `tests/e2e/test_history_pagination.py` proves the pager contract in a real browser against the real app + DB, then the regression set, the full gates, and the atomic commit.
## Work
1. `tests/e2e/test_history_pagination.py` — new file (the `test_chat_history.py` house pattern: a module docstring citing the phase + the TODO line + the run-in-isolation instructions `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` (DB up: `podman compose up -d db`); the `conftest` fixtures — `app_server` / `page`; `auth_helpers.login` for the admin cookie). The suite touches ONLY `saved_chats` (no KB reset idiom needed — it never drives the chat page):
- **fixture helper** (the `test_chat_history.py` DB-isolation idiom — distinctive titles, never assert on absolute row counts, delete the rows it creates in a `finally` via admin `DELETE /api/chats/<id>`): `make_chats(n)` — POST `/api/chats` (the public write surface) `n` conversations, each with a distinctive auto-title (e.g. `Pagination probe <i> of <n> — what is the capital of France?`) and a minimal `messages` payload (the house `_simple_conversation` shape); then, to make the `updated_at desc` order fully deterministic, re-Save (`PUT /api/chats/<id>` — public, bumps `updated_at` via `onupdate`) the rows in a controlled loop, so the last re-Saved chat is row 1. Pin the order by the first row of page 1 (the last re-Saved) and the first row of the last page (the earliest re-Saved) — never by the full middle order.
- `test_pager_shows_page_1_of_2_for_30_rows` — admin, `/history.html`: the table holds 25 rows, the pager is visible, `#history-pager-status` reads "Page 1 of 2", `#history-pager-prev` is `disabled`, `#history-pager-next` is enabled, `#history-status` carries "Showing 1–25 of 30.";
- `test_next_walks_to_page_2` — click Next: the table holds 5 rows (pinned by the fixture's earliest-re-Saved title first — the order), "Page 2 of 2", Next disabled, Prev enabled, the live region reads "Showing 26–30 of 30."; the 25 page-1 rows are GONE (replaced, never appended — the row count is exactly 5);
- `test_prev_walks_back_to_page_1` — from page 2, click Prev: 25 rows again, "Page 1 of 2", Prev disabled, "Showing 1–25 of 30.";
- `test_refresh_on_page_2_stays_on_page_2` — walk to page 2, click `#history-refresh` (wait for its re-enable), still 5 rows + "Page 2 of 2" (A5 — never reset to page 1), and the "Saved chats refreshed." line lands;
- `test_delete_all_rows_of_page_2_then_prev_single_page_hides_pager` — on page 2 (5 rows), use the inline two-step confirm (the `Delete? [Yes] [No]` pair appears — the owner-locked no-native-dialog pin) to delete all five: `#history-empty-row` becomes visible (the local delete contract — the pager is NOT recomputed on a local delete, D4); then click Prev: 25 rows, and the pager is now HIDDEN (total 25 → single page, A4 — the fresh `total` from the page-1 load corrects it);
- `test_single_page_hides_the_pager` — a 10-row fixture: the table lists all 10, the pager is fully hidden (A4 — no dead control);
- `test_anonymous_sees_gate_and_never_fetches` — no login: `#history-gate` visible, `#history-table-wrap` + `#history-pager` hidden, and NO `GET /api/chats` in the request log (the house log-pin idiom);
- `test_pager_mobile_375px` — viewport 375px: the pager buttons are full width (bounding-box pin — the phase-171 rhythm-suite idiom) and the table still scrolls horizontally inside its card;
- the standard a11y light pass (the house pattern from the phase-171/167 suites): the pager buttons' `:focus-visible` outline, the `aria-label`s exposed in the accessibility tree, the pager text contrast via the existing tokens (computed-style spot check), and the no-CDN pin (no external-origin `script`/`link` — the house integration check on the index page).
2. the regression set (green in isolation, UNCHANGED — task 01's `test_list_empty` pin update is the only touch to an existing test in this phase):
- `uv run pytest tests/e2e/test_chat_history.py -v --no-cov`
- `uv run pytest tests/e2e/test_history_page_width.py -v --no-cov`
- `uv run pytest tests/e2e/test_share_chat.py -v --no-cov`
- `uv run pytest tests/e2e/test_stale_saved_chats.py -v --no-cov`
- `uv run pytest tests/e2e/test_history_copy.py -v --no-cov`
- `uv run pytest tests/integration/test_chats_api.py -v`
3. the full gates:
- `uv run pytest` (the entire unit + integration suite — 100% green)
- `uv run pytest --cov=app --cov-report=term-missing` (app/ >90%)
- `uv run ruff check . && uv run pyright`
- the Documentation & Consistency final pass (README.md + AGENTS.md against the project as it stands — fix every inconsistency found, including pre-existing ones).
4. the atomic commit (the phase dir moves to `.agents/phases/complete/` **after** the commit):
```bash
git add app/ frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(history): paginate the saved-chat list — server-side ?limit/?offset with total + the view's pager"
```
## Testing & Quality
- E2E: the suite above green **in isolation**; the regression set green in isolation; the full gates green.
- The phase is "Complete" only when every test in the project passes 100% and the gates are green (the 00_phase.md contract).
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_history_pagination.py -v --no-cov` green in isolation.
- [ ] The five-file E2E regression set + the integration `test_chats_api.py` green (unchanged apart from task 01's wire-pin update).
- [ ] `uv run pytest` 100% green; `app/` coverage >90%; ruff + pyright clean.
- [ ] README.md + AGENTS.md verified consistent (the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,51 @@
# Phase 176 — Admin settings page: a usable Value-column width on mobile
**Source:** `TODO.md` L3 — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields." (the TODO's `image-1.png` screenshot: at a 375 px phone viewport the Value-column inputs render only 2–3 characters wide — "Br", "0.", "pr", "Wr", "Pc", "IN" — unreadable and unusable)
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-12)
**Context:** `frontend/index.html` — `#view-settings` (L1519), the `.table-wrap` card (L1596) holding `table.settings-table` with the static `thead` (L1599–1606: the Setting / Value / Source / Actions `th`s — the Value `th` at L1602 carries no class), `frontend/assets/settings.js` (the row render — `makeRow` builds `td.settings-name-cell` + `td.settings-value-cell` (the `.settings-control` control) + the Source badge cell + the Reset cell; the controls are type-correct per item and carry `width: 100%; min-width: 0`), `frontend/assets/styles.css` — `.settings-table` (L4742–4746: `width: 100%; min-width: 560px;` — the ONLY width floor, no per-column widths — the root cause: at 375 px the table renders at its 560 px floor and the Setting cell's long `white-space: nowrap` labels (L4754: `td { white-space: nowrap }`) devour it, collapsing the `min-width: 0` Value cell), the `.table-wrap` bounded-scroll card (L2836–2842: `overflow: auto; max-height: min(70vh, 64rem)` — horizontal scroll is the established contract; the ≤640px block's own comment: "the table's own horizontal scroll covers the columns"), the ≤640px mobile block (L6180–6529) with the existing phase-168 settings squeeze rules (L6484–6488), `tests/unit/test_settings_frontend_view.py` (the phase-168/171 static pin set, incl. `test_styles_css_settings_block_uses_existing_tokens_only` L516), `tests/e2e/test_admin_settings_layout.py` (phase 171 — the "no horizontal page overflow at 375px / 1440px" pin L482–486) and `tests/e2e/test_admin_settings_panel.py` (phase 168 — the regression set), PLAN.md §7.1 (≥44px touch targets on mobile; lists/tables ≥80–90% of container; the table scrolls INSIDE its card).
## Objective
On a phone, the Settings table's Value inputs are impossibly small (2–3 chars). The owner's direction: the table is already horizontally scrollable, so the inputs should be **slightly wider** — i.e. give the Value column a usable minimum width (and raise the table's scroll floor to match) so every `.settings-control` is readable and editable on a 375 px phone, absorbed by the card's existing horizontal scroll. The fix is **CSS-only, mobile-scoped** (the ≤640px block) — the desktop layout stays byte-identical, no `app/` code, no JS logic change.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the Value-column floor (task 01, owner-locked A2).** Inside the existing `@media (max-width: 640px)` block (L6180, appended beside the phase-168 settings squeeze rules L6484–6488), two rules:
- `.settings-table { min-width: 720px; }` — the mobile scroll floor (was 560 px, which the nowrap Setting labels exhaust). At a 375 px viewport the table renders at 720 px and scrolls horizontally INSIDE the `.table-wrap` card (overflow: auto) — the page itself never overflows (the phase-171 pin holds: the scroll is the card's, not the page's).
- `.settings-value-cell, .settings-value-th { min-width: 13rem; }` — the Value column (and its control, `width: 100%`) gets a 13 rem (≈208 px ≈ 14 mono chars at the control's 0.85 rem) floor at the phone. `13rem` is a plain length value — NO new CSS custom property (the house token rule B3 is about palette variables; a bare length literal in the mobile block is the same idiom as the existing `#settings-save { width: 100% }` rules there).
- To make the column floor deterministic (auto table layout can otherwise leave the `th` unconstrained), the Value `th` in `frontend/index.html` L1602 gains the class: `<th scope="col" class="settings-value-th">Value</th>` — static HTML (the no-HTML-string house rule binds the JS, not the templates; `settings.js` is untouched).
- **D2 — mobile-only scope (owner-locked A1).** Every new rule lives INSIDE the `@media (max-width: 640px)` block only. At ≥641 px (the 1440 px E2E desktop pin, and the 72 rem container ≈ 1152 px) the media query is inert: the table renders at card width as today, the Value column takes its natural auto share, and the desktop bytes of `styles.css`/`index.html` behavior are unchanged.
- **D3 — no other change (task 01).** `settings.js` (render / filter / save / reset / clear-all), the `.table-wrap` bounded-scroll + sticky-th contract (L2836–2848), the phase-171 `#settings-content` rhythm, the group bands, the Source badges, the mobile action-row squeeze rules (L6484–6488), `app/` server code — all byte-identical. Nothing animates (nothing for `prefers-reduced-motion` to still).
- **NOT touched:** `app/`, `settings.js`, the ≥641 px rules in `styles.css`, the other catalogue tables.
## Dependencies
- `171_admin_settings_layout` (complete) — the Settings page rhythm/band/cursor layout this phase extends, and the ≤640px mobile block's existing settings squeeze rules (L6484–6488) the new rules are appended beside.
- `168_admin_settings_panel` (complete) — the panel contract (the gate, the catalogue, the `.table-wrap` scroll card, the §7.4 Save lifecycle) that must stay behavior-identical.
## Tasks
1. `01_mobile_value_column_width.md` — the Value-column `min-width: 13rem` + the table `min-width: 720px` floor, both inside the ≤640px block; the `settings-value-th` class on the Value `th`.
2. `02_unit_pins.md` — `tests/unit/test_settings_frontend_view.py`: the static pins for the two mobile rules (present ONLY inside the ≤640px block) + the `settings-value-th` class pin on the static thead; the existing token-only pin stays green.
3. `03_e2e_mobile_value_width.md` — `tests/e2e/test_settings_mobile_value_width.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_settings_frontend_view.py` (the phase-168/171 static pin set, extended in task 02): the two new rules parse out of the ≤640px block with the exact values, the `settings-value-th` class is present on the settings `thead`'s Value `th`, and `test_styles_css_settings_block_uses_existing_tokens_only` stays green (no new palette variable — a bare length is not a token).
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_settings_mobile_value_width.py`, run **in isolation** — the width contract, proven in a browser at 375 px and 1440 px.
- **Coverage:** **>90%** on `app/` (this phase changes no `app/` code — the gate must stay green, not drop).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] At a 375 px viewport, every Settings `.settings-control` (the Value inputs) is ≥ 12 rem (≈192 px, the A3 pin threshold) wide — readable and editable (the E2E width pin).
- [ ] At 375 px the `.table-wrap` card scrolls horizontally (`scrollWidth > clientWidth`) while the PAGE does not overflow horizontally (the phase-171 pin at 375 px and 1440 px stays green).
- [ ] At a 1440 px viewport the Value input width is unchanged vs. pre-phase (the mobile rules are inert — no desktop regression; the phase-171 E2E suite stays green in isolation).
- [ ] The keyboard tab order still reaches filter → first row input → **Save settings** → **Clear all overrides** at 375 px (the phase-168/171 contract under the new column floor); the whole test issues ZERO external requests (rule 6).
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_settings_mobile_value_width.py -v --no-cov` green **in isolation**; `test_admin_settings_layout.py` (phase 171) + `test_admin_settings_panel.py` (phase 168) green in isolation.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-12, roadmap confirmation)
- **A1 — mobile-only scope.** The new width rules live ONLY in the existing `@media (max-width: 640px)` block; the desktop (≥641 px) layout stays byte-identical — the TODO is mobile-scoped.
- **A2 — the width values.** Table floor **720 px** (was 560 px — the phone scrolls the card horizontally, the established contract); Value column **min-width 13rem** → the input ≈ 208 px ≈ 14 mono characters, i.e. "slightly wider input fields". No new CSS custom property, no JS change.
- **A3 — the E2E pin threshold is 12 rem** (192 px) — a deliberate margin below the 13 rem (208 px) rule so the pin doesn't become brittle to sub-pixel rounding.
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): mobile settings table — a usable Value-column width (the card's horizontal scroll absorbs it)"
```
@@ -0,0 +1,28 @@
# Task 01 — the mobile Value column gets a usable width (the card's horizontal scroll absorbs it)
**Phase:** `176_mobile_settings_value_width` · **Source:** `TODO.md:3` — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields."
**Story:** n/a
## Objective
At a 375 px phone viewport the Settings table's Value inputs render 2–3 characters wide (the Setting cell's long `white-space: nowrap` labels exhaust the table's 560 px floor). Give the Value column a **13 rem floor** and raise the table's mobile scroll floor to **720 px** — both inside the existing `@media (max-width: 640px)` block, so the phone scrolls the card horizontally (the established contract) while every `.settings-control` is readable and editable. CSS only, plus the one `th` class that makes the column floor deterministic; the desktop (≥641 px) stays byte-identical (owner-locked A1).
## Work
1. `frontend/index.html` L1602 — the static `thead`'s Value cell: `<th scope="col">Value</th>` → `<th scope="col" class="settings-value-th">Value</th>`. Static HTML only (the no-HTML-string house rule binds the JS — `settings.js` stays byte-unchanged); the class exists so the column floor constrains the header cell, not just the `td`s, under auto table layout.
2. `frontend/assets/styles.css` — inside the existing `@media (max-width: 640px)` block (opens L6180), appended directly after the phase-168 settings squeeze rules (L6484–6488: `.settings-search` / `.settings-actions` stack + `#settings-save` / `#settings-clear-all` full width + `.settings-actions-cell` wrap), add ONE comment + two rules (the phase-176 header comment, stating the TODO L3 contract: the Value column's 13 rem floor makes the input ≈208 px ≈ 14 mono chars at the phone; the 720 px table floor + the card's `overflow: auto` (L2836) mean the phone scrolls the CARD, never the page):
- `.settings-table { min-width: 720px; }`
- `.settings-value-cell, .settings-value-th { min-width: 13rem; }`
No other rule moves. `13rem` / `720px` are bare length literals — NO new CSS custom property (B3 binds palette variables; a length in the mobile block is the idiom the neighboring `width: 100%` rules already use).
3. `frontend/assets/styles.css` L4742–4746 — update the `.settings-table` comment to note the mobile block (≤640px) raises the floor to 720 px for the Value column (one line; the rule itself stays here at 560 px for ≥641 px viewports).
- ASSUMPTION: A1 (mobile-only scope — the rules live ONLY in the ≤640px block; desktop byte-identical), A2 (the width values — 720 px table floor, 13rem Value floor) — both owner-locked 2026-10-12 in the phase-176 roadmap confirmation. No runtime guesses.
## Testing & Quality
- Unit: `tests/unit/test_settings_frontend_view.py` — the existing pins (incl. `test_styles_css_settings_block_uses_existing_tokens_only` L516) must stay green; the task-02 pins cover the new rules.
- Coverage: **>90%** on `app/` (unchanged code — the gate must stay green, not drop).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] In the ≤640px block of `styles.css`, the two new rules are present with the exact values (720 px / 13 rem) and NO `settings-value` rule appears outside that block (grep: the selectors occur only once, inside `@media (max-width: 640px) … }`).
- [ ] `frontend/index.html` L1602 carries `class="settings-value-th"`; `settings.js` is byte-unchanged (`git diff --stat` shows only `frontend/index.html` + `frontend/assets/styles.css` + the test pin).
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green; ruff + pyright clean.
- [ ] No `app/` change; the ≥641 px rules are byte-unchanged.
@@ -0,0 +1,28 @@
# Task 02 — unit pins: the two mobile width rules + the `settings-value-th` class
**Phase:** `176_mobile_settings_value_width` · **Source:** `TODO.md:3` — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields."
**Story:** n/a
## Objective
Pin task 01's CSS in `tests/unit/test_settings_frontend_view.py` the house way (the phase-168/171 static-pin idiom — `re.search` the rule body, assert the declarations): the two new rules exist with their exact values **inside the ≤640px block only**, and the `settings-value-th` class is on the settings `thead`'s Value `th`. The existing pins — incl. `test_styles_css_settings_block_uses_existing_tokens_only` (L516) — stay green: the new rules add a bare length, no `:root` custom property (B3 holds).
## Work
1. `tests/unit/test_settings_frontend_view.py` — a new section `# ---------- phase 176 (task 01): the mobile Value-column floor ----------` after the phase-171 section (after `test_settings_save_disabled_is_just_disabled_no_wait_cursor`, L714), with ONE test function `test_settings_mobile_value_column_floor` (the house one-test-per-contract idiom; the function docstring cites `TODO.md` L3 + owner-locked A1/A2, the way the phase-171 docstrings cite their TODO line):
- Extract the ≤640px block: `mobile = re.search(r"@media \(max-width: 640px\) \{([\s\S]*?)\n\}\n\n/\* Phase 46", css)` is too brittle — instead pin the rules by SELECTOR SCOPING: assert `css.count(".settings-value-cell, .settings-value-th")` is **exactly 1** and that the occurrence sits INSIDE the ≤640px block (locate the `@media (max-width: 640px)` opening at L6180 and the block's closing `}` at the end of the media region — the test may slice `css[mobile_open:media_close]` with the open/close markers; the phase-171 pin `test_settings_content_is_a_flex_column_with_the_shell_gap` (L589) already sets the rule-extraction idiom to follow).
- In the mobile block: the `.settings-table` rule declares `min-width: 720px` (the mobile floor — the base L4742 rule keeps `min-width: 560px` for ≥641 px; pin BOTH: the base `.settings-table` rule body contains `min-width: 560px` and NO `720px`, the mobile-block `.settings-table` rule body contains `min-width: 720px`).
- In the mobile block: the Value rule declares `min-width: 13rem` on both `.settings-value-cell` and `.settings-value-th`.
- `index.html` (via the file read the settings-view pins already use, `_read(FRONTEND / "index.html")`): the settings `thead` (the `table.settings-table` region — the `#settings-tbody` anchor) contains `<th scope="col" class="settings-value-th">Value</th>`.
2. `tests/unit/test_settings_frontend_view.py` — verify `test_styles_css_settings_block_uses_existing_tokens_only` (L516) still passes UNCHANGED (the new rules are inside the media block, not the settings block's `:root` scan scope; if its no-new-custom-property scan is a global grep for `--settings-` or similar, confirm it does not match `13rem`/`720px` — it cannot, they are bare lengths; run it and record the pass).
- ASSUMPTION: the media-block extraction uses open/closing markers (not a hardcoded line range) so the pin survives the two-line insertion above it — the phase-171 docstring's "the 640px mobile stacking composes with it" contract is the same robustness bar. (Not an owner decision: test-internal mechanics.)
## Testing & Quality
- Unit: `tests/unit/test_settings_frontend_view.py` — the new pin + all existing settings pins green.
- Coverage: **>90%** on `app/` (unchanged code).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_settings_frontend_view.py -v --no-cov` green, incl. the new `test_settings_mobile_value_column_floor` and the untouched `test_styles_css_settings_block_uses_existing_tokens_only`.
- [ ] The pin fails if either mobile rule is removed or moved out of the ≤640px block (mutation check: temporarily drop the rule → red → restore).
- [ ] `frontend/` and `app/` are byte-unchanged by this task (the diff is `tests/unit/test_settings_frontend_view.py` only).
- [ ] ruff + pyright clean.
@@ -0,0 +1,38 @@
# Task 03 — E2E: the mobile Value-column width contract + regression + gates + commit
**Phase:** `176_mobile_settings_value_width` · **Source:** `TODO.md:3` — "On mobile, the settnigs values are impossibly small. This is a horizontally scrollable table so we can have slightly wider input fields."
**Story:** n/a
## Objective
Prove the width contract in a real browser (AGENTS rule 4 — the phase's dedicated Playwright suite, run **in isolation**), prove the desktop is unchanged, run the regression set + full gates, and commit the phase atomically.
## Work
1. `tests/e2e/test_settings_mobile_value_width.py` — the phase suite, structured on `tests/e2e/test_admin_settings_layout.py` (phase 171) VERBATIM where the contract allows (module docstring citing `TODO.md` L3 + the test→contract mapping; the `_boot_app` / `app_server` fixture with `env["BOR_SETTINGS_DB"] = "1"`; the per-test `_clean_settings_state` truncation of `app_settings` / `ui_settings` + the `e2e-`-labeled token cleanup — never a TRUNCATE on `api_tokens`; the module app env block; `login` from `e2e.auth_helpers`; `expect` from `playwright.sync_api`). Tests (each at the stated viewport via `page.set_viewport_size({"width": …, "height": 900})`):
- **`test_mobile_value_inputs_are_usable_width`** — at **375 px**: the admin arrives at the settings view (the phase-168 gate flow); for a representative spread of rows (at least one `type="search"`/text input, one `select`, one textarea — the catalogue is 75 rows; pick three distinct control types), the `.settings-control` bounding-box width is **≥ 192 px** (12 rem — owner-locked A3, the deliberate margin under the 13 rem rule) AND ≥ 200 px wide is NOT asserted (the 12 rem pin is the contract; 13 rem ≈ 208 px is the rule). Also the `.settings-table`'s own width (`scrollWidth`) **exceeds** the `.table-wrap` card's `clientWidth` (the horizontal scroll is active on the phone — the TODO's own premise) while `document.documentElement.scrollWidth ≤` the 375 px viewport (the PAGE does not overflow — the phase-171 pin, the scroll is the card's).
- **`test_mobile_table_still_scrolls_and_page_does_not_overflow`** — at **375 px**: `.table-wrap` computed `overflow` is `auto`; the card's `scrollWidth > clientWidth`; the page-level `documentElement.scrollWidth` ≤ 375 px + 1 px rounding; the sticky `thead` contract survives (the first row's group band/`thead` is reachable — scroll the card and the `thead` `th`s stay pinned, computed `position: sticky`).
- **`test_desktop_value_width_is_unchanged`** — at **1440 px**: a fixed probe row (the same first `type="search"` control) has a width STRICTLY GREATER than at 375 px is the wrong shape — the contract is the media query is INERT: the 1440 px `.settings-control` width equals the pre-phase natural layout; pinned as: the mobile rules do not apply (computed `min-width` of `.settings-value-cell` at 1440 px is the UA `0px`/auto — i.e. `getComputedStyle` returns `0px` or `auto`, never `208px`/`208.0px`) and the `.settings-table` `scrollWidth` ≤ the card's `clientWidth` (no forced scroll at desktop — the 720 px floor binds only inside the media query).
- **`test_keyboard_order_and_no_external_requests`** — at **375 px**: the tab order reaches `#settings-filter` → the first row's Value control → `#settings-save` → `#settings-clear-all` (the phase-168/171 contract under the new column floor); the whole test's `page.on("request")` log contains ZERO external (non-127.0.0.1) requests (rule 6).
- **The DB-isolation docstring + module teardown** (the phase-171 idiom): the suite's only write is none (no save is issued — the width contract needs no dirty row); `app_settings` pinned empty at teardown.
2. Run the **regression set** in isolation (each its own run, per the phase-171 completion criteria):
- `uv run pytest tests/e2e/test_settings_mobile_value_width.py -v --no-cov` (THIS phase — must be green in isolation),
- `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` (phase 171 — the rhythm/band/cursor + the 375/1440 no-page-overflow pins),
- `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` (phase 168 — the panel contract).
3. Run the **full gates**: `uv run pytest` (green), `uv run pytest --cov=app --cov-report=term-missing` (**>90%** on `app/` — this phase changes no `app/` code: the gate stays green, not dropped), `uv run ruff check . && uv run pyright` (clean).
4. **Commit** the phase atomically (AGENTS rule 8):
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "fix(settings): mobile settings table — a usable Value-column width (the card's horizontal scroll absorbs it)"
```
then move the phase dir: `mv .agents/phases/todo/176_mobile_settings_value_width .agents/phases/complete/176_mobile_settings_value_width` (the dir move is part of the same phase completion — commit it with `git add .agents/ && git commit --no-gpg-sign -m "chore(agent): complete phase 176 mobile settings value width"` ONLY if the repo's convention is a separate commit; the phase-171 precedent shows a single commit — follow it: the dir move is folded into the atomic commit by `git add .agents/phases/` AFTER the `mv`).
- ASSUMPTION: A3 (the 12 rem / 192 px E2E pin threshold, the deliberate margin under the 13 rem rule) — owner-locked 2026-10-12. The probe-row choice (one text input, one select, one textarea) is executor mechanics within that lock: the three control types must exist among the 75 rows (the phase-168 catalogue guarantees text / number / select / textarea / password controls).
## Testing & Quality
- E2E (AGENTS rule 4): `tests/e2e/test_settings_mobile_value_width.py` green **in isolation**; phase 171 + phase 168 suites green in isolation (regression).
- Coverage: **>90%** on `app/` (unchanged code).
- Lint/types: `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_settings_mobile_value_width.py -v --no-cov` green in isolation (the 375 px input ≥ 192 px, the card scrolls, the page does not; 1440 px inert; keyboard + zero external requests).
- [ ] `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` and `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` green in isolation.
- [ ] `uv run pytest` green; `app/` coverage >90%; ruff + pyright clean.
- [ ] One atomic `--no-gpg-sign` Conventional Commit (the phase-171 single-commit precedent, dir move folded in); the phase dir lives in `.agents/phases/complete/176_mobile_settings_value_width/`.
@@ -0,0 +1,42 @@
# Phase 177 — The favicon's colors follow the effective theme palette
**Source:** `TODO.md` L3 — "We need the favicon to change colors to match the theme."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
**Context:** `frontend/assets/favicon.svg` (the current static art — a hexagon + center dot + 4 spokes whose three hardcoded colors map 1:1 onto the built-in palette: `#1a0f0f` = `surface`, `#f43f5e` = `brand`, `#fca5a5` = `brand_ink`), the five templates' L8 `<link rel="icon" type="image/svg+xml" href="/assets/favicon.svg">` (`frontend/index.html`, `frontend/document.html`, `frontend/login.html`, `frontend/shared.html`, `frontend/doc-edit.html`), `app/core/theming.py` (`BUILTIN_COLORS`, `COLOR_FIELDS`, `effective_settings()` — the DB-over-env/built-in resolver with NO process cache, `theme_style_tag()` + `inject_theme()` — the pre-paint tag pattern, `theme_csp_hash()`), `app/core/caching.py` (`CachingMiddleware`'s per-response short-lived-session theme read with the never-break DB-blip fallback; `/favicon.svg` is neither in `HTML_PAGES` nor under `/assets/` → passes through untouched), `app/main.py` `create_app()` (router includes end ~L199; the static catch-all mount at L252 — routes-first), `app/core/security_headers.py` (CSP `default-src 'self'; … img-src 'self' data:` — the phase-123 carve-out that makes a `data:` favicon URI legal; CSP3 governs `link[rel=icon]` by `img-src`), `frontend/assets/theme.js` (the admin Theme tab — `previewColor()` live preview, the `input` bindings, `saveTheme()`/`resetTheme()` + the `bor:view-refresh` re-show handler), `tests/unit/test_theming.py` (the static pin set incl. `test_builtin_colors_match_styles_css_root`), `tests/unit/test_theme_frontend.py` (the JS static-pin pattern), `tests/e2e/test_theme_save_and_coverage.py` + `test_theme_semantic_completion.py` (the admin theme-tab E2E pattern: `e2e/auth_helpers.login(page, app_url, next="/theme.html")`, the `#theme-*` inputs), AGENTS rules 4/6/8/9/10, PLAN §7.4 (the never-stale feedback contract).
## Objective
The browser-tab favicon must carry the EFFECTIVE theme palette (built-in or the owner's saved one), painted on first load like everything else — no red hexagon on a graphite theme. The favicon is served by the app from the same resolver that paints the page (`effective_settings`), and the admin Theme tab's live (unsaved) picks repaint it too (owner-confirmed A4).
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the pure renderer (task 01, A1).** `theming.favicon_svg(colors: dict[str, str]) -> str` — the SAME artwork (viewBox `0 0 64 64`, the hexagon path, the r=6.5 center circle, the 4 spokes, all stroke widths/caps unchanged) with only the three color roles swapped: hexagon `fill` → `surface`, hexagon `stroke` + circle `fill` → `brand`, spokes `stroke` → `brand_ink`. The input is the full 20-value effective dict (mirroring `theme_style_tag`'s self-contained signature; the function reads its three keys). **Byte-identical contract:** for the built-in palette the output is EXACTLY the current static file's `<svg>` element (lowercase hex, same attribute order) — pinned by a golden constant in `tests/unit/test_theming.py`; the static file's comment header is file packaging, not art, and is not part of the contract.
- **D2 — serving (task 02, A2/A3).** `GET /favicon.svg` (+ HEAD for static-file parity) — a no-prefix route in `create_app()` after the router includes and BEFORE the static catch-all mount (routes-first, the phase-51 shared-page pattern). Sync handler: one short-lived `SessionLocal()` session (the `caching.py` house pattern — NO process cache, so a runtime theme change is visible on the next request), `effective_settings()` → `favicon_svg()`; ANY exception → `logger.exception` + `favicon_svg(BUILTIN_COLORS)` (200, never 500 — the theme-tag's never-break contract, `caching.py`'s DB-blip pattern). Response: `media_type="image/svg+xml"`, `Cache-Control: no-cache` (the pages are no-cache; the palette changes at runtime, so the browser must revalidate — the route is NOT under `/assets/`, so no version token; freshness is revalidation, not busting). Deliberately OUTSIDE `/api/*`: no per-response byte-budget line (AGENTS rule 10 is `/api/*`-only), no auth dependency (the favicon is chrome, like the login page). `CachingMiddleware` passes it through untouched (neither an HTML page nor an asset); `SecurityHeadersMiddleware` still adds the CSP — no change needed (`img-src 'self' data:` covers the D4 preview).
- **D3 — the templates + the static file's deletion (task 03, A2).** All five L8 links → `href="/favicon.svg"`; `frontend/assets/favicon.svg` is **deleted** — the route is the single source (the built-in bytes stay pinned by D1's golden constant, so nothing is lost). The new href is not under `/assets/`, so `CachingMiddleware`'s `?v=` rewrite regex can never touch it (a pin keeps this explicit — the link stays bare).
- **D4 — the Theme tab live preview (task 04, A4 — owner-confirmed 2026-10-13).** `theme.js` gains a pure `themeFaviconDataUri(surface, brand, brandInk)` — the CLIENT-SIDE MIRROR of `theming.favicon_svg` (same artwork, same roles; a unit pin keeps the two from diverging — the `PAIRS`-mirror house pattern). The existing per-color `input` binding ALSO updates `link[rel="icon"].href` to the `data:` URI whenever one of the three driving fields (`surface`/`brand`/`brand_ink`) changes (rebuilt from the three inputs' current values; the other 14 picks never touch it). **Restore to `"/favicon.svg"`** on: Save success (after `applyServedTheme`), Reset success, the Reset-failure `clearPreview()` path, and the `bor:view-refresh` re-show — after every restore the SETTLED palette (re-read per request, no process cache) governs. CSP note (commented in the code): `data:` is legal for the favicon via the phase-123 `img-src 'self' data:` carve-out (CSP3: `link[rel=icon]` is governed by `img-src`; the carve-out is image-scoped — `data:` never becomes a script/style/fetch source).
- **NOT touched:** `CachingMiddleware`, the `#bor-theme` tag injection + CSP hash, the 8 semantic state colors (they are NOT in the artwork — a favicon has no state meaning), `app/` outside `theming.py`/`main.py`, `brand.js`, the other templates/assets, the `theme.js` save/reset/contrast machinery beyond the additions above.
## Dependencies
- `176_mobile_settings_value_width` (todo) — the immediately preceding phase (Protocol B ordering; no functional dependency).
- `92_theme_save_and_coverage` (complete) — the served-theme sync + the `#theme-*` input contract the D4 additions bind to.
- `91_admin_theme_tab` (complete) — the Theme tab's live-preview mechanism (`previewColor`) and the `bor:view-refresh` re-show hook.
## Tasks
1. `01_favicon_svg_renderer.md` — `theming.favicon_svg()`: the pure palette→artwork renderer, byte-identical for the built-in palette (golden pin) — A1.
2. `02_favicon_route.md` — `GET /favicon.svg` (+ HEAD) in `create_app()`: effective palette, built-in fallback on DB failure (200, never 500), `image/svg+xml` + `no-cache` — A2/A3.
3. `03_template_link_swap.md` — the five template links → `/favicon.svg`; `frontend/assets/favicon.svg` deleted (single source).
4. `04_theme_tab_live_preview.md` — the Theme tab's live picks repaint the favicon (the CSP-legal `data:` mirror, A4); Save/Reset/re-show restore the route URL; the dedicated E2E suite + the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_theming.py` (task 01): the golden byte-identical pin, the themed-role pin (the three hexes in their exact roles), the geometry-invariance pin (only the 4 color attributes differ); `tests/unit/test_theme_frontend.py` (task 04): the `data:` mechanism pin (the three driving field names, the exact `/favicon.svg` restore, the restore wiring).
- **Integration:** `tests/integration/test_favicon_route.py` (tasks 02–03): the three route states (default = golden bytes, themed = effective colors, DB failure = built-in fallback 200), the CSP header presence, the five template link pins (bare `/favicon.svg`, no `?v=`, no `/assets/favicon.svg` anywhere in the served HTML), the deleted file's 404.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_themed_favicon.py`, run **in isolation** — the full contract in a real browser (default/themed route bytes, the pre-paint tag non-regression, the live-preview `data:` repaint, the restore contract, zero external requests — rule 6).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`); **lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `GET /favicon.svg` on a default (no ui_settings row) deployment is 200 `image/svg+xml` with `Cache-Control: no-cache` and a body byte-identical to the current static art (the golden pin, the integration test).
- [ ] With a saved non-default palette (distinct `surface`/`brand`/`brand_ink`), the route's SVG carries those three hexes in their exact roles, and the served pages STILL inject the `#bor-theme` tag (the phase-91 pin stays green — no regression to the tag/CSP contract).
- [ ] A DB failure at request time serves the built-in art with 200 (never 500).
- [ ] All five templates link `href="/favicon.svg"` (bare — no `?v=`), `/assets/favicon.svg` is 404 (the file is deleted), and the favicon actually renders in the browser (the E2E).
- [ ] In the admin Theme tab, a live (unsaved) pick of `surface`/`brand`/`brand_ink` repaints the favicon via the `data:` URI; Save / Reset / re-show restore `href="/favicon.svg"` with the settled palette governing (the E2E).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov` green **in isolation**; `test_theme_save_and_coverage.py` + `test_theme_semantic_completion.py` (phases 92/93) green in isolation (no theme-tab regression).
- [ ] ONE atomic Conventional Commits commit (AGENTS rule 8, `--no-gpg-sign`) covering the phase + this directory + the cleared TODO.md.
@@ -0,0 +1,26 @@
# Task 01 — The pure favicon renderer in `theming.py`
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
## Objective
A pure, unit-testable renderer that produces the favicon artwork from the effective palette — the same art as today's static `frontend/assets/favicon.svg`, byte-identical for the built-in palette.
## Work
1. `app/core/theming.py` — add `favicon_svg(colors: dict[str, str]) -> str` next to `theme_style_tag()`. The output is EXACTLY the `<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">…</svg>` element (no comment header — the static file's comment is file packaging, not art). The artwork is unchanged: the hexagon path (`d="M32 4 55 18v28L32 60 9 46V18Z"`, `stroke-width="4"`, `stroke-linejoin="round"`), the center `<circle cx="32" cy="32" r="6.5">`, the spokes path (`d="M32 25.5V16M32 48v-9.5M25.5 32H16M48 32h-9.5"`, `stroke-width="3"`, `stroke-linecap="round"`). Only the three color roles swap, per the A1 mapping: hexagon `fill` ← `colors["surface"]`, hexagon `stroke` ← `colors["brand"]`, circle `fill` ← `colors["brand"]`, spokes `stroke` ← `colors["brand_ink"]`. The input is the full 20-value effective dict (mirroring `theme_style_tag`'s self-contained signature — the function reads its three keys); pure (dict in → str out, no DB, no app, no defensive branches — the `effective_settings`/`BUILTIN_COLORS` contract guarantees the values). Docstring: the A1 role table, the byte-identical contract, and the "no state colors" note (the 8 semantic state colors are not in the artwork — a favicon has no state meaning).
- ASSUMPTION: A1 (owner-confirmed) — the color mapping is hexagon fill→`surface`, hexagon stroke + center dot→`brand`, spokes→`brand_ink`; only these three identity colors drive the favicon.
2. `tests/unit/test_theming.py` — extend the pin set:
- `test_favicon_svg_builtins_is_byte_identical_to_the_static_art` — `favicon_svg(BUILTIN_COLORS)` equals a GOLDEN constant holding the current static file's `<svg>` element bytes. During this task's execution, verify the golden constant against the current `frontend/assets/favicon.svg` file (a one-time sanity check — the file is deleted in task 03, so the golden constant, not the file, is the durable contract).
- `test_favicon_svg_themed_carries_the_three_roles` — a palette with distinct `surface`/`brand`/`brand_ink` (e.g. `#102030` / `#4f46e5` / `#a5b4fc`) plus the other 14 built-ins: the three hexes appear in their exact attribute roles (assert by parsing the attributes, not by substring), and none of the other 14 hexes appears.
- `test_favicon_svg_geometry_is_invariant` — with a themed palette, everything except the four color attributes is byte-equal to the built-in output (viewBox, both path `d`s, circle `cx/cy/r`, stroke widths, `linejoin`/`linecap`, attribute order).
3. One-time sanity: diff the golden constant against `frontend/assets/favicon.svg`'s `<svg>` element and log the result (evidence for the hand-off; not a persistent test).
## Testing & Quality
- Unit: the three pins above (`tests/unit/test_theming.py`), run with `uv run pytest tests/unit/test_theming.py -v`.
- Coverage: **>90%** on this task's new code (a pure function — the pins drive it to ~100%).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_theming.py -v` green (new + existing pins).
- [ ] The one-time golden-vs-file sanity check passes (logged).
- [ ] Full test suite green; `app/` coverage >90%; ruff + pyright clean.
- [ ] No behavior change in completed work (a new function with no existing call site).
@@ -0,0 +1,32 @@
# Task 02 — Serve the themed favicon: `GET /favicon.svg`
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
## Objective
The favicon over the wire: a route that renders the artwork from the EFFECTIVE palette on every request (no process cache — a runtime theme change is visible on the next request), with the never-break DB-blip fallback.
## Work
1. `app/main.py` — in `create_app()`, after the router includes (ending at the `chats_shared_page_router` line ~L200) and BEFORE the static catch-all mount (`app.mount("/", …)` at L252, routes-first): register `app.api_route("/favicon.svg", methods=["GET", "HEAD"], include_in_schema=False)` (HEAD for static-file parity, the phase-76 `_shell_routes` parity idiom). The handler is a sync `def` (the house pattern for short-lived-session reads):
- open a short-lived `SessionLocal()` session (imported from `app.db`, the `caching.py` middleware's exact pattern — NO process cache, `try/finally` close);
- `effective = theming.effective_settings(db)` → `svg = theming.favicon_svg(effective)`;
- ANY exception (DB blip, pre-migration boot): `logger.exception("favicon: theme read failed — serving the built-in favicon")` and `svg = theming.favicon_svg(theming.BUILTIN_COLORS)` — **200, never 500** (the theme-tag's never-break contract, `caching.py`'s DB-blip pattern);
- return `Response(svg, media_type="image/svg+xml", headers={"Cache-Control": "no-cache"})`.
- Commented rationale in the code (house style): (a) the route is deliberately OUTSIDE `/api/*` → no per-response byte-budget line (AGENTS rule 10 is `/api/*`-only) and no auth dependency (the favicon is chrome — public like the login page); (b) `CachingMiddleware` passes it through untouched (neither in `HTML_PAGES` nor under `/assets/`) → the route's own `no-cache` is the freshness contract (the palette changes at runtime; the page is no-cache too, so the browser revalidates together with it); (c) `SecurityHeadersMiddleware` still adds the CSP — no change needed (`img-src 'self' data:` covers task 04's `data:` preview).
- ASSUMPTION: A2 (owner-confirmed) — serving shape: a new dynamic route `GET /favicon.svg` replaces the static file (task 03 deletes it).
- ASSUMPTION: A3 (owner-confirmed) — fault tolerance: a DB failure / pre-migration boot serves the built-in palette with 200 (never 500).
2. `tests/integration/test_favicon_route.py` — NEW, via the house `TestClient` fixture (follow the `tests/integration/test_ui_settings.py` app/db fixture pattern):
- **default state** (no `ui_settings` row): `GET /favicon.svg` → 200, `content-type: image/svg+xml`, `cache-control: no-cache`, body byte-identical to the golden art constant (import it from `tests/unit/test_theming.py` — one source), and the response carries the CSP header (the `SecurityHeadersMiddleware` pass-through).
- **themed state**: a `ui_settings` row (id 1) with distinct `surface`/`brand`/`brand_ink` (e.g. `#102030` / `#4f46e5` / `#a5b4fc`): the body carries the three hexes in their roles (attribute-parsed, same helper idiom as the unit pins) and none of the other 14; a second request after updating the row reflects the new palette immediately (no process cache — the runtime-change contract).
- **DB-failure state**: patch `effective_settings` (or `SessionLocal`) to raise → 200 + the built-in golden bytes (never 500).
- **HEAD parity**: `HEAD /favicon.svg` → 200, the same headers, empty body.
## Testing & Quality
- Integration: the four states above (`tests/integration/test_favicon_route.py`), run with `uv run pytest tests/integration/test_favicon_route.py -v`.
- Coverage: **>90%** on this task's new/modified code (the handler's success + fallback branches both driven).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_favicon_route.py -v` green (all four states).
- [ ] The route wins over the static mount (a file named `favicon.svg` in the static dir could not shadow it — the routes-first position holds).
- [ ] Full test suite green; `app/` coverage >90%; ruff + pyright clean.
- [ ] No behavior change in completed work (a new route only; `/assets/favicon.svg` still serves until task 03 deletes it).
@@ -0,0 +1,27 @@
# Task 03 — Point all five templates at the route; delete the static file
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
## Objective
Every page asks for the themed favicon from the route, and the static file stops existing — the route is the single source (the built-in bytes stay pinned by task 01's golden constant, so nothing is lost).
## Work
1. `frontend/index.html`, `frontend/document.html`, `frontend/login.html`, `frontend/shared.html`, `frontend/doc-edit.html` — L8 in each file: `<link rel="icon" type="image/svg+xml" href="/assets/favicon.svg">` → `<link rel="icon" type="image/svg+xml" href="/favicon.svg">`. Nothing else in any template changes.
2. Delete `frontend/assets/favicon.svg` (A2, owner-confirmed — the single-source decision; the one-time golden-vs-file sanity was done in task 01, so the deletion loses no bytes: the golden constant in `tests/unit/test_theming.py` is the durable contract).
3. `tests/integration/test_favicon_route.py` — extend with the template pins:
- each of the five served page bodies (`/`, `/document.html`, `/login.html`, `/shared.html`, `/doc-edit.html` via `TestClient`) contains `<link rel="icon" type="image/svg+xml" href="/favicon.svg">` and does NOT contain `/assets/favicon.svg`;
- the favicon link in the served HTML stays BARE — no `?v=` (the href is not under `/assets/`, so `CachingMiddleware`'s `_ASSET_REF_RE` can never touch it; the pin keeps that contract explicit against a future regex drift);
- `GET /assets/favicon.svg` → 404 (the file is gone);
- a tree scan of `frontend/` (the test walks the directory, the house idiom) asserts zero remaining references to `/assets/favicon.svg`.
## Testing & Quality
- Integration: the template pins above, run with `uv run pytest tests/integration/test_favicon_route.py -v`.
- Coverage: **>90%** on `app/` (no new `app/` code in this task — the gate must stay green, not drop).
## Completion Criteria
- [ ] All five templates link `href="/favicon.svg"`; the tree scan finds zero `/assets/favicon.svg` references.
- [ ] `GET /assets/favicon.svg` is 404; `GET /favicon.svg` still serves the golden bytes (the route is the single source now).
- [ ] In a running app, a browser tab loads the favicon from `/favicon.svg` (final browser proof in task 04's E2E).
- [ ] Full test suite green; `app/` coverage >90%; ruff + pyright clean.
- [ ] No behavior change in completed work (the favicon art is byte-identical on a default deployment — the golden pin holds).
@@ -0,0 +1,40 @@
# Task 04 — Theme-tab live preview + the dedicated E2E suite + the commit
**Phase:** `177_themed_favicon` · **Source:** `TODO.md:3` — "We need the favicon to change colors to match the theme."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-13)
## Objective
The admin Theme tab's live (unsaved) picks repaint the favicon too (owner-confirmed A4), the settled palette always wins after Save/Reset/re-show — and the phase's dedicated Playwright suite proves the whole contract in a real browser.
## Work
1. `frontend/assets/theme.js` —
- a pure helper `themeFaviconDataUri(surface, brand, brandInk)` → `"data:image/svg+xml," + encodeURIComponent(svg)`, where `svg` is the CLIENT-SIDE MIRROR of `theming.favicon_svg` (the same artwork, the same A1 roles: hexagon fill ← `surface`, hexagon stroke + circle fill ← `brand`, spokes stroke ← `brand_ink`). Guard with the existing `isHex` idiom: a non-hex input returns `null` (the link is left untouched — the color inputs are always valid hex, the guard is belt-and-braces; the inputs are the only source of the values, so there is no XSS path — the house "never build HTML" rule binds the DOM, the URI is built from constants + validated hexes).
- in the existing per-color `input` bindings (the live-preview loop): when the field is one of the three driving fields (`surface`, `brand`, `brand_ink`), ALSO set the `link[rel="icon"]` element's `href` to `themeFaviconDataUri(<the three inputs' current values>)` (rebuilt, not patched — a `surface` pick repaints with the current `brand`/`brand_ink` picks). The other 14 picks never touch the link.
- **restore to `"/favicon.svg"`** in the four settle points: `saveTheme()` success (after `applyServedTheme(settings)`), `resetTheme()` success (after the reconcile), the `resetTheme()` failure path (where `clearPreview()` runs — the picks are stale), and the `bor:view-refresh` re-show handler (after its `applyServedTheme`). After every restore the SETTLED palette governs — the route re-reads the DB per request (no process cache), so the restored URL is always fresh; the page is `no-cache`, so the browser revalidates the favicon with it.
- CSP comment (house style, per D4): `data:` is legal for the favicon via the phase-123 `img-src 'self' data:` carve-out (CSP3 governs `link[rel=icon]` by `img-src`; the carve-out is image-scoped — `data:` never becomes a script/style/fetch source).
- file-header docstring: extend with the favicon-preview paragraph (the mechanism, the four restore points, the CSP note — the theme.js header documents every behavior).
- ASSUMPTION: A4 (owner-confirmed 2026-10-13) — the live (unsaved) picks repaint the favicon via the CSP-legal `data:` URI; the settled palette always wins after Save/Reset/re-show. (The simpler alternative — saved palette only, no `data:` — was considered and declined.)
2. `tests/unit/test_theme_frontend.py` — extend the static pin set (the house JS-pin idiom — the pins parse `theme.js`'s source):
- the `data:image/svg+xml` URI mechanism is present (the literal prefix, `encodeURIComponent`);
- the favicon helper reads EXACTLY the three field names `surface`/`brand`/`brand_ink` (the driving-set pin — a fourth field or a missing one fails the pin);
- the restore string is EXACTLY `"/favicon.svg"` and is wired to all four settle points (save success, reset success, reset-failure `clearPreview` path, `bor:view-refresh`);
- the mirror's artwork constants match the server's (the viewBox + both path `d` strings + circle `r` — the divergence guard, the `PAIRS`-mirror house pattern).
3. `tests/e2e/test_themed_favicon.py` — NEW, the phase's dedicated Playwright suite (AGENTS rule 4), run in isolation, following the `test_theme_save_and_coverage.py` fixture idiom (`app_server`/`app_url`/`db_ready`, `e2e/auth_helpers.login(page, app_url, next="/theme.html")`, the `_clean` ui_settings reset):
- **default state:** `GET /favicon.svg` → 200 + the golden art bytes (fetched in the test); the five pages' HTML reference `/favicon.svg`; `/assets/favicon.svg` → 404;
- **themed state:** admin login → `PUT /api/ui-settings` with a distinct palette (e.g. `surface #102030`, `brand #4f46e5`, `brand_ink #a5b4fc`): the route's SVG carries the three hexes in their roles; a fresh page load STILL injects `#bor-theme` with the expected content (the phase-91 non-regression pin — the tag/CSP contract is untouched);
- **live preview:** admin opens `/theme.html` (the `#view-theme` view), sets the `#theme-surface` / `#theme-brand` / `#theme-brand-ink` inputs (the `input` event, the house `_fill_theme_form` idiom) → `link[rel="icon"]`'s href starts with `data:image/svg+xml` and, decoded, carries the three picked hexes in their roles; a non-driving pick (e.g. `#theme-bg`) does not change the href;
- **restore contract:** click **Save theme** → the href is EXACTLY `/favicon.svg` and the route now serves the saved palette's bytes; click **Reset to defaults** → the href is `/favicon.svg` and the route serves the built-in golden bytes; re-showing the tab (the `bor:view-refresh` path) restores the href too;
- **zero external requests** across the whole suite (AGENTS rule 6 — the house E2E assertion idiom: a `page.on("request")` collector asserting every URL is the local origin / a `data:` URI).
4. The atomic commit (AGENTS rule 8): ONE Conventional Commits commit, e.g. `feat(theming): serve the favicon from the effective theme palette`, `--no-gpg-sign` — covering the `app/` + `frontend/` + `tests/` changes, this phase directory (`.agents/` is tracked), and the cleared `TODO.md`.
## Testing & Quality
- Unit: the JS static pins (`tests/unit/test_theme_frontend.py`), run with `uv run pytest tests/unit/test_theme_frontend.py -v`.
- E2E: `tests/e2e/test_themed_favicon.py` run **in isolation** — `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov`.
- Coverage: **>90%** on `app/` (the JS is gated by the pins + the browser E2E — the house idiom); ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_theme_frontend.py -v` green (new pins).
- [ ] `uv run pytest tests/e2e/test_themed_favicon.py -v --no-cov` green **in isolation** (all six contract blocks).
- [ ] `test_theme_save_and_coverage.py` (phase 92) + `test_theme_semantic_completion.py` (phase 93) green in isolation (no Theme-tab regression).
- [ ] `uv run pytest` green, `app/` coverage >90% (`uv run pytest --cov=app --cov-report=term-missing`), `uv run ruff check . && uv run pyright` clean.
- [ ] The atomic commit is landed (Conventional Commits, `--no-gpg-sign`, `.agents/` + the cleared `TODO.md` included).
@@ -0,0 +1,49 @@
# Phase 178 — Split thinking levels: a chat level and a summary level, separately tunable
**Source:** `TODO.md` L3 — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-11)
**Context:** `app/config.py` L108–115 — `llm_thinking_level: str = "low"` (the phase-172 shared knob, directly after `llm_summary_model` L103–107, before `stream_thinking` L116), `app/core/settings_catalog.py` L154–155 — the single `llm_thinking_level` `SettingMeta` row (label "Thinking level", group `LLM (aipi)`; the completeness pin in `tests/unit/test_settings_catalog.py` — `set(CATALOG) == set(Settings.model_fields)` — makes every `Settings` field a build failure until it has a `SettingMeta`), `.env.example` L61 — `BOR_LLM_THINKING_LEVEL=low`, `app/rag/llm.py` — `_thinking_extra_body(settings)` (L49–57, reads `settings.llm_thinking_level`), `LLMClient.chat()` (L470, the one-shot `chat.completions.create` L430–452 — `extra = _thinking_extra_body(self.settings)` L432, `**extra_kwargs` L444; callers: `app/rag/summarizer.py` L242 + L359 document summaries, `app/rag/folder_summaries.py` L347, `app/rag/overview.py` L254 KB overview, `check_models` L805 the pre-sync probe), `LLMClient.chat_stream()` (L511, the kwargs dict L615–622 + `kwargs["extra_body"] = extra` L625–628), `app/api/chat.py` — the per-turn log line (L1272–1291, `thinking=%s` L1275, `settings.llm_thinking_level.strip() or "-"` L1289), the module docstring L14–19, `frontend/assets/settings.js` (the phase-168 generated panel — **no code change here**: `str` catalogue entries render as free-form text rows automatically), `tests/unit/test_config.py` L108–133 (default / env override / empty-stays-empty), `tests/unit/test_settings_catalog.py` L99–111, `tests/unit/test_llm_client.py` L1330–1440 (the mocked-client `extra_body` family), `tests/unit/test_chat_gate.py` L1550–1630 (the per-turn log-line family), `tests/integration/test_admin_settings_api.py` L521–607 + `test_admin_settings_live.py` L171–204, `tests/e2e/test_thinking_level.py` (the phase-172 Playwright suite — the pattern to extend).
## Objective
The operator tunes the chat turns' thinking level and the one-shot summaries' thinking level **independently**: `llm_chat_thinking_level` (renamed from the phase-172 `llm_thinking_level`, default `low` — chat stays at low, TODO L3) rides `chat_stream()`; `llm_summary_thinking_level` (default empty = **off** — the TODO's "none", the phase-172 locked A2 off-switch: the parameter is ABSENT from the request) rides the one-shot `chat()` (document summaries, folder summaries, KB overview, the `check_models` probe). Both are free-form text, both live in `.env` (`BOR_LLM_CHAT_THINKING_LEVEL` / `BOR_LLM_SUMMARY_THINKING_LEVEL`) and the admin Settings panel, both live-apply without a restart.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the two fields (task 01).** `app/config.py` LLM section: `llm_thinking_level` is **renamed** to `llm_chat_thinking_level` (default stays `"low"`) and `llm_summary_thinking_level: str = ""` is added next to it (both before `stream_thinking`). Plain `str`, no allowlist, no validator (phase-172 A4, kept). House docstrings name the env var, the default, the wire mapping and the off rule. `.env.example` L61: `BOR_LLM_THINKING_LEVEL=low` is **replaced** by `BOR_LLM_CHAT_THINKING_LEVEL=low` + `BOR_LLM_SUMMARY_THINKING_LEVEL=` (empty value, one-line comment each).
- **D2 — the consumers rename with the field (task 01, behavior-identical).** Every consumer of the old field is mechanically renamed to `llm_chat_thinking_level` in task 01 — `app/rag/llm.py` (module docstring, `_thinking_extra_body`), `app/api/chat.py` (module docstring, per-turn log line L1289) — and every test pin is renamed in place (unit `test_config.py` / `test_settings_catalog.py` / `test_llm_client.py` / `test_chat_gate.py`, integration `test_admin_settings_api.py` / `test_admin_settings_live.py`, e2e `test_thinking_level.py`). After task 01 the suite is green and behavior is **byte-identical** to pre-phase: both `chat()` and `chat_stream()` ride the (renamed) chat level, default `low`; the new summary field exists but is inert until task 03.
- **D3 — the wire split (task 03).** `_thinking_extra_body` takes the **level string** (pure function, no `Settings` read): `chat_stream()` passes `self.settings.llm_chat_thinking_level`; `chat()` passes `self.settings.llm_summary_thinking_level`. Trimmed non-empty → `{"reasoning_effort": <value>}`; empty/whitespace → `None` (the parameter is ABSENT — never a literal `"none"` or `"off"`, phase-172 A2). The `check_models` probe (L805) goes through `chat()` → summary level → off by default; no special-casing (it is a reachability ping). Embeddings path untouched.
- **D4 — the admin rows (task 02).** The single `SettingMeta("llm_thinking_level", …, "Thinking level")` becomes TWO rows: `llm_chat_thinking_level` → "Chat thinking level", `llm_summary_thinking_level` → "Summary thinking level" (both group `LLM (aipi)`, placed where the old row sat — after `llm_summary_model`, before `stream_thinking`). The panel (`settings.js`), the `GET` catalogue, the `PUT` / `DELETE` routes and the DB-over-env overlay are generic over the catalogue — the two free-form text rows, their save/reset and live-apply fall out with zero extra code. The completeness pin is the enforcement.
- **D5 — the per-turn line (task 04).** The per-turn log line's `thinking=<value>` (or `-` when empty) reads the **chat** level — the level the turn actually ran at; the summary level is a one-shot concern and gains no log field (the summary log lines have no knob field today). No `query_log` schema change (phase-172 A6, kept).
- **NOT touched:** `frontend/assets/settings.js`, `frontend/assets/styles.css`, the `app_settings` table shape, `app/core/settings_overlay.py` (generic), the embeddings path, `stream_thinking` (the phase-17 kill-switch for `thinking` SSE events), every other setting.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; this phase refines the committed phase-172 thinking-level machinery)
## Tasks
1. `01_settings_catalog_env.md` — the rename + the new field in `app/config.py` + the two `CATALOG` rows + `.env.example` + the mechanical consumer/test rename (suite green, behavior byte-identical).
2. `02_admin_row_contract.md` — the integration set / reset / live-apply contract for BOTH keys (stored raw, live without restart, empty storable as the summary off-switch).
3. `03_llm_extra_body_split.md` — `chat_stream()` → chat level, `chat()` → summary level; the mocked-client unit tests per method.
4. `04_per_turn_log_field.md` — `thinking=<chat level>` (or `-`) pinned: the line follows the chat knob, never the summary knob.
5. `05_e2e_admin_settings_rows.md` — `tests/e2e/test_split_thinking_levels.py` (the phase's dedicated Playwright suite, AGENTS rule 4), the regression set, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` — chat default `low` / env override / empty-stays-empty; summary default `""` / env override / empty-stays-empty (no coercion, no allowlist — D3/A2). `tests/unit/test_settings_catalog.py` — the completeness pin stays green (both fields have their `SettingMeta`; group/label/flags correct). `tests/unit/test_llm_client.py` — `chat_stream()` sends `extra_body={"reasoning_effort": <chat level>}` when the chat level is trimmed-non-empty and no `extra_body` when it is empty; `chat()` sends it from the **summary** level with the same absent-when-empty rule; an empty summary level (the default) keeps one-shot requests byte-identical to pre-phase-172; the embeddings path is unaffected. `tests/unit/test_chat_gate.py` — the per-turn line shows the chat level (with the summary level deliberately different — it must not leak into the line).
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — BOTH rows appear in `GET /api/admin/settings` (`str`, correct labels, effective defaults `low` / ``); `PUT` stores the raw string on each, the empty value is storable (the summary off-switch), the overlay resolves live without a restart; `DELETE` resets each to env.
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_split_thinking_levels.py`, run **in isolation** — both rows visible in the `LLM (aipi)` group (labels "Chat thinking level" / "Summary thinking level"), chat default `low`, summary default empty, both free-form text inputs (type, save, survives reload), both reset-to-env restore the defaults, the env-var display pins `BOR_LLM_CHAT_THINKING_LEVEL` / `BOR_LLM_SUMMARY_THINKING_LEVEL`; the `test_admin_settings_panel.py` + `test_admin_settings_layout.py` regression set stays green.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest -k "config and thinking" -v` shows chat default `low` + summary default `""` (env overrides for both).
- [ ] `GET /api/admin/settings` (admin) lists `llm_chat_thinking_level` and `llm_summary_thinking_level` in the `LLM (aipi)` group (`str`, defaults `low` / empty); `PUT` / `DELETE` round-trip each per the overlay contract; the old `llm_thinking_level` key no longer appears.
- [ ] With the levels set, `chat_stream()` carries `reasoning_effort` from the chat level and `chat()` from the summary level (unit-asserted via the mocked client); with a level empty, that method's request kwargs contain no `extra_body` (byte-identical to pre-phase-172 for the default one-shot path).
- [ ] The per-turn log line carries `thinking=<chat level>` (`-` when empty) — never the summary level.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` green **in isolation**; `test_admin_settings_panel.py` + `test_admin_settings_layout.py` (phase 171) green in isolation.
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(llm): split the thinking level into a chat level (default low) and a summary level (default off)`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-11, roadmap confirmation)
- **A1 — rename, not add-on:** `llm_thinking_level` is **renamed** to `llm_chat_thinking_level` (`BOR_LLM_THINKING_LEVEL` → `BOR_LLM_CHAT_THINKING_LEVEL`). Any DB-stored override under the old key becomes inert (it is no longer a catalogue row) and chat falls back to the env default `low` — chat behavior is preserved exactly as the TODO wants; no value migration.
- **A2 — the TODO's "none" = the phase-172 off-switch:** `llm_summary_thinking_level` defaults to empty (`""`) → `reasoning_effort` is ABSENT from one-shot `chat()` requests. Per phase-172's locked A2 ("nothing is sent as `off` or `null` — absence is the off"), no literal `"none"` string is ever sent.
- **A3 — scope:** `chat_stream()` (the chat turn) → chat level; every one-shot `chat()` (document summaries, folder summaries, KB overview, the `check_models` probe) → summary level; the embeddings path is untouched.
- **A4 — no frontend code change:** the panel is generated from the catalogue; the two free-form text rows appear automatically (phase-172 A5 pattern).
- **A5 — the per-turn line** keeps its single `thinking=<chat level>` field (`-` when empty), reads the chat level, and gains no summary field; no `query_log` schema change.
@@ -0,0 +1,69 @@
# Task 01 — The two settings + the two catalogue rows + `.env.example` + the mechanical rename
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
**Story:** n/a
## Objective
The two settings exist in `app/config.py` (chat default `low`, summary default empty = off, both free-form), in the admin catalogue (the completeness pin stays green), and in `.env.example` — and every existing consumer/test of the old single `llm_thinking_level` is renamed so the suite is green and behavior is **byte-identical** to pre-phase (both LLM paths still ride the chat level, default `low`).
## Work
1. `app/config.py` — in the LLM section, where the phase-172 field sits (L108–115, directly after the `llm_summary_model` block, before `stream_thinking` L116): **rename** `llm_thinking_level` → `llm_chat_thinking_level` (default stays `"low"`) and add the summary field next to it. House docstring style (env var named, default, wire mapping, off rule):
```python
#: The chat turn's thinking level (phase 178,
#: ``BOR_LLM_CHAT_THINKING_LEVEL``; the phase-172 knob, renamed) —
#: pi-style reasoning effort, free-form text exactly like
#: ``llm_chat_model`` (no allowlist — the endpoint is the source of
#: truth). Default ``low``. Rides the streaming chat-completions
#: requests to aipi as the top-level ``reasoning_effort`` parameter
#: (``app/rag/llm.py`` ``chat_stream``); the EMPTY value is the off
#: switch — the parameter is not sent at all.
llm_chat_thinking_level: str = "low"
#: The one-shot summary thinking level (phase 178,
#: ``BOR_LLM_SUMMARY_THINKING_LEVEL``) — rides the one-shot
#: ``chat()`` calls (document summaries, folder summaries, KB
#: overview, the ``check_models`` probe) as ``reasoning_effort``.
#: Default EMPTY = the off switch (phase-172 A2): the parameter is
#: not sent at all; the TODO's "none" means absence, never a
#: literal "none" string.
llm_summary_thinking_level: str = ""
```
No validator, no coercion — plain `str` (phase-172 A4, kept). Verify by reading: `llm_summary_model` (L103–107) and `stream_thinking` (L116–120) keep their shape byte-identical.
2. `app/core/settings_catalog.py` — replace the single row (L154–155) with TWO rows, placed exactly where the old row sat (after `llm_summary_model` L151–152, before `stream_thinking` L157):
```python
"llm_chat_thinking_level": SettingMeta(
"llm_chat_thinking_level", "LLM (aipi)", "Chat thinking level"
),
"llm_summary_thinking_level": SettingMeta(
"llm_summary_thinking_level", "LLM (aipi)", "Summary thinking level"
),
```
Nothing else: the panel (`settings.js`), the `GET` catalogue, the `PUT` / `DELETE` routes and the overlay are generic over the catalogue (D4).
3. `.env.example` — in the LLM block (L57–65), replace `BOR_LLM_THINKING_LEVEL=low` (L61) with:
```
BOR_LLM_CHAT_THINKING_LEVEL=low # the chat turn's thinking level (pi-style; free-form, rides reasoning_effort); empty = the parameter is not sent
BOR_LLM_SUMMARY_THINKING_LEVEL= # the one-shot summary thinking level (document/folder summaries, KB overview); empty (default) = the parameter is not sent
```
directly under `BOR_LLM_CHAT_MODEL=turbo` (L60), house comment style.
4. `app/rag/llm.py` — **mechanical rename, behavior-identical** (D2): the module docstring (L14–19 — rename `BOR_LLM_THINKING_LEVEL` / `default low` to `BOR_LLM_CHAT_THINKING_LEVEL`; the "rides every chat-completions request (one-shot `chat` and streaming `chat_stream`)" sentence stays TRUE after this task because both paths still ride the chat level; task 03 finalizes the wording) and `_thinking_extra_body` (L49–57, `settings.llm_thinking_level` → `settings.llm_chat_thinking_level`; docstring names the chat level). `chat()` (L432) and `chat_stream()` (L625) keep calling the same helper with the chat level — no split yet (that is task 03). The embeddings path is untouched.
5. `app/api/chat.py` — **mechanical rename, behavior-identical** (D2): the module docstring (L14–19, `BOR_LLM_THINKING_LEVEL` → `BOR_LLM_CHAT_THINKING_LEVEL`, "the effective chat level") and the per-turn log line argument (L1289, `settings.llm_thinking_level` → `settings.llm_chat_thinking_level`; the format string L1275 keeps `thinking=%s`).
6. `tests/unit/test_config.py` — L108–133: the three phase-172 tests become SIX: `llm_chat_thinking_level` (default `low` with `BOR_LLM_CHAT_THINKING_LEVEL` unset; env override `high`; empty env value stays empty) and `llm_summary_thinking_level` (default `""` with `BOR_LLM_SUMMARY_THINKING_LEVEL` unset; env override `high`; empty env value stays empty). Same monkeypatch pattern, no coercion asserted.
7. `tests/unit/test_settings_catalog.py` — L99–111: the single-row test becomes a BOTH-rows test (each `SettingMeta` in the `LLM (aipi)` group with the correct label, `str`-ish flags — not in `SECRETS`, not in `RESTART_REQUIRED`, not in `NON_OVERRIDABLE`); the completeness pin (`set(CATALOG) == set(Settings.model_fields)`) must stay green.
8. `tests/unit/test_llm_client.py` — L1330–1440: rename the kwarg `llm_thinking_level=` → `llm_chat_thinking_level=` in the existing family (behavior assertions unchanged — the helper still reads the chat level); the `chat()`-path tests keep passing as-is.
9. `tests/unit/test_chat_gate.py` — L1550–1630: rename the `Settings(..., llm_thinking_level=…)` pins → `llm_chat_thinking_level=…`; the log-line assertions (L1553, `BOR_LLM_THINKING_LEVEL` docstring) name `BOR_LLM_CHAT_THINKING_LEVEL` / the chat level.
10. `tests/integration/test_admin_settings_api.py` — L521–607: the `llm_thinking_level` row contract is renamed to `llm_chat_thinking_level` (env var `BOR_LLM_CHAT_THINKING_LEVEL`, default `low`; PUT/DELETE/anonymous-401 pins unchanged in shape). Task 02 extends it for the summary row.
11. `tests/integration/test_admin_settings_live.py` — L171–204: the live-apply test is renamed to `llm_chat_thinking_level` (same save/empty/delete pattern against the new key).
12. `tests/e2e/test_thinking_level.py` — rename the pins in place: `tr[data-key="llm_thinking_level"]` → `tr[data-key="llm_chat_thinking_level"]`, `BOR_LLM_THINKING_LEVEL` → `BOR_LLM_CHAT_THINKING_LEVEL`, the `Settings.model_fields["llm_thinking_level"]` env lookup → `llm_chat_thinking_level`. E2E runs only in isolation (the per-task gate covers unit + integration), but the file must not rot: task 05 turns it into `test_split_thinking_levels.py`.
- ASSUMPTION: A1 (phase-locked) — the rename orphans any stored `app_settings` row under `llm_thinking_level` (it becomes inert; chat falls back to env default `low`); no value migration is performed.
- ASSUMPTION: A2 (phase-locked) — the summary default is the empty string (the phase-172 off-switch: `reasoning_effort` ABSENT), never a literal `"none"`.
## Testing & Quality
- Unit: `tests/unit/test_config.py` (six thinking-level tests), `tests/unit/test_settings_catalog.py` (the completeness pin + both rows), `tests/unit/test_llm_client.py` (renamed family, unchanged assertions), `tests/unit/test_chat_gate.py` (renamed pins).
- Integration: `tests/integration/test_admin_settings_api.py` + `test_admin_settings_live.py` (renamed row contract, green).
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
## Completion Criteria
- [ ] `grep -rn "llm_thinking_level\b\|BOR_LLM_THINKING_LEVEL\b" app/ tests/ .env.example` returns **nothing** (the old name is gone everywhere; `llm_chat_thinking_level` / `BOR_LLM_CHAT_THINKING_LEVEL` in its place).
- [ ] `uv run pytest -k "config and thinking" -v` — chat default `low` + summary default `""` + env overrides for both.
- [ ] `uv run pytest` green (unit + integration), `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
- [ ] Behavior check (read, not run): `chat()` and `chat_stream()` both still ride the chat level; with both defaults (`low` / empty) and the old `.env` value, streaming requests are byte-identical to pre-phase-172.
- [ ] No behavior change in completed work.
@@ -0,0 +1,28 @@
# Task 02 — The admin row contract: both keys, set / reset / live-apply
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
**Story:** n/a
## Objective
The integration contract proves both rows behave per the overlay: they appear in the catalogue with the right group/label/default, `PUT` stores the raw string (empty included — the summary off-switch), the value resolves live without a restart, and `DELETE` resets to env — for the chat row AND the summary row, independently.
## Work
1. `tests/integration/test_admin_settings_api.py` — the phase-172 row contract (now `test_llm_chat_thinking_level_row_contract`, from task 01) gains a **summary twin** `test_llm_summary_thinking_level_row_contract` following the same shape, with the summary specifics:
- `GET /api/admin/settings` lists `llm_summary_thinking_level` in the `LLM (aipi)` group, `str`, label "Summary thinking level", `env_var` `BOR_LLM_SUMMARY_THINKING_LEVEL`, effective default `""` (with `BOR_LLM_SUMMARY_THINKING_LEVEL` deleted from the env for the test — the house `monkeypatch.delenv(…, raising=False)` pattern).
- `PUT {API}/llm_summary_thinking_level` with `{"value": "high"}` → 200, the raw string stored (`SELECT value FROM app_settings WHERE key = 'llm_summary_thinking_level'` → `"high"`), `get_settings().llm_summary_thinking_level == "high"` **without a restart**.
- `PUT` with `{"value": ""}` → 200 (the off-switch is storable — an empty stored value must not be rejected as "no value"), `get_settings().llm_summary_thinking_level == ""`.
- `DELETE` → 200, the row gone from `app_settings`, `get_settings().llm_summary_thinking_level == ""` (the env default).
- An anonymous (non-admin) `PUT` → the 401/403 pin, same as the chat twin.
- Independence pin: after `PUT`-ing the summary row `"high"`, the chat row's effective value is still `"low"` (the two overlays resolve independently — the whole point of the split).
2. `tests/integration/test_admin_settings_live.py` — the chat live-apply test (from task 01) gains a **summary twin** `test_llm_summary_thinking_level_live_apply_no_restart`: `delenv` `BOR_LLM_SUMMARY_THINKING_LEVEL` → default `""`; `_save(db, "llm_summary_thinking_level", "high")` → `get_settings().llm_summary_thinking_level == "high"` **NO restart**; `_save(…, "")` → `""`; `DELETE FROM app_settings WHERE key = 'llm_summary_thinking_level'` → back to `""`.
3. Verify by running: `uv run pytest tests/integration/test_admin_settings_api.py tests/integration/test_admin_settings_live.py -v` — both families green, and the old single-row test is gone (no `llm_thinking_level` key referenced anywhere: `grep -rn "llm_thinking_level\b" tests/` empty).
## Testing & Quality
- Integration: the two extended families above (row contract + live-apply, chat AND summary).
- Coverage: **>90%** on `app/` (this task is test-only — `app/` code is untouched, so the task-01 coverage must simply stay green).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_admin_settings_api.py -v -k "thinking" -v` — the chat + summary row contracts green (group/label/env-var/default, PUT raw storage incl. empty, DELETE reset, anonymous rejected, independence).
- [ ] `uv run pytest tests/integration/test_admin_settings_live.py -v -k "thinking"` — both live-apply twins green (no restart, empty storable, delete resets).
- [ ] `uv run pytest` green, `app/` coverage >90%.
- [ ] No behavior change in completed work.
@@ -0,0 +1,41 @@
# Task 03 — The wire split: `chat_stream()` → chat level, `chat()` → summary level
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
**Story:** n/a
## Objective
The one-shot `chat()` path (document summaries, folder summaries, KB overview, the `check_models` probe) sends `reasoning_effort` from the **summary** level — absent by default (A2) — while `chat_stream()` keeps sending it from the **chat** level (default `low`), so the two surfaces are independently tunable on the wire.
## Work
1. `app/rag/llm.py` — `_thinking_extra_body` (L49–57) becomes a pure function of the level string (drop the `Settings` read):
```python
def _thinking_extra_body(level: str) -> dict[str, str] | None:
"""The thinking level as the chat-completions extra body (phase 178).
Non-empty (trimmed) value → ``{"reasoning_effort": <value>}`` (the
OpenAI-standard top-level parameter, phase-172 A1); empty /
whitespace → ``None`` (the off switch — the parameter is ABSENT
from the request, phase-172 A2; never a literal ``"none"`` or
``"off"``).
"""
trimmed = level.strip()
return {"reasoning_effort": trimmed} if trimmed else None
```
2. `app/rag/llm.py` — `chat()`'s single-attempt body (L432): `extra = _thinking_extra_body(self.settings.llm_summary_thinking_level)`; the comment (L443–444) becomes "Phase 178 (A3/A2): the **summary** level rides as `reasoning_effort` (document summaries, folder summaries, KB overview, the `check_models` probe); the key is absent (never None) when off — the default (empty) keeps one-shot requests byte-identical to pre-phase-172."
3. `app/rag/llm.py` — `chat_stream()` (L625–628): `if (extra := _thinking_extra_body(self.settings.llm_chat_thinking_level)) is not None: …` (the comment L626–627 becomes "Phase 178 (A3/A2): the **chat** level rides as `reasoning_effort`; the key is absent (never None) when off.").
4. `app/rag/llm.py` — finalize the module docstring (L1–25, rewritten in task 01): replace the single-level paragraph with the split wording — the chat turn's level (`BOR_LLM_CHAT_THINKING_LEVEL`, default `low`) rides streaming `chat_stream`; the one-shot level (`BOR_LLM_SUMMARY_THINKING_LEVEL`, default empty = off) rides `chat` (document summaries, folder summaries, KB overview, the `check_models` probe); an empty value is the off switch (parameter absent, byte-identical to pre-phase-172); the embeddings path never carries it. Note the probe's consequence: `check_models` (L805) goes through `chat()` → off by default — it is a reachability ping, no special-casing.
5. `tests/unit/test_llm_client.py` — the mocked-client family (L1330–1440, renamed in task 01):
- The `chat()`-path tests (currently `llm_chat_thinking_level="low"` / `"low "` / `" "` at L1335–1368) are re-pointed at the **summary** kwarg: `llm_summary_thinking_level="high"` → the `chat.completions.create` call carries `extra_body={"reasoning_effort": "high"}`; `" high "` → trimmed `"high"`; `""` / `" "` → the call kwargs contain **no** `extra_body` key (never `None`, never a literal "none").
- The `chat_stream()`-path tests (L1385–1422) keep their shape: `llm_chat_thinking_level="high"` / `" high "` → `extra_body={"reasoning_effort": "high"}`; `""` / `" "` → no `extra_body` key.
- Add the cross-pins (the whole point of the split): client with `llm_chat_thinking_level="high"` + `llm_summary_thinking_level=""` (the defaults inverted) → `chat_stream()` carries `reasoning_effort="high"` while `chat()` carries no `extra_body`; the default client (chat `"low"` + summary `""`) → `chat()` is byte-identical to pre-phase-172 (no `extra_body`) while `chat_stream()` carries `reasoning_effort="low"`.
- The embeddings-path test (no `extra_body` ever) stays green untouched.
## Testing & Quality
- Unit: `tests/unit/test_llm_client.py` (the re-pointed + extended family above; the mocked-SDK-client pattern).
- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_llm_client.py -v -k "thinking" -v` — per-method `extra_body` behavior green (chat level → `chat_stream` only; summary level → `chat` only; absent-when-empty; cross-pins).
- [ ] Default-config read check: with chat `low` + summary `""`, a `chat()` request carries no `extra_body` and a `chat_stream()` request carries `reasoning_effort="low"` (the test pins above).
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work (the streaming path is byte-identical at the defaults; the one-shot path is off by default — the TODO's "summary should be none").
@@ -0,0 +1,25 @@
# Task 04 — The per-turn line: `thinking=<chat level>`, never the summary knob
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
**Story:** n/a
## Objective
The per-turn log line (PLAN §9, AGENTS rule 10) is pinned to the **chat** level — the level the turn actually ran at: `thinking=<trimmed chat level>` (`-` when empty) immediately before `thinking_chars=N`, with a cross-pin proving the summary level (set to a different value) never leaks into the line. No `query_log` schema change (phase-172 A6 / phase-178 A5, kept).
## Work
1. `app/api/chat.py` — the per-turn log line (L1272–1291) already reads the chat level after task 01's mechanical rename (`settings.llm_chat_thinking_level.strip() or "-"` L1289, format `thinking=%s` L1275): verify by reading, no functional change. Finalize the inline comment (L1289) to "phase 178 (A5): the effective **chat** thinking level, trimmed, `-` when empty — the level this turn ran at; the summary level is a one-shot concern and gains no field here."
2. `app/api/chat.py` — the module docstring (L14–19, renamed in task 01): finalize the wording — "The per-turn log line records `thinking=<level>` (the effective `BOR_LLM_CHAT_THINKING_LEVEL`, trimmed, `-` when empty) before `thinking_chars` (phase 172, A6; phase 178, A5 — the **chat** level; the one-shot `BOR_LLM_SUMMARY_THINKING_LEVEL` never appears on the line)."
3. `tests/unit/test_chat_gate.py` — the log-line family (L1545–1630, pins renamed in task 01: the default `thinking=low`, the `"high"` override, the `""` → `-` off-switch, the `" "` → `-` trim pin, the full field-order pin):
- Update the test docstring (L1550–1557) to name `BOR_LLM_CHAT_THINKING_LEVEL` / the chat level (phase 178, A5).
- Add the **cross-pin** (the split's proof on the log line): `Settings(_env_file=None, relevance_threshold=0.30, llm_chat_thinking_level="low", llm_summary_thinking_level="high")` → the line is `thinking=low thinking_chars=…` — the summary's `high` does NOT appear as the line's `thinking=` value.
- Add the inverse off-pin: `llm_chat_thinking_level=""` + `llm_summary_thinking_level="high"` → `thinking=- thinking_chars=…` (the chat off-switch logs `-` even though the summary knob is set).
## Testing & Quality
- Unit: `tests/unit/test_chat_gate.py` (the extended log-line family above).
- Coverage: **>90%** on `app/` (this task touches only the docstring/comment + tests — coverage must stay green).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_chat_gate.py -v -k "log" -v` — the full log-line family green: default `thinking=low`, override `high`, off-switch `-`, trim `-`, field order, AND the two cross-pins (summary level never leaks into `thinking=`).
- [ ] `grep -n "BOR_LLM_SUMMARY_THINKING_LEVEL" app/api/chat.py` returns nothing (the one-shot knob has no presence on the chat turn's log line, A5).
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
- [ ] No behavior change in completed work.
@@ -0,0 +1,30 @@
# Task 05 — The E2E suite: `test_split_thinking_levels.py` + regression + gates + commit
**Phase:** `178_split_thinking_levels` · **Source:** `TODO.md:3` — "Need a separate summary thinking level and chat thinking level. Default thinking level for summary should be none. Chat should stay at low"
**Story:** n/a
## Objective
The phase's dedicated Playwright suite (AGENTS rule 4 — one test file for the phase, run in isolation) proves the operator's acceptance in a real browser: TWO free-form thinking-level rows in the admin Settings panel (chat default `low`, summary default empty = the TODO's "none"), independently saveable/resettable with live-apply, plus the full phase gates and the atomic commit.
## Work
1. `tests/e2e/test_thinking_level.py` → `tests/e2e/test_split_thinking_levels.py` — `git mv` the phase-172 suite (the module scaffolding is the house pattern, kept: `_boot_app` / `_stop_app` / `_clean_settings_state` / the autouse `_clean` (truncates `app_settings` + `ui_settings`, deletes `e2e-`-labeled tokens, module teardown pins `app_settings` empty) / the `.env` mtime guard / the `_cookies` / `_admin_client` / `_catalog_item` helpers) and rewrite the module docstring (Source → `TODO.md` L3 this item; the test → contract mapping below; the env pins: the module app opts BACK IN over the conftest's global `BOR_SETTINGS_DB=0` pin (`env["BOR_SETTINGS_DB"] = "1"`), and pins `BOR_LLM_CHAT_THINKING_LEVEL=low` AND `BOR_LLM_SUMMARY_THINKING_LEVEL=` (empty string SET — so both badges read `env`, not `default`, and an operator's local `.env` cannot leak a different level into the suite; the house leak-guard pattern) and `BOR_LLM_CHAT_MODEL` (the "like chat model" reference)).
2. The three tests (renamed/extended from the phase-172 trio):
- `test_both_rows_render_free_form_with_defaults` — the `LLM (aipi)` group band is visible (the phase-171 band contract); a row labelled **Chat thinking level** with the `BOR_LLM_CHAT_THINKING_LEVEL` env-var display shows `low` with the `env` badge; a row labelled **Summary thinking level** with the `BOR_LLM_SUMMARY_THINKING_LEVEL` display shows an **empty** input with the `env` badge (the TODO's "none" = the off switch, A2); BOTH inputs are `type="text"` (free-form — NOT `number`/`select`, the A4 contract); the `llm_chat_model` row's input is the same control (the "like chat model" reference, one assertion pair); `tr[data-key="llm_thinking_level"]` is **absent** (the old key is gone, A1).
- `test_saved_levels_persist_without_restart` — the phase-168 §7.4 Save lifecycle per row: type `think-soft` (no allowlist — A4) into the summary row and `high` into the chat row (dirty hints appear), save both; the API catalogue (`_catalog_item`) AND the raw `app_settings` rows carry the saved values for EACH key independently; a reload of `/settings.html` still shows `think-soft` / `high` with the `DB` badges (no restart — the phase-168 contract); the independence pin: the two rows' controls are distinct and the saved values do not cross.
- `test_reset_returns_to_env_defaults` — the per-row Reset (the per-row `DELETE`): after the saves, both Resets are live; clicking the summary Reset lands the value at **empty** with the `env` badge, clicking the chat Reset lands `low` with the `env` badge (the overlay reset, task 02's contract, in the browser); the status lines show the house "… is back to the environment value." toast for each.
3. Regression set (in isolation, per the house E2E protocol): `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` (phase 168) and `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` (phase 171) — both green (they do not pin the thinking rows — verified: no `thinking` references — but the two new rows must not break the panel/layout contracts).
4. The full gates: `uv run pytest` (unit + integration) green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage **>90%**; `uv run ruff check . && uv run pyright` clean; `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` green **in isolation** (DB up: `podman compose up -d db`).
5. Commit (AGENTS rule 8): ONE atomic Conventional Commit, `--no-gpg-sign` — e.g. `feat(llm): split the thinking level into a chat level (default low) and a summary level (default off)` — body: the TODO L3 source, the rename (A1), the wire split (A3), the log line (A5), the two admin rows; then move the phase dir: `git mv .agents/phases/todo/178_split_thinking_levels .agents/phases/complete/178_split_thinking_levels` and include it in the commit (the `.agents/` tree is tracked — only `.agents/phase-sessions/` + `.agents/pipeline.log` are gitignored).
6. `TODO.md` — ALREADY CLEARED by the conversion (the file holds only the `# TODO` title; verify by reading — do not re-add the item).
## Testing & Quality
- E2E (AGENTS rule 4): `tests/e2e/test_split_thinking_levels.py` run **in isolation** (the three tests above).
- Coverage: **>90%** on `app/`.
- Lint/types: ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_split_thinking_levels.py -v --no-cov` green **in isolation** (both rows, defaults `low`/empty, free-form, save + reload persistence per key, reset to env defaults, old key absent).
- [ ] `uv run pytest tests/e2e/test_admin_settings_panel.py -v --no-cov` and `uv run pytest tests/e2e/test_admin_settings_layout.py -v --no-cov` green in isolation (regression).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] One atomic `--no-gpg-sign` Conventional Commit including the `.agents/` phase move; `TODO.md` holds only the `# TODO` title.
- [ ] No behavior change in completed work.
@@ -0,0 +1,41 @@
# Phase 179 — Git clone server freeze: investigate the owner's suspicion, then fix it
**Source:** `TODO.md` L3 — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-11)
**Context:** `app/api/sync.py` — `start_sync` (L421–438, `asyncio.create_task(_run_sync())` L438), `_run_sync` (L482, **async**, runs on the app's single uvicorn event loop), `await check_models(llm)` (L533 — already async HTTP probes), the short-lived `SessionLocal()` + `effective_sources(db)` (L538–540), the per-source `_check_cancelled()` boundary (L565), **`root = clone_or_pull(…)` (L577 — synchronous)**, **`file_commit_dates(root)` (L589 — synchronous)**, `sync_status` (L353–354 — "the UI polls this every 2 s"), `cancel_sync` (L442–459); `scripts/git_sync.py` — `clone_or_pull` (L96), `file_commit_dates` (L270), `run_git` (L320 — `subprocess.run` L342, **blocking**); `app/api/health.py` — `GET /api/health` (L13, mounted at `/api/health` via `app/main.py` L211); `scripts/import_docs.py` — the standalone script path (a SEPARATE process, never on the app's event loop — NOT a freeze vector); `app/api/doc_drafts.py` — `push_doc_draft` (a sync `def` → FastAPI threadpool — NOT a freeze vector); `app/api/sync.py` L233 (`from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates` — the names to monkeypatch); `tests/integration/test_sync_api.py` — the `sync_client` context-managed `TestClient` pattern (L176–179 — one app event loop across requests) + the env-fallback `BOR_GIT_SOURCES` driving (L64–65 docstring); `tests/e2e/test_sync_cancel.py` — the dedicated module-app E2E pattern (dedicated port, `file://` fixture repo via the `_git` helper, the `_wall_guard` module fixture); `tests/e2e/slow_llm.py` (not needed here — the mock LLM import is fast; the fixture's slowness is in the CLONE, not the import).
## Objective
Verify the owner's suspicion with a deterministic reproduction (a long `git clone` during a sync blocks the app's event loop, stalling the entire HTTP surface — including the 2 s sync-status poll the UI relies on), record the verdict, and — the suspicion being confirmed — make the sync's git operations run OFF the event loop so the server stays fully responsive for the whole duration of a long clone (and the phase-134 cooperative cancel keeps its contract).
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the repro is in-process and deterministic (task 01).** A new `tests/integration/test_sync_git_freeze.py` follows `test_sync_api.py`'s `sync_client` pattern (context-managed `TestClient` — one app event loop across requests). `app.api.sync.clone_or_pull` and `app.api.sync.file_commit_dates` are monkeypatched (the module-level names imported at L233) with stubs that record a `time.monotonic()` start/end and sleep 3 s. The probe test measures the wall-clock latency of a `GET /api/sync/status` issued while the stub is in its sleep. The probe asserts the **PRE-fix** behavior (the probe is blocked — latency ≥ 2.5 s of the 3 s stub) → the test is GREEN before the fix and IS the demonstration of the freeze; the verdict (**CONFIRMED** / not) + mechanism + the measured latency are recorded in the test module docstring. No app/ code is touched in task 01.
- **D2 — the fix is `asyncio.to_thread`, nothing else (task 02).** Inside `_run_sync`: `root = await asyncio.to_thread(clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch)` and `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)`. The module docstring and the phase-134 boundary comment (L565–576) update the semantics: a cancel registered while a git op is in flight no longer parks the EVENT LOOP — the git **worker thread** runs to completion (Python threads are not interruptible), the loop stays free (every other request, status poll included, is served), and the run still stops at the NEXT `_check_cancelled()` boundary (the phase-134 contract, unchanged). The task-01 probe flips to the post-fix assertion (probe latency < 1.5 s during the 3 s stub clone). The short-lived `SessionLocal()` / `effective_sources` DB calls stay on the loop (A3 — millisecond-scale, out of scope, noted in the docstring). `check_models` is already async HTTP — untouched.
- **D3 — the E2E pins the real pipeline (task 03).** `tests/e2e/test_sync_git_freeze.py` (the phase's dedicated Playwright suite, AGENTS rule 4): the dedicated module app on a dedicated port (the `test_sync_cancel` pattern — the session app is never started in the isolated run), mock LLM, a dedicated `BOR_SOURCES_DIR`, `BOR_GIT_SOURCES=file://<big repo>`, the shared E2E Postgres truncated before AND after every test. The `big_git_repo` fixture is a GENERATED local repo — N commits × ~1 MiB of near-incompressible (seeded pseudo-random) `.md` files — sized at authoring so a FRESH `git clone` of it takes ≥ 5 s on the test host (the pinned constants + the calibration note live in the module; `git` on PATH is the documented prerequisite, phase 28). Contract 1: while the run is in the clone phase, `GET /api/sync/status` polls (via `page.request`, ~2 s apart like the UI) each complete in < 1.5 s and the Sources page keeps updating with no network-error state; the run lands `success` and the sentinel file is in the KB. Contract 2 (phase 134 under to_thread): a cancel registered during the clone — the checkout dir is deleted first so the run re-clones (a long git phase again) — the status endpoint stays responsive (< 1.5 s) while the in-flight git thread finishes, and the run lands the `cancelled` terminal (error null, the UI's "Sync cancelled" neutral state). The `_wall_guard` module fixture pins the budget (the `test_sync_cancel` pattern — cut the fixture size if it drifts, never the guard). `E2E_REAL_LLM=1` is skipped (mock-only — the clone, not the import, is the timing fixture).
- **NOT touched:** `scripts/import_docs.py` (standalone process), `scripts/git_sync.py` (the git layer is unchanged — only its SCHEDULING changes), `app/core/docs_push.py` / `app/api/doc_drafts.py` (sync `def` → threadpool), `check_models`, the async LLM import path, the frontend, any other endpoint.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; this phase investigates + fixes the in-process sync path)
## Tasks
1. `01_repro_verdict.md` — the deterministic in-process repro (3 s stubbed clone, status-poll latency probe) that demonstrates the event-loop freeze and records the verdict in the test module docstring.
2. `02_offloop_git_ops.md` — the fix: `clone_or_pull` / `file_commit_dates` off the event loop via `asyncio.to_thread`, the docstring/boundary-note update, the probe flipped to the post-fix assertion.
3. `03_e2e_freeze_contract.md` — `tests/e2e/test_sync_git_freeze.py` (the phase's dedicated Playwright suite, AGENTS rule 4): a real long `file://` clone, the responsive-status-poll contract, the phase-134 cancel-during-clone contract, the full gates, the atomic commit.
## Testing & Quality
- **Integration:** `tests/integration/test_sync_git_freeze.py` — the probe: pre-fix (task 01) the status poll issued during the 3 s stubbed clone is blocked (≥ 2.5 s); post-fix (task 02) it completes in < 1.5 s; the run reaches a terminal state in both (fixture cleanliness). `test_sync_api.py` + `test_import_docs_git.py` stay green (no behavior change beyond loop-freeness).
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_sync_git_freeze.py`, run **in isolation** — the clone-phase status polls responsive (< 1.5 s each, ~2 s apart), the Sources page updating, the `success` terminal + sentinel in the KB; the cancel-during-clone → `cancelled` terminal with the status endpoint responsive while the git thread finishes.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the only `app/` change is two `await asyncio.to_thread(…)` lines + docstrings in `app/api/sync.py`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the post-fix assertion (probe < 1.5 s during the 3 s stub clone) and the verdict ("CONFIRMED" + mechanism + measured latency) recorded in the module docstring.
- [ ] `app/api/sync.py::_run_sync` runs the per-source git ops off the event loop (`asyncio.to_thread`); the phase-134 boundary note documents the worker-thread-runs-to-completion semantics; the DB calls stay on the loop (the A3 note).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` green **in isolation** (the responsive-poll contract + the cancel-during-clone contract, wall-time guard holds).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `fix(sync): run the sync's git clone/pull off the event loop so long clones no longer freeze the server`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-11, roadmap confirmation)
- **A1 — scope: investigate + fix.** The TODO item is an investigation ("find out if this is true"); the owner confirmed the phase goes on to the minimal fix. If the repro in task 01 had landed NOT-confirmed, task 02 would document the no-fix decision instead — the code-path analysis (the synchronous `subprocess.run` inside the async `_run_sync`) and the repro together confirm the freeze.
- **A2 — fix mechanism: `asyncio.to_thread`.** No new packages; the single-event-loop uvicorn is unchanged; a cancel at the git phase leaves the in-flight git worker thread running to completion (Python threads are not interruptible) while the event loop is free — the phase-134 cooperative-cancel contract (stop at the next boundary, `cancelled` terminal, no corruption) is preserved.
- **A3 — out of scope: the sync's short-lived DB calls.** `SessionLocal()` / `effective_sources` (L538–540) are millisecond-scale SQLAlchemy calls — not the reported symptom; they stay on the loop and are noted in the docstring as a known, bounded, non-fix.
- **A4 — the E2E slow clone is a generated local `file://` repo.** N commits × ~1 MiB seeded pseudo-random `.md` files, sized at authoring so a fresh clone takes ≥ 5 s on the test host; deterministic, no network, `git` on PATH is the documented prerequisite (phase 28).
@@ -0,0 +1,27 @@
# Task 01 — Reproduce the freeze in-process and record the verdict
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a
## Objective
A deterministic, in-process reproduction that proves (or disproves) the owner's suspicion: while a sync's git clone is in flight, the app's event loop is blocked and the whole HTTP surface — including `GET /api/sync/status`, the endpoint the UI polls every 2 s — is stalled. The verdict (CONFIRMED / not, the mechanism, the measured latency) is recorded in the test module docstring.
## Work
1. `tests/integration/test_sync_git_freeze.py` — a NEW file following `tests/integration/test_sync_api.py`'s house patterns:
- The `sync_client` fixture (L176–179 pattern): a **context-managed** `TestClient(fastapi_app)` — one app event loop across requests (the docstring L64–65 explains why the context manager is required). Admin login via the house helper (the `_login` pattern, `test_sync_api.py` L260–265).
- The git source via the **env fallback**: drive `BOR_GIT_SOURCES=file://<tmp repo>` the way `test_sync_api.py` does (a fresh `Settings` on the resolver's module; the dev `.env` never leaks in). The tmp repo is a real one-commit git repo built with the house `_git` helper pattern (`tests/e2e/test_sync_cancel.py` L156–195 shape: `git init -q`, `add -A`, `-c user.email=… -c user.name=… -c commit.gpgsign=false commit -qm`); the stub never actually clones it, so the repo's content is irrelevant — one small `.md` file suffices.
- **The stubs (the timing fixture):** `monkeypatch.setattr("app.api.sync.clone_or_pull", …)` and `monkeypatch.setattr("app.api.sync.file_commit_dates", …)` — the module-level names imported at `app/api/sync.py` L233 (patching the `scripts.git_sync` originals would not affect the runner). The `clone_or_pull` stub: on entry sets a `threading.Event` (`clone_started`) and records `time.monotonic()`, sleeps **3.0 s** (simulating a long clone), records the end, and returns the tmp repo's `Path`; the `file_commit_dates` stub returns `{}`. Both are plain sync functions — the point is that PRE-fix they run ON the event loop.
- **The probe test** `test_long_git_clone_freezes_the_event_loop`: `POST /api/sync` (admin) → wait for `clone_started` (bounded wait, ~10 s) → `t0 = time.monotonic()`, `client.get("/api/sync/status")`, `elapsed = time.monotonic() - t0` → **assert `elapsed >= 2.5`** (the probe request, issued into the middle of the 3 s stubbed clone, is BLOCKED — it cannot be served until the stubbed clone returns and the event loop is free again). That assertion IS the freeze demonstration and it is GREEN pre-fix. Then poll `GET /api/sync/status` to the terminal state (the stub returns, so the run completes: `success` with the import, or a clean terminal — whichever the stubbed root yields; pin whichever the suite shows and document it) so the fixture is left clean for the next test.
- **The module docstring (the verdict record):** the owner's TODO item quoted (L3), the code path (`POST /api/sync` → `asyncio.create_task(_run_sync())` — `app/api/sync.py` L438/L482 → the synchronous `clone_or_pull` L577 → `run_git`'s blocking `subprocess.run` — `scripts/git_sync.py` L342 — with no `to_thread`/`create_subprocess_exec` anywhere in `app/` or `scripts/`), the measured probe latency (e.g. "the status probe took ≈3.0 s of the 3.0 s stubbed clone — the event loop was stalled for the full git op"), and the verdict line: **CONFIRMED** (or NOT — if the probe came back fast, stop and flag it in the docstring + the phase report rather than proceeding to a fix).
- `GET /api/health` (`app/api/health.py` L13, mounted at `/api/health`) may be used as a SECOND probe in the same test (the same assertion — it is blocked too: any endpoint on the loop stalls); keep the status probe primary (it is the endpoint the UI's 2 s poll depends on).
2. Verify by running: `uv run pytest tests/integration/test_sync_git_freeze.py -v` — green (the probe is green with the PRE-fix assertion: it documents the freeze, it is not a failing test), and `uv run pytest` — the full suite green (no other test is affected: the stubs live inside this one test's monkeypatch).
## Testing & Quality
- Integration: the probe family above (the blocked-latency assertion, the terminal-state assertion, the admin/anonymous gating falls out of the house `sync_client` pattern — add the anonymous-403 pins if the house pattern does not already carry them in this file).
- Coverage: **>90%** on `app/` (this task is test-only — `app/` code is untouched, so the suite's coverage must simply stay green).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the pre-fix assertion (`elapsed >= 2.5` for the status probe issued mid-clone).
- [ ] The module docstring records: the TODO item, the code path, the measured latency, and the verdict (**CONFIRMED** — expected, per the code-path analysis).
- [ ] `uv run pytest` green.
- [ ] No `app/` code change in this task.
@@ -0,0 +1,28 @@
# Task 02 — The fix: the sync's git operations run off the event loop
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a
## Objective
With the freeze confirmed (task 01's verdict), make the server stay fully responsive for the whole duration of a long `git clone` / `git pull` (and the `file_commit_dates` walk): the sync's per-source git operations run in a worker thread via `asyncio.to_thread`, the phase-134 cooperative-cancel contract is preserved, and the task-01 probe flips to pin the post-fix behavior.
## Work
1. `app/api/sync.py` — `_run_sync` (L482), the per-source git block (L566–589):
- L577: `root = clone_or_pull(…)` → `root = await asyncio.to_thread(clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch)` (the three-argument call, phase 121/135, unchanged — only its scheduling changes).
- L589: `doc_dates_by_root[str(root)] = file_commit_dates(root)` → `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)`.
- `import asyncio` if not already imported at module level (check first — the module already uses `asyncio.create_task` at L438, so `asyncio` is imported; verify, do not duplicate).
- **The phase-134 boundary note (L565–576)** — update the comment: a cancel registered while a git op is in flight no longer parks the EVENT LOOP; the git **worker thread** runs its blocking `subprocess` to completion (Python threads are not interruptible — `_task.cancel()` unblocks the run at the `to_thread` await point / the run stops at the NEXT `_check_cancelled()` boundary), while every other request — status polls included — is served immediately. The observable contract is unchanged (phase 134: stop at the next boundary, `cancelled` terminal, no corruption).
- **The module docstring** — add the freeze note: pre-phase-179 the synchronous `clone_or_pull` / `file_commit_dates` (their `subprocess.run` calls — `scripts/git_sync.py` L342) ran ON the app's event loop, stalling the whole HTTP surface (including the 2 s `sync_status` poll) for the duration of a long clone (the TODO L3 suspicion — confirmed by `tests/integration/test_sync_git_freeze.py`); phase 179 runs them off the loop via `asyncio.to_thread`. Include the **A3 note**: the short-lived `SessionLocal()` / `effective_sources` DB calls (L538–540) remain on the loop — millisecond-scale, a known bounded non-fix, deliberately out of scope.
- `check_models` (L533) is untouched (already async HTTP probes). The kind-local branch (L590+) is untouched (no subprocess in it).
2. `tests/integration/test_sync_git_freeze.py` — flip the task-01 probe to the post-fix assertion: the status probe issued into the middle of the 3 s stubbed clone now completes in **< 1.5 s** (the event loop is free while the stubbed clone runs in the worker thread); keep the terminal-state assertion. Update the module docstring: the verdict section now reads CONFIRMED (task 01) → FIXED (phase 179, `asyncio.to_thread`), with the post-fix measured latency recorded (the probe ≈ instant, e.g. "the status probe took ≈0.01 s while the 3.0 s stubbed clone ran in the worker thread").
3. Verify by running: `uv run pytest tests/integration/test_sync_git_freeze.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -v` — all green (the flipped probe green; the sync families show no behavior change beyond loop-freeness), then `uv run pytest` — the full suite green.
## Testing & Quality
- Integration: the flipped probe (post-fix: status probe < 1.5 s during the 3 s stubbed clone) + the `test_sync_api.py` / `test_import_docs_git.py` regression set.
- Coverage: **>90%** on `app/` (the only `app/` change is two `await asyncio.to_thread(…)` lines + docstring/comment in `app/api/sync.py` — the suite's coverage must stay green).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the POST-fix assertion (probe < 1.5 s during the 3 s stub clone) and the docstring verdict updated (CONFIRMED → FIXED, post-fix latency recorded).
- [ ] `app/api/sync.py::_run_sync` awaits `asyncio.to_thread` for BOTH git ops; the phase-134 boundary note + the module docstring document the worker-thread semantics and the A3 DB note; `check_models` untouched.
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -v` green; `uv run pytest` green.
- [ ] No behavior change in completed work beyond loop-freeness (the sync pipeline's sequence, errors, and cancel terminals are unchanged).
@@ -0,0 +1,32 @@
# Task 03 — E2E: the status poll stays responsive through a real long clone (the phase's dedicated Playwright suite)
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a
## Objective
Pin the fix against the REAL pipeline (the owner's exact suspicion, end to end): a sync whose first action is a genuinely long `git clone` (a generated local `file://` repo) must keep the whole HTTP surface — the 2 s `GET /api/sync/status` poll and the Sources page — responsive for the entire duration of the clone, the run must land `success`, and the phase-134 cancel-during-clone contract must hold under the off-loop scheduling. Then run the full gates and commit the phase.
## Work
1. `tests/e2e/test_sync_git_freeze.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped — the CLONE, not the import, is the timing fixture), following `tests/e2e/test_sync_cancel.py`'s house pattern point for point:
- **Isolation:** a per-module app on a **dedicated port** (the conftest session app is never started in this isolated run), the mock LLM on the conftest port, a dedicated `BOR_SOURCES_DIR` for the fixture repo's checkout, `BOR_GIT_SOURCES=file://<big repo>` (the env-fallback row — the `test_sync_cancel` L256 shape), the shared E2E Postgres truncated before AND after every test.
- **The `big_git_repo` fixture (the timing fixture — A4):** a module-scoped helper (the `_git` pattern, `test_sync_cancel.py` L156–195) that generates N commits, each adding one ~1 MiB `notes/note-<k>.md` whose body is **seeded pseudo-random** text (deterministic — a fixed-seed generator, e.g. `random.Random(k)` over a word list — so the pack is near-incompressible and the clone time is stable; plain repeated text would compress away and the clone would be fast). **Calibrate at authoring time**: pick N (start ~200) so a FRESH `git clone file://<repo>` takes **≥ 5 s** on the test host (measure it during authoring, record the measured clone time in the module docstring, and pin N + the file size as module constants with the calibration note). The last file carries the sentinel marker (the `test_sync_cancel` sentinel shape).
- **Test 1 — `test_status_polls_responsive_during_long_clone`:** admin login (the house helper) → trigger the sync (the UI's sync button, or the API via `page.request` — the `test_sync_cancel` L100-ms `page.request` poll shape) → while the run reports `running` and is still in the clone phase (the status's `current_file` is null / no file positions — the import has not started), poll `GET /api/sync/status` via `page.request` at ~2 s intervals (the UI's poll cadence): **every poll must complete in < 1.5 s** (the UI's 2 s poll budget — pre-fix the polls issued during the clone would stall for the whole clone and this is the owner's frozen server, observable as hung status polls / a stalled Sources page) → the Sources page keeps updating (no network-error state, the status label keeps advancing) → the run lands the `success` terminal → the sentinel file is in the KB (the house drill-down/`GET /api/docs` assertion shape).
- **Test 2 — `test_cancel_during_clone_reaches_terminal`:** first DELETE the fixture repo's checkout dir under `BOR_SOURCES_DIR` (the run re-clones — a long git phase again), trigger a second sync, and while it is in the clone phase register the cancel (the UI's Cancel control or `POST /api/sync/cancel` via `page.request`) → **the status endpoint stays responsive** (the `page.request` polls during the in-flight git worker thread all complete in < 1.5 s — the A2 observable: the git thread runs to completion in the background while the loop serves requests) → the run lands the `cancelled` terminal (the phase-134 contract: `error: null`, the in-flight positions cleared, the UI's neutral "Sync cancelled" state — no error banner, the main button retry-ready).
- **The `_wall_guard` module fixture** (the `test_sync_cancel` pattern): budget the module (two syncs, the first a full clone + mock import of N files, the second a re-clone + cancel) — budget it at the measured clone time + import span + setup, pin the guard; **if the suite drifts, cut N, not the guard.**
- The module docstring: the owner's TODO item quoted, the fix (task 02's `asyncio.to_thread`), the fixture calibration (N, the measured clone time), the timing paragraph, the run-in-isolation command (`uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov`), the environment prerequisites (DB up: `podman compose up -d db`; `git` on PATH — the phase-28 documented prerequisite), the test→observable mapping (the house shape).
2. **The full gates** (the phase's final pass):
- `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` — green **in isolation**.
- `uv run pytest --cov=app --cov-report=term-missing` — the full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
- The regression spot-check in isolation: `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` (the phase-134 cancel E2E — the contract this phase must not have broken; run it AFTER the new suite so the DB truncate ordering is clean).
3. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/api/sync.py`, the two test files, and the `.agents/` tree — e.g. `fix(sync): run the sync's git clone/pull off the event loop so long clones no longer freeze the server` (the commit body names the TODO L3 investigation + the verdict). Move the phase dir to `.agents/phases/complete/179_git_clone_server_freeze/`.
## Testing & Quality
- E2E: the dedicated suite above (the responsive-poll contract + the cancel-during-clone contract, the `test_sync_cancel` regression spot-check).
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` green **in isolation** (every status poll < 1.5 s during the real clone, `success` terminal + sentinel in the KB, cancel-during-clone → `cancelled` terminal with the status endpoint responsive, wall-time guard holds).
- [ ] `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` green in isolation (the phase-134 contract intact).
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,54 @@
# Phase 180 — Markdown inline links, inline images, and the full heading ladder
**Source:** `TODO.md` L4 ("Markdown [links](some-address) aren't being render correctly and show up as plain text"), L5 ("Quad-level headers (####) aren't being rendered and show up as plain text") and the in-answer half of L3 ("Need SVG render support on the front end when citing … source")
**Story:** n/a (TODO-derived phase, Protocol B append — owner roadmap confirmation 2026-10-14)
**Context:** `frontend/assets/markdown.js` — the SHARED escape-first renderer used by all three pages (`index.html` chat bubbles + thinking, `document.html` viewer, `shared.html`) and by `document-modal.js` through `document.js`'s `renderDocument`. Pipeline order: step 1 fence protection (`\u0000CODE n\u0000`) → step 1b GFM pipe-table protection (`\u0000TABLE n\u0000`, phase 44) → step 2 `text = inline(escapeHtml(text))` → step 2b blockquotes (phase 138) → step 3 the paragraph pass (split on `\n{2,}`, block guard `/^<(h\d|ul|ol|pre|li|blockquote)/`) → step 4 restore. The `inline(s)` chain (the single source of every inline transform, also applied per table cell) is today, in order: code span → `**strong**` → `*em*` → `^### `→`<h4>` → `^## `→`<h3>` → `^# `→`<h3>` → `^\s*[-*] `→`<li>` → `<ul>` grouping → `^\d+\. `→`<li>`. Three verified gaps: **no link transform exists at all** (`renderMarkdown("[a](https://x)")` returns the text verbatim inside a `<p>`), **no image transform exists at all** (`![alt](src)` prints literally), and the heading ladder stops at `###` — `####`/`#####`/`######` match no rule and print as plain text (each rule requires a space after its hashes, so no rule partially eats a deeper level). `frontend/assets/styles.css` — the `:root` palette (`--ink`, `--ink-soft`, `--line`, `--surface`, `--bg`, the `--brand-*` family, `--radius*`) plus the precedent rule families: `.md-table`/`.md-table-wrap` (L539-557), `.md-quote` + the `.msg.user .bubble .md-quote` inversion (L561-588), `.source-image-img` / `.doc-image-img` (L981-1027, L5611-5640) for image treatment. `app/core/security_headers.py::CSP` (L67-68) — `default-src 'self'; base-uri 'none'; frame-ancestors 'none'; img-src 'self' data:`: served-path and `data:` images are ALREADY permitted, no CSP change is needed or allowed here. Test patterns: `tests/unit/test_markdown_blockquotes.py` (source pins + a `HAVE_NODE`-skipped node behaviour harness + a byte-identity fixture string) and `tests/e2e/test_markdown_blockquotes.py` (dedicated fixture tree `tests/fixtures/quote_docs/`, mock LLM trigger, chat + viewer + localStorage-restore assertions).
## Objective
`[text](url)` renders as a real, safe `<a>`, `![alt](src)` renders as a lazy `<img>`, and `####`–`######` render as `<h5>`/`<h6>` — in chat answers, the thinking block, saved/shared chats, the document viewer and the document modal, from the ONE shared renderer. Every input that contains none of the three constructs renders byte-identical to today, so the phase-44 table suite, the phase-138 blockquote suite and every renderer-dependent unit suite stay green unedited.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — one chain, one place (tasks 01–03).** All three fixes are edits to the same `inline(s)` chain in `frontend/assets/markdown.js`. Nothing else in the pipeline moves: fences (step 1) and tables (step 1b) keep their precedence, and because a table cell goes through `inline(escapeHtml(cell))`, links/images/deep headings inside a cell work for free (pinned in task 05).
- **D2 — escape-first is the security boundary, so NEVER unescape.** `inline()` always receives HTML-escaped text. The captured URL may therefore contain `&amp;`/`&#39;`/`&quot;` — that is *valid* in an `href`/`src` and must be passed through verbatim (never decoded, never re-escaped). Any attempt to decode would re-open the injection surface the renderer exists to close.
- **D3 — links (task 01).** `/\[([^\]]*)\]\(([^)\s]+)\)/g` (empty link text allowed, no whitespace inside the URL), applied AFTER the `*em*` rule and BEFORE the heading ladder. The URL is validated by an allowlist *function*, not a regex-in-replacement: allow `^https?://`, `^mailto:`, `^/`, `^#`, `^\.{1,2}/`; anything else (`javascript:`, `data:`, `vbscript:`, `file:`, bare `javascript&colon;` variants) returns the ORIGINAL matched text unchanged — it stays visible as inert escaped text, never a live element. Absolute `http(s)://` links get `target="_blank" rel="noopener noreferrer"`; `mailto:`, root-relative and `#fragment` links get neither (a same-origin target in the same tab is the correct behaviour). Output marker: `<a class="md-link" …>`.
- **D4 — images (task 02), matched BEFORE the link rule.** `/!\[([^\]]*)\]\(([^)\s]+)\)/g` → `<img class="md-img" src="…" alt="…" loading="lazy">`. The image rule runs FIRST so the `!` is never orphaned into `!<a>`. Same allowlist discipline: `/`-rooted served paths, `http(s)://` and `data:image/*` render; anything else stays inert text. `alt` is ALWAYS emitted (empty allowed) — an `<img>` without `alt` fails WCAG. Linked images (`[![alt](src)](url)`) get their OWN rule ahead of both (locked A6): `<a class="md-link" …><img class="md-img" …></a>` — otherwise the image rule leaves a stray `[` inside the link text.
- **D5 — the heading ladder (task 03, locked A4).** Insert `/^#### (.*)$/gm → <h5>$1</h5>` and `/^#{5,6} (.*)$/gm → <h6>$1</h6>` immediately BEFORE the existing `/^### /` rule (deeper-first is the readable order; the rules cannot cross-match because every rule requires a space after its hashes). `#` and `##` keep mapping to `<h3>` — existing behaviour, deliberately unchanged. The step-3 guard already matches `h\d`, so no paragraph-guard change is needed (pinned, not assumed).
- **D6 — CSS (task 04).** `.md-link` (underlined; `color: var(--brand-ink)` — 9.0:1 on `--surface`, the file's documented AA-safe brand text token — with `--brand-hover` on `:hover`, the shared 3px `--brand` `:focus-visible` ring already used at L163, and the `.msg.user .bubble .md-link { color: var(--bg) }` inversion following the `.md-quote` precedent at L588, 5.2:1 on the brand bubble), `.md-img` (`max-width:100%; height:auto; object-fit:contain` + the `.source-image-img` border/radius/surface language), and the `.doc-md` / `.bubble` / `.thinking-text` `h5`/`h6` type scale. Palette variables ONLY, no new hue (the monochrome invariant), every non-obvious color carrying its `/* n.n:1 on --x (AA) */` comment.
- **D7 — byte-identity is the regression contract (task 05).** A checked-in fixture string with no links, no image refs and no `####`+ heading, plus its byte-exact pre-phase renderer output (the phase-138 `NO_QUOTE_INPUT`/`NO_QUOTE_PRE_PHASE_OUTPUT` shape), pinned in the node behaviour harness.
- **Known, documented limits (recorded in the module docstring, not fixed):** inline code spans are NOT placeholder-protected (pre-existing: `**bold**` inside backticks already transforms), so markdown syntax inside an inline code span transforms too; a URL containing a literal `)` or whitespace is a miss; a size-less SVG (`viewBox` only, no `width`/`height`) in `.md-img` paints at the replaced-element default box — the vector sizing contract belongs to phase 181.
- **NOT touched:** `app/**` (no server change in this phase), the CSP string, `document.js`'s `renderDocument` structure, `app.js`'s chip/figure builders, `styles.css` families outside `.md-*` + the `h5`/`h6` scale.
## Dependencies
- `179_git_clone_server_freeze` (todo) — numbering/pipeline order only. The code dependency is the shared renderer at the head of `complete/` (phases 44 tables, 138 blockquotes).
## Tasks
1. `01_inline_links.md` — the `[text](url)` → `<a class="md-link">` transform with the protocol allowlist in `markdown.js`.
2. `02_inline_images.md` — the `![alt](src)` → `<img class="md-img">` transform (ahead of the link rule) + the linked-image rule.
3. `03_heading_ladder.md` — `####` → `<h5>`, `#####`/`######` → `<h6>` in the ladder.
4. `04_styles_markdown_inline.md` — `.md-link`, `.md-img` and the `h5`/`h6` type scale in `styles.css`.
5. `05_unit_and_e2e.md` — `tests/unit/test_markdown_inline_links_images.py` + the phase's dedicated Playwright suite `tests/e2e/test_markdown_inline_links_images.py`, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_markdown_inline_links_images.py` (new, the phase-138 pattern): source pins (the chain order code → strong → em → **image → linked-image → link** → headings → lists; the allowlist function; the exact `md-link`/`md-img` markers; no `unescape`/`decodeURIComponent` anywhere in the file; the `.md-link`/`.md-img` CSS rules reference palette vars only) + node behaviour (link renders with `target`/`rel`; `javascript:`/`data:` inert; relative + `#fragment` + `mailto:` link without `target`; `&amp;` in a query string survives verbatim; image renders with `alt` (empty alt allowed), disallowed src inert; linked image wraps without a stray `[`; `####`→`<h5>`, `#####`/`######`→`<h6>`, `###`→`<h4>` unchanged; a link inside a fence stays code; a link/image inside a table cell renders; byte-identity fixture).
- **E2E (AGENTS rule 4):** `tests/e2e/test_markdown_inline_links_images.py`, run **in isolation** (`uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov`, DB up).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — this phase adds no `app/` code, so the full suite staying green IS the gate.
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `[text](url)` renders as `<a class="md-link">` in a chat answer, the thinking block, a restored shared chat, the viewer and the document modal; `javascript:`/`data:` URLs stay visible as inert text
- [ ] `![alt](src)` renders as `<img class="md-img" loading="lazy">` with an `alt` present; a non-allowlisted src stays inert text
- [ ] `####` renders `<h5>`, `#####`/`######` render `<h6>`; `#`/`##`/`###` are unchanged
- [ ] a link/image/`####` inside a fenced code block is not transformed; a cell-level link renders in a `.md-table`
- [ ] an input with none of the three constructs renders byte-identical (phase-44 + phase-138 suites pass unedited)
- [ ] `uv run pytest` green; coverage TOTAL >90%; the phase's E2E green in isolation; `uv run ruff check . && uv run pyright` clean
- [ ] one `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate
## Locked decisions
- **A4 — `####` → `<h5>`; `#####` and `######` both → `<h6>`; `#`/`##` keep rendering as `<h3>` (owner-confirmed 2026-10-14, roadmap confirmation).**
- **A5 — link/image URL allowlist = `http:`, `https:`, `mailto:` (links), `/`-rooted + `./`/`../` relative paths, `#fragment`, and `data:image/*` (images). Everything else renders as inert text, never a live element (owner-confirmed 2026-10-14, roadmap confirmation).**
- **A6 — `[![alt](src)](url)` renders as a link wrapping the image (an explicit rule ahead of the two others), not as text with a stray bracket (assistant decision, surfaced here so the executor does not re-decide it).**
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(ui): render markdown links, inline images, and h5/h6 headings in the shared renderer"
```
@@ -0,0 +1,27 @@
# Task 01 — The link transform in the shared renderer
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:4` — "Markdown [links](some-address) aren't being render correctly and show up as plain text"
**Story:** n/a
## Objective
`renderMarkdown` turns `[text](url)` into `<a class="md-link" …>text</a>` in the shared `inline()` chain — through an explicit URL allowlist, escape-first, with every existing transform's output byte-identical.
## Work
1. `frontend/assets/markdown.js` — in `const inline = (s) => …`, insert one link `.replace()` **after** the `*em*` rule and **before** the `^### ` heading rule: `/(\[)([^\]]*)(\]\()([^)\s]+)(\))/g` written plainly as `/\[([^\]]*)\]\(([^)\s]+)\)/g` (empty link text allowed, no whitespace in the URL). The replacement is a FUNCTION, not a string, so the URL passes the allowlist check:
- `allowLinkUrl(url)` — returns the url for `^https?://`, `^mailto:`, `/`-rooted (`^/`), `^\.{1,2}/` relative, and `^#` fragment URLs; returns `null` for anything else (`javascript:`, `data:`, `vbscript:`, `file:`, `blob:`) — LOCKED A5;
- allowed → `<a class="md-link" href="<url>"[ target="_blank" rel="noopener noreferrer"]>text</a>`; the `target`/`rel` pair is added ONLY for `^https?://` (a `mailto:`/relative/fragment link stays in the same tab);
- not allowed → return the ORIGINAL matched substring untouched, so it renders as the inert escaped text it is today (never a live element, never dropped).
2. `frontend/assets/markdown.js` — pass the captured URL through VERBATIM: the chain's input is already `escapeHtml`-ed, so `&amp;`/`&#39;`/`&quot;` in a query string are already attribute-safe. No `decodeURIComponent`, no manual unescape, no re-escape (design D2 — this is the security boundary).
3. `frontend/assets/markdown.js` — module docstring: add the phase-180 paragraph (the chain order is now code → strong → em → **linked-image → image → link** (tasks 02 adds the first two ahead of this one) → headings → lists), the allowlist (LOCKED A5), the `target`/`rel` rule, and the documented limits (a URL containing a literal `)` or a space is a miss; inline code spans are not placeholder-protected, so markdown syntax inside `` `…` `` transforms — the pre-existing `**strong**` behaviour, unchanged here).
- ASSUMPTION: link text may itself contain inline markup (`**bold**`, `*em*`, `` `code` ``) — those rules run FIRST in the chain, so the text is already transformed when the link rule wraps it; no extra handling.
- ASSUMPTION: autolinks (`<https://x>`) and reference-style links (`[text][ref]`) stay out of scope — the TODO names the inline `[text](url)` form only.
## Testing & Quality
- Unit: pinned by task 05's file (source pins: the link rule sits between the `em` and heading rules, the marker is exactly `<a class="md-link"`, the file contains no `decodeURIComponent`/`unescape`; node behaviour: absolute link with `target="_blank" rel="noopener noreferrer"`, `mailto:`/relative/`#` link without them, `javascript:` + `data:` inert, `?a=1&b=2` keeping `&amp;`, an empty link text, a link inside a table cell).
- Coverage: n/a (frontend) — the suite + the pins are the gate.
## Completion Criteria
- [ ] node repro: `renderMarkdown("see [the docs](https://example.com/a?x=1&y=2)")` → a `<p>` containing `<a class="md-link" href="https://example.com/a?x=1&amp;y=2" target="_blank" rel="noopener noreferrer">the docs</a>`
- [ ] node repro: `renderMarkdown("[click](javascript:alert(1))")` contains no `<a` and no `href`
- [ ] every other existing construct (`strong`, `em`, `code`, `#`–`###`, lists, tables, fences, blockquotes) renders byte-identical to the pre-task renderer
- [ ] `uv run pytest` green (the renderer-dependent unit suites unedited); `uv run ruff check .` clean
@@ -0,0 +1,27 @@
# Task 02 — The image transform (and the linked-image rule) in the shared renderer
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … Also would be nice to click an svg and see a preview in the RAG tab rather than the xml" (the in-answer rendering half; the document-preview half is phase `181_svg_document_preview`)
**Story:** n/a
## Objective
`renderMarkdown` turns `![alt](src)` into a lazy `<img class="md-img">` (and `[![alt](src)](url)` into that image wrapped in a link), through the same URL allowlist as task 01 — so a diagram referenced inside markdown paints instead of printing its reference text.
## Work
1. `frontend/assets/markdown.js` — in the `inline()` chain, insert TWO rules **ahead of** task 01's link rule, in this order:
1. **linked image first:** `/\[!\[([^\]]]*)\]\(([^)\s]+)\)\]\(([^)\s]+)\)/g` → `<a class="md-link" href="url"[ target="_blank" rel="noopener noreferrer"]><img class="md-img" src="src" alt="alt" loading="lazy"></a>` (LOCKED A6 — without this rule the image rule consumes the inner `![…](…)` and the link rule then leaves a stray `[` in the link text). Both URLs are allowlist-checked (`allowLinkUrl` for the href, `allowImageUrl` for the src); either failing → the original text is returned untouched.
2. **image second:** `/!\[([^\]]*)\]\(([^)\s]+)\)/g` → `<img class="md-img" src="src" alt="alt" loading="lazy">`.
2. `frontend/assets/markdown.js` — add `allowImageUrl(url)` next to task 01's `allowLinkUrl`: allows `/`-rooted served paths (`/api/documents/<id>/image`, `/assets/…`), `^.{0,2}/` relative paths, `^https?://` and `^data:image/` — the `data:` case exists because the app's own CSP (`app/core/security_headers.py::CSP`, `img-src 'self' data:`) already permits `data:` images; everything else (`javascript:`, `file:`, non-image `data:`) returns `null` → the original escaped text stays visible and inert (LOCKED A5).
3. `frontend/assets/markdown.js` — `alt` is ALWAYS emitted, escaped-passthrough like the URL, empty allowed (`alt=""` for a decorative image) — an `<img>` with no `alt` attribute fails WCAG 1.1.
4. `frontend/assets/markdown.js` — docstring: record the final chain order (code → strong → em → **linked-image → image → link** → headings → lists), the image allowlist, and the documented limit that a size-less SVG (`viewBox` only) in `.md-img` paints at the replaced-element default box (vector sizing is phase 181's `04_svg_preview_render.md` contract).
- ASSUMPTION: no `onerror` handler is added — the renderer emits an HTML STRING into `innerHTML`-style surfaces and the no-CDN/CSP posture forbids inline handlers; a broken src degrades to the browser's alt text (the house pattern for the shared renderer). The JS `onerror` fallbacks live in `document.js`/`app.js`, which do not route through this function.
- ASSUMPTION: title syntax (`![alt](src "title")`) is a miss (the URL capture forbids whitespace) — out of scope.
## Testing & Quality
- Unit: pinned by task 05 (chain order pins the two image rules before the link rule; markers `<img class="md-img"` / `loading="lazy"`; `alt=` always present; node behaviour: served-path image, `data:image/*` image, disallowed-protocol image inert, linked image produces `<a class="md-link">` wrapping `<img class="md-img">` with no stray bracket, empty alt allowed, an image inside a table cell renders).
- Coverage: n/a (frontend).
## Completion Criteria
- [ ] node repro: `renderMarkdown('![topology](/api/documents/aaaa/image)')` → `<p><img class="md-img" src="/api/documents/aaaa/image" alt="topology" loading="lazy"></p>`
- [ ] node repro: `renderMarkdown('[![d](/a.svg)](https://example.com)')` → one `<a class="md-link"` containing one `<img class="md-img"`, and the output contains no stray `[` or `]`
- [ ] node repro: `renderMarkdown('![x](javascript:alert(1))')` contains no `<img` and no `src=`
- [ ] input with no image reference renders byte-identical; `uv run pytest` green
@@ -0,0 +1,27 @@
# Task 03 — Extend the heading ladder to `####`, `#####`, `######`
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:5` — "Quad-level headers (####) aren't being rendered and show up as plain text"
**Story:** n/a
## Objective
The ladder renders `#### x` as `<h5>x</h5>` and `##### x` / `###### x` as `<h6>x</h6>`; `#`, `##` and `###` keep their existing output exactly.
## Work
1. `frontend/assets/markdown.js` — in the `inline()` chain, insert TWO rules immediately BEFORE the existing `/^### (.*)$/gm → "<h4>$1</h4>"` rule (deeper-first, so the ladder reads top-down by depth):
- `/^#### (.*)$/gm` → `"<h5>$1</h5>"`
- `/^#{5,6} (.*)$/gm` → `"<h6>$1</h6>"` (LOCKED A4: both `#####` and `######` map to `<h6>` — the app's type scale stops at two levels below `###`, no `<h7>` exists in HTML)
The rules cannot cross-match each other or the existing ones because every heading rule requires a space after its hash run (`####` never matches `/^### /`).
2. `frontend/assets/markdown.js` — leave `/^## /` and `/^# /` mapping to `<h3>` UNTOUCHED (LOCKED A4 — the existing flattening of `#`/`##` to `<h3>` is deliberate; the phase adds levels below `###`, it does not re-rank the existing ones).
3. `frontend/assets/markdown.js` — the step-3 paragraph guard `/^<(h\d|ul|ol|pre|li|blockquote)/` already matches `h5`/`h6`, so NO change is needed there: pin it with a unit assertion rather than editing it.
4. `frontend/assets/markdown.js` — docstring: record the ladder (`#`/`##` → `h3`, `###` → `h4`, `####` → `h5`, `#####`/`######` → `h6`) so the deliberate flattening is documented, not re-discovered.
- ASSUMPTION: a heading line must start at column 0 of its line (the `m` flag, the existing rule shape) — an indented `#### x` stays plain text, matching `^### ` today.
## Testing & Quality
- Unit: pinned by task 05 (source pins: the two new rules sit before `/^### /`; the guard regex unchanged; node behaviour: `#### A` → `<h5>A</h5>`, `##### A`/`###### A` → `<h6>A</h6>`, `### A` → `<h4>A</h4>` unchanged, `## A`/`# A` → `<h3>` unchanged, a `####` line inside a fence stays code, a `####` heading immediately followed by a paragraph is not wrapped in `<p>`).
- Coverage: n/a (frontend).
## Completion Criteria
- [ ] node repro: `renderMarkdown("#### Deep")` → `<h5>Deep</h5>` (no `<p>` wrapper, no literal `####`)
- [ ] node repro: `renderMarkdown("##### Deeper\n###### Deepest")` → two `<h6>` elements
- [ ] node repro: a `####` line inside a ``` fence renders inside `<pre><code>` unchanged
- [ ] existing heading output byte-identical; `uv run pytest` green (phase-44/138 suites unedited)
@@ -0,0 +1,26 @@
# Task 04 — Styling for `.md-link`, `.md-img` and the `h5`/`h6` type scale
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:4–5` (link + heading rendering), `TODO.md:3` (inline image rendering)
**Story:** n/a
## Objective
The three new elements are legible and theme-correct in every renderer context (chat bubble — brain and user, the thinking block, the shared page, the viewer, the document modal), built from the existing `:root` palette only.
## Work
1. `frontend/assets/styles.css` — `.md-link` next to the `.md-quote` family (L561-588): `color: var(--brand-ink)` (the file's documented AA-safe brand text token, 9.0:1 on `--surface`), `text-decoration: underline` (a link must not rely on colour alone — WCAG 1.4.1), `text-underline-offset: 2px`, `:hover { color: var(--brand-hover) }`. The global `:focus-visible { outline: 3px solid var(--brand) }` (L162-166) already covers the link — do NOT add a competing focus rule, and do not reset the outline.
2. `frontend/assets/styles.css` — `.msg.user .bubble .md-link { color: var(--bg); }` — the exact inversion the `.md-quote` precedent applies at L588 (the user bubble is a `--brand` fill: `--bg` on `--brand` is 5.2:1, AA). Add the same inversion for `:hover` if the default `--brand-hover` fails contrast on the brand fill — a colour-mix toward `--bg` is acceptable, a new hue is not.
3. `frontend/assets/styles.css` — `.md-img`: `display: inline-block; max-width: 100%; height: auto; object-fit: contain;` plus the `.source-image-img` card language (`background: var(--surface)`, `border: 1px solid var(--line)`, `border-radius: var(--radius-sm)`) and `margin: 0.4rem 0`. No fixed width/height — an oversized screenshot must never widen the page.
4. `frontend/assets/styles.css` — the deep-heading type scale, ONE rule set for all renderer contexts (no chat-specific duplication): `.doc-md h5, .msg .bubble h5, .thinking-text h5 { margin: 0.8rem 0 0.3rem; font-size: 0.95rem; }` and the `h6` sibling at `0.88rem`, following the existing `.doc-md h3` / `.doc-md h4` rhythm (L5282-5283) — the scale stays monotonic (`h3` 1.15 → `h4` 1.0 → `h5` 0.95 → `h6` 0.88) and every level keeps a visible size difference from body text.
5. `frontend/assets/styles.css` — comment block above each rule in the house style: the phase number, the TODO source line, the contrast math for every non-obvious colour, and the "static content, no animation" note (the background-motion invariant).
- ASSUMPTION: `h5`/`h6` in the chat bubble get explicit sizing (they inherit UA defaults today, which are larger than `0.95rem`) so a deep heading in an answer cannot out-shout the answer text.
- ASSUMPTION: no `.md-img--vector` modifier here — vector (viewBox-only SVG) sizing belongs to `181_svg_document_preview/04_svg_preview_render.md`, which knows the document's format.
## Testing & Quality
- Unit: pinned by task 05 — the `.md-link`, `.md-img`, `h5`/`h6` rules exist and reference ONLY `var(--…)` palette tokens (no hex literals, no `rgb(`) — the established `:root`-drift pin pattern used by the blockquote/table CSS tests; the `.msg.user .bubble .md-link` inversion is present.
- Coverage: n/a (frontend).
## Completion Criteria
- [ ] a link in a brain bubble, a user bubble, the thinking block, the shared page, the viewer and the document modal is underlined and passes ≥4.5:1 against its surface
- [ ] Tab focus on a link shows the app's 3px `--brand` ring
- [ ] a wide image in an answer does not produce a horizontal scrollbar on the chat page at 375px and at 1440px
- [ ] `uv run pytest` green (the CSS-pinning unit suites unedited)
@@ -0,0 +1,36 @@
# Task 05 — Unit pins, the dedicated Playwright suite, the gates, the commit
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:3–5` (the acceptance evidence for tasks 01–04)
**Story:** n/a
## Objective
Lock the three renderer fixes with a source-pin + node-behaviour unit file and the phase's own Playwright E2E suite (AGENTS rules 4/9), run every gate, and land the single atomic commit.
## Work
1. `tests/unit/test_markdown_inline_links_images.py` (new — the `tests/unit/test_markdown_blockquotes.py` shape, copy its `HAVE_NODE = shutil.which("node") is not None` skip and its `subprocess` node harness):
- **source pins on `frontend/assets/markdown.js`:** the chain order is code → strong → em → linked-image → image → link → `^#### ` → `^#{5,6} ` → `^### ` → `^## ` → `^# ` → lists (assert the index of each rule's literal in the file is strictly increasing); the exact markers `<a class="md-link"`, `<img class="md-img"`, `loading="lazy"`, `<h5>`, `<h6>`; `allowLinkUrl` / `allowImageUrl` named functions exist; the file contains NO `decodeURIComponent` / `unescape` / `innerHTML`; the step-3 guard regex is still exactly `/^<(h\d|ul|ol|pre|li|blockquote)/`.
- **CSS pins on `frontend/assets/styles.css`:** `.md-link`, `.md-img`, the `h5` and `h6` rules exist and their colour values reference `var(--…)` only (no hex, no `rgb(`); `.msg.user .bubble .md-link` exists (the inversion).
- **node behaviour (skipped without node):** every repro listed in tasks 01–04's Completion Criteria, PLUS the byte-identity contract — a checked-in `NO_INLINE_INPUT` / `NO_INLINE_PRE_PHASE_OUTPUT` fixture pair (a paragraph + a list + a pipe table + a fence + a blockquote, i.e. everything the renderer already supported) asserted byte-equal.
2. `tests/fixtures/links_docs/` (new dedicated fixture tree — the phase-138 `quote_docs` precedent: a NEW tree, never the shared 13-doc fixture set, whose import count many suites pin). One document `homelab/inline-markdown.md` whose body carries: a `[absolute link](https://example.com/a?x=1&y=2)`, a `[relative link](/sources)`, a `mailto:` link, a `#anchor`, an inert `[oops](javascript:alert(1))` probe, `![topology](data:image/svg+xml;base64,…)` (a tiny inline SVG — CSP `img-src 'self' data:` already allows it, no bytes fixture needed), an inert `![bad](javascript:alert(1))`, a `[![linked](data:…)](https://example.com)`, and an `#### Deep header` + `##### Deeper` line, plus a pipe table with a `[cell link](https://example.com)` inside a cell.
3. `tests/e2e/test_markdown_inline_links_images.py` (new — the phase-138 E2E skeleton verbatim in structure: module-level fixture constants, `_import_fixtures` + `_run_in_thread` + `_reset_db` (truncate `chunks, documents, query_log, steering_notes, kb_overview` then re-import, `summary.added == 1`), `login(page, app_url, next="/")`, the mock LLM trigger answering with the byte-stable inline-markdown answer):
- `test_chat_inline_markdown_renders` — the brain bubble contains `a.md-link` elements for the absolute (with `target="_blank"`, `rel` containing `noopener noreferrer`, `href` preserving `&amp;`), relative and `mailto:` links; NO `a.md-link` for the `javascript:` probe (its text stays visible); `img.md-img` for the data-URL image with a non-empty `alt` and `loading="lazy"`; the linked image is an `a.md-link > img.md-img` with no stray bracket in its text; `h5`/`h6` elements exist; `table.md-table a.md-link` exists (cell-level transform); the turn is NOT deflected.
- `test_viewer_inline_markdown_renders` — the fixture document opens from the Sources page in the document modal (admin, `openDocumentModal`) and renders the same `a.md-link` / `img.md-img` / `h5` / `h6` shapes (the shared renderer contract across surfaces).
- `test_restore_keeps_inline_markdown` — reload the chat page: the localStorage restore path re-runs `renderMarkdown` and the restored brain bubble keeps its links, image and headings.
- `test_no_horizontal_overflow_at_mobile` — at `viewport={"width": 375, "height": 720}`, `document.documentElement.scrollWidth <= clientWidth` with the wide image rendered (the `.md-img` `max-width: 100%` contract).
4. Run the gates, in order, and fix what fails (do not baseline, do not skip):
- `uv run pytest`
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL >90%
- `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov` (in isolation, DB up: `podman compose up -d db`)
- `uv run ruff check . && uv run pyright`
5. Move the phase directory to `.agents/phases/complete/` and commit — see the overview's Commit block.
## Testing & Quality
- This task IS the phase's test gate. The unit file must fail if any of tasks 01–03's rules are reordered or removed, and must fail if task 04's CSS introduces a hard-coded colour.
- Coverage: **>90%** on `app/` (no `app/` code changes in this phase — the number must not move).
## Completion Criteria
- [ ] `uv run pytest` green, including the new unit file (and green with node absent — the skip path)
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov` green in isolation
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] one atomic commit, `--no-gpg-sign`: `feat(ui): render markdown links, inline images, and h5/h6 headings in the shared renderer`, containing `frontend/`, `tests/` and `.agents/`
@@ -0,0 +1,56 @@
# Phase 181 — SVG documents render as pictures, not as XML
**Source:** `TODO.md` L3 — "Need SVG render support on the front end when citing and svg source. Also would be nice to click an svg and see a preview in the RAG tab rather than the xml"
**Story:** n/a (TODO-derived phase, Protocol B append — owner roadmap confirmation 2026-10-14)
**Context:** **The root cause is ingestion, not the viewer.** `app/config.py::image_extensions` (L436) defaults to `png,jpg,jpeg,webp,gif,bmp` and `.env` mirrors it (`BOR_IMAGE_EXTENSIONS`), so an `.svg` can only enter the KB through `BOR_IMPORT_EXTENSIONS` — i.e. as a **text** document whose `content` is the raw XML, which `frontend/assets/document.js::renderDocument` then paints in `<pre class="doc-raw">`. That is exactly the "xml instead of a preview" the owner sees. The rest of the image machinery is already there and needs no change: `app/rag/retriever.py::source_ref_with_image` (L1010-1022) puts `image_url = /api/documents/<id>/image` on a cited image doc, `app/schemas.py::SourceRef.image_url` (L154-181) carries it (omitted when `None`), `app/api/docs.py::get_document_content` (L219-318) returns `is_image` + `image_url` + `format`, `app/api/docs.py::get_document_image` (L319-378) serves the stored bytes with `IMAGE_MIMES.get(suffix, IMAGE_FALLBACK_MIME)`, `frontend/assets/app.js::appendSources` (L2386+) renders a `.source-image` figure per `image_url` ref, `frontend/assets/document.js` renders the `.doc-image` `<img>` block for `doc.is_image` (L453-470, L560+), `frontend/assets/sources.js` (L2328) renders the KB-tree thumbnail. `app/rag/importer.py::_index_file` already prefers the image branch over the text path (L1417-1425) and `_index_image_file` (L3532+) owns bytes-digest → `_markdown_image_context` (phase 125) → `_describe_or_skip` → copy to `image_dir/<doc-id><ext>` → `is_image`/`image_path`/`content`/`summary`. `app/rag/summarizer.py::IMAGE_MIMES` (L152-159, dotted keys) is the one map for both the vision data-URL mime and the route's `Content-Type`; `describe_image` is a CHAT-model vision call (phase 122 LOCKED A3). `app/core/security_headers.py::CSP` (L67-68) + its middleware (which PRESERVES a CSP an inner layer set — L109-110) is the injection boundary. `app/rag/md_images.py::extract_image_refs` (the phase-125 `![alt](path)` scanner) already treats an `.svg` path like any other image. Test patterns: `tests/unit/test_image_documents.py` (toggle / ingest / describe-seam), `tests/integration/test_docs_api.py` (the image route), `tests/e2e/test_image_documents.py` + `tests/e2e/test_answer_images.py` (image E2E), `tests/unit/test_markdown_blockquotes.py` (source-pin + node harness).
## Objective
An `.svg` in a source tree becomes an **image document**: its own text (`<title>`/`<desc>`/`<text>`) is what gets indexed and retrieved, its bytes are served as `image/svg+xml` under a sandboxed CSP, and every citation surface — the chat figure, the document modal, the `/document.html` viewer, the Sources thumbnail — paints the rendered vector graphic instead of XML. Plus the phase-180 renderer, an SVG referenced as `![alt](/api/documents/<id>/image)` inside an answer or a document renders as a real image.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — SVG is a *vector* image document; the vision call is bypassed (tasks 01–02, locked A1).** OpenAI-compatible vision endpoints do not accept `image/svg+xml`, and adding a rasteriser dependency is out of scope. So an `.svg` takes the phase-122 image branch but its indexed text comes from the file itself: a new pure module `app/rag/svg_text.py` (the `app/rag/md_images.py` precedent — stdlib only, no I/O, fully unit-testable) returns the SVG's own words. `content`/`summary` follow the phase-122 image convention (`summary = content`) and the normal chunk pipeline embeds them; the phase-125 `image_context` (the `![alt](path)` paragraphs around a reference to the file) still prepends, unchanged.
- **D2 — the text of a diagram is its `<title>`, `<desc>` and `<text>` (task 01).** Matched by LOCAL name (namespace-agnostic: plain `<title>`, `svg:title`, and namespaced variants all count), in DOCUMENT order, whitespace-normalised, de-duplicated case-insensitively, joined with `\n`. Parsing is `xml.etree.ElementTree` — never a regex over untrusted markup — guarded against the billion-laughs/XE class by (i) refusing any input containing a `<!DOCTYPE` or `<!ENTITY` declaration (→ extraction failure, no parse attempt) and (ii) a `MAX_SVG_BYTES = 2 MiB` input cap (over-cap → extraction failure). XML entity refs inside text (`&amp;`) are decoded by the parser, which is the correct direction (the value is re-escaped on the way out by the renderer).
- **D3 — a textless SVG still gets honest, retrievable text (task 01, locked A2).** Many real diagrams carry no words at all (pure `<path>`s). Skipping them (the phase-122 "undescribed image is unsearchable noise" posture) would make them invisible and never citable — the opposite of what the owner asked. So `extract_svg_text` falls back to a structural line built ONLY from facts in the file: `Vector diagram "<file stem>" — <N> shapes (<the three most common element local names with counts>), viewBox <w>×<h>` (each clause omitted when the file does not state it, the stem always present). No invented prose, no model call.
- **D4 — ingestion mechanics (task 02).** `image_extensions` default gains `svg` (`.env.example` documents it; a live `.env` keeps its own value — the phase does not edit `.env`). `app/rag/importer.py::_index_image_file` branches on `full_path.suffix.lower() == ".svg"`: vector files take `extract_svg_text` INSTEAD of `_describe_or_skip`; everything else (copy to `image_dir`, digest rules, `is_image`/`image_path`, title = file stem, counters, the `on_chunk` hook) is unchanged and shared. Fail-soft stays LOCKED A3-shaped: extraction returning `None` (over-cap, DOCTYPE-bearing, or unparseable XML) → `summary.images_failed += 1`, a warning, no row, no orphan copy. **One extra guard is required for existing libraries:** `_index_file`'s fast path (L1403, `doc.file_hash == file_digest`) returns BEFORE the image branch, so a row that was indexed as TEXT when `svg` lived only in `BOR_IMPORT_EXTENSIONS` would stay a text doc forever even after the switch. Add a narrow mismatch guard — take the fast path only when the file's kind matches the row's (`settings.images` ∧ the extension is in `image_extension_set` ∧ `doc.is_image is False` ⇒ fall through to the image branch). Everything else keeps the phase-147 fast path byte-for-byte.
- **D5 — serving the bytes safely (task 03).** `IMAGE_MIMES[".svg"] = "image/svg+xml"` in `app/rag/summarizer.py` (one map, one truth — the vector branch never reaches `image_data_url`, so the vision path is unaffected). `get_document_image` keeps its public, id-as-credential posture (phase 124 LOCKED A4 — a shared chat shows the figure to a guest) and its `Cache-Control: private, max-age=3600`, and adds ONE response header for `image/svg+xml` only: `Content-Security-Policy: default-src 'none'; sandbox` (the `SecurityHeadersMiddleware` preserves an inner CSP, so the route's value wins) — an `<img>` can never execute SVG script, but a user who opens the URL directly must not get a script-capable same-origin document either. `X-Content-Type-Options: nosniff` is already global.
- **D6 — the frontend (task 04).** The modal, the viewer, the citation figure and the Sources thumbnail already key off `is_image`/`image_url`, so they get the SVG for free — the task is the sizing + honesty work: `.doc-image-img--vector` (applied in `document.js` where `doc.format === "svg"`) with `width: 100%` so a `viewBox`-only SVG (no intrinsic size, the CSS default replaced-element box is 300×150) fills the `.doc-image` card instead of sitting small in it; the onerror `.doc-image-unavailable` fallback stays as-is; a legacy SVG row still stored as TEXT keeps rendering its XML in `.doc-raw` (honest — there are no bytes behind it) until the next sync converts it.
- **D7 — the chat figure is a 96px-capped thumbnail** (`.source-image-img`, `max-height: 96px; max-width: 100%; height: auto`), so a size-less SVG inside it stays inside the cap; `SourceRef` carries no format field and NONE is added (the wire surface stays byte-identical for every text doc — phase-83/122 contract). Clicking the figure/chip opens the modal, which is the full-size preview (`doc.format` known there). Documented in the module comments.
- **NOT touched:** the CSP constant in `security_headers.py`, `app/api/chat_images.py::_CHAT_IMAGE_FILENAME_RE` (question attachments are raster-only by design — a vector question image is out of scope), any rasteriser dependency, `describe_image`'s prompt, the phase-124 answer-image path, the `SourceRef` schema, `.env` (user-owned), the PDF embedded-image pass (`pdf.py` — PDFs never embed SVG for this app's extractor).
## Dependencies
- `180_markdown_inline_links_images` (todo) — its `![alt](src)` transform is what makes `![alt](/api/documents/<id>/image)` inside an answer or a document render; this phase supplies the route that answers that URL. No code-level dependency beyond that (the ingestion, the route and the viewer work without it).
## Tasks
1. `01_svg_text_extraction.md` — the new pure `app/rag/svg_text.py` module: `<title>`/`<desc>`/`<text>` extraction with the XE/size guards and the structural fallback.
2. `02_svg_index_as_image.md` — `svg` in `image_extensions`, the vector branch in `_index_image_file`, and the fast-path kind-mismatch guard that converts existing text-SVG rows.
3. `03_svg_bytes_route_csp.md` — `IMAGE_MIMES[".svg"]` and the sandboxed `image/svg+xml` response from `GET /api/documents/{id}/image`.
4. `04_svg_preview_render.md` — the vector sizing class + CSS in the viewer, and the figure/modal/thumbnail verification (the "preview instead of XML" contract).
5. `05_unit_integration_e2e.md` — `tests/unit/test_svg_text_extraction.py`, `tests/integration/test_svg_document_image.py`, the phase's dedicated Playwright suite `tests/e2e/test_svg_document_preview.py`, the gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_svg_text_extraction.py` (new, pure-function, no DB): title+desc+text in document order, namespace-prefixed variants, whitespace normalisation, de-duplication, `&amp;` decoded, the `<!DOCTYPE`/`<!ENTITY` refusal, the 2 MiB cap, malformed XML → `None`, the structural fallback for a textless SVG (stem + shape counts + viewBox, and each clause absent when unknown). Importer-level: the vector branch indexes an SVG with NO vision call (the `_describe_or_skip` seam patched to FAIL the test if called), `content == extracted text`, `summary == content`, the copy lands at `image_dir/<id>.svg`, `is_image=True`, the `images_failed` fail-soft path, and the fast-path conversion of a pre-existing text row (same bytes, `file_hash` unchanged).
- **Integration:** `tests/integration/test_svg_document_image.py` (new): `GET /api/documents/{id}/image` for an SVG row → `200`, `Content-Type: image/svg+xml`, the sandbox CSP header present, `Cache-Control: private, max-age=3600`; a raster row's response headers UNCHANGED (no CSP key added); the four 404 shapes (unknown id, unparseable id, text doc, missing file on disk) unchanged.
- **E2E (AGENTS rule 4):** `tests/e2e/test_svg_document_preview.py`, run **in isolation** (`uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov`, DB up).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — `app/rag/svg_text.py` is new pure code and must be covered near-completely; the importer/route deltas are covered by the tests above.
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] an `.svg` in a source tree imports as `is_image` with `image_path` set, no vision call made, and its indexed text drawn from its own `<title>`/`<desc>`/`<text>`
- [ ] `GET /api/documents/<id>/image` serves `image/svg+xml` with `Content-Security-Policy: default-src 'none'; sandbox`
- [ ] clicking an SVG citation chip/figure in the chat opens the modal showing a rendered `<img>` — no XML text anywhere in the modal body
- [ ] the `/document.html` viewer renders the SVG full-width in its `.doc-image` card (not the 300×150 default box) with the extracted text below it
- [ ] a `![alt](/api/documents/<id>/image)` reference inside a markdown document paints the graphic (the phase-180 renderer + this route — the E2E's two-stage seeding, `test_markdown_image_ref_renders`)
- [ ] every raster/PDF image behaviour is unchanged (the phase-122/123/124/133/140 suites pass unedited)
- [ ] `uv run pytest` green; coverage TOTAL >90%; the phase's E2E green in isolation; `uv run ruff check . && uv run pyright` clean
- [ ] one `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate
## Locked decisions
- **A1 — an SVG's indexed text is extracted from the file itself (`<title>`/`<desc>`/`<text>`), the vision describe call is bypassed for `.svg`; no rasteriser dependency is added (owner-confirmed 2026-10-14: "SVG as image document").**
- **A2 — a textless SVG is indexed with a structural metadata line built only from facts in the file (stem, shape counts, viewBox) rather than skipped, so it stays retrievable and previewable (assistant decision made explicit here; it relaxes the phase-122 fail-soft skip for the vector case only — a genuinely unparseable or over-cap file still skips).**
- **A3 — SVG bytes render ONLY through `<img>`, never inlined into the DOM, and the bytes route carries `default-src 'none'; sandbox` so a direct navigation cannot execute script (owner-confirmed 2026-10-14, roadmap confirmation).**
- **A7 — `svg` joins the `BOR_IMAGE_EXTENSIONS` DEFAULT; the `BOR_IMAGES` toggle remains the single knob (phase 122 LOCKED A3). With `BOR_IMAGES=0` an `.svg` stays a text document (raw XML in the viewer) and the walk never admits it as an image — no special-casing of the off state (owner-confirmed 2026-10-14, roadmap confirmation).**
## Commit
```bash
git add app/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(kb): index SVGs as image documents and preview them instead of raw XML"
```
@@ -0,0 +1,33 @@
# Task 01 — `app/rag/svg_text.py`: the words inside a vector drawing
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … source" (the indexed-text half: an SVG document must be retrievable without a vision call)
**Story:** n/a
## Objective
A new pure module turns SVG bytes into the text that makes the document searchable — its own `<title>`/`<desc>`/`<text>` words, or an honest structural line when it has none — with no third-party dependency, no I/O, and no network.
## Work
1. `app/rag/svg_text.py` (new — the `app/rag/md_images.py` module shape: module docstring citing this phase + the LOCKED decisions, `from __future__ import annotations`, stdlib only, full type hints, pure functions):
- `MAX_SVG_BYTES: int = 2 * 1024 * 1024` and `MAX_SVG_TEXT_CHARS: int = 4000` (module constants, documented — the cap binds on the EXTRACTED text, never silently mid-word: truncate at the last newline/period before the cap).
- `@dataclass(frozen=True) class SvgText:` — `text: str` (the searchable text) and `structural: bool` (`True` when the fallback line was used, so the importer can log/count it).
- `def extract_svg_text(data: bytes, *, stem: str = "") -> SvgText | None`.
2. `app/rag/svg_text.py` — the guards, BEFORE any parse (LOCKED A3's sibling: untrusted XML):
- empty/whitespace-only input → `None`;
- `len(data) > MAX_SVG_BYTES` → `None` (the billion-laughs/memory guard, alongside the next one);
- the lower-cased bytes containing `<!doctype` or `<!entity` → `None` (a document type declaration is refused outright, which is what makes stdlib `ElementTree` safe here — no external entities, no entity expansion);
- `ElementTree.fromstring` raising `ParseError` → `None` (never re-raise: the caller's fail-soft is a skip, not a crash).
3. `app/rag/svg_text.py` — extraction: a `local_name(tag)` helper that strips the `{namespace}` prefix, then walk `root.iter()` (document order) and collect, for local name `title` / `desc` → the element's `.text` block (`"".join(el.itertext())` so nested markup is carried), and for `text` → `" ".join(el.itertext())` (this is what covers `<tspan>` children). Normalise each item with `" ".join(item.split())`, drop empties, de-duplicate case-insensitively keeping the FIRST occurrence, cap at `MAX_SVG_TEXT_CHARS`, and return `SvgText("\n".join(items), structural=False)` when at least one item survived (parser-decoded entities like `&amp;` come out decoded — the renderer re-escapes on display).
4. `app/rag/svg_text.py` — the structural fallback (LOCKED A2), used only when the walk produced nothing: `Vector diagram "<stem>" — <N> shapes (<top-3 local names by count>), viewBox <w>×<h>` built ONLY from the file — `N` counts descendant elements excluding `svg`/`title`/`desc`; the `shapes` clause is omitted when `N == 0` and the noun pluralises normally (`1 shape`, `2 shapes`); the `viewBox` clause is omitted when the root has no parseable `viewBox` (four whitespace/comma-separated numbers; the last two are `w`/`h`) and when present reports `w×h` with integral values formatted without a trailing `.0`; the stem is `""` → the clause `Vector diagram` alone (no empty quotes). The em-dash form matches the app's own doc-comment typography and is the string the tests pin.
5. `app/rag/svg_text.py` — module docstring: why the vision call is bypassed (LOCKED A1: vision endpoints reject `image/svg+xml`; no rasteriser dependency), the guard list, the document-order/dedupe rules, and the fallback contract (LOCKED A2: facts from the file, never invented prose).
- ASSUMPTION: `<image>` elements embedding a raster inside the SVG are NOT followed (their `href` is not text; the byte-embedded raster belongs to the phase-151 PDF path, not here).
- ASSUMPTION: `aria-label`/`role` attributes are not extracted — `<title>`/`<desc>` are the SVG-spec carriers and the fallback covers the rest.
## Testing & Quality
- Unit: `tests/unit/test_svg_text_extraction.py` starts here (completed in task 05) — every branch above is a pure-function test with a literal SVG string, no DB, no LLM, no filesystem: document order across `title`/`desc`/`text`, `tspan` joining, namespace-prefixed tags (`<svg:title>`, `xlink:` attributes ignored), whitespace normalisation, case-insensitive dedupe, `&amp;` decoded, the `<!DOCTYPE`/`<!ENTITY` refusals (both a `<!DOCTYPE svg [<!ENTITY x "boom">]>` wrapper and a bare `<!ENTITY`), the over-cap refusal, malformed XML → `None`, and the fallback (shape counts + viewBox present/absent/malformed, empty stem).
- Coverage: **>90%** on this module (target: near-100% — it is pure).
## Completion Criteria
- [ ] `uv run python -c "from app.rag.svg_text import extract_svg_text; print(extract_svg_text(b'<svg viewBox=\"0 0 8 6\"><title>Top</title><text>a</text></svg>', stem=\"topo\"))"` reports the two extracted items in order with `structural=False`
- [ ] a `<svg>` with only `<path>`s yields `Vector diagram "topo" — 1 shape (path)`-shaped text with `structural=True`
- [ ] `extract_svg_text(b'<!DOCTYPE svg [<!ENTITY x "y">]><svg><title>&x;</title></svg>')` returns `None`
- [ ] `uv run pytest tests/unit/test_svg_text_extraction.py -v` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,38 @@
# Task 02 — SVG joins the image set and indexes as a vector document
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … source. Also would be nice to click an svg and see a preview … rather than the xml"
**Story:** n/a
## Objective
`svg` is an image extension, and `_index_image_file` indexes an `.svg` from its bytes + extracted text instead of a vision call — so the row arrives with `is_image=True`, `image_path` set and searchable content. A pre-existing text-indexed SVG row is converted on the next sync.
## Work
1. `app/config.py` (L436) — `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` → `"png,jpg,jpeg,webp,gif,bmp,svg"`; extend the field's doc-comment: SVG is the one VECTOR member (LOCKED A1 — no vision call, `app.rag.svg_text` supplies its text) and `BOR_IMAGES` remains the single knob (LOCKED A7). The existing `_image_extensions_known` validator (L659-673) needs no change (`svg` is a well-formed token); `image_extension_set` (L899-909) yields `.svg`.
2. `.env.example` (the `# BOR_IMAGE_EXTENSIONS=…` line ~L167) and `README.md` (L794 prose, the L1239 settings table row) — add `svg` to the documented default, one sentence on the vector behaviour (indexed from its own `<title>`/`<desc>`/`<text>`; no vision call) and one on the `BOR_IMAGES=0` consequence (LOCKED A7: an `.svg` stays a text doc — raw XML in the viewer). Never touch the live `.env` (user-owned).
3. `app/rag/importer.py::_index_image_file` (L3532+) — the vector branch, at the `_describe_or_skip` call site (L3632) ONLY:
```
if full_path.suffix.lower() == ".svg":
extracted = extract_svg_text(full_path.read_bytes(), stem=full_path.stem)
content = None if extracted is None else extracted.text
else:
content = await _describe_or_skip(...) # unchanged, byte-identical
```
Keep everything after it exactly as it is — the `images_failed += 1` + warning + `return` fail-soft when `content is None` (LOCKED A2: only a genuinely unusable file skips; the structural fallback means a textless SVG does NOT), the `image_context` prepend (`f"{image_context}\n\n{content}"`, phase 125 — unchanged), the digest over the raw bytes, the `image_dir/<doc-id>.svg` copy written only after success, `title = full_path.stem`, `is_image=True`, `image_path`, `summary = content`, the chunk pipeline and the `on_chunk` hook.
4. `app/rag/importer.py` — import `extract_svg_text` next to the existing `app.rag.summarizer` imports (L356-359) and update the module docstring's image paragraph: the image branch is raster (vision-described) OR vector (`svg`, text-extracted, no model call).
5. `app/rag/importer.py` — a log-only `svg_indexed: int = 0` counter on `ImportSummary` (docstring: log-only, the phase-133 stats convention — the sync `detail` API shape is untouched) incremented on a successful vector index, plus one `logger.info("import: svg indexed source=%s path=%s structural=%s chars=%d", …)` line so a run's SVG handling is auditable (AGENTS rule 10).
6. `app/rag/importer.py::_index_file` — the fast-path kind guard (design D4). At the `if doc is not None and doc.file_hash == file_digest:` fast path (L1403), take it only when the row's KIND already matches how the file would be indexed now: compute the would-be image membership once (`settings.images` ∧ `match_extension(full_path, settings.image_extension_set) is not None`) and require `doc.is_image` to agree. That converts a row imported as TEXT (before `svg` joined the image set) — same bytes, same `file_hash` — into the image branch instead of returning early, while every correctly-classified file keeps the phase-147 fast path byte-for-byte. Add the explanatory comment (this is the migration; no alembic migration is needed — `is_image` flips through the normal index write).
- The reverse direction (a raster row when `BOR_IMAGES` is off) must NOT be forced through the image path (the walk never admits it and the phase-122 prune guard protects it) — the guard only applies when `settings.images` is on.
- ASSUMPTION: no dedicated backfill script (the phase-118/138 backfill pattern) — the next scheduled sync converts every SVG through the guard in step 6, which is idempotent and needs no new entrypoint.
- ASSUMPTION: an SVG in BOTH `BOR_IMPORT_EXTENSIONS` and `BOR_IMAGE_EXTENSIONS` becomes an image document — `_index_file` already prefers the image branch (L1417-1425) over the text path, so no walk-precedence change is made.
## Testing & Quality
- Unit (`tests/unit/test_image_documents.py` extended, or a new `tests/unit/test_svg_image_ingest.py` if the file's scope reads better — the executor's call, stated in the commit body): the vector branch indexes an SVG while the `_describe_or_skip` seam is patched to raise (proving NO vision call), `content` equals the extracted text, `summary == content`, one `is_summary` chunk, `is_image=True`, `image_path` ends with `.svg` inside `image_dir`, `svg_indexed == 1`; extraction failure (over-cap bytes) → `images_failed == 1`, no row, no copy; the `image_context` enrichment prepends for an SVG exactly as for a raster; the fast-path guard — a row with `is_image=False` and a matching `file_hash` is RE-indexed as an image on the next run, while an already-correct row (raster `is_image=True`, same hash) still takes the fast path (assert `_describe_or_skip` is NOT called).
- Regression: `tests/unit/test_image_documents.py`, `tests/integration/test_importer.py`, `test_import_docs_git.py`, `test_importer_dates.py` stay green — no raster/PDF behaviour changes.
- Coverage: **>90%** on the changed lines.
## Completion Criteria
- [ ] `Settings().image_extension_set` contains `.svg` with no env override
- [ ] importing a tree containing one `.svg` creates a row with `is_image=True`, an `image_path` under `image_dir`, and content from the SVG's own text — with the vision seam never invoked
- [ ] a second sync over the unchanged file reports it `unchanged` (the guard fires once, the fast path resumes)
- [ ] a pre-existing text-SVG row flips to `is_image=True` on the next sync (fixture test)
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,31 @@
# Task 03 — Serve the bytes as sandboxed `image/svg+xml`
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "…see a preview … rather than the xml" (the transport half: something must answer `/api/documents/<id>/image` with SVG the browser will paint, safely)
**Story:** n/a
## Objective
`GET /api/documents/{doc_id}/image` serves an SVG row as `image/svg+xml` behind a `default-src 'none'; sandbox` CSP, while every raster response keeps its current bytes, headers and 404 semantics unchanged.
## Work
1. `app/rag/summarizer.py::IMAGE_MIMES` (L152-159) — add `".svg": "image/svg+xml"`. Extend the map's doc-comment: SVG is the one vector entry, and the vector import branch (task 02) never routes through `image_data_url`, so this entry exists for the BYTES route only (`app/api/docs.py::get_document_image` reads the same map — one map, one truth).
2. `app/api/docs.py::get_document_image` (L319-378) — keep the lookup, the public id-as-credential posture (phase 124 LOCKED A4 — a shared chat shows the figure to a guest), the four 404 shapes (unknown id, unparseable id, text doc, missing file) and the `Cache-Control: private, max-age=3600`. Add ONE conditional header next to `media_type`:
```
headers = {"Cache-Control": "private, max-age=3600"}
if media_type == "image/svg+xml":
headers["Content-Security-Policy"] = "default-src 'none'; sandbox"
```
(LOCKED A3.) Raster responses gain NO new header — their header set is byte-identical.
3. `app/api/docs.py` — docstring on the route: why SVG needs the extra header — an `<img>` can never execute SVG script, but the URL is directly navigable, and without the policy a stored SVG would be a script-capable same-origin document; `sandbox` blocks scripting AND makes the response an opaque origin (no same-origin fetch/form capability), `default-src 'none'` blocks every subresource. Note that `app/core/security_headers.py::SecurityHeadersMiddleware` PRESERVES an inner CSP (L109-110), so this route's value is the one the browser sees, and that the middleware's global `X-Content-Type-Options: nosniff` still applies.
4. `README.md` — the security-headers section (where phase 82/123's CSP is described) gains one line: SVG document bytes are served sandboxed, with the one-sentence rationale.
- ASSUMPTION: no `Content-Disposition: attachment` — the graphic must render inline; `sandbox` is the stronger and less intrusive control.
- ASSUMPTION: the CSP is scoped to `image/svg+xml` (derived from the extension map, never from a client value) rather than applied to all image types — a raster needs no policy change, and adding one would risk the phase-123/124 image surfaces.
## Testing & Quality
- Integration: `tests/integration/test_svg_document_image.py` (new, task 05 completes it): an SVG row → `200`, `content-type == "image/svg+xml"`, `content-security-policy == "default-src 'none'; sandbox"`, `x-content-type-options == nosniff`, `cache-control == "private, max-age=3600"`, body bytes byte-equal to the stored copy. A raster (`.png`) row → NO `content-security-policy` key in the response headers and the existing headers unchanged. The 404 shapes: unknown uuid, a non-uuid string (`404`, not 422), a text doc, an image row whose copy was deleted from disk.
- Unit: `IMAGE_MIMES` contains `.svg` and the existing entries are untouched (the map is asserted by identity, not membership, so a silent removal fails).
- Coverage: **>90%** on the changed lines.
## Completion Criteria
- [ ] `curl -sD- -o/dev/null http://127.0.0.1:8000/api/documents/<svg-doc-id>/image` shows `content-type: image/svg+xml` and `content-security-policy: default-src 'none'; sandbox`
- [ ] the same request for a PNG row shows no `content-security-policy` header (and `content-type: image/png`)
- [ ] `uv run pytest tests/integration/test_svg_document_image.py tests/integration/test_docs_api.py -v` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,28 @@
# Task 04 — The preview instead of the XML (viewer, figure, modal, thumbnail)
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Also would be nice to click an svg and see a preview in the RAG tab rather than the xml"
**Story:** n/a
## Objective
An SVG document paints as a graphic at readable size in the document modal and the `/document.html` viewer, keeps its compact figure in the chat citation row, and keeps its Sources thumbnail — no XML text anywhere on those surfaces.
## Work
1. `frontend/assets/document.js::docImageBlock` (L571-591) — one addition: when `doc.format === "svg"`, set the img's class to `doc-image-img doc-image-img--vector` (keep the existing base class so every current rule still applies). Rationale to put in the comment: an SVG carrying only a `viewBox` has NO intrinsic size, so the replaced-element default box (300×150) would sit small inside the full-width `.doc-image` card; `doc.format` is the only place in the app that knows the type (the bytes URL is extension-less and `SourceRef` carries no format field by design — design D7). Do NOT inline the SVG into the DOM (LOCKED A3): `img.src = doc.image_url` stays `createElement` + property assignment, which is already the file's XSS contract.
2. `frontend/assets/styles.css` (next to `.doc-image-img`, L5623-5630) — `.doc-image-img--vector { width: 100%; height: auto; object-fit: contain; }` with the house comment (the no-intrinsic-size explanation, why `object-fit: contain` protects `max-height: 70vh` from squashing the drawing, static content/no animation). Palette-neutral: no new colour.
3. `frontend/assets/app.js::appendSourceImageFigure` (~L2485+) — NO structural change: the figure already keys off `image_url`, already carries the summary caption and the chip's navigation (the click → `openDocumentModal` path is the "preview in the RAG tab" the owner asked for). Add the comment recording the deliberate non-change: a size-less SVG inside the 96px-capped `.source-image-img` scales inside the cap, and the full-size preview is the modal (no `format` field is added to `SourceRef` — the wire contract stays byte-identical for text docs).
4. `frontend/assets/sources.js` (the `d.is_image` thumbnail branch, ~L2328) — verify the KB-tree thumbnail renders for an SVG row through the same `image_url` and add the same one-line comment; only touch the code if the E2E proves a real gap (report the finding in the commit body either way).
5. Legacy honesty — do NOT special-case it: a row still indexed as TEXT (an `.svg` under `BOR_IMAGES=0`, or not yet re-synced) keeps rendering its XML in `.doc-raw`, which is the truth about that row (task 02's fast-path guard converts it on the next sync). Comment this in `docImageBlock` so nobody "fixes" it by sniffing the content string.
- ASSUMPTION: no lightbox/zoom affordance is added — the modal at full width IS the preview the owner asked for; zoom is a separate feature request.
- ASSUMPTION: no "download the SVG" control — the bytes URL is directly openable and the existing "Copy path" control already covers provenance.
## Testing & Quality
- Unit: `tests/unit/test_document_viewer.py` extended — `docImageBlock` for `format === "svg"` yields `class="doc-image-img doc-image-img--vector"` with `src` = `image_url` and `alt` = summary (title fallback); for `format === "png"` the class string is unchanged (the base class only); the CSS pins `.doc-image-img--vector` exists, references no colour literal, and keeps `object-fit: contain`.
- E2E (task 05): the modal + viewer assertion that an SVG row renders an `<img>` with `naturalWidth > 0` and that the modal body contains no `<` XML text and no `<pre class="doc-raw">`.
- Coverage: n/a (frontend) — the unit pins + the E2E are the gate.
## Completion Criteria
- [ ] clicking an SVG source chip in a chat answer opens the modal showing the rendered graphic (no XML, no `.doc-raw` in the modal body)
- [ ] `/document.html?source=…&path=…svg` shows the graphic spanning the `.doc-image` card (width ≥ 600px at a 1280px viewport for a `viewBox`-only SVG) with the extracted text below it
- [ ] the chat citation figure and the Sources thumbnail show the graphic; the figure's caption is the extracted text
- [ ] a raster image document's viewer DOM is unchanged (the phase-122/133 suites pass unedited)
- [ ] `uv run pytest` green; `uv run ruff check .` clean
@@ -0,0 +1,38 @@
# Task 05 — Unit + integration + the dedicated Playwright suite, the gates, the commit
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` (the acceptance evidence for tasks 01–04)
**Story:** n/a
## Objective
Prove the whole SVG path — extraction, image-document ingestion, the sandboxed bytes route, and the preview at every surface — with the phase's own test files, run the AGENTS gates, and land the single atomic commit.
## Work
1. `tests/unit/test_svg_text_extraction.py` — completes task 01's pure-function matrix (document order, `tspan` joining, namespaced tags, whitespace/dedupe rules, entity decoding, `<!DOCTYPE`/`<!ENTITY` refusal, the 2 MiB cap, malformed XML → `None`, the structural fallback with its plural + clause-omission rules, `MAX_*` constants pinned).
2. `tests/unit/test_svg_image_ingest.py` (or the extension of `tests/unit/test_image_documents.py` chosen in task 02) — the vector branch with the `_describe_or_skip` seam patched to RAISE (the "no vision call" proof), `content`/`summary`/chunk shape, `is_image`/`image_path` under `image_dir`, `svg_indexed`, the extraction-failure skip (`images_failed`, no row, no copy), the phase-125 `image_context` prepend for an SVG, and the `_index_file` fast-path kind guard in BOTH directions (text row converts; correctly-classified row stays `unchanged`).
3. `tests/integration/test_svg_document_image.py` — task 03's contract (SVG → `image/svg+xml` + the sandbox CSP + `nosniff` + `Cache-Control` + exact bytes; PNG → NO CSP key; the four 404 shapes) plus `GET /api/documents/content` for an SVG row: `is_image: true`, `image_url` present and of the form `/api/documents/<uuid>/image`, `format: "svg"`, `content` = the extracted text. Add the wire-contract pin that ties the phases together: that `image_url` shape is accepted by phase 180's `allowImageUrl` (assert the literal path passes the renderer's allowlist regex — the `![alt](<image_url>)` synergy).
4. `tests/fixtures/svg_docs/` (new dedicated tree — never the shared fixture set) — `diagrams/topology.svg` (a hand-written multi-kilobyte diagram: a `viewBox` and NO `width`/`height`, a `<title>Homelab network topology</title>`, a `<desc>` sentence, two `<text>` labels, and a dozen `<path>`/`<rect>` shapes so retrieval has real words) + `notes/infra.md` (prose that names the diagram so the grounded question has a text co-citation) + `diagrams/blank.svg` (pure paths — the structural-fallback subject).
5. `tests/e2e/test_svg_document_preview.py` (new; the `tests/e2e/test_image_documents.py` module-app pattern — its own app instance with `BOR_IMAGES=true` and `BOR_IMAGE_DIR` pinned to a suite tmp dir so a local `.env` cannot leak either way, the phase-138 `_reset_db` truncate-then-`import_sources` seeding, `login(...)`, mock LLM, `E2E_REAL_LLM` skipped):
- `test_viewer_renders_svg_not_xml` — open the Sources row for `diagrams/topology.svg` (admin): the modal shows `img.doc-image-img--vector` with `naturalWidth > 0` and a non-empty `alt`; the modal body contains no `<pre class="doc-raw">` and no `<svg` XML text; the extracted `<title>` text appears as the description below the graphic; the "Full page" link's `/document.html` view shows the same graphic at ≥600px width at a 1280px viewport (the vector sizing contract).
- `test_chat_figure_and_modal_preview` — the grounded question whose answer cites the SVG doc shows `.source-image` whose `<img>` loads (`naturalWidth > 0`) with the caption from the extracted text; clicking the figure opens the modal with the full graphic (the "preview in the RAG tab" contract) and Escape closes it.
- `test_blank_svg_is_indexed_and_searchable` — `diagrams/blank.svg` exists as a document with the structural `Vector diagram "blank" — …` content (the LOCKED A2 fallback), listed on the Sources page with its thumbnail.
- `test_markdown_image_ref_renders` (two-stage seeding — copy the fixture tree to the suite's tmp sources dir, import it, read the SVG doc's id from the DB, then write `notes/inline.md` containing `![topology](/api/documents/<id>/image)` and re-import): the viewer for that document shows `img.md-img` pointing at the bytes route and loading (`naturalWidth > 0`) — the phase-180 renderer + this route end to end.
- `test_raster_and_pdf_surfaces_unchanged` — the smoke guard: a PNG doc in the same tree still renders through `.doc-image-img` (no `--vector` class) and the CSP header is absent on its bytes route.
6. Run the gates in order, fixing what fails (no baselining, no skipping):
- `uv run pytest`
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL >90%
- `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` (in isolation, DB up)
- `uv run ruff check . && uv run pyright`
- `uv run python -m scripts.api_payload_audit` — the phase-167 byte-budget line for the new SVG route is INFO-sized (an SVG fixture stays well under 256 KB).
7. Move the phase directory to `.agents/phases/complete/` and commit — see the overview's Commit block.
## Testing & Quality
- This task IS the phase's gate. If a test cannot be made deterministic, fix the product, not the assertion (the AGENTS rule 9 posture) — and record anything genuinely untestable in the commit body.
- Coverage: **>90%** on `app/`; `app/rag/svg_text.py` should sit at ≥95% (pure functions, no excuses).
## Completion Criteria
- [ ] `uv run pytest` green (new unit + integration files included)
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` green in isolation
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] `uv run python -m scripts.api_payload_audit` clean for the SVG route
- [ ] one atomic commit, `--no-gpg-sign`: `feat(kb): index SVGs as image documents and preview them instead of raw XML`, containing `app/`, `frontend/`, `tests/`, `.env.example` and `.agents/`
@@ -0,0 +1,79 @@
# Phase 182 — Verbatim file display: a `show_text` tool + `bor` locator fences the server resolves
**Story:** n/a (owner chat 2026-10-14 — "show me my pi.dev config" / "show me the quadlet for gufo" answers are re-typed from context and silently add/remove characters, so the answer text cannot be copy-pasted; the owner has to open the source bubble and ignore the written code block)
**Context:** `app/rag/agent.py` — the harness-aligned tool surface `AGENT_TOOLS` (L332: `ls` / `read` / `grep`) plus the 4th, DISPLAY tool `SHOW_IMAGE_TOOL` (L501) composed by `agent_tools(settings)` (L538), the shared identity resolver `_resolve_path` (L1357), the line-range validator `_read_range_args` (L1474) and the `OFFSET_PAST_END` (L671) / `BAD_RANGE_ARGS` (L628) teachings, the refusal block L571–L690 (`MISSING_SHOW_ARGS` L571, `NOT_AN_IMAGE_DOC` L581, `ALREADY_SHOWING` L585), `AgentHolder` (L1600–L1680: `shown_images`, `read_coverage`, `read_truncations`), `_execute_tool` (L1681, the `show_image` branch L2023–L2068 is the shape this phase mirrors); `app/rag/prompts.py` — `TOOLS_SECTION` (L293) and the appended-only-when-offered `SHOW_IMAGE_CLAUSE` (L379) + `build_high_prompt(show_image=…)` (L759); `app/api/chat.py` — the prompt/tool call site `show_image=settings.answer_images` (L529), the per-turn visible-content counter `content_chars` (L934), the `done`-frame builder (L1320–L1365, `answer_ref`/`answer_image` L1343–L1355), the pinned per-turn log line (L1272–L1291); `app/schemas.py` — `SourceRef` (L169), `ChatToolResultEvent` (L279), `ChatDoneEvent.answer_image` + its omission serializer (L305–L380), `ChatMessageRecord.answer_image` + `_drop_image_when_absent` (L1463, L1478); `app/config.py` — `read_max_chars` (L296) + its validator (L686), `answer_images: bool = False` (L539), `log_large_response_bytes` (L76); `app/core/settings_catalog.py` — `read_max_chars` (L267) / `answer_images` (L296) rows (the completeness pin in `tests/unit/test_settings_catalog.py` makes a new `Settings` field a build failure until it has a row); `frontend/assets/markdown.js` — `renderMarkdown(md)` (L43), fence-protection pass (L44–L48: fences → tables → escape+inline → blockquotes → paragraphs → restore, escape-first, no CDN); `frontend/assets/app.js` — the per-delta whole-bubble re-render `renderMarkdown(acc)` (L4322), the `done` handler (L4325–L4460), `appendAnswerImage` (L2598), the restore renderer (L2890–L2960), `copyShareLinkWithFallback` (L3250, the http-origin clipboard degradation), `rememberBrainTurn` save point (L4390–L4420); `frontend/assets/shared.js` — `addAnswerImage` (L573) + the record renderer (L619–L660); `frontend/index.html` L1696–L1701 and `frontend/shared.html` L197–L199 (classic `brand.js` + `markdown.js`, then the module script); `frontend/assets/styles.css` — `.answer-image` (L1028–L1059), the bounded-scroll-panel and `.cell-clip` conventions (PLAN §7.1, phases 156/167); `tests/e2e/mock_llm.py` — the scripted-trigger registry (L1162–L1520, `SHOW_IMAGE_TRIGGER` L2606, `_SHOW_CONFIRM_MARKER`, the "verified: no existing E2E question or fixture file contains the phrase" discipline); `tests/e2e/test_answer_images.py` (the two-app ON/OFF pattern), `tests/unit/test_frontend_*.py` (the JS/CSS text-pin convention).
## Objective
When the owner asks *"show me the quadlet for gufo"*, the exact bytes of the indexed file reach the chat answer — **typed by the server, never by the model** — in a panel with a path + line-range caption, a `VERBATIM` badge, line numbers, a working Copy button and a link to the source. The model only *names* the location (`show_text(path, offset?, limit?)`, or an inline ` ```bor <path> <range> ` locator fence where it wants the block to sit); the server resolves the slice from Postgres and the client places it. This kills the class of bug where the answer's code block silently differs from the file, and it removes the user-facing line-number problem entirely: the numbers shown are the server's resolved extent, not the model's claim.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the two mechanisms, one authority (A1).** (a) a 5th agent tool `show_text(path, offset?, limit?)`, the display sibling of `show_image`; (b) an **inline locator fence** the model may write in its answer — ` ```bor ` + the combined `source/path` + an optional range — so a resolved block can sit anywhere in the prose. **Both are resolution requests, never content requests**: the only party that supplies bytes is the server. `show_text` is the authority for teaching (a refusal teaches the model to self-correct before the user sees anything); the fence is placement sugar and ALSO resolves standalone (a fence with no prior `show_text` call still renders, because resolution is a plain `(source, path)` lookup, not a replay of a tool call).
- **D2 — server resolves, client places (A4/A7).** The client cannot resolve (no new route, A4), so: `app/api/chat.py` accumulates the turn's **clean visible answer text** (next to the existing `content_chars` counter, L934 — the post-scaffold-filter text only, never `thinking`), and at `done`-build time passes it through `app.rag.verbatim.resolve_answer_blocks(db, answer, holder.shown_texts, settings)`; the result rides the `done` frame as the ADDITIVE `answer_text` array. `markdown.js` turns a `bor` fence into `<div class="verbatim" data-bor-locator="<canonical>">` and **drops the model's fence body**; `app.js`/`shared.js` hydrate each placeholder by matching `data-bor-locator` against the record's `answer_text[].locator` (canonical-form match → unique path-only match → unresolved). A block the tool registered that the answer never locates renders APPENDED, at the `answer_image` position (below the answer, above the source clusters).
- **D3 — the model's bytes are NEVER painted (A7, the phase's core contract).** Inside a `bor` fence the model's body is discarded at render time — while streaming the placeholder reads `Retrieving the exact text…` (`role="status"`, polite by definition), and a fence the model has not finished typing shows that SAME pending state rather than raw backticks, at `done` it becomes the resolved panel or, when it cannot be resolved, the honest state `I could not verify this against the knowledge base — open <source>` (persona: radically honest, never a silent substitution and never the model's possibly-corrupt text). The E2E proves it with a mock that deliberately types a CORRUPTED body and asserts the panel carries the DB bytes. `renderMarkdown` gains a SECOND, OPTIONAL argument (`{ verbatim: true }`); default OFF keeps every existing call site — the document viewer (`document.js` L494), the thinking blocks, the user bubbles — byte-identical, and a document that legitimately contains a ` ```bor ` fence still renders its own text.
- **D4 — line ranges are the server's (A2).** `offset` (1-based) / `limit` are OPTIONAL and validated by the existing `_read_range_args` (L1474): omitted → the WHOLE document (the taught default for "show me X"); `offset` alone → to the end; `limit` overshoot → CLAMPED to the last line; `offset` past the last line → the `OFFSET_PAST_END`-style refusal naming the document's true line count (counts in nothing, consumes one round, invisible to the user). Slicing uses `content.splitlines()` with the SAME 1-based indexing as `read` (L1836) and `grep`, so every line number in the app agrees. The char cap (`settings.show_text_max_chars`) cuts the SLICE at a line boundary, appends the shared `TRUNCATION_MARKER` (`app/rag/retriever.py`) and shrinks `end_line` to the last fully-shown line — the caption never lies.
- **D5 — what the model gets back (A3).** The tool result is a confirmation + the truth + an instruction, then the exact slice:
`Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.` / `Document {source}/{path}:` / `date: …` / `{origin lines}` / `lines: {first}-{end} of {total_lines}` / the slice. The `Document …` + `date:` + `lines:` header contract is the one `read` established (the E2E mock's `_READ_RESULT_PREFIX` pin keeps parsing it). Returning the text is deliberate (A21: LLM calls are free) — the model's commentary can then never disagree with the panel.
- **D6 — new module, no import cycle (D2).** `app/rag/verbatim.py` owns `VerbatimBlock` (frozen dataclass: `source, path, title, start_line, end_line, total_lines, text, truncated, locator`), the pure half (`canonical_locator`, `slice_lines`, `fences_in`, `cap_slice`) and one DB function (`document_by_identity(db, combined)`, a bare `select(Document).where(source==, path==)` — the same `(source, path)` identity contract as `agent._resolve_path`, WITHOUT its did-you-mean suggestions). `agent.py` imports the pure half, so `verbatim.py` must never import `agent.py` (documented at the module top — protocol D2). Path traversal is a non-issue by construction: a traversal string is simply a non-existent row (404-shaped refusal, no filesystem access anywhere).
- **D7 — canonical locator (the matching key).** `source/path` for a whole document, `source/path#12-40` for a range. The tolerant input grammar (both the fence parser and the tool's `path` argument) accepts `source/path`, `source/path 12-40`, `source/path:12-40`, `source/path#12-40`, `source/path#L12-L40` and `source/path 12` (single line); anything unparseable is not a `bor` locator at all and renders as an ordinary code block (the body is then the model's own, exactly as today, with a Copy button and no `VERBATIM` badge). Resolution normalizes to the canonical form, so the client's match is a string compare.
- **D8 — caps, kill switch, observability (A8).** `answer_text: bool = True` (`BOR_ANSWER_TEXT`, the master switch — `false` = the tool is not offered, the prompt clause is absent, resolution does not run, no `answer_text` key: every frame and every byte-identity contract is exactly pre-phase), `show_text_max_chars: int = 12_000` (per block; `0` = inert, negative fails startup naming the field — the `agent_max_rounds`/`read_max_chars` pattern), `show_text_max_blocks: int = 5` (per answer, in answer order; `0` = inert, negative fails startup). Worst-case frame growth 5 × 12 000 = 60 KB against the 256 KB budget, which the phase-167 byte-budget line (`api … bytes=…`, `grep ' :: api '`) makes visible. The §9 per-turn line gains ONE additive trailing field `verbatim_blocks=N` (the blocks actually sent, tool-registered + fence-resolved, deduped) — `tests/unit/test_chat_gate.py`'s pinned line is updated in the same task, never silently.
- **D9 — resolution scope (the honesty posture).** Resolution runs on GROUNDED turns only (the agent ran). A `bor` fence hallucinated into a DEFLECTED answer has no registered blocks and the gate said "I haven't done anything like that" — it renders the unresolved state rather than leaking KB text through a turn that just disclaimed it. `done.sources` / `related` semantics are NOT changed (no regression to the citation surface): the panel itself carries the path, the line range, a Copy button and a link to the document viewer (the git origin link stays the viewer's job, phase 145's `origin_for_source` — the panel does not duplicate it) — the citation affordance sits at the point of use.
- **NOT touched:** `show_image` / `SHOW_IMAGE_TOOL` / `answer_image` (image display is out of scope, A6 — verified, not modified), the `ls`/`read`/`grep` contracts, the SSE event-type LIST (A15 stays at seven types — this phase rides `done`, no 8th event), `documents`/`chunks` schema (no migration: `saved_chats.messages` is raw JSONB, the phase-50/160 precedent), `document.js` (the viewer renders documents, not answers), `app/core/response_budget.py`, `scripts/*`, `frontend/assets/router.js`.
## Locked decisions (owner-confirmed 2026-10-14, from the phase-authoring interview)
- **A1 — BOTH mechanisms in this phase:** the `show_text` tool AND the inline ` ```bor ` locator fence. (The owner chose "A + B in one phase" over the tool alone.)
- **A2 — whole-file default + server clamp:** `offset`/`limit` optional; omitted = the whole document; overshoot clamps; past-the-end refuses to the model with the true total; the caption always shows the SERVER's resolved extent. No `expect`-anchor snapping (considered and deferred — the clamp plus the model's own `grep`/`read` line numbers cover it).
- **A3 — the tool returns the verbatim slice** plus the "do not retype" instruction (not a terse confirmation, not the read_coverage-conditional variant).
- **A4 — the bytes ride INLINE** in `done.answer_text` and persist in the `bor.chat.v1` brain record: no new bytes route, no new SSE event type, no new anonymous surface (a shared page renders from the share's own record, the same trust model as the phase-123 chat image).
- **A5 — Copy buttons on EVERY fenced code block in an answer** (live, restored and shared), but the `VERBATIM` badge + caption belong ONLY to a server-resolved block. The clipboard goes through the phase-55 degradation (a non-secure origin gets the inline select-to-copy fallback) — never a bare `navigator.clipboard` assumption.
- **A6 — images are out of scope:** `show_image` keeps images; `show_text` on an `is_image` document is REFUSED with a pointer to `show_image`. The phase's E2E asserts the answer figure is unchanged.
- **A7 — the model's fence body never reaches the DOM** (see D3). Unresolvable = honest, visible, non-copyable-as-truth state.
- **A8 — settings:** `BOR_ANSWER_TEXT` default `true` (master switch), `BOR_SHOW_TEXT_MAX_CHARS` default `12000`, `BOR_SHOW_TEXT_MAX_BLOCKS` default `5`; `0` = inert, negative = loud startup failure.
- **A9 — additive wire only:** `answer_text` is dropped when empty (absent, never `null`) — every frame without blocks, every pre-phase record and the whole default-off configuration stay byte-identical. No PLAN §2/§4 anchor changes beyond recording these revisions.
## Dependencies
- `124_answer_images` (complete) — the display-tool precedent this phase mirrors end to end (`SHOW_IMAGE_TOOL` → `holder.shown_images` → `done.answer_image` → `appendAnswerImage` → persistence → shared page).
- `131_read_line_ranges` / `141_read_range_dedupe_fix` (complete) — `offset`/`limit` + `_read_range_args` + the `lines: N-M of T` header contract and `splitlines()` indexing.
- `145_git_source_links` (complete) — the document viewer's `origin_for_source` link, which the panel links TO rather than duplicating.
- `167_api_payload_diet` (complete) — the byte-budget line that will show the `done`-frame growth.
- `180_markdown_inline_links_images` (todo, already authored) — independent of this phase; both touch `markdown.js` and `styles.css`, so run it FIRST (or rebase) to avoid a renderer conflict.
## Tasks
1. `01_settings_and_kill_switch.md` — the three `Settings` fields + validators + the three `CATALOG` rows + `.env.example` (+ their unit/integration pins). Inert: nothing reads them yet.
2. `02_verbatim_resolver_module.md` — `app/rag/verbatim.py`: the locator grammar, the canonical normalizer, `splitlines()` slicing, the line-boundary cap and `resolve_answer_blocks` (+ the resolver unit suite).
3. `03_show_text_tool.md` — `SHOW_TEXT_TOOL`, the `agent_tools` gate, the `_execute_tool` branch, the refusal set, `AgentHolder.shown_texts` (+ agent unit suite).
4. `04_prompt_teaching_clause.md` — `SHOW_TEXT_CLAUSE` inside `<tools>`, `build_high_prompt(show_text=…)`, the `app/api/chat.py` call site (+ byte-identity pins when off).
5. `05_done_frame_and_persistence.md` — the `VerbatimBlockRef` schema, `ChatDoneEvent.answer_text`, `ChatMessageRecord.answer_text`, the visible-text accumulator + resolution + `verbatim_blocks=N` in the per-turn line (+ integration/frame tests).
6. `06_markdown_locator_and_panel_css.md` — `renderMarkdown(md, opts)`'s opt-in `verbatim` pass → `.verbatim[data-bor-locator]` (body dropped; ordinary fences keep rendering exactly as today), and the `styles.css` panel/copy-button CSS contract. Inert until task 07 opts in and wraps `<pre>` nodes at the DOM level.
7. `07_hydration_and_copy_buttons.md` — `frontend/assets/verbatim.js` (+ `index.html`/`shared.html` script tags): hydrate-at-`done`, restore, shared page, error/settled paths, `rememberBrainTurn` persistence, copy buttons on every answer code block.
8. `08_e2e_and_docs_pass.md` — the mock triggers (`show_text` call, corrupt-body fence, fence-only, off-app), `tests/e2e/test_verbatim_show_text.py` in isolation, the regression set, the full gates, the README/AGENTS full consistency pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` (the three defaults; env overrides; `0` accepted as inert; each negative value fails validation naming the field); `tests/unit/test_settings_catalog.py` (the completeness pin green with the three new rows); `tests/unit/test_verbatim_resolver.py` (NEW — the locator grammar incl. every tolerant form + rejects, canonical normalization, whole-file/offset/limit/clamp/past-end, the line-boundary cap + `TRUNCATION_MARKER` + shrunk `end_line`, `fences_in` ordering + dedupe, tool-then-fence dedupe, `max_blocks` cut, an `is_image` doc refused, an unknown path unresolved, a fenced `bor` line inside an ordinary fence, an unclosed fence, an empty answer); `tests/unit/test_agent.py` (the `show_text` branch: success shape + headers, `shown_texts`, refusal set — missing args, non-string, unknown path did-you-mean, image doc, bad range args, offset past end, block cap, duplicate identical show; every refusal counts in nothing and executes nothing; `agent_tools` offers 4 vs 5 tools by flag); `tests/unit/test_prompts.py` (the clause is INSIDE `<tools>` before `</tools>`, the off-build is byte-identical to pre-phase, `TOOLS_SECTION` itself unchanged); `tests/unit/test_chat_gate.py` (the per-turn line's new `verbatim_blocks=N`, and `0` on a turn with none); `tests/unit/test_frontend_verbatim_markdown.py` + `tests/unit/test_frontend_verbatim_blocks.py` + a CSS pin module (the `test_frontend_*` text-pin convention: the `verbatim` option, the dropped body, `data-bor-locator`, the `.verbatim`/`.code-block`/`.verbatim-copy` markers, the new `<script>` tags in `index.html` + `shared.html` (plain relative `src` — no `cache-busting` token in the path, the `?v=` rewrite is the middleware's job, phase 160 A7), `shared.js` parity, no inline `onclick` — CSP `default-src 'self'`).
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` (all three rows: `bool` + two `int`s, correct groups/labels, live apply without a restart, reset-to-env); the chat API suite — a grounded turn with a `show_text` call and/or a fence yields `done.answer_text` with the SERVER's text and extent; a turn with none has NO `answer_text` key (byte-identity); `answer_text=false` yields the pre-phase frame byte-identically; the saved-chat round trip preserves `answer_text` (`extra="forbid"` accepts it — the phase-113 lesson) and `GET /api/chats/{id}` returns it.
- **E2E (AGENTS rule 4, task 08):** `tests/e2e/test_verbatim_show_text.py`, run **in isolation** — the panel's text equals the fixture file's bytes (the mock's corrupted fence body never appears anywhere in the DOM), the caption reads `source/path · lines 12–40 of N` from the SERVER's extent, `VERBATIM` badge + line numbers + Copy button copy EXACTLY the panel's bytes (clipboard read with granted permissions, or the fallback field), the panel's open link targets `/document?source=…&path=…`, an unresolvable locator renders the honest state with no model bytes, restored-after-reload and the anonymous shared page render the same panel, every ordinary code block has a Copy button and no badge, and the `show_image` figure still renders (A6 regression).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (setup, commands, configuration, features, conventions). This phase's user-visible additions: the `show_text` capability + the `bor` locator syntax in the feature list, the three `BOR_*` settings in the env table, and (in `AGENTS.md` rule 5's UI list) the verbatim-panel contract "an answer's code block is either verbatim-and-badged or plainly the model's — never a silently-substituted hybrid".
## Completion Criteria
- [ ] `ask "show me the quadlet for <x>"` on a running app (real or mock LLM) renders a panel whose text is byte-identical to `documents.content`'s resolved slice, with a caption whose numbers came from the server, and a Copy button whose clipboard content equals that slice exactly.
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` prints `True 12000 5`; each negative value refuses startup naming its field.
- [ ] `GET /api/admin/settings` lists `answer_text`, `show_text_max_chars`, `show_text_max_blocks`; `PUT`/`DELETE` round-trip per the overlay contract.
- [ ] A grounded turn's `done` frame carries `answer_text[]` with `source,path,title,start_line,end_line,total_lines,text,truncated,locator`; a turn without verbatim content carries NO key; with `BOR_ANSWER_TEXT=false` the frame is byte-identical to pre-phase and only FOUR tools are offered.
- [ ] The per-turn log line ends with `verbatim_blocks=N` (`grep` it on a shown turn and on a plain turn).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_verbatim_show_text.py -v --no-cov` green **in isolation**; `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_markdown_tables.py` (and the phase-180 suite if it landed first) green in isolation.
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
- [ ] no behavior change in completed phases (image answers, `read`/`grep`/`ls`, deflected byte-identity, the document viewer, saved-chat byte-identity).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(agent): display file contents verbatim with a show_text tool and server-resolved locator fences`); the phase dir moves to `.agents/phases/complete/`.
@@ -0,0 +1,33 @@
# Task 01 — Three settings and the kill switch (inert)
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Add the three `Settings` fields the phase runs on — the master `answer_text` switch and the two budgets — with their admin-panel rows and `.env.example` entries. Nothing reads them yet: after this task the app behaves byte-identically to pre-phase.
## Work
1. `app/config.py` — in the agent-tools neighborhood of `read_max_chars` (L296), add:
- `answer_text: bool = True` — the master switch for verbatim display (`BOR_ANSWER_TEXT`). Docstring: names the env var, the default `True`, and what `False` means (the `show_text` tool is not offered, the `<tools>` clause is absent, locator resolution does not run, the `done` frame carries no `answer_text` key — every byte-identity contract is exactly pre-phase). Model the docstring on `answer_images` (L539) but note the DEFAULT IS TRUE here (the owner chose the fix on by default; the switch is an escape hatch, not an opt-in).
- `show_text_max_chars: int = 12_000` (`BOR_SHOW_TEXT_MAX_CHARS`) — the per-block character budget for one displayed slice; `0` = inert (no block displays), a negative value is a typo and fails startup naming the field (the `_read_max_chars_non_negative` validator at L686 is the exact pattern).
- `show_text_max_blocks: int = 5` (`BOR_SHOW_TEXT_MAX_BLOCKS`) — the per-answer cap on displayed blocks; `0` = inert, negative fails startup (the `agent_max_rounds` validator at L678 is the pattern).
Add the two `@field_validator`s next to `_read_max_chars_non_negative`, each raising a `ValueError` that names the field (house rule: fail loud, never half-configured).
2. `app/core/settings_catalog.py` — three `SettingMeta` rows beside `read_max_chars` (L267): `answer_text` → `"Answer text"`, `show_text_max_chars` → `"Show text max chars"`, `show_text_max_blocks` → `"Show text max blocks"`, all in the `Agent tools` group (the `read_max_chars` group). The completeness pin in `tests/unit/test_settings_catalog.py` (`set(CATALOG) == set(Settings.model_fields)`) is the enforcement — without the rows the build fails.
3. `.env.example` — three lines in the agent-tools block, each with a one-line comment: `BOR_ANSWER_TEXT=true`, `BOR_SHOW_TEXT_MAX_CHARS=12000`, `BOR_SHOW_TEXT_MAX_BLOCKS=5`. State the `0`/negative semantics in the comments (the surrounding file's style).
4. No consumer changes. Do NOT touch `app/rag/agent.py`, `app/rag/prompts.py`, `app/api/chat.py`, `app/schemas.py` or any frontend file in this task.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` — defaults (`True`, `12000`, `5`); each env override applies; `BOR_SHOW_TEXT_MAX_CHARS=0` and `BOR_SHOW_TEXT_MAX_BLOCKS=0` are accepted (inert, not an error); each negative value raises a validation error whose message names the field (the `read_max_chars` test is the template). `tests/unit/test_settings_catalog.py` — the completeness pin green, the three rows carry the right type/group/label.
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — the three rows appear in `GET /api/admin/settings` (`bool` + two `int`, group `Agent tools`); `PUT` stores each raw value and it resolves live without a restart; `DELETE` resets each to the env default; the `0` values are storable.
- **Coverage:** **>90%** on this task's new/modified code (the validators are the only new logic — cover both branches of each).
## Completion Criteria
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` → `True 12000 5`.
- [ ] `BOR_SHOW_TEXT_MAX_CHARS=-1 uv run python -c "import app.config"` refuses with a message naming `show_text_max_chars` (same for `show_text_max_blocks`).
- [ ] The three rows are visible in the admin Settings panel's `Agent tools` group (or asserted by the integration test if the browser is not running).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the env table) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed work (no consumer reads the new fields yet).
@@ -0,0 +1,47 @@
# Task 02 — `app/rag/verbatim.py`: the locator grammar, the slicer, the resolver
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
The single source of truth for "which exact lines of which indexed document, and what are their bytes". Pure functions for the locator grammar and the slicing, one bare DB lookup, and `resolve_answer_blocks` which the API layer calls at `done` time. Inert: nothing calls it yet (tasks 03 and 05 wire it in).
## Work
1. Create `app/rag/verbatim.py`. Module docstring states the contract (PLAN §6's tool surface extension, phase 182 D1–D7): *the model names a location, this module supplies the bytes*; and the import rule — **this module must never import `app.rag.agent`** (agent.py imports the pure half of this module; a reverse import cycles). Allowed imports: `sqlalchemy`, `app.models.Document`, `app.rag.retriever.TRUNCATION_MARKER`, `app.config.Settings`, stdlib.
2. `LOCATOR_LANG = "bor"` and `@dataclass(frozen=True) class VerbatimBlock` — `source: str`, `path: str`, `title: str`, `start_line: int`, `end_line: int`, `total_lines: int`, `text: str`, `truncated: bool`, `locator: str` (the canonical form, built by the factory, never passed in by a caller). The dataclass docstring pins: `start_line`/`end_line` are 1-based and INCLUSIVE and describe exactly the lines `text` contains after any cap; `text` never carries a trailing newline (lines are joined with `"\n"`).
3. `canonical_locator(raw: str) -> str | None` — the tolerant parser of D7. Accepts, in order: a combined `source/path` (the first `/` splits it, the `read`/`grep` identity contract) followed by an optional separator + range — separators ` `, `:`, `#`, `#L`; ranges `12-40`, `12–40` (en dash), `L12-L40`, or a bare `12` (one line, end == start). Returns `source/path` for a whole document and `source/path#12-40` for a range (always the canonical `#` + plain digits, never the en dash, never the `L`). Returns `None` (NOT a locator) for: an empty/whitespace input, a value with no `/` in the path part, a non-numeric or non-positive bound, `start > end`, trailing junk after the range, or a leading `bor` token left in by the caller. No DB, no normalization of case (paths are case-sensitive row values).
4. `fences_in(answer: str) -> list[str]` — every `bor` fence's RAW locator string, in document order, deduped keeping the first occurrence. Rules: the opening line is a fence whose info string is exactly `` ```bor `` + whitespace + a non-empty remainder (a bare `` ```bor `` with no path is NOT a locator — it is an ordinary code block and must not be swallowed); the fence closes at the next line that is `` ``` `` (trimmed); an UNCLOSED fence contributes nothing (defensive — a half-streamed answer); the body is read and DISCARDED (this module never looks at what the model typed); nested `` ``` `` inside the body is not supported and is a documented limitation (the fence ends at the first closer, the same rule `markdown.js` applies).
5. `@dataclass(frozen=True) class SliceResult` — `text: str`, `first_line: int`, `last_line: int`, `total_lines: int`, `past_end: bool` — and `slice_lines(content: str, offset: int | None = None, limit: int | None = None) -> SliceResult`. Indexing is `content.splitlines()` with the SAME 1-based semantics as `read` (`app/rag/agent.py` L1836) and `grep`: `offset` omitted → 1; `limit` omitted → to the end; `end = min(offset + limit - 1, total_lines)` (an overshooting `limit` CLAMPS); `offset > total_lines` → `past_end=True` with empty text and the true `total_lines` (the caller builds the teaching refusal); `total_lines == 0` (an empty document) → `past_end=True`. Never raises for a range problem — the refusal is the caller's, the total always travels with the result.
6. `cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]` — the character budget applied at a LINE boundary, returning `(text_to_show, last_line_shown, truncated)`. `cap <= 0` raises `ValueError` (a mis-wired caller must be loud — the `show_text_max_chars == 0` kill switch is checked by `resolve_answer_blocks`, never by reaching here). When `len(text) <= cap`: `(text, last_line_of(text), False)`. Otherwise cut to the last whole line that fits within `cap` and return `(cut + "\n" + TRUNCATION_MARKER, first_line + n_lines_in_cut - 1, True)`; a first line LONGER than `cap` cuts that one line mid-way (the only sub-line cut, documented) and still ends with the marker. The marker is the shared `app.rag.retriever.TRUNCATION_MARKER` — never a second copy of the text.
7. `document_by_identity(db: Session, combined: str) -> Document | None` — split at the FIRST `/` (source names are directory basenames and never contain one); `select(Document).where(Document.source == source, Document.path == path)` → the row or `None`. Docstring: this is the same `(source, path)` identity contract as `app.rag.agent._resolve_path` REIMPLEMENTED as a bare lookup — no did-you-mean suggestions here (the tool branch already has `_resolve_path` for teaching; the fence resolver only needs yes/no), and the duplication exists to keep the import direction one-way. A traversal string is simply a row that does not exist.
8. `block_for(db, combined: str, *, offset: int | None = None, limit: int | None = None, cap: int, doc: Document | None = None) -> VerbatimBlock | None` — resolve (or take the caller's already-resolved `doc`), refuse `None` for: no document, `doc.is_image` (image display belongs to `show_image`, locked A6), an empty/past-end slice; otherwise `slice_lines` → `cap_slice` → the block with `locator = f"{source}/{path}"` (+ `f"#{first}-{last}"` when the request named a range or the cap cut it — the canonical form task 07 matches on). `title` is the row's title.
9. `resolve_answer_blocks(db, answer: str, shown: Sequence[VerbatimBlock], settings: Settings) -> list[VerbatimBlock]`:
- returns `[]` when `not settings.answer_text`, or `settings.show_text_max_blocks == 0`, or `settings.show_text_max_chars == 0` (the kill switches, checked HERE so no caller has to remember);
- for each `fences_in(answer)` raw string in order: `canonical_locator` → skip when `None`; `block_for(...)` → skip when `None` (unknown path, image doc, past end) — the caller sees simply "not among the blocks", which is what task 07 renders as the unresolved state;
- dedupe by canonical locator, the FENCE winning over a same-locator entry from *`shown`* (the fence controls placement);
- append the `shown` blocks the answer never located, in their original call order;
- truncate the list to `settings.show_text_max_blocks`.
Docstring pins the ordering rule (answer order first, then the unlocated tool blocks) — task 07's DOM order and the E2E assertions depend on it.
10. Update the module-level `__all__` and keep every public name docstring'd in the house style (contract + phase reference).
## Testing & Quality
- **Unit:** create `tests/unit/test_verbatim_resolver.py`.
- `canonical_locator`: `homelab/a.md` → itself; `homelab/a.md 12-40` / `:12-40` / `#12-40` / `#L12-L40` / `12–40` (en dash) → `homelab/a.md#12-40`; `homelab/a.md 12` → `#12-12`; `None` for `a.md` (no source), `""`, `" "`, `homelab/a.md 0-5`, `homelab/a.md 40-12`, `homelab/a.md x-y`, `homelab/a.md 12-40 extra`, `"bor homelab/a.md"`.
- `slice_lines`: whole file, offset-only, limit-only, both, overshooting limit clamps `last_line` to the total, `offset` past the end → `past_end` + true total, empty content → `past_end`, and the `splitlines()` agreement check (a fixture whose line `N` is asserted by content, mirroring what `grep` would report).
- `cap_slice`: under cap (untouched, `truncated=False`), cut at a line boundary (text ends with the shared `TRUNCATION_MARKER`, `last` = the last whole line shown, and the marker line is not counted as content), a single over-long first line, `cap=0` raises `ValueError`.
- `document_by_identity`: hit, miss, traversal-shaped miss, first-slash splitting when the path itself contains slashes.
- `block_for`: whole-document block, ranged block, canonical locator with/without a range, image doc → `None`, unknown path → `None`, past-end → `None`, capped block's `end_line` equals the lines actually shown.
- `resolve_answer_blocks`: no fences + no shown → `[]`; fence-only (no tool call) resolves; tool-only appends; fence + identical shown block → ONE block (fence wins, appears at the fence position); two fences keep answer order; `show_text_max_blocks=1` truncates; every kill-switch combination returns `[]`; an unresolvable fence is simply absent; a duplicated fence appears once.
- Session-fixture style follows the existing agent/resolver unit suites (the same in-memory/session fixture the `tests/unit/test_agent.py` helpers use).
- **Coverage:** **>90%** on this task's new/modified code (every branch above is a test).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_verbatim_resolver.py -v` green.
- [ ] `uv run python -c "import app.rag.verbatim as v; print(v.canonical_locator('homelab/a.md#L12-L40'))"` prints `homelab/a.md#12-40`.
- [ ] `grep -n "import" app/rag/verbatim.py` shows NO import of `app.rag.agent` (the one-way rule).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (nothing user-visible yet — the phase's final pass re-checks both project-wide).
- [ ] no behavior change in completed work (nothing calls this module yet).
@@ -0,0 +1,57 @@
# Task 03 — The `show_text` agent tool: name a location, the server shows the bytes
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
The 5th agent tool: `show_text(path, offset?, limit?)` displays an indexed document (or a line range of one) to the user verbatim. The handler resolves the slice through task 02, records it for the `done` frame, and hands the SAME bytes back to the model with an explicit "do not retype" instruction.
## Work
1. `app/rag/agent.py` — add `SHOW_TEXT_TOOL` right after `SHOW_IMAGE_TOOL` (L501), same OpenAI function shape:
- `name: "show_text"`.
- `description` (this IS the behavior contract — the phase-124 discipline): "Display part of a document to the user EXACTLY as stored — the user sees the server's bytes and can copy them. Call it when the user wants to see, copy or run a file (a quadlet, a config, a script): they copy from what is displayed, so do NOT retype the content yourself after calling it — describe it instead. `path` is the combined `source/path` string exactly as shown in `ls`. `offset` (1-based first line) and `limit` (line count) are optional — omit them to display the whole document; use them when `grep`/`read` told you which lines matter. The displayed line numbers come from the server, not from your arguments: an overshooting `limit` is clamped and an `offset` past the end is refused with the true line count. Only text documents — for an image use `show_image`. Call one tool at a time — wait for this result before your next call."
- `parameters`: `path` (string, required, the `read` "a bare document path will not resolve" teaching), `offset` and `limit` (integers, optional, the `read` wording reused).
2. `agent_tools(settings)` (L538) — append `SHOW_TEXT_TOOL` when `settings.answer_text`, keeping `AGENT_TOOLS` byte-identical and the list fresh per call (the `show_image` precedent; the offer order is `ls`, `read`, `grep`, `show_image`, `show_text`). Update the docstring (four base + two display tools, each gated by its own flag).
3. `AgentHolder` (L1600–L1680) — add `shown_texts: list[VerbatimBlock] = field(default_factory=list)` with a docstring paragraph in the `shown_images` style: the blocks in CALL order, capped by `settings.show_text_max_blocks`, read by `app.api.chat` after the stream for the `done` frame's `answer_text` (the `shown_images`/`read_docs` read-after-stream precedent); refused shows append nothing.
4. Refusal constants beside the `show_image` ones (L571–L590), each single-braced and pinned in the house style:
- `MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."`
- `SHOW_TEXT_NOT_A_TEXT_DOC` — one `{arg}` field: the doc is a standalone image; `show_image` displays images. (LOCKED A6 — never silently converted.)
- `SHOW_TEXT_BLOCK_CAP` — one `{max}` field: that many blocks are already displayed this answer — describe them instead of showing more.
- `SHOW_TEXT_ALREADY_SHOWN` — one `{locator}` field: that exact range is already on screen — say so instead of showing it again.
Reuse `BAD_RANGE_ARGS` (L628) and `OFFSET_PAST_END` (L671) verbatim for the range problems (the same teaching, the same echoed values — one copy of each string in the codebase).
5. `_execute_tool` (L1681) — a `show_text` branch placed next to the `show_image` branch (L2023), in this order:
1. `raw_path = call.arguments.get("path")`; non-string/blank → `MISSING_SHOW_ARGS`-shaped refusal (`MISSING_SHOW_TEXT_ARGS`).
2. `settings.answer_text` false (defensive — the tool was not offered) → the same refusal copy as the round-cap-off posture: state that the tool is unavailable; counts in nothing.
3. `doc, _source, _path = _resolve_path(db, arg)`; `doc is None` → `_no_document_refusal(db, arg)` (the phase-72 did-you-mean teaching, reused unchanged).
4. `doc.is_image` → `SHOW_TEXT_NOT_A_TEXT_DOC`.
5. `ok, err = _read_range_args(offset, limit)` — reuse the existing validator unchanged; `not ok` → `err`.
6. `len(holder.shown_texts) >= settings.show_text_max_blocks` → `SHOW_TEXT_BLOCK_CAP` (checked BEFORE the work; `0` therefore disables the tool's effect while keeping it offered — pinned in the tests).
7. `block_for(db, arg, offset=offset, limit=limit, cap=settings.show_text_max_chars, doc=doc)` → `None` (empty document / `offset` past the end) → `OFFSET_PAST_END.format(total_lines=...)` (compute the total with `len(doc.content.splitlines())`, the same expression `read` uses); a returned block whose canonical `locator` already appears in `holder.shown_texts` → `SHOW_TEXT_ALREADY_SHOWN`.
8. Success: `holder.shown_texts.append(block)`; `holder.tool_calls += 1` (executed-only counting, the house rule); return the result string in this EXACT shape (the `read`/`show_image` header contract the E2E mock's `_READ_RESULT_PREFIX` parses):
```
Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.
Document {source}/{path}:
date: {YYYY-MM-DD}
{origin lines — the same _origin_lines(origin) helper the read branch uses}
lines: {first}-{last} of {total_lines}
{block.text}
```
`block.text` already ends with `TRUNCATION_MARKER` when the char cap cut it; when `block.truncated` is true append one line, in the `READ_TRUNCATION_NOTICE` spirit, naming that the rest is NOT displayed and that the user can open the document (no `offset`/`limit` re-show loop is implied — the full text is a click away, and only the displayed slice is what the user copies).
- Every refusal path appends nothing, increments nothing, and consumes a round exactly like the existing refusals.
6. Module docstring — extend the numbered loop contract (the `show_image` paragraph's neighborhood) with the `show_text` contract and the locked rules A1–A3/A6: the model names a location, the server supplies the bytes, the displayed extent is the server's, images belong to `show_image`, the fence form is resolved at the API layer (task 05), not here.
## Testing & Quality
- **Unit (`tests/unit/test_agent.py`, the `show_image` family is the template):** `agent_tools` offers 4 tools with `answer_text=False` (byte-identical to pre-phase, `show_image` last) and 5 with it on (`show_text` present, correct order); a successful whole-document show (result headers in order, `lines: 1-N of N`, the slice verbatim, `shown_texts` length 1, `tool_calls` 1, `read_docs` unchanged); a ranged show (clamped `limit` reflected in the `lines:` header); a capped show (`TRUNCATION_MARKER` present + the extra notice line); every refusal — missing `path`, non-string `path`, unknown path (did-you-mean copy, no `shown_texts`), image doc, bad range args, `offset` past the end (the true total named), block cap (`max_blocks=1`, second call refused), duplicate identical show; each refusal asserts `tool_calls` and `shown_texts` untouched; `max_blocks=0` → the first call is refused with the cap copy.
- **Integration:** none required in this task (the frame wiring is task 05) — but `tests/integration/` must stay green, and any test that pins the exact tool-offer array is updated here, deliberately, with a comment.
- **Coverage:** **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_agent.py -v -k show_text` green, including all nine refusal cases and the counting assertions.
- [ ] With `answer_text=True`, a scripted agent turn returns a result whose 2nd line starts `Document ` and whose `lines:` header matches the slice actually returned.
- [ ] `grep -n "SHOW_TEXT_" app/rag/agent.py` shows the four new constants, each used by exactly one branch.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the tool surface list in README's agent-tools section) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed work (`ls`/`read`/`grep`/`show_image` results byte-identical; the pre-phase four-tool offer byte-identical when the flag is off).
@@ -0,0 +1,36 @@
# Task 04 — Teach the model to point instead of type: the `<tools>` clause
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
The HIGH prompt teaches two things it has never said: **the user copies from what is displayed, not from your prose**, and the exact ` ```bor ` locator-fence syntax for placing a displayed block. Gated by `settings.answer_text`, byte-identical when off.
## Work
1. `app/rag/prompts.py` — add `SHOW_TEXT_CLAUSE: str` right after `SHOW_IMAGE_CLAUSE` (L379), with a docstring comment in the same style (why it exists, what each sentence pins, and that `TOOLS_SECTION` stays byte-identical). The clause text must contain, in this order:
- the display contract: "`show_text` displays part of a document to the user EXACTLY as stored — the user copies from what is displayed, not from your prose.";
- when to call it: "Call it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a config, a script, a unit file — instead of writing the content into your answer yourself.";
- the do-not-retype rule (the phase's reason for existing): "After calling it, do NOT repeat the displayed lines in your answer — describe them, explain them, point at them. Anything you type yourself may differ from the file by a character, and the user cannot tell.";
- the placement syntax: "To put a displayed block exactly where you want it in your answer, write a locator line on its own: three backticks followed by `bor`, a space, the combined `source/path`, and optionally the line range (` 12-40`). Leave its body EMPTY — the app replaces it with the exact stored text. Never type the file's content inside a `bor` block.";
- the honesty-about-numbers rule: "Never guess line numbers — use the numbers `grep` and `read` reported. When you are not sure which lines, OMIT the range and display the whole document: the numbers you display are the server's, never yours.";
- the failure rule: "If `show_text` refuses, say what you could not display — do not reconstruct it from memory.";
- the image pointer: "For an image use `show_image` instead."
2. `build_high_prompt(...)` (L759) — add `show_text: bool = False` after `show_image`; when true, the clause rides INSIDE `<tools>` AFTER the `SHOW_IMAGE_CLAUSE` (when that is present) and before the closing `</tools>` tag, using the existing "section text minus the final `</tools>` + clause + closing tag" mechanism (L782–L791 documents it). Both clauses compose independently: `show_image` only → exactly the phase-124 output; `show_text` only → the three-tool copy + the text clause; both → image clause then text clause; neither → byte-identical to pre-phase. Update the docstring (a *show_text* paragraph mirroring the *show_image* one, incl. the note that the deflection prompt NEVER carries it — a deflected turn has no tools and D9 forbids resolving locators on it).
3. `app/api/chat.py` (the grounded `TurnPlan` builder, L516–L529) — pass `show_text=settings.answer_text` next to `show_image=settings.answer_images`, with the same style of phase comment.
4. Do NOT change `TOOLS_SECTION`, `build_deflect_prompt`, or any mock in `tests/e2e/mock_llm.py` (the new scripted triggers belong to task 08).
## Testing & Quality
- **Unit (`tests/unit/test_prompts.py` + the `tests/unit/test_prompt_lock.py` pins):** the off-build (`show_text=False`, the default) is byte-identical to the pre-phase string for every combination of the other arguments; the on-build contains the clause INSIDE `<tools>` (index of the clause text > index of `<tools>`, `< ` index of `</tools>`); `show_image=True, show_text=True` yields the image clause before the text clause; `TOOLS_SECTION` itself unchanged (the existing lock test still passes untouched); the clause names `show_text`, `bor`, the do-not-retype rule and the "omit the range" rule (the E2E and the persona depend on those exact ideas); the deflect prompt never contains `show_text` or `bor`.
- **Integration:** the grounded-turn integration tests stay green (the prompt string is not pinned byte-for-byte there beyond the `<tools>` marker); add one assertion that a grounded plan built with `answer_text=True` carries the clause and one that `answer_text=False` does not.
- **Coverage:** **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` green.
- [ ] `uv run python -c "from app.rag.prompts import build_high_prompt as b; print('show_text' in b([], show_text=True))"` prints `True`, and the same call without the flag prints `False`.
- [ ] A grounded request against a running app shows the clause in the model request (log/assert via the E2E mock's request capture in task 08 — here, the unit proof is enough).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the prompt/tool teaching summary) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed phases (deflected prompt byte-identical; the phase-124 clause position untouched).
@@ -0,0 +1,34 @@
# Task 05 — The `done` frame's `answer_text`, the visible-text accumulator, the log field
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Put the resolved bytes on the wire and in the persisted record: the turn's clean visible answer text is accumulated during streaming, resolved through `app.rag.verbatim` at `done`-build time, and sent as the ADDITIVE `answer_text` array; the per-turn log line gains `verbatim_blocks=N`.
## Work
1. `app/schemas.py` — `class VerbatimBlockRef(BaseModel)` beside `SourceRef` (L169): `source: str = Field(max_length=120)`, `path: str = Field(max_length=1000)`, `title: str = Field(max_length=500)` (the `SourceRef` caps, the same identity), `start_line: int = Field(ge=0)`, `end_line: int = Field(ge=0)`, `total_lines: int = Field(ge=0)`, `text: str = Field(max_length=600_000)` (the phase-83 answer-text cap — generous above the 12 000 default, so an operator raising `BOR_SHOW_TEXT_MAX_CHARS` does not hit the wire limit), `truncated: bool = False`, `locator: str = Field(max_length=2100)`. Docstring: the phase-182 display block; `text` is SERVER-supplied bytes from `documents.content`, never model output; 1-based inclusive lines.
2. `ChatDoneEvent` (L305) — `answer_text: list[VerbatimBlockRef] | None = None` + docstring in the `answer_image` style, and extend its `model_serializer` to POP the key when `None` **or empty** (absent, never `null`, never `[]`: every frame without blocks stays byte-identical to pre-phase — the pin that proves it lives in the integration suite). Note the two deliberate differences from `answer_image` in the docstring: it is a LIST (several blocks per answer, D1), and it carries the TEXT itself (A4 — no bytes route, no new anonymous surface, a shared page renders from its own record).
3. `ChatMessageRecord` (L1400–L1510) — `answer_text: list[VerbatimBlockRef] | None = None` on the BRAIN record, popped when `None`/empty in the existing `_drop_image_when_absent` serializer (extend its docstring). Required so `extra="forbid"` does not 422 the done-time auto-save of a turn with blocks (the phase-113 lesson, applied up front) and so the restore path + the shared page re-render the panels. No migration (`saved_chats.messages` is raw JSONB, the phase-50/160 precedent).
4. `app/api/chat.py`:
1. Next to the visible-content counter (`content_chars`, L934) add `visible_parts: list[str]` and append EVERY visible (post-scaffold-filter) content piece in the piece loop — the same pieces counted into `content_chars`, never `thinking` (D3: locators only ever come from the answer). Bounded by the model's output-token budget; note that in the comment.
2. In the `done` builder (L1320–L1365), after `answer_ref`: `blocks = resolve_answer_blocks(db, "".join(visible_parts), holder.shown_texts, settings) if not plan.deflected else []` (D9 — a deflected turn resolves NOTHING: no fence leaks KB text through a turn that just disclaimed it; the deflected path has no `holder`, so the guard is load-bearing). Convert with `VerbatimBlockRef(**dataclasses.asdict(b))` in order, pass `answer_text=blocks or None`.
3. Per-turn log line (L1272–L1291): append `verbatim_blocks=%d` after `scaffold_stripped=%d` (the count of blocks SENT, i.e. after dedupe + `show_text_max_blocks`; `0` on every turn without them, including deflected ones) and the module docstring's line-format block with the new field. This is the §9 additive revision — update `tests/unit/test_chat_gate.py`'s pinned line in the same commit, never silently.
4. `holder.shown_texts` is read AFTER the stream (the `shown_images` precedent) — do not reorder the generator's finally/cancel semantics (a cancelled turn still writes neither line nor `query_log` row).
5. Nothing in `app/schemas.py`'s existing validators changes; no new SSE event type (A15 stays at seven).
## Testing & Quality
- **Unit:** `tests/unit/test_chat_gate.py` (or the module that owns the per-turn line pins) — the line ends `… scaffold_stripped=N verbatim_blocks=0` on a plain grounded turn, `verbatim_blocks=2` on a turn whose holder has two blocks, `verbatim_blocks=0` on a deflected turn (and the deflected frame still carries no `answer_text`).
- **Integration:** the chat API/streaming suite — a scripted grounded turn that calls `show_text` yields a `done` frame whose `answer_text[0]` has the SERVER's `text`, the clamped extent, the canonical `locator`, and `truncated` false under the cap; a turn whose ANSWER contains a ` ```bor ` fence (no tool call) resolves the same block from the fence; fence + identical tool call produce exactly ONE entry; three blocks with `show_text_max_blocks=2` send two in answer order; a turn with no blocks serializes WITHOUT the `answer_text` key (byte-identity pin, exact-dict compare); `answer_text=False` yields the pre-phase frame byte-for-byte AND `verbatim_blocks=0`; a deflected frame carries no `answer_text` even when its (mocked) text contains a fence. The saved-chat suite — `POST /api/chats` with a brain record carrying `answer_text` is accepted (`extra="forbid"`), `GET /api/chats/{id}` returns it unchanged, and a record without it round-trips byte-identically to pre-phase.
- **Coverage:** **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] A scripted SSE turn with a `show_text` call returns `answer_text: [{source,path,title,start_line,end_line,total_lines,text,truncated,locator}]` and the text equals `documents.content` for that range exactly.
- [ ] `grep ' :: api ' ` on that request shows the `done`-bearing response's byte line (the phase-167 audit) — record the observed size in the phase's task notes if it approaches the 256 KB budget.
- [ ] The per-turn log line shows `verbatim_blocks=N`; a plain turn shows `0`.
- [ ] `uv run pytest` green (incl. the updated line pins), coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the SSE `done` frame contract in README's API section; AGENTS rule 10's log-line list) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed phases (`answer_image`, `sources`, `related`, `budget_exhausted`, the deflected byte-identity, the cancelled-turn contract).
@@ -0,0 +1,43 @@
# Task 06 — `markdown.js`: the opt-in `bor` locator pass (model bytes never painted) + the panel CSS
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Teach the shared renderer to recognize a ` ```bor ` locator fence and replace it — body and all — with a placeholder node, and write the panel's CSS. Inert: the pass is opt-in through a second argument that NO call site passes yet (task 07 opts the answer renderers in), so every existing render path stays byte-identical.
## Work
1. `frontend/assets/markdown.js` — `renderMarkdown(md)` becomes `renderMarkdown(md, opts)` (a SECOND, OPTIONAL argument; `opts && opts.verbatim === true` is the only truthy path). Header comment: the phase-182 pass, why the model's body is discarded, and the byte-identical-when-off contract.
2. Add the **verbatim pass BEFORE the existing fence pass** (order: verbatim → fences → tables → escape+inline → blockquotes → paragraphs → restore):
- it matches a fence whose info string is exactly `bor` + whitespace + a non-empty remainder: `` /^```bor[ \t]+\S[^\n]*\n(?:[\s\S]*?^```[ \t]*$|(?![\s\S]))/gm `` — the second alternative matches a fence left OPEN at the very end of the input, so a half-streamed locator shows the pending placeholder instead of raw backticks (a bare ` ```bor ` with no path still does NOT match: it is an ordinary code block and must survive to the fence pass untouched); note the deliberate asymmetry with the server's stricter `fences_in` (task 02 ignores unclosed fences — the server only ever sees a finished answer, the client sees a partial one);
- the matched locator text is run through the new local `borLocator(raw)` normalizer; if it returns `""` (unparseable), the match is put BACK byte-for-byte and left to the existing fence pass — a malformed locator degrades to whatever that pass does with this text today (no placeholder, no badge, the model's own body): this phase neither improves nor worsens it;
- a parsed one is replaced with `"\u0000VERBATIM" + n + "\u0000"` and recorded, and restored in the final restore step as:
`<div class="verbatim verbatim-pending" data-bor-locator="<escaped canonical>"><span class="verbatim-caption"><escaped source/path></span><span class="verbatim-state" role="status">Retrieving the exact text…</span></div>`
— every interpolated value goes through the existing `escapeHtml`; the model's body appears NOWHERE in the output (the phase's core contract, A7).
3. `borLocator(raw)` (module-local, docstring'd): trims; splits the path at the FIRST `/` (no `/` → invalid); accepts the separators ` `, `:`, `#` and `#L` before the range and the ranges `12-40`, `12–40` (en dash), `L12-L40`, `12`; returns `source/path` or `source/path#12-40` (canonical digits only), `""` for anything else (non-numeric, `< 1`, `start > end`, trailing junk). Documented as the CLIENT MIRROR of Python `app.rag.verbatim.canonical_locator` — the parity is pinned by the phase's E2E (both sides must agree or hydration misses), not by a unit test.
4. Do NOT wrap ordinary fences in markdown: task 07 wraps `<pre>` nodes at the DOM level (so the document viewer, the thinking blocks and every other caller keep byte-identical HTML).
5. `frontend/assets/styles.css` — a clearly-labelled phase-182 block. Palette variables ONLY (`--surface`, `--line`, `--ink`, `--ink-soft`, `--accent-*`, `--ok-*`; the phase-92 zero-literal rule — a monochrome theme must keep every state readable, WCAG "text + color, never color alone"):
- `.verbatim` — block, `margin` in the bubble's rhythm, `border: 1px solid var(--line)`, `border-radius: var(--radius-sm)`, `background: var(--surface)`, `min-width: 0`, `overflow: hidden`.
- `.verbatim-head` (row) + `.verbatim-caption` — the `source/path · lines 12–40 of 210` line, mono, `--ink-soft` (≥4.5:1), clipped with the existing `.cell-clip` ellipsis convention (phase 156) while the control's `aria-label`/`title` keeps the full text; `.verbatim-head` STICKS to the top of the scroll panel (`position: sticky; top: 0` + a matching background — PLAN §7.1's sticky-header rule for bounded panels).
- `.verbatim-badge` — a small TEXT badge reading `VERBATIM` (text, never an icon alone), `--ok` family, `text-transform: none` (no uppercase surprises for screen readers), and a `title` explaining "these bytes are copied from the knowledge base, not written by the model".
- `.verbatim-body` — the bounded scroll panel: `max-height: min(40rem, 70vh)`, `overflow: auto`, monospace, `white-space: pre`, `tab-size: 2`, `overflow-wrap: normal` (code scrolls horizontally, prose never does — phase 165's `word-wrap` split), `min-width: 0`.
- `.verbatim-lines` — the line-number gutter (`<ol>`/`<li>` or a `data-line` column, the implementer's choice, but numbers must be `user-select: none` so a Copy/select never drags line numbers into the clipboard), right-aligned, `--ink-soft`, `position: sticky; left: 0` optional.
- `.verbatim-pending` + `.verbatim-state` — the streaming/pending look (muted, no animation beyond what `prefers-reduced-motion` allows); `.verbatim-unresolved` — the honest failure state, `--err`/`--ink-soft`, no `VERBATIM` badge.
- `.code-block` (the task-07 DOM wrapper) + `.code-block-copy` / `.verbatim-copy` — a small button, `min-height: 44px` target on `@media (max-width: 640px)`, `:focus-visible` 3px ring like the rest of the app, `text-overflow: ellipsis` on its label, and a `.copied` state that changes TEXT (`Copy` → `Copied`), not color alone.
- The panel is a code surface, so it must stay readable in a light theme too: no hard-coded `#fff`/`#000`, no new hue.
## Testing & Quality
- **Unit (`tests/unit/test_frontend_verbatim_markdown.js` is not possible — use the project's text-pin convention, `tests/unit/test_frontend_verbatim_markdown.py`):** the `renderMarkdown(md, opts)` signature and the `opts.verbatim` guard exist; the verbatim pass runs BEFORE the fence pass; `\u0000VERBATIM` placeholders + the restore step exist; the output template contains `verbatim-pending`, `data-bor-locator`, `verbatim-caption`, `role="status"` and the pinned pending copy `Retrieving the exact text…`; `borLocator` exists and the separator/range tokens it handles are all present in its body; the ordinary-fence `<pre><code>` output string is unchanged in the source.
- **Unit (CSS, `tests/unit/test_frontend_verbatim_css.py`, the `test_frontend_cluster_css.py` pattern):** the phase-182 selectors exist; `.verbatim-body` has a `max-height` in the `min(…, 70vh)` form and `overflow: auto`; `.verbatim-head` is `position: sticky`; line numbers are `user-select: none`; no new hex/rgb literals in the phase-182 block (parse the block and assert every color is a `var(--…)`); the mobile block keeps a ≥44px control height; `prefers-reduced-motion` is honored by anything that animates.
- **Coverage:** **>90%** on the task's new/modified code (Python-side: the pin modules execute the assertions; there is no new `app/` code in this task, so `app/` coverage is unchanged — it must stay >90%).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_frontend_verbatim_markdown.py tests/unit/test_frontend_verbatim_css.py -v` green.
- [ ] Serving the app and calling `renderMarkdown("```bor homelab/a.md 12-40\nGARBAGE\n```", { verbatim: true })` in the browser console yields the placeholder DOM with `data-bor-locator="homelab/a.md#12-40"` and NO `GARBAGE` anywhere in the returned HTML; the same call WITHOUT the second argument returns the ordinary `<pre><code>` block.
- [ ] `renderMarkdown` output for the phase-44 table fixtures, the phase-138 blockquote fixtures and the phase-180 image fixtures is unchanged (the existing suites stay green).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the renderer pass list in README's frontend section) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed work (document viewer, thinking blocks, user bubbles, tables, blockquotes, inline images).
@@ -0,0 +1,43 @@
# Task 07 — Hydration, panels, and a Copy button on every answer code block
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Wire the client: resolve each placeholder against the frame's `answer_text`, render the panel (caption, `VERBATIM` badge, line numbers, Copy, viewer link), persist the blocks with the turn, and give every settled answer code block a Copy button. Live turn, restore-after-reload and the anonymous shared page all render identically.
## Work
1. Create `frontend/assets/verbatim.js` (classic script, like `markdown.js`) and include it AFTER `markdown.js` in `frontend/index.html` (L1696–L1697 block) and `frontend/shared.html` (L197–L198 block) — plain relative/absolute `src` exactly as the neighbors, the `?v=` token rewrite is the caching middleware's job (A19). Expose globals (the same pattern `markdown.js` uses), all DOM built with `createElement` + `textContent` (the frontend house rule — never an HTML string), all handlers via `addEventListener` (CSP `default-src 'self'`, no inline handlers):
- NO locator normalizer here — `markdown.js::borLocator` is the single implementation (task 06), and hydration compares the server's canonical `locator` against the already-normalized `data-bor-locator` attribute with a plain string compare. Re-implementing the grammar in a second file is the phase's main drift risk; do not.
- `buildVerbatimPanel(block, opts)` → the `.verbatim` node: `.verbatim-head` with `.verbatim-caption` (`source/path · lines 12–40 of 210`, or `source/path · all 210 lines` when the block is the whole document), `.verbatim-badge` (`VERBATIM`, `title` = "these bytes are copied from the knowledge base, not written by the model"), `.verbatim-copy` (`Copy`, `aria-label` = `Copy N lines from <source/path>` — the phase-156 clipped-label rule), an optional `.verbatim-open` link to the document (`opts.onOpen` hook: `app.js` passes the modal opener built from `documentUrl` + `openDocumentModal` (back → the chat, phase 153); `shared.js` passes a plain `/document?source=&path=` href — the shared page has no modal); `.verbatim-body` with the line-number gutter (`user-select: none`) and the text lines; the root gets `role="group"` + `aria-label` = `Verbatim text from <source/path><, lines N–M>`, and `data-bor-locator`. A truncated block (`block.truncated`) appends one muted line naming that the rest is in the document, plus the same open link. The git ORIGIN link stays the document viewer's job (phase 145) — the panel links to the viewer, it does not duplicate the origin.
- `hydrateVerbatim(root, blocks)` — for every `.verbatim[data-bor-locator]` in *root*: exact canonical-locator match in *blocks* → replace with the panel; else exactly ONE block sharing the same `source/path` → use it (the model's range was wrong; the panel shows the SERVER's extent — this is A2 made visible); else → the unresolved state: `.verbatim verbatim-unresolved` with the text `I could not verify this against the knowledge base — open <source/path>` (the link only when the path itself parsed) and NO badge, NO body, and none of the model's bytes. Never throws on a malformed/missing `blocks`.
- `appendUnlocatedVerbatim(wrap, blocks)` — the blocks no placeholder matched, appended in order at the `answer_image` position (below the answer text, above the source clusters, the phase-124/130 DOM order).
- `settleVerbatimPlaceholders(root, kind)` — turns leftover `.verbatim-pending` into the unresolved state (used by the error/stop paths so no panel ever sits at "Retrieving…" after a terminal frame).
- `enhanceAnswerCodeBlocks(root)` — every `<pre>` inside an answer bubble that is NOT inside a `.verbatim`: wrap in `.code-block` (once — idempotent on re-render) and append `.code-block-copy` with the same clipboard path. NO badge, NO caption — a `VERBATIM` badge belongs only to a server-resolved block (locked A5).
- `copyTextWithFallback(text, statusEl)` — the phase-55 degradation generalized from `app.js::copyShareLinkWithFallback` (L3250): try `navigator.clipboard.writeText`, on rejection render/return the inline select-to-copy fallback; returns `true` when the clipboard took it. `app.js`'s share path then DELEGATES to it (single implementation) — the owner-locked copy strings stay exactly as they are, `test_history_copy.py` / the share suites are the guard.
2. `frontend/assets/app.js`:
- the answer renderers pass the opt-in: the per-delta `renderMarkdown(acc)` (L4322), the `done`-time re-render (L4371) and the brain restore renderer (L2909) become `renderMarkdown(text, { verbatim: true })`. The USER bubble renderers (L1272, L1347, L2890) and every thinking-block render (L2841, L2939) stay as they are — user text and the scratchpad are not locators.
- the `done` handler (the `answer_image` neighborhood, L4340–L4360): `if (ev.answer_image) appendAnswerImage(...)` then `hydrateVerbatim(bubble, ev.answer_text || [])` then `appendUnlocatedVerbatim(wrap, ev.answer_text || [])` then `appendSources(...)`; `enhanceAnswerCodeBlocks(wrap)` runs once the turn settles (buttons appear with the settled answer, never mid-stream — pinned copy in the tests).
- persistence (the `rememberBrainTurn(...)` save point, L4390–L4420): `answer_text: ev.answer_text || undefined` (undefined drops the key — the `answer_image` pattern, so a pre-phase record stays byte-identical).
- the restore path re-renders + hydrates from `m.answer_text` (the `answer_image` restore precedent at L3001); a restored record with no `answer_text` renders its fences as the unresolved state (the record is what it is — never resurrect model bytes).
- the terminal failure paths (the `error` frame handler, the 300 s `TURN_TIMEOUT_MS` settle, the Stop action) call `settleVerbatimPlaceholders(bubble, "unverified")`.
- the Retry / edit-and-resend paths: nothing extra — they re-run the turn, the new turn hydrates on its own (verify the phase-120/159 suites stay green).
3. `frontend/assets/shared.js` — the record renderer (L619–L660, the `addAnswerImage` neighborhood) passes `{ verbatim: true }`, hydrates from `m.answer_text`, appends the unlocated blocks, enhances the settled code blocks, and uses the plain-href `onOpen` (no modal on the shared page). Parity is a hard requirement: the same blocks render the same panel on both surfaces.
4. `frontend/assets/styles.css` — extend task 06's block only where the real DOM needs it (states the JS adds: `.copied`, `.verbatim-unresolved`); palette variables only, no new literals.
## Testing & Quality
- **Unit (`tests/unit/test_frontend_verbatim_blocks.py`, the `test_frontend_*` text-pin convention):** `verbatim.js` is included after `markdown.js` in BOTH `index.html` and `shared.html`; `hydrateVerbatim` / `appendUnlocatedVerbatim` / `settleVerbatimPlaceholders` / `enhanceAnswerCodeBlocks` / `copyTextWithFallback` / `buildVerbatimPanel` exist; `app.js` passes `{ verbatim: true }` at the three brain renderers and does NOT pass it in the user/thinking renderers (assert by scoping the regex to the right call sites); the `done` handler hydrates BEFORE `appendSources`; `rememberBrainTurn` persists `answer_text` with `|| undefined`; `shared.js` reaches the same three globals (parity); no `onclick=`/`oncopy=` attribute strings anywhere in the new code (CSP); no `innerHTML += ` in the new code; the pinned unresolved copy `I could not verify this against the knowledge base`; the badge text `VERBATIM`; the copy button's `aria-label` template names the full path.
- **Unit (CSS):** the new state selectors exist and stay literal-free (task 06's module extends).
- **Coverage:** no new `app/` code; `app/` coverage must stay **>90%**, and the new pin module runs in the unit suite.
## Completion Criteria
- [ ] Live turn against a mock or real LLM: the panel shows, with the caption's numbers, the `VERBATIM` badge, line numbers, and a Copy button whose clipboard content equals the server text exactly; every ordinary code block in the same answer has a Copy button and NO badge.
- [ ] Reload the page: the restored bubble renders the identical panel. Open the Share link in an anonymous window: the shared page renders the identical panel.
- [ ] An answer with a ` ```bor ` fence whose path does not resolve shows the honest unresolved state with no model bytes; nothing sits at "Retrieving the exact text…" after a terminal frame (force it with the `slow_llm`/always-fail trigger).
- [ ] A `show_text` call whose block the answer never locates renders appended above the source clusters.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean; keyboard-only check: the copy button and the viewer link are reachable with a 3px `:focus-visible` ring, and the reduced-motion setting changes nothing jumpy.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the answer-panel feature list, AGENTS rule 5's UI contract line) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed phases (share-link copy strings, answer image, source clusters, related row, edit/resend, thinking timeline).
@@ -0,0 +1,50 @@
# Task 08 — Phase E2E suite, regression sweep, docs pass, commit
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
One dedicated Playwright file proving verbatim display end to end (including clipboard truth), the named regression suites green in isolation, the docs/README/AGENTS consistency pass, and the single atomic commit.
## Work
1. `tests/e2e/mock_llm.py` — three new scripted branches (module docstring updated; keep every existing branch byte-identical, and re-verify the phase-116 discipline: no new trigger phrase is a substring of, or contained in, any existing trigger or fixture question):
- `_SHOW_TEXT_TRIGGER = "show me the exact text of"` → request 1: a `show_text` tool call whose `path` is the parsed target (whole document, no `offset`/`limit`); the follow-up (request 2 carries the tool result starting with the new module constant `_SHOW_TEXT_CONFIRM_MARKER`, the `_READ_RESULT_PREFIX` precedent) → the final answer: prose + a ` ```bor ` fence carrying the target locator AND a DELIBERATELY CORRUPTED body (one fixture line with a character flipped and a trailing space) + a second fence for a second locator if the fixture has one. The corruption is the test: it must never reach the DOM.
- `_FENCE_ONLY_TRIGGER = "put the block inline"` → NO tool call; prose + one ` ```bor ` fence (resolved purely from the answer text — proves the standalone half of the mechanism).
- `_BAD_LOCATOR_TRIGGER = "show me a block I cannot find"` → NO tool call; prose + a ` ```bor ` fence pointing at a path that does not exist in the KB.
2. Fixture: seed (module-scoped truncate + `import_directory`-style insert, the `test_answer_images.py` idiom) TWO small text documents with distinctive, punctuation-rich content — one with a short line-count and obvious per-line identity (so a clamped range and its line numbers are checkable), one whose content includes a URL, backticks and an em dash (the escaping and the "no retype" proof). Fixture questions (asked by the happy-path spec, disjoint from every other trigger's question): *"show me the exact text of `baselab/quadlets/gufo.container`"*, *"put the block inline for `baselab/notes/verbatim-b.md 2-3`"*.
3. Create `tests/e2e/test_verbatim_show_text.py` (fixtures + style per `tests/e2e/README.md`; App A runs with the phase defaults, App B explicitly `BOR_ANSWER_TEXT=false`):
- **happy path (App A):** ask the show-me question → assert exactly FIVE tools were offered (assert on the request capture, the `test_agent_document_tools.py` idiom); a `.verbatim` panel exists whose caption shows `lines 1-N of N`; the panel's body text equals the fixture's content line-for-line; the badge `VERBATIM` is present; the answer's prose does NOT contain the corrupted line; `.verbatim-pending` is gone at settle; the panel's open link targets `/document?source=…&path=…` (the viewer, where the git origin link lives); the appended-block path renders above `.sources`.
- **clipboard truth:** grant `clipboard-read`/`clipboard-write` for the app origin (skip with a clear reason if the runner refuses), click the panel's Copy, read `navigator.clipboard.readText()`, assert byte equality with the fixture text fetched from the DB/API (no trailing newline, no line numbers); then click a plain code block's Copy in the same answer and assert IT copies exactly the model's block content (no line numbers, no badge).
- **fence only (App A):** the inline locator case resolves with no tool call in the request log, the model's body text is absent from `page.content()`, and no literal ` ```bor ` text remains in the DOM at settle (a half-typed fence must have become the pending state, then the panel).
- **unresolved (App A):** the bad-locator case renders the pinned unresolved copy, no `.verbatim-body`, no `VERBATIM`, and the model's bytes are absent.
- **wrong range corrected (App A):** drive the tool + a fence with an out-of-range `offset` (reuse the show-me trigger with a range suffix) → the panel shows the server's extent, not the model's claim; if the resolver refused the whole show, the honest unresolved state appears and the model's claimed numbers appear nowhere.
- **persistence:** after the happy path, reload `/history` and the open chat → the panel re-renders from the saved record; open the Share link in a second context (or `page.context().new_page()` in an incognito-style context) → the anonymous shared page shows the same panel; assert `/api/chat` for the same turn never emitted a second SSE event type (collect event names, assert `<= 7` distinct and `answer_text` came inside `done`).
- **line-number honesty:** assert the caption's numbers exist as a DOM range (`lines 1-… of …`) and that with a fixture of exactly N lines the `of N` matches — no invented numbers.
- **image regression (this phase's image check, per the owner):** a turn with `answer_image=True` still shows exactly one image bubble and NO `.verbatim` panel; `show_text` pointed at the image document (drive it via the tool branch if the mock can be pointed at the fixture image, else assert at unit level in task 03 and record it here) → the refusal copy naming `show_image`.
- **App B (`BOR_ANSWER_TEXT=false`):** exactly FOUR tools offered; asking the same show-me question yields no `.verbatim` element at all; a ` ```bor ` fence in the answer renders as an ordinary `<pre><code>` block; the `done` frame has no `answer_text` key; the Copy button on ordinary code blocks STILL works (A5 is flag-independent).
- **mobile:** at 390 px the panel scrolls horizontally without stretching the bubble and the Copy control is ≥44 px tall.
- **stale-UI check (owner's standing rule):** the pending → resolved transition never leaves a second panel behind (assert exactly one `.verbatim` per locator).
4. **Regression sweep (run in isolation, all green, no edits to the existing specs):** `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_read_range_tool_lines.py`, `test_read_truncation_cap.py`, `test_tool_path_teaching.py`, `test_tool_scaffolding_guardrails.py`, `test_markdown_tables.py`, `test_markdown_blockquotes.py`, `test_long_answers.py`, `test_history_copy.py`, `test_chat_persistence.py`, `test_stale_ui_copy.py`. If `phase 180_markdown_inline_links_images` is still in `todo/`, run ITS E2E too (it shares `markdown.js`) and record the result.
5. **Docs consistency pass:**
- `README.md` — the feature (what the user sees: panel, `VERBATIM` badge, Copy on every code block), the `show_text` tool + the ` ```bor ` fence in the agent-tools/API sections, the `done.answer_text` field, the three env vars, the "the model points, the server supplies the bytes" principle stated once in plain words.
- `AGENTS.md` — rule 5's UI contract line gains the phase-182 clause (an answer code block always has a Copy button; only a server-resolved block carries `VERBATIM`; a `bor` block's model bytes are never painted); rule 10's log-line list mentions `verbatim_blocks=N`.
- `.agents/PLAN.md` — the tool surface (`show_text` as the 5th tool, gated like `show_image`), the `done` frame's additive field, the new settings; state explicitly that A1–A6 and A15 are otherwise UNCHANGED.
- `git status` must show nothing stale; every file this phase touched is in the commit.
6. **Commit** (mandatory, one atomic commit for the phase — Conventional Commits, `--no-gpg-sign` per repo default):
```bash
git add -A && git commit --no-gpg-sign -m "feat(rag): show file contents verbatim with a show_text tool and locator blocks
- add show_text(path, offset?, limit?): the server slices documents.content
and displays it, so the user copies the KB's bytes instead of a retyped
copy that may differ by a character
- resolve \`\`\`bor source/path 12-40 locator fences in answers server-side;
the model's fence body is never painted, unknown locators say so honestly
- carried in the done frame's additive answer_text array and persisted with
the saved chat, so restore and the shared page re-render identically
- Copy button on every answer code block, VERBATIM badge only on
server-resolved ones; line numbers always come from the server
- gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the flag
off every frame, record and byte budget is exactly pre-phase"
```
Message in English (repo convention).
@@ -0,0 +1,57 @@
# Phase 183 — Mass prune must not load the embeddings it is deleting
**Source:** owner incident 2026-10-15 — "brain-of-code hit 4 GB of RAM while running the sync" (the sync `started repos=4` at 11:19:23, the run reached `_prune` at 12:14:14, pruned **3,008 litellm docs**, and the VM locked up hard enough that boot −1 has no shutdown record at all).
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Context:** `app/models.py` — `Document.chunks` (**L227–229**, `relationship(back_populates="document", cascade="all, delete-orphan")` — **no `passive_deletes`**), `Chunk` (L232), `Chunk.embedding` (L241, `Vector(EMBEDDING_DIM)` = 768), `Chunk.document` (L247); `app/rag/importer.py` — `_prune` (**L3860**, the one-shot `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` at **L3909**, the per-doc `session.delete(doc)` at **L3931**, the single `if pruned: session.commit()` at **L3934–3935**), `_replace_content_chunks` (L992, the `doc.chunks = [...]` ORPHAN-removal path at L1033), `_store_summary` (L1595, the `for old in [c for c in doc.chunks if c.is_summary]` loop at L1648); `app/api/git_sources.py` — `delete_git_source` (L855, the `db.delete(doc)` loop at **L926**, the docstring's "chunks + embeddings via the `all, delete-orphan` cascade" line at **L863**, `docs_pruned` at L919); `app/api/docs.py` — the document-delete endpoint's `db.delete(row)` (L700); `alembic/versions/0001_initial_schema.py` — **L54 `sa.ForeignKey("documents.id", ondelete="CASCADE")`** (the DB already cascades — production confirms: `chunks_document_id_fkey`, `confdeltype='c'`); `tests/unit/test_importer.py` + `tests/integration/test_importer_e2e.py` (the importer families), `tests/integration/test_source_removal_api.py` + `tests/e2e/test_source_removal_cleanup.py` (the removal families), `tests/e2e/test_force_push_resync.py` (the "upstream deleted the file, re-sync prunes it" shape).
## Measured baseline (the numbers this phase is written against; recorded 2026-10-15)
- A loaded 768-dim `Chunk.embedding` costs **24.8 KiB of Python per chunk** (list header + 768 float objects) against **3 KiB on disk** — the ORM round-trip is an 8× expansion, and pgvector hands back a `list[float]`, never a buffer.
- `session.delete(doc)` on a relationship **without** `passive_deletes` makes SQLAlchemy load the child collection first (it must know which children to orphan/delete). Measured against the dev DB with production-shaped rows (400 docs × 25 chunks, ~745-char chunk content): **RSS grows linearly at ~0.24 MB per document deleted** = 25 chunks × ~9.7 KiB resident, +241 MB over 400 deletes, and NOTHING is released before the single commit because `app/db.py:30` builds sessions with `expire_on_commit=False`.
- Applied to the incident run: 3,008 docs × ~24 chunks ≈ **720 MB of float lists materialised to delete 72k rows**, on top of the ~300 MB the same walk already retained — on a 3,904 MB VM with **no swap**, sharing RAM with a 3.1 GB Postgres.
## Objective
Make a mass prune cost O(documents) instead of O(chunks × 24.8 KiB): `Document.chunks` declares `passive_deletes=True` so the DB's already-present `ON DELETE CASCADE` removes the chunk rows without the ORM ever hydrating them, on both mass-delete surfaces (the importer's prune and the admin's source removal), with every user-visible behaviour (counts, log lines, image/page-copy cleanup, the phase-122/126 prune guards, the `changed_paths` feed into the folder summaries) byte-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the pin is statement count, not RSS (task 01).** Memory assertions are flaky; the SQLAlchemy `before_cursor_execute` event stream is not. A new `tests/integration/test_prune_cascade_statements.py` counts, for one prune, every SELECT whose SQL targets the `chunks` table. PRE-fix the count is **exactly one per pruned document** (the lazy child collection load) and that test is GREEN before any code change — it IS the demonstration, exactly as phase 179 task 01 pinned the freeze with a latency probe. The measured RSS delta (the ~0.24 MB/doc figure above) is recorded in the module docstring as the corroborating number, asserted only as a loose ceiling (`< 40 MB` for a 200-doc fixture, where pre-fix measures ≈ 124 MB) so a noisy CI box cannot fail the suite on allocator behaviour.
- **D2 — the fix is one keyword, and it is a semantic claim about the schema (task 02).** `app/models.py:227–229` becomes `relationship(back_populates="document", cascade="all, delete-orphan", passive_deletes=True)`. That is only correct because the FK really cascades — so task 02 also adds a **schema-drift guard test** that reads the live constraint (`pg_constraint` where `conrelid='chunks'::regclass and contype='f'` → `confdeltype == 'c'`) and fails loudly if a future migration ever drops `ON DELETE CASCADE`, because at that point `passive_deletes=True` would silently orphan chunk rows. **No new alembic revision** — 0001 L54 already declares it, and the incident box's live schema matches (A13 stays satisfied because nothing changes).
- **D3 — both mass-delete surfaces, one keyword's blast radius (task 02).** With `passive_deletes=True` the ORM stops hydrating children for EVERY `session.delete(document)`: the importer's `_prune` (L3931), `delete_git_source`'s `db.delete(doc)` loop (`app/api/git_sources.py:926` — the endpoint that took 47 s for 2,303 immich docs on 2026-10-15 11:18:30), and `app/api/docs.py:700`. Audit-and-note, not change: none of the three reads `doc.chunks` AFTER deleting it (verified: `_prune` only touches `image_path`/`page_count`/`source`/`path`; the git-sources endpoint only counts; the docs endpoint only logs). The `doc.chunks = [...]` reassignment inside `_replace_content_chunks` (L1033) is the **orphan-removal** path, which `passive_deletes` does NOT short-circuit — the collection still loads there (a re-embed needs the old rows gone and the new ones built); it runs only for CHANGED files (217 in the incident run), and it is explicitly OUT OF SCOPE (A2) — the docstring says so in one line so the next reader does not "discover" it.
- **D4 — the cascade contract is verified at the row level, not assumed (tasks 01 + 03).** Every prune test asserts `select(count()).select_from(Chunk).where(Chunk.document_id.in_(deleted ids)) == 0` — no orphan chunk rows, before and after the change — plus the existing `documents`-level assertions. This is the phase's non-negotiable: a delete that stops hydrating children must still delete the children.
- **D5 — the E2E pins the owner-visible prune (task 03).** `tests/e2e/test_prune_cascade_cleanup.py` (this phase's dedicated Playwright file, AGENTS rule 4), the `test_force_push_resync.py` + `test_source_removal_cleanup.py` house shapes (dedicated module app, mock LLM, `BOR_SOURCES_DIR`/`BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated before AND after each test): a fixture source of N files is synced to `success` from the Sources page, a subset is deleted from disk upstream, the re-sync's counts line reports `pruned=N-subset` and those documents vanish from the KB (the ls/tree row is gone, `GET /api/docs` 404s the path, the Documents list total drops), while the chunk rows are gone too (the test's own DB query — the same D4 assertion, end to end) and the surviving documents still answer a chat question (a partial prune never damages the RAG). `E2E_REAL_LLM=1` skipped.
- **NOT touched:** the `_prune` batching (phase 184), the fast-path column set (phase 185), `expire_on_commit` (deliberately left `False` — every request path relies on reading attributes after commit), `Chunk.embedding`'s type, the retrieval query, the frontend, and any LLM/vision path.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; phases 184 and 185 build on this one, in that order)
## Tasks
1. `01_prune_cascade_probe.md` — the deterministic statement-count probe (one `chunks` SELECT per pruned doc, PRE-fix, green before the fix) + the row-level orphan assertions + the measured RSS delta recorded in the module docstring.
2. `02_passive_deletes.md` — the fix (`passive_deletes=True`), the live-FK drift guard, the three delete-site audit, the probe flipped to the post-fix assertion (`0` chunk SELECTs), the git-sources endpoint docstring correction.
3. `03_e2e_prune_cascade.md` — `tests/e2e/test_prune_cascade_cleanup.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Integration:** `tests/integration/test_prune_cascade_statements.py` (the statement-count probe, the orphan-row assertions, the loose RSS ceiling + the recorded measurement).
- **Unit:** the existing importer prune pins stay green as written — the phase-122 images-off prune guard, the phase-126 parent-PDF rule, `changed_paths` population, the `summary.pruned` count (`tests/unit/test_importer.py`, `tests/unit/test_importer_ignore.py`); add a unit pin that `Document.__mapper__.relationships["chunks"].passive_deletes is True`.
- **Regression:** `tests/integration/test_importer.py`, `test_importer_e2e.py`, `test_source_removal_api.py`, `test_sync_api.py`, `test_git_sources_api.py` — all green, byte-identical counts and log lines.
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_prune_cascade_cleanup.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the operations note that a prune deletes chunk rows in the DATABASE (FK cascade) and that the operator-visible cost of a mass prune is bounded by documents, not chunks (the number the incident was missing).
- The `_prune` docstring (`app/rag/importer.py:3860`) gains the one-paragraph "why `passive_deletes` + the FK cascade, and why the re-embed path is different" note; `app/models.py:227` gains the one-line "do not remove without re-checking the FK's ON DELETE CASCADE (pinned by tests/integration/test_prune_cascade_statements.py)" warning.
## Completion Criteria
- [ ] `app/models.py::Document.chunks` declares `passive_deletes=True`; a unit pin asserts it.
- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with the POST-fix assertion (zero `chunks` SELECTs during a prune of N docs, `0` orphan chunk rows, RSS delta under the pinned ceiling) and the pre/post measurements + the incident reference in the module docstring.
- [ ] The live-FK drift guard is green and would fail if `ON DELETE CASCADE` were absent (proven by the test's own negative fixture or a documented, executed one-off check).
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
- [ ] `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` green **in isolation** (the counts line, the vanished documents, zero orphan chunks, the surviving KB still answers).
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `fix(models): prune with passive_deletes so deleting documents never hydrates their embeddings`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — mechanism: `passive_deletes=True`, not a bulk-`delete()` rewrite.** The relationship keyword keeps `session.delete(doc)` as the one delete shape every caller already uses (importer prune, source removal, doc delete), so there is no new control flow to get wrong; the bulk-rewrite variant belongs to phase 184's batching, which builds on this.
- **A2 — out of scope: the `doc.chunks = [...]` orphan path** (`_replace_content_chunks` L1033). `passive_deletes` does not cover collection replacement, and a re-embed genuinely needs the loaded collection; it runs for changed files only. Documented as a known, bounded, deliberate non-fix.
- **A3 — no migration.** The FK's `ON DELETE CASCADE` ships in 0001 L54 and is verified live; this phase adds a drift guard instead of a schema change.
- **A4 — the probe is statement-count based.** Deterministic, CI-safe, and it names the exact waste; the RSS figure is documentation plus a loose ceiling, never the assertion that can flake a build.
@@ -0,0 +1,29 @@
# Task 01 — Pin what a prune actually loads (the probe, green PRE-fix)
**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
**Story:** n/a
## Objective
A deterministic integration probe that shows — before any code changes — that deleting documents hydrates their chunk rows (and therefore every 768-dim embedding) into Python, one SELECT per document. The probe is GREEN pre-fix; task 02 flips it. The measured cost is recorded in the module docstring so the next reader never has to re-derive it.
## Work
1. `tests/integration/test_prune_cascade_statements.py` — a NEW file, house patterns:
- Session/DB fixtures per `tests/integration/test_importer.py` (the shared test DB, the per-test truncate, `SessionLocal` from `app.db`).
- A `seed_docs` helper: **200** `Document` rows, `source="cascade-probe"`, each with **25** `Chunk` rows whose `content` is ~745 chars of fixed text and whose `embedding` is a deterministic 768-float vector (`[i / 768 for i in range(768)]` rotated by the doc index — deterministic, and L2-normalised enough not to matter here). Commit.
- **The statement capture:** a `sqlalchemy.event` listener on the engine's `before_cursor_execute` that appends `(statement, params)` tuples for the duration of the test; a helper `chunk_selects(captured)` returns the statements that are `SELECT` and reference `"FROM chunks"` (case-insensitive, `chunks` word-boundary so `documents` never matches). Install in the fixture, remove in teardown (`event.remove`) so no other test in the session sees the listener.
- **The probe test** `test_prune_loads_every_chunk_collection`: build the prune inputs the way `_prune` receives them (`source_names={"cascade-probe"}`, `seen=set()` → everything prunes), call `app.rag.importer._prune(session, source_names, seen, images=False, summary=ImportSummary(), page_dir=None)` (the real signature, `app/rag/importer.py:3860`), then assert:
- `chunk_selects(captured) == 200` — **exactly one child-collection SELECT per deleted document.** This is the PRE-fix assertion and it is GREEN now. Mark it so the flip is unmissable: a module-level `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC` constant with the comment `# 1 pre-phase-183 (the lazy child load); 0 post-phase-183 — task 02 flips this`.
- every `Chunk` row of the 200 documents is gone (`select(count()).select_from(Chunk).where(Chunk.document_id.in_(ids)) == 0`) — the D4 no-orphan contract, asserted before the fix so post-fix it means "unchanged, and not by accident".
- the `documents` rows are gone, `_prune`'s return value is 200, and `ImportSummary.changed_paths` holds the 200 `(source, path)` pairs.
- **The loose memory ceiling** `test_prune_memory_delta_has_a_ceiling`: the same 200-doc prune with `VmRSS` read from `/proc/self/status` before and after (a tiny local helper inside the test module — no new `app/` code in this task), asserting the delta is **< 40 MB**, with the MEASURED value recorded in the docstring. Pre-fix this measures ≈ **48 MB** for 200 docs (200 × 25 × 24.8 KiB ≈ 124 MB of float lists, of which the allocator keeps what it can reuse — the point is the order, not the exact figure), so the pinned ceiling must be picked from the ACTUAL measured pre-fix number: set it to roughly **one third** of what the pre-fix run measures, i.e. a ceiling the pre-fix run VIOLATES if it can be made to (if the pre-fix delta comes in under 40 MB on this box, LOWER the ceiling to a value the post-fix run clears with room and document both numbers — the assertion's job is to catch the hydrate-the-embeddings behaviour, never to fail on allocator noise). If the delta cannot be made discriminating on this hardware, keep the test but assert only `< 40 MB` and state in the docstring that the statement count is the load-bearing pin.
2. **The module docstring (the measurement record):** the incident in one paragraph (the 2026-10-15 sync pruned 3,008 litellm docs; the app cgroup peaked at 3.3 GB twice that week; the VM has 3,904 MB, no swap, and shares RAM with a 3.1 GB Postgres); the mechanism (`app/models.py:227–229` has no `passive_deletes`, so `session.delete(doc)` must load the children; `app/db.py:30` sets `expire_on_commit=False`, so nothing is released before the single commit at `importer.py:3934`); the measured unit costs (**24.8 KiB** of Python per 768-dim embedding vs 3 KiB on disk; **~0.24 MB resident per document deleted** at 25 chunks/doc, measured 2026-10-15 against the dev DB with production-shaped rows); this test's own measured numbers (fill them in from the run); and the verdict line **CONFIRMED — one chunk SELECT per pruned document** (if the probe comes back with 0 selects, STOP and flag it in the docstring and the phase report instead of proceeding).
3. Verify: `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green; `uv run pytest` green (the event listener is scoped to the fixture, nothing else sees it).
## Testing & Quality
- Integration: the probe family above (statement count, orphan-row check, return value, `changed_paths`, the loose RSS ceiling).
- Coverage: **>90%** on `app/` — test-only task, `app/` untouched, so the gate must simply stay green.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 1` (the pre-fix assertion) and the 0-orphan-chunks assertion.
- [ ] The module docstring records the incident, the mechanism, the 24.8 KiB / 0.24 MB-per-doc unit costs, and this run's measured statement count + RSS delta, plus the **CONFIRMED** verdict.
- [ ] `uv run pytest` green; no `app/` change in this task.
@@ -0,0 +1,52 @@
# Task 02 — The fix: `passive_deletes=True`, with the FK contract pinned
**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
**Story:** n/a
## Objective
Stop the ORM from hydrating chunk rows (and their 768-dim embeddings) when a `Document` is deleted: declare `passive_deletes=True` on `Document.chunks` so the database's existing `ON DELETE CASCADE` does the work, prove the schema really cascades with a drift guard, and flip task 01's probe to the post-fix assertion.
## Work
1. `app/models.py` — `Document.chunks` (**L227–229**):
```python
chunks: Mapped[list[Chunk]] = relationship(
back_populates="document",
cascade="all, delete-orphan",
# Phase 183: deleting a Document removes its chunks IN THE DATABASE
# (chunks_document_id_fkey is ON DELETE CASCADE — alembic 0001 L54,
# pinned live by tests/integration/test_prune_cascade_statements.py).
# WITHOUT this keyword the ORM loads every child before deleting it,
# which hydrates every 768-dim embedding (~24.8 KiB of Python each):
# a 3,008-document prune cost ~720 MB and took the 2026-10-15 VM
# (3,904 MB, no swap) down. Do not remove this without re-checking
# the FK — the drift guard fails the suite if the cascade is gone.
passive_deletes=True,
)
```
Nothing else in `models.py` changes (`Chunk.document` L247, `Chunk.embedding` L241 untouched).
2. `app/rag/importer.py` — the `_prune` docstring (L3860): append the phase-183 paragraph — the children go through the FK cascade, so no chunk row is ever SELECTed here; the `changed_paths` / image-copy / page-copy cleanup is unchanged; and the **A2 note**: the re-embed path `_replace_content_chunks` (L992, `doc.chunks = [...]` at L1033) is collection REPLACEMENT, which `passive_deletes` does not cover and which legitimately needs the loaded collection — a known bounded non-fix.
3. `app/api/git_sources.py` — `delete_git_source`'s docstring (**L863**, "documents (chunks + embeddings via the `all, delete-orphan` cascade)") → correct it to name the FK cascade (phase 183): the rows go in the database, the app never loads them. No logic change in the endpoint (verified: the loop at L919–927 only counts `docs_pruned`, it never reads `doc.chunks`).
4. `tests/integration/test_prune_cascade_statements.py` — flip the probe: `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, comment updated (`0 post-phase-183 — the FK cascade; 1 pre-phase-183 was the lazy child load`). Keep every other assertion (0 orphan chunks, 0 documents, the return value, `changed_paths`, the RSS ceiling — now clearing it by a wide margin) and extend the docstring: `CONFIRMED → FIXED (phase 183, passive_deletes=True)` with the post-fix measured statement count and RSS delta.
5. **The drift guard** — same file, `test_chunks_fk_cascades_at_database_level`: query the live schema and assert the cascade exists, so `passive_deletes=True` can never silently orphan rows.
```python
row = session.execute(
text("select confdeltype from pg_constraint "
"where conrelid = 'chunks'::regclass and contype = 'f'")
).all()
assert row and all(r[0] == "c" for r in row) # 'c' = ON DELETE CASCADE
```
Prove the guard is real (not a tautology) in the task's own verification: run it once against a scratch DB where the constraint is recreated `ON DELETE NO ACTION` (a throwaway database the task creates and drops — never the shared test DB) and record in the test's comment that the guard fails there, plus a unit pin `Document.__mapper__.relationships["chunks"].passive_deletes is True` in `tests/unit/test_importer.py`.
6. **Audit note (no code change):** grep `session.delete` / `db.delete` for the `Document` deletes (`app/rag/importer.py:3931`, `app/api/git_sources.py:926`, `app/api/docs.py:700`) and confirm none reads `doc.chunks` after the delete; record the confirmation in the task's completion note. If any site DOES read the collection after deleting, stop and report it rather than papering over it.
7. Verify: `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green post-fix; then the families the keyword touches — `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/integration/test_sync_api.py tests/integration/test_git_sources_api.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the flipped probe + the live-schema drift guard + the five families above, byte-identical counts and log lines.
- Unit: the `passive_deletes` mapper pin in `tests/unit/test_importer.py`.
- Coverage: **>90%** on `app/` (the `app/` change is one keyword + docstrings).
## Completion Criteria
- [ ] `app/models.py::Document.chunks` has `passive_deletes=True` with the explanatory comment; the unit mapper pin is green.
- [ ] `uv run pytest tests/integration/test_prune_cascade_statements.py -v` green with `EXPECTED_CHUNK_SELECTS_PER_PRUNED_DOC = 0`, zero orphan chunks, and the docstring's CONFIRMED → FIXED record with both measured numbers.
- [ ] The drift guard is green on the test DB and documented as failing on a scratch DB whose FK lacks `ON DELETE CASCADE`.
- [ ] `_prune`'s and `delete_git_source`'s docstrings match the new behaviour (the A2 re-embed note included); the three-delete-site audit is recorded.
- [ ] `uv run pytest` green, `app/` coverage >90%, `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,34 @@
# Task 03 — E2E: a mass prune leaves nothing behind (the phase's dedicated Playwright suite)
**Phase:** `183_sync_passive_delete_cascade` · **Source:** owner incident 2026-10-15 — the sync hit 4 GB of RAM while pruning 3,008 documents
**Story:** n/a
## Objective
Pin the fix against the real pipeline the owner actually drives: upstream deletes files from a synced source, the admin re-syncs from the Sources page, and the prune removes exactly the vanished documents, leaves **zero orphan chunk rows**, keeps the surviving KB answerable, and reports the same counts line it always did. Then run the full gates, do the docs pass, and commit the phase.
## Work
1. `tests/e2e/test_prune_cascade_cleanup.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), following the `tests/e2e/test_force_push_resync.py` + `tests/e2e/test_source_removal_cleanup.py` house shapes point for point:
- **Isolation:** a per-module app on a dedicated port (the conftest session app is never started in the isolated run; `tests/e2e/conftest.py` — `APP_PORT` L32, `app_server` L106, `page` L219), the mock LLM on its conftest port, a dedicated `BOR_SOURCES_DIR` and `BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated **before AND after** every test, admin login via `tests/e2e/auth_helpers.py`.
- **Fixture:** a `prune_source` fixture dir with **9** importable `.md` files across two folders (`notes/keep-1..3.md`, `notes/gone-1..6.md`), one of the `keep-*` files carrying the sentinel phrase the chat test asks about.
- **Test 1 `test_deleted_files_prune_without_orphans`:** sync → assert the run reaches `success` and the Sources counts line reports `added=9` / `pruned=0` → delete the six `gone-*.md` files from disk → re-sync → assert the counts line reports `pruned=6` and `added=0` (the unchanged 3 stay unchanged) → the six documents are gone from the KB (the ls/tree row for `notes/gone-1.md` is absent, `GET /api/docs` for the path 404s via `page.request`, the Documents total is 3) → **zero orphan chunk rows** for the deleted document ids (the test's own `psycopg`/SQLAlchemy query against the E2E DB, the D4 assertion) and `select(count()).select_from(Chunk)` equals the sum of the 3 survivors' chunk counts → the run leaves no `documents` row whose `source` lost its file.
- **Test 2 `test_surviving_kb_still_answers_after_a_partial_prune`:** with the same pruned state, ask the chat the sentinel question (the `test_chat_rag.py` shape) and assert the answer cites the surviving sentinel document and the turn completes (no error state, no empty answer) — the prune did not damage retrieval.
- **Test 3 `test_source_removal_prunes_every_document_of_the_source`:** the admin path (`delete_git_source`, `app/api/git_sources.py:855`) through the Git sources UI or `page.request` `DELETE /api/git-sources/{id}` → every document AND every chunk row of that source is gone (the same orphan assertion), `docs_pruned` matches the seeded count via the log/response contract the endpoint already exposes, and the remaining source is intact.
- The module docstring: the incident reference, the fix (`passive_deletes=True` + the FK cascade), the test→observable mapping, the run-in-isolation command, the prerequisites (DB up: `podman compose up -d db`).
2. **Docs pass (the phase's final documentation review):** `README.md` + `AGENTS.md` reviewed in full against the post-phase app and every inconsistency fixed (pre-existing included). Add to `README.md`'s operations/sync section: a prune deletes chunk rows through the database cascade, so the cost of a mass prune scales with documents, not embeddings — and the `chunks` FK's `ON DELETE CASCADE` is load-bearing (pinned by `tests/integration/test_prune_cascade_statements.py`), so it must never be dropped.
3. **The full gates:**
- `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_source_removal_cleanup.py -v --no-cov` and `uv run pytest tests/e2e/test_force_push_resync.py -v --no-cov` — green in isolation (the two prune/removal E2E families this phase must not have broken; run them AFTER the new suite so the DB truncate ordering stays clean).
- `uv run pytest --cov=app --cov-report=term-missing` — the whole suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/models.py`, the two docstring updates, the new integration + E2E files, the unit pin and the `.agents/` tree — e.g. `fix(models): prune with passive_deletes so deleting documents never hydrates their embeddings`, body naming the 2026-10-15 incident and the measured 24.8 KiB/embedding, ~0.24 MB/doc. Move the phase dir to `.agents/phases/complete/183_sync_passive_delete_cascade/`.
## Testing & Quality
- E2E: the dedicated suite above + the two regression E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_prune_cascade_cleanup.py -v --no-cov` green **in isolation** (the counts line, the vanished documents, zero orphan chunks, the surviving KB answers, the source-removal variant clean).
- [ ] `tests/e2e/test_source_removal_cleanup.py` and `tests/e2e/test_force_push_resync.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the cascade-is-load-bearing note included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,59 @@
# Phase 184 — The sync's session stops growing with the knowledge base
**Source:** owner incident 2026-10-15 (as phase 183) — the import run held ONE session (`app/db.py:30` sets `expire_on_commit=False`) for the whole walk + prune, so every row it touched stayed resident until the run ended.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** `183_sync_passive_delete_cascade` (the batched prune deletes documents with the FK cascade doing the child work — it is only cheap because of `passive_deletes=True`).
**Context:** `app/db.py` — **L30 `SessionLocal = sessionmaker(bind=engine, autoflush=False, expire_on_commit=False, future=True)`** (left EXACTLY as it is — every request path reads attributes after commit); `app/rag/importer.py` — `import_sources` (**L708**, `owns_session = session is None` at **L878**, `session = SessionLocal()` at **L880**, the `total` pre-walk at L884–899, the per-source loop at L901, the per-file loop at **L916** with `progress(source, rel, summary.files, total)` at **L943** and `await _index_file(...)` at **L945**, the `EmbeddingError` → `session.rollback()` handler at L951–958, the per-source post-passes `_enrich_pending_images` (**L965**) and `_backfill_image_contexts` (**L972**), `_prune(...)` (**L979**), `session.close()` in the `finally` at L984–985), `_prune` (**L3860**, `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` at **L3909**, `session.delete(doc)` at **L3931**, the single commit at **L3934–3935**), `_delete_image_copy` (takes `image_path: str | None`), `_delete_page_copies(doc, page_dir)` (needs only `doc.id` + `doc.page_count`); `app/api/sync.py` — `import_sources(...)` called with **no `session=`** at **L725** (the importer owns it — this phase never changes that contract); `app/config.py` (the `BOR_`-prefixed `Settings` pattern, `chunk_target_chars` at L228 as the shape), `app/core/settings_catalog.py` — `GROUPS` at **L72** (the `"Import & sources"` group) and the `CATALOG` entry shape at L226, pinned complete by `tests/unit/test_settings_catalog.py::test_catalog_covers_exactly_model_fields` (**`set(CATALOG) == set(Settings.model_fields)`** — a new `Settings` field without a catalogue row FAILS the build); `tests/unit/test_importer.py`, `tests/integration/test_importer_e2e.py`, `tests/integration/test_sync_api.py`, `tests/e2e/test_sync_changed_only.py`, `tests/e2e/test_sync_summary_visibility.py`.
## Measured baseline (2026-10-15, this repo's dev DB, production-shaped rows)
- A `Document`'s `content` averages **15.2 KB** (17,407 docs, 252.6 MB of markdown in the incident DB), and the identity map keeps every one the walk touched — see phase 185 for the column-level fix.
- Objects are released by NOTHING during the run: `expire_on_commit=False` means a commit does not drop attributes, and the importer never expunges. The identity map is therefore the run's memory profile, and it is unbounded: it scales with (files walked) + (chunks created) + (chunks hydrateable per prune — phase 183).
## Objective
Bound the import run's resident set independently of the size of the knowledge base: `session.expunge_all()` on a fixed cadence at the one point where no previously-loaded object is reachable, and `_prune` released in committed batches instead of one transaction at the end of the whole import — with every count, log line, guard, cancel boundary and summary number byte-identical.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the pin is the identity map, not RSS (task 01).** `len(session.identity_map)` is exact, deterministic and it IS the retention surface. `tests/integration/test_import_session_retention.py` runs an import of N files through `import_sources` (the real entry point, the `test_importer_e2e.py` fixture shapes + the mock-embedder pattern from `tests/unit/test_importer.py`) and asserts the map size **at the end of the run**: PRE-fix it is `≈ docs + chunks + documents-touched` (record the measured number), and the test pins `>= N` to prove accumulation. The post-fix assertion (task 02) is `<= reset_files + slack`. RSS is recorded in the docstring only, never asserted (phase 183 A4's rule, inherited).
- **D2 — one new setting per new cadence, catalogue-compliant (tasks 02 + 03).** `sync_session_reset_files: int = 200` (`BOR_SYNC_SESSION_RESET_FILES`) and `sync_prune_batch: int = 500` (`BOR_SYNC_PRUNE_BATCH`). **`0` means "never reset / one transaction" = the exact pre-phase behaviour**, the operator's rollback knob without a re-deploy (the `BOR_LOG_LARGE_RESPONSE_BYTES` precedent). Both get `SettingMeta` entries in the `"Import & sources"` group (the completeness pin makes this mandatory, not optional), both documented in `.env.example` and `README.md`, neither `restart_required` (the importer reads `llm.settings` per run) and neither `secret`.
- **D3 — the reset point is the top of the per-file loop, and nowhere else (task 02).** Inside `import_sources`, at the top of the L916 loop body — after the PREVIOUS file's `_index_file` has returned, before this file's `progress(...)` hook at L943 — call `session.expunge_all()` every `sync_session_reset_files` files (`summary.files % reset == 0`), plus once at each source boundary before the two post-passes (L965/L972) so a source's walk residue never crosses into the next source. Proof of safety that the task must state in the comment: no object from a finished file is reachable afterwards (`_index_file` returns `None`, the progress/chunk/image hooks only assign fields on `_status`, `dates_map`/`ignore` are plain data, and `total` was computed before the loop), and the `EmbeddingError` handler's `session.rollback()` (L951–958) is safe with detached objects because it touches none. The two post-passes select their own rows fresh (`L3241`, `L3391`), so they are unaffected.
- **D4 — the prune is batched on light rows, not streamed cursors (task 03).** `_prune` replaces `select(Document).where(...).all()` with `select(Document).where(...).options(load_only(Document.id, Document.source, Document.path, Document.is_image, Document.image_path, Document.page_count)).order_by(Document.id)` — the exact column set the guards and the copy cleanup need (`is_image` + `path` for the phase-122 toggle guard and the phase-126 `_EMBEDDED_IMAGE_RE` parent rule, `image_path` for `_delete_image_copy`, `id` + `page_count` for `_delete_page_copies`) — and commits every `sync_prune_batch` deletes instead of once at the end (`sync_session_reset_files`'s sibling knob, `0` = the pre-phase single commit). Deletes stay `session.delete(doc)` (the phase-183 `passive_deletes` makes them child-load-free; a bulk `delete()` is NOT used, so `changed_paths`, the log line and the guards keep their exact per-document shape). `yield_per`/server-side cursors are explicitly rejected: a flush inside an open streaming result is a trap, and the light rows are ~1 KB each.
- **D5 — batch commits change the prune's atomicity, deliberately (A2).** Pre-phase a crash mid-prune rolled the whole prune back; post-phase a crash leaves a PREFIX pruned. That is safe here and must be stated in the docstring: `seen` is recomputed from the filesystem on every run, so the next sync converges; the import already commits per file throughout the walk (L1176–1190 and friends), so the run was never one transaction. The `summary.pruned` count is the count of documents actually deleted, unchanged in meaning.
- **D6 — the E2E pins "nothing changed, at scale" (task 04).** `tests/e2e/test_sync_session_reset.py`: a fixture source of **more files than the reset cadence** (the module pins `BOR_SYNC_SESSION_RESET_FILES=3` with 10 files, so the cadence fires repeatedly mid-run), synced from the Sources page → every document present exactly once, the counts line identical to a `BOR_SYNC_SESSION_RESET_FILES=0` run of the same fixture (the byte-identical assertion the phase lives on), a second sync reports all-`unchanged` with `pruned=0`, no 500s and no network-error state on the page, and the KB totals in the Sources stats match the DB. Plus the prune path with the batch pin at 2 (so a multi-batch prune is exercised through the UI).
- **NOT touched:** `app/db.py:30` (the global `expire_on_commit=False` — changing it would re-fetch rows across every request path for no benefit here), the importer's `owns_session` contract, the per-file index/summary/vision logic, the walk's column set (phase 185), `scripts/git_sync.py` (phase 186), the retrieval path, the frontend.
## Dependencies
- `183_sync_passive_delete_cascade` — required: batched `session.delete(doc)` without child hydration is the whole point of the batch cadence.
## Tasks
1. `01_identity_map_probe.md` — the deterministic identity-map probe (an `import_sources` run pinned to accumulate; the measurement recorded) plus the `sync_session_reset_files` / `sync_prune_batch` settings + catalogue + `.env.example` groundwork with the completeness pin green.
2. `02_session_reset_cadence.md` — the `expunge_all` cadence at the per-file loop top + source boundaries, the probe flipped to the bounded assertion, the safety-argument comment.
3. `03_batched_prune.md` — `_prune`'s light `load_only` select + per-batch commits + the atomicity note, the prune-side retention and statement-count pins.
4. `04_e2e_bounded_sync.md` — `tests/e2e/test_sync_session_reset.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the settings defaults + `0`-means-disabled semantics; `tests/unit/test_settings_catalog.py` completeness pin green with the two new fields in `"Import & sources"`; the existing prune guard pins (`images`-off skip, phase-126 parent rule, `changed_paths`) green as written.
- **Integration:** `tests/integration/test_import_session_retention.py` (the identity-map probe, both cadences), `tests/integration/test_importer*.py`, `test_sync_api.py`, `test_source_removal_api.py` — byte-identical counts and log lines.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_sync_session_reset.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md` + `.env.example`: the two new settings (default, `0` = pre-phase behaviour, what each bounds); one operations paragraph stating that the sync's memory is now bounded per-run rather than per-KB, and where the operator sets the ceiling.
## Completion Criteria
- [ ] `sync_session_reset_files=200` and `sync_prune_batch=500` exist on `Settings`, appear in `CATALOG` under `"Import & sources"`, in `.env.example`, in `README.md`; `uv run pytest tests/unit/test_settings_catalog.py -v` green.
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green with the POST-fix bound (identity-map size ≤ `reset_files + slack` with the cadence on, unbounded with it `0`) and both measured numbers in the module docstring.
- [ ] `import_sources` expunges only at the two proven-safe points (documented in the code comment); `_prune` deletes light rows in `sync_prune_batch` commits with the phase-122/126 guards and `changed_paths` intact.
- [ ] Every existing importer/sync/removal test is green with unchanged expectations (the byte-identical contract).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(import): bound the sync session with a reset cadence and batched prune commits`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — mechanism: periodic `expunge_all` + batched prune commits.** Chosen over "new session per source" (would re-derive the importer's `owns_session`/rollback semantics for a smaller win) and over "flip `expire_on_commit=True` for the sync session" (re-reads rows on every commit across a 27k-file walk and relies on implicit expiry instead of an explicit release).
- **A2 — the prune's atomicity changes to per-batch, accepted.** `seen` is recomputed from disk each run so a partial prune converges on the next sync; the import already commits per file, so no run-level transaction existed to lose.
- **A3 — both knobs default ON, `0` restores pre-phase behaviour exactly.** The operator keeps a no-redeploy rollback (the phase-167 `BOR_LOG_LARGE_RESPONSE_BYTES` precedent).
- **A4 — no server-side cursors / `yield_per`.** A flush inside an open streaming result is a correctness trap; the light prune rows cost ~1 KB each, so materialising them is not the problem this phase is fixing.
@@ -0,0 +1,33 @@
# Task 01 — Pin the run's retention (identity map) and lay the two knobs
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15 — the sync's memory grew with the KB, not the work
**Story:** n/a
## Objective
A deterministic integration probe on the real `import_sources` entry point showing that the run's SQLAlchemy identity map grows with every file touched and is never released (PRE-fix, green before the fix), plus the two new settings (`sync_session_reset_files`, `sync_prune_batch`) with their catalogue rows so the cadence tasks have somewhere to read their numbers from.
## Work
1. `app/config.py` — two new `Settings` fields, the `chunk_target_chars` (L228) shape, `BOR_` prefix from `env_prefix` (L38):
- `sync_session_reset_files: int = 200` — the number of files between identity-map releases inside `import_sources`; `0` = never (the pre-phase behaviour). Docstring names the phase and the `0` semantics.
- `sync_prune_batch: int = 500` — the number of pruned documents per commit inside `_prune`; `0` = one commit at the end (the pre-phase behaviour).
Both `int`, both validated to reject negatives (the `show_text_max_*` "negative refuses startup" precedent — follow whatever those fields do; if they validate in a `model_validator`, add these to the same validator).
2. `app/core/settings_catalog.py` — `CATALOG` entries in the **`"Import & sources"`** group (`GROUPS` L72, the group `chunk_target_chars` lives in is `"Chunking & embedding"` — use `"Import & sources"`, these are sync-run knobs): `"Sync session reset files"`, `"Sync prune batch"`. Neither `secret`, neither `restart_required`, both `overridable`. Verify `uv run pytest tests/unit/test_settings_catalog.py -v` — the `set(CATALOG) == set(Settings.model_fields)` pin fails until both entries exist, so this step is proven by that test going green.
3. `.env.example` — both keys with their defaults and a one-line comment each (`0` = pre-phase behaviour).
4. `tests/integration/test_import_session_retention.py` — a NEW file, house patterns (`tests/integration/test_importer_e2e.py` for the fixture-KB + mock-embedder shape, `tests/unit/test_importer.py` for the fake `LLMClient`):
- Fixture: a `sources` tmp dir with **12** `.md` files (each ~4 KB so each produces ≥2 chunks), a `Settings` with `sync_session_reset_files=0` and `sync_prune_batch=0` (the PRE-phase behaviour, explicitly), an isolated truncate of `documents`/`chunks` before and after.
- `test_import_accumulates_everything_it_touched`: run `await import_sources([root], llm, prune=False, session=session)` on a **fresh** session, then assert `len(session.identity_map) >= 12 * 2` (the 12 documents plus their chunks — the assertion that proves nothing is released; pin the exact measured figure in a module constant with the measured value in a comment). Record in the docstring: the map size, the chunk count, and `sum(1 for s in session.identity_map.all_states() if s.class_ is Document)`.
- `test_reset_cadence_is_configurable_off`: with `sync_session_reset_files=0` the map is exactly as the previous test measured (the `0` = pre-phase pin, which stays true after task 02 ships — it is the rollback guarantee, not a pre-fix only assertion).
- A module-level `_map_size(session)` helper (`len(session.identity_map)`) so tasks 02/03 assert through one door.
- Docstring: the incident paragraph, `app/db.py:30`'s `expire_on_commit=False` (which every request path depends on and this phase does NOT change), the measured numbers, the verdict **CONFIRMED — the run's identity map grows monotonically and nothing releases it**.
5. Verify: `uv run pytest tests/integration/test_import_session_retention.py tests/unit/test_settings_catalog.py -v` green; `uv run pytest` green.
## Testing & Quality
- Unit: the two settings' defaults, negative rejection, and the catalogue completeness pin.
- Integration: the retention probe (both cadence-off assertions).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] Both settings exist with the pinned defaults, `0`-means-disabled semantics, catalogue rows in `"Import & sources"`, `.env.example` entries.
- [ ] `uv run pytest tests/unit/test_settings_catalog.py -v` green (the completeness pin proves the catalogue work).
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green with the accumulation assertion and the docstring's measured numbers + **CONFIRMED** verdict.
- [ ] `uv run pytest` green; no behaviour change yet (the settings are read by nothing until tasks 02/03).
@@ -0,0 +1,45 @@
# Task 02 — The fix: release the identity map on a cadence at the safe point
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Bound the import run's resident set: `session.expunge_all()` every `sync_session_reset_files` files at the one point in `import_sources` where no previously-loaded object is reachable, plus once at each source boundary — with every count, log line and error path byte-identical, and task 01's probe flipped to the bounded assertion.
## Work
1. `app/rag/importer.py` — `import_sources` (L708):
- Read the knob once before the source loop: `reset_every = llm.settings.sync_session_reset_files` (`0` → disabled; resolve it ONCE so a mid-run settings overlay cannot change cadence between files).
- **The per-file reset** — at the TOP of the L916 loop body, before `summary.files += 1` / the `progress(...)` call at L943:
```python
# Phase 184 (task 02): the identity-map release. expire_on_commit=False
# (app/db.py:30) means a commit drops NOTHING, so without this the run
# holds every Document and Chunk it touched until the END of the import
# — the 2026-10-15 incident's second driver. This is the ONLY safe
# point: the previous file's _index_file has already returned (it
# returns None), the progress/chunk/image hooks only assign fields on
# the sync's own status object, dates_map/ignore are plain data, and
# `total` was computed before the loop. The EmbeddingError handler
# below calls session.rollback() and touches no object of its own, so
# detached state is irrelevant to it. reset_every == 0 → never
# (the exact pre-phase behaviour, the operator's rollback knob).
if reset_every and summary.files and summary.files % reset_every == 0:
session.expunge_all()
```
- **The source-boundary reset** — immediately before `await _enrich_pending_images(...)` (L965), the same guard (`if reset_every: session.expunge_all()`), with a one-line comment: the two post-passes (L965, L972) select their own rows fresh (L3241, L3391), so no object survives the reset.
- Do NOT reset inside `_index_file`, `_unchanged_doc`, `_store_summary`, or any post-pass — their locals are live across statements.
2. `tests/integration/test_import_session_retention.py` — extend, then flip:
- `test_reset_cadence_bounds_the_map`: the same 12-file fixture with `sync_session_reset_files=3` → after the run `assert _map_size(session) <= 3 + SLACK` (pick `SLACK` from the measurement — it must cover the last partial batch plus whatever a single file's own chunks contribute; document the arithmetic in the comment) and the import still reports `added=12`, the same chunk total, and the same documents as the cadence-`0` run (the byte-identical comparison the phase lives on — compare the two runs' `ImportSummary` fields and the DB's per-path chunk counts).
- Keep `test_import_accumulates_everything_it_touched` and `test_reset_cadence_is_configurable_off` (cadence `0` still accumulates → the rollback guarantee).
- `test_reset_cadence_survives_an_embedding_error`: a stub embedder that raises `EmbeddingError` on file #4 (with `sync_session_reset_files=2`, so a reset has already happened) → the run records `errors=1`, keeps going, and finishes with the other 11 files indexed; asserts nothing blows up on detached state (the handler's `session.rollback()` path).
- Docstring: `CONFIRMED → FIXED (phase 184)`, both measured map sizes (cadence 0 vs cadence 3), and the post-fix bound.
3. Verify: `uv run pytest tests/integration/test_import_session_retention.py -v`; then the families — `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_importer_dates.py tests/integration/test_sync_api.py tests/unit/test_importer.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the bounded-map test, the cadence-`0` guarantee, the error-path test, the byte-identical summary comparison.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `import_sources` calls `session.expunge_all()` at exactly the two documented points, guarded by `reset_every`, with the safety argument in the comment.
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green (bounded with the cadence on, unbounded with it `0`, error path clean, identical summaries).
- [ ] No existing importer/sync test needed an expectation change.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,51 @@
# Task 03 — The prune stops holding 17k full documents and releases in batches
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15 (the prune phase was where the box died)
**Story:** n/a
## Objective
Make `_prune`'s resident set independent of the size of the knowledge base: select only the columns its guards and cleanup need (not `content`, avg 15.2 KB × 17,407 rows), and commit every `sync_prune_batch` deletes instead of one transaction after the whole loop. The per-document semantics — the phase-122 images-off guard, the phase-126 parent-PDF rule, `changed_paths`, the `import: pruned …` log line, the returned count — stay byte-identical.
## Work
1. `app/rag/importer.py` — `_prune` (L3860):
- Replace **L3909** `docs = session.scalars(select(Document).where(Document.source.in_(source_names))).all()` with a light-column, deterministically ordered select:
```python
docs = session.scalars(
select(Document)
.options(
load_only(
Document.id, Document.source, Document.path,
Document.is_image, Document.image_path, Document.page_count,
)
)
.where(Document.source.in_(source_names))
.order_by(Document.id)
).all()
```
with `from sqlalchemy.orm import load_only` added to the module's SQLAlchemy imports, and a comment naming exactly why those six columns (the phase-122 toggle guard reads `is_image`; the phase-126 `_EMBEDDED_IMAGE_RE` parent rule reads `path`; `_delete_image_copy` takes `image_path`; `_delete_page_copies` needs `id` + `page_count`; the `seen` tuple and `changed_paths` need `source` + `path`) and why `content` is not among them (252 MB of markdown per full KB walk; phase 185 does the same job for the fast path).
- Add the batch commit. After `session.delete(doc)` / `pruned += 1` / the `logger.info("import: pruned …")` line, commit on the batch boundary and once at the end:
```python
if prune_batch and pruned and pruned % prune_batch == 0:
session.commit() # phase 184: a long prune no longer holds a huge transaction
...
if pruned:
session.commit()
```
`prune_batch = llm-free`: `_prune` has no `llm` argument, so pass the cadence in as a keyword — `prune_batch: int = 500` on the signature, and the caller at L979 passes `prune_batch=llm.settings.sync_prune_batch`. Keep `prune_batch=0` meaning "single commit at the end" (the pre-phase behaviour, and the default for the standalone `scripts/import_docs.py` path if it calls `_prune` directly — check and keep its call byte-identical by leaving the keyword default at the value that preserves its behaviour).
- Update the `_prune` docstring: the light-row contract, the batch-commit contract, and the **A2 atomicity note** (a crash mid-prune leaves a prefix pruned and the next sync converges, because `seen` is recomputed from the filesystem; the import already commits per file, so no run-level transaction existed).
2. `tests/integration/test_import_session_retention.py` — add the prune side:
- `test_prune_holds_no_content`: seed 30 docs × 5 chunks, prune them all, and assert (a) via a `before_cursor_execute` capture that the documents SELECT issued by `_prune` does NOT list `documents.content` in its column list (the same deterministic technique phase 183 task 01 uses for chunks — assert on the captured statement text), and (b) 0 orphan chunks afterwards.
- `test_prune_batch_commits`: `sync_prune_batch=10` over 30 prunable docs → assert the commit count (a `after_commit`/`after_transaction_end` event counter, or spy `Session.commit` with `monkeypatch`) is 3 (+ whatever the caller adds), that all 30 documents and 150 chunks are gone, and `summary.pruned == 30`; with `sync_prune_batch=0` the commit count is 1 and every other number is identical.
- `test_prune_guards_unaffected_with_light_rows`: the phase-122 images-off guard and the phase-126 parent-PDF rule keep working with `load_only` rows (an `is_image` doc with `images=False` survives; an embedded-image doc whose parent PDF rel is in `seen` survives) — this is the assertion that the six columns really are enough.
3. **Check the deferred-attribute trap explicitly:** anything inside the prune loop that touches a non-loaded attribute would emit a per-row lazy SELECT — the statement capture in `test_prune_holds_no_content` must assert **no per-document follow-up SELECT** (the captured count of `FROM documents` statements stays at 1 for the initial load). If a lazy load appears, the missing column goes into `load_only` (and the comment updates), it does not get papered over with `raiseload`.
4. Verify: `uv run pytest tests/integration/test_import_session_retention.py -v`; then `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_source_removal_api.py tests/unit/test_importer.py tests/unit/test_importer_ignore.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the no-`content` statement pin, the batch-commit pins (both cadences), the guard pins with light rows, the orphan-row pins.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `_prune` selects through `load_only(id, source, path, is_image, image_path, page_count)` ordered by `Document.id`, commits every `sync_prune_batch` (default 500, `0` = single commit), and the docstring records the column rationale + the A2 atomicity note.
- [ ] `uv run pytest tests/integration/test_import_session_retention.py -v` green (no `content` in the prune SELECT, no per-row lazy load, the batch counts, the guards, zero orphans).
- [ ] `scripts/import_docs.py`'s prune path (if any) keeps its pre-phase behaviour.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,35 @@
# Task 04 — E2E: the bounded sync still syncs everything (the phase's dedicated Playwright suite)
**Phase:** `184_sync_bounded_session` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the UI that the phase's two cadences change the run's memory profile and NOTHING else: the same fixture produces the same counts line with the cadence on and off, every document lands exactly once, a re-sync is all-`unchanged`, a multi-batch prune behaves, and the Sources page never stalls or errors. Then the full gates, the docs pass, and the atomic commit.
## Work
1. `tests/e2e/test_sync_session_reset.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_sync_changed_only.py` + `test_sync_summary_visibility.py` house shapes:
- **Isolation:** a dedicated module app + port (`tests/e2e/conftest.py` — `APP_PORT` L32, `app_server` L106, `page` L219), the mock LLM, dedicated `BOR_SOURCES_DIR` / `BOR_PDF_PAGE_DIR` under `tmp_path_factory`, the shared E2E Postgres truncated before AND after every test, admin login via `tests/e2e/auth_helpers.py`.
- **Fixture:** 10 `.md` files across two folders (`a/1..5.md`, `b/1..5.md`), one carrying the sentinel phrase.
- **Test 1 `test_reset_cadence_produces_the_identical_run`:** module sets `BOR_SYNC_SESSION_RESET_FILES=3` (so the cadence fires 3× through a 10-file walk). Sync from the Sources page → `success`; the counts line reports the 10 added (the phase-98 counts-line contract, `test_sync_summary_visibility.py` shape); the KB totals match `select(count())` on `documents` (10) and on `chunks` (>10); no network-error state and no 500 during the run's polls.
- **Test 2 `test_resync_is_unchanged_with_the_cadence_on`:** a second sync from the same fixture (nothing changed on disk) → `unchanged=10`, `added=0`, `pruned=0`, `success`, and the DB's chunk total is unchanged (no silent re-embed — the cadence must not be losing rows and re-adding them).
- **Test 3 `test_multi_batch_prune_from_the_ui`:** `BOR_SYNC_PRUNE_BATCH=2` (module-level env), delete 5 of the 10 files from disk, re-sync → the counts line reports `pruned=5`, the 5 documents are gone from the KB and from the ls/tree rows, the survivors' chunk total is unchanged, and 0 orphan chunk rows exist (the test's own DB query). The batch cadence therefore commits 3 times through the UI's own run.
- **Test 4 `test_cadence_zero_matches_cadence_three` (the byte-identical cross-check):** re-seed the fixture, run with `BOR_SYNC_SESSION_RESET_FILES=0` and compare the resulting `documents`/`chunks` per-path counts against test 1's recorded numbers (the test records its numbers into a module-level fixture or asserts against a fixed expected table — whichever is deterministic in this suite's house style; the assertion must be a real comparison, not a tautology). The two runs must agree on every path, every chunk count, and the counts line.
- Module docstring: the incident reference, the two knobs + their defaults + `0` semantics, the test→observable mapping, the run-in-isolation command, the prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` reviewed in full against the post-phase app; every inconsistency fixed, pre-existing included. `README.md` gains the operations paragraph (the sync's memory is now bounded per run; the two knobs; `0` restores the old behaviour) and `.env.example` matches the shipped defaults.
3. **The full gates:**
- `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` and `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` — green in isolation (the sync-suite neighbours this phase must not disturb; run after the new suite).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/config.py`, `app/core/settings_catalog.py`, `app/rag/importer.py`, the integration + E2E files, `.env.example`, the docs and the `.agents/` tree — e.g. `perf(import): bound the sync session with a reset cadence and batched prune commits`. Move the phase dir to `.agents/phases/complete/184_sync_bounded_session/`.
## Testing & Quality
- E2E: the dedicated suite above + the two regression E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_session_reset.py -v --no-cov` green **in isolation** (identical run with the cadence on, unchanged re-sync, multi-batch prune, the cadence-0 vs cadence-3 cross-check).
- [ ] `tests/e2e/test_sync_changed_only.py` and `tests/e2e/test_sync_git_freeze.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md`, `AGENTS.md` and `.env.example` accurate and consistent with the whole project.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,55 @@
# Phase 185 — The unchanged fast path stops fetching 15 KB of markdown per file
**Source:** owner incident 2026-10-15 (as phases 183/184) — every one of the ~27k files the walk inspected hydrated its `Document` row, `content` included, and `expire_on_commit=False` kept it resident for the whole run.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; sequenced after `183`/`184` because it touches the same walk and reuses their statement-capture technique.
**Context:** `app/rag/importer.py` — `_index_file` (**L1384**; the row load **`doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))` at L1452** — the ONLY place the fast path gets its row, `_file_hash` streamed digest at L1451, the fast-path guard at L1460–1469, the three sibling loads at **L1914** (PDF), **L2675** (embedded image), **L3674** (image) which the phase-147 fast path shares), `_row_kind_mismatch` (**L1352** — reads `doc.is_image` only), `_unchanged_doc` (**L1061** — `doc.file_hash = file_digest` at L1137, the **phase-118 summary-NULL backfill `if doc.summary is None: await _store_summary(..., content=doc.content ...)` at L1139–1142**, `_regate_strip` at L1145–1157 which reads `doc.page_count` + `doc.path` + `doc.full_path` for a `.pdf` with a strip, the D1 manual-date lock at L1163, the D4 date refresh at L1172), `_store_summary` (L1595), `ImportSummary.log()` (L520); `app/db.py:30` (`expire_on_commit=False`, unchanged); `tests/unit/test_importer_unchanged_fast_path.py` (the phase-147 matrix, whose docstring already pins "the phase-118 backfill on the fast path runs from the STORED `doc.content`"), `tests/unit/test_importer.py`, `tests/integration/test_importer.py`, `tests/integration/test_importer_e2e.py`, `tests/e2e/test_document_viewer.py` + `tests/e2e/test_document_summaries.py` (the surfaces that render stored `content` and summaries), `tests/e2e/test_sync_changed_only.py`.
## Measured baseline (2026-10-15, incident box)
`documents`: 17,407 rows, `avg(length(content)) = 15,217` chars, **252.6 MB of markdown total**. `_index_file` loads the whole row for every file it inspects — the incident run inspected ~27k files and re-indexed **217**, so the run paid ~250 MB of markdown resident (`expire_on_commit=False`) to discover that 99% of the KB had not changed.
## Objective
Keep the phase-147 unchanged fast path's behaviour exactly as it is and make it stop hydrating `Document.content`: the row load defers the one column the fast path does not read, so a full walk's memory scales with row metadata instead of markdown — while the phase-118 summary-NULL backfill (the one fast-path branch that genuinely needs the stored content) keeps working from the stored text.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `defer(Document.content)`, not a `load_only` allowlist (task 02).** The surgical form: `select(Document).options(defer(Document.content)).where(...)` at the row loads. A `load_only` allowlist would defer every column not named — including columns added by a future phase, which is how a "memory fix" turns into a mysterious N+1 six phases later. `defer()` names exactly what is dropped, so everything else keeps today's semantics, and pyright/the suite stay honest. Applies to the shared row load at **L1452** and the three sibling loads that share the phase-147 fast path — **L1914** (PDF), **L2675** (PDF-embedded image), **L3674** (image) — via ONE module-level helper so the four sites cannot drift: `_DOC_ROW_OPTS: tuple[Callable[[str], Any], ...] = (defer(Document.content),)` (or a tiny `_doc_row_stmt(source, rel)` helper returning the prepared `select` — the executor picks one, and a unit pin asserts every fast-path load uses it).
- **D2 — NO `raiseload`.** `noload`/`raiseload` would break the phase-118 backfill (L1139–1142) and any future legitimate reader. With plain `defer`, reading `doc.content` emits one lazy SELECT for that row and behaves exactly as today. The trade is explicit: unchanged files with a stored summary pay ZERO extra queries; a summary-NULL backfill pays ONE extra SELECT, which is a per-row, one-time, already-rare cost. The docstring says so.
- **D3 — the pin is the statement's column list, deterministic (task 01).** `tests/integration/test_import_fast_path_columns.py` captures SQL through `before_cursor_execute` (the phase-183 task 01 technique) and asserts, for an unchanged re-sync of a seeded KB: (a) the documents SELECT issued by `_index_file` does **not** mention `documents.content` in its select list — PRE-fix that assertion FAILS, so task 01 writes it as the *inverse* assertion (content IS present, green pre-fix) and task 02 flips it, exactly the 179 pattern; (b) exactly ONE documents SELECT per file (no lazy follow-up) for a doc WITH a summary; (c) for a doc with `summary IS NULL`, the backfill still runs and emits exactly ONE extra documents SELECT (the lazy `content` load) and produces the same summary chunk as today.
- **D4 — observable parity is non-negotiable.** Every count (`files/added/updated/unchanged/dates_updated`), every log word (`import: unchanged …`, `import: date-refreshed …`, `import: summary …`), the D1 manual-date lock, the phase-163 PDF strip re-gate and the phase-118 backfill result stay byte-identical — this phase changes WHICH COLUMNS travel, never what the run does. `tests/unit/test_importer_unchanged_fast_path.py` must go green with **no expectation edits** (if an expectation must change, the design is wrong — stop and flag).
- **D5 — the E2E pins the two surfaces `content` feeds (task 03).** `tests/e2e/test_fast_path_deferred_content.py`: (1) a doc whose `summary` is NULL in the DB (the test nulls it directly, the way phase 118's tests do) is re-synced from the Sources page and its summary comes back from the STORED content (the document page's summary renders + the summary chunk exists + `dates_updated`/counts stay honest); (2) a normal unchanged doc's viewer still shows the FULL stored text (no truncation, no blank) after a sync that ran with the deferral; (3) a changed file re-indexes and its viewer shows the NEW text. `E2E_REAL_LLM=1` skipped.
- **NOT touched:** `app/db.py:30`, the walk itself (`iter_importable_files`, L624 — its `sorted(root.rglob("*"))` materialisation is a separate, much smaller cost and stays), `_unchanged_doc`'s logic, `_store_summary`, the index/summary/vision pipelines, the retrieval path, the frontend, phases 183/184's files beyond the shared row-load line.
## Dependencies
— (none functionally; run after `183_sync_passive_delete_cascade` and `184_sync_bounded_session` so the walk's three memory fixes land in one coherent story)
## Tasks
1. `01_fast_path_statement_probe.md` — the statement-capture probe pinning that the fast-path row load hydrates `content` (PRE-fix, green now), the zero-extra-SELECT pin for a doc with a summary, and the backfill's lazy-load pin.
2. `02_defer_content.md` — the fix: the shared `_doc_row_stmt` / `defer(Document.content)` helper wired into the four fast-path row loads, the docstrings, the probe flipped.
3. `03_e2e_deferred_content.md` — `tests/e2e/test_fast_path_deferred_content.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_importer_unchanged_fast_path.py` green with **no expectation edits** (the phase-147 matrix: zero-LLM-call, D1/D4 date rules, the phase-118 backfill from stored content); `tests/unit/test_importer.py` green.
- **Integration:** `tests/integration/test_import_fast_path_columns.py` (the column-list + statement-count pins), `tests/integration/test_importer.py`, `test_importer_e2e.py`, `test_importer_dates.py`, `test_sync_api.py`.
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_fast_path_deferred_content.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md` (sync/import operations): one sentence — the unchanged fast path reads row metadata only, so a full-resync walk's cost is metadata-bound; the rare summary-NULL backfill re-reads the stored content (one query) and that is intentional.
- `app/rag/importer.py`: the module docstring's phase-147 fast-path paragraph and `_index_file`'s docstring gain the deferral note (with the D2 no-`raiseload` rationale), so nobody "optimises" it into a lazy-load landmine.
## Completion Criteria
- [ ] The four fast-path row loads share ONE deferral helper; `Document.content` is deferred and nothing else is.
- [ ] `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green with the POST-fix column-list assertion (no `documents.content` in the fast-path SELECT), the one-SELECT-per-file pin, and the backfill's single lazy-load pin, all measured numbers in the module docstring.
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py -v` green with zero edited expectations.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` green **in isolation** (backfill from stored content, full text in the viewer, changed file re-indexes).
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(import): defer Document.content on the unchanged fast path so a full walk no longer hydrates the KB`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — `defer(Document.content)` only.** Not a `load_only` allowlist (future columns would silently become lazy), not `undefer_group`s, not a DTO/read-model rewrite of the importer (too large a change to keep phase-147 parity provable).
- **A2 — no `raiseload`/`noload`.** The phase-118 backfill must keep reading the stored content; failing loudly there would be a behaviour regression, and the extra SELECT is per-row and one-time.
- **A3 — the walk itself (`iter_importable_files`) is out of scope** this phase — its `sorted(root.rglob("*"))` list is a one-off per source (tens of thousands of `Path`s, ~10 MB), orders of magnitude below the markdown this phase removes.
@@ -0,0 +1,27 @@
# Task 01 — Pin that the fast path hydrates the markdown it never reads
**Phase:** `185_sync_defer_content_fast_path` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
A deterministic, SQL-level probe proving that the phase-147 unchanged fast path SELECTs `documents.content` for every file it inspects (green PRE-fix, task 02 flips it), that it issues exactly one documents SELECT per unchanged file, and that the phase-118 summary-NULL backfill path behaves as documented.
## Work
1. `tests/integration/test_import_fast_path_columns.py` — a NEW file, house patterns (`tests/integration/test_importer_e2e.py`'s fixture-KB + fake-embedder shape; the statement capture is phase 183 task 01's: a `sqlalchemy.event` `before_cursor_execute` listener installed and removed in a fixture).
- Fixture: 4 `.md` files (each ~8 KB so `content` is unmistakably large), seeded through the real `import_sources` so rows look production-shaped, then a SECOND `import_sources` run over the same tree (the unchanged run) with the capture active. One of the four rows gets `documents.summary = NULL` committed before the second run (the phase-118 backfill case).
- Helpers: `doc_selects(captured)` → statements that are SELECTs `FROM documents`; `loads_content(stmt_text)` → True when the statement's select list names `content` (match on the aliased/plain column token, and state the matcher in a comment so it cannot match `content_hash`).
- `test_unchanged_run_hydrates_content_pre_fix`: assert `len(doc_selects(captured)) >= 4` and `all(loads_content(s) for s in doc_selects(captured))` — **the PRE-fix assertion**, with the marker comment `# task 02 flips this: no content in the fast-path SELECT`. Record the captured statement text for ONE file in the docstring (the evidence).
- `test_unchanged_file_is_one_select`: for a doc WITH a summary, the per-file documents statement count is exactly **1** (this pin must hold before AND after the fix — post-fix it is the guard against the deferral turning into a per-row lazy-load storm).
- `test_summary_null_backfill_reads_stored_content`: for the summary-NULL row, assert the second run produced a summary chunk (`is_summary` chunk exists, its content matches the stored markdown) and, before the fix, no extra documents SELECT is needed; structure the assertion so task 02's post-fix expectation is a documented constant: `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 0` → task 02 sets it to `1` (the single lazy `content` load) with the arithmetic in the comment.
2. Docstring: the measured baseline (17,407 rows, avg 15,217 chars, 252.6 MB), the mechanism (`_index_file` L1452 loads the whole row; `app/db.py:30` `expire_on_commit=False` keeps it), the captured statement evidence, and the verdict **CONFIRMED — the fast path SELECTs `content` for every file inspected**.
3. Verify: `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green; `uv run pytest` green.
## Testing & Quality
- Integration: the three pins above.
- Coverage: **>90%** on `app/` (test-only task).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green with the PRE-fix content-in-SELECT assertion and `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 0`.
- [ ] The one-SELECT-per-file pin is green and marked as must-stay-green post-fix.
- [ ] The docstring carries the baseline numbers, the captured statement, and the **CONFIRMED** verdict.
- [ ] `uv run pytest` green; no `app/` change in this task.
@@ -0,0 +1,53 @@
# Task 02 — The fix: defer `Document.content` on the fast-path row loads
**Phase:** `185_sync_defer_content_fast_path` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Make the unchanged fast path read row metadata instead of markdown: one shared statement helper that defers `Document.content` (and nothing else), wired into the four row loads that share the phase-147 fast path, with task 01's probe flipped and zero behaviour change.
## Work
1. `app/rag/importer.py` — add ONE module-level helper next to `_file_hash` (L687):
```python
def _doc_row_stmt(source: str, rel: str) -> Select[tuple[Document]]:
"""The importer's single Document-row statement (phase 185).
``defer(Document.content)`` — the unchanged fast path (phase 147) reads
file_hash/is_image/page_count/summary/dates, never the stored markdown,
and app/db.py builds sessions with expire_on_commit=False, so hydrating
content for EVERY inspected file is what took the 2026-10-15 run to
~250 MB on a 17k-document KB (avg 15,217 chars/row). ONE column is
deferred on purpose: a load_only allowlist would silently defer every
column a later phase adds, and the deferred read would then surface as
an unexplained N+1. No raiseload/noload: the phase-118 summary-NULL
backfill (``_unchanged_doc``) genuinely reads ``doc.content`` and must
keep working — it costs one extra SELECT for that one row.
"""
return (
select(Document)
.options(defer(Document.content))
.where(Document.source == source, Document.path == rel)
)
```
with `from sqlalchemy import select` already present and `from sqlalchemy.orm import defer` added (import style: match the module's existing import block).
2. Replace the four identical fast-path row loads with `session.scalar(_doc_row_stmt(source, rel))`:
- `_index_file` **L1452** (the text/branch dispatch),
- `_index_pdf_file` **L1914**,
- `_index_one_embedded_image` **L2675** (the `image_doc` variant — same statement shape, different variable name),
- `_index_image_file` **L3674**.
Keep every surrounding line untouched (`file_digest` computed first, the guard, the branch bodies). Grep for any OTHER `select(Document).where(Document.source ==` in the importer to confirm no fifth site exists; if one does (e.g. the post-passes at L3241/L3391 select differently shaped rows), leave it and say so in the task note — those are NOT fast-path loads.
3. Docstrings: the module docstring's phase-147 paragraph and `_index_file`'s docstring (L1440–1470 region) gain the deferral sentence; `_unchanged_doc`'s step-3 comment (the phase-118 backfill) gains "…which is the one fast-path read that pulls the deferred `content` back — one SELECT, one time, by design".
4. `tests/integration/test_import_fast_path_columns.py` — flip: `EXPECTED_CONTENT_IN_FAST_PATH_SELECT = False` (assert NO `content` token in the fast-path documents SELECT list) and `EXPECTED_EXTRA_DOC_SELECTS_FOR_BACKFILL = 1`; keep the one-SELECT-per-file pin green; docstring `CONFIRMED → FIXED (phase 185, defer(Document.content))` with the post-fix captured statement text (the evidence that markdown no longer travels).
5. `tests/unit/test_importer_unchanged_fast_path.py` — must go green **without editing an expectation**. If a unit test stubs the session (no real lazy load possible) and fails on the deferral, fix the TEST HARNESS honestly (e.g. the fake returns a real `Document` instance, which already has `content` set) rather than weakening a pin — and say in the task note which harness detail was adjusted.
6. Verify: `uv run pytest tests/integration/test_import_fast_path_columns.py tests/unit/test_importer_unchanged_fast_path.py -v`; then `uv run pytest tests/integration/test_importer.py tests/integration/test_importer_e2e.py tests/integration/test_importer_dates.py tests/integration/test_import_extensions_env.py tests/integration/test_sync_api.py -v`; then `uv run pytest`.
## Testing & Quality
- Integration: the flipped column pin, the one-SELECT pin, the single-lazy-load backfill pin.
- Unit: the phase-147 matrix green, zero edited expectations.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `_doc_row_stmt` exists and all four fast-path loads use it (grep evidence in the task note); only `Document.content` is deferred.
- [ ] `uv run pytest tests/integration/test_import_fast_path_columns.py -v` green post-fix with both flipped constants and the post-fix captured statement in the docstring.
- [ ] `uv run pytest tests/unit/test_importer_unchanged_fast_path.py -v` green with zero edited expectations (any harness adjustment documented).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,32 @@
# Task 03 — E2E: the deferred column still shows up where the owner reads it (the phase's dedicated Playwright suite)
**Phase:** `185_sync_defer_content_fast_path` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the UI that the deferral is invisible to the owner: the phase-118 summary-NULL backfill still produces a summary from the STORED content, an unchanged document's viewer still renders its full stored text, and a changed file re-indexes to the new text. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_fast_path_deferred_content.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_document_summaries.py` / `test_document_viewer.py` / `test_sync_changed_only.py` house shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, the shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`).
- **Test 1 `test_summary_null_backfill_survives_the_deferral`:** seed+sync a fixture with 2 files, then set `documents.summary = NULL` for one row directly (the way phase 118's tests do), re-sync from the Sources page → the run reports `unchanged` (not `added`/`updated`) and the document page shows a non-empty summary; the `is_summary` chunk exists for that document; exactly one summary chunk (no duplicate).
- **Test 2 `test_unchanged_document_viewer_shows_full_stored_text`:** re-sync without touching the files → open the document view and assert the rendered text equals the stored `content` (the sentinel phrase at the START and the END of the file both present, no truncation, no empty body) — the deferral must not have made the viewer's source row thinner.
- **Test 3 `test_changed_file_reindexes_under_the_deferral`:** rewrite one file on disk (new sentinel + changed length), re-sync → the run reports `updated=1`, the viewer shows the NEW text and no stale fragment, the chunk count changes, and a third sync reports all `unchanged` (the fast path still fires after a deferral-mediated write).
- Module docstring: the incident reference, the deferral (`defer(Document.content)`, one column, no `raiseload` — A1/A2), the test→observable mapping, the run-in-isolation command, prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` reviewed in full against the post-phase app; every inconsistency fixed, pre-existing included. `README.md` gains the one-sentence operations note (a full re-sync walk is metadata-bound; the summary-NULL backfill intentionally re-reads the stored content once).
3. **The full gates:**
- `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` and `uv run pytest tests/e2e/test_sync_changed_only.py -v --no-cov` — green in isolation (the fast path's UI neighbours).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `perf(import): defer Document.content on the unchanged fast path so a full walk no longer hydrates the KB` — covering `app/rag/importer.py`, the integration + E2E files, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/185_sync_defer_content_fast_path/`.
## Testing & Quality
- E2E: the dedicated suite + the two regression E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_fast_path_deferred_content.py -v --no-cov` green **in isolation** (backfill, full text, changed-file re-index).
- [ ] `tests/e2e/test_document_dates.py` and `tests/e2e/test_sync_changed_only.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,68 @@
# Phase 186 — The commit-date walk streams instead of buffering the whole `git log`
**Source:** owner incident 2026-10-15 (as phases 183–185) — `_run_sync` builds `doc_dates_by_root` from a full-history `git log --name-only` captured ENTIRELY in memory, per source, and keeps the result for the whole run.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; sequenced after 183–185 so the sync's memory work lands together.
**Context:** `scripts/git_sync.py` — the module docstring's **A11 "every git command goes through `run_git`" claim at L31**, `clone_or_pull` (L96), `_unshallow_if_shallow` (L221), **`_parse_commit_dates(output: str)` (L240)** — `for line in output.splitlines()` at L245-ish, the first-sighting-wins `dates.setdefault(path, commit)`, the `ValueError` paths; **`file_commit_dates` (L270)** — `output = run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)` (L298), the two fail-soft `except` arms returning `{}` with the `file_commit_dates: …` warnings, `run_git` (**L320**, `subprocess.run(argv, cwd=cwd, capture_output=True, text=True, env=env)` at **L342**, `GitSyncError` with `proc.stderr` on non-zero); `app/api/sync.py` — `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)` (**L646**, phase 179), the dict's consumer `dates_map = (doc_dates_by_root or {}).get(str(root), {})` in `app/rag/importer.py` (**L913**), `raw_date=dates_map.get(rel)` passed to `_index_file` (**L945–948**), the mtime fallback `file_mtime_datetime` (`app/rag/doc_dates.py` L57) and `normalize_doc_date`; `app/rag/doc_dates.py` (pure, unchanged); `tests/unit/test_git_sync.py` (the `_parse_commit_dates` + argv pins, which call it with **strings**), `tests/integration/test_git_file_dates.py`, `tests/integration/test_importer_dates.py`, `tests/e2e/test_git_source_dates.py`, `tests/e2e/test_document_dates.py`.
## Measured baseline (2026-10-15, the incident box's own checkouts, read-only `git` inside the app container)
| checkout | commits | `git log --name-only --format=@@%cI` lines | bytes |
|---|---|---|---|
| litellm | 54,502 | 346,829 | 15,077,351 |
| llama.cpp | 11,659 | 69,922 | 1,867,046 |
| vllm-radiance | 192 | 4,855 | 387,672 |
| gufo | 589 | 9,155 | 327,032 |
`run_git` returns that as ONE `str`; `_parse_commit_dates` then calls `.splitlines()`, materialising a second copy as a list of 346,829 `str` objects — tens of MB of transient peak per source on a 3,904 MB box, before the retained dict. The source removed that morning (`server`, nextcloud) is the case that shows how unbounded this is: the walk scales with the repository's ENTIRE history, not with its working tree.
## Objective
Stream the walk: parse the `git log` output line by line from the subprocess pipe so the run never holds the whole output, keep the phase-106 D2 contract byte-for-byte (first-sighting-wins last-commit dates, the same fail-soft arms, the same warning lines, mtime fallback), and amend the A11 single-site claim so the streaming variant is a documented, tested sibling rather than an exception hiding in the module.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `stream_git_lines(argv, cwd)` is the second git site, and A11 says so (task 02).** A generator in `scripts/git_sync.py`:
```python
def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
"""Stream one git command's stdout, line by line (phase 186, A11 sibling)."""
```
`subprocess.Popen(argv, cwd=cwd, stdout=PIPE, stderr=PIPE, text=True)`; stdout is iterated directly (`for line in proc.stdout`); **stderr is drained by a daemon thread into a list** so a chatty stderr can never deadlock the pipe while stdout is being consumed (the classic `Popen` + two-pipes trap — this is the reason not to `communicate()` at the end and not to `stderr=STDOUT`, which would pollute the parse). On `proc.wait() != 0` it raises `GitSyncError(f"git {' '.join(argv[1:])} failed (exit …): {stderr}")` — the SAME message shape as `run_git` (L344–347), so every caller's fail-soft handling is unchanged. The module docstring's L31 claim becomes: `run_git` is the buffered site, `stream_git_lines` the streaming one, and **both** are the only places a `subprocess` is spawned (a unit pin greps `subprocess.` occurrences in the file — the phase-179 AGENTS rule 11 "one git-invocation site" intent survives as "one for each shape, both owned here").
- **D2 — the parser takes any iterable; strings keep working (task 01 first).** `_parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]`: `if isinstance(lines, str): lines = lines.splitlines()` and everything else — the `@@%cI` commit-header rule, the whitespace path split, the `\`→`/` normalisation, the leading-`/` strip, `setdefault` first-sighting-wins, both `ValueError` arms — stays byte-identical, so `tests/unit/test_git_sync.py`'s existing string pins need no edits.
- **D3 — `file_commit_dates` consumes the stream, fail-soft unchanged (task 02).** `return _parse_commit_dates(stream_git_lines([...], cwd=dest))` inside the existing `try`, the existing `except GitSyncError` → the same warning + `{}`, the existing `except ValueError` → the same warning + `{}`. Two consequences documented in the docstring, not hidden: a mid-stream failure now surfaces AFTER some lines were parsed, and the method still returns `{}` because the caller falls back to mtimes (the phase-106 D2 fail-soft contract is about the RESULT, and the result is unchanged); and the returned dict — one entry per path in the repository's history — remains retained by `_run_sync` in `doc_dates_by_root` for the run (A3: out of scope; the dict is ~10% of the buffered cost this phase removes, and bounding it changes dates, not just memory).
- **D4 — the pins are comparative and deterministic (tasks 01 + 02).** Unit: a one-shot generator input proves single-pass consumption (a generator that raises on a second `iter()`), and the lazy-vs-buffered assertion measures `tracemalloc` peak for the SAME synthetic corpus (≈300k lines) through (a) one big `str` + `splitlines()` and (b) a generator, asserting `streamed < buffered * 0.6` with both numbers printed in the failure message. Integration: `tests/integration/test_git_file_dates.py` extended with a generated two-commit repo proving the post-fix dates equal the pre-fix dates for (i) a file only in commit 1, (ii) a file modified in commit 2 (its date is commit 2's), (iii) a path in a subdirectory, (iv) a file absent from history entirely (mtime fallback).
- **D5 — the E2E pins the owner-visible date (task 03).** `tests/e2e/test_git_log_stream_dates.py`: a local `file://` fixture repo with two dated commits (the `test_git_source_dates.py` `_git` helper shape, `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` pinned so the dates are exact), synced from the Sources page; the document date shown for the modified file equals commit 2's date and the untouched file shows commit 1's date (the phase-106 contract, now through the stream), a file git never committed falls back to its mtime, and the D1 manual-date lock still survives a re-sync. Whatever the walk currently does with a quoted non-ASCII path is recorded in the module docstring as-is and pinned as UNCHANGED behaviour (this phase does not change the parser's quoting semantics — it only changes how the bytes arrive).
- **NOT touched:** the `git log` ARGV (`--name-only --format=@@%cI` stays, pinned by the existing argv unit tests), `clone_or_pull` / `_unshallow_if_shallow` (they stay on `run_git`), `app/rag/doc_dates.py` (pure), the importer's `dates_map.get(rel)` lookup, the mtime fallback, the sync's cancel semantics (phase 134/179), the frontend.
## Dependencies
— (none functionally; run after `183`/`184`/`185`)
## Tasks
1. `01_parser_accepts_lines.md` — `_parse_commit_dates` takes `str | Iterable[str]` with byte-identical semantics, the one-shot-generator + comparative-memory unit pins, the existing string pins untouched.
2. `02_stream_git_log.md` — `stream_git_lines` (Popen + stderr thread + the `run_git`-shaped `GitSyncError`), `file_commit_dates` on the stream, the A11 docstring amendment, the grep pin, the integration date-parity tests.
3. `03_e2e_streamed_dates.md` — `tests/e2e/test_git_log_stream_dates.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_git_sync.py` — existing string pins green unedited; the new iterable/one-shot/lazy-memory pins; the `subprocess.` occurrence grep pin.
- **Integration:** `tests/integration/test_git_file_dates.py` (date parity on a generated repo, the fail-soft arms still returning `{}` with the same warnings), `tests/integration/test_importer_dates.py`, `test_import_docs_git.py`, `test_sync_api.py` (the `file_commit_dates` monkeypatch points keep working — `app.api.sync` imports the name, phase 179's probe still patches it).
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_git_log_stream_dates.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` plus full coverage of `scripts/git_sync.py`'s new function (the existing `tests/unit/test_git_sync.py` family covers this module today — keep it covered, not just `app/`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `AGENTS.md` rule 11 (the "never block the event loop" rule already cites `run_git`): one clause noting the streaming sibling exists and that BOTH are off-loop via `asyncio.to_thread` (phase 179) — do not rewrite the rule, extend the reference. `README.md`: nothing user-facing changes; add the ops note that the commit-date walk streams (so a huge repo no longer needs the whole `git log` in RAM) and that the per-source date dict is still held for the run (A3).
## Completion Criteria
- [ ] `_parse_commit_dates` accepts `str | Iterable[str]` with byte-identical parse semantics; the existing string-based unit pins are green with no edits.
- [ ] `stream_git_lines` exists, drains stderr on a thread, raises the `run_git`-shaped `GitSyncError`; `file_commit_dates` streams; the module docstring's A11 claim names both sites and the grep pin enforces it.
- [ ] The lazy-vs-buffered memory comparison and the one-shot generator pins are green with their measured numbers in the test docstrings.
- [ ] `uv run pytest tests/integration/test_git_file_dates.py tests/unit/test_git_sync.py -v` green (date parity across all four cases, both fail-soft arms unchanged).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (verified in the final pass).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(sync): stream the git commit-date walk instead of buffering git log in RAM`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — stream, do not shrink the argv.** No `--since`, no `--max-count`, no per-file `git log -1`: every one of those changes WHICH dates the KB stores, and the phase-106 D2 last-commit-date contract is a data contract, not a performance knob.
- **A2 — stderr on a worker thread.** The deadlock-free way to stream stdout while still reporting git's stderr; `stderr=STDOUT` would corrupt the parse, `stderr=DEVNULL` would lose the error text `GitSyncError` carries today.
- **A3 — the retained `doc_dates_by_root` dict is out of scope.** Bounding it (by age, by working-tree membership, or by streaming it to a temp file) changes document dates or adds a new subsystem; this phase removes the transient whole-`git log` copy, which is the unbounded piece.
- **A4 — `run_git` keeps its shape.** Every other caller (`clone_or_pull`, `_unshallow_if_shallow`, `app/core/docs_push.py`) keeps the buffered site; migrating them is churn without a measured cost.
@@ -0,0 +1,26 @@
# Task 01 — The parser takes a stream (semantics untouched)
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Widen `_parse_commit_dates` to accept any iterable of lines (keeping `str` working so every existing pin is untouched), and pin in tests that it consumes its input exactly once and that streaming allocates materially less than the buffered path for the same corpus.
## Work
1. `scripts/git_sync.py` — `_parse_commit_dates` (**L240**): signature `def _parse_commit_dates(lines: str | Iterable[str]) -> dict[str, datetime]:` with the single line `if isinstance(lines, str): lines = lines.splitlines()` at the top; rename the loop variable to match; `from collections.abc import Iterable` added. Every parsing rule stays byte-identical: the `@@`-prefixed `%cI` header (`datetime.fromisoformat(line[2:])`), the blank-line skip, `line.split()` for multiple paths per line, `raw_path.replace("\\", "/").lstrip("/")`, `dates.setdefault(path, commit)` (newest-first, first sighting wins), and BOTH `ValueError` arms (a malformed date, a path line before any commit header). The docstring gains: "*lines* may be a string or any iterable of lines (phase 186 — `stream_git_lines` feeds it a live pipe); a string input behaves exactly as before."
2. `tests/unit/test_git_sync.py` — extend (existing string-based tests untouched):
- `test_parser_accepts_a_line_iterable`: the same corpus the existing tests use, fed as `iter([...])`, produces the identical dict.
- `test_parser_consumes_exactly_once`: a one-shot generator wrapper that raises `AssertionError("re-iterated")` if `iter()` is called twice on it; parse succeeds → proves the parser does not materialise-then-rescan (and rules out an accidental `list(lines)` inside the loop).
- `test_streamed_parse_alloculates_less_than_buffered`: build ONE synthetic corpus of ~300,000 lines in the exact real shape (3,000 commits × 100 paths, `@@2026-01-01T00:00:00+00:00` headers), then measure `tracemalloc` peak over (a) `_parse_commit_dates(one_big_str)` (the buffered path's own `splitlines()`) and (b) `_parse_commit_dates(iter_of_lines_from_a_generator)` (each line built lazily from a fixed seed), asserting `streamed_peak < buffered_peak * 0.6` with both numbers in the assertion message. Record both numbers + the machine in the docstring; if the real ratio on this box differs, keep the 0.6 factor only if the pre/post split genuinely clears it and document the measured ratio (never weaken to `<= buffered_peak`, which is a tautology).
- Corpus justification in the module docstring (not a test): the incident box's measured sizes — litellm 54,502 commits / 346,829 lines / 15 MB; llama.cpp 69,922 lines; vllm-radiance 4,855; gufo 9,155 — which is why the synthetic corpus is ~300k lines.
3. Verify: `uv run pytest tests/unit/test_git_sync.py -v`; `uv run pytest` green.
## Testing & Quality
- Unit: the three new pins + every existing pin green unedited.
- Coverage: **>90%** on `app/` and `scripts/git_sync.py`'s parser stays fully covered.
## Completion Criteria
- [ ] `_parse_commit_dates` accepts `str | Iterable[str]`; no parsing rule changed; the docstring says so.
- [ ] The one-shot-generator pin is green (no re-iteration, no hidden `list()`).
- [ ] The streamed-vs-buffered memory comparison is green with both measured peaks recorded in the docstring.
- [ ] `uv run pytest tests/unit/test_git_sync.py -v` and `uv run pytest` green.
@@ -0,0 +1,37 @@
# Task 02 — The fix: `stream_git_lines` + `file_commit_dates` off the buffer
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Add the streaming git site and put the commit-date walk on it, so no `git log` output is ever held as one string — with `run_git`'s error contract, the phase-106 D2 fail-soft result, and the A11 single-site rule kept intact (as two documented siblings).
## Work
1. `scripts/git_sync.py` — new function next to `run_git` (L320):
```python
def stream_git_lines(argv: list[str], cwd: Path) -> Iterator[str]:
```
- `proc = subprocess.Popen(argv, cwd=cwd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=None)` (the same env semantics as `run_git`: `None` inherits — do NOT introduce the env-replacement trap the module docstring warns about).
- stderr drained by a daemon thread appending to a list, started before stdout is consumed (A2: the deadlock-free shape — never `communicate()` after iterating stdout, never `stderr=STDOUT`, never `stderr=DEVNULL`).
- `for line in proc.stdout or (): yield line` (guard the `text=True` optional type for pyright), then `proc.stdout.close()`, `err_thread.join()`, `rc = proc.wait()`; `if rc != 0: raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {rc}): " + "".join(stderr_lines).strip())` — the message shape copied from `run_git` L344–347 so every caller's fail-soft text is unchanged.
- `finally`-safe: if the generator is closed early (`GeneratorExit`), terminate the process (`proc.terminate()` guarded by try/except) so a partial consume never leaks a `git` child.
- Module docstring: amend the **L31** claim → `run_git` (buffered) and `stream_git_lines` (streaming, phase 186) are the ONLY two places a git subprocess is spawned; both own the argv→`GitSyncError` contract; both are reached from async code only through `asyncio.to_thread` (phase 179 / AGENTS rule 11).
2. `file_commit_dates` (**L270**): `output = run_git([...])` (L298) → `return _parse_commit_dates(stream_git_lines(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest))` inside the existing `try`, keeping BOTH `except` arms and their exact warning strings. Add the docstring notes: the walk streams (no whole-`git log` in RAM — the incident's transient peak); a mid-stream failure still yields `{}` + the mtime fallback (the D2 contract is about the RESULT); and the returned per-source dict stays retained by `_run_sync` for the whole run — **A3, out of scope, named**.
3. `tests/unit/test_git_sync.py` — add:
- `test_stream_git_lines_yields_lines_and_raises_run_git_shaped_error` — run it against a real throwaway git repo (the house `_git` helper): success path yields the expected lines; `git log` in a non-repo dir raises `GitSyncError` whose message matches `^git log failed \(exit \d+\): ` (the `run_git` shape).
- `test_stream_git_lines_reports_stderr_without_deadlocking` — a command that writes >100 KB to stderr (e.g. `git -c core.quotepath=bogus …` is contrived; use `git log` in a repo with a corrupt arg that spews usage, or `git` with `--help` on stderr via a bad flag) and assert the call raises with a non-empty stderr excerpt and completes in a bounded time (a wall-clock guard, e.g. `< 10 s`).
- `test_only_two_git_spawn_sites` — read `scripts/git_sync.py` and assert `subprocess.` occurrences are exactly inside `run_git` and `stream_git_lines` (the A11 grep pin), and that no other module in `app/` or `scripts/` spawns `subprocess` for git (`grep`-style assertion over the repo; if the docs-push path legitimately uses `run_git`, it passes by construction).
- Keep every existing `run_git`/argv pin green (the `git log` argv is unchanged — A1).
4. `tests/integration/test_git_file_dates.py` — extend with the date-parity fixture (a generated repo, `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` pinned per commit): file only in commit 1 → commit 1's date; file modified in commit 2 → commit 2's date; a file in a subdirectory → its own last-commit date; a file that exists but was never committed → `{}` entry, so the importer's mtime fallback applies (assert the empty lookup, not the resulting date, to keep this test about `file_commit_dates`). Also `test_file_commit_dates_fail_soft_unchanged`: a non-directory and a non-repo directory both return `{}` with the existing warning lines (wording pinned).
5. Verify: `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py tests/integration/test_import_docs_git.py tests/integration/test_sync_api.py -v`; then `uv run pytest`.
## Testing & Quality
- Unit: the streaming site (success, error shape, stderr safety, spawn-site grep).
- Integration: date parity + both fail-soft arms + the phase-179 `app.api.sync.file_commit_dates` monkeypatch point still exercised by `test_sync_git_freeze.py`.
- Coverage: **>90%** on `app/`; `scripts/git_sync.py` fully covered by the unit family.
## Completion Criteria
- [ ] `stream_git_lines` exists with the stderr-thread + terminate-on-close shape and the `run_git`-shaped `GitSyncError`; the module docstring names both spawn sites and the grep pin enforces it.
- [ ] `file_commit_dates` streams, returns byte-identical dicts for the parity fixture, and keeps both fail-soft arms and warnings.
- [ ] `uv run pytest tests/unit/test_git_sync.py tests/integration/test_git_file_dates.py -v` green.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
@@ -0,0 +1,34 @@
# Task 03 — E2E: the dates the owner reads come through the stream (the phase's dedicated Playwright suite)
**Phase:** `186_git_log_streaming_dates` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Pin the phase-106 D2 promise through the browser with the streaming walk: after syncing a real two-commit `file://` repo, the dates shown for each document are that file's LAST-COMMIT dates (not the sync date, not the clone date), uncommitted files fall back to mtime, and a manual date correction survives a re-sync. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_git_log_stream_dates.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped), the `tests/e2e/test_git_source_dates.py` + `test_document_dates.py` shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, `BOR_GIT_SOURCES=file://<fixture repo>`, the shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`, the `_git` helper with `GIT_AUTHOR_DATE`/`GIT_COMMITTER_DATE` set per commit so dates are exact and stable).
- Fixture repo: commit A (dated 2026-01-05) adds `notes/old.md` + `notes/changed.md`; commit B (dated 2026-03-20) modifies `notes/changed.md` only; plus an untracked `notes/untracked.md` written directly into the CHECKOUT after cloning (or, if the sync re-clones, into the source dir the run walks — whichever the house fixture model uses; state which in the docstring).
- **Test 1 `test_document_dates_are_the_last_commit_dates`:** sync from the Sources page → `notes/old.md`'s document date is 2026-01-05 and `notes/changed.md`'s is 2026-03-20 (assert via `GET /api/docs/{id}` through `page.request` and the document page's date surface), neither equal to the run's own timestamp.
- **Test 2 `test_uncommitted_file_falls_back_to_mtime`:** the untracked file's document date equals its file mtime (the fail-soft contract through the streaming path) and the run reaches `success` with no error banner.
- **Test 3 `test_manual_date_survives_a_resync` (the D1 lock):** set a document's date through the existing date-edit surface, re-sync, and the manual date is unchanged while the other documents refresh from the stream.
- **Test 4 `test_resync_is_unchanged_and_dates_hold`:** a second sync with nothing changed reports all `unchanged` and identical dates (the stream re-parses, it does not drift).
- Non-ASCII/quoted path behaviour: record what the walk does TODAY with a `core.quotepath`-quoted path in the module docstring and pin it as UNCHANGED behaviour (whatever the run shows — this phase changes transport, not parsing; if the behaviour is wrong, note it for a future phase rather than fixing it here).
2. **Docs pass:** `README.md` + `AGENTS.md` in full, every inconsistency fixed (pre-existing included). `AGENTS.md` rule 11 gains the one-clause reference to `stream_git_lines` (both sites off-loop via `asyncio.to_thread`); `README.md` gains the ops note (the commit-date walk streams; the per-source date dict is still held for the run — phase 186 A3).
3. **The full gates:**
- `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_git_source_dates.py -v --no-cov` and `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` — green in isolation (the date families this phase must not disturb).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `perf(sync): stream the git commit-date walk instead of buffering git log in RAM` — covering `scripts/git_sync.py`, the unit + integration + E2E files, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/186_git_log_streaming_dates/`.
## Testing & Quality
- E2E: the dedicated suite + the two date E2E families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_git_log_stream_dates.py -v --no-cov` green **in isolation** (last-commit dates, mtime fallback, the D1 manual lock, the unchanged re-sync).
- [ ] `tests/e2e/test_git_source_dates.py` and `tests/e2e/test_document_dates.py` green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the rule-11 clause included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,54 @@
# Phase 187 — Make the sync's memory a logged, visible number
**Source:** owner incident 2026-10-15 — reconstructing "how much RAM did the sync actually use?" meant reading `systemd` unit-teardown lines from three boots (`brain-of-code-app.service: … 3.3G memory peak`), and boot −1 (the run that killed the box) had NO teardown record at all because the VM never shut down cleanly.
**Story:** n/a (incident-derived, owner directive 2026-10-15)
**Depends on:** none functionally; lands after 183–186 so the numbers it reports describe the fixed sync.
**Context:** `.agents/PLAN.md` §9 Observability (the log-format authority — PLAN is never edited by an agent, so the new fields are documented in `README.md` + `AGENTS.md` rule 10 and the owner reconciles PLAN); `AGENTS.md` rule 10 ("Amply log, visibly feed back"); `app/core/response_budget.py` (the house precedent this phase mirrors: a pure-stdlib `app/core/` module, one greppable logger name — `grep ' :: api '`, `BOR_LOG_LARGE_RESPONSE_BYTES`, and `scripts/api_payload_audit.py` as the offline audit); `app/api/sync.py` — the `SyncStatus` dataclass (**L276**, phase-64 "keep the final counts in the terminal" convention), `sync_status()` (**L385**, its additive-keys contract "Additive only — every pre-existing key keeps its name, type, and meaning"), the `sync: started repos=%d origin=%s git=%d local=%d` line (**L591–594**), the phase transitions (`_status.phase = "import"` **L718**, `"overview"` **L743**, `"summaries"`), `sync: done detail=%s` (**L903**), the cancel terminal's `sync: cancelled by user state=cancelled files_done=%d files_total=%d` (**L933–936**), the failed terminal (`logger.exception("sync: failed")` **L938**); `app/rag/importer.py` — `ImportSummary.format_counts` (**L513**) + `ImportSummary.log()` (**L520**, the cron-safe greppable summary line PLAN §9 names); `frontend/assets/sources.js` — the sync progress box + result line (`setSyncProgress(…)` around **L1325**, the label/counts helper at **L1080–1100** and **L1378**, the aria-live announcements), `frontend/index.html` (the progress box markup), `tests/unit/test_sync_api*.py` / `tests/integration/test_sync_api.py` (the status-shape pins), `tests/e2e/test_sync_summary_visibility.py`, `tests/e2e/test_sync_progress_box.py`, `tests/e2e/test_sync_status_freshness.py`, `tests/e2e/test_sync_cancel.py`.
## Objective
Make "how much memory did that sync use?" answerable from one log line and visible on the page that started it: a small `app/core/memory.py` reading current + peak RSS, appended to every sync lifecycle line and the importer's summary line, plus an additive `rss_peak_mb` key on `GET /api/sync/status` and a `peak <N> MB` clause on the Sources page's sync result line.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — `app/core/memory.py` never raises and never lies.** Three pure helpers: `rss_mb() -> int | None` (Linux: parse `VmRSS` from `/proc/self/status`; non-Linux/failure: `resource.getrusage(RUSAGE_SELF).ru_maxrss // 1024` as the best available proxy), `rss_peak_mb() -> int | None` (`ru_maxrss // 1024` — the process high-water mark), and `cgroup_peak_mb() -> int | None` (read `/sys/fs/cgroup/memory.peak` when present, else `None`). Docstring states the honest nuance the incident turned on: **inside a container the process high-water and the host's cgroup `memory.peak` are different numbers, and the cgroup number includes page cache** (the incident box's app cgroup showed 2.16 GB of `file` against 211 MB of `anon` while idle) — so the app logs the PROCESS number and the operator's dashboard may show a larger one; neither is wrong, and the log says which it is.
- **D2 — the format is append-only, in the §9 style (task 02).** `sync: started … rss_mb=N`, and every terminal (`sync: done detail=… rss_mb=N rss_peak_mb=M`, `sync: cancelled by user … rss_mb=N rss_peak_mb=M`, and the failed funnel) gains the same two trailing fields; `ImportSummary.log()` gains a trailing ` rss_mb=N`. A `None` renders as `rss_mb=?` — a helper `fmt_mb(v: int | None) -> str` keeps that in one place. No existing field is renamed, reordered or re-typed (the phase-167/182 append-only convention), so every existing `grep` keeps working.
- **D3 — sampled, not polled per file (task 03).** `_status` gains `rss_mb: int | None` and `rss_peak_mb: int | None`, reset at run start; sampled at the run start, at each `_status.phase` transition (`import` L718, `overview` L743, `summaries`), and in EVERY terminal (which keep the final value next to the counts — the phase-64 convention). The per-file progress hook samples at most once per 5 s via a `time.monotonic()` guard, so a 27k-file walk costs ~600 `/proc` reads, not 27k, and the operator watching the page sees the number move (PLAN §7.4's never-stale contract).
- **D4 — the status key and the UI clause are additive and admin-only (task 03).** `sync_status()` returns two new keys (`rss_mb`, `rss_peak_mb`, `int | null`); `sources.js` appends ` · peak 512 MB` to the existing result/counts line — a clause on text that already exists, never a new control, never replacing a state string, and the aria-live announcement keeps its current wording (a screen-reader user gets the same announcement plus the clause; the peak is supplementary, so it must not become the announcement's only content). The Sources view is admin-only (PLAN B5), which is the exposure guard; no memory number appears on any anonymous surface (chat, shared chat, health).
- **D5 — an offline reader for the log (task 02, small).** Document `grep ' :: ' | grep ' rss_peak_mb='` in `README.md`'s operations section. NO new audit script: `scripts/api_payload_audit.py` exists for bytes; a memory equivalent would be a second parser for one number. Explicitly out of scope (A3).
- **NOT touched:** `PLAN.md` (never), the sync pipeline's sequence/errors/cancel semantics, the response-budget logger, the chat's per-turn line (this phase is about the sync), the container/quadlet memory limits (owner's own task), the retrieval path.
## Dependencies
— (none; after 183–186 in the roadmap so the reported numbers describe the bounded sync)
## Tasks
1. `01_memory_helper.md` — `app/core/memory.py` (the three helpers + `fmt_mb`), unit-tested with `/proc` and `resource` stubbed (Linux path, missing-file path, garbage-file path, non-Linux fallback).
2. `02_sync_log_lines.md` — the four sync lifecycle lines + `ImportSummary.log()` gain `rss_mb=` / `rss_peak_mb=`, the `fmt_mb` single-sourcing, the log-format assertions in the integration tests, the README operations note.
3. `03_status_key_and_ui.md` — the `SyncStatus` fields + sampling (start, phase transitions, terminals, the 5 s-throttled progress sample), the two additive `sync_status()` keys, the `sources.js` clause + aria-live behaviour, the shape pins.
4. `04_e2e_peak_memory.md` — `tests/e2e/test_sync_peak_memory_visible.py` (the phase's dedicated Playwright suite), the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_core_memory.py` — every helper branch (real `/proc` on Linux + stubbed failure modes, the `None` rendering, the kB→MB arithmetic), `tests/unit/test_settings*` untouched.
- **Integration:** `tests/integration/test_sync_api.py` — the lifecycle lines carry the fields (caplog assertions on the `sync: started` / `sync: done` / `sync: cancelled` / `sync: failed` shapes with `rss_mb=?` accepted on a platform without `/proc`), the status payload's two new keys with their types and the terminal-keeps-final-value convention, every pre-existing key byte-identical.
- **E2E (AGENTS rule 4, task 04):** `tests/e2e/test_sync_peak_memory_visible.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `AGENTS.md` rule 10: the required sync lines now carry `rss_mb=` / `rss_peak_mb=` (extend rule 10's parenthetical, do not rewrite it). `README.md`: the operations grep, what each number means, and the container-vs-cgroup caveat (page cache included in the host's number) written in one short paragraph. PLAN §9 is the owner's to reconcile — flag it in the phase report, do not edit it.
## Completion Criteria
- [ ] `app/core/memory.py` exists with the four documented functions, every branch unit-tested, and never raises.
- [ ] All four sync lifecycle lines + `import: summary …` carry `rss_mb=` (and the terminals `rss_peak_mb=`), appended after the existing fields; no existing field changed.
- [ ] `GET /api/sync/status` exposes `rss_mb` / `rss_peak_mb` (`int | null`), keeps the final value in every terminal, and every pre-existing key is unchanged.
- [ ] The Sources page's sync result line shows the ` · peak <N> MB` clause with no new control and no regression in the existing state strings.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the PLAN §9 reconciliation flagged, not performed).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(observability): log and show the sync's RSS so memory is a number, not a forensic exercise`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — the app logs PROCESS RSS, not the cgroup number**, with the caveat documented (`anon` vs `file` differ by an order of magnitude and the cgroup figure includes page cache — demonstrated during the incident investigation).
- **A2 — append-only to the existing lines and keys.** No new log line type, no renamed field, no separate metrics endpoint (A12: no aux services), no Prometheus.
- **A3 — no memory audit script** this phase; the grep is documented instead.
- **A4 — sampling is throttled (5 s) and never per-file**, and the UI clause is text on a line that already exists on an admin-only view.
@@ -0,0 +1,26 @@
# Task 01 — `app/core/memory.py`
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
One pure-stdlib module that answers "how much RAM is this process using, and what was its high-water mark" without ever raising, mirroring `app/core/response_budget.py`'s shape (no FastAPI, no imports from `app.api`, logger only if needed).
## Work
1. `app/core/memory.py`:
- `rss_mb() -> int | None` — parse `VmRSS` from `/proc/self/status` (kB → MB, integer, floor); any `OSError`/`KeyError`/parse failure returns `None`. Non-Linux fallback: `resource.getrusage(resource.RUSAGE_SELF).ru_maxrss // 1024` (documented as a high-water proxy, not a current value).
- `rss_peak_mb() -> int | None` — `resource.getrusage(RUSAGE_SELF).ru_maxrss // 1024`, `None` only if `resource` is unavailable (wrap in a guarded import for platforms without it).
- `cgroup_peak_mb() -> int | None` — read `/sys/fs/cgroup/memory.peak` (bytes → MB) when it exists and parses, else `None`. Docstring: inside a container this file is usually absent (the host reads it), and the host's number counts PAGE CACHE too — the incident box's app cgroup showed ~2.16 GB `file` against ~211 MB `anon` while idle, and `systemd` reported `3.3G memory peak` for the unit while the uvicorn process was a fraction of that. This module therefore reports PROCESS numbers and this distinction is written here once, so no future reader conflates them.
- `fmt_mb(v: int | None) -> str` — `"?"` for `None`, else `str(v)` (so log lines never print `rss_mb=None`).
- Module docstring: the incident (one paragraph), the three helpers, the never-raise contract, and the container/cgroup caveat above.
2. `tests/unit/test_core_memory.py` — every branch, with the filesystem and `resource` stubbed: a real read (Linux runner: `/proc/self/status` has `VmRSS`, so `rss_mb()` is a positive int), a missing file, an unparseable file, a file without a `VmRSS` line, the non-Linux `resource` fallback path (patch the `/proc` open to raise + patch `getrusage`), `rss_peak_mb()`'s arithmetic, `cgroup_peak_mb()` present/absent/garbage, `fmt_mb(None) == "?"` / `fmt_mb(0) == "0"` (0 is a value, not missing).
3. Verify: `uv run pytest tests/unit/test_core_memory.py -v`; `uv run ruff check . && uv run pyright` clean; `uv run pytest` green.
## Testing & Quality
- Unit: 100% branch coverage of the new module (well above the >90% gate).
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `app/core/memory.py` with the four functions, the never-raise contract, and the container/cgroup caveat documented.
- [ ] `uv run pytest tests/unit/test_core_memory.py -v` green, every branch covered.
- [ ] No import cycle (`app.core.memory` imports stdlib only — prove it in the module docstring's "pure stdlib" line and by the absence of `app.` imports).
@@ -0,0 +1,28 @@
# Task 02 — The lifecycle lines carry the numbers
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Append `rss_mb=` to the sync's start line, `rss_mb=`/`rss_peak_mb=` to all three terminals, and `rss_mb=` to the importer's cron-safe summary line — append-only, so every existing grep and every existing field keeps working.
## Work
1. `app/api/sync.py` — the four lifecycle log sites, using `app.core.memory.rss_mb/rss_peak_mb/fmt_mb` (imported at module level, matching the file's import style):
- **L591–594** `sync: started repos=%d origin=%s git=%d local=%d` → append ` rss_mb=%s` (`fmt_mb(rss_mb())`).
- **L903** `sync: done detail=%s` → append ` rss_mb=%s rss_peak_mb=%s`.
- **L933–936** `sync: cancelled by user state=cancelled files_done=%d files_total=%d` → append the same two fields.
- **L938** the failed funnel: `logger.exception("sync: failed")` stays as-is (the traceback), and the FAILED line gains a dedicated `logger.error("sync: failed state=failed files_done=%d files_total=%d rss_mb=%s rss_peak_mb=%s error=%s", …)` immediately after the `_status.error = _sanitize_error(str(e))` assignment (sanitized exactly like the status field — AGENTS/security: the raw exception string never reaches the log un-sanitized). Keep `logger.exception` so the traceback survives.
2. `app/rag/importer.py` — `ImportSummary.log()` (**L520**): append ` rss_mb=%s` to the existing `import: summary …` line (the `format_counts()` helper at L513 stays untouched — the memory field belongs to the LOG line, not the counts formatter, so the CLI's other consumers of `format_counts()` are byte-identical).
3. `tests/integration/test_sync_api.py` — caplog assertions (the house caplog pattern): after a run to `success`, the captured records include a `sync: started … rss_mb=` line and a `sync: done … rss_mb=… rss_peak_mb=…` line whose values are `?` or a positive integer; the same for the cancelled terminal (reuse the existing cancel test's fixture) and the failed terminal (reuse the existing failure test — `sync: model check failed` or a stubbed raise). Assert the PRE-existing prefixes are unchanged (`line.startswith("sync: started repos=")` etc.) — the append-only pin.
4. `tests/integration/test_importer_e2e.py` (or the importer family's log-pin home): the `import: summary …` line ends with ` rss_mb=<int|?>` and its earlier fields are unchanged.
5. `README.md` — the operations grep (`grep ' :: ' … | grep ' rss_peak_mb='`), the field meanings, and the process-vs-cgroup caveat in one short paragraph (D1/A1). `AGENTS.md` rule 10: extend the parenthetical to mention the sync's new fields (do not rewrite the rule).
6. Verify: `uv run pytest tests/integration/test_sync_api.py tests/integration/test_importer_e2e.py -v`; `uv run pytest`; ruff + pyright.
## Testing & Quality
- Integration: the four lifecycle terminals + the importer summary line pin the fields and the unchanged prefixes.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] All four sync lifecycle lines and `import: summary …` carry the new fields, appended after the existing ones; the failed terminal gains the sanitized one-line summary while `logger.exception` still writes the traceback.
- [ ] `uv run pytest tests/integration/test_sync_api.py -v` green (fields present, pre-existing prefixes byte-identical).
- [ ] `README.md` + `AGENTS.md` rule 10 updated; `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,30 @@
# Task 03 — The status key and the Sources-page clause
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Put the number where the operator is: two additive keys on `GET /api/sync/status` (sampled at the run start, each phase transition, every terminal, and at most every 5 s during the walk) and a ` · peak <N> MB` clause on the Sources page's existing sync result line.
## Work
1. `app/api/sync.py`:
- `SyncStatus` (**L276**): two new fields `rss_mb: int | None = None`, `rss_peak_mb: int | None = None`, documented as phase-187 additive state with the phase-64 "terminals keep the final value" convention and the run-start reset.
- A single private sampler on the module: `_sample_rss(force: bool = False) -> None` — reads `app.core.memory.rss_mb()` / `rss_peak_mb()`, stores both, keeps a `time.monotonic()` guard so a non-forced call is a no-op within 5 s (D3/A4); forced at the run start, each `_status.phase = "import"|"overview"|"summaries"` transition (**L718** and friends), and every terminal (success L903 path, cancelled L933 path, failed L938 path) so the final value is exact.
- `sync_status()` (**L385**): append `"rss_mb"` and `"rss_peak_mb"` to the dict (the file's own words: "Additive only — every pre-existing key keeps its name, type, and meaning") and document the two keys + the terminal convention in its docstring.
- The per-file `_hook` (L677) calls `_sample_rss()` (unforced → throttled).
2. `frontend/assets/sources.js`: on the terminal render path, append the clause to the EXISTING result/counts text — ` · peak 512 MB` — sourced from `status.rss_peak_mb ?? status.rss_mb`, and render NOTHING when both are null (never `peak ? MB`). Touch the result-line helper used at **L1080–1100** / **L1378** and the terminal/announce path around **L1314–1330**; the existing state strings ("Sync cancelled", the counts line, the error copy) keep their exact wording with the clause appended, and the aria-live announcement keeps announcing what it announces today plus the clause (no new announcement). No new DOM control, no layout shift on the 375px viewport (the clause wraps inside the existing line's `.cell-clip`-free text; verify visually in the E2E at mobile width).
3. Pins:
- `tests/integration/test_sync_api.py`: the payload carries both keys with `int | None`; after a run the terminal payload's `rss_peak_mb` is non-null on Linux and `>= rss_mb` when both are set; idle (never-run) reports `null`/`null`; every pre-existing key/value shape asserted unchanged (the additive pin); anonymous `GET /api/sync/status` still 403 (no new anonymous surface — A4).
- `tests/unit/test_sync_status_shape.py` (or this file's existing home for the status-shape matrix): the key set is exactly the old set ∪ {`rss_mb`, `rss_peak_mb`}.
- `frontend`: the clause appears in the terminal line and NOT in `running`; a null peak renders no clause.
4. Verify: `uv run pytest tests/integration/test_sync_api.py -v`; the three sync E2E neighbours in isolation (`test_sync_summary_visibility.py`, `test_sync_progress_box.py`, `test_sync_status_freshness.py`) still green (they assert the existing strings — the clause must not break them; if one asserts an exact full-line equality, update it to the clause-appended form and say so in the task note); `uv run pytest`; ruff + pyright.
## Testing & Quality
- Integration: the shape + additive + admin-gating pins above.
- Coverage: **>90%** on `app/`.
## Completion Criteria
- [ ] `SyncStatus` + `sync_status()` carry the two additive keys; sampling happens at start / phase transitions / every terminal and is throttled to 5 s in the per-file hook.
- [ ] The Sources page shows ` · peak <N> MB` on the terminal line, renders nothing when the value is null, and adds no new control.
- [ ] The three existing sync E2E families are green (any exact-string update documented).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
@@ -0,0 +1,34 @@
# Task 04 — E2E: the operator sees the peak (the phase's dedicated Playwright suite)
**Phase:** `187_sync_peak_memory_line` · **Source:** owner incident 2026-10-15
**Story:** n/a
## Objective
Prove through the browser that the number is there and honest: a real sync from the Sources page ends with a `peak <N> MB` clause on the result line, the status payload's `rss_peak_mb` is a positive integer that survives the terminal, the running state shows no clause, and nothing anonymous can read it. Then the full gates, the docs pass, the atomic commit.
## Work
1. `tests/e2e/test_sync_peak_memory_visible.py` — a NEW module, mock-only, the `tests/e2e/test_sync_progress_box.py` / `test_sync_summary_visibility.py` house shapes (dedicated module app + port, mock LLM, dedicated `BOR_SOURCES_DIR`, shared E2E Postgres truncated before AND after each test, admin login via `tests/e2e/auth_helpers.py`).
- **Test 1 `test_result_line_shows_the_peak`:** sync a small fixture → after the `success` terminal, the progress box's result line matches `/peak \d+ MB/`, the existing wording ("Sync complete"/the counts line) is still present, and the aria-live region contains the same clause.
- **Test 2 `test_status_payload_reports_rss`:** during `running` and after the terminal, `GET /api/sync/status` via `page.request` → `rss_peak_mb` is a positive int in the terminal, `>= rss_mb` when both are set, and pre-existing keys are unchanged (the additive pin end to end). During `running` the result line shows NO `peak` clause (the clause is terminal-only in the UI).
- **Test 3 `test_anonymous_cannot_read_memory`:** logged-out `GET /api/sync/status` → 403 (the A4 exposure guard, unchanged).
- **Test 4 `test_cancelled_terminal_keeps_the_peak`:** cancel a run (the `test_sync_cancel.py` shape) → the neutral "Sync cancelled" state still renders, with the peak clause present (terminals keep the value).
- **Mobile width:** assert the clause at 375 px does not overflow the progress box (the PLAN §7 container rule).
- Module docstring: the incident reference, the fields + the process-vs-cgroup caveat pointer to `README.md`, the test→observable mapping, the run-in-isolation command, prerequisites (`podman compose up -d db`).
2. **Docs pass:** `README.md` + `AGENTS.md` in full against the post-phase app, every inconsistency fixed (pre-existing included); confirm rule 10's extension and the operations grep landed in task 02; flag the PLAN §9 reconciliation in the phase report (never edit PLAN.md).
3. **The full gates:**
- `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` — green **in isolation**.
- `uv run pytest tests/e2e/test_sync_progress_box.py -v --no-cov`, `tests/e2e/test_sync_status_freshness.py -v --no-cov`, `tests/e2e/test_sync_cancel.py -v --no-cov` — green in isolation (the surfaces the clause touches).
- `uv run pytest --cov=app --cov-report=term-missing` — full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
4. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit — e.g. `feat(observability): log and show the sync's RSS so memory is a number, not a forensic exercise` — covering `app/core/memory.py`, `app/api/sync.py`, `app/rag/importer.py`, `frontend/assets/sources.js`, the tests, the docs and `.agents/`. Move the phase dir to `.agents/phases/complete/187_sync_peak_memory_line/`.
## Testing & Quality
- E2E: the dedicated suite + the three sync UI families in isolation.
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_peak_memory_visible.py -v --no-cov` green **in isolation** (clause on both terminals, absent while running, payload types, anonymous 403, mobile width clean).
- [ ] The three regression E2E families green in isolation.
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; ruff + pyright clean.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project; the PLAN §9 reconciliation flagged for the owner.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,62 @@
# Phase 188 — The vector search stops full-scanning 414,220 embeddings
**Source:** found while investigating the 2026-10-15 memory incident — `pg_indexes` for `chunks` lists `chunks_pkey`, `ix_chunks_document_id`, `ix_chunks_tsv` (GIN) and **no index on `embedding`**, so every chat turn runs `ORDER BY embedding <=> :vec LIMIT 100` (A7's cosine tier) as a sequential scan of a 3 GB relation: per-turn CPU on the interactive path, plus 3 GB of page-cache churn on a 3,904 MB box that the sync also needs.
**Story:** n/a (incident-adjacent, owner directive 2026-10-15)
**Depends on:** none functionally; LAST of the six incident phases — it is the only one that touches the schema and the retrieval quality contract, so it lands after the sync's memory work is in.
**Context:** `.agents/PLAN.md` §2 — **A3** (PostgreSQL 17 + pgvector, cosine `<=>`, one system), **A6** (dim **768**, fixed at table creation, a mismatch fails loudly), **A7** (hybrid: **cosine top-100** ∪ FTS top-30 → RRF k=60 → full text of the top-N=2 documents), **A8** (the honesty gate: deflect only when best cosine < `BOR_RELEVANCE_THRESHOLD` 0.62 AND (zero FTS hits OR best cosine < `BOR_LEXICAL_SUPPORT_FLOOR` 0.35)), **A13** (every migration ships a TESTED downgrade), **A16** (per-phase unit + integration + >90% + one dedicated E2E file), **A21** (LLM calls are free; the interactive path's latency is what matters); `app/models.py` — `Chunk.embedding: Mapped[list[float] | None] = mapped_column(Vector(EMBEDDING_DIM))` (**L241**); `app/rag/retriever.py` — **`_vector_candidates(db, question_embedding, limit)` (L431)** building `distance = Chunk.embedding.cosine_distance(question_embedding)` (**L440**) `.where(Chunk.embedding.is_not(None))` (**L444**) `.order_by(distance).limit(limit)` (**L446**), the fusion caller at **L690–711** (`vector = _vector_candidates(db, question_embedding, v_n)`), `_lexical_candidates` (L632); `app/config.py` (the `BOR_` `Settings` pattern, `relevance_threshold` / `lexical_support_floor` / `suggested_docs` as the retrieval-knob neighbours); `app/core/settings_catalog.py` — `GROUPS` (**L72**, the `"Retrieval & honesty"` group) + the `set(CATALOG) == set(Settings.model_fields)` completeness pin; `alembic/versions/` — head is **`0028`** (`uv run alembic heads`), so this phase adds **`0029`**; `db/Containerfile` — **`ARG PGVECTOR_VERSION=v0.8.0`** (HNSW + `halfvec` available; verified 0.8.0 on both the dev and the production instance); `scripts/eval_retrieval.py` (+ `tests/fixtures/retrieval_battery.txt`, the phase-119/162 regression battery — `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt`, `AIPI_KEY` required); `benchmarks/model_benchmarks.csv` (the recorded-measurement home); `tests/e2e/test_retrieval_quality.py`, `tests/e2e/test_search_recall_small_sources.py`, `tests/e2e/test_chat_rag.py`, `tests/unit/test_retriever.py`, `tests/unit/test_retriever_recency.py`.
## Production numbers (read 2026-10-15, read-only)
414,220 chunks, **all 414,220 with a non-null `embedding`**, `chunks` relation ≈ **3,000 MB**, database 3,147 MB, `shared_buffers=128MB`, `work_mem=4MB`, host 3,904 MB with **no swap**. pgvector **0.8.0**.
## Objective
Add an HNSW cosine index on `chunks.embedding` through a reversible migration built `CONCURRENTLY` (A3's one-system rule respected — no new service, no new dependency), give the retrieval tier an explicit `hnsw.ef_search` knob that can never be set below the A7 top-100 candidate count, and prove with a deterministic recall pin plus the live retrieval battery that the approximate index does not move the top hits, the `top_score`, or the honesty-gate verdicts.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the index is on the existing `vector(768)` column, `vector_cosine_ops` (task 01).** `CREATE INDEX CONCURRENTLY … USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 64)` — the pgvector defaults' neighbourhood, which keeps the graph buildable on a 4 GB box. `halfvec(768)` (half the index size) was considered and REJECTED for this phase (A2): it needs `embedding::halfvec(768)` in `_vector_candidates` AND a quantisation-recall study, i.e. two quality risks in one phase. The index is smaller-with-halfvec; if the prod build does not fit, that is a follow-up phase, not a silent compromise here.
- **D2 — `CONCURRENTLY` needs autocommit, and an interrupted build leaves an INVALID index (task 01).** Migration `alembic/versions/0029_chunks_embedding_hnsw.py` runs its DDL inside `with op.get_context().autocommit_block():` and, before creating, drops any existing INVALID index of that name (`select indisvalid from pg_index join pg_class …` → `DROP INDEX CONCURRENTLY IF EXISTS ix_chunks_embedding_hnsw` when `indisvalid` is false), then re-checks `indisvalid` afterwards and raises if the build did not validate — because a `CONCURRENTLY` build that dies (a 3,904 MB box is exactly where it dies) leaves an index that costs insert time and buys nothing, silently. Upgrade and downgrade both run in the autocommit block; the downgrade is `DROP INDEX CONCURRENTLY IF EXISTS …` and is TESTED (A13) by `alembic downgrade -1` / `upgrade head` in the task's verification.
- **D3 — `ef_search` is explicit and clamped (task 02).** `retrieval_ef_search: int = 100` (`BOR_RETRIEVAL_EF_SEARCH`, `"Retrieval & honesty"` catalogue group): `_vector_candidates` issues `SET LOCAL hnsw.ef_search = :ef` (per-transaction — no leakage between requests, no global GUC change; A12 keeps state where it belongs) immediately before the SELECT, and a value below the A7 candidate count is CLAMPED UP to it with one WARNING line (`retrieval: ef_search=%d raised to %d (A7 candidate count)`) rather than silently degrading recall. Negative values refuse startup the way the phase-182 `show_text_max_*` knobs do.
- **D4 — recall is pinned twice: deterministic in CI, live against the battery (tasks 03 + 04).**
- **CI (deterministic, no LLM):** seed ~5,000 chunks with fixed-seed 768-dim vectors, compute the EXACT cosine top-100 in Python (`numpy`-free is fine — a pure-Python dot product over the seeded corpus, or reuse the existing helper if `tests/` already has one), then assert the index-backed query's top-100 set overlaps the exact set by **≥ 0.99** at `ef_search=100` and that `1 - distance` (`top_score`) matches the exact value to 6 decimals for the returned rows. Plus the plan pin: `EXPLAIN (FORMAT JSON)` for the real `_vector_candidates` statement names `ix_chunks_embedding_hnsw` (and does NOT show `Seq Scan on chunks`).
- **Live (A8's honesty contract):** the retrieval battery, index OFF vs ON, on the 1,000-document live-replica KB (the `restore-test-db` skill) with `AIPI_KEY` set — every question's `top_score`, top-5 document set and deflection verdict must be identical (a difference means RAISING `ef_search`, never accepting it). The before/after numbers are recorded in `benchmarks/` and the phase report; the build's wall time and the box's free RAM before/after are recorded too (they are the deploy-runbook numbers).
- **D5 — the build is an operator step with a runbook (task 01 + README).** The migration is correct, but building HNSW over 414k × 768 on the production 3,904 MB box needs a documented pre-flight: free-RAM check, `maintenance_work_mem` set explicitly for the session (e.g. 512 MB, never left at the 4 MB `work_mem` default), the build run against the postgres container directly (`podman exec … psql`) if the app container is memory-limited (the owner's separate task), `autovacuum` left running afterwards, and the INVALID-index recovery path. `README.md` gains that runbook section; the phase's own CI build runs against the small test corpus, not 414k rows.
- **NOT touched:** the embedding dim (A6), the `<=>` operator and the hybrid fusion (A7), the thresholds themselves (A8 — unchanged unless the battery forces an `ef_search` raise), the `tsv` GIN index, `chunks`/`documents` columns, the importer/sync (phases 183–186), the frontend, any new dependency or service (A3/A12).
## Dependencies
— (none; deliberately the LAST of the six incident phases — schema + retrieval-quality risk after the sync's memory work)
## Tasks
1. `01_hnsw_migration.md` — migration `0029` (autocommit block, `CONCURRENTLY`, the INVALID-index pre-drop + post-validate), the tested downgrade, the migration tests.
2. `02_ef_search_setting.md` — `retrieval_ef_search` on `Settings` + the catalogue group + `.env.example`, `SET LOCAL` in `_vector_candidates`, the clamp-and-warn rule, the negative-refuses-startup pin.
3. `03_plan_and_recall_pins.md` — the `EXPLAIN` plan pin (index used, no seq scan) and the deterministic ≥0.99 overlap / exact-`top_score` recall pin on a fixed-seed corpus.
4. `04_recall_battery_gate.md` — the live battery index-off vs index-on on the 1,000-doc replica KB; the recorded numbers in `benchmarks/` + the phase report; the `ef_search` raise loop if anything moved.
5. `05_e2e_index_chat_and_runbook.md` — `tests/e2e/test_vector_index_recall.py` (the phase's dedicated Playwright suite), the `README.md` build runbook, the full gates, the docs pass, the atomic commit.
## Testing & Quality
- **Unit:** the `retrieval_ef_search` default/clamp/negative rules; the migration module's helpers (the INVALID-index decision function is a pure function over a query result, so it unit-tests without a DB).
- **Integration:** `tests/integration/test_chunks_hnsw_index.py` — upgrade/downgrade round-trip on the test DB, `pg_index.indisvalid` asserted, the INVALID-index recovery path exercised by creating an INVALID index deliberately (`… WHERE false` won't produce INVALID; use the documented `UPDATE pg_index SET indisvalid=false` superuser trick or a deliberately interrupted build — pick the one the test DB allows and document it), the `EXPLAIN` plan pin, the deterministic recall pin. `tests/unit/test_retriever.py` + `test_retriever_recency.py` + `tests/integration/test_chat_db_sessions.py` green with `SET LOCAL` in play.
- **Live gate:** the retrieval battery off/on, identical verdicts, recorded.
- **E2E (AGENTS rule 4, task 05):** `tests/e2e/test_vector_index_recall.py`, run **in isolation**.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- Final pass reviews `README.md` and `AGENTS.md` in their entirety against the post-phase app and fixes every inconsistency found, pre-existing ones included.
- `README.md`: the `0029` deploy section (the D5 build runbook: pre-flight free RAM, `maintenance_work_mem`, `CONCURRENTLY` semantics, the INVALID-index recovery, expected build time from task 04's measurement) and the `BOR_RETRIEVAL_EF_SEARCH` knob (default 100, the clamp, "raise it if recall matters"). `AGENTS.md`: the quick-reference gains nothing new unless a command changes — the retrieval battery line already exists; keep it accurate.
## Completion Criteria
- [ ] `alembic/versions/0029_…` creates `ix_chunks_embedding_hnsw` `CONCURRENTLY` inside an autocommit block, pre-drops an INVALID index of that name, post-validates `indisvalid`, and its downgrade is tested (`upgrade head` → `downgrade -1` → `upgrade head` green on the test DB, A13).
- [ ] `retrieval_ef_search` (default 100) is applied with `SET LOCAL` in `_vector_candidates`, clamped up with a WARNING below the A7 candidate count, negative refuses startup, and it appears in `CATALOG` (`"Retrieval & honesty"`) + `.env.example` + `README.md`.
- [ ] The `EXPLAIN` pin shows the index used and no seq scan; the deterministic recall pin shows ≥0.99 top-100 overlap and exact `top_score`s.
- [ ] The live battery ran index-off and index-on over the 1,000-doc replica with IDENTICAL top scores, top-5 sets and deflection verdicts, and the numbers (plus build time + free-RAM readings) are recorded in `benchmarks/` and the phase report.
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` green **in isolation**.
- [ ] `README.md` and `AGENTS.md` accurate and consistent with the whole project (the runbook included).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `perf(retrieval): index chunks.embedding with HNSW and pin recall so cosine stops full-scanning 414k vectors`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-15)
- **A1 — HNSW on `vector(768)` with `vector_cosine_ops`, built `CONCURRENTLY`.** No query rewrite, no new dependency, `<=>` uses it directly (A3).
- **A2 — `halfvec` quantisation is NOT part of this phase** (it needs a statement cast + its own recall study); revisit only if the production build does not fit in RAM.
- **A3 — recall is a gate, not a hope.** The honesty gate (A8) reads `top_score`; an approximate index that changes a top hit can flip a deflect. The deterministic ≥0.99 pin AND the live battery must pass, and the remedy is a higher `ef_search`, never a relaxed assertion.
- **A4 — `SET LOCAL`, per transaction.** No global GUC mutation, no connection-pool-wide state (A12: no aux services, restart clears state).
- **A5 — the production build is an operator action with a runbook**, executed after the owner's container-memory work lands; the phase's CI never builds a 414k-row graph.

Some files were not shown because too many files have changed in this diff Show More